Prev Next whats_new_2021

@(@\newcommand{\B}[1]{ {\bf #1} } \newcommand{\R}[1]{ {\rm #1} } \newcommand{\W}[1]{ \; #1 \; }@)@ This is dismod_at-20221105 documentation: Here is a link to its current documentation .
Changes and Additions to Dismod_at During 2021

mm-dd

12-31
An error message in db2csv_command was missing its assert. This has been fixed. In addition, all the sys.exit(msg) calls were changed to assert False, msg to make then easier to debug.

12-30
Improve the error message that prints the rates when an integrand can not be computed; e.g., prevalence can not be computed when S and C are zero.

12-27
The system_command_prc routine used to print the error message when return_stderr was false and file_stderr was None. Now it puts the error message in the assertion that gets generated.

12-22
Fix x-axis tick labels in plot_data_fit plots.

12-14
  1. Add the user_plot_curve.py example and fix the title in user_plot_rate_fit.py , user_plot_data_fit.py .
  2. API Change: Move rate_set to the last plot_rate_fit argument.
  3. The system_command_prc function now puts quotes around command line arguments that has spaces (when it prints a command).
  4. The plot_rate_fit and plot_data_fit routines now have a command line interface in dismodat.py.


12-09
Add the file_stdout , file_stderr , and write_command arguments to system_command_prc.

12-08
The distutils module has been deprecated. Replace all use of distutils.dir_util.mkpath with os.makedirs.

12-05
  1. Add the perturb_command . This is useful for the cascade where the mean of the prior is sometimes very close to the optimization estimate.
  2. The csv2db The modify were deprecated.


11-29
In plot_curve, add the assumption that the plot data values are non-negative and not all zero.

11-27
  1. Add the driver option to dock_dismod_at.sh so it can use either docker or podman.
  2. Advance the dock_dismod_at.sh dismod_at_version to 20211121. This requires re-running
       ./dock_dismod_at.sh image mixed
     because the version of cppad_mixed changed.


11-19
Add the plot_curve utility.

11-16
Advance to cppad_mixed-20211025. This improves the warm_start option in the fit command.

11-15
Change plot_data_fit so that it plots with respect to age and time (as well as index).

11-14
  1. The connection_file function was added.
  2. The user_average_integrand.py and user_csv2db.py examples were added to the user_example list.
  3. Improve the error messages in get_name_type .


11-07
  1. Make the plot_data_fit arguments integrand_list and plot_title optional. In addition, its return value n_fit_dict was changed from a list to a dictionary.
  2. API Change: The changes on 11-07 and 11-06 are API changes to plot_data_fit.


11-06
  1. The plot_data_fit routine would crash when one of the integrands had no data. This has been fixed.
  2. The plot_title argument was added to the plot_rate_fit and plot_data_fit functions. In addition, the max_plot argument was added to plot_data_fit.


11-05
  1. The plot_data_fit function now plots data that is held out in green .
  2. API Change: Split the out column in the data.csv into the d_out and s_out columns.


11-03
Fix bug in plot_rate_fit . To be specific, if more that one page of times (of ages) was needed to plot a rate as a function of age (time), only the last page was included in the pdf output file.

10-31
There was a bug in the calculation of max_cov_diff during an init_command . This has been fixed.

10-25
  1. Use the term standard error (not the rate name) to label standard errors plotted by plot_rate_fit .
  2. Print the plot location at the end of user_plot_rate_fit.py and user_plot_data_fit.py .


10-24
The following arguments were added to system_command_prc : print_command , return_stdout , return_stderr . The default settings for these arguments makes system_command_prc backward compatible.

10-23
Balancing child nodes was added to the hold_out command.

10-15
The bnd_mulcov_command was not affecting the limits of the covariate multipliers. This has been fixed.

10-14
Add more error detection and reporting to get_name_type and plot_rate_fit .

10-12
An error report for null covariate reference values was added. In addition, using nulls in undocumented cases was explicitly forbidden; see null .

10-11
Add the plot_rate_fit python function.

10-07
Add the plot_data_fit python function.

10-06
Add a documentation to the user_no_children.py example.

09-01
If the file name for a database in a command did not exist, most commands gave the error message:
   Could not find table or column in table.
   detected in age table
 This has been changed to
   Cannot find the database file_name
 where file_name is the name used for the database.

08-30
The hold_out command did not properly detect and report when the integrand_name was not valid. This has been fixed.

08-25
Advance dock_dismod_at.sh from version 20210812 to version 20210825.

08-24
Change the some python exits with an error message to raising an assert exception with the same message. (This is more useful in the python debugger.)

08-23
The was a bug in checking that the child nodes in the nslist_pair_table were children of the parent node, when the node_id for the parent node was not zero and the parent node was specified using parent_node_name . This has been fixed.

08-22
Change hes_fixed_table so that it does not include fixed effects that have equal lower and upper limits.

08-21
The sim_data utility was moved to average_integrand . It is still considered under construction.

08-18
  1. Change the system_command_prc function so its result is a python string (instead of bytes); e.g., see user_trace_init.py .
  2. If an error occurs during system_command_prc , print stderr as a string (instead of bytes) and raise an AssertionError instead of exiting the current program.
  3. Add documentation in avgint table discussing when node_id should be null.


08-17
There was a problem using a mulcov integrand in the avgint_table (the integrand name begins with mulcov_). To be specific, these integrands would not get included in the predict table avgint subset . This has been fixed.

08-15
Add the replace_table utility.

08-12
  1. Advance to cppad_mixed-20210812. This fixes an install problem in cppad_mixed-20210714.
  2. Fix the syntax in the python system_command_prc function. To be specific, stderr is not returned.
  3. Advance dock_dismod_at.sh from version 20210624 to version 20210812; see the items between 06-24 and 08-12 in this section.


08-08
  1. API Change: The transformed standard deviation is now computed before the standard deviation adjustment, hence @(@ \sigma @)@ no longer depends on the model_variables . As a consequence for log scaled densities, the adjusted standard deviation is now in log of the data space (it used be have the same units as the data). This fixes a problem with simulating data that has meas_noise covariate multipliers.
  2. Change the definition of meas_delta in data.csv so that it is still in data units even when the adjusted standard deviation is in log of data space.
  3. Correct the title in the user_hes_random.py example.
  4. Test the user_fit_meas_noise.py example using more values for the meas_noise_effect .
  5. Move user_sample_simple.py to user_sample_asy_sim.py .


07-29
The Mac changed how bash worked and this broke it's install. To be specific, export DYLD_LIBRARY_PATH no longer works (export works for most shell variables but not others). The bin/get_cppad_mixed.sh was changed to adapt to this problem including advancing to cppad_mixed-20210714. In addition, the specific_compiler setting was added to bin/run_cmake.sh to make it easier to build using MacPorts.

07-25
Fix a bug in the system_command_prc error reporting. To be specific, when an error ocurred during a system command, the command return code was not properly reported.

07-24
  1. The sim_data utility is under construction. It which is slower and easier to use than the simulate_command . It also can be used for independent testing of the dismod_at integrators.
  2. Remove cen_log_students from the log scaled density list (it is not yet a valid density).


07-22
  1. Group the output tables in the fit command documentation.
  2. Add the hes_random table table to the output of the fit command.
  3. The default value for trace_init_fit_model was changed to false and it was added to the option_default list.


07-14
  1. Advance to cppad_mixed-20210714. This enabled adding the trace_init_fit_model option; see user_trace_init.py for an example.
  2. Change bin/get_cppad_mixed.sh so that example_install.sh so that it automatically replaces all the special requirements . This catches changes in the install of eigen, ipopt, and cppad without have to remove the old install directory.
  3. Change system_command_prc so that it return standard output and prints standard error when an error occurs.


07-05
Fix two bugs in the changes on 07-04. One ocurred when using zero sum constraints . The other was in the db2csv_command (the python code value true was corrected to True).

07-04
If a child node does not have any data, its rate random effects automatically use a random bound of zero; see bound_random . This removes them from the optimization problem and yields the same results. The variable.csv column m_bnd was changed to bound to included this new feature in its output.

07-02
There was a bug in the set option command. To be specific, if you set an option that was not yet in the table to the empty string, an empty string (instead of null) was put in the option table. This has been fixed.

06-24
  1. Add the data_density_command and include its settings in the data.csv file columns density , eta , nu .
  2. Remove the log density wish list item which is no longer necessary after the improvement to log densities on 05-20 .
  3. Improve the usage message with dismod_at is executed with no arguments.
  4. Advance to version 20210624 of cppad_mixed and dock_dismod_at.sh.


06-22
  1. Add the bnd_mulcov_table , bnd_mulcov_command and columns m_diff , and m_bnd to variable.csv
  2. Change the section name user to user_example and sort the user examples.


06-18
  1. Change the compression intervals from age and time vectors to age and time scalars (use the same intervals for all integrands).
  2. Sort the table in the data_flow section.


06-17
Add options that compress intervals . Include this setting in the age_lo , age_up, time_lo, time_up columns in the data.csv file output by db2csv_command . For an example; see user_compress.py .

06-15
  1. Move the default option values from option_table to option_default section.
  2. Re-order and improve the headings in the option_table documentation.
  3. Include hold_out_command and init_command in the data flow command output table . In addition, add some missing commas in that table.
  4. Improved documentation for hold_out .
  5. Fix bug in hold_out_command when no data is being held out (the old hold_out values were not erased).


06-14
  1. Improve the documentation for the age average grid .
  2. Group zero sum constraints in option table documentation.
  3. Group Extra Columns in option table documentation.
  4. Group Age Average Grid in option table documentation.


06-13
  1. The db2csv_command would fail, if the previous fit had a warm_start , with the following message: 
     
   Previous fit command in log table used simulated data but
   cannot find data_sim table
    This has been fixed.
  2. Advance to cppad_mixed-20210613. Add the mixed_info_table table and the db2csv mixed_info.csv output file.


06-12
  1. Add the hold_out_command .
  2. Change out column of the data.csv to include the result of all the hold_out settings. This has since been changed to the d_out and s_out columns.


06-11
  1. Add the hold_out column to the data_subset table.
  2. Check that all the files output by the init_command are present when executing commands other than init and set.


06-10
  1. Add the hold_out_integrand option and user_hold_out_1.py example.
  2. Advance to cppad_mixed-20210610. This fixes a warning message during bin/get_cppad_mixed.sh on ubuntu.


06-09
  1. Remove _table from the links in data_flow where they are in lists of tables and it is not necessary.
  2. Change the fixed_trace table's name to trace_fixed.
  3. The trace_fixed_table and ipopt_info_table were not being deleted by the init_command . This has been fixed.


06-08
  1. The warm_start documentation was improved.
  2. Remove the tracing optimization and warn start wish list items because they have been completed.


06-07
  1. Advance to cppad_mixed-20210607 and add restoration to the fixed_trace table.
  2. Add a discussion heading at the top of database sections that did not have a heading after the jump table at the top of the section; e.g., the covariate table discussion heading.


06-06
Advance to cppad_mixed-20210606. This enable adding the fixed_trace_table , for an example usage; see fixed trace table in the warm start example.

06-01
The warm_start option was added to the fit command. This is much better than just modifying the start_var_table because it also sets the initial Lagrange multipliers. For an example, see user_warm_start.py .

05-31
Advance to cppad_mixed-20210531. This is in preparation for adding a warm start option to the fit command.

05-29
  1. Advance to cppad_mixed-20210528. This speeds up and improves the adaptive derivative_test . It also improves the checking of bounds, complementarity, and the gradient of the Lagrangian at the final solution. It also improves the detection of when Lagrangian multipliers should be non-zero because a constraint is active; e.g., see lagrange_value in the fit_var table.
  2. The new version of cppad_mixed found identifiability problems with the user_subgroup_mulcov.py example. This has been fixed by change the prior for the child and subgroup random effect from uniform to a very broad Gaussian.


05-21
  1. Advance to dock_dismod_at.sh version 20210416. Building this new image will require executing 
    
   ./dock_dismod_at image mixed
   ./dock_dismod_at image dismod_at
    The dismod_at.mixed image is necessary because the version of cppad_mixed has changed since the last advancement of dock_dismod_at.sh.
  2. In the user_data_sim_py example the code 
    
   residual = (math.log(z + eta + delta) - math.log(mu + eta)) / sigma
    was incorrect and was changed to 
    
   residual = (math.log(z + eta) - math.log(mu + eta)) / sigma
  3. Change the default setting for quasi_fixed from 'true' to 'false'.
  4. Change user_fit_sim.py quasi_newton setting from 'true' to 'false' and make the maximum allowable error smaller.


05-20
During a simulate_command , the calculation of data_sim_delta and data_sim_stdcv in the data_sim_table often resulted in zero or negative values which does not make sense and would cause dismod_at to abort with the message
   ... simulate_command sim_stdcv is nan or less than zero ...
 This problem has been fixed using the adjusted standard deviation to replace data_sim_delta and the minimum CV standard deviation to replace data_sim_stdcv . These replacements use the original measurements meas_value instead of the simulated measurements data_sim_value to define the corresponding transformed standard deviation .
  1. Add the user_sim_log.py example to highlight this change.
  2. Adapt the user_data_sim.py examples to this change.
  3. API Change: The data_sim_delta and data_sim_stdcv columns were removed from the data_sim_table . The sim_stdcv and sim_delta columns were removed from the data.csv .


05-19
Add missing source code in the documentation for the covid_19.py and residual.py examples.

05-18
Advance to cppad_mixed-20210517. This change, and changes to dismod_at, catch and report throws of std::exception both in the output and in the log_table .

05-15
Advance to cppad_mixed-20210515. This avoids running the cppad_mixed tests during example_install.sh . It also fixes a problem with the optimization of cppad function objects.

05-07
Advance to cppad_mixed-20210507. This should solve some install problems on the msys and cygwin systems.

05-06
If a meas_delta , in the data.csv created by db2csv command, is greater than the maximum python float, report it as empty.

04-16
  1. The fit both command was failing when zero_sum_child_rate was specified for a rate that did not have any children. This has been fixed.
  2. There was a syntax error in the test/user/fit_sim.py program. This has been fixed.
  3. Advance to dock_dismod_at.sh version 20210416. This includes the bug fix above.


03-02
Advance to cppad_mixed-20210302. This places the release source directories Ipopt.git, Mumps.git, ASL.git in external/release/cppad_mixed/external (similarly for the debug case). Previously Mumps.git was below ASL.git and Ipopt.git was below Mumps.git.

01-28
Advance to dock_dismod_at.sh version 20210128. Most importantly this includes the new meas_noise_effect options added on 01-10 . For those at IHME, it is also has the new fit_ihme.py program.

01-22
The set option command no longer requires the option table it to be consistent with all the other input tables. It only has to be self consistent. This enables one to use the set option command to change the rate_case to a value that is consistent with the rest of the database.

01-21
The program db_simplify was changed to fit_ihme.py because it fits ihme databases. It now gets installed and is part of the User API. You can use it, if you have access to the ihme databases, but it is not part of this documentation (specifications). If you execute fit_ihme.py with no arguments, you will get a usage message that give you some idea how to use it.

01-19
The file name was removed from the log table entry at the start of a dismodat.py command. This is so that all log messages with message_type command start with begin and then the name of the command.

01-11
Change the order of the columns in the file variable.csv so that the fit results fit_value are next to the corresponding lower and upper limits.

01-10
  1. The user_data_sim.py example was changed so that it uses the same notation as the new version of the data_sim_table documentation and tests more cases.
  2. The bug fix of the value of data_sim_stdcv on 01-09 was corrected (better testing show it was incorrect).
  3. Add following meas_noise_effect cases: add_std_scale_none , add_var_scale_none .


01-09
  1. The documentation for the data_sim_table was improved.
  2. Fix a bug in the value of the data_sim table column data_sim_stdcv when meas_noise_effect is add_var_scale_log and the density is linear .


01-08
  1. The definition of the transformed standard deviation @(@ \sigma @)@ was extended to apply to all densities, not just log scaled densities. In addition, the definition was move from the statistic section to a place where the different cases are easier to understand; see sigma in data_like , fixed_value , random_value , smooth_dage , smooth_dtime .
  2. In the definition of the value prior for the random effects V_j^u the term @[@ D \left( \theta_j \W{,} \mu_j \W{,} \lambda_j \sigma_j \W{,} d_j \right) @]@ was corrected to be @[@ D \left( u_j \W{,} \mu_j \W{,} \lambda_j \sigma_j \W{,} d_j \right) @]@
Input File: omh/whats_new/2021.omh