Changes and Additions to Dismod

whats_new_2019

@(@\newcommand{\B}[1]{ {\bf #1} } \newcommand{\R}[1]{ {\rm #1} } \newcommand{\W}[1]{ \; #1 \; }@)@This is dismod_at-20221105 documentation: Here is a link to its current documentation . Changes and Additions to Dismod_at During 2019
mm-dd

12-07
Spaces in a group_name are now forbidden. A check was added to make sure there are no such spaces.

12-06
Document the fact that there are no subgroup covariate multipliers that affect the measurement noise; see meas_noise under subgroup_smooth_id in the mulcov table. In addition, report an error if these multipliers are requested.

12-05

The mulcov table group_id documentation what corrected to prohibit the null value. In addition, error checking was added to make sure that these are valid subgroup table group_id values.
The error messaging for the subgroup table was using the wrong subgroup_id , this has been fixed.
Advance dock_dismod_at.sh to version 20191205. This requires running ./dock_dismod_at.sh image dismod_at

12-04

Add the user_zsum_mulcov_meas.py example which demonstrates using the zero_sum_mulcov_group option with subgroup measurement covariate multipliers.
Include both group and subgroup in the variable.csv file created by the db2csv command.
Advance dock_dismod_at.sh to version 20191204. This requires running ./dock_dismod_at.sh image dismod_at

11-30
Add the user_zsum_mulcov_rate.py example which demonstrates using the zero_sum_mulcov_group option with subgroup rate covariate multipliers.

11-29
The zero_sum_random option was changed to zero_sum_child_rate . A warning is printed when zero_sum_random is used and this change is made automatically. In addition, the old2new command will make this change to the option table .

11-28

Add the user_bilevel_random.py example demonstrating fitting two levels of random effects using subgroup covariate multipliers .
Fix the coverage check in user_cascade.py . To be specific, change abs( (mean - truth) ) / std to abs( (fit - truth) ) / std.
Add warnings to create_database when subgroup (group) information is missing for the data table and avgint table ( mulcov table ) tables.

11-26

The meas_covariate.py example was changed to be an example using group covariate multipliers; see user_group_mulcov.py . In addition, a discussion of this example was added.
The constant weighting function was removed from the user_example examples (it is now the default weighting and hence no longer needed).
Change user_fit_meas_noise.py so that it used groups to identify a subset of that data that has extra measurement noise.
The meas_std_effect option_name was removed. It was deprecated (see 04-08 below) and should have been replaced by meas_noise_effect .

11-25
Use system_command_prc to simplify the get_started examples.

11-24

An old2new_command was added. This time it adds the subgroup_table and other related information, to the database in a backward compatible fashion; see old2new_command and 11-22 below. In addition, directions for obtaining this backward compatibility was added to the create_database function.
The db2csv_command was modified in the following ways: The group/sub column as added to variable.csv (changed to group and subgroup on 12-04 ). The subgroup and group columns were added to data.csv The subgroup column was added to predict.csv

11-22
The following changes below were made to the database input tables:

The subgroup_table was added.
The subgroup_id column was added to the data table and avgint table .
The smooth_id column in the mulcov table was changed to group_smooth_id . In addition, the group_id and subgroup_smooth_id columns were added to the mulcov table.

In addition, the subgroup_table argument was added to the create_database routine.

11-20

The automatic conversion of meas_std to meas_noise has been removed; see 04-08 below.
The old2new command that added the mulcov_id column to the var table was removed; see 04-20 and 04-23 below.

11-18
Group all of the sections for the prior of the model variables; see variable_prior . In addition, simplify the differences using smooth_dage and smooth_dtime .

11-17
Fix some out of date statements and simplify the documentation for the value prior for the random effects random_value . To be specific, the corrections are necessary because we can now have different smoothings for different child nodes; see the rate table child_nslist_id .

10-24
The order of the variables in the var_table was changed so that the random effects come first. This is in preparation for adding more random effects. It also makes it easier to find specific variables in the var table and the variable.csv file.

10-09
Advance dock_dismod_at.sh to version 20191008, which includes to changes under 10-08 below. This requires running



   ./dock_dismod_at.sh image dismod_at

10-08

Fix a bug in the zero_sum_child_rate option. This bug caused user_const_value.py to fail when one switched the role of iota and omega ; see test/user/const_value.py.
Advance to cppad_mixed-20191008. This adds a check on the zero sum random constraint that would have detected the bug mentioned in the previous item.

10-06
The bound_random documentation was improved by stating that it does not apply to random effects that have equal lower and upper limits.

10-02
Add discussion at the top of the user_zsum_child_rate.py example.

10-01

The system_command_prc utility was added used to simplify the user_example examples.
The new user_mulstd.py example demonstrates estimating standard deviation multipliers.

09-30
Use a newer version of the program that generates the documentation omhelp. The search utility now shows the title and section tag in separate areas and hides the other keywords. Hitting return or double clicking in the title area, in addition to choosing the Goto button, will go to the selected section.

09-24

Add mention that the install scripts under special requirements also serve as examples of how to install the corresponding packages. In addition, run_cmake.sh serves as an example of how to install dismod_at.
The csv2db_command wish list items were removed because we assume that the dismod_at user can write code in some scripting language and can read the Python user_example examples.

09-23

The comments in the dock_dismod_at.sh Dockerfile were improved. This will help, people who understand docker, with the install procedure.
The version of ubuntu in dismod_at.base was changed from 18.04 to 19.10. This requires running ./dock_dismod_at.sh image base ./dock_dismod_at.sh image dismod_at This fixed a warning during the install of cppad ; see cmake mailing list .

09-22

Advance dock_dismod_at.sh to version 20190922.
The value null in the smooth grid table for dage_prior_id and dtime_prior_id can now be used to specify an uninformative prior; i.e., uniform from minus infinity to plus infinity. (The corresponding null prior id wish list item was removed; see 09-20 .)
The user_const_value.py example has been changed to use more grid points for the fixed rate effects than for the random rate effects. In addition, the omega constraint uses the new null feature for the dage_prior_id and dtime_prior_id in the smooth grid table. Furthermore, some problems with the example have been fixed.
Improve the error message when the child value priors do not satisfy their requirements.
The random seed in user_cascade.py was a fixed value, this has been changed to seed of the clock for better testing.
Add discussion of choosing the random seed and reporting its value when an error occurs to the random_seed example.. In addition, this example was changed so that the start_var_table and scale_var_table are equal to the truth_var_table .

09-21

The rate_case and checkpointing wish list items were under the csv2db Command heading. This was a mistake and they have been moved to be under the ODE Solution heading.
The change of random failure in the user_censor.py example was reduced by reducing the number of values that get censored. To be specific, the line
```
   meas_std = 2.0 * iota_true;
```
was changed to
```
   meas_std = 1.5 * iota_true;
```

09-20

The sampling random effects items was removed from the with list because the user_sample_sim.py method seems to work well.
An asymptotic statistics item and null prior_id item were added to the wish list

09-19

Move the sample command asymptotic example from user_asymptotic.py to user_sample_asy.py .
The prior_sim_table values for the random effects are no longer included when sampling the fixed effects; see the simulate method for the sample command.
Add the example user_sample_sim.py for the sample_command with the simulate method.
Advance dock_dismod_at.sh to version 20190919. A modified version of bin/run_cmake.sh was in the dismod_at.base which caused a problem when building dismod_at.image. This has been fixed but you should rebuild dismod_at.base to get this fix.

09-17

The constant weight change on 09-13 below introduced a bug in the user_speed.py example. This has been fixed.
Advance dock_dismod_at.sh to version 20190917.
Add instructions for logging output from the dock_dismod_at.sh program.

09-15
Discussion was added to the user_const_value.py example. In addition, it was modified to show how to constrain child rates for omega (as one might do if they replaced all cause mortality data by constraints on other cause mortality).

09-13

The constant weighting can now be specified using null as weight_id in the data table and avgint table . This weighting can also be specified by the empty string in create_database; see weight . The user_example examples were simplified using this option.
Advance dock_dismod_at.sh to version 20190913.

09-12

Advance the dock_dismod_at.sh dismod_at_version to 20190912.
If zero_sum_child_rate is chosen for a rate, the simulated prior values for that corresponding child rate effects at each grid point now sum to zero; see zero_sum_child_rate in the prior_sim table documentation.
The method_random option was added. This gives a different choice for optimizing the random effects.
The zero_sum_child_rate value in option.csv written by the db2csv command was false when it should have been empty. This has been fixed.
The examples that used simulated data were changed to set the start_var_table and scale_var_table equal to the truth_var_table ; see starting point and scaling for the example/user/fit_sim.py example.

09-11

Increase speed and reduce chance of failure for test/user/fit_sim.py. The problem here is that the simulated values for the mean of the prior of the random effects do not sum to zero. This should be fixed.
The simulate_index was added to error messages printed when fitting simulated data.

08-24

The documentation now explicitly states that the empty string is stored as a null when it is used for the value in a set option command. In addition, the documentation for the option_table now explicitly states which default values are null (instead of using other words like empty). Furthermore, the possible combinations of null and not null for the parent node id and name are discussed under the heading name or id .
A new version of dock_dismod_at.sh automatically detects (and suggests solutions) when the docker daemon is not running or when you do not have the necessary permissions to run docker.

07-25

The dock_dismod_at.sh command now has two stages to its build process; dismod_at.base and dismod_at.image . This avoids having to rebuild the entire image every time.
Extra debugging instructions were added to the dock_dismod_at.sh command.
Advance to cppad-20190725. This fixes a warning when building images using dock_dismod_at.sh.

07-24
Advance to cppad_mixed-20190724. This gives better error reporting during initialization. It also automatically optimizes w.r.t. the random effect before initializing terms that require the Hessian w.r.t. the random effects to be positive definite.

07-20
Advance to cppad_mixed-20190720. This fixes a print during the testing of dismod_at.

07-19
Advance to cppad-20190719. This fixes a problem installing the CppAD binary library on the Mac.

07-13
There was a mistake in the value of dismod_at_hash for dismod_at_version 20190710 of dock_dismod_at.sh. This has been fixed.

07-10

Advance to cppad_mixed-20190710. This makes the Newton method for the fixed effects more robust (quasi_fixed false case). To be specific, one is less likely to an error message indicating that the Hessian w.r.t. the random effects is not positive definite.
There is a new dismod_at_version in dock_dismod_at.sh. In addition, there are new instructions to help with one of the possible errors during the dock_dismod_at.sh build command.

07-06
Improve the documentation for the max_num_iter option. Most importantly, make it clear what value is returned for the random effects when max_num_iter_fix is -1.

06-28
Add discussion of relation ship between the measurement standard deviation meas_std and the adjusted standard deviation delta(theta) to the meas_std documentation.

06-27
The relation between of the standard deviations and the measured value (the mean value) was not clear for the log scaled densities. This has been fixed by adding to a discussion about log scaling the standard deviations in the data table (the prior table ).

06-26
The meas_delta column of the data.csv file output by the db2csv command has values when the previous fit corresponding to simulated data (not the measured value and standard deviation). This has been fixed by setting meas_delta to empty when the previous fit is for simulated data. In addition, more discussion was added for fit_var, fit_data_subset tables and the simulate_index .

06-25
The build option for dock_dismod_at.sh was not updating the docker image when the remote repository changed. This has been fixed by adding a git pull and checkout of a particular dismod_at_version in the Dockerfile created by dock_dismod_at.sh. In addition, more detection and discussion was added for removing containers .

06-24
Some minor improvements were made to the dock_dismod_at.sh documentation, as well as the comments in its bash script.

06-23
If you have the docker package, you can now avoid the install_unix process using dock_dismod_at.sh script.

06-22
Print value used for PKG_CONFIG_PATH and fix set exit status for example_install.sh .

06-21
Automatically set PKG_CONFIG_PATH during the bin/install_cppad_mixed.sh script which is used by example_install.sh .

06-07

Advance to cppad_mixed-20190607. This gives better error messaging when nan occurs and build_type is debug.
A paragraph discussing Ipopt options was added to the fit command documentation.
If there was only a lower, or only an upper, limit in the prior for a variable, Ipopt would sometimes start the optimization at a point that was too far from the start_var point. This has been fixed by prohibiting a lower limit without an upper (or an upper limit without a lower); see upper in prior table documentation.
Extra documentation was added for the bound_frac_fixed option (and bound_push was set to zero).

06-02
The version number in the title for the level of the documentation dismod_at was not updating. This started with version 20190521 and has been fixed.

06-01

The ihme_db.sh script was improved and documented.
In some cases, avg_age was still being used instead of age_avg, see the discussion about the avg_age on 2018-10-30 . This has been fixed.
This discussion of the age_avg_table in the data_flow and init_command was improved.

05-24
Needed to also add a git checkout master for problem mentioned on 05-21 below.

05-21
There was a problem with the cppad and cppad_mixed install scripts. They sometimes failed because the corresponding repository was out of date. This has been fixed by adding a git pull to the corresponding scripts.

05-12
Add a censored Laplace entry to the wish list.

05-07
Advance to using cppad-20190507. This enabled a fix of the cen_gaussian calculation. To be specific, when the start_var_table values were very poor, and there was zero measurement values, one minus the error function was numerically zero and its log was nan. This has been fixed using the complementary error functions which was recently added to CppAD.

05-05
The new example user_data_sim.py detected a problem in the simulation of log scaled data. This problem has been fixed.

05-03
API Change: The data.csv file columns Delta and delta have been changed to meas_stdcv and meas_delta respectively. The change of names was motivated by a problem using sqlite when two column names that only differed by case; i.e., Delta and delta.

The new column data_sim_stdcv, in the data_sim table, is value of Delta that is used when fitting the corresponding simulated data. This corrects a bug in fitting simulated log scaled data that has meas_noise covariates which adjust the measurement standard deviation; see delta_i .
The value Delta , is now referred to as the minimum cv standard deviation .
The sim_stdcv column has been added to the data.csv file..
The documentation of the method used to create the simulated data was improved and corrected.

05-02

Add the cen_log_gaussian and cen_log_laplace densities.
Partially fix a bug in fitting simulated data for log scaled densities. There is still problem when there are meas_noise covariates, that hopefully will be fixed soon.

05-01

Due to the change in prior_sim_table on 04-30, log scaled distributions can now simulate negative means. This was causing a message saying that residual_density: mu was negative. This check has been changed to make sure @(@ \mu + \eta @)@ is positive.
Add a discussion of the difference between means for the censored distributions.
A check as added to make sure that variable values, that are possibly provided by the user, are within their specified limits.

04-30

The censored Laplace density cen_laplace now works. See the discussion of nonsmooth densities in the data table.
Change the prior_sim_table by entering the a simulated Laplace random variate, instead of zero, when the mean for prior Laplace distribution is zero.
Improve the documentation of censoring of simulated values in the prior_sim table.

04-29

The user version of the documentation no longer includes things that are only meant for developers and are not part of the API. (The developer documentation can be built by starting omhelp with the file dev.omh.)
The jump tables at the top of the whats new sections were replace by drop Headings menus because they were long and not very informative.
The user_censor.py example was extended to include simulating censored data using the simulate_command .

04-28
Remove some dead code that was repeated between different user_example examples. In addition, use a local system_command function to simplify the user examples.

04-27

Add the cen_gaussian density; see the example user_censor.py .
Change the definition of the log-transformed standard deviation sigma so that it is different for data than for priors. This avoids the need for reversed residuals.
Fix bug in previous change to density table (should no longer need densities that are not used).

04-26

The density_table no longer needs to contain all possible entries and the user can choose the order of the entries.
Add a wish list entry for keyword arguments in create_database .

04-25
Simplify the discussion of residuals and densities by placing the ones that are similar next to each other so it is easier to compare them.

04-24
Add a plan for censoring negative data .

04-23

The detection and reporting of problems, when computing the value of an integrand , has been improved. For example, attempting to divide by zero.
The old2new_command was added.

04-22

Improve the user_cascade.py example. For example, use a Laplace prior to get a better estimate of alpha , when fitting with n1 as the parent.
In the user_example examples, move functions that to define smoothings inside the corresponding function that defines a database; see fun .

04-21
First complete version of user_cascade.py ; i.e., first version that passes priors for fitting one level down from the top level.

04-20

Improve the user_predict_fit.py example.
Change the install test so that make check does a re-try when it gets a failure. This makes the probability of a random failure much lower.
Add the mulcov_mulcov_id integrand name ; see the user_predict_mulcov.py example / test.
The mulcov_id column was added to the var table. This identifies which variables a mulcov_mulcov_id integrand name correspond to; see init_command.py .

04-18
First version of user_cascade.py that gets good coverage by modeling the measurement noise covariate as model misspecification and using it to expand the posterior intervals.

04-17

Add the user_cascade.py example. This example is currently under construction; i.e., not yet complete.
Change ode_step_size in user_csv2db.py from 1.0 to 5.0 (to make automated testing faster when running in debug mode).

04-15
Improve the user_students.py example (this example has since moved to user_data_density.py ).

04-14
Improve the user_lasso_covariate.py example.

04-12
A discussion was added at the top of the user_change_grid.py example.

04-11

There was a bug in the db2csv computation of the adjusted standard deviation meas_delta for the log density cases. This has been fixed.
Remove the Ode Grid section from the documentation. It has been replaced by the average integrand time step and the age average grid .

04-10

The parent node , not just its descendants , is included in the node restrictions for the data_subset table. This and has been fixed in the data subset documentation and other places where it was not correctly stated.
The @(@ x_{i,j} @)@ values are now called covariate difference to distinguish them from the covariate values in the data table and the avgint table.
Discussion for computing the parent and child rates were added to the avgint table.

04-09

In the case where the deprecated name meas_std is used, the corresponding var_type was mulcov_meas_noise . This has been changed to mulcov_meas_std for backward compatibility.
In the data.csv file, the column meas_value was moved to be next to avgint and meas_std was moved to be next to Delta . This makes comparisons between these columns easier.
Add the adjusted standard deviation column meas_delta to the data.csv file written by the db2csv command.
The discussion of data weighted residuals was improved by defining reversed weighted residuals .

04-08

The covariate multiplier type meas_std was deprecated (and was later removed). It should be changed to meas_noise .
The option meas_std_effect option was deprecated. It should be changed to meas_noise_effect .

04-06
The documentation for the fit_var_value column of the fit_var table was improved.

02-16
Add the script bin/speed_16_18.sh which compares the last version of in 2016 with the last version in 2018.

01-26
Extra wording was added to the description of age in the age table and time in the time table to make it clear that these values do not need to be in a specific order. The jump_at_age.py example was modified so that its ages and times were not in increasing order. In addition, a discussion was added at the beginning of the jump_at_age.py example.

01-23
There was a bug in the var table where age_id and time_id were sometimes incorrect for variables were covariate multipliers. This has been fixed.

01-22

A weighting wish list item was added (and was completed on 09-13 ).
The user_ill_condition.py example was added.
The data_sim_table documentation formula for log scaled data was @[@ y = \exp(e) * ( A + \eta ) @]@ The correct formula is @[@ y = \exp(e) * ( A + \eta ) - \eta @]@ The documentation has been fixed (the implementation in software was correct).

01-16
Add the scale column to the variable.csv file create by the db2csv command. In addition, group the start, sale, fit_value, truth, and sam_avg columns next to each other because they are all values for the model variables.

01-15
Change the option table minimum_meas_cv to redirect one to the integrand table.

01-10
Reduce the change of random failure in test/user/fit_meas_noise.py during testing .

Input File: omh/whats_new/2019.omh