Prev Next whats_new_2018

@(@\newcommand{\B}[1]{ {\bf #1} } \newcommand{\R}[1]{ {\rm #1} } \newcommand{\W}[1]{ \; #1 \; }@)@ This is dismod_at-20221105 documentation: Here is a link to its current documentation .
Changes and Additions to Dismod_at During 2018

mm-dd

12-19
The add_var_scale_log would sometimes fail to converge when the initial value of corresponding covariate multiplier gamma_j (a, t) was zero. This was because the derivative of the object w.r.t. this multiplier was zero even though zero is not the solution. This has been fixed by changing the definition when adding variances to use gamma_j for the variance (instead of the standard deviation). The user_fit_meas_noise.py example was modified so that it demonstrates how the new definition affects the scaling of gamma .

12-15
Add options for how the measurement noise effect meas_noise_effect is defined. The gives the choice of adding variances or standard deviations . It also gives the choice to scale the added standard deviations for non-log densities.

12-14
Add user_fit_meas_noise.py , an example that fits gamma using simulated data.

10-30
  1. The avg_age_split option was changed to the age_avg_split option. This way the age_avg_table does not appear to have the same primary key as the age_table .
  2. The age average grid is now output in the age_avg_table to make age_avg_split easier to use.
  3. There was a bug in the age_avg_split option when two splits occurred in the same ode_step_size interval. This has been fixed.
  4. There was a bug in the age_avg_split option when the minimum and maximum age in the age_table were equal. This case now results in an error message.
  5. If the set option command had an empty string for a value, the empty string was placed in the option table. This violated the text can be null, not empty rule and has been fixed.
  6. If the age_avg_split was not sorted, the program would error even if the set_command was trying to fix it. This has been changed so the set command can fix it.
  7. A check of the file avg_age.csv was added to the db2csv_command.py example.


10-22
Start work on an math_abstract to the mathematics of dismod_at.

10-21
Change the way that average integrands are computed; see numeric_average . This enables using a step size that varies with age; see age average grid . Here is one case of a speed comparison for this change using a uniformly spaced grid: before 4.9 seconds, after 3.6 second. Here is the memory comparison for the same case: before 288.7 MB, after 121.8 MB. This test takes advantage of the fact that the new method only integrates one cohort when age_lower = age_upper and time_lower = time_upper . It does not use age_avg_split to vary the spacing of the age grid.

10-13
  1. The ode_step_size documentation was changed to reflect the fact that it is also used for approximating the integrals in the average integrand.
  2. The documentation for the age corresponding to pini was fixed. To be specific, the corresponding age is the minimum age in the age_table , not the age corresponding to age_id zero.


10-05
Fix typos in the Data or Avgint Table Notation . To be specific, change @(@ a_i @)@ to @(@ b_i @)@, @(@ s_i @)@ or @(@ t_i @)@ where appropriate.

09-30
  1. The n_data_per_child argument to the speed test has been replaced by ode_step_size .
  2. Advance to cppad-0180925 and cppad_mixed-0180925. This had the following effect timing result for bin/speed.sh:
    user time memory
    before 10.395 sec 1.265 GB
    after 9.376 sec 1.247 GB

09-28
The program did not report an error when a weight_id in the data table was null. This has been fixed.

09-24
The ode wish list entry has been extended and moved to the front of the list.

09-14
An empty smooth table would cause the program to crash. An error message is now printed for this case.

08-30
The documentation for the child_smooth_id and child_nslist_id was improved. To be specific, these specify priors for the child rate effects and not for the child rates.

08-29
Advance to cppad-20180827 and cppad_mixed-20180829 (working on speed).

08-22
  1. Add dismodat.py to the documentation under the command section.
  2. Advance to cppad-20180820 and cppad_mixed-20180821. This uses a new CppAD feature and resulted in the following change for user_speed.py : old_time=9.33 seconds, new_time=7.39 seconds.


08-13
The fit_data_subset table had a an extra column (that was not in the API) named data_subset_id which has been removed. This extra column was identically equal to the fit_data_subset_id column. Note that fit_data_subset_id is a foreign key for data_subset_id in the data_subset table. This follows the foreign key convention for the dismod_at database.

08-09
  1. Fix some bugs in csv2db_command .
  2. Advance to cppad-20180809 and cppad_mixed-20180809. This fixes a bug in cppad (that originally showed up in dismod_at testing).


08-08
Edit the prevalence ODE wish list item.

08-07
Add the csv2db_command wish list items (removed on 2019-09-24 ).

08-06
  1. The measurement integrand name, in the discussion of the get started database, has been corrected from mtother to susceptible.
  2. The data column data_info was changed to c_data_info and a discussion of this column was added to the get started example.


07-31
Put more emphasis on the difference between the parent rates , child rate effects , and adjusted rates .

07-30
  1. Add the build_type to the usage message that is printed when no command line arguments are give to dismod_at.
  2. Some dismod_at commands (e.g., predict_command ) were not trapping and reporting exceptions. This has been fixed.
  3. The following sentence was added to the prevalence ODE item: This formulation of the ODE also have the advantage that it is stable even as @(@ S + C \rightarrow 0 @)@; i.e, it will not get the associated nans cause by division of zero by zero.


07-27
Add the prevalence ODE wish list item.

07-26

API Change
The sample_index in the predict table was changed to be null (not zero) when the source for the last predict command is not sample . This also change the s_index in the predict.csv file generated by the db2csv command to be empty (not zero) for this case.

Other Changes
  1. Improve the discussion of the mtother data in the db2csv command.
  2. Execute db2csv_command at end of user_csv2db.py example.
  3. A predict_command was added to the end of the user csv2db.py example and a discussion of these predictions was added.


07-25
  1. Add mention that, during a db2csv command, the age-time grid for rates is the same as the user's choice of grid for mtother .
  2. Add discussion of why the default the prefix for eigen is different from the other prefixes.
  3. Change the default value of build_type to debug. You have to change this to get the faster (release) version of dismod_at.
  4. Improve discussion of the paths needed during and after the install process.
  5. Advance to cppad_mixed-20180725. This should improve detection (during the install ) of when the default settings for extra_cxx_flags do not work.


07-23
  1. Advance to cppad-20180723.
  2. Fix unix_time for the end of the commands that use dismodat.py was mistakenly reported as the same as the begin time. This has been fixed.
  3. Added the csv2db_command to help one get started using dismod_at.
  4. More discussion has been added for when an init is required after a set option command.


07-21
Add restrictions on (a, t) arguments to weight_table fun(a, t) and smooth_table fun(a, t) in create_database.

07-13
Advance to cppad_mixed-20180713.

06-30
Advance to cppad-20180630, cppad_mixed-20180630, and remove checkpoint_newton_step from run_cmake.sh . This should make be faster and use less memory when quasi_fixed is false. For the user_speed.py example with n_children=30 , n_data_per_child=20 , and quasi_fixed=false the elapsed time improved from 14.1 to 8.3 seconds.

06-29
Advance to cppad-20180627 and cppad_mixed-20180629. Again this is expected to be an improvement to the optimization of cppad functions when build_type is release.

06-21
  1. Add text about failures during the dismod_at install correctness tests .
  2. The init_command and db2csv_command would hang if parent and node_id were equal in the node table. An error message is now generated for any case where a node was a descendant of itself.


06-20
Advance to cppad-20180620 and cppad_mixed-20180620. This should use less memory, and increase speed, during the optimization of cppad functions, which is only done when build_type is release.

06-18
Change the set command so that the possible sources include the scale, truth tables.

06-14
Advance to cppad-20180613 and cppad_mixed-20180614. This should use less memory during the optimization of cppad functions, which is only done when build_type is release. In addition, it should fix a problem with cppad_mixed finding the file cholmod.h on some systems.

06-09
Advance to using version 20180609 of cppad_mixed . This changes suitesparse from a special requirement to a system requirement .

06-04
Advance to using version 20180604 of cppad and cppad_mixed . This will probably have no effect but might reduce the amount of memory dismod_at uses.

05-24
A wish list meas_std entry was added and the minimum_meas_cv entry was removed (because it was completed).

05-23
Warning: this is not a backward compatible change to the user API. The minimum_meas_cv option was removed from the option_table and the column minimum_meas_cv was added to the integrand table. This enable one to used different minimum measurement coefficients of variation for different integrands.

05-21
Advance to using cppad-20180521 , and cppad_mixed-20180521 . This should both reduce memory usage.

05-14
There was a bug in the simulation of age difference and time difference prior means introduced on 04-24 . To be specific @(@ \sigma @)@ was being used where @(@ \delta @)@ should have been.

05-09
A minimum_meas_cv and a command diagram entry were added to the wish list. The minimum_meas_cv entry has been completed; see minimum_meas_cv .

05-08
Round off error could make some values in fit_var_table not satisfy the lower and upper limits for the corresponding variables. This has been fixed.

05-07
Advance to using cppad-20180507 , and cppad_mixed-20180507 . This should both increase speed and reduce memory usage.

05-05
  1. The text describing the purpose of the prior table , smooth table , and smooth_grid table was improved.
  2. The model_variables section was improved. To be specific, the term subvector was changed to group of variables and more mention was made of the corresponding function.


05-04
  1. Advance to using cppad-20180503 , and cppad_mixed-20180503 . This should reduce the amount of memory used without adversely affecting its speed.
  2. Use child rate effect when referring to the a random effect u_ik . This distinguishes it from the corresponding rate for that child r_ik .


05-02
  1. It is possible for the start_var_table values to violate the lower and upper limits for a variable. An error message that specifies which variable's starting value violates the constraints is printed. (The fit used to terminate with no message when this occurred.)
  2. The sample simulate used to start its fits at the truth_var_table values. This has been changed to use the start_var_table values.


04-30
  1. A description of the log entries when using the dismodat.py syntax was added.
  2. The documentation for log entry at the end of a command was corrected from done to end .
  3. The sql_command was modified to return its result .
  4. The under construction warning was removed from the modify_command .


04-26
  1. The columns sim_v, sim_a, sim_t were added to the variable.csv file.
  2. The modify_command was marked as under construction.


04-25
  1. Add Instructions for running one get_started or user .
  2. Improve the user_fit_random.py example.
  3. Improve user_fit_fixed.py and move it to user_fit_fixed_both.py .
  4. Add the user_sample_simple.py example.
  5. There was a bug in the sample_command and the fit simulate_index when the data set was empty. These have been fixed.
  6. The prior_sim_table was note being used to replace the prior means when fitting with a simulate_index . This has been fixed.


04-24
Change the simulate command to also output the prior_sim_table . Use this table to include variation of the prior means when using the sample command with the simulate method; see simulation in the discussion of the posterior distribution for maximum likelihood estimates.

04-23
Improve the description for simulation from the posterior distribution of maximum likelihood estimators that use priors.

04-22
Add a proof that prevalence does not depend on other cause mortality; see prev_dep .

04-21
Warning: this is not a backward compatible change to the user API. Change the name of the simulate_table to data_sim_table . In the data sim table change simulate_value to data_sim_value and simulate_delta to data_sim_delta.

04-16
  1. Add a linear matrix @(@ B @)@ to the prior in Lemma 2 so that it can apply to computing the posterior statistics when we have difference priors.
  2. Add simulation to the posterior simulation discussion.


04-13
  1. Add some text to make it easier to follow the final step in the proof of Lemma 2 of the posterior distribution section.
  2. Print an error and exit when the set option name is not a valid choice (the program used to put the invalid option in the option table).


04-12
  1. There is a problem with using the sample command simulate method. The new posterior model section discusses the problem.
  2. The requirements for the last age (last time) priors were changed so they do not need to be null; see null under dage_prior_id and dtime_prior_id . This made it possible to modify the example user_change_grid.py so that it removes the last age (instead of one in the middle of the grid) without having to change any dage_prior_id values. There is a problem with using the sample command simulate method. The model section posterior discusses the problem.


04-11
Instructions for removing an age or time from a smoothing grid were added. This is useful when one determines (through fitting) that there is not enough data to determine a variable, or that a variable is nearly linear in a region.

04-10
The fit_random.py user example has been improved including a discussion of what is going on.

04-09
There was a mistake in the discussion of the get_started_db.py example. The adjusted other cause mortality for this example is @(@ \exp ( \alpha x ) \omega @)@ not @(@ \alpha x \omega @)@. This has been fixed.

04-08
Some uses have complained that it is hard to navigate the documentation. The links at the top of every page have been changed in an attempt to make this clearer.

04-07
  1. Add limited_memory_max_history_fixed to the option table. The default for this option was previously hard coded into the program and could not be changed. As it turns out, we seem to get better convergence with larger values for this option when quasi_fixed true.
  2. The documentation for the optimizer options have been split up into three groups: fixed and random , fixed only , and random only .


04-06
  1. The init command documentation was improved to include a list of the deleted tables .
  2. The depend_var_table was added to the set of tables deleted by the init command because it must be the same length as the var_table or the db2csv_command will fail.
  3. The db2csv command was improved so that it prints an error message when the length of any of the variable tables is not the same as the var_table .
  4. Advance to cppad_mixed-20180406. This allows for testing some new ideas, but does not yet have any effect on dismod_at.


03-21
In the definition of delta_i , The text @(@ delta_i ( \theta ) @)@ was changed to @(@ \delta_i ( \theta ) @)@.

03-13
The documentation of the restrictions on the priors for the random effects has been improved. See child value priors and child difference priors . In addition, the error messages when these restrictions are violated has also been improved.

03-10
The install download procedure was changed to use the following command to determine the version number: 
 
   grep '^SET(dismod_at_version' CMakeLists.txt

03-08
There was a mistake in the error message for wrong column type detected in an input table. To be specific, a syntax error would occur on the line 
 
   msg  = 'expected type ' + ty ++  ' for column ' + name
This was fixed by changing ++ to +.

03-06
  1. Some users were using the type names INTEGER, REAL, and TEXT. The database specifications have been extended to allow for these type names , and the necessary changes were made.
  2. The error messaging when db2csv_command detected an invalid primary key has been improved.
  3. Change db2csv_command so it checks the type of all the columns in the input tables.


02-20
Advance to cppad_mixed-20180220. This fixes a problem, for some compilers, where some friend functions in cppad_mixed.hpp were not defined.

02-17
Change the user_diabetes.py example so that it constrains the child omega rates to vary with age and time using the parent_age_grid and parent_time_grid .

02-14
  1. Add child_age_grid and child_time_grid to the diabetes example problem parameters.
  2. Improve documentation of child_nslist_id .


02-13
The changes below are for user_diabetes.py :
  1. Add random_seed and tolerance_fixed to the problem parameters.
  2. Change fit_with_noise_in_data to be false, so less data can result in tests passing; i.e, makes automated testing faster.
  3. Change max_abs_rel_err to accept_rel_err .
  4. Change ode_step_size to be larger to make the automated testing run faster.


02-12
The changes below are for user_diabetes.py :
  1. Add two child nodes (random effects) to the node_list .
  2. Changes so the example with random effects runs faster: Increase the tolerance_fixed . Reduce the number of points in the age and time grids.
  3. Set the scale_var_table equal to the start_var_table . This avoids nans during evaluation of the objective with random effects.
  4. Optimize first with no random effects and then use that as a starting point for optimizing with random effects.
  5. Fix a list by changing it from time_index_rate_parent to time_index_rate_child.
  6. Change number of time grid points in random effects from two to one (random effects are constant for this example).


02-11
  1. Advance to cppad_mixed-20180208 (no change from user's perspective).
  2. Change db2csv_command so that it displays eta_v when ever it affects the fixed effects scaling ; see prior_info .
The changes below are for user_diabetes.py :
  1. Remove two priors that are no longer being used.
  2. Change the priors for the rates so use log scaling during the optimization process.
  3. Add derivative_test_fixed , and mulcov_dict to the set of problem parameters.
  4. Increase the number of points in the age and time grids.
  5. Correct documentation for covariate multipliers .


02-09
  1. Detect and report an error when a data table density is log_students and eta is less than zero or null.
  2. Update the values used for option_name in the option_table.py example.


02-08
  1. Add start_var to the possible sources for the set command.
  2. Advance to cppad_mixed-20180208. This fixes a bug in the Laplace density in the data table when quasi_fixed was false.
  3. Detect and report an error when the density in the data table is nonsmooth .


02-05
Advance to cppad_mixed-20180205. This is expected to improve the scaling of the fixed effect optimization (in some cases).

02-04
The variable.csv start column was only correct when var_type was rate. This has been fixed. The changes below are for user_diabetes.py :
  1. Add truth2start , accept_rel_err , to the problem parameters.
  2. Add fitting of multipliers for the following covariates: market scan 2000 on meas_value , body mass index and sex on the rate_value for iota.


02-03
The age and time limits in the data.csv file were rounded off (as are most of its other columns) to reduce the space used when viewing the columns in a spread sheet program.

The changes below are for user_diabetes.py :
  1. Make the standard deviations in the parent rate differences problem parameters .
  2. Change node_list example to only the parent node.
  3. Add integrand_list , ode_step_size , meas_repeat , quasi_fixed , to the problem parameters.


02-02
All these changes are to user_diabetes.py which is still under construction.
  1. Added age and time variation to omega and chi.
  2. Increased number of optimizer iterations because there is feed back from the initial iota to prevalence at all ages.
  3. Increase number of age grid points to 9 so ages are closer to being integer valued.
  4. The problem parameters were moved into the documentation section of the this example.


02-01
Added time variation to iota in the user_diabetes.py example. This example is still under construction, but it has some interesting techniques for simulating data and testing models.

01-31
  1. The start column was added to the variable.csv file created by the db2csv_command .
  2. The db2csv_command.py example was improved to better demonstrate (and test) which field in variable.csv are empty


01-30
Improve the documentation for the smooth_table argument to the create_database python function.

01-28
  1. Add mention that sample_index is always zero when the source for the predict command is fit_var or truth_var .
  2. Mention that under normal circumstances, derivative_test should be none.


01-26
  1. A special discussion of the prior mean, in the case where the density is uniform , was added.
  2. Improve error message when number of unique ages (times) smooth_grid_table does not agree with the corresponding n_age ( n_time ) in smooth table.
  3. The definition of the adjusted standard deviation for the log transformed densities was changed so that the standard deviation effect multiplies Delta .
  4. There was a bug in how db2csv_command determined the reference value for covariates in the avgint_table . This has been fixed.


01-25
Add mention that the meas_value is not used during the simulate command.

01-24
Change the specification for the smooth_table function
   (v, da, dt) = fun(a, t)
 so that None can be returned for da and dt when there is no such forward difference. In addition, change the user_example examples to use this feature.

01-23
  1. Improve documentation of which data points require a separate solution of the ODE . In addition, we added a discussion of the purpose of the ODE grid.
  2. Advance to cppad_mixed-20180123. This fixed a bug in cppad_mixed-20180122.


01-22
  1. Advance to cppad_mixed-20180122. This only makes sure the new version works and should not result in any change to dismod-at.
  2. Improve the instructions for combining both debug and release versions of dismod_at on the same system.


01-21
  1. Include discussion of the weighted residual in the simulate_command documentation.
  2. Some other minor improvements and corrections were made to the documentation.
  3. The calculation of the data_sim_value was corrected to allow for negative values in linear cases. (The mistake fixed here was made during the changes on 01-20 ; see below.)
  4. Check for and abort with an error message when there is a null value in the data table meas_value column.


01-20
  1. The method used by the simulate_command was simplified and documented. To be specific, the average integrand, instead of the simulated data value, was used to compute the transformed standard deviation.
  2. The fields meas_value and meas_std in the data_sim table were changed to data_sim_value and data_sim_delta .


01-18
  1. The flags -Wpedantic, -Wshadow and -Wconversion were added to the default value of extra_cxx_flags . In addition, fix the corresponding warning messages generated by gcc during the dismod_at part of the instal process.
  2. Fix an un-used variable warning when build_type is release.
  3. Replace the default acceptable tolerance by the tolerances specified in the options table. (The acceptable tolerance is used when the optimizer is having trouble obtaining the desired tolerance.)
  4. Advance to cppad_mixed-20180117. This fixes warnings with the new compiler flags above.


01-16
The data.csv file did not include the sim_value and sim_std columns when there was no simulate_index in the previous fit command. If there is a data_sim_table the default value zero is used for simulate_index when it does not appear in the previous fit command.

01-12
There was a bug in the change on 01-09. To be specific, the case where a child had no random effects , and the child had data with no rate covariates, was not handled correctly. This has been fixed.

01-09
A change was made to speed up the calculations for measurements that have the reference value for all the rate covariates; see rate_value in the mulcov table documentation.
Input File: omh/whats_new/2018.omh