Jacob Barhak Ph.D. http://sites.google.com/site/jacobbarhak/ th ISPOR 18 Annual international Meeting 18-22 May 2013
THE REFERENCE MODEL SCORES FITNESS OF MODELS AND POPULATIONS In a Nutshell Competitive Disease Forecast A League of Disease Models/Consumers Report Rotate Populations
Population 1
Population Characteristics: Age, Male, Race, SBP, DBP, A1c, Smoke, BMI, HDL, LDL, Trig, TC, ACR, Lipid Ratio, Age At Diagnosis Of Diabetes, AF, Survive Stroke, Survive MI, MicroAlbuminuria, MacroAlbuminuria, Diabetes Type 2, Treated for Hypertension, Family History CHD, Rheumatoid Arthritis, Townsend Index, A1c Change, BMI Change, DBP Change, SBP Change, HDL Change, LDL Change, Trig Change, TC Change, Smoke Change, Year
Population 2
Process CHD
No CHD
MI
Stroke
Process Competing Mortality
Inputs are based on secondary data: • Published Risk Equations • Published Clinical Trials: i.e. no real individual data • Other publications
Rotate equation/ hypothesis variations
End of Trial
A1c
7.48
7.24
BMI
28
28
DBP
80.5
74.3
SBP
145
137.9
HDL
1.25
1.25
LDL
3.11
2.65
Trig
1.64
1.59
Without Biomarker change, Use the baseline distribution in risk equation
1
2
3
*
2
*
*
3
4
5
*
*
*
*
6
7
*
*
5 6
*
7
*
8
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
*
*
*
*
4
*
*
*
*
*
*
*
*
*
*
* *
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
* *
*
Outputs: 1. Clinical outcomes / Deaths 2. Costs / Quality of Life 3. Population / Equation / Hypothesis fitness
*
9
*
*
With biomarker change, from 1st year use end of trial distribution in risk equation
Best Model
MODELS Model Characteristics
A1c changes BMI changes BP changes Lipid change Smoke changes MI Equation # Stroke Equation # CHD Death Equation # Stroke Death Equation # Treatment Improvement Correction
0 0 0 0 0 1 1 1 1 0
0 0 0 0 0 1 1 1 1 1
0 0 0 0 0 1 2 1 1 0
0 0 0 0 0 1 2 1 1 1
0 0 0 0 0 1 3 1 1 0
0 0 0 0 0 1 3 1 1 1
0 0 0 0 0 1 4 1 1 0
0 0 0 0 0 1 4 1 1 1
CONCLUSIONS: 1) Modelers should talk in terms of best fitting model rather than best model since in many cases, the best model changes according to specifics of the query. 2) Disease modelers who develop risk equations should add a temporal correction term within their risk equations to prevent model outdating. 3) The fitness score function allows comparing behavior of different populations using a similar metric and provides a visual explanation of current understanding of disease progression.
Potentially access more populations and cover more phenomena
*
*
*
*
RESULTS: The fitness score matrix uses color coding and ranking to visually demonstrate the fitness between 6/34 populations/cohorts and 64 combinations of published risk equations and hypotheses. The results show that different combinations of risk equations behave differently on different population cohorts. For each query, the system ranks the models. Models that implement the following two corrections generally behaved better: 1) Temporal correction for treatment improvement 2) Biomarker change introduced in the first year.
Other Death
Alive
Risk Equation #
ADVANCE Collaborative Group, Intensive Blood Glucose Control and Vascular Outcomes in Patients with Type 2 Diabetes, NEJM. 2008 Jun 12;358(24):2560-72. doi:10.1056/NEJMoa0802987. Epub 2008 Jun 6.
METHODS: 1) Extraction of multiple published disease models/risk equations for cardiovascular disease and multiple diabetic population distributions and outcomes. The populations used were UKPDS, ASPEN, ADVANCE, ACCORD, NDR, KP. 2) Implementation of these components and systematic cross validation of models against populations using micro-simulation and relying on computing power. 3) Defining a fitness score to convert multiple outcome differences into a single number. 4) Defining different queries with weights to rank model / population fitness. The methods avoid using individual data and rely on more accessible summary data.
Death
Stroke Death
Survive Stroke
Population n
1
Example From Table 1 in:
OBJECTIVES: To figure out what disease models equations/hypothesis work best on what populations.
CHD Death
Survive MI
Parameter: BioMarker/State/Treatment Hypothesis 1: Biomarker Change:
ABSTRACT
Process Stroke
No Stroke
Baseline
High Performance Computing
0 0 0 0 0 2 1 1 1 0
0 0 0 0 0 2 1 1 1 1
0 0 0 0 0 2 2 1 1 0
0 0 0 0 0 2 2 1 1 1
0 0 0 0 0 2 3 1 1 0
0 0 0 0 0 2 3 1 1 1
0 0 0 0 0 2 4 1 1 0
0 0 0 0 0 2 4 1 1 1
0 0 0 0 0 3 1 1 1 0
0 0 0 0 0 3 1 1 1 1
0 0 0 0 0 3 2 1 1 0
0 0 0 0 0 3 2 1 1 1
0 0 0 0 0 3 3 1 1 0
0 0 0 0 0 3 3 1 1 1
0 0 0 0 0 3 4 1 1 0
0 0 0 0 0 3 4 1 1 1
0 0 0 0 0 4 1 1 1 0
0 0 0 0 0 4 1 1 1 1
0 0 0 0 0 4 2 1 1 0
0 0 0 0 0 4 2 1 1 1
0 0 0 0 0 4 3 1 1 0
0 0 0 0 0 4 3 1 1 1
0 0 0 0 0 4 4 1 1 0
0 0 0 0 0 4 4 1 1 1
1 1 1 1 1 1 1 1 1 0
1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 2 1 1 0
1 1 1 1 1 1 2 1 1 1
1 1 1 1 1 1 3 1 1 0
1 1 1 1 1 1 3 1 1 1
1 1 1 1 1 1 4 1 1 0
1 1 1 1 1 1 4 1 1 1
1 1 1 1 1 2 1 1 1 0
1 1 1 1 1 2 1 1 1 1
1 1 1 1 1 2 2 1 1 0
1 1 1 1 1 2 2 1 1 1
1 1 1 1 1 2 3 1 1 0
1 1 1 1 1 2 3 1 1 1
1 1 1 1 1 2 4 1 1 0
1 1 1 1 1 2 4 1 1 1
1 1 1 1 1 3 1 1 1 0
1 1 1 1 1 3 1 1 1 1
1 1 1 1 1 3 2 1 1 0
1 1 1 1 1 3 2 1 1 1
1 1 1 1 1 3 3 1 1 0
1 1 1 1 1 3 3 1 1 1
1 1 1 1 1 3 4 1 1 0
1 1 1 1 1 3 4 1 1 1
1 1 1 1 1 4 1 1 1 0
1 1 1 1 1 4 1 1 1 1
1 1 1 1 1 4 2 1 1 0
1 1 1 1 1 4 2 1 1 1
1 1 1 1 1 4 3 1 1 0
1 1 1 1 1 4 3 1 1 1
1 1 1 1 1 4 4 1 1 0
1 1 1 1 1 4 4 1 1 1
40.0 36.7 36.2 14.6 18.4 24.8 25.4 20.5 15.9 10.7 35.1 46.6 28.7 36.0 75.8 80.6 65.3 68.0 55.8 45.2 41.5 44.4 9.2 18.3 42.7 81.5 151.5 60.3 11.3 13.3 40.5 54.5 59.3 44.4
27.6 33.1 31.2 22.9 28.2 28.8 29.4 6.8 16.0 16.6 19.9 24.4 22.6 22.2 45.5 53.8 42.3 43.9 40.0 20.3 15.6 17.3 8.2 14.8 35.8 64.7 123.1 50.2 11.5 17.9 39.3 58.6 55.0 48.2
37.8 33.1 31.4 15.3 20.8 23.7 23.3 18.2 12.4 12.3 28.8 30.8 22.2 35.4 65.6 73.3 64.1 59.6 51.8 45.0 41.9 44.9 11.4 21.0 50.9 84.7 128.0 62.3 15.0 17.8 42.1 51.8 53.3 43.1
32.2 36.0 33.8 22.6 26.5 30.6 32.9 11.6 20.4 16.6 24.3 23.2 23.1 26.1 53.2 59.5 44.0 48.2 50.4 25.6 23.8 23.6 8.3 18.1 41.5 73.4 123.7 54.4 14.0 17.7 42.5 56.2 53.0 42.3
39.2 33.3 34.9 16.7 20.1 24.3 24.6 20.8 13.6 10.5 29.9 36.9 30.1 33.5 72.9 74.2 54.5 63.5 61.2 49.7 49.1 46.9 9.9 24.8 53.7 83.8 138.0 63.5 12.1 16.3 40.3 49.8 51.0 38.4
35.4 36.3 38.6 24.9 31.8 32.4 30.4 15.8 29.1 16.5 28.7 32.2 22.8 26.9 59.8 60.2 45.1 47.2 51.2 31.3 32.4 28.6 10.5 20.5 42.5 72.9 125.8 56.1 13.5 16.7 38.5 52.2 54.5 41.7
40.4 35.4 33.9 16.5 20.2 25.8 24.7 20.6 12.7 10.7 31.0 38.6 26.8 34.7 78.2 77.6 60.0 60.3 53.1 50.5 45.4 48.9 10.5 25.1 53.8 88.9 164.2 67.7 13.2 14.9 39.3 52.9 52.0 43.8
32.6 34.7 37.1 23.5 28.1 28.4 33.0 12.7 20.9 16.4 25.4 29.6 22.2 25.6 51.6 58.2 48.2 48.6 46.4 28.8 27.4 29.0 9.2 19.2 41.5 80.0 145.8 58.8 15.0 15.0 36.1 52.7 50.8 44.3
38.2 34.3 34.6 11.5 18.4 17.6 17.3 18.7 25.7 8.8 54.4 55.3 54.1 63.2 77.1 81.9 60.6 68.8 55.1 58.9 53.7 60.2 10.4 18.4 42.3 92.1 162.2 64.2 10.9 9.6 28.6 46.5 76.6 35.0
33.9 42.7 40.5 17.9 28.3 22.3 27.6 16.9 41.5 17.2 44.6 45.1 43.8 46.0 49.1 57.0 43.8 49.3 50.9 32.1 30.3 32.0 10.3 17.7 34.0 72.2 141.9 55.8 12.5 12.6 33.6 55.7 76.7 41.4
41.1 33.8 35.4 12.0 16.9 14.6 19.0 13.2 22.8 6.9 46.3 48.7 48.5 56.0 75.6 76.7 60.5 64.1 54.9 60.2 56.6 56.9 11.4 21.6 54.3 96.8 140.4 71.8 12.9 12.1 30.6 46.5 61.4 34.0
39.0 45.8 43.3 20.0 29.5 23.6 26.2 17.0 43.0 15.4 41.6 45.3 43.4 49.6 60.0 60.4 50.8 55.2 52.4 39.7 36.3 36.7 10.8 21.8 46.2 85.4 149.5 60.8 14.1 16.1 33.9 60.3 77.4 40.2
38.5 31.8 33.4 12.9 14.6 17.7 15.6 15.6 28.3 7.5 50.5 52.4 50.6 57.6 70.2 72.9 53.9 60.0 50.3 64.8 62.0 64.6 12.3 22.9 51.5 91.6 151.8 69.3 10.9 12.4 26.9 47.3 58.6 34.7
39.0 46.2 44.9 21.2 29.0 21.7 25.6 27.4 49.1 16.4 52.3 51.2 46.9 58.9 59.5 66.2 55.0 54.3 54.8 48.8 45.5 50.9 12.2 20.1 47.0 89.4 152.3 64.3 12.7 14.6 30.1 59.4 70.6 46.8
40.7 30.6 31.9 11.0 17.3 14.7 16.0 14.1 22.6 7.9 49.2 60.0 48.6 58.3 73.5 70.8 57.7 60.5 59.2 65.4 59.1 64.6 12.7 22.9 50.8 102.0 180.9 72.0 11.2 12.3 27.7 49.8 60.7 32.4
33.1 43.2 44.0 19.9 27.8 23.3 24.2 19.4 42.9 15.6 45.7 51.0 47.1 52.6 57.7 59.6 48.3 52.1 56.4 45.5 39.2 44.1 11.4 23.0 45.0 94.0 169.1 63.2 11.8 11.3 32.2 55.2 73.7 44.7
37.9 43.8 43.1 23.4 27.0 34.0 32.5 20.2 18.7 17.4 67.2 71.3 70.9 75.4 94.4 92.9 78.2 75.9 65.2 49.8 48.7 50.3 10.6 37.4 64.3 82.6 110.9 70.3 21.8 30.1 57.7 60.7 46.5 48.1
21.6 21.8 22.9 10.7 9.6 8.2 10.0 45.3 25.6 15.4 14.0 9.3 23.3 15.3 15.9 15.3 23.8 18.2 15.0 35.6 33.8 33.7 4.7 5.9 13.4 31.9 88.6 28.7 7.3 1.6 11.1 14.8 37.8 12.7
33.4 43.1 40.3 25.2 29.6 31.4 30.4 14.8 16.0 18.0 58.3 60.4 65.9 71.4 85.7 89.1 68.0 73.9 64.7 53.0 46.2 49.9 14.6 41.9 75.5 83.1 93.5 68.9 20.6 30.0 54.5 56.3 26.2 48.9
30.0 27.9 28.4 11.0 11.6 11.5 12.2 47.2 28.8 16.6 10.6 8.4 20.6 13.9 18.4 19.5 21.6 20.8 12.5 34.7 35.0 35.1 6.3 10.2 24.0 37.2 87.7 34.0 9.2 9.2 10.4 5.4 38.0 9.1
34.3 47.7 39.0 23.0 29.1 32.6 29.8 16.1 18.6 16.6 62.5 65.6 64.1 72.0 81.2 88.3 69.2 78.3 61.6 54.2 53.0 54.3 13.6 41.4 73.8 89.3 97.5 69.8 20.1 30.5 56.4 59.2 20.8 49.6
35.3 28.5 30.0 12.7 11.4 7.3 9.3 51.1 32.3 16.5 12.2 12.3 12.6 7.4 22.0 23.8 20.8 18.3 13.9 37.2 36.0 35.9 7.0 10.9 21.4 42.8 95.4 34.2 8.7 7.5 6.9 7.8 36.1 8.0
32.7 41.3 43.4 22.4 30.2 31.6 34.8 16.7 19.0 18.5 67.4 67.0 65.4 76.9 84.1 90.5 76.3 76.3 64.2 53.8 54.4 53.2 12.8 44.6 75.1 95.2 130.4 75.7 20.4 32.0 58.6 57.4 31.3 48.2
30.5 28.9 28.8 12.2 11.5 7.6 9.3 48.0 28.6 16.5 12.1 10.1 14.1 5.2 20.8 17.9 23.1 20.4 11.5 35.6 35.4 34.6 6.6 10.2 20.6 40.3 108.3 36.1 5.9 5.2 7.3 6.3 39.7 7.2
26.7 23.2 24.1 12.4 14.6 21.8 17.1 22.7 12.2 6.8 45.7 45.6 50.8 40.9 64.0 60.6 51.2 48.1 35.8 44.1 35.2 40.7 8.9 21.1 40.4 73.7 136.9 60.0 12.3 12.1 31.8 54.9 78.1 39.8
24.0 22.5 21.1 10.4 10.2 9.8 11.2 37.1 23.8 12.2 12.3 7.9 20.0 19.7 12.8 13.3 18.4 17.0 15.6 30.4 29.7 29.4 6.0 6.7 12.1 32.8 83.9 28.0 6.1 0.7 9.3 17.6 9.6 12.3
25.6 22.2 20.9 13.5 9.0 21.9 17.8 18.2 7.3 10.3 39.8 34.5 49.1 41.6 58.4 50.2 51.4 41.3 37.4 46.6 37.7 41.4 9.8 22.4 43.7 71.7 115.1 59.6 14.0 15.1 34.4 53.4 67.5 40.8
31.0 25.8 26.7 9.8 11.1 8.3 11.1 37.1 25.4 12.6 7.6 7.1 17.3 14.2 18.9 16.2 17.3 15.1 8.5 30.0 31.2 29.7 7.5 9.2 22.4 45.3 94.0 35.0 7.9 9.3 11.6 13.5 7.6 10.4
23.3 23.8 22.5 12.2 13.6 23.5 16.7 18.8 10.4 7.2 42.3 36.3 49.2 43.9 58.8 51.8 51.7 38.3 38.4 51.1 42.9 44.1 13.6 24.1 45.6 73.1 124.4 57.6 14.0 11.6 35.0 56.2 62.3 43.6
34.1 29.8 31.4 10.2 9.8 8.7 8.8 40.8 28.3 11.9 8.9 6.8 11.3 8.1 18.8 17.6 17.0 15.5 9.7 33.2 34.4 33.8 8.2 10.5 23.3 47.6 99.0 36.3 7.5 7.6 7.7 13.9 8.3 7.4
27.5 22.4 23.5 10.9 10.3 21.6 17.9 22.9 10.8 8.8 47.9 36.4 44.2 44.9 62.4 60.8 47.1 40.7 37.0 53.7 44.6 49.3 11.6 21.9 43.8 91.1 151.7 70.9 12.1 13.7 30.0 56.1 72.3 41.1
32.6 27.9 26.5 8.6 11.4 9.0 9.9 39.0 25.8 10.5 10.2 8.5 10.3 9.5 19.5 17.5 17.0 15.4 8.7 30.8 30.9 32.7 7.1 8.2 21.5 50.7 116.2 37.8 7.2 5.9 8.2 12.6 17.7 12.2
39.8 34.7 37.0 16.2 18.0 22.9 19.8 22.0 12.3 11.0 31.9 29.3 21.3 27.4 64.1 64.9 56.7 53.0 47.4 43.3 36.0 40.6 10.3 20.7 44.5 77.1 149.3 62.1 11.7 15.9 38.2 50.9 63.1 45.0
26.2 32.6 31.9 22.7 22.0 29.5 25.5 9.8 11.2 15.4 16.8 20.3 22.9 21.2 41.2 43.6 34.7 35.4 36.2 19.1 15.2 17.9 7.7 16.7 33.7 65.7 120.2 52.1 13.2 13.5 38.3 58.5 51.4 45.0
43.2 33.4 32.1 15.1 17.0 24.9 21.4 19.7 6.9 9.8 21.9 24.6 23.3 26.6 58.7 61.2 49.2 46.5 44.4 44.3 37.2 39.4 11.2 23.1 54.4 85.8 132.0 65.9 14.4 16.7 39.6 54.9 44.9 39.6
31.8 35.4 33.5 21.7 23.3 27.3 25.9 9.9 14.1 17.0 20.0 22.7 21.1 19.6 47.2 47.5 41.9 40.4 39.9 23.4 20.2 23.4 9.1 22.4 42.5 71.2 129.1 53.4 13.2 17.7 40.0 57.2 53.8 50.0
40.6 33.9 36.3 17.5 16.2 24.0 22.0 18.0 13.5 9.6 21.8 27.2 22.2 25.6 61.3 66.0 46.8 46.7 43.4 50.4 41.7 43.3 11.1 23.2 50.7 83.0 137.6 65.1 13.8 16.9 36.6 52.4 48.9 41.7
33.5 41.1 37.2 21.1 25.7 29.7 27.5 20.0 21.6 17.0 21.7 27.4 19.6 23.4 51.6 51.0 39.1 42.9 45.3 32.6 28.5 29.2 10.5 21.5 42.5 78.6 133.2 54.3 12.8 15.1 39.5 56.8 53.0 44.5
41.4 36.2 37.4 14.5 17.8 24.4 21.3 20.2 12.1 7.8 30.1 30.1 23.1 32.3 64.8 63.1 47.4 49.2 46.0 49.5 40.1 45.3 11.6 22.8 51.8 94.6 159.4 64.8 13.8 14.7 34.0 55.0 51.3 45.7
30.7 34.6 34.5 22.7 24.4 31.8 30.8 12.3 16.2 17.6 22.2 27.5 18.4 23.2 43.8 47.0 36.8 38.9 38.3 29.6 26.0 26.0 8.1 19.2 43.6 78.6 150.1 60.9 12.6 16.8 36.2 52.0 55.4 45.6
39.4 31.2 35.6 11.6 14.7 18.4 15.5 18.1 21.4 8.8 45.4 43.0 45.8 43.2 66.0 68.3 54.8 47.3 40.3 58.5 44.2 51.7 10.6 18.4 45.7 92.9 167.3 63.1 11.5 9.8 28.3 47.1 73.6 35.3
32.5 40.1 36.9 21.3 25.3 25.3 23.9 17.5 31.2 18.5 36.4 32.0 36.2 35.2 42.8 43.8 41.3 32.2 35.9 30.9 21.5 28.3 9.2 18.7 36.8 79.3 143.5 54.2 11.8 12.6 28.3 58.3 72.8 41.6
36.6 34.1 33.2 11.6 12.4 14.1 13.9 12.8 18.3 6.5 32.5 35.5 43.7 35.5 57.5 56.9 52.1 40.3 48.4 56.3 43.7 51.2 10.8 26.3 49.9 91.2 147.7 63.3 11.6 13.6 32.0 47.3 63.4 36.9
35.9 44.1 42.7 22.1 23.2 28.0 22.5 18.9 36.2 17.4 33.0 33.7 36.7 34.7 50.4 51.7 42.7 37.0 45.5 37.2 29.7 35.9 11.8 20.7 43.2 88.4 145.8 58.7 15.0 14.1 32.4 62.3 67.7 43.7
37.8 32.1 31.6 15.5 14.5 16.9 15.0 16.1 21.9 8.4 39.2 32.4 41.3 43.4 58.2 54.3 45.1 42.7 45.6 61.2 50.4 57.0 12.0 23.0 48.6 91.5 155.1 63.8 12.4 11.0 29.7 46.4 56.4 36.0
41.8 43.8 43.3 20.7 23.9 23.7 21.1 25.8 39.6 16.2 37.9 38.8 40.6 38.7 49.9 49.9 43.9 36.0 44.8 43.3 37.9 42.7 13.1 20.8 46.0 88.0 151.6 61.8 13.2 12.7 30.7 59.6 71.6 40.5
40.4 30.8 33.6 12.1 14.0 14.7 16.3 14.3 18.9 6.7 43.6 41.6 44.0 44.0 62.0 59.3 45.7 45.8 48.6 62.5 54.0 59.0 12.8 21.5 48.6 91.4 182.2 72.5 12.8 13.0 28.3 47.4 63.3 36.2
38.3 43.4 46.0 21.3 22.2 22.5 22.2 19.0 37.7 15.3 37.3 37.1 36.1 39.7 51.5 48.9 44.1 36.9 43.9 42.4 38.3 37.1 12.0 19.8 45.9 87.8 168.1 67.8 11.6 14.3 35.2 56.2 71.9 46.3
36.0 46.4 43.6 22.7 19.5 33.1 22.9 18.9 11.0 20.1 47.5 51.5 45.1 50.2 73.1 66.9 59.8 57.6 48.8 44.5 35.1 38.3 11.3 36.9 71.0 86.4 109.6 71.9 21.0 29.1 58.7 58.0 47.7 49.3
21.5 23.0 22.4 12.2 12.1 9.1 11.5 49.9 32.0 16.7 18.9 16.6 30.0 18.5 20.9 17.6 27.8 24.6 18.6 37.7 40.0 37.7 4.3 6.1 15.4 31.3 85.4 28.4 6.1 2.8 9.0 16.9 38.5 12.9
35.5 43.3 41.6 20.9 20.1 31.5 22.1 16.1 5.9 18.9 43.7 46.4 49.2 46.4 66.9 62.0 51.2 49.4 52.1 45.6 36.2 39.0 11.9 39.3 73.1 85.1 89.6 68.3 21.9 32.1 55.4 56.8 25.4 52.2
30.4 28.1 29.2 15.8 13.7 8.6 11.3 47.4 31.4 17.4 17.0 11.6 28.5 17.3 20.3 19.1 25.6 22.3 12.6 36.1 38.2 36.9 6.5 11.3 22.8 40.8 89.0 32.4 9.2 9.2 11.5 5.4 41.4 7.7
35.0 44.3 43.5 21.3 19.5 35.7 23.0 17.1 10.7 16.9 46.0 49.0 50.8 51.7 59.5 62.9 57.1 56.2 51.4 48.5 42.0 44.4 13.9 42.0 70.8 86.7 96.5 72.2 18.6 31.2 54.7 59.0 21.9 47.8
34.1 29.8 29.7 15.5 12.4 9.5 10.3 51.4 35.2 17.3 14.6 11.9 16.6 13.5 21.9 19.2 24.2 21.3 14.5 39.8 39.9 39.4 6.4 11.0 23.9 44.3 91.9 35.0 9.5 7.6 9.9 5.8 40.3 7.2
36.0 43.2 44.2 23.0 18.3 32.1 23.3 16.9 10.1 17.9 50.3 48.3 50.0 54.0 70.3 62.7 56.1 55.9 46.7 50.6 41.3 45.2 10.9 41.8 73.6 96.8 131.8 72.9 19.8 29.5 52.2 55.7 31.3 49.9
29.3 28.2 27.4 12.0 13.4 7.5 10.5 49.5 32.3 17.2 14.7 12.0 19.0 10.1 22.5 19.7 24.5 22.8 13.1 38.9 38.7 37.5 5.9 9.0 23.8 43.7 106.2 34.1 7.8 3.7 6.6 7.4 40.8 8.6
Hypothesis 2: Temporal Correction for Treatment Improvement: Each model variation was executed: With Temporal Treatment Correction
Without Temporal Treatment Correction
UKPDS ASPEN
ADVANCE
ACCORD KP
NDR
28.4 21.6 24.1 14.0 19.5 19.9 23.7 22.3 17.4 9.1 62.9 71.7 68.8 78.1 87.4 89.4 78.5 73.0 55.0 46.4 45.4 49.3 8.5 19.4 41.0 73.3 140.3 61.2 11.0 14.0 31.7 51.4 75.7 41.7
23.7 22.2 22.2 12.1 7.6 10.6 11.9 38.2 18.0 12.0 11.2 8.1 19.5 19.2 14.5 19.7 15.8 11.7 11.9 28.9 27.1 28.6 6.6 6.0 14.7 35.4 91.7 28.1 5.3 1.8 11.8 17.2 3.3 14.3
25.2 23.2 23.7 12.1 21.7 20.7 22.7 20.3 9.8 8.9 59.7 62.1 61.7 67.2 78.9 81.0 70.6 73.3 57.5 48.9 42.7 45.1 10.9 19.9 46.6 81.6 119.8 59.1 13.9 13.7 35.0 54.4 70.6 42.6
28.8 26.8 24.5 12.3 10.7 12.5 14.0 39.9 20.5 12.4 8.8 12.8 15.8 18.3 22.1 27.8 19.3 17.2 16.5 28.5 27.0 28.6 7.3 11.4 21.8 42.0 94.6 36.7 9.0 8.3 15.0 10.8 6.0 9.7
27.0 22.6 22.9 12.8 17.3 21.9 20.8 21.9 16.2 7.7 62.8 63.7 61.6 68.0 76.9 84.6 67.7 66.7 55.8 48.5 50.2 49.9 11.4 26.7 47.5 71.9 117.2 62.7 12.7 13.4 34.7 52.7 68.5 42.5
34.9 29.0 31.6 10.2 9.0 7.4 9.9 39.4 25.3 12.5 15.4 19.1 13.1 18.0 27.1 28.0 20.4 20.5 17.3 35.0 31.0 33.2 7.7 10.2 24.2 49.9 99.8 37.1 9.2 6.4 7.9 14.5 6.9 10.6
25.3 21.5 23.1 11.9 17.0 23.6 22.2 23.5 10.8 10.6 59.8 62.8 62.8 72.6 78.3 85.5 70.2 71.4 53.8 52.3 55.0 51.6 12.4 23.2 49.7 89.6 155.8 66.0 11.9 12.3 37.7 54.6 68.7 45.4
31.2 29.1 26.7 9.1 9.9 7.2 11.6 40.3 19.1 11.5 12.2 14.4 13.7 19.5 22.2 28.0 18.0 16.4 11.9 32.8 32.3 29.4 6.8 7.3 21.4 49.8 116.3 37.8 6.9 6.0 9.4 12.5 17.2 7.1
OVERALL MODEL RANKING RESULTS Model Characteristics Sorted
Method_A1c Method_BMI Method_BP Method_Lipids Method_Smoke Method_MI Method_Stroke Method_DeathCHD Method_DeathStroke Method_TimeImprove Weighted Mean
1 1 1 1 1 1 1 1 1 1
0 0 0 0 0 1 1 1 1 1
0 0 0 0 0 4 1 1 1 1
1 1 1 1 1 1 2 1 1 1
1 1 1 1 1 4 1 1 1 1
0 0 0 0 0 1 2 1 1 1
1 1 1 1 1 1 3 1 1 1
0 0 0 0 0 4 2 1 1 1
0 0 0 0 0 4 3 1 1 1
19.384 20.191 21.622 22.664 22.814 22.919 23.265 23.301 23.841
0 0 0 0 0 4 4 1 1 1
1 1 1 1 1 1 4 1 1 1
0 0 0 0 0 1 3 1 1 1
1 1 1 1 1 4 2 1 1 1
1 1 1 1 1 4 3 1 1 1
1 1 1 1 1 4 4 1 1 1
0 0 0 0 0 1 4 1 1 1
1 1 1 1 1 2 1 1 1 1
0 0 0 0 0 2 1 1 1 1
23.92 24.293 24.561 24.584 24.979 24.997 25.048 41.618 43.138
1 1 1 1 1 2 2 1 1 1
0 0 0 0 0 2 2 1 1 1
1 1 1 1 1 1 2 1 1 0
0 0 0 0 0 2 3 1 1 1
1 1 1 1 1 2 4 1 1 1
0 0 0 0 0 2 4 1 1 1
1 1 1 1 1 3 1 1 1 1
1 1 1 1 1 2 3 1 1 1
1 1 1 1 1 1 3 1 1 0
1 1 1 1 1 1 1 1 1 0
0 0 0 0 0 3 1 1 1 1
1 1 1 1 1 2 3 1 1 0
1 1 1 1 1 2 1 1 1 0
1 1 1 1 1 2 2 1 1 0
0 0 0 0 0 2 2 1 1 0
1 1 1 1 1 3 3 1 1 0
1 1 1 1 1 3 2 1 1 0
45.39 45.947 45.984 46.484 46.517 46.637 46.954 47.201 47.606 48.054 48.305 48.784 48.995 49.249 49.797 49.893 50.228
1 1 1 1 1 1 4 1 1 0
1 1 1 1 1 3 2 1 1 1
0 0 0 0 0 1 3 1 1 0
0 0 0 0 0 2 3 1 1 0
0 0 0 0 0 1 2 1 1 0
0 0 0 0 0 1 1 1 1 0
0 0 0 0 0 2 1 1 1 0
1 1 1 1 1 3 1 1 1 0
1 1 1 1 1 2 4 1 1 0
1 1 1 1 1 3 3 1 1 1
0 0 0 0 0 3 3 1 1 0
1 1 1 1 1 3 4 1 1 0
1 1 1 1 1 3 4 1 1 1
0 0 0 0 0 3 2 1 1 0
0 0 0 0 0 3 1 1 1 0
0 0 0 0 0 3 2 1 1 1
0 0 0 0 0 2 4 1 1 0
0 0 0 0 0 3 4 1 1 1
0 0 0 0 0 3 3 1 1 1
50.47 50.575 50.652 51.027 51.147 51.213 51.319 51.614 51.682 51.809 52.312 52.432 52.627 52.801 52.956 53.445 53.539 54.148 54.212
1 1 1 1 1 4 2 1 1 0
0 0 0 0 0 1 4 1 1 0
1 1 1 1 1 4 3 1 1 0
1 1 1 1 1 4 1 1 1 0
0 0 0 0 0 4 2 1 1 0
1 1 1 1 1 4 4 1 1 0
0 0 0 0 0 4 3 1 1 0
54.63 55.122 55.353 55.509 58.025 58.034 58.667 59.531
Best 20 models have temporal treatment correction !
Best Model
0 0 0 0 0 3 4 1 1 0
Temporal treatment correction is a good idea! It accounts for model outdating
0 0 0 0 0 4 1 1 1 0
0 0 0 0 0 4 4 1 1 0
60.4 62.932
Time
Adjustment Time
FITNESS: LOW SCORE = GOOD FITNESS UKPDS33 Conventional UKPDS33 Intensive UKPDS33 Full ASPEN All Placebo ASPEN All Atorvastatin ASPEN Primary Placebo ASPEN Primary Atorvastatin ASPEN Secondary Placebo ASPEN Secondary Atorvastatin ASPEN Full ADVANCE Standard ADVANCE Intensive ADVANCE Asia Standard ADVANCE Asia Intensive ADVANCE EME Standard ADVANCE EME Intensive ADVANCE Eastern Europe Standard ADVANCE Eastern Europe Intensive ADVANCE Full ACCORD BP Standard Therapy ACCORD BP Intensive Therapy ACCORD BP Full KP 20-34 KP 35-50 KP 50-65 KP 65-75 KP 75+ KP Total NDR 20-34 NDR 35-50 NDR 50-65 NDR 65-75 NDR 75+ NDR Total
Simulated Time Stamp
Model Data Time Interval
Fitness Score: Simulated Outcomes at time T
Observed Outcomes at time T
X1
Y1
X2
Y2
Xn
Yn
Better fitness
Z=f(X,Y) Fitness of Model M and Population P
Z
Additional Query information leads to: • Ranking Models per populations • Ranking Populations per model • Overall Model Ranking
Worst fitness
References: • Barhak J., Isaman D.J.M., Ye W., Lee D.: Chronic disease modeling and simulation software. Journal of Biomedical Informatics, Volume 43, Issue 5, October 2010, Pages 791–799, http://dx.doi.org/10.1016/j.jbi.2010.06.003 • Barhak J., The Reference Model in the Mount Hood #6-2012 validation challenge and the uncertainty challenge. The Mt hood challenge 6, June 7-8, 2012. Johns Hopkins Mt. Washington Conference Center. • You Tube Video - The Reference Model for Disease Progression: Online: http://www.youtube.com/watch?v=7qxPSgINaD8 • Barhak J., The Reference Model for Disease Progression. Poster presentation, SciPy 2012, Austin Tx, 18-19 July 2012. SciPy 2012 proceedings can be accessed publically online: https://github.com/Jacob-Barhak/scipy_proceedings/blob/2012/papers/Jacob_Barhak/TheReferenceModelSciPy2012.rst • J. Barhak, The Reference Model for Chronic Disease Progression. 2012 Multiscale Modeling (MSM) Consortium Meeting, October 22-23, 2012, Abstract: http://www.imagwiki.nibib.nih.gov/mediawiki/images/7/77/Reference_Model_for_Chronic_Disease_Progression_Barhak.pdf . Poster: http://www.imagwiki.nibib.nih.gov/mediawiki/images/c/c4/PosterTheReferenceModel_IMAGE_MSM_Submit_2012_10_17.pdf
Acknowledgments: The GPL modeling framework was supported by the Biostatistics and Economic Modeling Core of the MDRTC (P60DK020572) and by the Methods and Measurement Core of the MCDTR (P30DK092926), both funded by the National Institute of Diabetes and Digestive and Kidney Diseases. The modeling framework was initially defined as GPL and was funded by Chronic Disease Modeling for Clinical Research Innovations grant (R21DK075077) from the same institute. The Reference Model was developed independently without financial support.
