Skip to main content

Advertisement

Log in

Risk Prediction Models in Perioperative Medicine: Methodological Considerations

Current Anesthesiology Reports Aims and scope Submit manuscript

Abstract

Purpose of Review

Risk prediction models hold enormous potential for assessing surgical risk in a standardized, objective manner. Despite the vast number of risk prediction models developed, they have not lived up to their potential. The aim of this paper is to provide an overview of the methodological issues that should be considered when developing and validating a risk prediction model to ensure a useful, accurate model.

Recent Findings

Systematic reviews examining the methodological and reporting quality of these models have found widespread deficiencies that limit their usefulness.

Summary

Risk prediction modelling is a growing field that is gaining huge interest in the era of personalized medicine. Although there are no shortcuts and many challenges are faced when developing and validating accurate, useful prediction models, these challenges are surmountable, if the abundant methodological and practical guidance available is used correctly and efficiently.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

Papers of particular interest, published recently, have been highlighted as • Of importance •• Of major importance

  1. Nashef SA, Roques F, Sharples LD, et al. EuroSCORE II. Eur J Cardiothorac Surg. 2012;41(4):734–44 discussion 744–735.

    PubMed  Google Scholar 

  2. Moons KGM, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ. 2009;338:b375.

    PubMed  Google Scholar 

  3. Collins GS, Jibawi A, McCulloch P. Control chart methods for monitoring surgical performance: a case study from gastro-oesophageal surgery. Eur J Surg Oncol. 2011;37:473–80.

    CAS  PubMed  Google Scholar 

  4. Altman DG, Vergouwe Y, Royston P, Moons KGM. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:b605.

    PubMed  Google Scholar 

  5. Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. BMJ. 2009;338:b604.

    PubMed  Google Scholar 

  6. Le Manach Y, Collins G, Rodseth R, et al. Preoperative score to predict postoperative mortality (POSPOM): derivation and validation. Anesthesiology. 2016;124(3):570–9.

    PubMed  Google Scholar 

  7. Gurm HS, Seth M, Kooiman J, Share D. A novel tool for reliable and accurate prediction of renal complications in patients undergoing percutaneous coronary intervention. J Am Coll Cardiol. 2013;61:2242–8.

    PubMed  Google Scholar 

  8. Copeland GP, Jones D, Walters M. POSSUM: a scoring system for surgical audit. Br J Surg. 1991;78:355–60.

    CAS  PubMed  Google Scholar 

  9. Vuylsteke A, Pagel C, Gerrard C, et al. The papworth bleeding risk score: a stratification scheme for identifying cardiac surgery patients at risk of excessive early postoperative bleeding. Eur J Cardiothorac Surg. 2011;39:924–30.

    PubMed  Google Scholar 

  10. Wessler BS, Lai YhL, Kramer W, et al. Clinical prediction models for cardiovascular disease: tufts predictive analytics and comparative effectiveness clinical prediction model database. Circ Cardiovasc Qual Outcomes. 2015;8(4):368–75.

    PubMed  PubMed Central  Google Scholar 

  11. Damen JAAG, Hooft L, Schuit E, et al. Prediction models for cardiovascular disease risk in the general population: a systematic review. BMJ. 2016;353:i2416.

    PubMed  PubMed Central  Google Scholar 

  12. Kleinrouweler CE, Cheong-See FM, Collins GS, et al. Prognostic models in obstetrics: available, but far from applicable. Am J Obstet Gynecol 2016;214(1):79–90 e36.

    PubMed  Google Scholar 

  13. Shariat SF, Karakiewicz PI, Margulis V, Kattan MW. Inventory of prostate cancer predictive tools. Curr Opin Urol. 2008;18:279–96.

    PubMed  Google Scholar 

  14. Ettema RG, Peelen LM, Schuurmans MJ, Nierich AP, Kalkman CJ, Moons KG. Prediction models for prolonged intensive care unit stay after cardiac surgery: systematic review and validation study. Circulation. 2010;122(7):682–9 687 p following p 689.

    PubMed  Google Scholar 

  15. Bouwmeester W, Zuithoff NP, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):e1001221.

    PubMed Central  Google Scholar 

  16. • Collins GS, de Groot JA, Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol. 2014;14:40. Provides an overview of the conduct and reporting of external validation studies.

  17. Collins GS, Mallett S, Omar O, Yu LM. Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC Med. 2011;9:103.

    PubMed  PubMed Central  Google Scholar 

  18. Collins GS, Omar O, Shanyinde M, Yu LM. A systematic review finds prediction models for chronic kidney were poorly reported and often developed using inappropriate methods. J Clin Epidemiol. 2013;66:268–77.

    PubMed  Google Scholar 

  19. Nayak S, Edwards DL, Saleh AA, Greenspan SL. Performance of risk assessment instruments for predicting osteoporotic fracture risk: a systematic review. Osteoporos Int. 2014;25(1):23–49.

    CAS  PubMed  Google Scholar 

  20. Altman DG. Prognostic models: a methodological framework and review of models for breast cancer. Cancer Investig. 2009;27(3):235–43.

    Google Scholar 

  21. Collins GS, Michaëlsson K. Fracture risk assessment: state of the art, methodologically unsound, or poorly reported? Curr Osteoporos Rep. 2013;10:199–207.

    Google Scholar 

  22. Roques F, Nashef SAM, Michel P, et al. Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg. 1999;15:816–23.

    CAS  PubMed  Google Scholar 

  23. Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg. 1999;16(1):9–13.

    CAS  PubMed  Google Scholar 

  24. Collins GS, Moons KGM. Comparing risk prediction models. BMJ. 2012;344:e3186.

    PubMed  Google Scholar 

  25. Masconi KL, Matsha TE, Erasmus RT, Kengne AP. Recalibration in validation studies of diabetes risk prediction models: a systematic review. Int J Stat Med Res. 2015;4:347–69.

    Google Scholar 

  26. Toll DB, Janssen KJ, Vergouwe Y, Moons KG. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol. 2008;61(11):1085–94.

    CAS  PubMed  Google Scholar 

  27. • Moons KG, de Groot JA, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014;11(10):e1001744. Provides a framework and gives guidance for conducting systematic reviews of prediction model studies.

    PubMed  PubMed Central  Google Scholar 

  28. Vickers AJ, Cronin AM. Everything you always wanted to know about evaluating prediction models (but were too afraid to ask). Urology. 2010;76(6):1298–301.

    PubMed  Google Scholar 

  29. • Peat G, Riley RD, Croft P, et al. Improving the transparency of prognosis research: the role of reporting, data sharing, registration, and protocols. PLoS Med. 2014;11:e1001671. An article stressing the importance of planning prediction model studies and if possible to register the study and publish the study protocol.

    PubMed  PubMed Central  Google Scholar 

  30. Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med. 2000;19(4):453–73.

    CAS  PubMed  Google Scholar 

  31. Moons KG, Kengne AP, Grobbee DE, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012;98:691–8.

    PubMed  Google Scholar 

  32. Moons KG, Kengne AP, Woodward M, et al. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012;98:683–90.

    PubMed  Google Scholar 

  33. Steyerberg EW, Moons KGM, van der Windt DA, et al. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381.

    PubMed  PubMed Central  Google Scholar 

  34. Tripepi G, Heinze G, Jager KJ, Stel VS, Dekker FW, Zoccali C. Risk prediction models. Nephrol Dial Transplant. 2013;28(8):1975–80.

    PubMed  Google Scholar 

  35. Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014;35:1925–31.

    PubMed  PubMed Central  Google Scholar 

  36. Hickey GL, Blackstone EH. External model validation of binary clinical risk prediction models in cardiovascular and thoracic surgery. J Thorac Cardiovasc Surg. 2016. doi:https://doi.org/10.1016/j.jtcvs.2016.04.023.

    PubMed  Google Scholar 

  37. Kattan MW, Hess KR, Amin MB, et al. American Joint Committee on Cancer acceptance criteria for inclusion of risk models for individualized prognosis in the practice of precision medicine. CA Cancer J Clin. 2016. doi:https://doi.org/10.3322/caac.21339.

    Google Scholar 

  38. Wynants L, Collins GS, van Calster B. Key steps and common pitfalls in developing and validating risk models: a review. BJOG Int J Obst Gynaecol. 2016 (in press).

  39. •• Moons KGM, Altman DG, Reitsma JB, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1–W73. Provides methodological guidance on information to report when publishing a prediction model study.

  40. Harrell FE Jr. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. 2nd ed. New York: Springer; 2015.

    Google Scholar 

  41. • Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. New York: Springer; 2009. Key text book on various aspects on prediction modelling.

  42. Groenwold RH, Moons KG, Pajouheshnia R, et al. Explicit inclusion of treatment in prognostic modelling was recommended in observational and randomised settings. J Clin Epidemiol. 2016. doi:https://doi.org/10.1016/j.jclinepi.2016.03.017.

    PubMed  Google Scholar 

  43. Cheong-See FM, Allotey J, Marlin N, et al. Prediction models in obstetrics: understanding the treatment paradox and potential solutions tothe threat it poses. BJOG Int J Obst Gynaecol. 2016;123:1060–4.

    CAS  Google Scholar 

  44. Biesheuvel CJ, Vergouwe Y, Oudega R, Hoes AW, Grobbee DE, Moons KG. Advantages of the nested case-control design in diagnostic research. BMC Med Res Methodol. 2008;8:48.

    PubMed  PubMed Central  Google Scholar 

  45. Sanderson J, Thompson SG, White IR, Aspelund T, Pennells L. Derivation and assessment of risk prediction models using case-cohort data. BMC Med Res Methodol. 2013;13:113.

    PubMed  PubMed Central  Google Scholar 

  46. Sun GW, Shook TL, Kay GL. Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol. 1996;49(8):907–16.

    CAS  PubMed  Google Scholar 

  47. Peduzzi P, Concato J, Feinsten AR, Holford TR. Importance of events per independent variable in proportional hazards regression analysis. 2. Accuracy and precision of regression estimates. J Clin Epidemiol. 1995;48(12):1503–12.

    CAS  PubMed  Google Scholar 

  48. Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol. 1996;49(12):1373–9.

    CAS  PubMed  Google Scholar 

  49. Ogundimu EO, Altman D, G., Collins GS. Simulation study finds adequate sample size for developing prediction models is not simply related to events per variable. J Clin Epidemiol. 2016 (in press).

  50. Courvoisier DS, Combescure C, Agoritsas T, Gayet-Ageron A, Perneger TV. Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. J Clin Epidemiol. 2011;64:993–1000.

    PubMed  Google Scholar 

  51. Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2016;35:214–26.

    PubMed  Google Scholar 

  52. Vergouwe Y, Steyerberg EW, Eijkemans MJC, Habbema JDF. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol. 2005;58(5):475–83.

    PubMed  Google Scholar 

  53. Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. 2016;74:167–76. doi:https://doi.org/10.1016/j.jclinepi.2015.12.005.

    PubMed  Google Scholar 

  54. Masconi KL, Matsha TE, Echouffo-Tcheugui JB, Erasmus RT, Kengne AP. Reporting and handling of missing data in predictive research for prevalent undiagnosed type 2 diabetes mellitus: a systematic review. EPMA J. 2015;6(1):7.

    PubMed  PubMed Central  Google Scholar 

  55. van der Heijden GJMG, Donders ART, Stijnen T, Moons KGM. Imputation of missing values is superior to complete case analysis and the missing-indicator method in multivariable diagnostic research: a clinical example. J Clin Epidemiol. 2006;59(10):1102–9.

    PubMed  Google Scholar 

  56. Vach W. Some issues in estimating the effect of prognostic factors from incomplete covariate data. Stat Med. 1997;16:57–72.

    CAS  PubMed  Google Scholar 

  57. Little RJA, Rubin DB. Statistical analysis with missing data, vol. 2nd. Hoboken, NJ: Wiley; 2002.

    Google Scholar 

  58. White IR, Royston P, Wood AM. Multiple imputation using chained equations: issues and guidance for practice. Stat Med. 2011;30(4):377–99.

    PubMed  Google Scholar 

  59. Moons KG, Donders RA, Stijnen T, Harrell FE Jr. Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol. 2006;59(10):1092–101.

    PubMed  Google Scholar 

  60. Donders AR, van der Heijden GJ, Stijnen T, Moons KG. Review: a gentle introduction to imputation of missing values. J Clin Epidemiol. 2006;59(10):1087–91.

    PubMed  Google Scholar 

  61. Sterne JAC, White IR, Carlin JB, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393.

    PubMed  PubMed Central  Google Scholar 

  62. Vergouwe Y, Royston P, Moons KGM, Altman DG. Development and validation of a prediction model with missing predictor data: a practical approach. J Clin Epidemiol. 2010;63(2):205–14.

    PubMed  Google Scholar 

  63. Marshall A, Altman DG, Holder RL. Comparison of imputation methods for handling missing covariate data when fitting a Cox proportional hazards model: a resampling study. BMC Med Res Methodol. 2010;10:112.

    PubMed  PubMed Central  Google Scholar 

  64. Marshall A, Altman DG, Holder RL, Royston P. Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines. BMC Med Res Methodol. 2009;9:57.

    PubMed  PubMed Central  Google Scholar 

  65. Turner EL, Dobson JE, Pocock SJ. Categorisation of continuous risk factors in epidemiological publications: a survey of current practice. Epidemiol Perspect Innov. 2010;7:9.

    PubMed  PubMed Central  Google Scholar 

  66. Altman DG. Problems in dichotomizing continuous variables. Am J Epidemiol. 1994;139:442.

    CAS  PubMed  Google Scholar 

  67. • Collins GS, Ogundimu EO, Cook JA, Le Manach Y, Altman DG. Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model. Stat Med. 2016. An article illustrating the loss of predictive accuracy when continuous measurements are categorised.

  68. Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med. 2006;25:127–41.

    PubMed  Google Scholar 

  69. van Walraven C, Hart RG. Leave ‘em alone: why continuous variables should be analyzed as such. Neuroepidemiology. 2008;30:138–9.

    PubMed  Google Scholar 

  70. Vickers AJ, Lilja H. Cutpoints in clinical chemistry: time for fundamental reassessment. Clin Chem. 2009;55:15–7.

    CAS  PubMed  Google Scholar 

  71. Bennette C, Vickers A. Against quantiles: categorization of continuous variables in epidemiologic research, and its discontents. BMC Med Res Methodol. 2012;12:21.

    PubMed  PubMed Central  Google Scholar 

  72. Dawson NV, Weiss R. Dichotomizing continuous variables in statistical analysis: a practice to avoid. Med Decis Mak. 2012;32:225–6.

    Google Scholar 

  73. Royston P, Sauerbrei W. Multivariable model building: a pragmatic approach to regression anaylsis based on fractional polynomials for modelling continuous variables. Chichester: Wiley; 2008.

    Google Scholar 

  74. Royston P, Altman DG. Regression using fractional polynomials of continuous covariates: parsimonious parametric modelling. Appl Stat. 1994;43(3):429–67.

    Google Scholar 

  75. Sauerbrei W, Royston P, Binder H. Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat Med. 2007;26(30):5512–28.

    PubMed  Google Scholar 

  76. Sullivan LM, Massaro JM, D’Agostino RB Sr. Presentation of multivariate data for clinical use: the Framingham study risk score functions. Stat Med. 2004;23:1631–60.

    PubMed  Google Scholar 

  77. Seel RT, Steyerberg EW, Malec JF, Sherer M, Macciocchi SN. Developing and evaluating prediction models in rehabilitation populations. Arch Phys Med Rehabil. 2012;93(Suppl 2):S138–53.

    PubMed  Google Scholar 

  78. Steyerberg EW, Eijkemans MJ, Habbema JD. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol. 1999;52(10):935–42.

    CAS  PubMed  Google Scholar 

  79. van Buuren S, Oudshoorn CGM: Multivariate imputation by chained equations: MICE V1.0 User’s Manual, vol. PG/VGZ/00.038. Leiden: TNO Preventie en Gezonheid; 2000.

  80. Bouwmeester W, Twisk JW, Kappen TH, van Klei WA, Moons KG, Vergouwe Y. Prediction models for clustered data: comparison of a random intercept and standard regression model. BMC Med Res Methodol. 2013;13:19.

    PubMed  PubMed Central  Google Scholar 

  81. Steyerberg EW, Harrell FE Jr, Borsboom GJJM, Eijkemans MJC, Vergouwe Y, Habbema JDF. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54:774–81.

    CAS  PubMed  Google Scholar 

  82. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–87.

    PubMed  Google Scholar 

  83. • Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21(1):128–138. Paper describing many of the key performance measures to calculate when validating a prediction model.

    PubMed  PubMed Central  Google Scholar 

  84. •• Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015;162(1):55–63. Key paper on issues to report when publishing a study developing or validating a prediction model.

    PubMed  Google Scholar 

  85. Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem. 2008;54(1):17–23.

    CAS  PubMed  Google Scholar 

  86. Pencina MJ, D’Agostino RB, Song L. Quantifying discrimination of Framingham risk functions with different survival C statistics. Stat Med. 2012;31:1543–53.

    PubMed  Google Scholar 

  87. Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med. 2014;33(3):517–35.

    PubMed  Google Scholar 

  88. Miller ME, Langefeld CD, Tierney WM, Hui SL, McDonald CJ. Validation of probabilistic predictions. Med Decis Mak. 1993;13(1):49–58.

    CAS  Google Scholar 

  89. Cox DR. Two further applications of a model for binary regression. Biometrika. 1958;45:562–5.

    Google Scholar 

  90. Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6.

    PubMed  PubMed Central  Google Scholar 

  91. Baker SG, Cook NR, Vickers A, Kramer BS. Using relative utility curves to evaluate risk prediction. J R Stat Soc A. 2009;172:729–48.

    Google Scholar 

  92. Kanis JA, Oden A, Johnell O, et al. The use of clinical risk factors enhances the performance of BMD in the prediction of hip and osteoporotic fractures in men and women. Osteoporos Int. 2007;18(8):1033–46.

    CAS  PubMed  Google Scholar 

  93. Kattan MW, Marasco J. What is a real nomogram? Semin Oncol. 2010;37(1):23–6.

    PubMed  Google Scholar 

  94. Collins GS. How can I validate a nomogram? Show me the model. Ann Oncol. 2015;26:1034–5.

    CAS  PubMed  Google Scholar 

  95. Hickey GL, Grant SW, Caiado C, et al. Dynamic prediction modeling approaches for cardiac surgery. Circ Cardiovasc Qual Outcomes. 2013;6:649–58.

    PubMed  Google Scholar 

  96. Strobl AN, Vickers AJ, Van Calster B, et al. Improving patient prostate cancer risk assessment: moving from static, globally-applied to dynamic, practice-specific risk calculators. J Biomed Inform. 2015;56:87–93.

    PubMed  PubMed Central  Google Scholar 

  97. Hickey GL, Grant SW, Murphy GJ, et al. Dynamic trends in cardiac surgery: why the logistic EuroSCORE is no longer suitable for contemporary cardiac surgery and implications for future risk models. Eur J Cardiothorac Surg. 2013;43:1146–52.

    PubMed  Google Scholar 

  98. Collins GS, Altman DG. Design flaws in EuroSCORE II. Eur J Cardiothorac Surg. 2012;43:871.

    PubMed  Google Scholar 

  99. Collins GS, Altman DG. Calibration of EuroSCORE II. Eur J Cardiothorac Surg. 2013;43(3):654.

    PubMed  Google Scholar 

Download references

Funding

Jennifer De Beyer has received research funding through a Grant from Cancer Research UK.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Gary S. Collins.

Ethics declarations

Conflict of Interest

Authors declares that they have no conflict of intrerst.

Human and Animal Rights and Informed Consent

This article does not contain any studies with human or animal subjects performed by any of the authors.

Additional information

This article is part of the Topical collection on Research Methods and Statistical Analyses.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Collins, G.S., Ma, J., Gerry, S. et al. Risk Prediction Models in Perioperative Medicine: Methodological Considerations. Curr Anesthesiol Rep 6, 267–275 (2016). https://doi.org/10.1007/s40140-016-0171-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40140-016-0171-8

Keywords

Navigation