Original ArticleDecision analytic modeling was useful to assess the impact of a prediction model on health outcomes before a randomized trial
Introduction
Diagnostic or prognostic prediction models can be used to support management decisions such as subsequent testing, treatment, or lifestyle changes. Developed prediction models require external validation to ensure they have adequate predictive performance [1], [2], [3], [4]. However, good predictive performance does not imply that implementation in clinical practice will improve health outcomes or reduce health care costs. The impact of using risk prediction models in clinical practice on patient health and monetary outcomes can be evaluated in impact studies, such as comparative longitudinal (ideally (cluster) randomized) trials, in which care directed by the prediction model is compared with usual care [5], [6], [7], [8], [9], [10].
Impact studies for prediction models are infrequent, most likely due to their complexity, long follow-up, associated high costs, and lack of regulatory requirements [7], [8], [9], [11], [12], [13]. In addition, the benefits observed in such impact studies have typically been smaller than expected or even lacking [14], [15], [16]. An approach using a decision analytic model (DAM) may prove useful, making use of evidence available at the time an impact study is being considered. A DAM could provide insight in the conditions under which a prediction model is likely to result in favorable health outcomes or costs when implemented in clinical practice.
Decision analytic modeling is a method that integrates multiple sources of evidence to assess the downstream cost-effectiveness of applying a prediction model in daily practice [7], [8], [9], [17], [18]. Constructing a DAM forces researchers to think about the pathway through which (multiple alternative) complex interventions can lead to health and monetary benefits, such as variation in the interplay between the model predictions and subsequent patient management based on these predicted risks. DAMs also allow for uncertainty on parameters, such as distribution of predicted probabilities or effectiveness of treatment, to be taken into account. In addition, downstream effects of hypothetical scenarios can be analyzed, by varying values of parameters for which there is little or no evidence. The results are then used to inform decisions for an individual patient or health care policy. DAMs have also been proposed and performed before conducting longitudinal comparative trials to assess impact of (complex) therapeutic interventions and diagnostic tests [19], [20], [21], although they are still rare for diagnostic or prognostic prediction models. An explanation for this could be that using DAMs to assess impact is more complex for prediction models than for interventions, as the former would not only need to include accuracy of predictions but also downstream effects of, for example, benefits and harms of subsequent tests. In addition, lack of available evidence on compliance with management recommendations from a prediction model based on the predicted risk, and informed deviation (ID) from that compliance (i.e., whether there is incremental value of a clinician's experience on top of predictions provided by a model) may also explain the limited number of DAMs assessing impact of prediction models before conducting a formal large-scale, long-term, costly, empirical impact study. Although DAMs are particularly ideal to estimate the impact when evidence is lacking, namely by simulating multiple (hypothetical) scenarios.
In this article, we demonstrate how to assess the potential impact of a prediction model on patient health outcomes and health care costs using a DAM approach, specifically focusing on the effect of compliance with management recommendations. We will use the HEART score prediction model for diagnosis of major adverse cardiac events (MACE) in patients with chest pain as a case study [22]. This article will conclude by providing generic guidance on how to perform a DAM-based assessment to estimate the impact of using a prediction model in daily practice and elaborate on how the results of such DAM can inform the design and conduct of a subsequent prospective comparative prediction model impact study.
Section snippets
Case study
We compared implementation of the HEART score prediction model to usual care in a DAM as an example of how compliance with management recommendations from a prediction model influences the impact of that model on patients' health outcomes, health care costs, and cost-effectiveness. The HEART score provides an excellent example for illustrating the usefulness of a DAM, as model development [22] and several external validations have shown that the HEART score can correctly predict and stratify
Results
In usual care, the average proportion of patients with missed MACE was estimated at 0.016 (95% confidence interval 0.007–0.027) or an average of 16 MACE in discharged patients per 1,000 individuals presenting with chest pain at the ED. The average cost per patient in usual care was €2,870 [29].
Discussion
We have shown how a DAM can be used to estimate the potential health-economic impact of using a diagnostic or prognostic prediction model in practice, using only data and information available before performing a costly, long-term, randomized impact trial. We illustrated this for various hypothetical scenarios if the HEART score prediction model were to be implemented in clinical practice.
Generating a DAM for impact assessment of a prediction model forces researchers to think about its main
CRediT authorship contribution statement
Kevin Jenniskens: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Visualization, Writing - original draft, Writing - review & editing. Ghizelda R. Lagerweij: Data curation, Formal analysis, Investigation, Methodology, Visualization, Writing - original draft, Writing - review & editing. Christiana A. Naaktgeboren: Conceptualization, Investigation, Supervision, Writing - review & editing. Lotty Hooft: Writing - review & editing.
References (53)
- et al.
Prediction models need appropriate internal, internal-external, and external validation
J Clin Epidemiol
(2016) - et al.
External validation is necessary in prediction research: a clinical example
J Clin Epidemiol
(2003) - et al.
A capture-recapture analysis demonstrated that randomized controlled trials evaluating the impact of diagnostic tests on patient outcomes are rare
J Clin Epidemiol
(2012) - et al.
Systematic review of health economic impact evaluations of risk prediction models: stop developing, start evaluating
Value Health
(2017) - et al.
Early stage cost-effectiveness analysis of a BRCA1-like test to detect triple negative breast cancers responsive to high dose alkylating chemotherapy
Breast
(2015) - et al.
A prospective validation of the HEART score for chest pain patients at the emergency department
Int J Cardiol
(2013) - et al.
Barriers and facilitators perceived by physicians when using prediction models in practice
J Clin Epidemiol
(2016) - et al.
Current complications of diagnostic and therapeutic cardiac catheterization
J Am Coll Cardiol
(1988) - et al.
Decision analysis and cost-effectiveness analysis
Semin Spine Surg
(2009) - et al.
Integrated prediction and decision models are valuable in informing personalized decision making
J Clin Epidemiol
(2018)
Early-stage valuation of medical devices: the role of developmental uncertainty
Value Health
Feasibility of the headroom analysis in early economic evaluation of innovative diagnostic technologies with no immediate treatment implications
Value Health
Prognosis Research Strategy (PROGRESS) 3: prognostic model research
PLoS Med
Prognosis and prognostic research: validating a prognostic model
BMJ
Impact of risk scoring on decision-making in symptomatic moderate carotid atherosclerosis
Br J Surg
Implementation and effects of risk-dependent obstetric care in The Netherlands (expect study II): protocol for an impact study
JMIR Res Protoc
Prognosis and prognostic research: application and impact of prognostic models in clinical practice
BMJ
Risk prediction models: II. External validation, model updating, and impact assessment
Heart
Evaluating the impact of prediction models: lessons learned, challenges, and recommendations
Diagn Progn Res
Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration
Ann Intern Med
Impact analysis studies of clinical prediction rules relevant to primary care: a systematic review
BMJ Open
Impact of risk assessments on prophylactic antiemetic prescription and the incidence of postoperative nausea and vomiting: a cluster-randomized trial
Anesthesiology
Procalcitonin-guided use of antibiotics for lower respiratory tract infection
N Engl J Med
Predictive risk stratification model: a randomised stepped-wedge trial in primary care (PRISMATIC)
Health Serv Deliv Res
Methods for the economic evaluation of health care programme
Emerging use of early health technology assessment in medical product development: a scoping review of the literature
Pharmacoeconomics
Cited by (3)
Statistical perspectives on using hepatocellular carcinoma risk models to inform surveillance decisions
2023, Journal of HepatologyDecision Analytic Modeling for Global Clinical Trial Planning: A Case for HIV-Positive Patients at High Risk for Mycobacterium tuberculosis Sepsis in Uganda
2023, International Journal of Environmental Research and Public HealthExternal validation of prognostic models: What, why, how, when and where?
2021, Clinical Kidney Journal
Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
- 1
Shared first author.