Elsevier

Journal of Clinical Epidemiology

Volume 115, November 2019, Pages 106-115
Journal of Clinical Epidemiology

Original Article
Decision analytic modeling was useful to assess the impact of a prediction model on health outcomes before a randomized trial

https://doi.org/10.1016/j.jclinepi.2019.07.010Get rights and content

Abstract

Objective

To demonstrate how decision analytic models (DAMs) can be used to quantify impact of using a (diagnostic or prognostic) prediction model in clinical practice and provide general guidance on how to perform such assessments.

Study Design and Setting

A DAM was developed to assess the impact of using the HEART score for predicting major adverse cardiac events (MACE). Impact on patient health outcomes and health care costs was assessed in scenarios by varying compliance with and informed deviation (ID) (using additional clinical knowledge) from HEART score management recommendations. Probabilistic sensitivity analysis was used to assess estimated impact robustness.

Results

Impact of using the HEART score on health outcomes and health care costs was influenced by an interplay of compliance with and ID from HEART score management recommendations. Compliance of 50% (with 0% ID) resulted in increased missed MACE and costs compared with usual care. Any compliance combined with at least 50% ID reduced both costs and missed MACE. Other scenarios yielded a reduction in missed MACE at higher costs.

Conclusion

Decision analytic modeling is a useful approach to assess impact of using a prediction model in practice on health outcomes and health care costs. This approach is recommended before conducting an impact trial to improve its design and conduct.

Introduction

Diagnostic or prognostic prediction models can be used to support management decisions such as subsequent testing, treatment, or lifestyle changes. Developed prediction models require external validation to ensure they have adequate predictive performance [1], [2], [3], [4]. However, good predictive performance does not imply that implementation in clinical practice will improve health outcomes or reduce health care costs. The impact of using risk prediction models in clinical practice on patient health and monetary outcomes can be evaluated in impact studies, such as comparative longitudinal (ideally (cluster) randomized) trials, in which care directed by the prediction model is compared with usual care [5], [6], [7], [8], [9], [10].

Impact studies for prediction models are infrequent, most likely due to their complexity, long follow-up, associated high costs, and lack of regulatory requirements [7], [8], [9], [11], [12], [13]. In addition, the benefits observed in such impact studies have typically been smaller than expected or even lacking [14], [15], [16]. An approach using a decision analytic model (DAM) may prove useful, making use of evidence available at the time an impact study is being considered. A DAM could provide insight in the conditions under which a prediction model is likely to result in favorable health outcomes or costs when implemented in clinical practice.

Decision analytic modeling is a method that integrates multiple sources of evidence to assess the downstream cost-effectiveness of applying a prediction model in daily practice [7], [8], [9], [17], [18]. Constructing a DAM forces researchers to think about the pathway through which (multiple alternative) complex interventions can lead to health and monetary benefits, such as variation in the interplay between the model predictions and subsequent patient management based on these predicted risks. DAMs also allow for uncertainty on parameters, such as distribution of predicted probabilities or effectiveness of treatment, to be taken into account. In addition, downstream effects of hypothetical scenarios can be analyzed, by varying values of parameters for which there is little or no evidence. The results are then used to inform decisions for an individual patient or health care policy. DAMs have also been proposed and performed before conducting longitudinal comparative trials to assess impact of (complex) therapeutic interventions and diagnostic tests [19], [20], [21], although they are still rare for diagnostic or prognostic prediction models. An explanation for this could be that using DAMs to assess impact is more complex for prediction models than for interventions, as the former would not only need to include accuracy of predictions but also downstream effects of, for example, benefits and harms of subsequent tests. In addition, lack of available evidence on compliance with management recommendations from a prediction model based on the predicted risk, and informed deviation (ID) from that compliance (i.e., whether there is incremental value of a clinician's experience on top of predictions provided by a model) may also explain the limited number of DAMs assessing impact of prediction models before conducting a formal large-scale, long-term, costly, empirical impact study. Although DAMs are particularly ideal to estimate the impact when evidence is lacking, namely by simulating multiple (hypothetical) scenarios.

In this article, we demonstrate how to assess the potential impact of a prediction model on patient health outcomes and health care costs using a DAM approach, specifically focusing on the effect of compliance with management recommendations. We will use the HEART score prediction model for diagnosis of major adverse cardiac events (MACE) in patients with chest pain as a case study [22]. This article will conclude by providing generic guidance on how to perform a DAM-based assessment to estimate the impact of using a prediction model in daily practice and elaborate on how the results of such DAM can inform the design and conduct of a subsequent prospective comparative prediction model impact study.

Section snippets

Case study

We compared implementation of the HEART score prediction model to usual care in a DAM as an example of how compliance with management recommendations from a prediction model influences the impact of that model on patients' health outcomes, health care costs, and cost-effectiveness. The HEART score provides an excellent example for illustrating the usefulness of a DAM, as model development [22] and several external validations have shown that the HEART score can correctly predict and stratify

Results

In usual care, the average proportion of patients with missed MACE was estimated at 0.016 (95% confidence interval 0.007–0.027) or an average of 16 MACE in discharged patients per 1,000 individuals presenting with chest pain at the ED. The average cost per patient in usual care was €2,870 [29].

Discussion

We have shown how a DAM can be used to estimate the potential health-economic impact of using a diagnostic or prognostic prediction model in practice, using only data and information available before performing a costly, long-term, randomized impact trial. We illustrated this for various hypothetical scenarios if the HEART score prediction model were to be implemented in clinical practice.

Generating a DAM for impact assessment of a prediction model forces researchers to think about its main

CRediT authorship contribution statement

Kevin Jenniskens: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Visualization, Writing - original draft, Writing - review & editing. Ghizelda R. Lagerweij: Data curation, Formal analysis, Investigation, Methodology, Visualization, Writing - original draft, Writing - review & editing. Christiana A. Naaktgeboren: Conceptualization, Investigation, Supervision, Writing - review & editing. Lotty Hooft: Writing - review & editing.

References (53)

  • A. Girling et al.

    Early-stage valuation of medical devices: the role of developmental uncertainty

    Value Health

    (2010)
  • K.J. Van Nimwegen

    Feasibility of the headroom analysis in early economic evaluation of innovative diagnostic technologies with no immediate treatment implications

    Value Health

    (2014)
  • E.W. Steyerberg et al.

    Prognosis Research Strategy (PROGRESS) 3: prognostic model research

    PLoS Med

    (2013)
  • D.G. Altman et al.

    Prognosis and prognostic research: validating a prognostic model

    BMJ

    (2009)
  • B. Dharmarajah et al.

    Impact of risk scoring on decision-making in symptomatic moderate carotid atherosclerosis

    Br J Surg

    (2014)
  • P. van Montfort et al.

    Implementation and effects of risk-dependent obstetric care in The Netherlands (expect study II): protocol for an impact study

    JMIR Res Protoc

    (2018)
  • K.G. Moons et al.

    Prognosis and prognostic research: application and impact of prognostic models in clinical practice

    BMJ

    (2009)
  • K.G. Moons et al.

    Risk prediction models: II. External validation, model updating, and impact assessment

    Heart

    (2012)
  • T.H. Kappen et al.

    Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

    Diagn Progn Res

    (2018)
  • K.G. Moons et al.

    Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration

    Ann Intern Med

    (2015)
  • E. Wallace et al.

    Impact analysis studies of clinical prediction rules relevant to primary care: a systematic review

    BMJ Open

    (2016)
  • T.H. Kappen et al.

    Impact of risk assessments on prophylactic antiemetic prescription and the incidence of postoperative nausea and vomiting: a cluster-randomized trial

    Anesthesiology

    (2014)
  • D.T. Huang et al.

    Procalcitonin-guided use of antibiotics for lower respiratory tract infection

    N Engl J Med

    (2018)
  • H. Snooks et al.

    Predictive risk stratification model: a randomised stepped-wedge trial in primary care (PRISMATIC)

    Health Serv Deliv Res

    (2018)
  • M. Drummond et al.

    Methods for the economic evaluation of health care programme

    (2005)
  • M. IJzerman et al.

    Emerging use of early health technology assessment in medical product development: a scoping review of the literature

    Pharmacoeconomics

    (2017)
  • Cited by (3)

    Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

    1

    Shared first author.

    View full text