Skip to main content
SpringerLink
Log in

Influence of Modeling Choices on Value of Information Analysis: An Empirical Analysis from a Real-World Experiment

  • Original Research Article
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Background

Value of information (VOI) analysis often requires modeling to characterize and propagate uncertainty. In collaboration with a cancer clinical trial group, we integrated a VOI approach to assessing trial proposals.

Objective

This paper aims to explore the impact of modeling choices on VOI results and to share lessons learned from the experience.

Methods

After selecting two proposals (A: phase III, breast cancer; B: phase II, pancreatic cancer) for in-depth evaluations, we categorized key modeling choices relevant to trial decision makers (characterizing uncertainty of efficacy, evidence thresholds to change clinical practice, and sample size) and modelers (cycle length, survival distribution, simulation runs, and other choices). Using a $150,000 per quality-adjusted life-year (QALY) threshold, we calculated the patient-level expected value of sample information (EVSI) for each proposal and examined whether each modeling choice led to relative change of more than 10% from the averaged base-case estimate. We separately analyzed the impact of the effective time horizon.

Results

The base-case EVSI was $118,300 for Proposal A and $22,200 for Proposal B per patient. Characterizing uncertainty of efficacy was the most important choice in both proposals (e.g. Proposal A: $118,300 using historical data vs. $348,300 using expert survey), followed by the sample size and the choice of survival distribution. The assumed effective time horizon also had a substantial impact on the population-level EVSI.

Conclusions

Modeling choices can have a substantial impact on VOI. Therefore, it is important for groups working to incorporate VOI into research prioritization to adhere to best practices, be clear in their reporting and justification for modeling choices, and to work closely with the relevant decision makers, with particular attention to modeling choices.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availability Statement

The datasets generated and/or analyzed during the current study are available from the corresponding author upon reasonable request.

References

  1. von Niederhausern B, Guyatt GH, Briel M, Pauli-Magnus C. Academic response to improving value and reducing waste: a comprehensive framework for INcreasing QUality In patient-oriented academic clinical REsearch (INQUIRE). PLoS Med. 2018;15(6):e1002580.

    Article  Google Scholar 

  2. Moher D, Glasziou P, Chalmers I, Nasser M, Bossuyt PMM, Korevaar DA, et al. Increasing value and reducing waste in biomedical research: who’s listening? Lancet. 2016;387(10027):1573–86.

    Article  PubMed  Google Scholar 

  3. Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. J Health Econ. 1999;18(3):341–64.

    Article  CAS  PubMed  Google Scholar 

  4. Claxton K, Neumann PJ, Araki S, Weinstein MC. Bayesian value-of-information analysis. An application to a policy model of Alzheimer’s disease. Int J Technol Assess Health Care. 2001;17(1):38–55.

    Article  CAS  PubMed  Google Scholar 

  5. Ades AE, Lu G, Claxton K. Expected value of sample information calculations in medical decision modeling. Med Decis Mak. 2004;24(2):207–27.

    Article  CAS  Google Scholar 

  6. Groot Koerkamp B, Spronk S, Stijnen T, Hunink MG. Value of information analyses of economic randomized controlled trials: the treatment of intermittent claudication. Value Health. 2010;13(2):242–50.

    Article  PubMed  Google Scholar 

  7. Briggs A, Sculpher M, Claxton K. Decision modelling for health economic evaluation. Oxford: Oxford University Press; 2006.

    Google Scholar 

  8. Claxton K, Sculpher M, Drummond M. A rational framework for decision making by the National Institute For Clinical Excellence (NICE). Lancet. 2002;360(9334):711–5.

    Article  PubMed  Google Scholar 

  9. Claxton KP, Sculpher MJ. Using value of information analysis to prioritise health research: some lessons from recent UK experience. Pharmacoeconomics. 2006;24(11):1055–68.

    Article  PubMed  Google Scholar 

  10. Kim DD, Trikalinos TA, Wong JB. Leveraging cumulative network meta-analysis and value of information analysis to understand the evolving value of medical research. Med Decis Mak. 2019;39(2):119–29.

    Article  Google Scholar 

  11. Jackson CH, Bojke L, Thompson SG, Claxton K, Sharples LD. A framework for addressing structural uncertainty in decision models. Med Decis Mak. 2011;31(4):662–74.

    Article  Google Scholar 

  12. Bojke L, Claxton K, Sculpher M, Palmer S. Characterizing structural uncertainty in decision analytic models: a review and application of methods. Value Health. 2009;12(5):739–49.

    Article  PubMed  Google Scholar 

  13. Koffijberg H, Rothery C, Chalkidou K, Grutters J. Value of information choices that influence estimates: a systematic review of prevailing considerations. Med Decis Mak. 2018;38(7):888–900.

    Article  Google Scholar 

  14. Steuten L, van de Wetering G, Groothuis-Oudshoorn K, Retel V. A systematic and critical review of the evolving methods and applications of value of information in academia and practice. Pharmacoeconomics. 2013;31(1):25–48.

    Article  PubMed  Google Scholar 

  15. Scoggins JF, Ramsey SD. A national cancer clinical trials system for the 21st century: reinvigorating the NCI Cooperative Group Program. J Natl Cancer Inst. 2010;102(17):1371.

    Article  PubMed  PubMed Central  Google Scholar 

  16. SWOG. About Us. 2018. https://oldweb.swog.org/Visitors/AboutUs.asp. Accessed 8 May 2018.

  17. Bennette CS, Veenstra DL, Basu A, Baker LH, Ramsey SD, Carlson JJ. Development and evaluation of an approach to using value of information analyses for real-time prioritization decisions within SWOG, a Large Cancer Clinical Trials Cooperative Group. Med Decis Mak. 2016;36(5):641–51.

    Article  Google Scholar 

  18. Carlson JJ, Kim DD, Guzauskas GF, Bennette CS, Veenstra DL, Basu A, et al. Integrating value of research into NCI Clinical Trials Cooperative Group research review and prioritization: a pilot study. Cancer Med. 2018;7(9):4251–60.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Meltzer DO, Hoomans T, Chung JW, Basu A. Minimal modeling approaches to value of information analysis for health research. Med Decis Mak. 2011;31(6):E1–22.

    Article  Google Scholar 

  20. Sanders GD, Neumann PJ, Basu A, Brock DW, Feeny D, Krahn M, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA. 2016;316(10):1093–103.

    Article  PubMed  Google Scholar 

  21. Expected Value of Sample Information (EVSI). York: York Health Economics Consortium; 2016. http://www.yhec.co.uk/glossary/expected-value-of-sample-information-evsi/. Accessed 2 June 2019

  22. Ryen L, Svensson M. The willingness to pay for a quality adjusted life year: a review of the empirical literature. Health Econ. 2015;24(10):1289–301.

    Article  PubMed  Google Scholar 

  23. Greenland S. Modeling and variable selection in epidemiologic analysis. Am J Public Health. 1989;79(3):340–9.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Greenland S, Daniel R, Pearce N. Outcome modelling strategies in epidemiology: traditional methods and basic alternatives. Int J Epidemiol. 2016;45(2):565–75.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Lee PH. Is a cutoff of 10% appropriate for the change-in-estimate criterion of confounder identification? J Epidemiol. 2014;24(2):161–7.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Djulbegovic B, Kumar A, Soares HP, Hozo I, Bepler G, Clarke M, et al. Treatment success in cancer: new cancer treatment successes identified in phase 3 randomized controlled trials conducted by the National Cancer Institute-sponsored cooperative oncology groups, 1955 to 2006. Arch Intern Med. 2008;168(6):632–42.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Djulbegovic B, Kumar A, Glasziou PP, Perera R, Reljic T, Dent L, et al. New treatments compared to established treatments in randomized trials. Cochrane Database Syst Rev. 2012;10:MR000024.

    PubMed  PubMed Central  Google Scholar 

  28. Basu A, Meltzer D. Decision criterion and value of information analysis: optimal aspirin dosage for secondary prevention of cardiovascular events. Med Decis Mak. 2018;38(4):427–38.

    Article  Google Scholar 

  29. Johnson N, Lilford RJ, Brazier W. At what level of collective equipoise does a clinical trial become ethical? J Med Ethics. 1991;17(1):30–4.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Latimer NR. Survival analysis for economic evaluations alongside clinical trials—extrapolation with patient-level data: inconsistencies, limitations, and a practical guide. Med Decis Mak. 2013;33(6):743–54.

    Article  Google Scholar 

  31. R Core Team R. A Language and Environment for Statistical Computing. 2018. http://www.R-project.org/. Accessed 20 Oct 2017.

  32. Kim DD, Neumann PJ. Comparative Modeling to Inform Health Policy Decisions: A Step Forward. Ann Intern Med. 2019;. https://doi.org/10.7326/M19-2797.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Philips Z, Claxton K, Palmer S. The half-life of truth: what are appropriate time horizons for research decisions? Med Decis Mak. 2008;28(3):287–99.

    Article  Google Scholar 

  34. Bojke L, Grigore B, Jankovic D, Peters J, Soares M, Stein K. Informing reimbursement decisions using cost-effectiveness modelling: a guide to the process of generating elicited priors to capture model uncertainties. Pharmacoeconomics. 2017;35(9):867–77.

    Article  PubMed  Google Scholar 

  35. Neumann PJ, Kim DD, Trikalinos TA, Sculpher MJ, Salomon JA, Prosser LA, et al. Future directions for cost-effectiveness analyses in health and medicine. Med Decis Mak. 2018;38(7):767–77.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for the Evaluation of Value and Risk in Health, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, 800 Washington St., Box 63, Boston, MA, 02111, USA

    David D. Kim

  2. Department of Pharmacy, University of Washington, Seattle, WA, USA

    Gregory F. Guzauskas, Anirban Basu, David L. Veenstra & Josh J. Carlson

  3. Flatiron Health, New York, NY, USA

    Caroline S. Bennette

  4. Hutchinson Institute for Cancer Outcomes Research, Fred Hutchinson Cancer Research Center, Seattle, WA, USA

    Scott D. Ramsey

Authors
  1. David D. Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gregory F. Guzauskas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Caroline S. Bennette
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anirban Basu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David L. Veenstra
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Scott D. Ramsey
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Josh J. Carlson
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

DDK and JJC conceived and designed the study, and SDR secured funding for this study. DDK and GFG were responsible for conducting empirical analyses. DDK wrote the initial manuscript and acted as overall guarantor for the overall content of this article. All authors elaborated, discussed, and approved the final version of the manuscript submitted for publication.

Corresponding author

Correspondence to David D. Kim.

Ethics declarations

Funding

Financial support for this study was provided in part by an award from the Patient-Centered Outcomes Research Institute (PCORI) [ME-1303-5889; PI: Scott Ramsey]. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, and writing and publishing the report. All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), or its Board of Governors or Methodology Committee.

Conflict of interest

DDK, GFG, CSB, AB, DLV, SDR, and JJC have no other conflicts of interest to declare in the subject matter discussed in this manuscript.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 200 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, D.D., Guzauskas, G.F., Bennette, C.S. et al. Influence of Modeling Choices on Value of Information Analysis: An Empirical Analysis from a Real-World Experiment. PharmacoEconomics 38, 171–179 (2020). https://doi.org/10.1007/s40273-019-00848-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-019-00848-8

Search

Navigation