Skip to main content
SpringerLink
Log in

Reporting Guidelines for the Use of Expert Judgement in Model-Based Economic Evaluations

  • Original Research Article
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Introduction

Expert judgement has a role in model-based economic evaluations (EEs) of healthcare interventions. This study aimed to produce reporting criteria for two types of study design to use expert judgement in model-based EE: (i) an expert elicitation (quantitative) study; and (ii) a Delphi study to collate (qualitative) expert opinion.

Methods

A two-round online Delphi process identified the degree of consensus for four core definitions (expert; expert parameter values; expert elicitation study; expert opinion) and two sets of reporting criteria in a purposive sample of experts. The initial set of reporting criteria comprised 17 statements for reporting a study to elicit parameter values and/or distributions and 11 statements for reporting a Delphi survey to obtain expert opinion. Fifty experts were invited to become members of the Delphi process panel by e-mail. Data analysis summarised the extent of agreement (using a pre-defined 75 % ‘consensus’ threshold) on the definitions and suggested reporting criteria. Free-text comments were analysed using thematic analysis.

Results

The final panel comprised 12 experts. Consensus was achieved for the definitions of expert (88 %); expert parameter values (83 %); and expert elicitation study (83 %). The panel recommended criteria to use when reporting an expert elicitation study (16 criteria) and a Delphi study to collate expert opinion (11 criteria).

Conclusion

This study has produced guidelines for reporting two types of study design to use expert judgement in model-based EE: (i) an expert elicitation study requiring 16 reporting criteria; and (ii) a Delphi study to collate expert opinion requiring 11 reporting criteria.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Dakin H, Devlin N, Feng Y, Rice N, O’Neill P, Parkin D. The influence of cost-effectiveness and other factors on NICE decisions. Health Econ. 2015;24(10):1256–71.

    Article  Google Scholar 

  2. Brennan A, Chick SE, Davies R. A taxonomy of model structures for economic evaluation of health technologies. Health Econ. 2006;15(12):1295–310.

    Article  PubMed  Google Scholar 

  3. Excellence NIfC. Guide to the methods of technology appraisal. London: NICE; 2004.

  4. Sculpher MJ, Claxton K, Drummond M, McCabe C. Whither trial-based economic evaluation for health care decision making? Health Econ. 2006;15(7):677–87.

    Article  PubMed  Google Scholar 

  5. Marsden G, Wonderling D. Cost-effectiveness analysis: role and implications. Phlebology. 2013;28(suppl 1):135–40.

    Article  PubMed  Google Scholar 

  6. Annemans L, Genesté B, Jolain B. Early modelling for assessing health and economic outcomes of drug therapy. Value Health. 2000;3(6):427–34.

    Article  CAS  PubMed  Google Scholar 

  7. IJzerman MJ, Steuten LM. Early assessment of medical technologies to inform product development and market access. Appl Health Econ Health Policy. 2011;9(5):331–47.

  8. Iglesias CP. Does assessing the value for money of therapeutic medical devices require a flexible approach? Expert Rev Pharmacoecon Outcomes Res. 2015;15(1):21–32.

    Article  PubMed  Google Scholar 

  9. Morgan MG. Use (and abuse) of expert elicitation in support of decision making for public policy. Proc Natl Acad Sci. 2014;111(20):7176–84. doi:10.1073/pnas.1319946111.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Kuhnert PM, Martin TG, Griffiths SP. A guide to eliciting and using expert knowledge in Bayesian ecological models. Ecol Lett. 2010;13(7):900–14. doi:10.1111/j.1461-0248.2010.01477.x.

    Article  PubMed  Google Scholar 

  11. Mullen PM. Delphi: myths and reality. J Health Organ Manag. 2003;17(1):37–52.

    Article  PubMed  Google Scholar 

  12. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.

    CAS  PubMed  Google Scholar 

  13. Sullivan W, Payne K. The appropriate elicitation of expert opinion in economic models. Pharmacoeconomics. 2011;29(6):455.

    Article  PubMed  Google Scholar 

  14. Rauch W. The decision delphi. Technol Forecast Soc Change. 1979;15(3):159–69.

    Article  Google Scholar 

  15. O’Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, et al. Uncertain judgements eliciting experts’ probabilities. Chichester: Wiley; 2006.

    Book  Google Scholar 

  16. Gosling JP. Methods for eliciting expert opinion to inform health technology assessment. In: Vignette Commissioned by the MRC Methodology Advisory Group. Medical Research Council (MRC) and National Institure for Health Research (NIHR). 2014. https://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjr7Krwu_LMAhWGLsAKHbylCNUQFgghMAA&url=https%3A%2F%2Fwww.mrc.ac.uk%2Fdocuments%2Fpdf%2Fmethods-for-eliciting-expert-opinion-gosling-2014%2F&usg=AFQjCNGhWG6GGK0oAPr0IYtYCBttaC3Jnw&sig2=aRuqYgYzvqJ1ibAH8EDoHA&bvm=bv.122676328,d.ZGg. Accessed 21 June 2016.

  17. Grigore B, Peters J, Hyde C, Stein K. Methods to elicit probability distributions from experts: a systematic review of reported practice in health technology assessment. Pharmacoeconomics. 2013;31(11):991–1003.

    Article  PubMed  Google Scholar 

  18. Evans C, Crawford B. Expert judgement in pharmacoeconomic studies. Pharmacoeconomics. 2000;17(6):545–53.

    Article  CAS  PubMed  Google Scholar 

  19. Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2):e1000217.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311(7001):376–80.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Johnson SR, Tomlinson GA, Hawker GA, Granton JT, Feldman BM. Methods to elicit beliefs for Bayesian priors: a systematic review. J Clin Epidemiol. 2010;63(4):355–69.

    Article  PubMed  Google Scholar 

  22. Kadane J, Wolfson LJ. Experiences in elicitation. J R Stat Soc Ser D Stat. 1998;47(1):3–19.

    Article  Google Scholar 

  23. Kinnersley N, Day S. Structured approach to the elicitation of expert beliefs for a Bayesian-designed clinical trial: a case study. Pharm Stat. 2013;12(2):104–13.

    Article  PubMed  Google Scholar 

  24. Knol AB, Slottje P, van der Sluijs JP, Lebret E. The use of expert elicitation in environmental health impact assessment: a seven step procedure. Environ Health. 2010;9(1):19.

    Article  PubMed  PubMed Central  Google Scholar 

  25. O’Hagan A. SHELF: the SHeffield ELicitation Framework. 2010. http://www.tonyohagan.co.uk/shelf/. Accessed 26 Feb 2016.

  26. Morris DE, Oakley JE, Crowe JA. A web-based tool for eliciting probability distributions from experts. Environ Model Softw. 2014;52:1–4.

    Article  Google Scholar 

  27. Morris E, Oakley J. MATCH uncertainty elicitation tool. 2014. http://optics.eee.nottingham.ac.uk/match/uncertainty.php/. Accessed 26 Feb 2016.

  28. Cantrill J, Sibbald B, Buetow S. The Delphi and nominal group techniques in health services research. Int J Pharm Pract. 1996;4(2):67–74.

    Article  Google Scholar 

  29. Melton LJ 3rd, Thamer M, Ray NF, Chan JK, Chesnut CH 3rd, Einhorn TA, et al. Fractures attributable to osteoporosis: report from the National Osteoporosis Foundation. J Bone Miner Res. 1997;12(1):16–23.

    Article  PubMed  Google Scholar 

  30. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. BMJ. 2013;346:f1049.

    Article  PubMed  Google Scholar 

  31. Evers S, Goossens M, De Vet H, Van Tulder M, Ament A. Criteria list for assessment of methodological quality of economic evaluations: consensus on Health Economic Criteria. Int J Technol Assess Health Care. 2005;21(02):240–5.

    PubMed  Google Scholar 

  32. Ramos MCP, Barton P, Jowett S, Sutton AJ. A systematic review of research guidelines in decision-analytic modeling. Value Health. 2015;18(4):512–29.

    Article  PubMed  Google Scholar 

  33. Lancsar E, Louviere J. Conducting discrete choice experiments to inform healthcare decision making: a user’s guide. Pharmacoeconomics. 2008;26(8):661–77.

    Article  PubMed  Google Scholar 

  34. Oxford Dictionaries. Oxford University Press. 2016. http://www.oxforddictionaries.com. Accessed 26 Feb 2016.

  35. Garthwaite PH, Kadane JB, O’Hagan A. Statistical methods for eliciting probability distributions. J Am Stat Assoc. 2005;100(470):680–701.

    Article  CAS  Google Scholar 

  36. The Free Dictionary. Farlex. 2016. http://www.thefreedictionary.com. Accessed 26 Feb 2016.

  37. Macmillan Dictionary. Macmillan Publishers. 2016. http://www.macmillandictionary.com/. Accessed 26 Feb 2016.

Download references

Acknowledgments

CPI contributed to the design and the conduct of the study, commented on drafts of the manuscript and produced the final version of the manuscript. AT contributed to the design and the conduct of the study, analysed the data, commented on drafts of the manuscript and approved the final version of the manuscript. WR contributed to the design and the conduct of the study, commented on drafts of the manuscript and approved the final version of the manuscript. KP conceived the idea for this study, contributed to the design and the conduct of the study, produced a first draft of the manuscript, commented on subsequent drafts and approved the final version of the manuscript. We also want to thank the following members of the Health Economics Elicitation Group for their contribution to this study: Laura Bojke, Qi Cao, Ali Daneshkhah, Christopher Evans, Bogdan Grigore, Anthony O’Hagan, Vincent Levy, Claire Rothery, Fabrizio Ruggeri, Ken Stein, Matt Stevenson and Patricia Vella Bonanno.

Author information

Authors and Affiliations

  1. Department of Health Sciences, Centre for Health Economics and the Hull and York Medical School, University of York, York, UK

    Cynthia P. Iglesias

  2. Manchester Centre for Health Economics, The University of Manchester, 4th Floor, Jean McFarlane Building, Oxford Road, Manchester, M13 9PL, UK

    Alexander Thompson & Katherine Payne

  3. Helmholtz Zentrum München, German Research Center for Environmental Health, Institute of Health Economics and Health Care Management, Neuherberg, Germany

    Wolf H. Rogowski

  4. Department of Health Care Management, Institute of Public Health and Nursing Research, Health Sciences, University of Bremen, Bremen, Germany

    Wolf H. Rogowski

Authors
  1. Cynthia P. Iglesias
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexander Thompson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wolf H. Rogowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Katherine Payne
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Katherine Payne.

Ethics declarations

No funding was received for this study and the authors of this paper, Cynthia P. Iglesias, Alexander Thompson, Wolf H. Rogowski and Katherine Payne, declare that there is no conflict of interest regarding its publication.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 904 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Iglesias, C.P., Thompson, A., Rogowski, W.H. et al. Reporting Guidelines for the Use of Expert Judgement in Model-Based Economic Evaluations. PharmacoEconomics 34, 1161–1172 (2016). https://doi.org/10.1007/s40273-016-0425-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-016-0425-9

Keywords

Search

Navigation