Abstract
An issue of substantial importance is the monitoring and improvement of health care facilities such as hospitals, nursing homes, dialysis units or surgical wards. In addressing this, there is a need for appropriate methods for monitoring health outcomes. On the one hand, statistical tools are needed to aid centers in instituting and evaluating quality improvement programs and, on the other hand, to aid overseers and payers in identifying and addressing sub-standard performance. In the latter case, the aim is to identify situations where there is evidence that the facility’s outcomes are outside of normal expectations; such facilities would be flagged and perhaps audited for potential difficulties or censured in some way. Methods in use are based on models where the center effects are taken as fixed or random. We take a systematic approach to assessing the merits of these methods when the patient outcome of interest arises from a linear model. We argue that methods based on fixed effects are more appropriate for the task of identifying extreme outcomes by providing better accuracy when the true facility effect is far from that of the average facility and avoiding confounding issues that arise in the random effects models when the patient risks are correlated with facility effects. Finally, we consider approaches to flagging that are based on the Z-statistics arising from the fixed effects model, but which account in a robust way for the intrinsic variation between facilities as contemplated in the standard random effects model. We provide an illustration in monitoring survival outcomes of dialysis facilities in the US.
Similar content being viewed by others
References
Ash AS, Fienberg SE, Louis TA, Normand ST, Stukel TA, Utts J (2012) Statistical issues in assessing hospital performance. COPSS-CMS White Paper Committee. The report can be found at http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/Downloads/Statistical-Issues-in-Assessing-Hospital-Performance.pdf
Austin PC, Naylor CD, Tu JV (2001) A comparison of a Bayesian vs. a frequentist method for profiling hospital performance. J Eval Clin Pract 7:35–45
Efron B (2004) Large-scale simultaneous hypothesis testing: the choice of a null hypotheses. J Am Stat Assoc 99:96–104
Efron B (2007) Size, power and false discovery rates. Ann Stat 35:1351–1377
Efron B, Morris C (1973) Stein’s estimation rule and its competitors—an empirical Bayes approach. J Am Stat Assoc 68:117–130
Hampel FR, Ronchetti EM, Rousseauw PJ, Stahel WA (1998) Robust statistics: the approach based on influence functions. Wiley, New York
Jones HE, Spiegelhalter DJ (2011) The identification of “unusual” health-care providers from a hierarchical model. Am Stat 65:154–163
Kipnis P, Escobar GJ, Draper D (2010) Effect of choice of estimation method on inter-hospital mortality rate comparisons. Med Care 48:458–465
Liu J, Louis TA, Pan W, Ma JZ, Collins AJ (2003) Methods for estimating and interpreting provider-specific standardized mortality ratios. Health Serv Outcomes Res Methodol 4:135–149
Neuhaus JM, Kalbfleisch JD (1998) Between- and within-cluster covariate effects in the analysis of clustered data. Biometrics 54:638–645
Normand ST, Shahian DM (2007) Statistical and clinical aspects of hospital outcomes profiling. Stat Sci 22:206–226
Normand ST, Glickman ME, Gatsonis CA (1997) Statistical methods for profiling providers of medical care: issues and applications. J Am Stat Assoc 92:803–814
Ohlssen DI, Sharples LD, Spiegelhalter DJ (2006) A hierarchical modelling framework for identifying unusual performance in health care providers. J R Stat Soc A 170:865–890
Pan W (2002) A note on the use of marginal likelihood and conditional likelihood in analyzing clustered data. Am Stat 56:171–174
Shahian DM, Normand ST, Torchiana DF, Lewis SM, Pastore JO, Kuntz RE, Dreyer PI (2001) Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 72:2155–2168
Spiegelhalter DJ (2005) Handling over-dispersion of performance indicators. In: Qual saf health care, vol 14, pp 347–351
Spiegelhalter DJ (2005) Funnel plots for comparing institutional performance. Stat Med 24:1185–1202
Spiegelhalter DJ, Sherlaw-Johnson C, Bardsley M, Blunt I, Wood C, Grigg O (2012) Statistical methods for healthcare regulation: rating screening and surveillance (with discussion). J R Stat Soc A 175:1–25
Acknowledgements
The authors would like to thank Professors Yi Li, Douglas Schaubel and Min Zhang for helpful discussions and comments, and Ms. Rena Sun for carrying out the calculations in Sect. 5. We also acknowledge with thanks the comments from the Editors and Referees on this paper, which helped to improve the presentation. This work was supported in part by contract M000336 from the Centers for Medicare and Medicaid Services (CMS), although the opinions presented here are not necessarily those of the CMS.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kalbfleisch, J.D., Wolfe, R.A. On Monitoring Outcomes of Medical Providers. Stat Biosci 5, 286–302 (2013). https://doi.org/10.1007/s12561-013-9093-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12561-013-9093-x