Figures
Abstract
Background and Purpose
A case-mix adjustment model has been developed and externally validated, demonstrating promise. However, the model has not been thoroughly tested among populations in China. In our study, we evaluated the performance of the model in Chinese patients with acute stroke.
Methods
The case-mix adjustment model A includes items on age, presence of atrial fibrillation on admission, National Institutes of Health Stroke Severity Scale (NIHSS) score on admission, and stroke type. Model B is similar to Model A but includes only the consciousness component of the NIHSS score. Both model A and B were evaluated to predict 30-day mortality rates in 13,948 patients with acute stroke from the China National Stroke Registry. The discrimination of the models was quantified by c-statistic. Calibration was assessed using Pearson’s correlation coefficient.
Results
The c-statistic of model A in our external validation cohort was 0.80 (95% confidence interval, 0.79–0.82), and the c-statistic of model B was 0.82 (95% confidence interval, 0.81–0.84). Excellent calibration was reported in the two models with Pearson’s correlation coefficient (0.892 for model A, p<0.001; 0.927 for model B, p = 0.008).
Citation: Yu P, Pan Y, Wang Y, Wang X, Liu L, Ji R, et al. (2016) External Validation of a Case-Mix Adjustment Model for the Standardized Reporting of 30-Day Stroke Mortality Rates in China. PLoS ONE 11(11): e0166069. https://doi.org/10.1371/journal.pone.0166069
Editor: Xiaoying Wang, Massachusetts General Hospital, UNITED STATES
Received: June 22, 2016; Accepted: October 22, 2016; Published: November 15, 2016
Copyright: © 2016 Yu et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data are from the China National Stroke Registry, owned by the Tiantan Clinical Trial and Research Center for Stroke. Yilong Wang you can access the dataset by contacting Yilong Wang at 'yilong528@gmail.com'. The database is legally owned by Tiantan Clinical Trial and Research Center for Stroke with privacy of over 20,000 individuals in China and 132 hospitals. However, interested researchers may contact Yuesong Pan (panys2000@163.com) or Yilong Wang (yilong528@gmail.com) for further information. The information of 132 hospital participating in our study can be accessed online in the article "The China National Stroke Registry for patients with acute cerebrovascular events: design, rationale, and baseline patient characteristics" in International Journal of Stroke (doi: 10.1111/j.1747-4949.2011.00584.x).
Funding: This study was funded by the Ministry of Science and Technology and the Ministry of Health of the People’s Republic of China: National Science and Technology Major Project of China (2008ZX09312_008) and State Key Development Program of (for) Basic Research of China (2009CB521905).
Competing interests: The authors have declared that no competing interests exist.
Introduction
Predicting 30-day stroke mortality rates in patients with acute stroke plays an important role in evaluating the prognosis of stroke patients [1,2]. Most previously published models have limited utility in clinical practice, as a result of being validated either in ischemic stroke [3,4] only or in hemorrhagic stroke [5,6] only, rather than in both. Although some models have been validated to accurately predict the mortality after ischemic stroke in Asian populations [7,8], they were not generalizable to both ischemic stroke and hemorrhagic stroke. As a result, these models cannot be used to compare the quality of care among different stroke treatment centers.
Case-mix adjustment models have been developed to meet the need of reporting case mix-adjusted mortality outcomes for stroke treatment services. However, most of these models included multiple variables, including the Oxfordshire community stroke project (OCSP) classification [9,10], plasma glucose levels [11], or those difficult to obtain, such as the GCS verbal score [12]. These limitations may explain why none of these case-mix adjustment models have been incorporated into routine clinical practice.
Recently, Benjamin D. Bray et al developed and validated a model to predict the risk of death for patients with acute stroke within 30 days after admission [13]. This model was derived from a large, prospective national registry of unselected cases of acute stroke in hospitals in England and Wales. The model was relatively simple and easy to implement. Two final models were eventually developed. Model A included age (<60, 60–69, 70–79, 80–89, and ≥90 years), presence of atrial fibrillation on admission, National Institutes of Health Stroke Severity Scale (NIHSS) score on admission, and stroke type (ischemic versus primary intracerebral hemorrhage). Model B was similar to Model A, but included only the consciousness component of the NIHSS score (NIHSS _1A) instead of all of the components comprising the NIHSS score [13]. However, there were few data on whether the model was suitable to use for Asian patients with acute stroke. Research has consistently shown that the incidence of stroke has been surging dramatically in low-to middle income countries. The average onset age for first stroke among individuals in China is approximately 10 years younger than among individuals in Western countries, and individuals in China have a higher percentage of hemorrhagic stroke[14]. Therefore, heterogeneity exists among stroke patients in Asian and Western countries. The aim of our study was to evaluate the accuracy of both model A and model B using data from the China National Stroke Registry (CNSR).
Methods
External validation cohort information
The CNSR is a prospective, nationwide hospital-based stroke registry created between September 2007 and August 2008. Eligible patients were enrolled if they were ≥18 years old and had ischemic stroke, transient ischemic attack, intracerebral hemorrhage, or subarachnoid hemorrhage within 14 days of the index event from 132 hospitals across China. The stroke events were confirmed by brain CT or MRI within 14 days after the onset of symptoms. The design, rationale and baseline characteristics of the CNSR have been reported elsewhere [15]. The registry was approved by the central Institutional Review Board at Beijing Tiantan Hospital. Informed written consent was obtained from all patients or their designated relatives. Authors had no access to identifying participant information during or after data collection. Our present study included only patients with acute ischemic stroke or primary intracerebral hemorrhagic stroke.
Risk factors definition
The baseline severity of neurological impairment was evaluated by the National Institute of Health Stroke Scale (NIHSS) within 24 h after admission. These data were collected via an interview conducted by trained research coordinators. Other data, including patient demographics (e.g., gender, age) and vascular risk factors, were extracted from the medical records. Baseline vascular risk factors included history of hypertension (history of hypertension or antihypertensive drug use), stroke or TIA (defined as being confirmed in a medical chart), dyslipidemia (history of dyslipidemia or lipid-lowering drug use), diabetes mellitus (history of diabetes mellitus or hypoglycemic drug use), atrial fibrillation (history of atrial fibrillation confirmed by at least one electrocardiogram or presence of this type of arrhythmia during hospitalization) and history of coronary heart disease, current or previous smoking, etc.
Outcome measures
The main outcome of interest was all-cause mortality rate within 30 days after admission, which was confirmed via telephone by trained research personnel at Beijing Tiantan Hospital.
Statistical analyses
Age and NIHSS scores were reported as medians (interquartile range); all the other values were percentages. Significance testing of age and NIHSS was performed via t-test for continuous variables and the others were by χ 2 test for categorical variables. Discrimination of the case-mix adjustment model in our study was assessed by receiver operating curve analysis and estimation of the area under the receiver operator curve (c-statistic) and by calibration by plot of observed versus predicted mortality with 10 deciles of predicted risk.
The case-mix adjustment model on which our analysis was based has been published elsewhere [13]. The final two models (Model A and Model B) produced from Bray were both used directly and validated in our study. Comparison between the observed and predicted mortality was assessed by Pearson’s correlation coefficient. Given the large sample size used in our study, we did not use Hosmer and Lemeshow goodness-of-fit test, which is known to be sensitive to sample size, to assess calibration. All analyses above were conducted using SAS (version 9.4; SAS Institute, Cary, NC).
Results
Among the 22,216 patients enrolled in the CNSR, there were 12,415 patients with ischemic stroke and 3,255 patients with primary intracerebral hemorrhagic stroke who consented to participate in follow-up. After excluding 1,722 patients without NIHSS_1A information, a total of 13,948 patients with complete 30-day mortality information were included in our study. Comparing the demographics of CNSR patients with stroke with those in the Sentinel Stroke National Audit Program (Table 1), CNSR patients were younger (median, 67 versus 77 years of age), more likely to be women (68.0% versus 50.2%), were more likely to have had hemorrhagic stroke (20.7% versus 10.5%), and had a lower mortality rate at 30 days (10.9% versus 13.2%). There were no clinically significant differences in baseline characteristics between patients included and those excluded; however, there was a slightly higher proportion of history of stroke or TIA and dyslipidemia among the patients included (Table 2).
Based on the case-mix adjustment model A with full NIHSS score, the c-statistic of our external validation cohort was 0.80 (95% confidence interval, 0.79–0.82). For the case-mix adjustment model B, which included only the NIHSS_1A rather than the full NIHSS, the c-statistic of our external validation cohort was 0.82 (95% confidence interval, 0.81–0.84) (Fig 1). Excellent calibration was reported in the plot of observed versus predicted mortality rates in model A (Pearson correlation coefficient 0.892; P = 0.0005) and model B (Pearson correlation coefficient 0.927; P = 0.008) (Fig 2).
Receiver operating characteristic (ROC) curve analysis of the case-mix adjustment stroke model A(A)and model B (B) in the China National Stroke Registry (CNSR) data set.
Observed vs predicted 30-day mortality after admission of case-mix stroke for model A(A) and model B(B), according to 10 deciles of predicted risk in the external validation sample from the China National Stroke Registry (CNSR) data set. Overall, observed and expected mortality rates were highly correlated (Pearson’s correlation coefficient 0.892 in model A and 0.927 in model B), which indicates good calibration.
Discussion
Our study showed that the two final simple models developed by Benjamin D. Bray et al could be used to predict the 30-day mortality rates in unselected type of stroke patients in the Chinese population. Model B, which included only the consciousness component of the NIHSS score, showed excellent performance, similar to that of Model A, suggesting the importance of the consciousness component of the physical examination. Thus, 30-day mortality might be evaluated based on Model B in stroke patients without complete NIHSS information, which is important in non-stroke centers.
Many studies have shown age to be a potent predictor of mortality after stroke [4,5,11,16–18], including both ischemic and hemorrhagic stroke. Atrial fibrillation is an important marker for cardioembolic stroke, which is associated with poor outcomes in thrombolytic therapy and with high mortality [19], probably due to the large infarction volume, a high risk of hemorrhage transformation, and lack of collateral blood supply. NIHSS score at admission, as a stroke severity marker, is a well-established and important predictor of mortality after stroke [7,20,21]. Although level of consciousness is only a component of the NIHSS score, it can forecast mortality after stroke both in the early phase and the long-term phase [22,23]. It can reflect the severity of stroke, possibly because severe loss of consciousness is often due to brain stem, large artery embolic or hemorrhagic stroke.
Many other case-mix adjustment models predicting mortality after stroke require complex assessments (Table 3), which limits their feasibility in routine stroke care [12,24]. The case-mix adjustment model in our study contains only items that could be easily extracted from the in-hospital medical records in China, and it is simple. Thus, it may be helpful for governments to monitor the stroke care services in different hospitals and for health insurers to analyze the cost-effectiveness of stroke care.
There were limitations to this study. First, the centers participating in the CNSR might not be representative of the varying types of Chinese hospitals because the study recruited more centers in urban areas than in rural areas. Second, there might be selection bias due to the difference in history of stroke or dyslipidemia between the patients included and those excluded. Third, patients with undetermined stroke type were not included in the model derivation and validation; however, CT scanning is very common in China and is available 24 hours a day, 7 days a week in 99.2% of the hospitals, according to the China National Stroke Registry.
Conclusions
In our study, we found that the case-mix adjustment model could be used to accurately predict 30-day mortality in Chinese patients with acute stroke.
Acknowledgments
We thank all participating hospitals, colleagues, nurses, and imaging and laboratory technicians for their contributions to the CNSR.
Author Contributions
- Conceptualization: PY LG Yilong W.
- Formal analysis: YP.
- Funding acquisition: Yongjun W.
- Investigation: PY RJ.
- Methodology: Yilong W. LL.
- Project administration: XM.
- Resources: XM.
- Software: YP.
- Supervision: Yilong W. LG.
- Validation: YP.
- Writing – original draft: PY JJ.
- Writing – review & editing: RJ XT XW.
References
- 1. Carandang R, Seshadri S, Beiser A, Kelly-Hayes M, Kase CS, Kannel WB, et al. (2006) Trends in incidence, lifetime risk, severity, and 30-day mortality of stroke over the past 50 years. JAMA 296: 2939–2946. pmid:17190894
- 2. Feigin VL, Lawes CM, Bennett DA, Barker-Collo SL, Parag V (2009) Worldwide stroke incidence and early case fatality reported in 56 population-based studies: a systematic review. Lancet Neurol 8: 355–369. pmid:19233729
- 3. Saposnik G, Kapral MK, Liu Y, Hall R, O'Donnell M, Raptis S, et al. (2011) IScore: a risk score to predict death early after hospitalization for an acute ischemic stroke. Circulation 123: 739–749. pmid:21300951
- 4. Papavasileiou V, Milionis H, Michel P, Makaritsis K, Vemmou A, Koroboki E, et al. (2013) ASTRAL score predicts 5-year dependence and mortality in acute ischemic stroke. Stroke 44: 1616–1620. pmid:23559264
- 5. Hemphill JC 3rd, Bonovich DC, Besmertis L, Manley GT, Johnston SC (2001) The ICH score: a simple, reliable grading scale for intracerebral hemorrhage. Stroke 32: 891–897. pmid:11283388
- 6. Tuhrim S, Dambrosia JM, Price TR, Mohr JP, Wolf PA, Hier DB, et al. (1991) Intracerebral hemorrhage: external validation and extension of a model for prediction of 30-day survival. Ann Neurol 29: 658–663. pmid:1842899
- 7. Zhang N, Liu G, Zhang G, Fang J, Wang Y, Zhang X, et al. (2012) A risk score based on Get With the Guidelines-Stroke program data works in patients with acute ischemic stroke in China. Stroke 43: 3108–3109. pmid:23042659
- 8. Zhang N, Liu G, Zhang G, Fang J, Wang Y, Zhang X, et al. (2013) External validation of the iScore for predicting ischemic stroke mortality in patients in China. Stroke 44: 1924–1929. pmid:23652267
- 9. Myint PK, Clark AB, Kwok CS, Davis J, Durairaj R, Dixit A, et al. (2014) The SOAR (Stroke subtype, Oxford Community Stroke Project classification, Age, prestroke modified Rankin) score strongly predicts early outcomes in acute stroke. Int J Stroke 9: 278–283. pmid:23834262
- 10. Abdul-Rahim AH, Quinn TJ, Alder S, Clark AB, Musgrave SD, Langhorne P, et al. (2016) Derivation and Validation of a Novel Prognostic Scale (Modified-Stroke Subtype, Oxfordshire Community Stroke Project Classification, Age, and Prestroke Modified Rankin) to Predict Early Mortality in Acute Stroke. Stroke 47: 74–79. pmid:26578661
- 11. Wang Y, Lim LL, Levi C, Heller RF, Fischer J (2001) A prognostic index for 30-day mortality after stroke. J Clin Epidemiol 54: 766–773. pmid:11470384
- 12. Counsell C, Dennis M, McDowall M, Warlow C (2002) Predicting outcome after acute and subacute stroke: development and validation of new prognostic models. Stroke 33: 1041–1047.
- 13. Bray BD, Campbell J, Cloud GC, Hoffman A, James M, Tyrrell PJ, et al. (2014) Derivation and external validation of a case mix model for the standardized reporting of 30-day stroke mortality rates. Stroke 45: 3374–3380.
- 14. Liu L, Wang D, Wong KS, Wang Y (2011) Stroke and stroke care in China: huge burden, significant workload, and a national priority. Stroke 42: 3651–3654.
- 15. Wang Y, Cui L, Ji X, Dong Q, Zeng J, Wang Y, et al. (2011) The China National Stroke Registry for patients with acute cerebrovascular events: design, rationale, and baseline patient characteristics. Int J Stroke 6: 355–361.
- 16. Wang Y, Lim LL, Heller RF, Fisher J, Levi CR (2003) A prediction model of 1-year mortality for acute ischemic stroke patients. Arch Phys Med Rehabil 84: 1006–1011.
- 17. Weimar C, Ziegler A, Konig IR, Diener HC (2002) Predicting functional outcome and survival after acute ischemic stroke. J Neurol 249: 888–895.
- 18. Weimar C, Konig IR, Kraywinkel K, Ziegler A, Diener HC, German Stroke Study Collaboration. (2004) Age and National Institutes of Health Stroke Scale Score within 6 hours after onset are accurate predictors of outcome after cerebral ischemia: development and external validation of prognostic models. Stroke 35: 158–162.
- 19. Wang XG, Zhang LQ, Liao XL, Pan YS, Shi YZ, Wang CJ, et al. (2015) Unfavorable Outcome of Thrombolysis in Chinese Patients with Cardioembolic Stroke: a Prospective Cohort Study. CNS Neurosci Ther 21: 657–661.
- 20. Smith EE, Shobha N, Dai D, Olson DM, Reeves MJ, Saver JL, et al. (2013) A risk score for in-hospital death in patients admitted with ischemic or hemorrhagic stroke. J Am Heart Assoc 2: e005207.
- 21. Cheung RT, Zou LY (2003) Use of the original, modified, or new intracerebral hemorrhage score to predict mortality and morbidity after intracerebral hemorrhage. Stroke 34: 1717–1722.
- 22. Henon H, Godefroy O, Leys D, Mounier-Vehier F, Lucas C, Rondepierre P, et al. (1995) Early predictors of death and disability after acute cerebral ischemic event. Stroke 26: 392–398.
- 23. Sheikh K, Brennan PJ, Meade TW, Smith DS, Goldenberg E (1983) Predictors of mortality and disability in stroke. J Epidemiol Community Health 37: 70–74.
- 24. Teale EA, Forster A, Munyombwe T, Young JB (2012) A systematic review of case-mix adjustment models for stroke. Clin Rehabil 26: 771–786.
- 25. Fullerton KJ, Mackenzie G, Stout RW (1988) Prognostic indices in stroke. Q J Med 66: 147–162.
- 26. Allen CM (1984) Predicting the outcome of acute stroke: a prognostic score. J Neurol Neurosurg Psychiatry 47: 475–480.
- 27. Gompertz P, Pound P, Ebrahim S (1994) Predicting stroke outcome: Guy's prognostic score in practice. J Neurol Neurosurg Psychiatry 57: 932–935.
- 28. Frithz G, Werner I (1976) Studies on cerebrovascular strokes. II. Clinical findings and short-term prognosis in a stroke material. Acta Med Scand 199: 133–140.