Evaluating alternative risk-adjustment strategies for surgery

doi:10.1016/j.amjsurg.2004.07.032

The American Journal of Surgery

Volume 188, Issue 5, November 2004, Pages 566-570

Presented at the 28th Annual Symposium of the Association of VA Surgeons, Richmond, Virginia, April 25–27, 2004

https://doi.org/10.1016/j.amjsurg.2004.07.032 Get rights and content

Abstract

Background

Comparison of institutional health care outcomes requires risk adjustment. Risk-adjustment methodology may influence the results of such comparisons.

Methods

We compared 3 risk-adjustment methodologies used to assess the quality of surgical care. Nurse reviewers abstracted data from a continuous sample of 2,167 surgical patients at 3 academic institutions. One risk adjustor was based on medical record data (National Surgical Quality Improvement Program [NSQIP]) whereas the other 2, the DxCG and Charlson Comorbidity Index (CCI), primarily used International Classification of Disease-9 (ICD-9) codes. Risk-assessment scores from the 3 systems were compared with each other and with mortality.

Results

Substantial disagreement was found in the risk assessment calculated by the 3 methodologies. Although there was a weak association between the CCI and DxCG, neither correlated well with the NSQIP. The NSQIP was best able to predict mortality, followed by the DxCG and CCI.

Conclusion

In surgical patients, different risk-adjustment methodologies afford divergent estimates of mortality risk.

Section snippets

Materials and methods

Data for this study were collected as part of the initial feasibility trial of the NSQIP in non–Veteran's Affairs hospitals [1]. In this trial, beginning in 1999, 3 academic medical centers (Emory University, Atlanta, GA; University of Michigan, Ann Arbor, MI; University of Kentucky, Lexington, KY) volunteered the time of a dedicated surgical nurse reviewer who was trained in NSQIP methodology. At each academic center, these nurse reviewers used NSQIP protocols to abstract clinical data from

Results

Of the 3 risk-adjustment systems, the risk scores for CCI and DxCG were relatively highly correlated (ρ = 0.56), whereas the risk score for the NSQIP was essentially uncorrelated with the CCI (ρ = −0.0076) and weakly correlated with the DxCG (ρ = 0.04) (Table 1). This result is likely caused by the similarity in data used by the DxCG and CCI: both are derived from ICD-9 codes. In contrast, the NSQIP is derived from a different data source: hospital medical record review.

The NSQIP and the DxCG

Comments

It is widely acknowledged that patients are shepherded systematically into different treatment protocols and to different providers based on patient characteristics [11], [12]. In particular, more complex problems often are sent to the providers and organizations that are deemed to offer higher quality. As a result, underlying differences in patient populations may distort comparisons of observed differences in the quality of care. For example, when Medicare compared mortality rates across

References (22)

W.R. Best et al.
Identifying patient preoperative risk factors and postoperative adverse events in administrative databases: results from the Department of Veterans Affairs National Surgical Quality Improvement Program
J Am Coll Surg
(2002)
S. Khuri et al.
Risk adjustment of the postoperative mortality for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study
J Am Coll Surg
(1997)
J. Daley et al.
Risk adjustment of the postoperative morbidity rate for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study
J Am Coll Surg
(1997)
J. Daley et al.
Validating risk-adjusted surgical outcomes: site visit assessment of process and structureNational VA Surgical Risk Study
J Am Coll Surg
(1997)
M.E. Charlson et al.
A new method of classifying prognostic comorbidity I longitudinal studies: development and validation
J Clin Epidemiol
(1987)
P.S. Romano et al.
Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: differing perspectives
J Clin Epidemiol
(1993)
O. Birim et al.
Lung resection for non-small-cell lung cancer in patients older than 70: mortality, morbidity, and late survival compared with the general population
Ann Thorac Surg
(2003)
A.S. Fink et al.
The National Surgical Quality Improvement Program in non-VA hospitals: initial demonstration of feasibility
Ann Surg
(2002)
Khuri SF, Daley J, Henderson W, et al. The Department of Veterans Affairs' NSQIP. The first national, validated,...
A. Ash et al.
Finding future high-cost cases: comparing prior cost versus diagnosis-based methods
Health Serv Res
(2001)

Y. Zhao et al.

Disease burden profiles: an emerging tool for managing managed care

Health Care Manage Sci

(2002)

Cited by (44)

Decreased rates of 30-day perioperative complications following ASD-corrective surgery: A modified Clavien analysis of 3300 patients from 2010 to 2014
2019, Journal of Clinical Neuroscience
The Clavien-Dindo grading allows for broad comparison of perioperative surgical complications, and a temporal analysis of complications following ASD-corrective surgery. NSQIP database was utilized from 2010 to 2014 to isolate patients. Complications were stratified by Clavien complication (Cc) grade, and patients grouped by highest Cc grade: I, II, III, IV, V. Secondary analysis grouped by minor (I, II, III) and severe (IV, V). Comorbidity burden was assessed with a NSQIP-modified Charlson Comorbidity Index (CCI) and frailty was measured with a 5-factor modified frailty index (mFI). From 2010 to 2014, 2971 patients (57 yrs, 58% F) underwent surgery for ASD (3.4 ± 4.1 levels; surgical approach: 46% anterior, 44% posterior, 10% combined), the rate of which increased 0.01% to 0.13. 32% suffered >1 complication. Patient breakdown by Cc grade: 0% I, 25% II, 3% III, 4% IV, 1% V. Severe Cc patients were more comorbid than minor Cc (CCI 2.8 vs 1.8), had longer operative times (394 min vs 251), and higher rates of osteotomy (29% vs 13%) and iliac fixation (16% vs 5%). Overall CCI (2.1–1.7) and perioperative complication rates (55–29%) decreased, despite increasing surgical invasiveness (2.8–4.5) and increasing frailty score (0.14 ± 0.15 vs 0.16 ± 0.16). Rates of Clavien grade II (39.80–22.20%) and IV (9.40–3.50%) complications also decreased, indicative of surgical improvements and effective preoperative patient selection. The decrease in CCI and increase in the modified frailty score may show that we are becoming more cognizant of discerning of comorbidities, but likely to not to have taken into account frailty, which may have an impact on future health socioeconomics.
Usefulness of administrative databases for risk adjustment of adverse events in surgical patients
2016, Cirugia Espanola
Conocer la validez de las bases de datos clínico-administrativas para realizar ajustes de riesgo en el estudio de los efectos adversos que pueden sufrir los pacientes intervenidos quirúrgicamente.
Se estudiaron 1.602 episodios de hospitalización de pacientes intervenidos quirúrgicamente durante los años 2008 y 2010 en un hospital terciario situado en el norte de España. Se analizaron 40 variables de comorbilidades que recoge el Colegio Americano de Cirujanos en el National Surgical Quality Improvement Program (NSQIP) a través de 2 fuentes de información: el conjunto mínimo básico de datos (CMBD) y los datos extraídos de la historia clínica completa (HC), considerada como patrón oro. Se catalogaron las variables en función de que los valores de sensibilidad (S), valor predictivo positivo (VPP) y kappa (K) fueran superiores a 0,6.
La media de comorbilidades por paciente fue de 1,6 mediante la HC y de 0,95 a través del CMBD (p < 0,0001). El CMBD no detectó ningún caso en 13 comorbilidades (estas supusieron el 8% de las comorbilidades detectadas con la HC). De las 27 comorbilidades restantes, 5 cumplieron los 3 criterios establecidos (S, VPP y k > 0,6), 2 enfermedades cumplieron 2 criterios, 12 cumplieron al menos uno y 8 no cumplieron ninguno.
Las bases de datos clínico-administrativas detectaron comorbilidades como diabetes o hipertensión arterial pero no aportaron suficiente información para recoger las variables necesarias para hacer el ajuste de riesgo propuesto por el NSQIP para la medición de los efectos adversos en pacientes quirúrgicos.
The aim of this study was to assess the usefulness of clinical-administrative databases for the development of risk adjustment in the assessment of adverse events in surgical patients.
The study was conducted at the Hospital of Navarra, a tertiary teaching hospital in northern Spain. We studied 1602 hospitalizations of surgical patients from 2008 to 2010. We analysed 40 comorbidity variables included in the National Surgical Quality Improvement (NSQIP) Program of the American College of Surgeons using 2 sources of information: The clinical and administrative database (CADB) and the data extracted from the complete clinical records (CR), which was considered the gold standard. Variables were catalogued according to compliance with the established criteria: sensitivity, positive predictive value and kappa coefficient > 0.6.
The average number of comorbidities per study participant was 1.6 using the CR and 0.95 based on CADB (p < .0001). Thirteen types of comorbidities (accounting for 8% of the comorbidities detected in the CR) were not identified when the CADB was the source of information. Five of the 27 remaining comorbidities complied with the 3 established criteria; 2 pathologies fulfilled 2 criteria, whereas 11 fulfilled 1, and 9 did not fulfil any criterion.
CADB detected prevalent comorbidities such as comorbid hypertension and diabetes. However, the CABD did not provide enough information to assess the variables needed to perform the risk adjustment proposed by the NSQIP for the assessment of adverse events in surgical patients.
Focused Risk Analysis: Regression Model Based on 5,314 Total Hip and Knee Arthroplasty Patients from a Single Institution
2014, Journal of Arthroplasty
We aimed to identify significant demographic, preoperative comorbidity and surgical predictors for major complications for use in the development of a risk prediction tool for a well-defined population as Total Joint Arthroplasty (TJA) patients. Data on 5314 consecutive patients who underwent primary total hip or knee arthroplasty from October 1, 2008 through September 30, 2011 at a single institution were used in a multivariate regression analysis. The overall incidence of a primary endpoint (reoperation during same admission, extended length of stay, and 30-day readmission) was 3.8%. Significant predictors include certain preexisting genitourinary, circulatory and respiratory conditions; ASA >2; advanced age and prolonged operating time. Mental health conditions demonstrate a strong predictive effect for subsequent serious complication(s) in TJA patients and should be included in a risk-adjustment tool.
Optimizing risk-adjusted outcome measures: A moving target.: Invited Commentary on: Variability of NSQIP assessed surgical quality based on age and disease process
2014, Journal of Surgical Research
Improving quality improvement
2012, American Journal of Surgery
Citation Excerpt :
Graphically, the ideal receiver operating curve area is 1. No risk adjustment strategy is ideal, but the NSQIP does well within a receiver operating curve of .94, the DxCG .59, and the CCI .52.14 Hospital discharge data are routinely recorded on medical records.
Racial Differences and Disparities in Cancer Care and Outcomes. Where's the Rub?
2012, Surgical Oncology Clinics of North America

View all citing articles on Scopus

View full text

Scientific paperEvaluating alternative risk-adjustment strategies for surgery

Abstract

Background

Methods

Results

Conclusion

Section snippets

Materials and methods

Results

Comments

J Am Coll Surg

J Am Coll Surg

J Am Coll Surg

J Am Coll Surg

J Clin Epidemiol

J Clin Epidemiol

Ann Thorac Surg

The National Surgical Quality Improvement Program in non-VA hospitals: initial demonstration of feasibility

Ann Surg

Finding future high-cost cases: comparing prior cost versus diagnosis-based methods

Health Serv Res

Disease burden profiles: an emerging tool for managing managed care

Health Care Manage Sci

Scientific paper
Evaluating alternative risk-adjustment strategies for surgery