Effectiveness of the APACHE II Scoring System in an Intensive Care Unit: Results of a Prospective Study

Background: The high cost of intensive care unit (ICU) services and limitations in health resources can prompt managers to use predictive scoring systems. Objective: To evaluate the role of APACHE II as a scoring system to predict outcomes and to compare actual and expected mortality rates. Methods: This prospective study was conducted in a10-bed, mixed ICU at Namazi University Hospital, a teaching hospital in Shiraz, Iran. All patients were included consecutively and data were collected during the first 24 h of admission. Statistical analyses were done with SPSS v.16 software. The differences were considered statistically significant at a P value of <0.05. Results: From June to November of 2013, data were available for 110 (61.4%) ICU admission. The patients’ mean (SD) age was 55.1 (17.7) years. Mean APACHE II score was 17.85±7.4. The total standardized mortality ratio (SMR) was 0.92, and the ratio ranged from 0.48 in medical patients to 1.22 in post-elective surgery patients. With the sensitivity 18.5% and specificity 92.8%, the highest correct classification was obtained at predicted death risk 0.6 (74.5%). Conclusion: We recommend the use of this outcome prediction score for decision-making and classifying patients based on the degree of severity of their diseases in our setting. Further work is needed with larger sample sizes to more precisely determine the generalisability of our results and evaluate validity of this outcome prediction score.


INTRODUCTION
The high cost of intensive care unit (ICU) services and limitations in health resources can prompt managers to use predictive scoring systems. The earliest scoring systems for outcome prediction date back to 1863 [1][2][3][4][5][6]. Currently, outcome prediction scoring systems are use for different purposes such as resource allocation, comparison of ICU performance, assessment of patient outcomes and matching in clinical trials [6,7]. The Acute Physiology and Chronic Health Evaluation II (APACHE II) instrument, developed by Knaus in 1985, is one of the tools used most frequently to obtain outcome prediction scores [1][2][3][4][5][6][7][8][9]10].
Many evaluations have been published of outcome prediction scoring systems used for intensive care patients around the world. Some of these studies evaluated the efficacy of care in ICUs with the APACHE II score, and concluded that this index is appropriate for patient classification and mortality prediction [11][12][13][14][15][16]. In our setting we have not previously used specific criteria or scoring systems to evaluate our patients' severity of disease in ICUs. The differences in patient populations, care facilities and ICU staffing have made it difficult to extrapolate the results of other studies to our ICU.
We conducted this study to evaluate the potential usefulness of APACHE II as an outcome prediction scoring system, and to compare actual and expected mortality rates in the ICU of Namazi University Hospital in Shiraz, Iran.

METHODS
This prospective study was conducted from June to November, 2013, in a10-bed, mixed ICU at Namazi University Hospital, a tertiary teaching center in Shiraz, southern Iran. The study method was approved by the institution's Research Ethics Committee. During the study period all patients were included consecutively except for those who were younger than 18 years old, who spent less than 24 hours in the ICU, or whose laboratory data were incomplete.
We designed a data collection form based on the APACHE II scoring system. Basic demographic data were also recorded for all patients, including sex, type of admission (medical, post-elective surgery, post-emergency surgery), diagnosis, length of ICU stay.
After appropriate training, the head nurse of the ICU coordinated data collection for each patient during the first 24 h after admission. For patients who were readmitted, only the first admission was analyzed. Final outcome of the patients was followed until hospital discharge. In post surgery patients the Glasgow Coma Scale was used only after the patient had recovered from the effects of anesthesia. For intubated patients their capacity to understand regardless of their ability to speak was considered to calculate this score.
During the study period, two reviewers checked the data for transcription errors and completeness. At the end of the study period, quality control was carried out by the researchers. During 2012 a pilot study of 30 patients was done to evaluate the methods, definitions and analyses and ensure maximum reliability.
To calculate the APACHE II score, the relevant scores were recorded for age and the values of 12 physiological variables: body temperature, heart rate, mean arterial pressure, respiratory rate, oxygenation, arterial pH, serum sodium, serum potassium, serum creatinine, hematocrit, white blood cell count, Glasgow Coma Scale score, and information on chronic diseases. We calculated the predicted death rate based on the equation developed by Knaus in 1985: (Ln(R/1−R) = −3.517 + (APACHE II score × 0.146) + (0.603, only for post-emergency surgery) + (diagnostic category weight). The standard mortality ratio (SMR) was calculated by dividing the actual mortality by expected mortality.

Statistical Analysis
Continuous variables are reported as mean (standard deviation) and median (inter-quartile) values. Categorical variables are presented as absolute numbers and percentages. One-way ANOVA and Mann-Whitney were used for ordinal variables. The predictive capability of the index was assessed using the receiver operating characteristic (ROC) curve, through a 2x 2 decision matrix. SPSS v. 16 software (SPSS Inc., Chicago, IL, USA) was used for all statistical analyses. The differences were considered statistically significant at a P value of <0.05 for all analyses.

RESULTS
From June to November 2013, a total of 187 patients were admitted to the ICU at ……University Hospital. We excluded 27 patients because they were younger than 18 years of age, were admitted for observation, or spent less than 24 hours in the ICU. Another 50 patients were excluded because of missing data for physiological variables; the result that was missing most frequently was arterial blood gas values. Complete data were available for 110 (61.4%) of the patients admitted to the ICU during the study period. Table 1 summarizes their main characteristics according to medical and surgical status. The patients ranged in age from 16 to 91 years, with a mean (SD) age of 55.1 (17.7) years. Medical patients were on average older than surgical post elective and surgical post emergency patients, 61 years vs. 54 and53 years, respectively. About two thirds of the patients (68) were men and 32 were women; this sex distribution was the same in all three categories of patients. Ninety four patients (85.5%) were admitted to the ICU for surgical indications. The median (inter quartile) length of ICU stay was 7 (3-18) days for medical patients and 6 (3-11) days for post elective and post emergency patients. Mean APACHE II score in our sample of patients was 17.85±7.4. On the basis of diagnostic categories, the lowest APACHE II score was 13.5±3.1 in neurologic patients, and the highest score was 21.1±8.8 in trauma patients.
According to diagnostic category, the most frequent diagnoses were gastrointestinal disease and cancer (22.7%).
We divided the patients into five groups according to their APACH II scores (Fig. 1): score 0-7(5 patients), 8-15(42 patients), 16-23(32 patients), 24-31 (23 patients), and 32(5 patients). All patients in the first group were discharged, whereas only 40% of the patients in the last group were discharged. 3) in post-emergency surgery patients. Actual mortality for the whole sample was 24.5%; the highest percentage was in medical patients (25.0%). Expected mortality in medical patients was 51.7%, which was higher than in postelective surgery (20.6%) and post-emergency surgery patients (26.6%). The overall SMR was 0.92, ranging from 0.48 in medical patients to 1.22 in post-elective surgery patients. Table 3 shows sensitivity, specificity and correct classification for each level of predicted death risk. For death risk level 0.5, the sensitivity was 29.6%, the specificity was 88% and the correct classification was 73.9%. With the sensitivity 18.5% and specificity 92.8%, the highest correct classification was obtained at predicted death risk 0.6(74.5%).

DISCUSSION
Although outcome prediction scoring systems have been widely used and accepted throughout the world, the differences between health care facilities, providers and patient populations make it important to evaluate their validity in different settings [1,17]. This study was designed to use the APACHE II scoring system to compare real and predicted mortality in patients admitted to an ICU, and to evaluate usefulness of this tool for our study population. The mean age of our patients was 55.1±17.7 years. Although medical  patients were older on average than surgical patients, this difference was not statistically significant. Increasing age was associated with higher APACHE II scores and subsequently higher predicted mortality, but a significant association between age and real mortality was not seen. The average age of our patients was higher than in Pakistan (51.2±17.9 years), Brazil (50±19), and another Iranian population in Tehran (49.2±18.4), but lower than in the Netherlands (56±15.9) [1,2,7,18].
In the current study, patients were not equally distributed across the three indications for ICU admission (medical, post-emergency surgery and post-elective surgery). In a study by Knaus et al., 47% of the patients were admitted to the ICU for medical indications, and in a study by Chiavone et al. this proportion was similar. Among our patients, however, only 14.5% had a medical indication for ICU admission. The difference can be explained by the fact that 60% of all beds in our general ICU are assigned to surgical patients [1,10].
Median (inter quartile) length of ICU stay was 9 (3-18)days in patients who died, and 6(3-10) days in those who survived and were discharged from the ICU; this difference, however, was not statistically significant (P=0.49). Mean length of ICU stay in a similar study in our region was 9.95±8.7, with a significant difference between patients who were discharged and those who died [2]. The distribution of patients in different APACHE II score categories showed that most patients were classified in the intermediate categories, a finding consistent with observations from earlier studies [7,10].
As the APACHE II score increased, so did the real death rate. For example, the death rate for patients with a score of 8-14 was 19.0%, whereas for patients with a score of 24-31, the death rate was 34%. These findings are similar to those reported by Naved et al. and Knaus et al. who found a significant relationship between APACHE II scores and real mortality rate [7,10].
So, we can use of this outcome prediction score to stratify such patients based on the degree of severity of their diseases, as showed in the study by knaus et al. and other published studies [1,7,8,10].
In our study the overall observed mortality rate was 24.5%. The observed mortality rates in other studies have ranged from 16.9% to 35.5% [1,15]. Differences in the observed rates may be explained by the fact that in our study we considered only ICU mortality but not hospital mortality as the observed mortality rate. Total expected mortality rate was substantially higher than what was actually recorded, with an SMR of 0.92; however, this value differed in different types of patients. In post-elective surgery patients the expected mortality rate was lower than the actual mortality rate, with an SMR higher than one (SMR: 1.22). Although these results differ from some published studies (1), they are consistent with those in American hospitals where the SMR ranged from 0.59 to 1.58, and those in Europe hospitals, which reported an SMR that ranged from 0.7 to 1.39 [19,20].
In analysis of the receiver operating characteristics curve, area under the curve was 0.651± 0.059, which was higher than the random prediction, but is poor to predict correctly the mortality rate of our patients. It was lower than 0.80, what was reported by Chiavone and Knaus et al. [1].
At the level of 0.5 critical score, we correctly classified 73.6% of patients. The best correction classification prediction was obtained at critical score 0.6, which correctly classification 74.5% of patients. This correction classification less than, which reported in previous studies [1,10].

LIMITATIONS
The following potential limitations should be considered. First, hospital information systems in our country are not designed for research projects, so data mining and data extraction were very difficult and time-consuming. Second, this study was carried out at a single ICU in a tertiary teaching center. Although Namazi University Hospital is the major reference center for southern of Iran, our results cannot be extrapolated to all ICUs in our region.

CONCLUSION
The most obvious finding to emerge from this study is that with increasing APACHE II score, the risk of mortality increased. But discriminative power and predictive capacity of APACHE II were fair. So we recommend the use of this outcome prediction score for to stratify patients based on the degree of severity of their diseases. Further work is needed with larger sample sizes to more precisely determine the generalisability of our results and evaluate validity of this outcome prediction score.