Comparison of Reliability and Validity of the Chinese Four-Level and Three-District Triage Standard and the Australasian Triage Scale

Emergency triage is an important tool for prioritizing urgent or critical patients, and its effect needs to be investigated and evaluated. This observational study aimed to compare the reliability and validity of the Chinese four-level and three-district triage standard (CHT) and the Australasian Triage Scale (ATS) in an adult emergency department of a general hospital in China. From 2016-01 to 2017-01, twelve nurses independently performed on-site triage of 254 patients and 1552 patients to assess the scales' reliability and validity, respectively. The interrater reliability, as assessed by the weighted k scores, was 0.686 (95% CI 0.608–0.757) for the CHT and 0.731 (95% CI 0.663–0.790) for the ATS, and the k scores between the CHT and the ATS were 0.630 (95% CI 0.594–0.669). Temperature, respiration, pulse, blood oxygen saturation, waiting time, treatment time, emergency disposition, hospitalization rate, and mortality were significantly associated with the triage levels of the CHT and ATS (p < 0.001). The area under the receiver operating characteristic (AUROC) curve values of the CHT and ATS for predicting intensive care treatment were 0.845 (95% CI: 0.825–0.866) and 0.740 (95% CI: 0.715–0.765), respectively. The reliability and validity of the CHT and ATS were moderate, and both of them can be used to identify critical patients in emergency departments. It is necessary to further improve the triage system in terms of structure and content.


Introduction
With the increase in population growth, population aging, and hospital survival rates, boarding and crowding in emergency departments (EDs) has become a global problem [1][2][3]. However, patients' reasons for visiting the ED are varied: some because of the urgency of their health status itself and some because of their perceived structural environment and individual motivation [4]. Because of its urgency, indefiniteness, and complexity, emergency medicine faces serious challenges in preventing adverse events. When patients came to seek emergency aid, they first undergo triage. e first assessment of patients by triage nurses is a key factor in determining how the patient's ED experience will progress [4]. Many triage systems have been developed to help doctors and nurses make accurate decisions [5]. It is necessary to analyze the reliability and validity of the triage system to improve its accuracy in the future.
At present, four major emergency triage scales are used to determine the priority of care for a patient: the Australasian Triage Scale (ATS), the Canadian Triage and Acuity Scale (CTAS), the Manchester Triage System (MTS), and the Emergency Severity Index (ESI). Many scholars have studied these scales and have shown that their overall reliability is moderate [6][7][8][9]. However, differences have been revealed in the internal consistency of the system between nurses [7] and between nurses and doctors [8]. Some scholars have also compared and evaluated the reliability and validity of different triage scales, such as the CTAS and ESI, [9] the ESI and MTS, [10] and the MTS and ATS [11]. Few studies have examined the connection between the triage category and the length of stay or mortality rate. It has been reported that the ESI is associated with the length of stay or mortality rate [12] and inversely associated only with 6-month survival [13].
In China, the Ministry of Health announced " e guidelines for the classification of emergency patients (draft for solicitation)" in September 2011, which divided an ED into "three districts" (red, yellow, and green) and the patient's condition into "four levels" (acute and dangerous, severe acute, emergency, and nonemergency) [14]. ese guidelines were simplified as " e Chinese four-level and three-district triage standard" (CHT) [14]. In China, only approximately half of hospitals use the formal scale, predominantly 4-tier (43%) or 5-tier (34%) scales [15]. So far, the implementation of the CHT in hospitals has rarely been reported [16]. us, few data are available that can be used to compare this standard with others. e earliest triage tool, the ATS, was developed in Australia and is widely used [6]. Although its use in China has not been reported, the ATS, like the CHT, was carried out by specifically trained and experienced registered nurses [17]. erefore, we determined and compared the reliability and validity of the CHT and the ATS by analyzing their association with patient physiological status and clinical outcome.

Study Subjects.
is observational study took place in a tertiary academic teaching hospital admitting more than 100 thousand emergency patients each year, which was conducted between January 2016 and January 2017. e hospital's adult ED is dedicated to handling patients older than 14 years. ose who died before hospital admission, refused treatment, or had incomplete data were excluded.
During the study period, all patients arriving at the ED were triaged by emergency nurses, who had at least three years of experience, working in the rescue area. All nurses were approached about participating in the study after being trained to use the CHT and ATS. A total of 12 nurses participated in the study. ere were two nurses (an instructor nurse and an assistant nurse) per shift, who performed the triage duties for emergency patients; and the assignments were randomized by the head nurse. All patients included in the study were registered between 08:00 and 22:00 because there were two nurses on duty during this period. We collected data on 1585 patients according to the 0.01 accuracy of a simulation of power based on 1000 samples [18]. However, 1552 valid samples were ultimately obtained.

Triage Tools
e Chinese Four-Level and ree-District Triage Standard (CHT). According to the CHT [14], each patient was categorized into one of the four acuity levels. Each level defined the urgency of the condition: category 1 (dangerous and acute) � critically ill patients needing to immediately receive life-saving interventions; category 2 (severe acute) � the patient's condition is likely to deteriorate to grade 1 within a short time or may cause serious disability, and treatment should be arranged as soon as possible; category 3 (emergency) � no signs of life-threatening or serious disabilities within a short period, thus, necessitating attention within a certain period of time; and category 4 (nonemergency).
en, based on the acuity level, the nurse assigned the patient to one of the three treatment areas: the red zone, yellow zone, or green zone [14]. e emergency division of this research hospital was as follows: the red zone included the resuscitation room for category 1 and the rescue room for category 2, both of which were open intensive care units and in which patients were treated by the same group of emergency doctors and nurses; the yellow and green areas were emergency clinics for categories 3 and 4. In general, patients assigned to categories 3 and 4 were treated in chronological order. If the patient's condition changed during the waiting period or if the triage nurse considered it necessary, the patient was seen earlier or sent to the red zone immediately.

e Australasian Triage Scale (ATS).
e ATS is a 5level triage system that requires the user to select from a standardized set of patient statuses [19]. It is endorsed by the Australasian College for Emergency Medicine (ACEM). e ATS was translated into Chinese for ease of use in China using Brislin's translation model [20]. First, two bilingual nurses (one of whom was an emergency specialist nurse) translated the original scale into Chinese; then, the Chinese version was back-translated by a third bilingual expert, who did not see the original scale. is process was repeated to ensure that the final Chinese version of the ATS expressed the original intention. e triage assessment involves the presenting problem, the patient's general appearance, and pertinent physiological observations, comprising 79 indicative clinical descriptors. ED patients are allocated to categories 1 through 5 (1: immediate, 2: 10 minutes, 3: 30 minutes, 4: 60 minutes, and 5: 120 minutes) [19]. Any patient identified as ATS category 1 or 2 should be taken immediately into an appropriate assessment and treatment area.

Data Collection and Outcome Definitions.
e CHT has already been used in our ED. Nurses applied the CHT and assessed the patient's general condition, respiratory and circulatory state, and complaint upon arrival to the triage area, and then classified the patient by means of the ATS, which only requires the most important information to be matched with the triage items to determine the triage level.
e first stage of the study involved a reliability analysis. During the first week, the instructing nurse and the assistant nurse used the two scales, the CHT and the ATS, to triage the same patient. Both nurses were blinded to each other's triage assessment and level assignment. During this process, 254 patients were assessed to compare the reliability of the CHT and ATS across different nurses. Next, one of the nurses continued to use the CHT and ATS to triage the same patient. A total of 1298 patients were assessed. e comparison of reliability of the ATS and CHT was based on the 254 samples from the triage performed by the instructing nurse during the first week plus the data from the additional 1298 patients. e doctor receiving time, disposal time, ED admission status, hospital admission rate (including deaths), and length of stay (LOS) were collected from the electronic hospital information systems. Patients who were transferred or signed out to return home were asked to confirm their outcomes via telephone at one month after discharge. e definitions of the outcomes are as follows: waiting time � time from emergency registration to being seen by a doctor; treatment time � time from emergency registration to the administration of intervention measures, such as oxygen, fluids, medications, and physician health advice; length of stay � time from emergency registration to discharge from the hospital; hospitalization rate � the proportion of inpatients to the total number of patients at each triage level; and mortality rate � the proportion of deaths to the total number of patients at each triage level.

Statistical Analyses.
Statistical analysis was performed using SPSS 21.0. First, the collected paperboard data were entered into the SPSS software and checked by two researchers. e degree of reliability between the CHT and the ATS was calculated using the weighted k scores. To facilitate statistical analysis, we merged ATS categories 3 and 4 into category 3 because ATS category 3 was defined as potentially life-threatening and situationally urgent, and category 4 was defined as potentially serious, situationally urgent, and significantly complex or severe, which is equivalent to CHT category 3. ATS category 4 in the figure and the table of this study corresponds to ATS category 5, which is equivalent to CHT category 4 (nonemergency). en, a parametric analysis was performed after verifying the normal distribution of the continuous variables using the Kolmogorov-Smirnov test. All continuous variables had nonnormal distributions, and all were reported as 50th, 25th, and 75th percentiles. Statistical evaluation of the data was performed using the Kruskal-Wallis test, followed by pairwise comparisons. Categorical variables were reported as numbers and proportions. A chi-square test was used to analyze categorical variables, and the Bonferroni method was used to adjust the level of the test. Paired sample analysis was considered statistically significant if the adjusted p value was less than 0.05.

Characteristics of the Study Subjects.
A total of 1585 patients were triaged. Of these, eleven cases involved prehospital death, ten cases involved rejection of treatment, and twelve cases involved incomplete data, leaving 1552 patients for analysis, of which 56.96% were male and 43.04% were female. e patient age ranged from 14 to 99 years, with a median (Q25, Q75) age of 58.0 (44.0, 69.0) years. ere were 1137 internal medicine cases, 133 surgical cases, 219 neurology cases, 31 neurosurgical cases, and 32 others. Fortyfour patients had died at follow-up, when 150 follow-up calls were made one month after discharge.

Reliability Comparison of the Triage Criteria.
Of the 12 triage nurses participating in the study, seven were intermediate nurses (two with more than 20 years of emergency experience) and five were junior nurses in terms of professional title. ere was no difference between the two groups (the instructor nurses and the assistant nurses) in the distribution of professional titles. e reliability analysis of the same triage tool between the two groups of nurses was based on the data from the 254 initial patients. e total consistency k scores was 0.686 (95% CI 0.608-0.757) between the two groups of nurses for the CHT and 0.731 (95% CI 0.663-0.790) for the ATS. e comparison of reliability between the CHT and ATS originated from the data collected when the same nurse applied these two triage tools to allocate the same patient. e overall consistency k scores was 0.654 (95% CI 0.622-0.689) and 0.630 (95% CI 0.594-0.669) when ATS grades 3 and 4 were combined. e results are shown in Table 1.

Distribution of Triage Categories for the Triage Criteria.
e number of patients in each category of the triage criteria is shown in Figure 1. ere were differences in distribution between the CHT group and ATS group (p < 0.001). When the two triages were compared as four-level triages, the distribution of triage categories was shown to be significantly different (p � 0.003). Except for categories 1 and 4, the intercomparisons between the other groups were statistically significant, p < 0.05.

Relationships of Triage Category with Physiological
Indicators and Outcomes. Physiological indicators such as temperature, respiration, pulse, and blood oxygen saturation were strongly associated with the CHT and the ATS triage categories (p < 0.001). After adjusting for multiple comparisons, the respiratory rates of patients in category 1 and category 2 were significantly higher than those of patients in categories 3 and 4 (p < 0.05), and the pulse rate of patients in category 1 was significantly higher than that of patients in the other categories (p < 0.05). e blood oxygen saturation value of category 1 and category 2 was significantly lower than that of category 3 and category 4 patients (p < 0.05).
Similar results were found for the following triage outcomes: waiting time, treatment time, emergency disposition, hospitalization rate, and mortality (p < 0.001). e hospitalization rates of patients classified as category 2 according to both the CHT and ATS were significantly higher than those of patients in categories 3 and 4 (p < 0.05), but the length of stay and mortality were significantly higher for category 1 patients (p < 0.05). e results are shown in Table 2. Figure 2

Discussion
In this study, the reliability of the CHT was similar to that of the ATS. Overall, the CHT triage validity was consistent with that of the ATS. A lower triage category number was related to deterioration of health status, a faster response speed, increased hospitalization time, a higher hospitalization rate, and a higher mortality rate.
According to the Fleiss grade [21] and using 0.65-0.70 as an acceptable range, it was suggested that the CHT and ATS had good reliability. e internal consistency k scores of the ATS was 0.731, which was consistent with the k scores of 0.64∼0.82 in the ATS scenario evaluation [22,23] and slightly higher than that in the CHT k score of 0.686 in this study. Nurses tended to combine the CHT with clinical experience because the CHT is a comprehensive assessment based on vital signs rather than precise digital evaluation. Nevertheless, the ATS has corresponding items, the nurses matched the patient's condition with each item. However, the AUROC curve value of the CHT 0.845 was slightly higher than the ATS value of 0.740, as shown in Figure 2. In this study, two nurses independently triaged the same patient on the spot at the same time, and the results fully reflected the actual situation.
Based on the analysis of the population distribution, CHT and ATS category 2 and category 3 patients accounted for 30% to 40% and 50% to 60% of patients, respectively, indicating that relatively large numbers of potential crisis or emergency patients visited the ED of this tertiary hospital.
is provincial general hospital is one of the main acceptance points for the transfer of critically ill patients from regional hospitals. Most of the patients were middle aged or elderly, with an average age of 58.0 (44.0, 69.0) years. Some data have shown that the rate of emergency visits has been increasing every year, especially for the elderly and for patients in triage categories 2 and 3 [2,24]. is finding reflects not only the population increasing and aging but also the urgency and complexity of patients' conditions; such patients require more observation and treatment than can be provided in outpatient visits.
In the past, many studies evaluated triage validity in terms of hospitalization rate, mortality rate, medical resources, and medical expenses [10,11,25]. is study also evaluated physiological indicators, hospitalization rate, mortality rate, and other parameters. Respiration, pulse rate, and oxygen saturation were closely related to different categories, but there was no significant correlation in blood  pressure. Earlier studies also showed that the worsening of vital signs, such as awareness, breathing, and oxygen saturation, increased the probabilities of ICU stay and death [26,27]. Some studies have reported that systolic blood pressure is closely related to disease severity. Hypotension (<90 mmHg) is closely associated with hospitalization and death [28]; conversely, hypertension has a protective effect [27]. No difference in blood pressure by triage category was found in our study, possibly because no stratification analysis was performed. Age (  e waiting time and treatment time were used to further evaluate the effect of medical staff complying with the systems of triage criteria. e waiting times of patients in category 1 and category 2 were shorter than those of patients in category 3 and category 4. When a patient was assigned to category 1, he or she was immediately sent to the resuscitation room by the triage nurses. At the same time, the doctors and nurses were informed to launch interventions such as respiratory support and circulatory support. Some category 2 patients were directly sent to the rescue room by nurses, and some were admitted to the intensive care unit after being evaluated by the doctors in the diagnosis area. e waiting time of category 2 patients (2.0 min) was very short, almost equal to that of category 1 patients (1.0 min), which indicated that the triage nurses and doctors paid close attention to these patients and prioritized their care. In addition, it may also be that because triage category 1 and 2 patients were immediately moved to the red zone, where a doctor would presumably see the patient soon, the difference in waiting time was not obvious. However, the treatment time for category 1 patients (1.0 min) was significantly shorter than for category 2 patients (8.0 min), which reflected the rapid response to patients with life-threatening conditions. e category 1 patients were critical and often deteriorated rapidly to death before hospitalization. is factor may have led to a higher mortality rate but a lower hospitalization rate in category 1 relative to category 2. No patient in category 4 was found to have died.
Our study confirmed that the CHT and the ATS were significantly correlated with the patients' condition, emergency treatment, and clinical outcomes, which can distinguish the severity of emergency patients. e AUROC curve value of the CHT was slightly higher, at 0.845 (95% CI: 0.825-0.866), than that of the ATS, at 0.740 (95% CI: 0.715-0.765), which might also be explained by the triage way of CHT. Based on the triage level of patients, the CHT triage standard stipulates the patients' treatment area. On the one hand, this way helps shorten the time of obtaining treatment for critical patients, and on the other hand, it can enhance nurses' risk awareness and send critically ill patients to the open ICUs (the resuscitation room or rescue room) to avoid delayed treatment.

Limitations
is study has several limitations. First, the data were obtained from a single setting. In the course of triage, some patients were transferred from other hospitals and had usually undergone clinical assessments, accompanied by auxiliary examinations and even a definitive final diagnosis, which was beneficial for the triage. e information collected in this study was based on paperboard surveys, and there may be typical sample cases. Hence, future studies with larger network samples are needed to better interpret our findings. Second, although this study was based on the real scenarios, this method could not be completely blinded because the assistant nurse could see which zone the patient was sent to by the instructing nurse and could adjust his or her CTS and ATS triage level accordingly for the subsequent patients. e presence of this "learning curve" could have contributed to the higher ATS consistency of 0.731 in this study than that of 0.64 [22] obtained from simulated triage scenarios based on actual ED patients. It also happened in the CHT consistency results. e accuracy of the triage was also influenced by nurses' knowledge and experience, the external environment, the institutional level, and bed availability. Finally, this research was conducted in an adult ED, so the results cannot be applied to pediatric patients.

Conclusions
is study compared the reliability and the validity of the CHT and the ATS. e reliability levels of the CHT and the ATS were considered moderate. Both systems can be used to identify critical patients in EDs. In order to improve the reliability and validity of emergency triage, it is necessary to further study the triage system in terms of structure and content.

Data Availability
e data used to support the findings of this study are restricted by the Ethics Committee of the Second Xiangya Hospital, China (2016/S014).