Challenges in the validation of triage systems at emergency departments

doi:10.1016/j.jclinepi.2009.07.009

Journal of Clinical Epidemiology

Volume 63, Issue 4, April 2010, Pages 384-388

https://doi.org/10.1016/j.jclinepi.2009.07.009 Get rights and content

Abstract

Objectives

Triage systems, developed by consensus of experts and based on decision rules, are typically not validated. The objective is to discuss the challenges to evaluate the reliability and validity of triage systems.

Study Design and Setting

Theoretical–conceptual approach to validate triage systems.

Results

The consensus-based triage systems have to be applied to a broad population with a variety of signs and symptoms. For the individual patient-specific decision, rules are used and the outcome measure is, typically, one of five prognosis-specific urgency categories. In contrast, prediction rules in diagnostic research are developed for a narrow specific subpopulation and based on a combination of parameters to predict presence of a specific diagnosis. Reliability is based on case scenario and simultaneous triage studies. The first step in triage validation is to decide on the best proxy for prognosis, “the reference standard” for the urgency classification. The next step is modification of the triage decision rules, including a multivariate approach. The final step is the validation in different settings and to evaluate the impact in clinical practice.

Conclusion

Triage should be viewed as diagnostic research and would benefit if it would use the available methodology in diagnostic research.

Introduction

Triage aims to determine the clinical priority of patients based on their presenting features. The objective is to decrease morbidity and mortality. Triage systems have to identify patients who need immediate attention on one hand but also to recognize patients who can safely wait or may not need emergency care at all on the other hand. The Manchester Triage System (MTS), the Canadian Triage and Acuity Scale (CTAS) and the Emergency Severity Index (ESI) Score, and the Australasian Triage Scales are the most commonly used triage systems in emergency departments (EDs) for children and adults [1], [2], [3], [4], [5], [6], [7].

All triage systems are based on consensus opinion of experts. The experts design decision trees to support clinical risk assessment or predictions of resource use to define urgency levels. Triage systems have to be simple to allow nurses to quickly assess the patient and have to be applicable for the broad population with a large variability in signs and symptoms in the emergency care setting.

The evaluation of triage systems involves assessment of reliability and validity. Reliability refers to the degree of intraobserver variability and interobserver variability. The validity refers to the degree to which the triage system predicts the “true” urgency. The internal validity is based on studies in a single setting (typically, the setting that was involved in the development or modification of that system). The external validity is based on studies in different settings. Most studies on triage systems focus on the interrater agreement [7], [8], [9], [10]. The results of the interrater agreement studies depend on the unambiguousness and completeness of the system and the accuracy of the nurse. The extensive use of triage systems in daily practice at ED and general practitioner out of hours offices contrasts sharply with the limited number of studies that validate triage systems. The validity of the MTS was studied retrospectively in a subgroup of patients with high-risk chest pain and seriously ill patients [11], [12]. We prospectively internally validated the MTS in pediatric emergency care in 13,000 children with an independent reference standard [4]. For the ESI, CTAS, pediatric Soterion Rapid Triage System, and Australasian Triage Systems, the trends in resource use, length of stay, admission, hospital charges, and mortality were used to validate the urgency levels [5], [6], [7]. An expert panel, which allocated an urgency level based on the clinical information of the ED visit at presentation, was used in some case scenario studies to validate triage systems [13], [14].

The objective of this article is to discuss the challenges in triage research: How to evaluate the reliability and validity of triage systems? Why are validation studies on the underlying decision rules of triage systems lacking? And how to apply the methodology of diagnostic research for the validation of triage systems?

Section snippets

The development of decision rules in major triage systems

The commonly used triage systems are all based on expert opinion of physicians and ED nurses. The systems are based on acuity of presenting problem and features or predicted resource consumption or a combination of these items. Most triage systems categorize the patients in one out of five levels of urgency.

The MTS is based on an algorithm of flowcharts specific for the patient's presenting features and specific discriminators. In the MTS, 52 flowcharts are available. Each flowchart includes

Reliability studies

The intrarater agreement (determination of variability within a single triagist re-rating the same patient) and interrater agreement (determination of variability between different triagists rating the same patient) of triage systems depend on the following criteria. The triagists must be experienced with clinical features of patients presenting at the ED and well trained for the particular triage tool. Furthermore, the triage system must be unambiguous and contain enough diversity of

Implementation of triage systems

Clinical prediction rules focus on specific groups of patients, and implementation in clinical practice at large scale is lacking [19], [20]. However, triage systems are implemented in many EDs worldwide (Table). Application of and compliance to triage rules is rather high, varying from 50% to 99.9% for ESI to 95% for the MTS [2], [4], [25]. What can we learn from the successful implementation of the triage systems?

The triage systems are developed by experts in the field of ambulatory emergency

Conclusions

Triage should be viewed as a diagnostic research and would benefit if it would use the methodology developed in diagnostic research. Evaluation should be shifted from a focus on consensus to the validation of the decision rules in the triage systems with a reference standard, including a multivariate approach and external validation in different settings. Possible lessons for diagnostic research are that triage is applied at large scale due to implementation in the workflow at the ED and that

References (26)

S. Gouin et al.
Evaluation of the pediatric Canadian triage and acuity scale in a pediatric ED
Am J Emerg Med
(2005)
P.A. Maningas et al.
The use of the Soterion rapid triage system in children presenting to the emergency department
J Emerg Med
(2006)
O.T. Rutschmann et al.
Interactive triage simulator revealed important variability in both process and outcome of emergency triage
J Clin Epidemiol
(2006)
P.G. Zimmermann
The case for a universal, valid, reliable 5-tier triage acuity scale for US emergency departments
J Emerg Nurs
(2001)
C.J. Biesheuvel et al.
Polytomous logistic regression analysis could be applied more often in diagnostic research
J Clin Epidemiol
(2008)
A. Van den Bruel et al.
The evaluation of diagnostic tests: evidence on technical and diagnostic accuracy, impact on patient outcome and cost-effectiveness is needed
J Clin Epidemiol
(2007)
R.J. Cooper
Emergency department triage: why we need a research agenda
Ann Emerg Med
(2004)
D.R. Eitel et al.
The Emergency Severity Index triage algorithm. Version 2 is reliable and valid
Acad Emerg Med
(2003)
R. Van Gerven et al.
Systematic triage in the emergency department using the Australian National Triage Scale: a pilot project
Eur J Emerg Med
(2001)

M. Van Veen et al.

Manchester triage system in paediatric emergency care: prospective observational study

BMJ

(2008)

M.R. Baumann et al.

Evaluation of the Emergency Severity Index (version 3) triage algorithm in pediatric patients

Acad Emerg Med

(2005)

S.L. Dong et al.

Reliability of computerized emergency triage

Acad Emerg Med

(2006)

Cited by (73)

The effectiveness of primary care streaming in emergency departments on decision-making and patient flow and safety – A realist evaluation
2022, International Emergency Nursing
Primary care streaming was implemented in UK Emergency Departments (EDs) to manage an increasing demand for urgent care. We aimed to explore its effectiveness in EDs with different primary care models and identify contexts and mechanisms that influenced outcomes: streaming patients to the most appropriate clinician or service, ED flow and patient safety.
We observed streaming and interviewed ED and primary care staff during case study visits to 10 EDs in England. We used realist methodology, synthesising a middle-range theory with our qualitative data to refine and create a set of theories that explain relationships between contexts, mechanisms and outcomes.
Mechanisms contributing to the effectiveness of primary care streaming were: quality of decision-making, patient flow, redeploying staff, managing patients across streams, the implementation of governance protocols, guidance, training, service evaluation and quality improvement efforts. Experienced nurses and good teamworking and strategic and operational management were key contextual factors.
We recommend service improvement strategies, operational management, monitoring, evaluation and training to ensure that ED nurses stream patients presenting at an ED seeking urgent care to the most appropriate clinicians for their needs in a safe and efficient manner.
The new emergency department “Tuscan triage System”. Validation study
2021, International Emergency Nursing
Citation Excerpt :
The only intervention to prevent the occurrence of consultations between nurses was a strong recommendation to respect the rules in order to safeguard the validity of the study. Lastly, as highlighted by previous studies, the use of triage clinical case vignettes could have produced potentially different results from those obtained if the validity and reliability assessments were performed on actual ED patients, or using other reference standards [6,8,28,31]. The TTS showed excellent content and face validity.
A new organizational framework was recently implemented in Tuscan Emergency Departments (EDs), including specific low-priority streaming. A new ED triage system, named “Tuscan Triage System” (TTS), was devised with the purpose of applying this reorganization.
A validation study was designed with the primary aims of assessing the content, face, and criterion validities, and the inter-rater reliability of the TTS. The secondary aim was to estimate the differences in triage level assignation between the previous “Regional Triage System” (RTS) and the TTS. Twenty-four nurses trained for the TTS were enrolled to assign TTS priority levels to 100 triage clinical case vignettes drawn up by the developers of the TTS (reference standard).
The Content Validity Index – Scale/Average (S-CVI/Ave) of TTS was 0.98. Concerning to face validity, the S-CVI/Ave was 1. The highest adherence of triage level assignation to the reference standard was for levels 1 and 2. The Krippendorff α value was 0.808. Undertriage and overtriage were 10.45% and 14.29%, respectively. Overall, the comparation between RTS and TTS showed a marked shift of level assignation towards TTS low priority levels.
The TTS seems to be safe. These results should be confirmed through studies in the real clinical settings.
Performance of the Manchester Triage System in patients with dyspnoea: A retrospective observational study
2020, International Emergency Nursing
Citation Excerpt :
A low specificity might indicate a well-performing triage system and treatments in the ED able to prevent negative outcome, such as mortality. Therefore, several authors have suggested using subjective outcome measures, such as the re-evaluation of the correct triage based on expert review [22,23]. Although this pragmatic approach might be important to evaluate the quality of the MTS in place, e.g. triage nurses’ adherence with the MTS recommendations, a hard outcome, such as mortality might still be a better outcome to assess the MTS’s ability to assess patients’ urgency and severity [24].
Between 1% and 7.5% of patients admitted to the emergency department (ED) suffer from dyspnoea. The Manchester Triage System is one of Europe’s most used triage systems considering five levels to prioritize patients in the ED: level 1 (red), immediate; level 2 (orange), very urgent; level 3 (yellow), urgent; level 4 (green), standard; level 5 (blue), non-urgent. With this study we aimed to evaluate the performance of the Manchester Triage System in patients with dyspnoea.
With this retrospective, observational study we analysed data from 4′076 patients with dyspnoea accessing the ED in one Italian hospital between January 1, 2017 and June 30, 2019. To determine the sensitivity and specificity we computed the area under the curve (AUC) of the receiver operating characteristics (ROC) comparing the sensitivity and specificity for each of the five priority codes and seven-day mortality.
A total of 3.6% (n = 145) of the patients died within seven days after access to the ED. From the patients dying within seven days, 6.2% (9/145) had a blue or green priority code, 17.9% (26/145) had a yellow priority code and 75.9% had an orange or red priority code. A high priority code (orange or red) had a sensitivity of 75.9%, a specificity of 66.1%, a PPV of 7.6% and a NPV of 98.7%. The AUC was 0.734 (CI 95% 0.695–0.773, p < 0.001).
The Manchester Triage System showed acceptable sensitivity and negative predictive value in patients with dyspnoea, yet a low specificity related to the risk of death. The system is safe, resulting in a low under-triage rate for death. Improving the clinical assessment during nurses’ triage (e.g. considering clinical history and auscultation of lungs) and prospective validation studies with surrogate endpoints is needed in this population.
Triage of patients with fever: The Manchester triage system's predictive validity for sepsis or septic shock and seven-day mortality
2020, Journal of Critical Care
Citation Excerpt :
The best methodology to evaluate triage systems is still debated [2]. According to some authors, the best way to evaluate the validity of a triage system is to use criteria established a priori or to re-evaluate individual triage systems by experts from different fields [27]. Although this may appear pragmatic it may not represent an objective outcome such as mortality, hospitalization in ICU or the need for an urgent procedure [2].
Up to 15% of patients arrive in the emergency department suffering from fever. Triage is their first contact and is responsible for the stratification of patients according to the severity of the condition for which they are presenting at the emergency department. The aim of this study is to assess the predictive validity of the Manchester Triage System in patients with fever for sepsis or septic shock and seven-day mortality.
The sensitivity, specificity and negative predictive value of the Manchester Triage System was assessed by priority code allocation towards seven-day mortality and the diagnosis of sepsis or septic shock.
A total of 3831 patients were evaluated in the emergency department for fever between 1 January 2017 and 30 June 2019. Of these, 1.9% were diagnosed with sepsis or septic shock. Using the Manchester Triage System to predict diagnosis of sepsis or septic shock provided a sensitivity of 88.7%, a specificity of 50.1% and a negative predictive value of 99.5%. For seven-day mortality, sensitivity was 44.4%, specificity was 92.3% and the negative predictive value was 99.3%.
The Manchester Triage System has demonstrated high sensitivity and negative predictive value in patients with fever diagnosed with sepsis or septic shock. For patients with sepsis or septic shock one-third of cases with an incorrectly assigned priority code were caused by incorrect application of the Manchester Triage System.
Reliability and validity of three international triage systems within a private health-care group in the Middle East
2020, International Emergency Nursing
To measure and compare the reliability and validity of three international triage systems (CTAS, MTS and ESI) when applied to patient presentations in the emergency centres of a private health-care group in the United Arab Emirates.
The ability of triage systems to sort patients into categories based on the urgency of their need and time to physician is a key indicator. Three international triage systems are being used for this purpose in private emergency centre settings.
Bespoke reference scenarios, 50 vignettes (10 per severity level) were created and validated by a local expert panel. Nurses performing triage at four emergency centres in the Emirate of Dubai completed online surveys to categorise the vignettes based on the triage system they used in their emergency centre.
Overall inter-rater reliability per triage category was substantial for allocations in category one, moderate for those in categories two and five, and fair for those in categories three and four. Agreement between raters and the reference standard was consistent throughout all four emergency centres. The accuracy of triaging allocations into categories one, two and five were good, whereas allocations in categories three and four were less accurate.
International triage systems focus on the identification of more urgent cases and perform poorly in discriminating between those that are less serious, which is less ideal in a setting where less-serious cases are more prevalent.
Validation of a modified South African triage scale in a high-resource setting: a retrospective cohort study
2023, Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine

View all citing articles on Scopus

View full text

CommentaryChallenges in the validation of triage systems at emergency departments

Abstract

Objectives

Study Design and Setting

Results

Conclusion

Introduction

Section snippets

The development of decision rules in major triage systems

Reliability studies

Implementation of triage systems

Conclusions

Am J Emerg Med

J Emerg Med

J Clin Epidemiol

J Emerg Nurs

J Clin Epidemiol

J Clin Epidemiol

Ann Emerg Med

The Emergency Severity Index triage algorithm. Version 2 is reliable and valid

Acad Emerg Med

Systematic triage in the emergency department using the Australian National Triage Scale: a pilot project

Eur J Emerg Med

Manchester triage system in paediatric emergency care: prospective observational study

BMJ

Evaluation of the Emergency Severity Index (version 3) triage algorithm in pediatric patients

Acad Emerg Med

Reliability of computerized emergency triage

Acad Emerg Med

Commentary
Challenges in the validation of triage systems at emergency departments