Development and validity of the Korean Psychiatric Triage Algorithm


 Background Psychiatric triage algorithm can improve the quality of care by providing a systematic method for classifying the severity of patients experiencing psychiatric crisis in the emergency room and by determining the optimal time frame for intervention. The aim of the present study was to test the professional validity and clinical validity of a psychiatric triage algorithm developed for use in South Korea.Methods The initial algorithm was developed through systematic literature review. Its validity was then verified by 10 experts. Based on results of expert validity, the algorithm was modified and the final algorithm was developed.Results Its clinical validity was then verified by 30 emergency room nurses who had used triage. Four questions of expert validity results with a CVI of 0.8 or less were revised to reflect expert opinion. The clinical validity of the final modified algorithm was 2.9 ~ 3.47.Conclusion The use of highly validity psychiatric triage algorithm by ER (Emergency Room) nurses not only can improve the quality of care, but also can improve patient outcomes and experience.


Introduction
A psychiatric emergency is a situation in which emergency measures are required because people with various aggravated or acute illnesses are at risk of harming themselves or others [1]. Psychiatric emergencies, unlike physical emergencies, are notable because there is a risk of harm to individuals and the society, including others [2].
An increasing number of patients from all over the world visit emergency medical centers each year due to psychiatric emergencies [3]. In particular, Korea is ranked the 1st among Organization for Economic Cooperation and Development (OECD) countries with a suicide rate of 28.6%. Suicide death is one fatal consequence of psychiatric emergencies [4]. Therefore, the importance of intervention in psychiatric emergency patients is emerging [4].
In psychiatric emergencies, early detection of emergencies, rapid access to emergency care, and provision of specialized emergency mental health care should be provided organically [1]. Among these tasks, classifying the severity of psychiatric emergency patients by determining clinical characteristics quickly and accurately is the most important rst step. This task is done by the emergency room nurse and is called triage. Patients who are misclassi ed as mild may deviate from medical attention and receive no immediate help, resulting in long stays in emergency medical centers [5]. This causes overcrowding of the emergency medical center and poses a burden of medical expenses on the patient [5,6]. On the contrary, categorizing the severity of psychiatric emergency patients to be higher than it actually is can increase social costs to cover the health insurance [7], deteriorate the quality of nursing, and lower the satisfaction of both patients and employees [8].
Although emergency medical center nurses often meet patients who experience psychiatric emergencies more than in the past [7], they are trained on physical diseases as before [9]. Therefore, it is often more di cult to interpret psychiatric emergency patient's response than when dealing with a patient who complains of physical problems [9]. Moreover, objective criteria and references to interpret psychiatric emergency patient's response are limited [9].
In addition, nurses in emergency centers need to be accurate in their clinical situations, even in busy emergency situations. They must professionally deal with violent and aggressive behaviors of patients protesting over long waiting time [10]. Such an environment can be very stressful for professionals. The severity of psychiatric emergency patients may be misjudged by the prejudice of medical staff involved in the classi cation of subjects with psychiatric symptoms [10]. To more accurately classify psychiatric emergency patients at the clinical site, objective judgment criteria are needed [9].
Looking at Triage tools used in emergency rooms worldwide, Emergency Severity Index (US, 1999), Types of tools used to classify the severity of psychiatric emergency patients in emergency care settings vary by country. The Manchester Triage [11] and Canadian Triage and Acuity Scale (CTAS) [12] are typically used. Although these tools are suitable for physical emergencies that require preferential treatment for life-threatening symptoms, they are not entirely suitable for categorizing psychiatric emergencies that focus on social dysfunction [13]. For example, hostility, agitation, thought disturbance, positive symptoms of schizophrenia, suspiciousness and irritability, reduced social functioning, poor selfcare in appearance, tone, and behavior can be immediately observed in psychiatric patients experiencing a psychiatric emergency [14]. However, these symptoms might not be real emergencies based on immediate life threats, although they can be exacerbated by self-harm or violence against others over time without proper treatment [15]. In addition, mental symptoms can be overlooked because they focus on solving physical problems when they are accompanied by mental and physical problems [13].
For this reason, Australia uses the Hobart Mental Health Triage Scale (MHTS) [16] and the South Eastern Sydney Area Health Service (SESAHS) [17], to classify patients with psychiatric emergencies into four levels and ve levels, respectively. They categorize emergency situations to these levels and set the standard of maximum waiting time. South Eastern Sydney Area Health Service also categorizes symptoms to be observed or reported during classi cation. Although these tools could result in reduced waiting times in emergency centers, they did not include physical symptoms, making it di cult to judge situations involving both mental and physical symptoms [18].
The Crisis and Triage Rating Scale (CTRS) [19] was developed in the United Kingdom. It was rst introduced in Korea in 2005 for crisis management for mentally ill patients in the community. It is currently being used in mental health promotion centers [20]. This measure has three categories: risk, support system, and cooperative capacity. It showed a high predictive validity for psychiatric emergency patients [19]. With this measure, subjects who receive less than 9 points are suggested to be hospitalized [19]. However, CTRS is not really a risk-based classi cation, but rather an inpatient and discharge of the patient. It does not include physical conditions that accompany mental emergencies [21]. In addition, there is a limit in measuring the severity if the patient is not cooperative because knowledge of the patient's social and family support system is required [18].
Color-Risk Psychiatric Triage [18] was an algorithm developed in Mexico in 2016. Physical assessments were performed rst, followed by psychological evaluation. Actual or potential risks were considered to be the main cause of psychiatric emergency classi cation. Color-Risk Psychiatric Triage's classi cation criteria are accurate. It is easy to check the severity of the classi cation using color codes that indicate priority [18]. However, as a severity classi cation tool for doctors, an appropriate prescription is provided for each severity, which is somewhat different from the nurse's triage work. In addition, it is complicated to use for triage.
In the emergency room, the time it takes for a patient to see a doctor depends on the outcome of the triage performed by the nurse. Triage is administered prior to the face of the doctor and is not intended for prescription. That is, the prescription obtained through this does not meet the purpose of the triage administered by the nurse, so it is inappropriate to utilize Color-Risk Psychiatric Triage. This requires the development and evaluation of severity classi cation criteria for use by nurses working in triage during psychiatric emergencies.
Thus, the purpose of this study was to develop a triage algorithm to determine the severity of patients who visited the emergency room due to psychiatric emergencies.

Design
This was a methodological study to develop Triage algorithm for determining the severity of patients who visited the emergency room due to psychiatric emergencies. KISS, DBPia, and NDSL were searched. Google scholar articles in the last 10 years was searched using key terms of 'Mental health illness', 'psychiatric emergency', 'triage', 'decision aid', 'assessment', 'Validity', 'reliability', 'algorithm' by combining AND and OR. Based on quality evaluation of selected documents, three researchers revised items and paths of the algorithm suitable for Korean situation through three meetings.
Validation and correction of content validity for initial algorithm For content validity, eight to twelve expert panels are recommended to be established [22]. Therefore, in this study, a total of 10 expert panels were formed, including two specialist in emergency medicine, one psychiatrist, two mental health specialist with over 10 years of experience in a psychiatric ward as nurses, two nurses with more than 10 years of experience in emergency medical center, one mental health specialist with over 10 years of experience in emergency department as a social worker. To collect data, experts were sent an e-mail with a description of the study and a consent form. They were asked to reply if they would like to participate in the study within two weeks. In the case of a reply, the researcher set up an appointment with an expert, met with the expert, and collected the consent form and questionnaire. Initial algorithms were presented to 10 experts for content validity. After that, we checked the validity of the initial algorithm through 16 questions. For each question, participants were asked to rate 1 to 4 points, ranging from 'not very appropriate' to 'very appropriate'. For additional comments, a blank space was provided for each item in the questionnaire.

Clinical validation of the modi ed algorithm
In considering test use, the practitioners must also consider clinical validity [23]. These test properties refer to the accuracy with which a test identi es a patient's clinical status. For the clinical validity test, 25 patients were classi ed as severe based on previous studies [24]. A total of 30 data were collected considering the possibility of withdrawal from the study. To verify the usefulness, adequacy, and convenience of the modi ed algorithm, a total of 30 nurses (12 nurses and 18 nurses who performed triage in the emergency department of C University Hospital and B University Hospital in Seoul, respectively) participated in this study. Recruitment notices, a description of the study, and a consent form were kept in the break room to voluntarily complete the questionnaire and put it in the collection box. Through this method, nurses participating in the study were informed of this study and voluntarily agreed to participate in the study. Evaluation of the usefulness, adequacy, and convenience of the algorithm was carried out using a tool developed by Paul [1999]. For each question, participants were asked to rate 1 to 4 points, ranging from 'not so' to 'Really so'. For additional opinions, blank space was left for each question in the questionnaire. The higher the score, the higher the practical suitability of the algorithm. Nurses who participated in this study evaluated the clinical validity of the modi ed algorithm based on their experience of classifying patients who visited the emergency room due to psychiatric problems. All 30 questionnaires were used for analysis without withdrawal from the study or unfaithful responses.

Final algorithm con rmation
The nal algorithm was con rmed by verifying that there was no di culty or problem for nurses to apply the algorithm to patients through practical suitability veri cation of the modi ed algorithm. Figure 1 shows the research process.
Data analysis To con rm the content validity of the initial algorithm, CVI was calculated for each item. Based on Lynn [1986], the content validity index (CVI) was calculated for 16 items. Items with CVI of 0.8 or higher were selected. Items with CVI below 0.8 were supplemented by re ecting the revised opinion provided by experts.
To con rm the clinical validity of the modi ed algorithm, average frequency and standard deviation of the algorithm's frequency of use, usefulness, adequacy, and convenience were calculated.

Initial Algorithm Development
A total of ten documents were searched (Fig. 2).

Content validity of initial algorithm
The expert group to verify the validity of the initial algorithms included two professors of mental health medicine, two professors of emergency medicine, one head nurse, one nurse who had worked in a psychiatric ward for more than 10 years, two who had worked in an emergency room for more than 10 years, one suicide case manager, and one social worker. Two of them had bachelor's degrees. Four of them had master's degrees. Four of them had doctoral degrees. The average working experience was 16.3 ± 8.5 years.
Is it appropriate to con rm that the physical problem of the patient who came to the emergency room is the result of suicide attempt? .9 2 − 1.
Is it reasonable to classify an emergency if a physical problem is the result of a suicide attempt?
Is it reasonable to use KTAS if physical problems are not the result of suicide attempts? 1.0

3.
If aggressive behavior is observed, is it reasonable to classify as urgent?
Is it reasonable to have a psychiatrist's face-to-face medical treatment immediately in an urgent? .8

4.
Is it reasonable to classify as an emergency if you have severe agitation / suicide attempts?
The criteria for severe agitation are based on the Excited Component of the Positive and Negative Syndrome Scale. Is the classi er valid?
Is it appropriate to have a psychiatrist visit in 15 to 30 minutes in an emergency? .7

− 3.
Is it appropriate to report to an emergency medical specialist if an emergency is found that causes physical damage due to suicide attempts? .9

5.
Is it reasonable to classify as a para-emergency if you show mild agitation? 1.0 5 − 1.
The criteria for mild agitation are based on the Excited Component of the Positive and Negative Syndrome Scale. Is the classi er valid?
Is it reasonable to have a psychiatrist visit within 30 to 60 minutes if you are paraemergency? .7
If it is non-emergency, is it reasonable to have a psychiatrist visit within 60 to 120 minutes?
If it is non-emergency, is it reasonable to make a discharge or outpatient appointment if the patient does not complain of a subjective psychiatric emergency?

Expert opinion on initial algorithms
For item 1 (Is it appropriate to distinguish between mental and physical problems by the patient's address?), there was an opinion that it could be di cult. In the case of suspicion of appealing psychiatric problems as a result of physical problems, there was an opinion that it was necessary to consider physical causes rather than approach psychiatric problems. There were also opinions that it would be more appropriate to classify based on address and past history and that there was a need to prioritize patients who had a complex problem.
For item 4 − 2 (Would it be appropriate to have a psychiatrist visit within 15 to 30 minutes in case of an emergency?), there was an opinion that it would be realistic to treat a resident within 30 ~ 60 minutes of a psychiatric emergency because it would be practically di cult if the resident did not reside in the emergency room.
For question 5 − 2 (Is it appropriate to have a psychiatrist visit within 30 to 60 minutes in case of paraemergency?), in the case of para-emergency, there was an opinion that face-to-face medical treatment of emergency medical specialists or residents who resided in the emergency room would be more realistic than psychiatrist within 30 to 60 minutes.
For question 6 − 1 (Is it appropriate to have a psychiatrist visit within 60 to 120 minutes for nonemergencies?), in the case of non-emergency, there was an opinion that it would be appropriate to visit the resident outpatient after the resident or the emergency clinic within 2 hours rather than the psychiatrist. Based on results of content validity analysis of these expert groups, the modi ed algorithm was derived by modifying and supplementing the initial algorithm (Fig. 3).

Validation of clinical validity of the modi ed algorithm
General characteristics of the triage nurse group for clinical validation of the modi ed algorithm are as follows (  The average rating of items to validate clinical validity was 2.9 ± 0.707 ~ 3.47 ± 0.571. Items that the algorithm helped in nursing activities and the importance of the algorithm had the highest ratings (3.47 ± 0.571 and 3.47 ± 0.629, respectively). On the other hand, the degree to which the algorithm helped in costeffectiveness was relatively low (average 2.9 ± 0.607) ( Table 3).

Discussion
The purpose of this study was to develop an algorithm for evaluating the severity of patients who experienced psychiatric crisis in Korean emergency room and to nd out whether it could be applicable to emergency room nurses. For algorithm development, the initial algorithm was revised after verifying the expert validity by considering previous studies. The clinical validity was then tested by educating emergency room nurses to use the modi ed algorithm and evaluating it based on previous experience.
For Item 1 (Physical problems are prioritized when both mental and physical problems are appealed.), the expert content validity index was less than 0.8. Thus, the severity classi cation of psychiatric emergency patients with physical problems can be achieved more consistently with this tool than with the Crisis and Triage Rating Scale [19] which does not consider physical problems. This allows us to prioritize patients with complex problems. In addition, for items 4 − 2, 5 − 2, and 6 − 1, considering the situation in Korea where psychiatrist does not reside in the emergency room, the psychiatrist, not the resident, performs face-to-face treatment. The time required for face-to-face medical treatment is too short. This point was considered to be lacking the reality. Therefore, the criteria were changed to 'face-to-face treatment by resident within 60 minutes in case emergency' for 4 − 2, 'face-to-face treatment by resident within 90 minutes in case of para-emergency' for item 5 − 2, and 'face-to-face treatment by resident within 120 minutes in case non-emergency' for item 6 − 1. Through this, the effectiveness of the algorithm was improved so that it could be applicable to the clinical eld.
After clinical practitioners received training on modi ed and complementary algorithms, the practical suitability was veri ed based on the experience of classifying patients who experienced psychiatric crisis. As a result, the item that the algorithm helped nursing activities and the importance of the algorithm scored 3.47 points out of 4 points. Algorithms can often be used for nurses' clinical judgment or nursing decision making. Algorithms are visually organized, easy to focus on, easy to understand and help nurses acquire knowledge for problem-solving [25]. While the patient's severity classi cation varied greatly according to nurses' individual competencies, the modi ed algorithm seems to be helpful in practice by providing a consistent and uni ed nursing provision and suggesting theoretical basis and criteria that are easy to perform.
In addition, the item of 'was the term easy to understand?' in the clinical validity evaluation of this study scored 3.37 which was a relatively high score. Korean Triage and Acuity Scale previously used in Korea had a different classi cation according to working experience of ER nurses due to an ambiguous standard setting [26]. In particular, the training for nurses in the emergency room is mainly about physical symptoms [9], making it di cult to determine the severity of psychiatric symptoms by the ambiguous standard of Korean Triage and Acuity Scale.
Therefore, we tried to improve problems of the existing Korean Triage and Acuity Scale in the development of the algorithm of this study. First, to clarify the criterion for the concept of 'agitation', which was a highly abstract concept among algorithms, the criteria for 'agitation' were presented based on the Positive and Negative Syndrome Scale (PANSS-EC) [27]. In addition, this study was written in one chapter to simplify the explanation of the algorithm path and the reference that could be used when using the algorithm. By providing detailed criteria on one side of the algorithm, it is possible to increase the consistency of performance based on the rationale without disturbing the clarity of the algorithm path.
On the other hand, the questionnaire asking whether the algorithm had cost-saving effect scored the lowest at 2.9 points. Items asking whether the algorithm reduced time in nursing activities scored 3.13 points. Items asking whether it was easy to use scored 3.13 points, which was relatively low. Stages for dividing psychiatric emergencies varied among triage tools. Although as many as seven stages have been distinguished [14], there are generally four to ve stages. The Color-Risk Psychiatric Triage used in the emergency room to classify the severity of patients experiencing psychiatric crises was developed to address only psychiatric crises. Therefore, it is an algorithm that understands and re ects psychiatric crisis well. The Color-Risk Psychiatric Triage divides the psychiatric crisis into ve stages [18].
For those with phases 2 and 3 showing no difference in mediation, the psychiatrist should treat them face to face within 15 to 30 minutes [18]. Triage nurses must quickly and accurately classify patients with limited information in complex, dynamic decision-making processes in a fraction of the time. To simplify the decision-making stage according to the role of the triage nurse, the algorithm consisted of 4 stages of psychiatric crisis by integrating stages 2 and 3.
Despite these considerations, the cost-effectiveness could have been underestimated because the algorithm was not yet familiar to nurses at the time of validating the clinical validity, making it more timeconsuming and di cult for nurses to use it for making decisions. In fact, two nurses wrote their comments "It's easy to use, but you have to watch it every time. I think it will improve if I learn it through repeated use". Also, because the algorithm was not computerized, nurses might have felt ine cient. In fact, there was an opinion that 'computerization is required for faster access for use in the emergency room'. Therefore, it is necessary to reevaluate emergency room nurses who are accustomed to using the algorithm after computerizing the algorithm developed in this study. There is also a need to look at the long-term cost-effective aspect. Prior studies have shown that determining the appropriate severity of emergency room patients helps reduce the length of emergency room stay and medical costs for emergency room patients [5].
This study has several limitations. First, we calculated the CVI value based on the opinions of experts and modi ed the algorithm to re ect the experts' comments, but did not go through the process of reviewing it again. However, clinical validity resulted in clinical suitability. In the future, a study to verify this algorithm will need to be reviewed again. And second, the feasibility analysis was not performed on patients who visited the emergency room. Thus, the cost-effectiveness of the algorithm developed in this study could not be con rmed. It is necessary to conduct a study that compares outcome variables such as the length of emergency room stay before and after applying the algorithm developed in this study, or the degree of agreement between results of the nurse's triage and hospitalization.

Conclusions
In this study, we developed an algorithm based on literature review and veri ed its validity using expert content validity and clinical validity. Through the use of the algorithm developed in this study, it is expected that the severity of psychiatric crisis patients who visit the emergency room can be systematically and consistently classi ed. It will help reduce the length of stay of patients. It can also help ensure patient safety and save unnecessary medical and social expenses.

COMPETING INTERESTS
All the authers declare that they have no competing interests.

FUNDING
There is no funding for this study. All data collection and analysis fees were paid by the autyhers.