Prognostic accuracy of eight triage scores in suspected COVID-19 in an Emergency Department low-income setting: An observational cohort study

Introduction Previous studies deriving and validating triage scores for patients with suspected COVID-19 in Emergency Department settings have been conducted in high- or middle-income settings. We assessed eight triage scores’ accuracy for death or organ support in patients with suspected COVID-19 in Sudan. Methods We conducted an observational cohort study using Covid-19 registry data from eight emergency unit isolation centres in Khartoum State, Sudan. We assessed performance of eight triage scores including: PRIEST, LMIC-PRIEST, NEWS2, TEWS, the WHO algorithm, CRB-65, Quick COVID-19 Severity Index and PMEWS in suspected COVID-19. A composite primary outcome included death, ventilation or ICU admission. Results In total 874 (33.84 %, 95 % CI:32.04 % to 35.69 %) of 2,583 patients died, required intubation/non-invasive ventilation or HDU/ICU admission . All risk-stratification scores assessed had worse estimated discrimination in this setting, compared to studies conducted in higher-income settings: C-statistic range for primary outcome: 0.56–0.64. At previously recommended thresholds NEWS2, PRIEST and LMIC-PRIEST had high estimated sensitivities (≥0.95) for the primary outcome. However, the high baseline risk meant that low-risk patients identified at these thresholds still had a between 8 % and 17 % risk of death, ventilation or ICU admission. Conclusion None of the triage scores assessed demonstrated sufficient accuracy to be used clinically. This is likely due to differences in the health care system and population (23 % of patients died) compared to higher-income settings in which the scores were developed. Risk-stratification scores developed in this setting are needed to provide the necessary accuracy to aid triage of patients with suspected COVID-19.


Introduction
Less resilient health care provision combined with uneven vaccination means that Emergency Department (EDs) in low-and middle-income countries (LMICs), especially in low-resource settings such as Sudan, still may be overwhelmed when there is high COVID-19 prevalence [1,2].Only around 20 % of the population of Sudan are estimated to be fully vaccinated.Use of Emergency Department (ED) risk-stratification scores can allow patients who need treatment in hospital to be identified quickly.Disposition decisions in LMICs are largely based on clinician gestalt and available clinical experience [3].Clinical risk-stratification scores may help clinicians with less experience rapidly identify those who need treatment in hospital and increase transparency of decision-making.
The AFEM COVID-19 Mortality Scale (AFEM-CMS) was developed for an Emergency Department setting in the low-resource setting of Sudan and showed good discrimination (C-statistic 0.78), albeit on a small sample size (467 patients), without external validation [4].Acuity scores developed in high income settings, such as the Pandemic Respiratory Infection Emergency System Triage (PRIEST) score are able to show accurate prediction of a composite outcome of death or organ support in patients with suspected COVID-19 in higher income settings and the PRIEST score is recommended for use in the ED for risk-stratification by the American College of Emergency Physicians to identify patients who may need inpatient treatment [5][6][7][8].The LMIC-PRIEST score was developed in an ED setting in the Western Cape, South Africa (a middle income country).The LMIC-PRIEST score includes physiological cut-offs based on routine practice in South Africa and includes comorbidities (Heart disease and diabetes), in place of functional status in the PRIEST Score [9].The accuracy of NEWS 2, PRIEST, LMIC-PRIEST and other clinical risk-stratification scores have not previously been assessed in a low resource setting.If such scores accurately predict serious adverse outcomes based on information available at initial triage they could help identify very low-risk patients who could be discharged immediately in order to help mitigate the risk of hospitals being overwhelmed during periods of increased COVID prevalence.
We aimed to estimate the ability of existing clinical risk-stratification score to predict risk of death or need for respiratory support in those with suspected COVID-19 infection support in Sudan (a low resource setting).

Setting
Our observational retrospective cohort study was conducted in nine government referral hospitals in Khartoum State, Sudan.Study data were derived from an electronic registry of consecutive patients treated for COVID-19 in COVID-19 centres at these nine government hospitals.The study period encompasses the ancestral Wuhan strain, Beta and Delta Waves.Sudan has two types of hospitals: district and referral.District hospitals are in rural and peri-urban areas and provide less specialist care, while referral hospitals are in urban centres and provide more advanced and specialist care.During the study period patients with suspected COVID-19 pandemic received similar levels of care in centrally managed COVID isolation units in both district and referral hospitals.Screening based on clinical suspicion (COVID-19 testing was very limited in this setting) was introduced during the Covid-19 pandemic in all twenty-one public hospital Emergency Units (EUs) and in nine hospitals a dedicated primary isolation area (Covid-19 centre) was developed to hold, treat and then safely transfer patients with suspected Covid-19 to five secondary Covid-19 treatment hospitals.The already fragile health system in Sudan came under significant pressure, especially during the early phases of the pandemic, with some hospitals closing due to staff sickness.Due to fear infection and knowledge of the pressure the emergency health care system was under, a high degree of patient led population pre-selection occurred, with only the sickest patients attending EUs.No scoring system was used routinely to assess patient acuity and patients were treated in EUs irrespective of severity of illness.

Data collection
At the nine study sites data were extracted and deidentified from paper records into a secure electronic database stored locally.Physiological parameters and presenting complaints from initial presentation to the Covid-19 centres were collected by clinical staff using electronic forms.These were the first physiological parameters recorded prior to treatment being initiated.Comorbidities and symptoms were recorded as unstructured free text and were extracted using natural language processing with manually checking of data extraction in a subset of patients prior to analysis [16].If comorbidities were not documented in available records they were assumed not to be present.Recorded observation which appeared implausible were assumed to be recorded in error and changed to missing (Supplementary Material 2).Some physiological variables including temperature and conscious level were not routinely measured or recorded and were excluded from analysis when calculating triage scores.

Inclusion criteria
All patients aged 16 or more years between 3rd January 2020 and 14th December 2021 who were treated at participating COVID-19 centres.Patients were treated in COVID centres based on a clinical suspicion of COVID-19 infection and all patients were included irrespective of acuity.Patients with incomplete demographic (age or sex) information were excluded.

Outcome
The primary outcome was a composite of any of: non-invasive ventilation, intubation in Emergency Unit Covid-19 centre, admission to higher dependency care (HDU/ICU) or inpatient death.

Analysis
The accuracy of eight clinical risk-stratification scores were estimated for our study outcomes [5,8,[10][11][12][13][14]. Discrimination of each score was estimated in terms of the receiver operating characteristic (ROC) curve and the area under the ROC curve (c-statistic).Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for each score threshold was estimated at thresholds which have been recommended for use clinically: 0 vs 1+ CRB-65; 0-1 vs 2+ NEWS2; 0-2 vs 3+ PMEWS; 0-4 vs 5+ PRIEST; 0 vs 1 WHO score; TEWS 0-2 vs 2+ Quick COVID Severity Index 0-3 vs 4+ [13, 17,18].We also estimated accuracy at every score threshold to assess the impact of using different thresholds to guide hospital admissions.We excluded patients on greater than 10 L/min oxygen from the analysis of the Quick COVID Severity Index as this formed part of the outcome in the development study [8].All analyses were performed in Python 3.8.8.pandas version: 1.4.4 [19].

Sample size
We used a convenience sample size of data collected on patients with C. Marincowitz et al. suspected COVID-19 presenting to the nine-participating hospitals 3rd January 2020 and 14th December 2021.
We a priori estimated a desired precision of discrimination for the AUC (C-statistic) based on 6000 patients with an event rate of the primary outcome of 5 %.Assuming an AUC of 0.8, based on discrimination of triage scores in other studies [6], this would provide a confidence interval width of 0.06 (95 % CI 0.77 to 0.83).Our study cohort was smaller but had a higher event rate and provided estimated confidence interval widths of a similar size.
Khartoum State Ministry of Health, and the University of Cape Town Human Research Ethics Committee (HREC 450/2020) gave approval for the study.Data were anonymised at each study site prior to being pooled and made available for analysis..

Study population
Fig. 1 shows how the study cohort was selected and Table 1 show the demographic and clinical details of the 2583 patients in our study cohort.Overall, 874 (33.84 %, 95 % CI:32.04 % to 35.69 %) died, required intubation/non-invasive ventilation in COVID-19 Centres or HDU/ICU admission, 596 (23.07 %, 95 % CI: 21.49 % to 24.73 %) patients died and 59 (2.28 %, 95 % CI: 1.77 % to 2.93 %) had a HDU/ICU admission.1248 (48.3 %) of patients had available COVID test results and 77.5 % of tested patients had confirmed infection.There were only around 50 critical care beds available at secondary isolation hospitals.Patients receiving non-invasive ventilation were routinely managed in non-ICU settings and this accounts for small number of patients admitted to HDU/ICU despite 300 patients receiving ventilatory support.

Triage score performance
All triage scores assessed had poor estimated discrimination (C-statistic range 0.56-0.64)for the primary outcome (Fig. 2).Estimated discrimination of the risk-stratification scores was higher for the secondary outcome of death (C-statistic range 0.59-0.69)(Supplementary Material 3) and lower for secondary outcome of HDU/ICU admission (Cstatistic range 0.46-0.65)(Supplementary Material 4).
Table 2 presents the estimated sensitivity, specificity, positive and negative predictive values at recommended score thresholds for use clinically for the primary composite outcome.Supplementary Material 5 presents estimates for sensitivity, specificity, positive and negative predictive values at every possible score threshold.Tables 3 and 4 present equivalent estimates at recommend score thresholds to Table 2 for the secondary outcomes (1) death and (2) HDU/ICU admission.
At previously recommended thresholds three risk-stratification scores (NEWS2, PRIEST and LMIC-PRIEST) had high estimated sensitivities (≥0.95) for the primary outcome (Table 2).However, the baseline risk of 33.8 % meant that low-risk patients identified at these thresholds had an 8 % and 17 % risk of the primary outcome.High sensitivities were achieved at the expense of poor specificity, with very few patients below the previously recommended thresholds.

Discussion
This study assessed for the first time the accuracy of eight clinical triage scores in the low resource setting of Sudanese COVID-19-units.All risk-stratification scores assessed had worse estimated discrimination compared to higher-income-settings in which the scores were developed and validated [5,8,[10][11][12][13][14]. In the study setting there were high rates of death (23.1 %) and the primary composite outcome (33.8 %).This far exceeds the rates estimated in the original PRIEST and LMIC-PRIEST cohorts and represents a much sicker cohort of patients [5,9].Due to fears of hospital acquired infection and restrictions on available emergency care due to the pandemic, patients only attended hospital as a last resort.This preselection of patients contributed to the high baseline risk, which meant that although previously recommended thresholds for risk-stratification scores to aid discharge decision-making could achieve high (>0.95,PRIEST and LMIC-PRIEST scores) sensitivities for the primary outcome, low-risk patients identified by these thresholds still had an around 15 % risk of the primary outcome.In this context, triage tools intended to rule out severe disease may not be of clinical use.

Comparison to previous literature
Few studies have been conducted assessing outcomes or potential triage methods for patients with suspected COVID-19 in Sudan.In a study conducted during the first wave of COVID-19 case fatality rates varied between 40 % and 70 % from April to September 2020 in government referral hospitals [4].In a study conducted COVID-19 treatment hospitals from May 2020 to May 2021 in Gezira State, Sudan, 61.2 % of admitted patients were reported to have received either non-invasive or invasive ventilatory support [20].The AFEM COVID-19 Mortality Scale (AFEM-CMS) was developed using data from 467 patients with confirmed COVID-19 who presented at two government referral hospitals [4].The mortality scale achieved a C-statistic of 0.78 (95 % CI:0.737-0.813) on internal validation when predicting death, which compares to estimated c-statistics of 0.59 to 0.69 for the triage scores assessed in this study.However, a key predictor in the AFEM-CMS is conscious level which we found to be poorly collected and recorded within the available emergency COVID-19 centre dataset.
Within higher resource settings of the UK and South Africa the triage scores assessed consistently had higher estimated discrimination for the primary outcome, C-statistics: UK 0.61 to 0.8 and Western Cape, South Africa 0.7 to 0.82 [6,21].This compared to estimated C-statistics of 0.56 to 0.64 in our Sudanese study cohort.This could be partly explained by some predictors (conscious level and temperature) included in some triage scores being unavailable.However, the strongest predictors of adverse outcomes were previously found to be need for supplemental oxygen, saturations and respiratory rate, all of which were available [5,9].More plausibly, differences in triage score performance may reflect differences between the Sudanese study population and population characteristics in higher-income settings.The Sudanese population had a high adverse event rate (33.8 %), only 26 % of patients did not require supplemental oxygen on presentation, mean presenting saturations were 84 % and mean initial respiratory rate was 31.In a UK ED setting the prevalence of adverse outcomes was 22.1 %, 68.4 % of patients did not initially require supplemental oxygen, mean presenting saturations were 94.9 % and mean respiratory rate was 23.1.In a South African ED setting 3.45 % of patients experienced the primary outcome, 91.9 % of patients did not require supplemental oxygen on presentation, mean presenting saturations were 96.1 % and mean respiratory rate was 18.6.The differences in the prevalence of outcome and predictors between this Sudanese validation cohort and the development cohorts may have contributed to spectrum effects and the differences in estimated performance [22].

Limitations
This study appears to be the largest multi-centre cohort study of patients with suspected COVID-19 being treating in the emergency care setting of Sudan.However, data collection was reliant on clinical staff extracting clinical predictors and outcomes from available clinical notes.Although intended to be a consecutive cohort of patients, there may be risk of bias from identification and inclusion of patients in the registry.Information bias may result from predictor misclassification or erroneous data, particularly for recording of comorbidities.The effect of missing data was not assessed using sensitivity analyses.Some predictors used in the triage scores assessed, including conscious level and temperature, were not routinely measured or recorded which may have affected the estimated accuracy.Outcomes including death were only measured if they occurred in hospital and therefore, for the small number of patients discharged from COVID-19 centres, death in the community may have been missed.Additionally, it was not possible to assess prediction of outcomes within specific time periods.The study population is based on those treated in COVID-19 centres due to a

Implications
Both emergency medicine and formal triage of patient acuity on arrival to emergency care are still being developed in Sudan with no consistent triage system used in our study setting [23].Reconfiguration of Emergency Department services in order to isolate patients with suspected COVID-19 further disrupted care pathways of patients and our study population was selected from patients treated in COVID-19 centres set up in participating hospitals emergency units.Within this setting none of the triage scores assessed showed sufficient accuracy to be used to select patients who could be safely rapidly discharged following initial assessment.Moreover, some components of the triage scores assessed including conscious level and temperature were not routinely recorded which may affect the feasibility of implementing such scores in this setting.The high adverse event rate and physiological indicators of respiratory failure in the study cohort shows a large degree of population selection had already occurred before patients attended the COVID-19 centres with almost all patients requiring further inpatient treatment (only 16 % of patients were discharged).
Whether the assessed tools would have any clinical use in a population of less seriously ill patients in Sudan (potentially in a community or prehospital setting) requires further research.However, our study indicates the assessed clinical scores should not be adopted in the setting of Emergency Units in Sudan.Within this setting where intensive care capacity is limited, research is needed to develop accurate riskprediction, and help determine which patients are most likely to benefit from respiratory and other forms of organ support.

Conclusion
Patients treated in emergency COVID-19 centres in Sudan had high rates of death and need for respiratory support compared to those presenting to Emergency Departments in higher-income settings.No triage score assessed was accurate enough to be used clinically.Further research is needed to identify predictors of adverse outcomes and develop risk-stratification scores which can be used to clinically riskstratify patients in this setting.

Dissemination of results
In addition to promotion of the results through study and funder website, findings will be presented informally to clinicians at data collection sites.

Authors Contribution
Authors contributed as follow to the conception or design of the work; the acquisition, analysis, or interpretation of data for the work; and drafting the work or revising it critically for important intellectual content: CM 20%, MH 15%, YO 15%, PH 7%, DM 7%, SG 7%, PB 7%, GF 7%, LS 7% and LW 7%.All authors approved the version to be published and agreed to be accountable for all aspects of the work.

Table 1
Study population characteristics.

Table 2
Accuracy triages scores primary outcome (95 % confidence interval).* Patients on greater than 10 L/min oxygen were excluded from analysis.

Table 3
Accuracy triages scores death (95 % confidence interval).* Patients on greater than 10 L/min oxygen were excluded from analysis.
* Patients excluded if had <3 predictor parameters.**Patients on greater than 10 L/min oxygen were excluded from analysis.C.Marincowitz et al.