Ratio of Oxygen Saturation to Inspired Oxygen, ROX Index, Modified ROX Index to Predict High Flow Cannula Success in COVID-19 Patients: Multicenter Validation Study

Introduction High-flow nasal cannula (HFNC) is a respiratory support measure for coronavirus 2019 (COVID-19) patients that has been increasingly used in the emergency department (ED). Although the respiratory rate oxygenation (ROX) index can predict HFNC success, its utility in emergency COVID-19 patients has not been well-established. Also, no studies have compared it to its simpler component, the oxygen saturation to fraction of inspired oxygen (SpO2/FiO2 [SF]) ratio, or its modified version incorporating heart rate. Therefore, we aimed to compare the utility of the SF ratio, the ROX index (SF ratio/respiratory rate), and the modified ROX index (ROX index/heart rate) in predicting HFNC success in emergency COVID-19 patients. Methods We conducted this multicenter retrospective study at five EDs in Thailand between January–December 2021. Adult patients with COVID-19 treated with HFNC in the ED were included. The three study parameters were recorded at 0 and 2 hours. The primary outcome was HFNC success, defined as no requirement of mechanical ventilation at HFNC termination. Results A total of 173 patients were recruited; 55 (31.8%) had successful treatment. The two-hour SF ratio yielded the highest discrimination capacity (AUROC 0.651, 95% CI 0.558–0.744), followed by two-hour ROX and modified ROX indices (AUROC 0.612 and 0.606, respectively). The two-hour SF ratio also had the best calibration and overall model performance. At its optimal cut-point of 128.19, it gave a balanced sensitivity (65.3%) and specificity (61.8%). The two-hour SF≥128.19 was also significantly and independently associated with HFNC failure (adjusted odds ratio 0.29, 95% CI 0.13–0.65; P=0.003). Conclusion The SF ratio predicted HFNC success better than the ROX and modified ROX indices in ED patients with COVID-19. With its simplicity and efficiency, it may be the appropriate tool to guide management and ED disposition for COVID-19 patients receiving HFNC in the ED.


INTRODUCTION
Coronavirus disease 2019 (COVID- 19) is an infectious disease caused by the severe acute respiratory syndrome coronavirus 2 that has infected millions of individuals worldwide. 1 Its emergence has been regarded as a worldwide public health emergency that has prompted the transformation of healthcare systems, including those of the emergency department (ED). 2 Appropriate and effective treatment of acute hypoxemic respiratory failure (AHRF) due to COVID-19 early in the ED is essential to improve patients' outcomes.
High-flow nasal cannula (HFNC), an oxygen-delivering technique whereby heated and humidified air is delivered with positive pressure generated, has been proven to be an effective initial respiratory support measure for patients with AHRF of many etiologies in both inpatient and ED settings. [3][4][5][6] It has also been used successfully as a non-invasive airway management strategy for AHRF in COVID-19 patients. 7,8 Nevertheless, HFNC therapy should always be administered with caution, as its failure may result in delayed intubation and increased mortality. 9,10 Therefore, predicting HFNC success or failure and determining the optimal timing of treatment escalation to invasive mechanical ventilation are critical to avoid delayed intubation and possibly prevent mortality.
The ROX (respiratory rate oxygenation) index, a ratio of oxygen saturation (SpO 2 )/fraction of inspired oxygen (FiO 2 ) to respiratory rate, has been demonstrated to be a reliable predictor of HFNC success for AHRF patients in the intensive care unit (ICU) and inpatient settings. 11,12 Many studies have externally validated the ROX index as a predictor of HFNC outcomes for COVID-19 patients; however, reported results were inconsistent, possibly due to different clinical settings and cut-points employed, as well as heterogeneous population. [13][14][15][16] The modified ROX index is another index incorporating heart rate (HR) to the original ROX index, which has been shown to be a good predictor of HFNC outcomes for HFNC application post-extubation. 17 However, no studies have validated its utility in COVID-19 patients.
The SpO 2 /FiO 2 (SF) ratio can also be employed as a predictive marker for HFNC outcomes. 18 The SF ratio may even have superior prognostic utility to the ROX or the modified ROX indices for COVID-19 pneumonia, a specific condition in which patients usually do not present with an abnormal respiratory pattern despite severe hypoxia. 19 Therefore, respiratory rate, a component of both types of ROX indices, may not be a good predictor of HFNC outcomes in COVID-19 patients. Although a previous study reported a superior prognosticating ability of the SF ratio over the ROX index in COVID-19 patients, it was a single-center study conducted in an inpatient setting. 20 No studies have validated and compared the SF ratio with the ROX index or its modified version in the ED setting, where HFNC is usually initiated earlier in the disease course. Consequently, we conducted this study to evaluate and compare the prognostic utility of the SF ratio, the ROX index,

Study Process and Data Collection
Using International Classification of Diseases, 10 th Revision, codes, we retrospectively reviewed consecutive patients visiting the participating EDs with the diagnosis of COVID-19 infection made before or within the index ED visit. Their electronic health records were reviewed to determine whether they had received HFNC in the ED. We used this data if all the inclusion and exclusion criteria were satisfied. With retrospective chart review performed by trained data abstractors at each study center, 22 we recorded the patients' baseline characteristics, physiologic parameters, relevant blood examination results, HFNC settings, co-treatments, and important clinical outcomes.
Parameters required for calculating the SF ratio, the ROX index, and the modified ROX index were recorded before HFNC application (hour 0) and at 2 hours after HFNC initiation. All these parameters were measured while the patients were still in the ED awaiting disposition. The decisions to initiate HFNC to the patients, adjust HFNC settings, and escalate the treatment toward a more invasive respiratory support measure were determined by the attending physicians at each study center. Another study coordinator double-checked the recorded data in the electronic case-report forms to ensure the reliability and accuracy of the study data.

Study Parameters and Outcomes
At 0 and 2 hours after HFNC initiation, we calculated three parameters and assessed them for their utility in predicting HFNC outcomes: the SF ratio; the ROX index; and the modified ROX index. The SF ratio was calculated from the ratio of SpO 2 to FiO 2 . 18 The ROX index was calculated from the ratio of SpO 2 / FiO 2 to respiratory rate, 11 and the modified ROX index was defined as the ratio of the ROX index over heart rate multiplied by 100. 17 The primary outcome was HFNC success, defined as no requirement of mechanical ventilation following HFNC treatment at HFNC termination. The secondary outcome was overall treatment failure, defined as a requirement of escalation to mechanical ventilation following HFNC termination or mortality at hospital discharge. 23,24

Statistical Analyses
We employed descriptive statistics to describe patients' characteristics. Categorical data is reported as frequency and percentage. Continuous variables are reported as mean and standard deviation (SD) or median and interquartile range for normally distributed and non-normally distributed data, respectively, evaluated based on histograms and Q-Q plots. We compared these variables between the success and failure groups by using the chi-squared or Fisher exact test for categorical data and an independent t-test or the Mann-Whitney U test for continuous data.
As an external validation study, the predictive performance of the SF ratio, the ROX index, and the modified ROX index for the study outcomes were assessed primarily with their discrimination and calibration capacities, coupled with other additional analyses. 25 We chose the parameter with an overall superior ability over the others among all analyses performed as the best parameter in predicting the study outcomes. [25][26][27] We reported the discrimination of each parameter with the area under the receiver operator characteristics curve (AUROC) and its 95% confidence interval (CI). We also made comparisons between the AUROCs of the study parameters for each study outcome. 28 Calibration was reported with calibration plots and the Hosmer-Lemeshow test. 29,30 Moreover, we evaluated overall model performance using the Nagelkerke R-squared. A parameter that could yield a higher R-squared value should perform better than others. 31,32 We also evaluated the clinical usefulness of the parameters at the optimal cut-off values according to the Youden index by reporting their sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, negative predictive value, positive predictive value, and diagnostic odds ratio. We also performed univariate and multivariate logistic regression analyses to identify independent predictors of adverse clinical outcomes: HFNC failure and overall treatment failure.
Age, gender, body mass index, day of symptoms, Charlson Comorbidity Index, Sequential Organ Failure Assessment (SOFA) score, D-dimer, C-reactive protein, and steroid were determined a priori based as existing evidence as potential associating variables with HFNC outcomes to be adjusted for in the multivariate models. We included variables with univariate P-value <0.2 in the multivariate regression model for each outcome. Nonetheless, each multivariate model evaluated only one potential predictor value among the three parameters at one time point to avoid multicollinearity.

Study Population
Between January 1-December 31, 2021, a total of 978 COVID-19 patients visited the participating EDs. Of these, 184 patients were treated with HFNC initiated in the ED and 11 (6%) had do-not-intubate status. Consequently, 173 patients were included in the study. Their characteristics are shown in Table 1 higher mean age than the success group (Table 1). Otherwise, baseline demographics and initial physiologic variables were generally comparable between the two groups. Nevertheless, the failure group required higher FiO 2 on HFNC, were on HFNC for a shorter duration, and had more complications and longer hospital length of stay than the success group (Table 1).

Study Parameters
There were no missing values for any parameters evaluated at either the 0-or 2-hour time points. We included all 173 samples in the analyses for both time points because no outcome events occurred prior to the two-hour time point at which the second parameter values were measured. The mean values of the three indices were lower in the HFNC failure group than in the success group; however, only the parameters measured at two hours were significantly different between the groups for both study outcomes ( Table 2). Distributions of the parameter values among the study population are shown in Figure 1. For all parameters, a higher proportion of patients with higher parameter values were those with overall treatment success, implying strong associations between the parameter values and overall treatment failure ( Figure 1D-1F). However, such trends and associations were not as prominent for any parameters with HFNC success ( Figure 1A-1C).

Parameters' Performance
The ROC curves of all parameters are shown in Figure  1S, and their AUROCs are presented in Table 3. The SF ratio measured at two hours post-HFNC application had the highest discriminating capacity (AUROC 0.651, 95% CI 0.558-0.744), followed by the two-hour ROX index (AUROC 0.612, 95% CI 0.516-0.707), and two-hour modified ROX index (AUROC 0.606, 95% CI 0.512-0.700). However, none of these AUROCs were significantly different from each other (P-value for difference among AUROCs=0.80). Similarly, the two-hour SF ratio could yield higher AUROC for overall treatment failure than the two-hour ROX and modified ROX indices (Table  3), but these AUROCs were also not significantly different (P-value=0.21). The parameters measured at hour 0 all had lower discrimination than those at two hours for both study outcomes ( Table 3).
The two-hour SF ratio was the parameter with the best overall performance based on the Nagelkerke R-squared for both study outcomes (Table 3). Calibration based on the Hosmer-Lemeshow tests also showed that the two-hour SF ratio calibrated well with both outcomes, especially with overall treatment failure. Although the calibration plots in Figure 2 imply that there may have been an underestimation of the probability of both outcomes in the lowest and highest quintiles of probability predictions for the SF ratio that was worse than the other two parameters, these plots need to be interpreted with caution due to the small number of sample and event rates at the very low and very high predicted probabilities, possibly resulting in over/underestimation of risks due to random noise.
The two-hour SF ratio at the optimal cut-point of 128.19 for predicting HFNC success yielded the most balanced sensitivity (65.3%) and specificity (61.8%) compared to twohour ROX>3.23 and two-hour modified ROX>4.27 (Table 4). Although it could also detect the lowest proportion of patients (56.6%), the rate of false positives was the lowest (21.4%)

0.001
Note: P-value for differences in AUROC among any parameters for HFNC success = 0.799, for overall treatment failure = 0.213. HFNC, high-flow nasal cannula; SpO2, pulse oximetry; FiO2, fraction of inspired oxygen; ROX, respiratory rate oxygenation index. Table 3. Prognostic performance of the parameters before and after high-flow nasal cannula (HFNC) application in predicting HFNC success and overall treatment failure.

Ruangsomboon et al.
SF Ratio, ROX, Modified ROX to Predict HFNC Success in COVID-19 Patients      Table 5. Univariate and multivariate analyses of factors predicting high-flow nasal cannula outcomes.
Note: a Optimal cut-points of SpO2/FiO2 ratio were 128.19 for HFNC failure and 119.38 for overall treatment failure, b optimal cut-points of ROX were 3.23 for HFNC failure and 4.36 for overall treatment failure, c optimal cut-points of Modified ROX were 4.27 for HFNC failure and 4.06 for overall treatment failure. Variables with univariate P-value< 0.2 were included in the multivariate logistic regression models. Only one strongest parameters in predicting each outcome among SpO2/FiO2 ratio, ROX, and modified ROX were included in the multivariate models to avoid multicollinearity. OR, odds ratio; CI, confidence interval; GFR, glomerular filtration rate; SpO2, pulse oximetry; FiO2, fraction of inspired oxygen; ROX, respiratory rate oxygenation index.
the strongest predictor of the outcomes based on the univariate regression results (Table 5) and because it had the most superior  diagnostic ability (based on Table 3 (Table 5). For overall treatment failure, the two-hour SF ratio>119.38 was also significantly and strongly associated with the outcome (aOR 0.19, 95% CI 0.08-0.38; P<0.001) ( Table 5).

DISCUSSION
To the best of our knowledge, this study was the first to directly compare the prognostic utility of the SF ratio, the ROX index, and the modified ROX index for AHRF patients secondary to COVID-19 in the ED setting. We found that the SF ratio measured at two hours post-HFNC application was the best predictor of HFNC and overall treatment success since it could yield the highest discriminating ability and overall performance index, as well as good calibration, well-balanced diagnostic accuracy indices, and strong association with HFNC and overall treatment failure.
High-flow nasal cannula has been recommended as the respiratory and oxygenation support measure for patients with AHRF due to COVID-19 since it has been shown to provide many physiologic benefits and may reduce adverse outcomes, such as mechanical ventilation rate. 7,8 Therefore, it has been increasingly used in many settings, including the ED, where the disease trajectory may differ from inpatient or ICU settings given that HFNC is initiated earlier in the disease course. Still, patients with failed HFNC treatment may end up requiring mechanical ventilation, and a delay in this process may result in mortality. 9 To avoid these undesirable outcomes, it is necessary to employ adequate and appropriate patient monitoring using effective and efficient instruments.
The ROX index, the most widely validated prognostic marker of HFNC outcomes, has been shown by many studies to also have acceptable prognostic utility in COVID-19 patients. [14][15][16]27 However, only one single-center study has evaluated its utility in the ED setting. 34 Moreover, no multicenter studies have compared the ROX index to the SF ratio, a more convenient and possibly more relevant tool for COVID-19 considering its pathophysiology and general patient characteristics, or the modified ROX index, a relatively more complex measure that also incorporates another vital sign (HR) that may be related to the disease severity and progression.
Consequently, the present study has added to the current body of evidence that the SF ratio, the simplest parameter among the three, could outperform the ROX and the modified ROX indices in predicting HFNC success for COVID-19 patients in the ED setting. Although the discrimination based on AUROC of the SF ratio was not significantly higher than those of the other parameters, it was still superior to the others by a wide range of other statistical analyses, including calibration, overall model performance, diagnostic accuracy indices based on the optimal cut-point, and associations with the outcome based on regression analyses. The superiority of the SF ratio over other more complex parameters could have been because patients with COVID-19 usually present with silent hypoxia, a condition in which other physiologic parameters can appear normal despite very low oxygenation. 19 As a result, the study parameter with only variables relevant to oxygenation was more highly related and predictive of the outcome. The present study yielded concordant results with a study by Kim et al, who evaluated 133 COVID-19 patients receiving HFNC treatment in an inpatient setting and found that the SF ratio at one hour provided superior AUROC to that of the ROX index. 20 However, it is important to note that although the SF ratio offered a more balanced sensitivity and specificity than the other two indices for HFNC success, the ROX index yielded a largely higher sensitivity and the proportion of detectable patients. This controversy might have been because the Youden index employed in these analyses may not be the most appropriate method to identify the optimal cut-point for this outcome as it only focused on the highest product of sensitivity combined with specificity without considering their balance. This matter was evidenced by a very high sensitivity and low specificity for the ROX index compared to the other two parameters.
Interestingly, we found that the SF ratio clearly had better predictive ability than the other indices for overall treatment failure based on all statistical analyses performed. Its performance was even better than the SF ratio for HFNC success. This result adds to the current body of evidence that mortality could have been another measure of adverse clinical outcomes of COVID-19 in patients treated with HFNC that predictive scoring systems, especially the SF ratio, could be able to predict accurately. With its unique clinical progression, disease-specific mortality could imply severe disease deterioration despite initial successful HFNC weaning. 24 Nevertheless, despite the SF ratio having the highest AUROC, the AUROCs found in our study were generally lower than in other studies in COVID-19 patients. [14][15][16]20,33 Such a contrast could have been explained partly by different settings and population between this study and previous studies (ED vs non-ED). Also, it could have been because of the characteristics of the population and setting specific to Thailand, a middleincome country, where the quality of care and available healthcare resources are much more limited than in other higher income countries. Our generally higher mortality and HFNC failure rates compared to other studies of higher income countries may reflect our limited-resources situation. Also, the events of inhospital mortality and mechanical ventilation experienced by our study population might not have arisen directly due to COVID-19 but also partly due to limited healthcare provisions and suboptimal quality of care. These issues might have explained the lower discrimination capacity of the study parameters in the present study. Nevertheless, they underlie the importance of the present study as the much higher validation AUROCs in previous studies from higher income countries would not have been applicable to our scenario.
Nonetheless, from the present study, it was still appropriate to conclude that the SF ratio was superior to the ROX index and the modified ROX index in predicting HFNC success and SF Ratio, ROX, Modified ROX to Predict HFNC Success in COVID-19 Patients Ruangsomboon et al.
overall treatment failure for emergency COVID-19 patients not only because of its superior performance over a wide range of statistical analytic methods but also because the SF ratio is easier to calculate at bedside, thereby being more efficient to be used in the ED.

LIMITATIONS
There were some limitations to this study. First, the study was conducted in a middle-income country, which may limit its generalizability even though it involved multiple EDs of hospitals with varying levels of care. Second, the data was collected retrospectively, which may have caused possible errors and corresponding bias associated with the nature of a retrospective study. Third, we only measured the parameters at two hours post-HFNC application and not at any later time points because it was the longest duration that all the patients were still in the EDs; therefore, the clinical utility of the parameters could also involve aiding in ED disposition decision-making. The other reason was that there were many missing variables at later time points because the physiologic parameters were not monitored simultaneously among inpatient units of participating hospitals. Nevertheless, had the parameters been followed for longer than two hours, their prognostic utility and their relative ability in prognostication could have changed. Regardless, the applicability of those findings to the ED setting may be limited.

CONCLUSION
The SF ratio measured two hours after high-flow nasal cannula initiation was better than the ROX index and the modified ROX index at predicting HFNC success in patients with acute hypoxemic respiratory failure secondary to COVID-19 in the ED setting. Compared to the other two ROX indices, the two-hour SF ratio had the greatest prognostic utility, as well as the utmost simplicity and bedside efficiency. Therefore, it may be an appropriate tool to guide appropriate disposition, further management, and potential escalation therapy for COVID-19 patients treated with HFNC therapy in the ED.