Predicting an Unfavourable Course in Patients Hospitalised for COVID-19: The PREDICT-COVID Study

Background: Clinical decision tools that have been proposed to predict the clinical course of patients admitted to hospital with COVID-19 are poorly presented and are at high risk of selection bias. The aim of the study was to propose a prediction clinical tool to predict an unfavourable outcome at the admission of a SARS-CoV2 infected patient that was carefully developed using a large learning database and that was developed from models derived from artificial intelligence. Methods: . The PREDICT-COVID study is a post hoc analysis of the Noso-Cor study, a multicenter prospective, observational study. All patients infected by SARS-CoV2 hospitalized in one of the 11 Lyon-University hospitals since 8-March-2020 have been included. The PREDICT-COVID database was split in two separate datasets: the learning dataset (80%) was used for the development of the model and the validation dataset (20%) for internal validation. The primary composite outcome was the need for mechanical ventilation or admission into an intensive care unit, or death within 21 days of admission. Results: . Data from 823 patients were analysed: age 70.6±16·9 years;body mass index 26.7±5·4 kg/m 2 and median number of comorbidities was 2. Out of the 44 recorded variables, 11 that were the most linked to the primary outcome criteria were retained to develop the optimised risk prediction tool. At admission the 5 most informative predictors were, in descending order: C-Reactive Protein, neutrophil-to-lymphocyte ratio, aspartate transaminase, shortness of breath, and prothrombin time. The ten-fold cross validation of the optimised model had an area under the ROC curve of 0.76±0.06. The performance of the developed Bayesian model to predict the primary outcome of the validation dataset had a mean area under the ROC curve of 0.78, sensitivity of 60%, and specificity of 77%. Conclusions: . The proposed optimised prediction tool that uses 11 routinely determined variables to predict an unfavourable course at admission for COVID-19 had satisfactory performance. For an external validation, the PREDICT-COVID prediction tool is available online at: https://www.hed.cc/?a=covid&n=NETCRIT21J.neta Trial registration: The Noso-Cor study was registered on ClinicalTrials (NCT04290780). The present analysis was registered on ClinicalTrials (NCT04412031) the June 2, 2020.


Background
The COVID-19 pandemic has raised health concerns around the world. In France, as of May 29, cases had reached 145,746, and 18,260 patients had died in hospital [1]. The clinical signs of COVID-19 that lead patients to hospital are well known: fever, cough, fatigue, headache, and shortness of breath [2,3]. The most frequently reported diagnostic and prognostic predictors of COVID-19 are age, body temperature, lymphocyte count, and lung imaging characteristics. Flu-like symptoms and neutrophil counts are frequently predictive in diagnostic models, while co-morbidities, sex, C-reactive protein and creatinine are frequent prognostic factors [4][5][6][7][8][9][10]. With more than 90% of deaths occurring in patients over the age of 60 years, and a predominance of males, French data are consistent with that reported worldwide [1,[4][5][6][7][8][9]. In Italy [11], a higher case-fatality rate (7.2%) compared to China (2.3%) was attributed to a higher proportion of elderly people [12], obviously if the methods for classifying and reporting deaths due to COVID-19 disease were the same.
In many countries, the capacity of intensive care units (ICU) was exceeded, and this was also the case in France in the north-east and in the Paris regions [1]. Thus, it is of major interest to forecast as soon as possible the clinical course of COVID-19 for individuals and for healthcare decision makers to anticipate and predict the need for ICU and resuscitation beds. To this end, many prediction models have been developed; so far 3 types have been developed: diagnostic models to detect COVID-19 infection, models to predict hospital admission for COVID-19 pneumonia in the general population, and models to predict progression to a serious or critical condition [10]. As point out by Wynants et al. [10], these proposed models are poorly presented and have a high risk of bias, raising concerns that their predictions may be unreliable when applied in daily practice.
Using techniques derived from arti cial intelligence, Bayesian methods provide additional insight into the interpretation of data by providing probability estimates of clinical interest and by allowing prior evidence to be taken into account to provide posterior probabilities that re ect natural sequential learning and thus facilitate medical decision making [13]. Using data available from the Noso-Cor study [14], the main objective of the present analysis was to propose a new prediction risk tool, carefully developed using a Bayesian network, and make it available to physicians and healthcare decision makers for predicting an unfavourable course at admission for COVID-19.

Materials And Methods
This is a post hoc analysis of the Noso-Cor study [14], that is an international multicenter prospective, observational, hospital-based study in adults and children. The data were restricted to patients hospitalised at one of the 11 Lyon University hospitals. Demographic and clinical data were collected using speci cally designed case report forms. An internal monitoring was implemented to detect transcription errors. The Noso-Cor study included all patients with a cough and/or fever greater than 37.8 °C at admission for whom SARS-Cov-2 infection was con rmed by reverse transcriptase polymerase chain reaction (RT-PCR) from a nasopharyngeal swab. The Noso-Cor study was approved by the ethics committee of Ile de France V on March 8, 2020 (Comité de Protection des Personnes Ile de France 5). Under French law, patients received a written information form and gave their oral consent to the use of their anonymised data for research purposes. The trial was registered on ClinicalTrials (NCT04290780). The study, which started the March 8, 2020, is still on-going. The present analysis that considered data of patients with a positive SARS-Cov-2 RT-PCR was also registered on ClinicalTrials (NCT04412031) the June 2, 2020.

Database
Anonymised data for patients who were tested positive for Sars-Cov2 and hospitalised in one of the thirteen hospitals of the Hospices Civils de Lyon (Lyon, France) were extracted from the Noso-Cor database REF 16 on May 20, 2020. The data extracted concerned the following. Clinical course of the patient during the 21 days following admission: date of symptoms onset, date of admission to the hospital, date of admission to ICU, as well as vital status at 7, 15, and 21 days. Patient characteristics at admission: age, sex, body mass index (BMI), smoking status, alcoholism, and whether or not they were a healthcare professional. Clinical signs at admission: temperature, cough, fatigue, irritability, abnormal lung auscultation, shortness of breath, pain, myalgia, nausea, runny nose, headache, ageusia, anosmia, diarrhoea, nasopharyngeal discomfort, red eye, confusion, heart failure, sore throat, and coma. Comorbidities: cardiac comorbidities, hypertension, liver comorbidities, hypothyroidism, rheumatological comorbidities, neurological comorbidities, diabetes (type 1 or 2), renal comorbidities, pulmonary comorbidities, cancer, and immunode ciency; these were condensed into a single variable which is the sum of the total number of comorbidities. Paramedical exams at admission: Lung X-ray, C-Reactive Protein (CRP), haemoglobin, white cell count, lymphocyte count, creatinine, natraemia, serum potassium, prothrombin time, aspartate transaminase (AST), alanine transaminase (ALT), and lactate dehydrogenase (LDH).
For Bayesian modelling, continuous variables were divided into classes of clinical relevance (age, BMI, temperature) or into ve classes (best tested discretisation).
The database was separated in two separate datasets at random; the learning dataset that contained 80% of the data was used for the development of the models (Bayesian network and logistic regression) and the validation dataset that contained 20% of the data was used for the internal validation of the model.

Primary outcome
The primary composite outcome included: need for mechanical ventilation, admission into an ICU because of multiple organ dysfunction syndrome or death within 21 days of admission. For all patients included in the present analysis (who were all SARS-CoV-2 -infected), the need for mechanical ventilation, transfer to an intensive care unit, or the occurrence of death was considered, regardless of preexisting conditions that may have contributed to or caused the need for mechanical ventilation, transfer to an intensive care unit, or death.
Prediction model A Bayesian model was used to develop the prediction tool. A Bayesian network is a directed acyclic graph that includes nodes and arrows. Each node represents a variable (and its modalities), and each arrow represents a probabilistic dependency between the parent variable and the child variable. The Tree Augmented Naive (TAN) algorithm was used to build the structure of the Bayesian network. The TAN algorithm applies three rules: i) each node is independently linked to the target node (i.e. primary criteria); ii) each node is also linked to a unique parent node; iii) amongst all possible structures, the structure that maximises the overall mutual information (mutual information measures the strength of the relationship between each variable and the target) between nodes is selected. The next step consists in estimating conditional probability tables, using the expectation maximisation algorithm. The expectation maximisation algorithm is an iterative procedure to compute the maximum likelihood estimation in the presence of missing or hidden data. In the maximum likelihood estimation, the aim was to estimate the model parameters for which the observed data were the most likely. This step is the so-called "learning step" that allows to develop the Bayesian network using the learning PREDICT-COVID dataset. The software used to create the model was RapidMiner Studio ® version 9.7 with the W-BayesNet (TAN) Weka 3 ® Machine Learning Group extension.

Optimisation of the prediction tool
The binding strengths between the variables and the target (unfavourable course) were evaluated by the proportion of variance reduction [15]. This method based on variance decomposition makes it possible to completely explore the space of the inputs to the model and to take into account interactions as well as non-linear responses [15]. The sensitivity analysis makes it possible to classify the variables of the model and to reduce their number, keeping in board the most relevant variables. Thus, to improve the ergonomics of the model and make it usable in clinical practice, the number of variables was reduced while preserving the performance of the prediction network. A regression logistic model was also used to con rm the performances of the Bayesian model. The software used to create the logistic regression model was RapidMiner Studio ® version 9.7 Weka 3 ® Machine Learning Group extension.
The performance of the optimised developed prediction tool used a 10-fold cross validation to determine mean ± standard deviation (SD) area under the receiver-operating-characteristics (AUC-ROC) curve and accuracy. Performance of the optimised developed model to predict the clinical course of the patients of the validation dataset was investigated using the AUC-ROC curve, sensitivity, speci city, positive predictive value (PPV), and negative predictive value (NPV).
The software used was RapidMiner Studio ® version 9.7 with the W-BayesNet (TAN) Weka 3 ® Machine Learning Group extension.
Statistics and validation of the models Quantitative variables are described in terms of means and SD if normal distribution, or median and interquartile range [IQR], and qualitative variables in percentages. The software used was MedCalc ® version 11.5.1.0. The validation dataset that contained 20% of the data was used for the internal validation of the model. The performance of the model was assessed by C statistic.

Study population
A total of 1100 patients on May 20, 2020 were selected. Wrongly selected patients (nosocomial infection, n = 266; under 18 years old, n = 3), and patients whose admission date was unknown (n = 8) were excluded from this analysis; thus, data from 823 patients were analysed. The mean ± SD age of the patients was 70.6 ± 16.9 years (range: 19 to 102 years), BMI was 26.7 ± 5.4 kg/m² (range: 11 to 46 kg/m²), and the median total number of comorbidities was 2 (range: 2 to 8). The most frequent comorbidity was hypertension (45.1%), and 23.4% of the patients were diabetics. The most frequent clinical signs reported at admission were abnormal lung auscultation (68.0%), fatigue (67.0%), and cough (66.7%; Supplementary Table 1). There was no signi cant difference between patients with an unfavourable and those with a favourable outcome with regards to sex, age, smoking status, temperature, renal comorbidity, diabetes, lung auscultation, fatigue, cough, shortness of breath, diarrhoea, pain, headache, runny nose, myalgia, coma at admission, lung X-Ray, creatininaemia, C-Reactive-Protein, prothrombin time ratio, white blood cells count, neutrophils to lymphocytes ratio and AST were signi cantly different (Supplementary Table 2).
The mean proportion of missing values was 5.2%, ranging from 0% for age and sex, to 25.0% for smoking status (Supplementary Table 1). Variables with more than 25% missing values (LDH 66.7% and alcohol consumption 43.7%) were excluded from the analysis.
Models to predict the primary composite outcome: need for mechanical ventilation; admission into an ICU because of multiple organ dysfunction syndrome or death within 21 days of admission Using a Bayesian network, variables were rst classi ed according to the variance reduction, ( Table 1). The 11 retained variables that were the most linked to the primary outcome criteria are listed in Table 2. The 5 most informative variables to predict the primary outcome, in descending order, were: CRP, neutrophil to lymphocyte ratio, AST, shortness of breath, and prothrombin time (Table 1).  The ten-fold cross validation of the optimised Bayesian model developed using the learning dataset (80% of the patients, n = 658) and the 11 analysed variables provided a mean ± SD AUC-ROC of 0.76 ± 0.06 and a mean ± SD error rate of 29.2 ± 6.5%.
The ten-fold cross-validation of the logistic regression model developed using the learning dataset (80% of the patients, n = 658) and the 11 analysed variables for the primary composite outcome had a mean ± SD AUC-ROC of 0.76 ± 0.08 and a mean ± SD error rate of 27.4 ± 11.0%.
The performance of the developed optimised Bayesian model to predict the primary outcome in the validation dataset (the remaining 20% of the patients, n = 165) had an AUC-ROC of 0.78, an accuracy of 77.6%, a sensitivity of 67.9%, a speci city of 76.2%, a PPV of 59.4%, and NPV of 82.5% (Table 3). The structure of the optimised Bayesian network using 11 variables to assess the primary outcome criteria is presented in Fig. 1.

Discussion
Using data from a large and recent prospective cohort of COVID-19 patients, a new prediction tool using a Bayesian network was built to predict the need for mechanical ventilation, transfer to an intensive care unit or death within 21 days of hospital admission.
A total of 32 clinical decision tools to predict the clinical course of patients admitted to hospital with COVID-19 have been proposed. Among these, 23 aimed to predict the risk of mortality and 8 were designed to predict progression to a more serious or critical illness However, as noted by Wynants et al. [10], the proposed clinical prediction tools are poorly presented and are at high risk of selection bias.
Su cient information is not provided to replicate these studies concerning the selection of patients and data, and the methodology used. Few studies report missing data, and, even when this was the case, the authors do not explain how these values were replaced, which is crucial for the construction of clinical prediction tools. Internal validation is not always carried out or clearly presented, and without a careful internal validation an over tting of the model to the data is to be expected explaining that their reported performance is probably optimistic [10]. Contrary to previously published models, the methodology used to develop the prediction model proposed herein follows the published recommendations concerning information on source data, the presentation of inclusion and exclusion criteria for the prospectively included population, the explanation of the judgment criterion, the management of missing data, the explanation of the model used, the methodology used for internal validation, and the model performance measures and their interpretations [16].
The internal validity tests con rmed that the performance of the PREDICT-COVID prediction tool described herein was satisfactory, and with only 11 variables that are commonly used, the prediction tool is easily usable in clinical practice. The performance of the logistic regression model, which is the most commonly used method in medicine to calculate the risk of an event according to exposure, was similar to that of the Bayesian model. However, Bayesian models have many advantages over logistic regression models; for instance, as they do not use any a priori hypothesis, explanatory variables can be co-linear (e.g. white blood cell count and neutrophil to lymphocyte ratio), and as they are based on conditional probabilities, the associations between variables are taken into account even if they are not linear. In addition, Bayesian networks are applicable in case of missing data that are frequent in clinical practice, which could be considered the most important advantage as in absence of a single variable a logistic regression model will not be able to calculate a prediction score for an individual. This explains, in part, why Bayesian models are increasingly used to develop risk prediction tools [17].
A selection bias is highly likely in the proposed clinical prediction tools. For example, Chinese clinical prediction tools were constructed using data from hospitalised populations that are much younger (less than 50 years of age) than those hospitalised in Europe or the USA, and whose mortality is much lower (less than 5%). Other clinical prediction tools were built using learning dataset of less than 100 patients. Furthermore, some clinical prediction tools have been performed on particular populations and use nonroutinely performed laboratory or radiological data (e.g. IL-6 [18], coronary calci cations [19]) when a patient is admitted to hospital for COVID-19. It is likely, that prescription of speci c laboratory or radiological exams by a trained physician is more predictive than the results of the speci c exams per se. A very high mortality rate in these studies con rms this hypothesis.
The clinical characteristics and clinical course of COVID-19 patients hospitalised in Lyon were similar to those reported in California [6], New York City [9], and Italy [5,7,8], which supports the generalisability of the proposed prediction tool. As reported in the analysis published byWynants et al [10], the most frequently reported diagnostic and prognostic predictors of covid-19 are age, body temperature, lymphocyte count and lung imaging characteristics. Flulike symptoms and neutrophil counts are frequently predictive in diagnostic models, while co-morbidities, sex, C-reactive protein and creatinine are frequent prognostic factors. In agreement, 7 of the 11 selected prognostic variables included in the model proposed herein are included among those cited in the other proposed models. Although increasing age is reported as a risk for poor outcomes [8,9,[20][21][22][23], herein patients with an unfavourable outcome were slightly but signi cantly older, but this was not retained in the prediction model presented herein. Similarly, elevated BMI has also been reported to be a risk factor [22,23], but this was not the case herein.
This suggests that age and BMI may explain a higher rate of hospitalisation in COVID-19 patients rather than an unfavourable outcome. Another interesting point is that among the 11 variables retained herein, the majority were laboratory parameters, and these represented 6 of the 7 most predictive variables.
Among these biological parameters, only aspartate transaminase and prothrombin time are not among the parameters most frequently retained by other predictive tools.
The present study has some limitations. The endpoint was independent of the co-morbidities of the patient infected with SARS-Cov-2. This method was chosen because there is no consensus to formally attribute death to SARS-Cov-2 and comorbidities certainly in uence the patient's course. Furthermore, as recommended by Piccininni et al [8]. total mortality captures indirect deaths, such as those related to a healthcare system under crisis, yielding a more complete picture of the pandemic's consequences. The total number of explanatory variables was limited due to the increased mathematical combinatory; and was related to the number of subjects included in the learning dataset. After a classi cation based on variance reduction, somewhat arbitrarily, only the 11 most relevant variables were selected. Thus, using 11 variables, the prediction model offers a good compromise between performance and ergonomics. As Lindsell et al. [24] point out, our model, which can be used as both a prognostic and predictive model, does not guide the health care team on the effectiveness of treatment, which is more informative than a simple prognosis. The main limitation of the present study is that, although carefully developed from especially collected data and controlled by internal monitoring, the results have to be externally validated using similar data sources and prove its effectiveness in a pragmatic trial.

Conclusions
The proposed optimised prediction tool that uses 11 routinely determined variables to predict an unfavourable course at admission for COVID-19 had satisfactory performance. For an external validation, the PREDICT-COVID prediction toll is available online at: https://www.hed.cc/? a=covid&n=NETCRIT21J.neta