Development and external validation of prediction models for critical outcomes of unvaccinated COVID-19 patients based on demographics, medical conditions and dental status

Background Multiple prediction models were developed for critical outcomes of COVID-19. However, prediction models using predictors which can be easily obtained in clinical practice and on dental status are scarce. Aim The study aimed to develop and externally validate prediction models for critical outcomes of COVID-19 for unvaccinated adult patients in hospital settings based on demographics, medical conditions, and dental status. Methods A total of 285 and 352 patients from two hospitals in the Netherlands were retrospectively included as derivation and validation cohorts. Demographics, medical conditions, and dental status were considered potential predictors. The critical outcomes (death and ICU admission) were considered endpoints. Logistic regression analyses were used to develop two models: for death alone and for critical outcomes. The performance and clinical values of the models were determined in both cohorts. Results Age, number of teeth, chronic kidney disease, hypertension, diabetes, and chronic obstructive pulmonary diseases were the significant independent predictors. The models showed good to excellent calibration with observed: expected (O:E) ratios of 0.98 (95%CI: 0.76 to 1.25) and 1.00 (95%CI: 0.80 to 1.24), and discrimination with shrunken area under the curve (AUC) values of 0.85 and 0.79, based on the derivation cohort. In the validation cohort, the models showed good to excellent discrimination with AUC values of 0.85 (95%CI: 0.80 to 0.90) and 0.78 (95%CI: 0.73 to 0.83), but an overestimation in calibration with O:E ratios of 0.65 (95%CI: 0.49 to 0.85) and 0.67 (95%CI: 0.52 to 0.84). Conclusion The performance of the models was acceptable in both derivation and validation cohorts. Number of teeth was an additive important predictor of critical outcomes of COVID-19. It is an easy-to-apply tool in hospitals for risk stratification of COVID-19 prognosis.


Introduction
The novel coronavirus disease 2019 (COVID- 19) pandemic has presented an important and urgent threat to global health since its outbreak in December 2019. COVID-19 does not only affect the respiratory tract, but it also affects other organs in human body i.e., lungs, liver, kidney, heart, and vessels [1]. Respiratory failure and acute respiratory distress syndrome (ARDS) are the most common serious complications of COVID-19 infection [2]. The relative excess deaths (excess mortality) from all causes in 2020 was 20% to 30% in Europe [3]. Compared with patients who were not vaccinated, patients who were vaccinated with at least two doses have significantly lower risk of mortality and ICU admission [4,5]. While the COVID-19 vaccines have provided strong protection against serious illness, hospitalization, and death [4][5][6][7], around 1/3 of the worldwide population is still unvaccinated until 2 nd July 2022 [8].
Multiple risk factors have been shown to be associated with the critical outcomes of COVID-19. Those risk factors include patients' clinical symptoms, systemic disease, lifestyle habits, demographics, and radiographic, laboratory, immunological, inflammatory and hematologic markers [9][10][11][12][13][14][15]. In addition, poor oral health, in particular periodontitis, was also shown to be associated with the critical outcomes of COVID-19 [16][17][18][19]. Patients with periodontitis may have 5 times higher odds of hospitalization, 6 times higher odds of requiring assisted ventilation, and 7 times higher odds of COVID-19-related death than participants without periodontitis [18]. This may be because of the damage to lower airways due to the aspiration of periodontal pathogens, exacerbation of the cytokine storm through the low-grade chronic systemic inflammation, and pulmonary vessels vasculopathy due to the dissemination of the virus through the ulcerated gingival epithelium [18]. Number of teeth is very commonly used in previous clinical studies [20,21], as a clinical indicator for periodontal diseases. This is because the diagnosis of periodontal diseases requires professional dental clinical and radiographic examinations by dentists. The number of teeth can be measured easily and reliably by clinicians or even patients themselves [22], which may facilitate the assessment of periodontal status in non-dental settings. Lower number of teeth has been shown to be significantly associated with higher risk of critical outcomes of COVID-19 [23,24].
The high contagiousness, high ICU admission rate, and high mortality of COVID-19 have led to tremendous increases in the demand for hospital beds and shortage of medical equipment. Therefore, there is an urgent need for a pragmatic risk stratification tool that allows for the early identification of COVID-19 patients who are likely to be at the highest risk of ICU admission and death [25]. This can help clinicians and policymakers make evidence-based decisions on the management of COVID-19 patients and optimize resource allocation. Recently, multiple prediction models have been developed for the prediction of the prognosis of COVID-19 patients [26,27]. Those prediction models varied in their predictors and performance of the models. A large number of prediction models carry difficulties in their application for the rapid risk stratification of general COVID-19 patients at their first intake in hospitals. This is because some predictors cannot be easily obtained without professional devices or lab tests, such as C reactive protein, peripheral oxygen saturation, urea level, white cell count, and lymphocytes [26]. Furthermore, many prediction models showed moderate performance in aspects of discrimination and calibration, and beard no benefit to clinical decision-making [26]. In addition, the oral condition and the dental status were never considered potential predictors in the previously developed models.
Therefore, the present study aimed to develop and externally validate prediction models for the early and rapid stratification of critical outcomes of COVID-19 patients using predictors which can be easily obtained in clinical practice, including patients' demographic characteristics, and medical conditions (including dental status).

Materials and methods
The present study was carried out based on Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [28].

Study design and population
The study was designed as a retrospective cohort study. We included consecutive hospitalized patients and outpatients from the Isala Hospital (Zwolle, the Netherlands) who were diagnosed with COVID-19 between January 2020 and May 2021 as the derivation cohort for the development of the prediction models. We included the consecutive hospitalized patients and outpatients from Northwest Clinics (Alkmaar, the Netherlands) who were diagnosed with COVID-19 between January 2020 and July 2021 as the validation cohort. The eligibility criteria are as below: • Patients were adults (≥18 years); • Patients visited the Department of Oral and Maxillofacial Surgery (OMFS) of the hospitals up to five years prior to the COVID-19 diagnosis; • Patients were unvaccinated against COVID-19; • Patients were diagnosed with COVID-19 based on a positive SARS-CoV-2 real-time reverse transcription-polymerase chain reaction (rRT-PCR) on swab material, sputum, or bronchoalveolar lavage samples.

Ethical statement
This study was approved by the Medical Ethics Committee, Isala Academy, Zwolle, the Netherlands (200710) on 22 nd February 2021, and taken over by Northwest Academy, Alkmaar, the Netherlands (L021-054) on 20 th September 2021. The requirement for informed consent was waived because the study concerns a Health Services Research project that, under the Medical Research Involving Human Subjects Act (WMO), is not considered medical scientific research (https://www.ccmo.nl/onderzoekers/ aanvullende-informatie-over-bepaalde-soorten-onderzoek/niet-wmo-onderzoek/dossieronderzoek). We only used retrospective datasets, in which the data were not collected specifically for the study and there was no physical involvement of the subjects in the research. The patient data were provided anonymously. This study was done in accordance with the Declaration of Helsinki guidelines for human research, 1964, and amended in 2013 (64th World Medical Association General Assembly, Fortaleza, Brazil).

Potential predictors
The potential predictors included patients' demographic characteristics, medical conditions, and dental status. The predictors on demographic characteristics included: • Sex (Female/Male); • Age in years at the diagnosis of COVID-19; • Body mass index (BMI). The BMI was calculated based on the height and weight noted in the most recent patients' electronic health records with the maximum retrieval period of one year.
The predictors on medical conditions included: • The information on medical conditions was first collected from electronic health records. When a medical condition was not mentioned in a patient file, but the corresponding medication was documented (e.g. metformin and/or insulin, statins, and antihypertensive drugs), the patient was considered to have such medical disorder. Besides, the smoking status of patients was also collected from the patients' health records. The patients were classified into current smokers, previous smokers, and non-smokers.
The predictors on dental status included: • Number of remaining natural teeth excluding the third molars (ranging from 0 to 28). The number of remaining natural teeth was measured by counting all teeth visible on the OPG including radices relicta. Pontics of fixed partial dentures and prosthetic dentures were not counted as teeth. In the analysis, the number of remaining teeth was classified into 0 teeth, 1-19 teeth, and 20-28 teeth based on the commonly used cutoffs [20,29]. • Number of implants the patients received.
Both of the two dental variables above are commonly used as proxy clinical indicators for dental health, such as periodontal diseases and dental caries [30,31]. The predictors on dental status were collected based on the orthopantomogram (OPG) taken within the past five years.

Outcome (endpoint)
The endpoint of the study was the presence or absence of the critical outcomes of COVID-19. The course and outcome of COVID-19 was classified into (1) ambulatory; (2) hospitalized; (3) ICU admission or death, based on the WHO Clinical Progression Scale [32]. In the study, the critical outcomes were defined as ICU admission or death, while the non-critical outcomes were defined as ambulatory or hospitalized without ICU admission [32,33].

Missing data
The multiple imputation technique was used for the missing values in both the derivation and validation datasets. We created m = 30 imputed datasets with 10 iterations and used predictive mean matching (PMM) for imputing the missing values. All the potential predictors and the outcome variables were included in the imputation model. In the imputation model, number of remaining teeth and number of implants were included as continuous variables.

Development of the models 2.4.2.1. Screening of potential predictors and modeling.
Multicollinearity of the potential predictors was tested using the variance inflation factor (VIF). When a VIF value of a predictor was higher than 5 [34], collinearity was considered present and the predictor was excluded from the following analysis.
Two prediction models were developed for the prediction of death only and for the prediction of the critical outcome due to COVID-19 (death or ICU admission combined). For each outcome variable, the univariate association between each potential predictor and the outcome variable was first assessed with univariate logistic regression analyses. Predictors with a p-value of ≤0.15 were selected for the subsequent multivariate analyses. Multivariate binary logistic regression analysis with backward selection (predictors with p > 0.15 were removed from the models) was used to further assess the association of potential predictors with the outcome in the multivariate setting, and to develop the prediction model.

Shrinkage factor.
To prevent the overfitting of the current model that has been developed from a derivation dataset and for over-optimism of a model applied in similar future populations, the regression coefficients of the predictors in the models were multiplied by a shrinkage factor [35,36]. A shrinkage factor ranges from 0 to 1 and was derived using the bootstrapping procedure with 100 bootstrap samples.

Calibration.
Calibration is defined as the agreement between predicted outcomes and observed outcomes [37]. The calibration of the models was assessed by plotting the predicted individual outcomes against the observed actual outcomes. For this, study members were grouped into deciles based on their predicted probabilities for the outcomes. The prevalence of the outcome events within each decile represents the observed probability. The mean of the individual predicted probabilities within each decile represents the predicted probability. In the calibration plot, the observed and predicted probabilities were compared across the range of predicted risk. The overall observed: expected ratio (O:E ratio) [37] was also used for the assessment of the overall calibration of the models. The O:E ratio was obtained by dividing the prevalence of the outcomes (observed) with the mean of individual predicted probabilities of the outcomes (expected) within the cohort [38]. An O:E ratio <1 indicates an overestimation of the models, while an O: E ratio >1 indicates an underestimation of the models [39]. An O:E ratio between 0.8 and 1.2 indicates that the calibration of the model is acceptable [39]. The calibration of the multivariate models was also assessed using the Hosmer-Lemeshow goodness-of-fit statistic test (HL test). A p-value of >0.10 in the HL test indicates that the model fits the observed data [40].

Discrimination.
Discrimination is defined as the ability to differentiate between those with and those without the outcome event [37]. The area under the receiver-operating characteristic curves (AUC) was used to assess the discrimination of the models [41]. An AUC of 0.70 to 0.80 indicates that the discrimination of the models is acceptable, while an AUC of ≥0.80 indicates that the discrimination of the models is excellent to outstanding [42].
The optimal cutoff for the predicted probability of the models was defined as the predicted probability with the maximum sum of sensitivity and specificity in the receiver-operating characteristic curve (ROC).

Clinical values.
Clinical values of the models at the optimal cutoff for predicted probability were assessed using prevalence (prior probability) and posterior probabilities of the outcome events. The posterior probability was defined as positive predictive value (PPV) and negative predictive value (NPV). The (added) predictive value of the models for ruling in an increased risk of the outcome events was defined as the PPV minus prevalence, while that for ruling out an increased risk of the outcome events was defined as NPV minus complement of prevalence.

Scoring system.
A clinical prediction rule for the outcome events was developed to provide an estimate for individual patients of their absolute risk of developing the outcome events. For the final multivariate binary logistic regression models, the individual probability (P) of the outcome events is predicted with the following formula: Where β is the shrunken regression coefficient of a predictor in the models.
To facilitate the calculation of the predicted probabilities of the outcome events in individual patients separately, the multivariate logistic regression models were converted to a score chart. The score of each included predictor in the score chart was produced by the shrunken regression coefficients being divided by the smallest regression coefficient of the predictors and subsequently rounded. Line charts were then developed to help determine the predicted probability of the outcome events.

External validation of the models
To assess the general applicability of the models, the derived prediction models were externally validated based on the validation cohort. In the validation cohort, the predicted probability for the outcome event of each patient was calculated based on the developed prediction models in the derived cohort mentioned above. The performance of the models in the validation cohort was also assessed in aspects of calibration and discrimination. The prevalence, PPV, NPV, and the added predictive values of the models in the validation cohort were calculated based on the cutoff for predicted probability established in the validation cohort.
All the statistical procedures mentioned above were performed based on the imputed datasets via SPSS software 27.0 (IBM, New York, USA) and R software 4.0.4 (R Development Core Team, Vienna, Austria). The discrimination, calibration, added values, and scoring system of the models were all assessed based on the shrunken regression coefficients.

Characteristics of included participants
Fig . 1 shows the flowchart of the patients included in the present study. Table 1 presents the main characteristics and the information on the missing values of the potential predictors and the outcome variables of both derivation and validation cohorts. In the derivation cohort, a total of 285 unvaccinated patients with the diagnosis of COVID-19 (138 females and 147 males) were enrolled, while 27 patients were excluded because they have been vaccinated, their vaccination status were unknown, or they were aged <18 years. The mean age ± standard deviation (SD) of the 285 patients was 61.1 ± 17.0 years. The mean age ±SD of male patients was 63.8 ± 14.6 years, while that of female patients was 58.2 ± 18.9 years. The mean BMI of the patients was 28.0 ± 5.3, and mean number of teeth was 15.9 ± 11.6. Eight percent of the patients were smokers. The prevalence of medical conditions ranged from 8% to 34%. Of the 285 patients, 48 patients (17%) died due to COVID-19, and 62 patients (22%) developed the critical outcomes (i.e. ICU admission or death) due to COVID-19. In the validation cohort, a total of 352 unvaccinated patients (199 females and 153 males) were enrolled in the study, while 187 patients were excluded because they have been vaccinated or their vaccination status were unknown. The mean age ±SD of the patients was 55.4 ± 21.8 years. The mean age ±SD of male patients was 60.7 ± 20.2 years, while that of female patients was 51.3 ± 22.2 years. The mean BMI of the patients was 26.9 ± 5.0, and mean number of teeth was 20.0 ± 10.4. Eight percent of the patients were smokers. The prevalence of medical conditions ranged from 6% to 34%. Of the 352 patients, 39 patients (11%) died, and 52 patients (15%) developed the critical outcomes. There is a statistically significant difference in patients' demographics, dental status, some medical conditions, and outcome variables between derivation and validation cohorts. In order to provide an overview on the distribution of each predictor on the outcomes for both derivation and validation cohorts, Table 2 is presented based on multiple imputation.

Screening of potential predictors and modeling
Before modeling, we checked the collinearity of all the potential predictors using VIF values. The VIF values of all the predictors were lower than 5. Therefore, the multicollinearity between predictors was ignorable and all the predictors were included for further analysis. Next, we pre-screened the potential predictors individually using univariate binary logistic regression analyses; these results are presented in Table 3. In the univariate analyses, when death due to COVID-19 was regarded as the endpoint, age (OR: 1.105, 95% CI: 1.     (Table 4). This indicates that COVID-19 patients with older age, lower number of teeth, and presence of DM, COPD, and HT have higher risk of obtaining critical outcomes due to COVID-19.

Performance of the prediction models
As described above, two prediction models were developed. The next step was to assess the performance of the models, in aspects of discrimination and calibration. Discrimination of the models was assessed with AUC. The original AUCs of the models for death and critical outcome were 0.86 (95% confidence interval [95%CI], 0.80-0.91) and 0.81 (95%CI, 0.75-0.86), respectively ( Fig. 2A and B). To avoid the overestimation of the prediction, the AUCs were shrunken by the shrinkage factors. The shrinkage factors of the two models for death and critical outcome were 0.90 and 0.88, respectively. The shrunken AUCs of the models based on the bootstrapping were 0.85 and 0.79, respectively, which indicated that the discrimination of the two models was both good to excellent. To assess the calibration of the two models, the calibration plots and O:E ratios were used. The calibration plots (Fig. 3A and B) showed that there was a good fit between the predicted probability and actual probability of the outcomes in both models because most plotted dots were lying close to the diagonal lines. The O:E ratios of the two models were 0.98 (95%CI, 0.76-1.25) and 1.00 (95%CI, 0.80-1.24), respectively, which indicated that the overall calibration of the two models was excellent. With resulting values for the HL tests of 0.64 and 0.61, the two models were shown to have good fit.

Clinical added values of the prediction models
To assess the clinical added values of the models, the optional cutoffs for the predicted probability, PPV, and NPV were determined. The optimal cutoffs for the predicted probability of the two models were both 0.15. Table 5 presents the prevalence, sensitivity, specificity, PPV, and NPV of the two models. When death was the endpoint, the prevalence of death, PPV and NPV were 0.17 (95%CI: 0.13 to 0.22), 0.38 (95%CI: 0.29 to 0.47) and 0.98 (95%CI: 0.94 to 0.99), respectively. The added value of the model for ruling in the Table 3 Univariate binary logistic regression analysis of the potential predictors for death and critical outcomes of COVID-19 based on the derivation cohort (N = 285).   risk of death, which was calculated as PPV minus prevalence, was 0.21 (95%CI, 0.11-0.31), while that for ruling out the risk of death, which was calculated as NPV minus the complement of the prevalence, was 0.14 (95%CI, 0.10-0.19) (Table 5A) (Table 5B). This indicates that the clinical added values of the model were significantly  The optimal cutoff for the predicted probability was 0. 15. b The optimal cutoff for the predicted probability was 0.14. c The optimal cutoff for the predicted probability was 0.18; PPV, positive predictive value; NPV, negative predictive value; CI, confidence interval; critical outcome, death or ICU admission.

N. Su et al.
sufficient for both ruling in and out the critical outcomes due to COVID-19.

Score chart and line charts of the prediction models
To enhance the clinical usefulness of the models, we transformed the models into a score chart (Table 6) and two line charts ( Fig. 4A  and B). A clinician can easily calculate the sum scores of a patient for prediction of death and critical outcome and determine the corresponding predicted probabilities based on the sum scores by using Fig. 4A and B. The cutoffs of the sum scores of the two models were 83 and 79, respectively.
For example, a patient was diagnosed with COVID-19 in the hospital. He was 60 years old with a total of 15 remaining teeth. He had a history of COPD and HT but had no DM and CKD. Therefore, based on the score chart (Table 6), the sum score for death can be calculated as 1*60 + 17+0 + 13 = 90, whereas the sum score for critical outcome is 1*60 + 20+21 + 0+16 = 117. Both of the two sum scores of the patient were above the cutoff scores (83 and 79), and therefore the patient has a high risk of death or critical outcome due to COVID-19. Based on Fig. 4A and B, the predicted probability of the patient for death and critical outcome is around 23% and 42%.

External validation
To determine whether the prediction models performed well in a different cohort, the discrimination, calibration, and added values of the models were assessed based on the validation cohort. The AUCs of the models for death and critical outcome based on the validation cohort were 0.85 (95%CI, 0.80-0.90) and 0.78 (95%CI, 0.73-0.83), respectively, which indicated that the discrimination of the models was good to excellent in the validation cohort ( Fig. 2C and D). Based on the calibration plots ( Fig. 3C and D), the calibration of the models was acceptable in general, but an overestimation was observed for the patients with high predicted risks of the outcomes in both models. The overall O:E ratios of the two models were 0.65 (95%CI, 0.49-0.85) and 0.67 (95%CI, 0.52-0.84), respectively, which also indicated an overestimation of the models. The optimal cutoffs for the predicted probability of the two models were 0.14 and 0.18, respectively. Table 5 also presents the prevalence, sensitivity, specificity, PPV, and NPV of the two models based on the validation cohort. When death was the endpoint, the prevalence of death, PPV and NPV were 0.11 (95%CI: 0.08 to 0.15), 0.27 (95%CI: 0.20 to 0.35) and 0.99 (95%CI: 0.97 to 1.00), respectively. The added value of the model for ruling in the risk of death was 0.16 (95% CI, 0.08-0.25) in addition to the prevalence, while that for ruling out the risk of death was 0.10 (95%CI, 0.07-0.14) in addition to the complement of the prevalence. When critical outcome was the endpoint, the prevalence of critical outcomes, PPV and NPV were 0.15 (95%CI: 0.11 to 0.19), 0.28 (95%CI: 0.22 to 0.35) and 0.96 (95%CI: 0.93 to 0.98), respectively. The added value of the model for ruling in the risk of critical outcome was 0.13 (95%CI, 0.05-0.21) in addition to the prevalence, while that for ruling out the risk of critical outcome was 0.11 (95%CI, 0.07-0.16) in addition to the complement of the prevalence. This indicates that the added values of the prediction models were significantly sufficient for both ruling in and out the death and critical outcomes due to COVID-19 when the Table 6 Score chart of the models for prediction of death and critical outcomes of COVID-19.

Predictors
Death Critical outcome CKD, chronic kidney diseases; HT, hypertension; DM, diabetes mellitus; COPD, chronic obstructive pulmonary diseases; critical outcome, death or ICU admission. The algorithms for the calculation of an individual's sum scores for death and critical outcome due to COVID-19 were presented below: Sum score for death = 1*Age + 17*presence of 1-19 teeth + 13* presence of 0 teeth + 11*presence of CKD + 13*presence of HT. Sum score for critical outcome = 1*Age + 20* presence of 1-19 teeth + 18* presence of 0 teeth + 21*presence of HT + 16*presence of DM + 16*presence of COPD. models were used in the validation cohort.

Discussion
In the present study, two prediction models for death and critical outcome (death or ICU admission) in unvaccinated COVID-19 patients were derived and externally validated, based on the predictors which can be easily obtained in clinical practice, including patients' demographic characteristics, medical conditions, and dental status. Number of teeth was an important predictor for death and/or critical outcomes due to COVID-19. In the models, older age, lower number of remaining natural teeth, and presence of CKD, HT, DM, and COPD were the important predictors for death and/or critical outcomes due to COVID-19. The performance of the models, in aspects of discrimination and calibration, were acceptable in both derivation and validation cohorts. The clinical added values of the prediction models for ruling in and out the poor prognosis of COVID were sufficient. To our best knowledge, this is the first study which considered dental status as a potential predictor for the prognosis of COVID.
In the study, the number of remaining natural teeth was considered an important independent predictor for critical outcomes, in particular, for death. Number of teeth is a commonly used proxy variable for periodontitis. Up to date, multiple biological mechanisms have been implicated in the etiopathogenetic link between periodontitis and critical outcomes of COVID-19 [18]. For example, periodontitis can increase the Angiotensin-converting enzyme 2 (ACE2) expression in the respiratory epithelium [43]. Because ACE2 is found to be a molecular target used by SARS-CoV-2 to infect human cells [44], the increase of ACE2 expression can increase the viral infectiveness in the lower airways and therefore worsening the courses of COVID-19 [18,43]. Besides, periodontitis is associated with upregulation of the neutrophil extracellular traps, which could predispose to more severe lung damage and ARDS in COVID-19 progression [45]. In addition, periodontitis is associated with the chronic low-grade elevation of systemic markers of inflammation, such as interleukin (IL)-1, IL-6, and C-reactive protein [46]. Such inflammatory markers may exacerbate COVID-19 cytokine storm and therefore worsening the progress of COVID-19 [46]. Another potential mechanism is that the ulcerated epithelium of periodontal pockets, caused by periodontitis, can facilitate the entry of the virus into the hematic circle and then to the small vessels in the lung periphery [47]. This may induce pulmonary vasoconstriction, immunothrombosis, and subsequent acute respiratory distress syndrome ARDS [18].
With regard to the performance of the two prediction models, the discrimination in both derivation and validation cohorts was found to be good to excellent, with AUCs ranging from 0.79 to 0.86. This indicated that the prediction models may have an excellent ability to differentiate the patients with the critical outcomes from those without. The calibration of the models in the derivation cohort was good based on both the calibration plots and O:E ratios. However, in the validation cohort, the O:E ratios of the prediction models were 0.65 and 0.67, respectively, which indicated an overestimation of prediction models. Based on the calibration plots, the overestimation may mainly occur when the predicted risk was larger than 0.4. This indicated that when the predicted risk of a patient was larger than 0.4, the actual risk of the patient may be lower. However, the optimal cut-off predicted risks of the models in the validation cohort were 0.14 and 0.18, respectively, which was much lower than 0.4. Therefore, even if the overestimation was present for the high predicted risks, it is not very likely to bias the risk stratification of the patients based on the optimal cut-offs.
The clinical added predictive values of the models for ruling in and out death were 0.21 and 0.14, respectively in the derivation cohort. The added values for ruling in and out critical outcomes were 0.18 and 0.19, respectively, in the derivation cohort. All the added values were statistically significant based on their 95%CIs. In the validation cohort, the added values for both ruling in and out the outcome events tended to be lower than that in the derivation cohort, but the added values were still statistically significant. Therefore, in general, the performance of the prediction models in both derivation and validation cohorts was acceptable.
The present models can provide clinicians with information on the prognosis of COVID-19 patients and help clinicians with the early identification and triage of COVID-19 patients, thus aiding in delivering proper care, reducing the fatality rates of patients, and optimizing the use of in-hospital resources [48,49]. The present study can also increase the awareness of clinicians on the close link between dental health and general health of individuals. In addition, the predictors included in the models could also be collected at an early stage, outside the hospitals. With the findings of the present study, primary caregivers can identify patients who may have a higher risk for poor prognosis of COVID-19 in their practice before the patients are infected. General practitioners and dentists can create the awareness with these patients about the possible prognosis if infected with COVID-19 and they can caution the patients to take necessary precautions to prevent the infection.
In interpreting the findings of the present study, some limitations should be taken into consideration. First, the number of remaining teeth of the patients was not counted at the diagnosis of COVID-19. Instead, it was counted based on the OPG taken within the past five years, and the time between the OPG and the diagnosis of COVID-19 varied between patients. As the number of teeth may be decreasing slowly over time due to caries, periodontal diseases, trauma, etc., it can be expected that patients who took the OPG more recently may tend to have lower number of teeth than the same patients if they had taken the OPG earlier. In the analysis, the different periods in the patients between the OPG and the diagnosis were not corrected for the number of teeth, which may bias the association between number of teeth and the critical outcomes. However, we categorized the number of teeth into three ordinal categories (0 teeth, 1-19 teeth, and 20-28 teeth) in the analysis, rather than including the continuous number of teeth in the models directly. This can, on one hand, minimize the bias to a large extent, and on the other hand, simplify the counting of the number of teeth by medical clinicians and prevent the impact of miscounting on the prediction to a large extent. Second, 25%-33% of the included patients had missing values in number of teeth and number of implants. One of the reasons for the missing values was that the patients may take the OPG more than five years before the diagnosis of COVID-19, which was outdated and may not validly reflect the present dental health. Another reason was that those patients did not have an indication for an OPG. Therefore, those missing values were assumed to missing at random (MAR). The multiple imputation technique was used to deal with the missing values, which can preserve the sample size and reduce the bias of the MAR data caused by the missing values when the proportion of missingness is relatively large [50]. Third, the predictors on medical conditions were all routinely collected from patients' electronic health records, which were more accessible, informative, standardized, and reliable than patients' self-reported methods [51,52]. This, however, may hinder the generalizability of the prediction models in the hospitals where the patients' health information was not well and comprehensively documented. Fourth, the COVID variant types were not identified in the included patients, therefore, the models were not corrected for variant types. All the included patients were diagnosed with COVID-19 between January 2020 and July 2021, which indicated that those patients probably had a mixture of variant types, including Alpha, Beta, Gamma, and Delta [53]. Those variant types may have different clinical manifestations, transmissibility, morbidity, and mortality of COVID-19 [54]. Therefore, not correcting for the variate types of COVID in the models may impair the performance of the prediction models. In addition, from December 2021, the Omicron variant rose rapidly around the world and became dominant in many countries, including the Netherlands. Omicron seems to be milder with lower mortality but more transmissible than the previous variants based on the limited evidence so far [55][56][57]. Whether the prediction models can be generalized to the patients with Omicron variants is still unknown and needs to be further validated. However, recent evidence showed that a prediction model for COVID-19 critical outcomes which was developed based on the unvaccinated patients in early 2020 still showed acceptable performance when it was externally validated in the patients with Omicron variants and mixed vaccination status [58]. Fifth, increasing number of patients were reinfected with COVID-19. The prior infection of COVID may be associated with the lower risk of death of the reinfection [59], so it may be an important predictor for the critical outcomes of COVID-19. However, patients' prior infection was not considered in the included study. This is because patients' prior infection needs to be confirmed with antibody or serology tests, which cannot be rapidly obtained and may hinder the application of the prediction models in clinical practice for the rapid triage of patients. Last, in the previous studies, several clinical (e.g. oxygen saturation [60]), radiographic, and laboratory markers (e.g. serum C-reactive protein, serum lactate delydrogenase [48]) have been shown to be significantly associated with the poor prognosis of COVID-19 and those markers have been considered potential predictors for prognosis of COVID-19 in several previous studies [26,48,60]. In addition, several immunological markers have been found to be significantly associated with prognosis of COVID-19. For example, Devalaraja-Narashimha et al. [10] found the importance of specific complement pathway components as prognostic biomarkers of prognosis of COVID. Multiple complement markers were significantly associated with increased mortality of COVID-19, including high plasma sC5b-9, C3a, factor Bb levels, and low mannan-binding lectin levels [10]. Bekbossynova et al. [11] reported that the increase of central memory T cells and decrease of effector memory T cells in peripheral blood may be potential biological markers for prognosis of COVID-19. Adding those markers as potential predictors in the prediction models may increase the performance of the models and improve the predictive accuracy. However, those markers were not included as potential predictors in the present study. This is because those markers were not routinely collected for the ambulant COVID patients (i.e. patients with mild symptoms) in the study. Besides, we mainly aimed to develop prediction models which can be easily used for a rapid risk stratification at patients' first intake in hospitals. Therefore, only the variables which can be easily and rapidly obtained in clinical practice were considered the potential predictors. Considering that the performance of the models has already been satisfactory based on the predictors we used, adding more clinical, radiographic, and laboratory predictors may add very limited values in the model performance due to ceiling effect while increasing the complexity of the models.
It should note that the aim of the study was to find out the significant independent predictors which have statistical associations with the critical outcomes of COVID-19 to reach the best prediction performance. Whether those predictors also have causal associations with the critical outcomes is not the focus of the study. Therefore, the potential confounders are not needed for the adjustment in the present study.
Future researchers are suggested to further validate and update the prediction models in the COVID-19 patients with the Omicron variant and in the vaccinated patients. Besides, it is recommended to assess whether the performance of the models improves when adding the variant types of COVID-19 and the history of COVID-19 infection as separate predictors.

Conclusions
COVID-19 patients with older age, lower number of teeth, and presence of CKD and HT have higher risk of death due to COVID-19, while COVID-19 patients with older age, lower number of teeth, and presence of DM, COPD, and HT have higher risk of obtaining critical outcomes due to COVID-19. The two prediction models for death and critical outcomes due to COVID have been developed and externally validated. The performance of the models, in aspects of discrimination and calibration, was acceptable in both derivation and validation cohorts. The added predictive values were considerable for both ruling in and ruling out the death and the critical outcome in decision-making. The study suggests that number of teeth is an important independent predictor for death and critical outcomes due to COVID-19 and that the models can be used as a reliable screening tool for early and rapid risk stratification of unvaccinated COVID-19 patients at intake in hospital settings.

Author contribution statement
Naichuan Su; Marie-Chris H.C.M. Donders: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.
Jean-Pierre T. F. Ho: Conceived and designed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data. Valeria Vespasiano; Jan de Lange: Conceived and designed the experiments; Contributed reagents, materials, analysis tools or data. Bruno G. Loos: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data.

Funding statement
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement
Data will be made available on request.

Declaration of interest's statement
The authors declare no competing interests.