Development and Validation of a Prognostic Model for Multi-Drug-Resistant Non-Hospital-Acquired Bloodstream Infection

Bloodstream infections (BSI) are an increasing cause of admissions to hospitals. Non-hospital-acquired BSI are defined by blood cultures that are positive less than 48 hours after admission, but a relevant difference exists between community-acquired and healthcare-associated (HCA) BSI in terms of risk of multidrug resistance (MDR). We planned a retrospective study in three different cohorts in order to develop and to temporally and spatially validate an easy and rapid prognostic model for identifying MDR non-hospital-acquired (non-HA) BSI. The pathogens most involved in BSI are Staphylococcus spp. and Escherichia coli, responsible for about 75% of all MDR isolated. The model includes age, gender, long-term care facility admission, immunocompromise, any recent invasive procedures and central line placement, recent intravenous treatment and antibiotic treatment. It shows an acceptable performance, especially for intermediate probabilities of MDR infection, with a C-index of 70%. The model was proposed in a nomogram that could allow better targeting of antibiotic therapy for non-HA BSI admitted in hospital. However, it should be further validated to determine its applicability in other populations.


Pathogen-Related Prognostic Risk Factors
Prognosis partly varies according to the pathogen. A recent systematic review established that the most important predictors of mortality in patients with MRSA (Methicillin-Resistant Staphylococcus aureus) bloodstream infection are age, patient condition, timing and appropriateness of antimicrobial treatment, surgical intervention and disease severity as evaluated by the APACHE II score [5]. The incidence of BSI due to Escherichia coli is higher than that of Staphylococcus aureus bacteremia, but the mortality is lower: the in-hospital and 30-day case-fatality is approximately 10-15%; mortality is highly dependent on age, hospital acquisition and co-morbidity, initial presentation with septic shock/severe disease and resistance to fluoroquinolones and third-generation cephalosporin resistance [1]. Although less frequent than E. coli among bacteremia isolates, BSI due to extended-spectrum beta-lactamase (ESBL)-positive K. pneumoniae carries a worse prognosis, including more frequent intensive care unit admission and higher 30-day case-fatality or in-hospital mortality. Many European and North American studies show an association between ESBL-positive Enterobacterales and both excess length of stay and increased mortality rate [1,[6][7][8]. Klebsiella pneumoniae is also the most prominent among carbapenem-resistant Gram-negative bacteria causing BSI. The case-fatality of BSI due to carbapenem-resistant Klebsiella is higher than that due to carbapenem-susceptible K. pneumoniae, and the likelihood of initially inappropriate therapy and of suboptimal definitive therapy was significant and a major prognostic factor for poor outcomes before the availability of newer antibiotics. A particularly high risk of fatal outcome after BSI due to carbapenem-resistant Gram-negative bacteria in general has been reported for immunocompromised patients [1].

Host-Related Prognostic Risk Factors
In patients with a history of malignancy, the most common predictors of mortality among BSI caused by Enterobacteriaceae are septic shock, pneumonia and ICU admission; furthermore, children and hematological malignancy are associated with higher mortality [6][7][8]. The relationship between some malignancies and prolonged bacterial infection has already been shown. For example, the etiology of gastric cancer is related to alcohol, smoking and unhealthy diet, but also to H. pylori infection and the following changes in gastric microbial population as the cancer develops [9][10][11].
A recent Japanese study shows that an estimated glomerular filtration rate (eGFR) lower than 30 mL/min/1.73 m 2 , procalcitonin of at least 100 ng/mL and primary infectious Antibiotics 2023, 12, 955 3 of 13 foci in the gastrointestinal tract or in the respiratory system are independent prognostic factors for short-term survival in patients with BSI [12].
Other studies evaluated prognostic factors in more selected clusters of patients: a machine-learning-based study showed that the most important predictors of death in patients with concomitant candidemia and bacteriaemia are age, serum creatinine level, leukocyte and lymphocyte counts, total bilirubin level, procalcitonin level, endotoxic shock, length of stay in the intensive care unit (ICU), length of stay in the hospital and total parental nutrition [13]; another recent Chinese cohort study indicates that ICU admission, coronary heart disease, biliary infection and the use of tigecycline are independent prognostic factors of 90-day mortality in elderly people (≥65 years old) with both typical (e.g., transplant recipient, hematological malignancy) and atypical (e.g., diabetes mellitus, liver cirrhosis, burns, postoperative) immunosuppression status, and that a decrease in body mass index is a protective factor [14].

Bloodstream Infection Classification
BSI are usually categorized as community-acquired (CA-BSI) or hospital-acquired (HA-BSI) based on the timing of the positivity of blood cultures (i.e., before or after 48 h since the arrival in hospital, respectively) [15]. HA-BSI pose a higher risk of morbidity and mortality compared with CA-BSI. However, an additional, intermediate, category of BSI, healthcare-associated (HCA-BSI), has been proposed [16]. This category is similar to CA-BSI in terms of the timing of culture results, but it also has at least one risk factor from among the following: recent hospitalization (i.e., at least 2 days in the last 3 months); admission in a long-term care facility in the last 30 days; antibiotic treatment (at least 5 days in the last month); intravenous therapy for chronic ulcers in the last month; hemodialysis in the last 30 days; and immunocompromise [15]. The risk of multidrug resistance (MDR) in the non-HA BSI, specifically in the HCA-BSI, has been increasing in the past few years [17].
Due to this possible higher risk of MDR among some non-HA BSI, and specifically among HCA-BSI compared with CA-BSI, we hypothesized the need of a rapid, empiric antibiotic treatment for these patients, and we aimed to develop a prognostic model for identifying non-HA BSI at high risk of MDR pathogens and to internally and externally validate such a model. In the cohort used for temporal external validation, the female patients numbered 149 and 121 among the CA-and HCA-BSI cases (p = 0.03). The median ages of the enrolled patients were 73.5 years (range 18-97) for CA-BSI and 73 years (range 24-94) for HCA-BSI (p = 0.98).

Demographic Characteristics and Risk Factors
In the spatial external validation cohort, the female patients numbered 65 and 56 among the CA-and HCA-BSI cases (p = 0.818). The median ages of the enrolled patients for spatial external validation were 72.4 years (range 18-95) for CA-BSI and 73.4 years (range 25-90) for HCA-BSI (p = 0.629). Table 1 summarizes the baseline characteristics in each cohort, including the risk factors for HCA-BSI. In the model development cohort, immunocompromise was due to prolonged steroid treatment in 116 patients (20.9%), chemotherapy in 80 (14.4%), and/or radiation therapy in 10 (1.8%) in the last month, and/or treatment for previous transplantation (solid or hematological) or for other medical conditions in 58 patients (10.4%), splenectomy in 11 (2%) and known human immunodeficiency virus infection in 4 (0.7%).
In the temporal and spatial validation cohorts, respectively, prolonged steroid treatment was reported in 66 and 30 patients (10.8% and 11.9%), chemotherapy in 36 and 19 (5.9% and 7.5%), and/or radiation therapy in 3 and 2 (0.5% and 0.8%) in the last 30 days, and/or treatment for previous solid or hematological transplantation or for other conditions in 23 and 11 (3.8% and 4.4%), splenectomy in 3 and 2 (0.5% and 0.8%) and known human immunodeficiency virus infection in 6 and 1 (1% and 0.4%). Table 2 reports the additional risk factors evaluated for MDR. Gram-negative agents were detected in 51.6% of the patients in the development cohort, and in 53.7% and 59% of the patients in the temporal and spatial validation cohorts, respectively.  Table 3 shows the results of MDR pathogens identified in the blood cultures as single agents. A polymicrobial positivity was found in 30, 34 and 21 patients, in the development, temporal and spatial validation cohorts, respectively. In the development cohort, 51 patients suffered from septic shock (9.2%), compared with 58 (9.5%) in the temporal validation cohort, and 18 (7.1%) in the spatial validation cohort.

Prognostic Model Development
Based on the data available in the development cohort and previous data already published and suggested as potential predictors in the international literature, we identified eight predictors of MDR among non-HA BSI: age, gender, long-term care facility admission, immunocompromise, any recent invasive procedures, any central venous catheterization, recent intravenous treatment and antibiotic treatment. This model showed an area under the receiver operating characteristic curve of 71.6% (95% confidence interval 61.8-72.4%). Table 4 shows the odds ratios (OR) of MDR for each of these variables in the uni-and multivariable models. Figure 1 shows the calibration of the developed prognostic model along with its nomogram. It was built using the already reported covariates. Such a model had a C-index of 70%.

Model External Validation
The prognostic model built in the development cohort was validated in two different cohorts. One of them was collected in the same hospital (i.e., Città della Salute e della Scienza University Hospital, Molinette) but over a different time period. This cohort included 609 patients with 177 MDR BSI.  1 any type of immunocompromised state, such as recent (i.e., in the last 30 days) or ongoing chemotherapy, and/or radiotherapy, and/or immunosuppressive therapy; prior solid organ or bone marrow transplant, splenectomy, congenital or acquired immunodeficiency. 2 any invasive procedures in the last 4 days, including urinary catheterization, any type of drainage, percutaneous endoscopic gastrostomy, enteral and/or endovascular nutrition, arterial catheterization with or without additional procedures. 3 central lines including those needed for urgent hemodialysis.

Temporal Study
The prognostic model built in the development cohort was validated in two different cohorts. One of them was collected in the same hospital (i.e., Città della Salute e della Scienza University Hospital, Molinette) but over a different time period. This cohort included 609 patients with 177 MDR BSI. Figure 2 reports the predicted versus observed probability of MDR BSI (calibration) as calculated in this temporally different cohort.
The C-index was 75.9%.

Spatial Study
A third cohort was collected for performing an additional external validation of the proposed prognostic model. This cohort referred to an entire single year of blood cultures including 253 patients and 83 MDR BSI infections. The data were collected in a different University-affiliated hospital in Turin, Italy (San Giovanni Bosco Hospital-Local Healthcare Unit of the Città di Torino). Figure 3 shows the calibration of the model in such a cohort (spatially external validation). The C-index was 77%.  Figure 2 reports the predicted versus observed probability of MDR BSI (calibration) as calculated in this temporally different cohort.
The C-index was 75.9%.

Spatial Study
A third cohort was collected for performing an additional external validation of the proposed prognostic model. This cohort referred to an entire single year of blood cultures including 253 patients and 83 MDR BSI infections. The data were collected in a different University-affiliated hospital in Turin, Italy (San Giovanni Bosco Hospital-Local Healthcare Unit of the Città di Torino). Figure 3 shows the calibration of the model in such a cohort (spatially external validation). The C-index was 77%.

Discussion
We have developed and validated an eight-variable prognostic model for the risk of MDR non-HA BSI in three different cohorts of patients admitted to hospital.
Our results suggest this possible prognostic model for high risk for MDR BSI among the non-HA BSI. This model was developed and then separately validated in three different cohorts. The external validation was repeated using two sets of data; one of them was collected in the same University Hospital, the second one in a different University-affiliated hospital in Turin, Italy. The double validation was performed in order to better understand the validity of the prognostic model. This model was summarized in a nomogram, a fast-performing tool for assessing the risk of MDR at a patient's admission to hospital.
The need to identify high risk of MDR among non-HA BSI, usually due to frequent

Discussion
We have developed and validated an eight-variable prognostic model for the risk of MDR non-HA BSI in three different cohorts of patients admitted to hospital.
Our results suggest this possible prognostic model for high risk for MDR BSI among the non-HA BSI. This model was developed and then separately validated in three different cohorts. The external validation was repeated using two sets of data; one of them was collected in the same University Hospital, the second one in a different University-affiliated hospital in Turin, Italy. The double validation was performed in order to better understand the validity of the prognostic model. This model was summarized in a nomogram, a fast-performing tool for assessing the risk of MDR at a patient's admission to hospital.
The need to identify high risk of MDR among non-HA BSI, usually due to frequent contacts with the healthcare systems (for this reason, called HCA-BSI), is linked to the different MDR risk within the large group of patients presenting for BSI and with a blood culture positivity before 48 h after admission. During the past few decades, the number of patients who were treated and followed up outside the hospital has gradually increased, and this has changed the number of patients affected by BSI and the most frequent pathogens [1,2]. For these reasons, the idea of "healthcare-related" infection is a relatively new one, but it is crucial for the correct management of patients affected by BSI [15].
Our data confirmed a significant different mortality risk for CA-and HCA-BSI, except for the spatial validation cohort, where the difference was present but without any significance; likely this is due to its smaller size compared with the other 2 cohorts.
Moreover, as previously reported in the international literature, the pathogens most likely involved in BSI are Staphylococcus spp., and Escherichia coli [19]. In our study, they are responsible of about 75% of all MDR isolated.
The presented model was developed using readily available data. Unlike previous studies and prognostic models [12][13][14]20,21], all parameters reflect patient demographics (i.e., age and gender) and recent medical history data. It shares some parameters with the healthcare-associated infection definition [15] by adding information about the need of central lines and any recent invasive procedures. Some of these procedures might be very frequent, as the placement of a urinary catheter, not only in the Emergency Department for acute illness, but also among out-of-hospital patients for worsening of chronic diseases (e.g., solid or hematologic neoplasm).
In general, the prognostic model shows a good performance across the three validation cohorts. In the development cohort, it had a better performance for the intermediate risk of MDR infection, between 20 and 70%. In the validation cohorts, performance remained acceptable for the same probabilities of MDR infection.
Our study has limitations. Firstly, we collected data in two large hospitals, a University one and a University-affiliated one. The patients referred to similar hospitals might be partially selected by some types of diseases (e.g., hematological and/or advanced oncological diseases) that are not usually treated in all hospitals. This situation might then have an impact on the generalizability of our model as well as the absence of comorbidities in the prognostic models. The inclusion of these comorbidities might have an impact on model performance.
Secondly, we restricted our analysis to the first blood cultures collected for each patient included in the study (for each cohort). It is likely that some patients might be admitted in hospital more than once during each study period, specifically for patients suffering from chronic diseases with high risk of immunocompromise. However, we chose to externally validate our model twice, and its performance remained acceptable. Based on the features of the cohorts we used, the model is not applicable to different settings like out-of-hospital clinics or intensive care units.
The study also presents some strengths. It is based on three large cohorts collected in different period of time. This allowed us to develop the model with enough degrees of freedom and to externally validate it. This second procedure was performed twice. The first external validation was based on data collected in the same hospital where the development cohort was collected, but in a different time frame (2012-2013 vs. 2014-2015). A temporal validation, performed in more-recently treated patients, is a possible way for assessing model performance [22]. However, for better understanding the real performance of the suggested prognostic model, we chose a second external validation, a spatial (or, so called, geographical) one, in a different site. This second process allowed us to confirm our results, but, as already well described in the literature [22], with the drawback of a smaller cohort than the development and temporal validation cohorts. This also allowed us to underline the general consistency of the model.

Materials and Methods
The present study was held in the Città della Salute e della Scienza di Torino University Hospital and in the San Giovanni Bosco University affiliated hospital (Local Healthcare Unit of the Città di Torino), Turin, Italy. Data were retrospectively collected based on the laboratory information system results on positive blood cultures.
The hospital Ethics Committee approved the study protocol (n. 0033874). Informed consent was waived because of the retrospective nature of the study and because the analysis used anonymous data. All data were collected from the laboratory information system, and then from medical records, by five investigators (A.C., A.B., P. Pet., E.G. and M.G.).
The primary outcome of the study was to predict the risk of MDR infection among the non-HA BSI (i.e., CA-and HCA-BSI).
For the development cohort, we recruited adult patients (i.e., age ≥ 18 years) with positive blood cultures admitted at the Città della Salute e della Scienza di Torino, Molinette Hospital between January 2012 and December 2013. Samples collected in intensive care, neurosurgery and cardiac surgery units were excluded. Patients with multiple admissions were included in the study only at their first admission, and data relative to subsequent admissions were excluded.
Since the study outcome was MDR in non-HA BSI, we excluded all positive cultures collected more than 48 hours after admission to hospital, as well as those collected in the Emergency Department while patients were waiting for admission due to hospital overcrowding.
Demographic and clinical characteristics of enrolled patients were reported as median and range for continuous data or number and percentage for ordinal data, as appropriate.
The continuous and the categorical variables were tested by using the non-parametric Wilcoxon rank-sum test for unmatched samples, and the Chi-squared or Fisher's exact test, as appropriate, respectively [23,24].
Multivariable analysis was used to predict MDR among non-HA BSI in the development cohort. Covariates tested were age, gender, long-term care facility admission, immunocompromise, any recent invasive procedures, any central venous catheterization, recent intravenous treatment and antibiotic treatment. The decision to include these variables was taken a priori, based on prior knowledge and the available literature [15,16,22,[25][26][27]. The number of predictors was kept to a minimum in order to avoid overfitting and to maximize the use of the predictive model in future applications.
The accuracy of the proposed model was reported as the area under the receiver operating characteristic curve (AUC ROC) [28]. The model was internally validated by bootstrapping (1000 replications) to obtain overfit-corrected estimates of discrimination and calibration [22].
The discrimination ability was measured using the optimism-corrected C-index [29]. Apparent and internally validated (after a 1000 replication bootstrapping procedure) calibration was visually assessed by evaluating the plot of the observed vs. predictive probability of MDR BSI [22].
A second cohort, using the same inclusion criteria, was identified in the same hospital (Città della Salute e della Scienza di Torino, Molinette Hospital), between January 2014 and December 2015. This cohort was used for an external validation of the model in a different time frame (i.e., temporal validation).
A third cohort, again based on the same criteria for enrollment, was collected using the data referring to a different University-affiliated hospital, the San Giovanni Bosco Hospital, Local Healthcare Unit of Città di Torino, Turin, Italy. This hospital shares a similar population of patients and the period of collection was the same as that for the temporal external validation cohort in the Città della Salute e della Scienza di Torino, Molinette Hospital. This last cohort was also used for external validation (i.e., between January 2014 and December 2015, spatial external validation of the model). Figure 4 briefly summarizes the data collection for these three cohorts.
pital, Local Healthcare Unit of Città di Torino, Turin, Italy. This hospital shares a similar population of patients and the period of collection was the same as that for the temporal external validation cohort in the Città della Salute e della Scienza di Torino, Molinette Hospital. This last cohort was also used for external validation (i.e., between January 2014 and December 2015, spatial external validation of the model). Figure 4 briefly summarizes the data collection for these three cohorts.  A nomogram, a graphical presentation, was proposed as an easy-to-use tool for applying the prognostic model. It is based on the calculation of the linear predictor. A reference line allows the user to score each variable for the patient and identify a total score corresponding to the linear predictor that has to be transformed in the prediction read at the bottom of the nomogram [30,31].
Analyses were performed using STATA 17 (Stata Corp TX, College Station, TX, USA) and R version 3.6.3 (The R Foundation for Statistical Computing, 2020).

Conclusions
This study proposed the development and double validation of a prognostic model for MDR non-HA BSI in hospital admitted adult patients. This model, along with the easyto-perform nomogram, could allow better targeting of empiric antibiotic therapy for BSI before the results of blood cultures become available. However, further prospective studies are needed to confirm the present results and to better define the impact of each covariate.
Author Contributions: G.M., E.P. and E.L. take responsibility for the content of the manuscript, including the data and the analyses; G.M., P.P. (Paolo Peasso) and F.G.D.R. conceived the study; E.P. and E.L. contributed to the study design; E.P. contributed to the statistical analyses; E.P., S.C. and E.L. wrote the first draft of the manuscript and prepared figures and tables; E.P., G.M., F.G.D.R., A.C., A.B., P.P. (Paolo Peasso), P.P. (Paolo Petitti), E.G., M.G., I.C., S.C., R.C., V.G., F.A. and E.L. contributed to data acquisition and interpretation of the results and provided critical input into manuscript drafting and revisions. All authors attest to meeting the four ICMJE.org authorship criteria. All authors have read and agreed to the published version of the manuscript.
Funding: This research received no external funding.

Institutional Review Board Statement:
The study was conducted in accordance with the Declaration of Helsinki. The of Città della Salute e della Scienza di Torino University Hospital Ethics Committee approved the study protocol (n. 0033874).

Informed Consent Statement:
This is a retrospective study so patients' consent to participate was waived according to hospital Ethics Committee.

Data Availability Statement:
The data are available upon reasonable request to the corresponding author.