https://orcid.org/0000-0001-9671-8161
Figures
Abstract
Background
To develop a clinical prediction model to identify children at risk for revisits with serious illness to the emergency department.
Methods and findings
A secondary analysis of a prospective multicentre observational study in five European EDs (the TRIAGE study), including consecutive children aged <16 years who were discharged following their initial ED visit (‘index’ visit), in 2012–2015. Standardised data on patient characteristics, Manchester Triage System urgency classification, vital signs, clinical interventions and procedures were collected. The outcome measure was serious illness defined as hospital admission or PICU admission or death in ED after an unplanned revisit within 7 days of the index visit. Prediction models were developed using multivariable logistic regression using characteristics of the index visit to predict the likelihood of a revisit with a serious illness. The clinical model included day and time of presentation, season, age, gender, presenting problem, triage urgency, and vital signs. An extended model added laboratory investigations, imaging, and intravenous medications. Cross validation between the five sites was performed, and discrimination and calibration were assessed using random effects models. A digital calculator was constructed for clinical implementation. 7,891 children out of 98,561 children had a revisit to the ED (8.0%), of whom 1,026 children (1.0%) returned to the ED with a serious illness. Rates of revisits with serious illness varied between the hospitals (range 0.7–2.2%). The clinical model had a summary Area under the operating curve (AUC) of 0.70 (95% CI 0.65–0.74) and summary calibration slope of 0.83 (95% CI 0.67–0.99). 4,433 children (5%) had a risk of > = 3%, which was useful for ruling in a revisit with serious illness, with positive likelihood ratio 4.41 (95% CI 3.87–5.01) and specificity 0.96 (95% CI 0.95–0.96). 37,546 (39%) had a risk <0.5%, which was useful for ruling out a revisit with serious illness (negative likelihood ratio 0.30 (95% CI 0.25–0.35), sensitivity 0.88 (95% CI 0.86–0.90)). The extended model had an improved summary AUC of 0.71 (95% CI 0.68–0.75) and summary calibration slope of 0.84 (95% CI 0.71–0.97). As study limitations, variables on ethnicity and social deprivation could not be included, and only return visits to the original hospital and not to those of surrounding hospitals were recorded.
Citation: Nijman RG, Borensztajn DH, Zachariasse JM, Hajema C, Freitas P, Greber-Platzer S, et al. (2021) A clinical prediction model to identify children at risk for revisits with serious illness to the emergency department: A prospective multicentre observational study. PLoS ONE 16(7): e0254366. https://doi.org/10.1371/journal.pone.0254366
Editor: José Moreira, Fundacao Oswaldo Cruz, BRAZIL
Received: November 18, 2020; Accepted: June 25, 2021; Published: July 15, 2021
Copyright: © 2021 Nijman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data are available from https://data.hpc.imperial.ac.uk/resolve/?doi=8375&access=.
Funding: No specific funding was received for this study. RN was supported with a Thrasher Research Fund grant (Award number 12830) and was awarded an NIHR academic clinical lectureship (CL-2018-21-007) to do this work. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: aOR, adjusted Odds Ratio; AUC, Area Under the receiving operating characteristic Curve; AVPU, Alert-Verbal-Pain-Unresponsive; CI, confidence interval; ED, emergency department; IQR, interquartile range; Iv, intravenous; LR, likelihood; MTS, Manchester triage system; OR, odds ratio; PICU, paediatric intensive care unit; VIF, variance inflation factor
Introduction
A small, but not negligible proportion of children (2.2% to 7%) will return to the emergency department (ED) within the course of their illness after an initial discharge from the ED [1–6]. Rates of children with a revisit to the ED increased in recent years, partly contributing to the overcrowding of emergency departments [5]. Until now, most interest has been in understanding revisits which do not require emergency care, and avoiding those revisits that could be managed safely in the community or with appropriate safety netting information. Several factors at the time of an initial ED visit, such as young age, nighttime presentation and high acuity level, have been linked to an increased risk of having a revisit to the ED [3, 4, 7–10]. Yet, less is known about children returning to the ED with a serious illness, seen in 13 to 42% of the children with an ED revisit [3, 4, 7, 9, 11].
At present, no prospective study has looked at determinants associated with return visits in children in EDs across Europe, let alone with a focus on return visits with serious illness. Increasing our understanding of determinants associated with return visits to EDs with serious illness will aid the identification of patients that can be discharged safely with a low risk for clinical deterioration, allowing for improved ED discharge flow and alleviating ED crowding. It will also create opportunities for more targeted safety netting and follow up for those patients at a higher risk. Hence, we aimed to develop a clinical prediction model and digital calculator to support clinicians identifying children at high and low risks of a return visit with serious illness.
Methods
We first looked at the number of return visits with a serious illness amongst children who had previously presented to the ED in five hospitals across Europe. We then looked at characteristics of the initial (‘index’) ED visit and the association with a return visit with a serious illness. Finally, we developed and validated a clinical prediction to estimate the risk of an individual patient to return to the ED within seven days of initial ED discharge, following TRIPOD guidelines for the development and validation of clinical prediction models.
Design, participants and setting
The study was embedded in the TrIAGE (TRiage Improvements Across General Emergency departments) project, aiming to improve the early recognition of seriously ill children at the ED [12, 13]. We performed a secondary analysis of data collected as part of a prospective observational multicentre cohort study in five European EDs (Erasmus MC, Rotterdam, the Netherlands (EMC); Maasstad Hospital, Rotterdam, the Netherlands (MHR); St Mary’s Hospital, London, United Kingdom (SMH); Hospital Fernando da Fonseca, Lisbon, Portugal (HFF); Medizinische Universitaet Wien, Vienna, Austria (MUW)), with data collected between 2012 and 2015 (Table 1). Data analyses were planned and executed following completion of the original data collection. Consecutive children aged 0 to 16 years who presented to EDs for unscheduled healthcare were included in the study. Universal health care coverage was available across the participating sites, with similar and free of charge access to urgent and emergency care, and primary care. Primary care with either general practitioners and/or primary care paediatricians was available in all the countries in our study. Patients who were re-directed towards a primary care provider or urgent care centre after presenting to the ED at the time of their index visit were not eligible for this study, as we did not record complete clinical data for these children. Patients who left against medical advice or who left before being seen were only eligible if sufficient baseline data were available for that visit. Children who were admitted following their index visit were subsequently excluded from analysis, as they have a different risk profile for a revisit with serious illness [5, 14]. A waiver for the need of patient informed consent was obtained at all participating hospitals.
Data collection and predictor variables
In all hospitals data on: patient characteristics, triage urgency, presenting problem, vital signs, ED interventions and procedures, treatment, and outcomes were recorded in the electronic health records in a standardised manner as part of routine clinical practice. Predictor variables were chosen based on their known association with return visits, patient urgency and severity scores [15, 16]. Data were extracted from the hospital’s electronic systems and checked for completeness, validity and outliers. Season, time and day of presentation were derived from the patient’s arrival date and time. Seasons were divided in Winter (21/12–20/03), Spring (21/03–20/06), Summer (21/06–20/09), Autumn (21/09–20/12). Date and time of presentation were grouped into: Weekday evenings (Monday–Friday; 18.00–23.00 hrs), Weekday nights (Monday–Friday; 23.00–07.00 hrs), Weekend days (Saturday–Sunday; 08.00–18.00 hrs), Weekend evenings (Saturday–Sunday; 18.00–23.00 hrs), Weekend nights (Saturday–Sunday; 23.00–07.00 hrs), and Weekdays (Monday–Friday; 08.00–18.00 hrs). Triage urgency was determined using the third edition of the Manchester Triage System [17]. This is a validated 5-level urgency classification system that allocates an urgency category using specific discriminators for 52 flowcharts detailing varying presenting problems [18]. These flowcharts were re-classified into 11 main groups of presenting symptoms (S1 Table). Low urgency levels standard (level 4) and non-urgent (level 5), as well as emergent (level 1) and very urgent (level 2) were combined in the multivariable regression analyses due to low number of cases in the highest and lowest category. Vital signs were measured at the time of triage by the triage nurse, following advanced paediatric life support standards [19]. Heart rate (beats per minute) and respiratory rate (breaths per minute) were categorised using the age-specific Advanced Paediatric Life Support threshold values. Oxygen saturations were dichotomised at <94%; body temperature at > = 38.0 degrees Celsius. For level of consciousness, Verbal, Pain, and Unconsciousness on the Alert-Verbal-Pain-Unresponsive (AVPU) scale were combined into ‘reduced level of consciousness’. Interventions and procedures in the ED included: any laboratory tests performed (ie. blood, urine, cerebral spinal fluid, other), any imaging done (ie. Xray, ultrasound, CT, MRI), and any intravenous medications and/or fluids administered. Immediate lifesaving interventions included airway and breathing support, haemodynamic support, emergency procedures, and emergency medications (S1 Table). Blood pressure, capillary refill and lifesaving interventions were not used for prediction modelling owing to low number of values measured in children across all institutions (blood pressure available for 8%; capillary refill for 47%); and/or <10 cases with abnormal values for children with serious illness at return visit (Table 2). In three hospitals (EMC, SMH, MUW), co-morbidity was coded into complex and non-complex by trained medical students blinded for the outcome of this study according to the definitions by Simon et al. [20].
Outcome measures
Serious illness was defined as the need for hospital admission or paediatric intensive care unit (PICU) admission or death in ED after an unplanned return visit within 7 days of the index visit. Characteristics of children admitted to PICU at the time of their revisit were described in detail separately (S2 Table).
Statistical analysis
Firstly, we performed uni- and multi-covariate logistic regression to look at the association between the characteristics of the index visit and a return visit with a serious illness, reporting crude unadjusted and adjusted odds ratios (OR) with 95% confidence intervals. For the three hospitals (Erasmus cohort, London cohort, Vienna cohort) with available data on co-morbidity we performed an additional analysis to determine the role of co-morbidity on return visits with serious illness in these institutions. Our cohort provided sufficient power for both the derivation and cross-validation studies, estimating the required sample using the recommended approach by Riley et al. (S1 Appendix) [21–23].
Secondly, we developed two clinical prediction models using logistic regression. A first model included clinical characteristics of the index visit (ie: ‘clinical model’), and then the use of laboratory investigations, intravenous medications or fluids, and imaging were added to the model (ie.: ‘extended model’) as a second step. We included the five hospitals as a categorical variable to account for case mix differences between the cohorts. All variables were pre-selected, and variables were included independent from their statistical contribution to the models based on previous literature. No variables were excluded from the final prediction models based on statistical contribution or when adding the variables of the extended model [15]. The overall performance of the extended model was compared with the clinical model using the likelihood ratio test. Missing data for vital signs (Table 2) were imputed using a multiple imputation model including a hospital variable, age, available vital signs, triage urgency, presenting problem, and discharge destination. This resulted in twenty-five iterative databases, including data from all five cohorts, on which we performed statistical analysis after which results were pooled [24]. Imputation was performed using the multivariate imputation by chained equations.
We initially derived the prediction model in four of the five cohorts, and then cross-validated the model in the fifth cohort that was left out. We repeated this “leave-one-out”approach as a method for cross-validating the prediction model four times with each of the five cohorts serving as an independent validation cohort once [25]. Each time, we adjusted the intercept by estimating the linear predictors for the validation cohort without using the hospital variable and then recalculating an intercept, thus acknowledging differences between the hospitals. External validity of the model was determined by estimating the discrimination by means of the Area Under the operating characteristic Curve (AUC), and model calibration by means of calibration plots, the calibration coefficient, or calibration slope, and change in intercept of the model. The AUC is used to estimate the ability to discriminate between patients with and without a return visit with serious illness, with values ranging between 0.5 (indiscriminate test) and 1 (optimal test). Calibration refers to the level of agreement between predicted risks and observed outcome; the ideal calibration coefficient, or slope, being one, indicating perfect agreement between observed and predicted risks. The calibration intercept reflects the extent that predictions are systematically too low or too high (‘calibration-in-the-large’), e.g. due to differences in the underlying incidence between cohorts, with an optimal value of zero [26]. The five estimates of AUCs and calibration coefficients of each of the five validation cohorts were then pooled by means of random effects models to provide summary estimates. Finally, we developed a clinical prediction model combining all available data from the five cohorts and with a refitted intercept to allow for hospital differences. An apparent optimism corrected AUC was calculated by means of bootstrapping (500 iterations). We performed bias corrected calibration with 200 bootstrapping iterations to demonstrate goodness of fit for the final overall prediction models. Variance inflation factors were calculated to assess collinearity between predictor variables. Diagnostic performance measures, including sensitivity, specificity and likelihood ratios (LRs) were calculated for different risk thresholds of both the clinical and the extended model. We used R statistical software version 4.0.0 for all our analyses, including use of the rms, metaphor, MICE, pmsampsize, pROC and epiR packages.
Ethics statement
The study was approved by the participating institutions’ medical ethical committees: Medical Ethics Committee Erasmus MC (MEC-2013-567), Maasstad Ziekenhuis Board of Directors (Protocol L2013-103), Imperial College London Joint Research Compliance Office (Reference number: 14SM2164; Ethics reference number 14/WA/1051), Comissão de Ética para a Saúde do Hospital Prof. Dr. Fernando Fonseca EPE (Reunião de 06 de Dezembro de 2017), Ethik Kommission Medizinische der Medizinischen Unversität Wien (EK Nr: 1405/2014). All waived the requirement for informed consent.
Results
Rates of revisits
109,482 children with an ED visit were included, of whom 98,561 children (90%) were discharged from ED (Fig 1). A total of 1,026 children (1.0%) returned to the ED with serious illness out of a total of 7,891 children who represented to the ED. Rates of revisits with serious illness varied between the hospitals (range 0.7–2.2%); unadjusted odds ratios for variables of interest and their risk of a return visit with serious illness are shown in Table 2. Twenty-three children (0.02%) were admitted to PICU and one child died at revisit (characteristics of PICU patients presented in S2 Table).
Flowchart with exclusion criteria and the included population.
Characteristics of return visits with serious illness
Several clinical characteristics of the index visit were associated with a revisit with serious illness. These included young age, female gender, time of presentation during the weekend daytime and weekday night-time, and higher triage urgency (Table 3). Presenting problems such as shortness of breath, gastro-intestinal problems, neurological problems and soft tissue infections were also linked to an increased risk of a return visit with serious illness, as were any laboratory tests performed, any imaging done, and any intravenous medications and/or fluids administered. Presentations for trauma and wounds made a revisit with serious illness less likely.
Comorbidity was significantly associated with a return visit with serious illness in the three hospitals for which we had these data, both for children with non-complex (aOR 2.36, 95% CI 1.82–3.05)) and children with complex co-morbidities (aOR 3.39, 95% CI 2.67–4.29) (S3 Table).
Cross validation of the clinical and extended prediction models
Table 4 presents the overall discrimination and calibration of the “leave-one-out”cross-validation of the clinical and the extended model. Intercept coefficients ranged from -0.91 to 0.10 for the clinical model showing some inter-hospital variation for calibration-in-the-large. Discrimination of the clinical and extended models were similar with a summary AUC of 0.70 (95% CI 0.65–0.74) for the clinical model and a summary AUC of 0.71 (95% CI 0.68–0.75) for the extended model (S1 and S2 Figs). Calibration was satisfactory for both models with a summary calibration slope of 0.83 (95% CI 0.67–0.99) for the clinical model and a summary calibration slope of 0.84 (95% CI 0.71–0.97) for the extended model (S3 and S4 Figs).
Derivation of the final clinical and extended prediction models
Next, we combined all cohorts and derived a clinical model predicting the risk of a revisit with serious illness based on clinical characteristics (Table 3; model coefficients are presented in S4 Table). 4,433 children (5%) had a risk of > = 3%, which was useful for ruling in a revisit with serious illness, with positive likelihood ratio 4.41 (95% CI 3.87–5.011) and specificity 0.96 (95% CI 0.95–0.96) (Table 5). Increasing this high-risk threshold to 4% or above resulted in improved ‘rule in’ value (positive likelihood ratio 6.23 (95% CI 5.11–7.61) and specificity 0.99 (95% CI 0.98–0.99)), but at the cost of being applicable to fewer children (n = 1,523, 2%). 37,546 (39%) had a risk <0.5%, which was useful for ruling out a revisit with serious illness (negative likelihood ratio 0.30 (95% CI 0.25–0.35), sensitivity 0.88 (95% CI 0.86–0.90)). An extended model had similar overall diagnostic performance for the low risk threshold of 0.5% and high-risk threshold of 4% (Table 5). The extended model improved risk classification for a modest proportion of children (Fig 2). The clinical model based on all cohorts combined had an optimism corrected apparent AUC of 0.73 (95% CI 0.70–0.76); the extended model also including intravenous medications and/or fluids, imaging, and laboratory investigations had improved optimism corrected AUC of 0.75 (95% CI 0.72–0.78; Likelihood Ratio test for overall model performance improvement, p <0.001). Bias corrected calibration showed goodness of fit of both the clinical and extended models (S5 Fig). Variance inflation factors were between 1 and 6 for all predictor variables indicating limited collinearity between predictor variables (S6 Table).
Barplots showing the distributions of the predicted risks for the clinical model (top) and extended model (bottom).
Discussion
Principal findings
Young age under two years, high urgency, and type of presenting problem were highly associated with a return visit to the emergency department with a serious illness. Rates of revisits with serious illness varied between the five hospitals, ranging from 0.7 to 2.2%, and were in line with the lower end of previously reported rates [3, 4, 7, 9, 11]. A clinical prediction model, including the variables season, time and day of presentation, age, gender, presenting problem, triage urgency, and vital signs, was able to select a high number of children with a low risk of return visit with serious illness, as well as identify those with the highest risks. Initial derivation and cross-validation of the model showed the robustness of the model in all settings. We developed a digital calculator to enable clinical implementation. For illustration, in our case example shown in Fig 3, a 3 year boy with shortness of breath had a 3.1% predicted risk of having a return visit with a serious illness, which was between the >70th—≦80th centiles (S5 Table). The predicted probabilities for a revisit with serious illness remained relatively low, even for children with an increased risk, mostly as a result of the overall low prevalence (1.0%). Performing laboratory tests and imaging, as well as administration of intravenous medications were all associated with an increased risk of a revisit with serious illness and improved overall risk prediction, even as the diagnostic performance for low and high-risk threshold were similar for the clinical and extended model.
A clinical scenario with calculation of the predicted risks for a return visit with serious illness after discharge from the emergency department, using the digital calculator at http://www.rimon.nl/rvc/.
Comparison to previous literature
Reasons for children to represent with a serious illness after initial ED visit are diverse. One might be concerned about medical errors in the decision making process during the first visit [27, 28]. However, in one study, return visits resulting in hospital admission were rarely the result of poor quality of ED care at first visit [29]. Another study also suggested that only a minority of return visits with serious illness were the result of a failure to make an appropriate diagnosis, a failure to instigate appropriate treatment, or a failure to investigate fully, and that the large majority of these children with a revisit resembled disease progression [30]. For example, in meningococcal disease is it well known that children might present with non-specific signs and symptoms at first, with the severity of the meningococcal septicaemia only becoming manifest at a later stage of the disease [31]. There might also be parental factors involved, such as a refusal to be admitted in first instance, non-adherence to prescribed therapy at home, or a failure to attend a follow up appointment [30]. This all contributes to the importance of empowering parents to provide informed monitoring at home, and providing them with clear instructions on when to return to the ED [32].
More advanced modelling techniques have been used to provide clinicians with clinical tools to predict return visits to the ED [1, 33, 34]. However, to our knowledge, none focused on the risk of return visits with serious illness, and none presented a tool ready for clinical implementation. From the outset, we felt it was more important to understand the group of children at risk for clinical deterioration, rather than preventing avoidable revisits, even though both have significant implications for appropriate use of scarce resources.
One of the difficulties was capturing the vast spectrum of problems clinicians are confronted with in the ED in a single tool [16]. For certain conditions, such as children at risk of dehydration or children at risk for severe infection, specific clinical features were identified predictive of clinical deterioration [35, 36]. Although those predictors might be important, they only apply to a select proportion of the children visiting the ED, and this will need to be positioned against the convenience of a ‘one-fits-all’ tool. Our model assigned higher risks of serious illness at revisit to children presenting with specific problems, such as shortness of breath, gastro-intestinal problems, and neurological problems.
Tachycardia has been shown to be associated with an increased risk of a revisit, as well as with revisits with serious illness, yet we did not find evidence of this [11, 37]. We did not include heart rate at the time of discharge; however, Daymont et al. looked at this and only found limited additional value for different ways of including heart rate as a predictor of return visits with serious illness [37]. The role of other vital signs as predictors of serious illness needs further exploring.
Some variables, such as social economic status, ethnicity, language fluency, and health literacy were previously shown to be associated with revisits as well as revisits with serious illness, yet were not included in the model [2–4, 7, 10]. Studies had inconsistent and sometimes contradicting risk profiles for these variables [36]. Crowding of EDs has previously been associated with both an increased as well as a reduced risk [2–4]. Moreover, Akenroye et al. found that ED crowding was an increased risk for revisits, but not for a revisit with admission to hospital [3], highlighting that risk factors for return visits differ from risk factors for revisits with serious illness. Notably, ethnicity and variables derived from the use of postal codes, such as deprivation scores, cannot routinely be used in most European countries without explicit informed consent and hence were not included in our models.
Implications for clinical practice and future research
The limited variability in calibration slopes testifies that this model can be used in a diverse range of clinical settings [25]. The differing underlying incidence of serious revisits, as also reflected in the intercept values of the cross-validation studies (‘calibration-in-the-large’, Table 4), indicate that the local incidence should be taken into account when interpreting the risk predictions. If the local incidence is known, one could adjust our model for local use by adjusting the intercept of the model. Knowing which children are at increased risk for clinical deterioration following discharge from ED can contribute to more tailored, patient-specific follow-up [15]. How this follow-up should look will be highly setting dependent, with some institutions having the capacity to organise planned face to face review appointments in ED or by an ambulatory care team, and in other settings this might be done via telephonic consultation or with the involvement of primary care. Many factors contribute to the development of serious illness, including natural disease progression and treatment compliance. By implementing tailored safety netting and follow-up, this will help early detection of children with progressing disease, preventing delayed presentations after an initial health care visit, as well as ensuring treatments are continued at home, and are given in a timely and appropriate manner. Examples of the latter could include antibiotic tolerance in young children, appropriate inhaler technique of bronchodilators, and reinforcement of hydration strategies. More senior clinicians will likely have an experienced feel for those patients at highest risk for deterioration. Our models enable this for more junior physicians, who see the large majority of unwell children in acute care facilities, and for settings with no dedicated paediatric emergency care team. Also, our prediction model can complement decision making in acute paediatric conditions where a watchful waiting approach is justified and a return visit can be interpreted as a marker of good clinical care, such as in children with non-specific abdominal pain deemed to be at low risk for appendicitis at first assessment. Importantly, any intervention with individualised follow-up or targeted safety netting will need to be studied carefully, in combination with cost-effectiveness analyses, as it might inadvertently have unwanted effects. One study showed an increase in the rate of return visits following the introduction of telephone calls after ED discharge [38]. In future, studies should also look into the reasons for revisiting the ED, such as parental factors, potential of medical errors, and natural disease course, as they will be useful in fine-tuning the specific interventions for safety netting and follow-up. In addition, the potential for machine learning techniques to improve our prediction model should be explored.
Strengths
This is the first study including a large cohort of children using a prospective multicentre design with data from five different European EDs. The design allowed for standardised, near complete data on predictors known to be related with urgency systems and severity scores. We opted for including clinical variables with minimal interuser variability that were all readily available in routine clinical practice. For example, the reliability and validity of the MTS triage urgency classification was shown previously [39]. We also converted our understanding of determinants of revisits with serious illness into a clinical tool that can be used in clinical practice. We included the clinical variables independent of their statistical contribution to the model to ensure validity of the model and enhance clinical uptake. Although one might argue that the discriminative ability of the model is only moderate, the tool is able to identify a large number of children who can be discharged safely, and also identifies a select and well-defined group of patients that can be targeted for more tailored safety netting advice and follow-up strategies.
Limitations
One of the limitations was the over-representation of patients from the larger of the five hospitals, and we adjusted for this by including the hospitals as a categorical variable whilst developing and validating the prediction models. Generally, findings across the sites were similar (data not presented), as was also confirmed by the cross-validation analyses. Furthermore, one can argue the accuracy of the outcome definition for serious illness; however, defining serious illness by the need for hospitalisation at revisit was least subjective to a doctor’s impression, and with limited variability between hospitals [40]. We performed a secondary analysis with a broader outcome definition of serious illness, including those who underwent lifesaving interventions or had triage urgency ‘emergent’ at revisit. This yielded only 16 additional cases, which did not affect our overall findings, supporting the validity of our outcome definition. Despite our large sample size, it did not allow for a separate model for children admitted to PICU at revisit (S2 Table). In addition, we lack data to show the impact of revisits to other EDs than the one originally visited, and future studies will need to provide detailed insight into where children with a serious illness re-present during an episode of acute illness. As our study hospitals have an important regional referral function with limited admission capacity, hospital admission was warranted only to provide therapeutic of supportive treatment that would not have been able in the community or by other health care providers in the large majority of cases. The role of other types of admission, such as for social reasons, will need to be studied in future. Also, only a relatively small number of variables were included resulting in a low explained variance of our models, meaning future studies should explore the incremental predictive value of additional variables, such as social economic status and language. We did not consider language fluency, as reliably determining parental or carer’s language proficiency was not feasible. Another important candidate predictor will be co-morbidity, which was an important predictor of return visits with serious illness in three out of five cohorts with data on co-morbidity available. Furthermore, the validity and applicability of our prediction models still needs to be proven outside Europe and in different health systems., specifically in those without primary care or universal health care coverage. Lastly, the data used for this study are from the period prior to the COVID-19 pandemic. The COVID-19 pandemic has impacted local healthcare delivery systems globally, and future validation studies will need to study long term impact of the current COVID-19 pandemic.
Conclusion
We developed a prediction model and a digital clinical calculator that can aid physicians identifying those children at highest and lowest risks for developing a serious illness after initial discharge from the ED, allowing for more targeted safety netting advice and follow-up.
Supporting information
S1 Table. Definitions of presenting symptoms and lifesaving interventions.
https://doi.org/10.1371/journal.pone.0254366.s002
(PDF)
S2 Table. Characteristics of the index visit of children admitted to PICU at the time of revisit.
https://doi.org/10.1371/journal.pone.0254366.s003
(PDF)
S3 Table. Clinical prediction models for the three hospitals with data on co-morbidity available.
https://doi.org/10.1371/journal.pone.0254366.s004
(PDF)
S4 Table. Regression coefficients of the final clinical prediction models.
https://doi.org/10.1371/journal.pone.0254366.s005
(PDF)
S1 Fig. Area under the receiver operating curve (AUC) for validation cohorts.
https://doi.org/10.1371/journal.pone.0254366.s008
(PDF)
S2 Fig. Forest plots for cross validation area under the receiver operating curves (AUC).
https://doi.org/10.1371/journal.pone.0254366.s009
(PDF)
S3 Fig. Forest plots for cross validation calibration slopes.
https://doi.org/10.1371/journal.pone.0254366.s010
(PDF)
S4 Fig. Calibration plots for cross-validation cohorts.
https://doi.org/10.1371/journal.pone.0254366.s011
(PDF)
Acknowledgments
We would like to thank Rikke Jorgensen for her dedication as a research nurse to the study at the St Mary’s hospital–Imperial College NHS Healthcare Trust. Similarly, we would like to acknowledge Pinky Rose Espina, paediatric resident at Medizinische Universitaet Wien, Vienna, for her contributions.
References
- 1. Bergese I, Frigerio S, Clari M, Castagno E, De Clemente A, Ponticelli E, et al. An Innovative Model to Predict Pediatric Emergency Department Return Visits. Pediatr Emerg Care 2019 Mar;35(3):231–6. pmid:27741066
- 2. Michelson KA, Lyons TW, Bachur RG, Monuteaux MC, Finkelstein JA. Timing and location of emergency department revisits. Pediatrics 2018 May;141(5). pmid:29650806
- 3. Akenroye AT, Thurm CW, Neuman MI, Alpern ER, Srivastava G, Spencer SP, et al. Prevalence and predictors of return visits to pediatric emergency departments. J Hosp Med 2014 Dec;9(12):779–87. pmid:25338705
- 4. Saunders NR, To T, Parkin PC, Guttmann A. Emergency Department Revisits by Urban Immigrant Children in Canada: A Population-Based Cohort Study. J Pediatr 2016 Mar;170:218–26. pmid:26711849
- 5. Cho CS, Shapiro DJ, Cabana MD, Maselli JH, Hersh AL. A national depiction of children with return visits to the emergency department within 72 hours, 2001–2007. Pediatr Emerg Care 2012 Jul;28(7):606–10. pmid:22743754
- 6. Alessandrini EA, Lavelle JM, Grenfell SM, Jacobstein CR, Shaw KN. Return Visits to a Pediatric Emergency Department. Pediatr Emerg Care [Internet] 2004 Mar;20(3):166–71. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15094574. pmid:15094574
- 7. Goldman RD, Kapoor A, Mehta S. Children admitted to the hospital after returning to the emergency department within 72 hours. Pediatr Emerg Care 2011 Sep;27(9):808–11. pmid:21878830
- 8. Chaudhari PP, Monuteaux MC, Bachur RG. Management of Urinary Tract Infections in Young Children: Balancing Admission With the Risk of Emergency Department Revisits. Acad Pediatr 2019 Mar;19(2):203–8. pmid:29864523
- 9. Goh GL, Huang P, Kong MCP, Chew SP, Ganapathy S. Unplanned reattendances at the paediatric emergency department within 72 hours: A one-year experience in KKH. Singapore Med J 2016 Jun;57(6):307–13. pmid:27353384
- 10. Gallagher RA, Porter S, Monuteaux MC, Stack AM. Unscheduled return visits to the emergency department: The impact of language. Pediatr Emerg Care 2013 May;29(5):579–83. pmid:23603647
- 11. Wilson PM, Florin TA, Huang G, Fenchel M, Mittiga MR. Is Tachycardia at Discharge From the Pediatric Emergency Department a Cause for Concern? A Nonconcurrent Cohort Study. Ann Emerg Med [Internet] 2017;70(3):268–276.e2. Available from: http://dx.doi.org/10.1016/j.annemergmed.2016.12.010 pmid:28238501
- 12. Zachariasse JM, Nieboer D, Maconochie IK, Smit FJ, Alves CF, Greber-Platzer S, et al. Development and validation of a Paediatric Early Warning Score for use in the emergency department: a multicentre study. Lancet Child Adolesc Heal [Internet] 2020 Aug 1;4(8):583–91. Available from: pmid:32710839
- 13. Zachariasse JM, Maconochie IK, Nijman RG, Greber-Platzer S, Smit FJ, Nieboer D, et al. Improving the prioritization of children at the emergency department: Updating the Manchester Triage System using vital signs. PLoS One [Internet] 2021 Feb 9;16(2):e0246324. Available from: pmid:33561116
- 14. Sabbatini AK, Kocher KE, Basu A, Hsia RY. In-hospital outcomes and costs among patients hospitalized during a return visit to the emergency department. JAMA—J Am Med Assoc 2016 Feb;315(7):663–71. pmid:26881369
- 15. De Vos-Kerkhof E, Geurts DHF, Wiggers M, Moll HA, Oostenbrink R. Tools for “safety netting” in common paediatric illnesses: A systematic review in emergency care. Arch Dis Child 2016 Feb;101(2):131–9. pmid:26163122
- 16. Seiger N, Maconochie I, Oostenbrink R, Moll HA. Validity of different pediatric early warning scores in the emergency department. Pediatrics [Internet] 2013 Oct 1;132(4):e841–50. Available from: http://pediatrics.aappublications.org/cgi/doi/10.1542/peds.2012-3594. pmid:24019413
- 17.
Kevin Mackway-Jones JMand JW. Presentational flow chart index. Second edi. Emergency Triage. Blackwell Publishing Ltd.; 2013. 62–63 p.
- 18. Van Veen M, Steyerberg EW, Ruige M, Van Meurs AHJ, Roukema J, Van Der Lei J, et al. Manchester triage system in paediatric emergency care: Prospective observational study. Bmj 2008 Sep;337(7673):792–5. pmid:18809587
- 19.
Phillips BM, Mackway-Jones K, Molyneux E, Wieteska S. Advanced paediatric life support. 2nd ed. Vol. 7, Current Paediatrics. Maarsen: Reed business; 1997. 48–52 p.
- 20. Simon TD, Cawthon ML, Stanford S, Popalisky J, Lyons D, Woodcox P, et al. Pediatric medical complexity algorithm: A new method to stratify children by medical complexity. Pediatrics 2014 Jun;133(6):e1647–54. pmid:24819580
- 21. Riley RD, Snell KI, Ensor J, Burke DL, Harrell FEJ, Moons KG, et al. Minimum sample size for developing a multivariable prediction model: PART II—binary and time-to-event outcomes. Stat Med 2019 Mar;38(7):1276–96. pmid:30357870
- 22. Riley RD, Ensor J, Snell KIE, Harrell FE, Martin GP, Reitsma JB, et al. Calculating the sample size required for developing a clinical prediction model. BMJ [Internet] 2020;368. Available from: https://www.bmj.com/content/368/bmj.m441 pmid:32188600
- 23. Archer L, Snell KIE, Ensor J, Hudda MT, Collins GS, Riley RD. Minimum sample size for external validation of a clinical prediction model with a continuous outcome. Stat Med [Internet] 2021;40(1):133–46. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.8766 pmid:33150684
- 24. Donders ART, van der Heijden GJMG, Stijnen T, Moons KGM. Review: A gentle introduction to imputation of missing values. J Clin Epidemiol 2006 Oct;59(10):1087–91. pmid:16980149
- 25. Steyerberg EW, Nieboer D, Debray TPA, van Houwelingen HC. Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration. Stat Med 2019 Sep;38(22):4290–309. pmid:31373722
- 26. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology 2010 Jan;21(1):128–38. pmid:20010215
- 27. Najaf-Zadeh A, Dubos F, Pruvost I, Bons-Letouzey C, Amalberti R, Martinot A. Epidemiology and aetiology of paediatric malpractice claims in France. Arch Dis Child 2011 Feb;96(2):127–30. pmid:20660524
- 28. Sen G, Keene J, Raine J. An analysis of successful litigation claims in childhood fatalities in England. Eur J Pediatr [Internet] 2012;171(11):1657–60. Available from: https://doi.org/10.1007/s00431-012-1793-6 pmid:23011745
- 29. Cheng J, Shroff A, Khan N, Jain S. Emergency Department Return Visits Resulting in Admission: Do They Reflect Quality of Care? Am J Med Qual 2016 Nov;31(6):541–51. pmid:26160967
- 30. DePiero AD, Ochsenschlager DW, Chamberlain JM. Analysis of pediatric hospitalizations after emergency department release as a quality improvement tool. Ann Emerg Med 2002 Feb;39(2):159–63. pmid:11823770
- 31. Thompson MJ, Ninis N, Perera R, Mayon-White R, Phillips C, Bailey L, et al. Clinical recognition of meningococcal disease in children and adolescents. Lancet 2006;367(9508):397–403. pmid:16458763
- 32. Almond SC, Summerton N. Diagnosis in general practice: Test of time. BMJ 2009 Jun;339(7711):43–5. pmid:19528115
- 33. Hu YH, Tai CT, Chen SCC, Lee HW, Sung SF. Predicting return visits to the emergency department for pediatric patients: Applying supervised learning techniques to the Taiwan National Health Insurance Research Database. Comput Methods Programs Biomed 2017 Jun;144:105–12. pmid:28494994
- 34. Lee EK, Yuan F, Hirsh DA, Mallory MD, Simon HK. A clinical decision tool for predicting patient care characteristics: patients returning within 72 hours in the emergency department. AMIA Annu Symp Proc 2012;2012:495–504. pmid:23304321
- 35. Freedman SB, Thull-Freedman JD, Rumantir M, Atenafu EG, Stephens D. Emergency department revisits in children with gastroenteritis. J Pediatr Gastroenterol Nutr 2013 Nov;57(5):612–8. pmid:23820403
- 36. De Vos-Kerkhof E, Geurts DHF, Steyerberg EW, Lakhanpaul M, Moll HA, Oostenbrink R. Characteristics of revisits of children at risk for serious infections in pediatric emergency care. Eur J Pediatr 2018 Apr;177(4):617–24. pmid:29397418
- 37. Daymont C, Balamuth F, Scott HF, Bonafide CP, Brady PW, Depinet H, et al. Elevated Heart Rate and Risk of Revisit With Admission in Pediatric Emergency Patients. Pediatr Emerg Care 2018 Jul;Publish Ah.
- 38. Goldman RD, Wei JJ, Cheyne J, Jamieson B, Friedman BC, Lin GX, et al. Impact of follow-up calls from the pediatric emergency department on return visits within 72 hours a randomized controlled trial. Pediatr Emerg Care 2014 Sep;30(9):613–6. pmid:25162691
- 39. Van Veen M, Moll HA. Reliability and validity of triage systems in paediatric emergency care. Scand J Trauma Resusc Emerg Med [Internet] 2009 Aug;17(1):38. Available from: https://doi.org/10.1186/1757-7241-17-38 pmid:19712467
- 40. Van Ierland Y, Seiger N, Van Veen M, Moll HA, Oostenbrink R. Alarming signs in the Manchester Triage System: A tool to identify febrile children at risk of hospitalization. J Pediatr [Internet] 2013;162(4):862–866.e3. Available from: http://www.sciencedirect.com/science/article/pii/S0022347612011365 pmid:23149176