Improving the quality of hospital care for children by supportive supervision: a cluster randomized trial, Kyrgyzstan

Abstract Objective To determine whether periodic supportive supervision after a training course improved the quality of paediatric hospital care in Kyrgyzstan, where inappropriate care was common but in-hospital postnatal mortality was low. Methods In a cluster, randomized, parallel-group trial, 10 public hospitals were allocated to a 4-day World Health Organization (WHO) course on hospital care for children followed by periodic supportive supervision by paediatricians for 1 year, while 10 hospitals had no intervention. We assessed prospectively 10 key indicators of inappropriate paediatric case management, as indicated by WHO guidelines. The primary indicator was the combination of the three indicators: unnecessary hospitalization, increased iatrogenic risk and unnecessary painful procedures. An independent team evaluated the overall quality of care. Findings We prospectively reviewed the medical records of 4626 hospitalized children aged 2 to 60 months. In the intervention hospitals, the mean proportion of the primary indicator decreased from 46.9% (95% confidence interval, CI: 24.2 to 68.9) at baseline to 6.8% (95% CI: 1.1 to 12.1) at 1 year, but was unchanged in the control group (45.5%, 95% CI: 25.2 to 67.9, to 64.7%, 95% CI: 43.3 to 86.1). At 1 year, the risk ratio for the primary indicator in the intervention versus the control group was 0.09 (95% CI: 0.06 to 0.13). The proportions of the other nine indicators also decreased in the intervention group (P < 0.0001 for all). Overall quality of care improved significantly in intervention hospitals. Conclusion Periodic supportive supervision for 1 year after a training course improved both adherence to WHO guidelines on hospital care for children and the overall quality of paediatric care.


Introduction
Hospital care of an adequate quality is essential for health and well-being and is a basic component of human rights. 1,2 Moreover, reducing inequalities in the quality of care is a primary objective of Health 2020, 1 a strategic policy document issued by the 53 Member States in the World Health Organization's (WHO) European Region. 1 Although data on the quality of hospital care for children in countries with constrained resources are starting to accumulate, [3][4][5][6][7][8][9][10][11][12][13][14][15][16] very little is known about which interventions are most effective for enhancing quality. The approach most commonly advocated by WHO for improving case management for common children's diseases has been the dissemination of evidence-based guidelines, usually combined with staff training. [4][5][6][7]17 Many countries implementing the Integrated Management of Childhood Illness strategy developed by WHO and the United Nations Children's Fund have adopted this approach. However, recent evidence indicates that training alone does not ensure better case management and, even after training, it may be difficult to incorporate new knowledge into practice. [18][19][20] Supportive supervision for staff has been proposed as an intervention for increasing adherence to clinical guidelines. [20][21][22] However, little high-quality evidence, for example from randomized controlled trials, is available on the efficacy of supportive supervision in general or, more specifically, on its use for improving the quality of hospital care for common childhood conditions in low-and middle-income countries. 17 Previous assessments of the quality of paediatric care in the Commonwealth of Independent States showed that in-hospital postnatal case fatality rates were low but inappropriate case management was common and characterized by unnecessary hospitalization, overdiagnosis and overtreatment, possibly associated with wasted resources and a risk to children's health. [3][4][5][6] The main underlying reasons were a lack of up-to-date, evidence-based guidelines and the persistence of outdated clinical practices. [3][4][5][6] In Kyrgyzstan, a lower-middle-income country in the WHO European Region, the mortality rate in children younger than 5 years has decreased steadily from 63 per 1000 live births in 1994 to 24 per 1000 in 2013. 23 In 2012, the Kyrgyz Ministry of Health and WHO agreed to implement a project aimed at improving the quality of hospital paediatric care. This project included an impact evaluation in the form of a cluster randomized controlled trial. This paper presents the findings of this cluster randomized controlled trial, which was designed to determine whether periodic supportive supervision, provided after a standard Objective To determine whether periodic supportive supervision after a training course improved the quality of paediatric hospital care in Kyrgyzstan, where inappropriate care was common but in-hospital postnatal mortality was low. Methods In a cluster, randomized, parallel-group trial, 10 public hospitals were allocated to a 4-day World Health Organization (WHO) course on hospital care for children followed by periodic supportive supervision by paediatricians for 1 year, while 10 hospitals had no intervention. We assessed prospectively 10 key indicators of inappropriate paediatric case management, as indicated by WHO guidelines. The primary indicator was the combination of the three indicators: unnecessary hospitalization, increased iatrogenic risk and unnecessary painful procedures. An independent team evaluated the overall quality of care. Findings We prospectively reviewed the medical records of 4626 hospitalized children aged 2 to 60 months. In the intervention hospitals, the mean proportion of the primary indicator decreased from 46.9% (95% confidence interval, CI: 24.2 to 68.9) at baseline to 6.8% (95% CI: 1.1 to 12.1) at 1 year, but was unchanged in the control group (45.5%, 95% CI: 25.2 to 67.9, to 64.7%, 95% CI: 43.3 to 86.1). At 1 year, the risk ratio for the primary indicator in the intervention versus the control group was 0.09 (95% CI: 0.06 to 0.13). The proportions of the other nine indicators also decreased in the intervention group (P < 0.0001 for all). Overall quality of care improved significantly in intervention hospitals. Conclusion Periodic supportive supervision for 1 year after a training course improved both adherence to WHO guidelines on hospital care for children and the overall quality of paediatric care.

Methods
We conducted a cluster, randomized, parallel-group trial in the Chui, Issyk-Kul and Talas Regions of northern Kyrgyzstan and involved 20 district and regional public hospitals that treated children (Fig. 1). Hospitals were taken as the unit of randomization to avoid contamination between practitioners at the same location. After a baseline assessment using WHO's Hospital care for children: quality assessment and improvement tool, 24 which compares case management with guidelines in WHO's Pocket book of hospital care for children, 25 10 hospitals were randomized to the intervention while 10 continued with usual care (Fig. 2). For randomization, hospitals were first stratified by geographical distribution (i.e. west versus east) and by type (i.e. regional versus district hospitals), then randomized by extraction of opaque sealed envelopes prepared by WHO.   25 The 10 case management indicators were also obtained from Hospital care for children: quality assessment and improvement tool. 24 The assessment at time point T2 took place shortly after the training course and at almost the same time as the first supportive supervision visit.

Research
Improving paediatric care in Kyrgyzstan Marzia Lazzerini et al.
Doctors in charge of children from the intervention hospitals attended a 4-day, WHO training course on WHO's guidelines on hospital care for children. 25 In collaboration with the health ministry, the WHO country office coordinated the course, which was held in the capital Bishkek, supervised by the WHO Regional Office for Europe. Subsequently, a team of eight senior national paediatricians, who had undergone training on both WHO's guidelines and supportive supervision methods, provided supervision for doctors, nurses and managers every 2 months for 1 year. The WHO country office regularly checked that supportive supervision was provided on time. During each hospital visit, two paediatricians provided supportive supervision over 1 or 2 days based on a peer-to peer, plan-do-study-act model, 26 which involved: (i) identifying and agreeing on the actions needed to improve the quality of care; (ii) implementing those actions; (iii) monitoring progress; and (iv) discussing any additional actions needed. Document templates were used for these activities and training was reinforced at each subsequent visit.

Outcomes
The primary aim of the study was to determine whether or not case management was inappropriate when compared with guidelines in WHO's Pocket book of hospital care for children; 25 this was evaluated using 10 key indicators in WHO's Hospital care for children: quality assessment and improvement tool (Table 1). 24 Our primary indicator was the combination of the three indicators: unnecessary hospitalization, an increased risk of iatrogenic effects and unnecessary painful procedures. Our primary study outcome was the change in the mean proportion of hospitalized children, who had medical records documenting the primary indicator. For both intervention and control groups, 10 trained data collectors obtained data prospectively from paper-based medical records on the 10 indicators.  Fig. 2). At each time point, data collectors examined in each hospital the medical records of a random sample of 35 to 40 children aged 2 to 60 months who had been hospitalized in the previous 2 months at each hospital and who presented with: (i) a cough or breathing difficulties; (ii) diarrhoea; or (iii) fever -all common in childhood. Data collectors filled in a paper-based template for the indicators. Subsequently, two data collector coordinators transferred this information into a predefined electronic spreadsheet, which they then sent to the study investigators by email. The study investigators checked the spreadsheets for internal consistency after each data collection.
At the baseline assessment (i.e. time point T1), health staff, data collectors, patients and data analysts were all blinded to the study allocation group. During the intervention (i.e. T2 to T6), neither hospital staff nor the supportive supervision team could be blinded but they were not involved in data collection or data analysis. Data collectors knew the allocation group but were not involved in data analysis. However, children and their families were blinded to both the allocation group and the characteristics of the intervention.
In addition, an independent team of international consultant paediatricians assessed the overall quality of care delivered at each hospital at time points T1 and T6 using WHO's Hospital care for children: quality assessment and improvement tool (Fig. 2), adapted for use in the WHO European Region. 24 The tool systematically assesses different components of the health system that contribute to quality of care -it includes three main sections and a total of 17 subsections. The main sections are: (i) hospital support systems; (ii) case management; and (iii) policies and organization of care. Using structured checklists in the assessment tool, the team attributed a score to each of approximately 250 items based on standards derived from WHO recommendations and other relevant guidelines. 24,25 During the evaluations, a summary score was derived for each of the 17 subsections: it ranged from 0 for totally inadequate care to 3 for care that met international standards.
To ensure data quality, we (i) fieldtested the template used for collecting data on indicators of inappropriate case management, which included an explicit definition of each indicator, before being used in the study; (ii) field-tested data collectors' performance before the study and their understanding was reinforced, if necessary, to ensure consistency; (iii) rotated data collectors among hospitals to reduce the influence of subjectivity; (iv) determined the reliability of the data collectors in evaluating medical records by comparing their evaluations with those of a team of independent international paediatricians at time points T1, T2 and T6 (Fig. 2); and (v) monitored the completeness and internal consistency of the data collected by an external independent data analyst at regular intervals after each supportive supervision visit.
The ethical committees of the Kyrgyz State Medical Institute of Postgraduate and Continuous Training and of the Institute for Maternal and Child Health IRCCS Burlo Garofolo in Italy approved the study. In addition, the director of each participating hospital gave informed consent. The privacy of children and their families was protected by collecting data anonymously. Data are reported in accordance with the Consolidated Standards of Reporting Trials (CONSORT) statement for cluster randomized controlled trial 27 and the trial is registered with ClinicalTrials.gov under the identifier NCT02001116.

Statistical analysis
Using data from the baseline assessment study, we calculated the sample size required by taking into account: (i) the fixed number of clusters; (ii) the intracluster correlation coefficient determined at the baseline assessment; (iii) the expected effect of the intervention; and (iv) the desired power of the study. 28,29 In the baseline assessment, the proportion of the primary indicator was 45% and the intracluster correlation coefficient was 0.16. Under the hypothesis that the proportion of children who satisfied the conditions for the primary indicator would decrease after the intervention by 35 percentage points in the intervention group and by 10 percentage points in the control group, we estimated that a total of 640 children (i.e. 32 at each of the 20 facilities) had to be evaluated at each time point to detect a significant difference between the two groups with a power of 80% and a significance of 90% (α = 5%, two-sided test).
We present categorical variables as absolute numbers, proportions and risk ratios (RRs) with 95% confidence intervals (CIs) and quantitative vari- (i) The child had "no pneumonia: cough or cold" according to WHO pocket book criteria and did not satisfy the criteria for "severe pneumonia" (thereby requiring hospitalization) but was hospitalized; (ii) the child had "some dehydration" according to WHO criteria and did not satisfy the criteria for "severe dehydration" but was hospitalized Incorrect diagnosis Failure to comply with WHO pocket book recommendations on diagnosis (i) The child had "no pneumonia: cough or cold" according to WHO pocket book criteria but was diagnosed with "pneumonia" or "severe pneumonia"; (ii) the child had "some dehydration" according to WHO criteria but was diagnosed with "severe dehydration" Incorrect treatment Failure to comply with WHO pocket book recommendations on treatment (i) The child should have received treatment for "no pneumonia: cough or cold" according to WHO pocket book criteria but was treated for "pneumonia" or "severe pneumonia"; (ii) the child had "some dehydration" according to WHO criteria but was treated for "severe dehydration" Inconsistent diagnosis and treatment There was no consistency between diagnosis and treatment or the diagnosis was not described clearly in the patient's chart (i) The child was diagnosed with "pneumonia" according to WHO pocket book criteria but was treated for "severe pneumonia"; (ii) the child had "some dehydration" according to WHO criteria but was treated for "severe dehydration" Increased iatrogenic risk Administration of two or more unnecessary drugs as indicated by a failure to comply with WHO pocket book recommendations on case management The administration of any unnecessary drug (i.e. a drug not recommended by the WHO pocket book) whose efficacy was not proven and which had possible adverse effects, e.g. steroids or antibiotics for diarrhoea, sedatives for children with fever and so-called cardiotonic or neuroprotective drugs in children without a clear indication for such treatment Unnecessary painful procedures Performance of unnecessary invasive procedures as indicated by a failure to comply with WHO pocket book recommendations on case management (i) Intramuscular or intravenous antibiotic injections in a child who, according to WHO pocket book criteria, should have been treated with an oral antibiotic (e.g. a child with "pneumonia" and no vomiting or signs of "severe pneumonia"); (ii) administration of intravenous fluids to a child who, according to WHO criteria, could have been rehydrated orally (e.g. a child with diarrhoea but with "no dehydration" or "some dehydration" according to WHO criteria and without repetitive vomiting Inadequate monitoring Inadequate monitoring for the clinical diagnosis received, as indicated by WHO pocket book criteria (i) Failure to monitor the respiratory rate at least twice a day in a child with a respiratory infection; (ii) failure to monitor weight at least twice a day in a child with diarrhoea; (iii) failure to monitor neurological status at least twice a day in a child with meningitis Nutritional status not assessed Failure to assess growth or identify malnutrition (i) Failure to assess a child's growth adequately, as indicated by WHO pocket book criteria (i.e. measurement of both weightfor-age and height-for-age and comparison with WHO 2006 growth standards); (ii) failure to identify a child with acute or chronic malnutrition according to WHO criteria Incorrect use of intravenous fluids Fluids not used in accordance with WHO pocket book recommendations (i) Intravenous fluids were prescribed when they were not needed (e.g. when the child was able to drink); (ii) the wrong type of fluid was given (e.g. a hypotonic solution); (iii) the wrong quantity was given (i.e. too much or too little); (iv) fluids were given at the wrong interval WHO Table 2 gives details of the hospitals involved in the study. There was no significant difference in any characteristic between intervention and control groups. In total, data on 4626 children were collected and analysed over the six time points: 2340 in the intervention group and 2286 in the control group. At baseline, there was no significant difference between the groups for any of the 10 indicators of inappropriate case management. During the study period, similar trends were observed for all indicators (Fig. 3). In particular, the primary indicator decreased significantly from 46.9%  T1 T2 T3 T4 T5 T6   T1 T2 T3 T4 T5 T6   T1 T2 T3 T4 T5 T6   T1 T2 T3 T4 T5 T6   T1 T2 T3 T4 T5 T6   T1 T2 T3 T4 T5 T6   T1 T2 T3 T4 T5 T6   T1 T2 T3 T4 T5 T6   T1 T2 T3 T4 T5 T6   T1 T2 T3 T4 T5 T6 Control group Intervention group 95% CI CI: confidence interval. Notes: Groups were randomized and each group consisted of 10 hospitals. The timing of assessments at time points T1 to T6 is described in Fig. 2. The intervention comprised periodic supportive supervision for 1 year after a standard World Health Organization 4-day training course. Details of the number of children's medical records examined at each time point are given in Fig. 1. The combined negative indicator was the combination of the three indicators of: (i) unnecessary hospitalization; (ii) increased iatrogenic risk; and (iii) unnecessary painful procedures. The indicators were obtained from the World Health Organization's Hospital care for children: quality assessment and improvement tool. 24 and are described in Table 1.  24 In contrast, summary scores at the end of the intervention period were significantly higher in the intervention than the control group (Table 3). Additional details of the changes observed and of residual problems that are amenable to future quality improvements are available from the corresponding author.

Discussion
We found that a standard, 4-day, WHO training course followed by periodic supportive supervision from national, trained paediatricians every 2 months for 1 year significantly reduced inappropriate case management of hospitalized children and improved the overall quality of paediatric care. Our findings add to the existing literature [30][31][32] and indicate that supportive supervision can improve both adherence to clinical guidelines and the overall quality of care.
Our study was a pragmatic trial because it was conceived as part of an implementation project rather than being performed in a so-called pure study setting. Nevertheless, the supportive supervision team comprised highly motivated and trained staff and external monitoring was carried out regularly to ensure that supportive supervision was  The results of the study cannot be directly generalized to situations in which a lack of equipment, drugs, supplies or human resources is a major problem or to places where a different form of supportive supervision is being used (e.g. lower-intensity supportive supervision, less-well-trained staff or no external monitoring).
In line with other studies on quality improvement interventions, our outcomes were process outcomes (i.e. indicators of case management quality compared with reference guidelines) rather than health outcomes. 30, 31 We did not assess mortality or morbidity because in-hospital, postneonatal mortality in countries in the WHO European Region is too low to be used for evaluating interventions and because, to be reliable, data on morbidity (such as the proportion of children with complications from common diseases) should be collected at hospital, primary care and community levels. For this study, we selected the measures of inappropriate case management on the basis of previous experience with evaluating the quality of hospital care in countries in the Commonwealth of Independent States, which showed that unnecessary hospitalization, incorrect diagnosis, incorrect treatment, a lack of monitoring and a lack of attention to comprehensive care (e.g. to nutrition) were the main problems. 3-6 These assessment methods are the most reliable and universally accepted, standardized, quantitative indicators currently available for evaluating the quality of paediatric hospital care in settings where mortality is low. Should better indicators become available in the future, they should be considered for use in further research.
A limitation of the study is that the data collectors were not blinded to the allocation group. However, information about the 10 indicators of inappropriate case management was collected from medical records, which are legal documents and, as such, should be considered reliable, and the assessment was based on predefined case definitions, criteria and reference standards, 25 In addition, the reliability of the data collectors' evaluations of medical records was examined at time points T1, T2 and T6 and found to be very high: Cohen's kappa coefficient for inter-rater agreement between data collectors and a team of international independent paediatricians was 0.82, 0.89 and 0.91 at the three time points, respectively. Moreover, it is unlikely that the large differences in quality of care observed between intervention and control hospitals can be explained by the influence of subjectivity.
We did not include cost, patient satisfaction or the satisfaction of health-care providers as study outcomes. However, the administrative data collected suggested that the cost of drugs, especially parenteral drugs, in most intervention hospitals had decreased and interviews with staff and patients indicated anecdotally that satisfaction had increased. Future studies should include a cost-effectiveness analysis and evaluate staff and patient satisfaction. In addition, future research should investigate which approach to supportive supervision results in the greatest improvements in different settings. The existing literature suggests that supportive supervision must be tailored to the local context to some extent. For example, low-or moderate-intensity supportive supervision can be effective in some settings, whereas high-intensity supervision may be needed in others to bring about real changes in behaviour, knowledge and practices. [19][20][21][30][31][32][33] In our study, improvements in all indicators of inappropriate case management were already apparent by time point T3 (i.e. within 6 months of the first supportive supervision visit), which suggested that the intensity of supportive supervision could be decreased in the following period should improvements be observed.
The ideal providers of supportive supervision may vary according to the setting. In our study and a study in Kenya, 30 the supportive supervision team comprised paediatricians; of two trials performed in primary care, the providers were paediatricians in one 31 and medical officers and staff supervising the Integrated Management of Childhood Illness in the other. 32 The characteristics of the ideal provider may depend on who is receiving supportive supervision: adequately trained medical officers may be able to provide effective supportive supervision to primary care staff, whereas a team of experienced paediatricians may be needed for hospital staff.
The intervention hospitals in our study should be further assessed after a longer period of time to determine whether the improvements observed are maintained. Future studies in these hospitals and in different contexts elsewhere should identify the most cost-effective way of providing effective supportive supervision over the long term. Furthermore, projects should also aim to improve case management in primary care and to strengthen referral systems, thereby ensuring better coordination between different levels of care. Our results indicate that policy-makers should consider using supportive supervision to increase adherence to evidence-based guidelines for paediatric hospital care. ■

Improving paediatric care in Kyrgyzstan
Marzia Lazzerini et al.