Developing global indicators for quality of maternal and newborn care: a feasibility assessment

Abstract Objective To assess the feasibility of applying the World Health Organization’s proposed 15 indicators of quality of care for maternal and newborn health at health-facility level in low- and middle-income settings. Methods Six of the indicators are about maternal health, five are for newborn health and four are general cross-cutting indicators. We used data collected routinely in facility registers and obtained as part of facility assessments from 963 health-care facilities specializing in maternity services in 10 countries in Africa and Asia. We made a feasibility assessment of the availability of data and the clarity of indicator definitions and identified additional information and data collection processes needed to apply the proposed indicators in real-life settings. Findings Of the indicators evaluated, 10 were clearly defined, of which four could be applied directly in the field and six would require revisions to operationalize them. The other five indicators require further development, with one of them being ready for implementation by using information readily available in registers and four requiring further information before deployment. For indicators that measure coverage of care or availability of services or products, there is a need to further strengthen measurement. Information on emergency obstetric complications was not recorded in a standard manner, thus limiting the reliability of the information. Conclusion While some of the proposed indicators can already be applied, other indicators need to be refined or will need additional sources and methods of data collection to be applied in real-world settings.


Introduction
Improving the quality of care for maternal and newborn health is important if health outcomes for mothers and babies are to continue to improve. This will require a renewed global focus. 1 According to 2015 estimates, annually 303 000 women die during pregnancy, childbirth or in the postnatal period, 2.6 million babies are stillborn and 2.7 million babies die within 1 month of birth. [2][3][4] The majority of these deaths occur in lowand middle-income settings and are preventable. Ensuring quality care is provided to every mother and newborn during this period is critical for maternal and newborn survival.
Monitoring of progress towards the achievement of millennium development goals 4 and 5 -i.e. reduce child mortality and improve maternal health, respectively -focused initially on measurement of coverage of evidence-based costeffective interventions such as antenatal care and skilled birth attendance rates. 5 Although intervention coverage rates (i.e. the number of people receiving an intervention or service from among those who need it) have been increasing rapidly, 6,7 it is widely acknowledged that the quality of care provided for mothers and babies is rarely evidence-based and womencentred. Uptake (and coverage) of care and quality of care are also linked; numerous examples exist in the literature describing where and how poor quality of care has deterred women from accessing services even where these were available, close by and affordable. [8][9][10][11] The United Nations (UN) Commission on Information and Accountability for Women and Children's Health was established in 2011 to enhance local, national and global accountability for women and children's health. The commission identified 10 recommendations to be adopted by countries. 12 The first set of recommendations focused on better information for results, calling for countries to strengthen vital registration and health information systems, to focus on a core set of harmonized maternal newborn and child health indicators, and to invest in information, communication and technology to strengthen their national health information systems at all levels. More recently, Countdown to 2015 reports acknowledged the need to monitor quality of care as well as coverage of interventions, and the corresponding need for better data to do this. 13 These developments are also identified in the sustainable development goals, with recognition that reaching such targets as reducing the maternal mortality ratio to under 70 per 100 000 live births by 2030, will require continued efforts to improve quality of care, underpinned by the availability of robust evidence. [14][15][16][17][18] In response to the need for more and better data, the World Health Organization (WHO) was asked to propose trace indicators for quality of maternal and newborn health care provided at health-care facility level that could be used for global comparisons. After consultation with a wide range of international stakeholders and experts in quality of care, a core set of 15 indicators was proposed (Box 1). 19 The indicators were thought to be good markers of lifesaving interventions and were obtained via consensus, but have not been formally assessed to determine whether they complement or link to data already routinely collected for maternal and newborn health. This paper describes a study of the feasibility of applying the proposed indicators in low-and middle-income settings. We assessed the availability of data in existing facility records and the clarity of indicator definitions, and identified additional information and processes needed to collect the data in real-life settings.
Objective To assess the feasibility of applying the World Health Organization's proposed 15 indicators of quality of care for maternal and newborn health at health-facility level in low-and middle-income settings. Methods Six of the indicators are about maternal health, five are for newborn health and four are general cross-cutting indicators. We used data collected routinely in facility registers and obtained as part of facility assessments from 963 health-care facilities specializing in maternity services in 10 countries in Africa and Asia. We made a feasibility assessment of the availability of data and the clarity of indicator definitions and identified additional information and data collection processes needed to apply the proposed indicators in real-life settings. Findings Of the indicators evaluated, 10 were clearly defined, of which four could be applied directly in the field and six would require revisions to operationalize them. The other five indicators require further development, with one of them being ready for implementation by using information readily available in registers and four requiring further information before deployment. For indicators that measure coverage of care or availability of services or products, there is a need to further strengthen measurement. Information on emergency obstetric complications was not recorded in a standard manner, thus limiting the reliability of the information. Conclusion While some of the proposed indicators can already be applied, other indicators need to be refined or will need additional sources and methods of data collection to be applied in real-world settings.
Quality of maternal and newborn care Barbara Madaj et al.

Data sources
For all indicators except one (G3), the information used in this evaluation was based on the results of a baseline facility assessment conducted by the Centre for Maternal and Newborn Heath at the Liverpool School of Tropical Medicine. This was part of a capacity-building programme implemented between January 2012 and December 2015 aiming to improve the availability and quality of emergency obstetric and newborn care (Making it Happen programme). 20 The facilities surveyed were selected by the respective ministries of health and included public health facilities designated to provide maternity services in purposively selected geographical and administrative areas (n = 53). With the exception of Zimbabwe, where only central-level referral hospitals were surveyed (n = 5), the sample included facilities offering either basic or comprehensive emergency obstetric care. This classification indicates the complexity of care provided (with comprehensive emergency obstetric care facilities being required to offer caesarean section and blood transfusion services in addition to basic package of care) and broadly serves as an indication of the facility size. 23 The primary data, which were subsequently used in the feasibility assessment, were collected prospectively according to the respective programme protocol using a standardized tool incorporating elements from the WHO and Averting Maternal Death and Disability health facility assessment tools. 24 Information was collected during health-facility visits by trained data collectors who interviewed health-care providers. Data were verified using routine facility data sources, e.g. labour ward and operating theatre registers, newborn care unit registers and patient discharge registers.
Permission to conduct the facility assessments was granted by the respective ministries of health. Our audit covered existing data that were already available in standard, anonymized records. No information which would compromise the confidentiality or privacy of patients or staff was recorded or included in the analysis.

Data extraction and analysis
For this assessment we first identified and extracted the data required to measure each indicator. This enabled us to assess the availability of the information in routinely collected facility records and, where possible, the completeness of the records across countries. From discussions among the research team, who were experienced in conducting facility assessments, we examined the clarity of the indicator definitions against the information currently available in facility records. For indicators where no data were readily available, and based on the team's clinical and research expertise, we assessed the feasibility of obtaining the necessary information. We also assessed the approach and methods

Box 1. Indicators for assessing quality of maternal and newborn health services proposed by the World Health Organization and stakeholders a,b
Maternal health M1: Proportion of antenatal care visits at which blood pressure was measured M2: Proportion of women with severe pre-eclampsia or eclampsia treated with magnesium sulfate injection M3: Proportion of women receiving oxytocin within 1 min of birth of infant M4: Proportion of women with prolonged labour M5: Intrapartum stillbirth rate M6: Proportion of women with severe systemic infection or sepsis in postnatal period, including readmissions Newborn health N1: Proportion of health facilities with functional bags and masks (two neonatal mask sizes) in the delivery areas of maternity services N2: Proportion of newborns who received all four elements of essential care: • immediate and thorough drying • immediate skin-to-skin contact • delayed cord clamping • initiation of breastfeeding in the first hour N3: Proportion of health facilities in which kangaroo mother care is operational, by level of facility N4: Facility neonatal mortality rate disaggregated by birth weight: > 4000 g, 2500-3999 g, 2000-2499 g, 1500-1999 g, < 1500 g N5: Proportion of health facilities offering maternity services certified by the Baby-friendly Hospital Initiative and recertification no later than 2 years afterwards

General indicators
G1: Proportion of health facilities that have stock-outs of essential lifesaving medicines for mothers and newborns in a specified period G2: Proportion of maternal and perinatal and child c deaths occurring in a facility that were reviewed G3: Proportion of health facilities with soap and running water or alcohol-based rub available in labour, childbirth, neonatal and paediatric wards G4: Proportion of health facilities with safe, uninterrupted oxygen supply in childbirth, neonatal and paediatric wards a The abbreviations (M1, M2, etc.) are used to facilitate referencing in this paper and were not in the original report. b This table excludes four child indicators proposed in the consultation 19 because these were not part of this assessment. c This analysis excludes the child death reviews. Source: World Health Organization, 2014. 19 Research Quality of maternal and newborn care Barbara Madaj et al.
needed to measure the indicator at the health-facility level, using alternative measures at facility level. Finally, in cases where the proposed indicator could not be assessed in full, we developed proxy measures for which routine data and data collection systems are readily available.
We present the assessment findings for each indicator by summarizing the descriptive information and by analysing the availability of the required data using descriptive statistics. Table 2 provides a summary of all proposed indicators based on the clarity of definitions and the availability of routine information to assess each indicator. Overall, 10 of the 15 indicators were considered to be clearly defined in their current format. However, using available facility registers, data would in principle be immediately accessible only for four indicators (M5, N1, N5 and G3), while the other six (M1, M2, M3, N2, N3 and G4) would require additional sources of information to operationalize them. Among the five indicators which require some further development, one (G1) could be implemented with currently available information, while the remaining four (M4, M6, N4 and G2) would need supplementary information.

Results
We further analysed each proposed indicator with regard to its potential for application in real-world settings.
The key findings regarding the availability of data for each indicator and discussion of alternative indicators or additional methods of assessing the indicator are summarized in Table 3 (available at: http://www.who.int/bulletin/volumes/95/6/16-179531). Table 4 (available at: http://www.who.int/bulletin/volumes/95/6/16-179531) shows the availability of data across countries. Country-specific differences were noted with regard to individual indicators. However, across all countries, emergency obstetric complications posed a challenge because existing registers lack dedicated space for recording cases and consequently information on complications is not recorded in a standard manner, thus limiting the reliability of  22 c For all countries, the areas and facilities for the surveys were selected by the respective ministries and represented geographical and administrative areas which were identified as needing capacity-building around emergency obstetric care. For indicator G3, the data came from a facility survey in Sierra Leone, conducted as part of a study assessing the effect of the Ebola virus disease outbreak on availability, uptake and demand for essential maternal and newborn health services. d Basic emergency obstetric care facilities are required to offer the following services: administer parenteral antibiotics, administer uterotonic drugs (i.e. parenteral oxytocin), administer parenteral anticonvulsants for pre-eclampsia and eclampsia (i.e. magnesium sulfate), manually remove the placenta, remove retained products (e.g. manual vacuum extraction, dilation and curettage), perform assisted vaginal delivery (e.g. vacuum extraction, forceps delivery) and perform basic neonatal resuscitation (e.g. with bag and mask). 23 e In addition to the seven services at basic level, comprehensive emergency obstetric care facilities are expected to provide blood transfusion services and perform surgery (e.g. caesarean section). 23 f South Africa data were not available for M1, M4, M6, N4 and G2; Zimbabwe data were not available for N4. g G3 is the proportion of health facilities with soap and running water or alcohol-based rub available in childbirth, neonatal and paediatric wards. h Only central-level referral hospitals were surveyed.

Quality of maternal and newborn care
Barbara Madaj et al.
the information. Additionally, on a more practical level, anecdotal evidence from fieldworker notes suggests assessments in larger facilities required consulting records and registers from various wards and sources and therefore took more time and effort to consolidate the findings. Data on the number of women giving birth and the number of babies born were available at all facilities. Our surveys did not collect information for indicators M1, M4, M6, N4 and G2 in South Africa and indicator N4 in Zimbabwe, thus affecting the denominators used in calculations in the assessment.

Maternal health indicators
M1: Proportion of antenatal visits at which blood pressure was measured. Data available for the assessment of the indicator did not include information from antenatal clinics. Instead, a proxy measure was derived using the availability of blood pressure monitors of any type in maternity services. Generally, data on the availability of these monitors were accessible at facilities and only 3% of facilities overall (29/830) could not provide the information (Table 3).
M2: Proportion of women with severe pre-eclampsia or eclampsia treated with magnesium sulfate injection. Data on women treated with magnesium sulfate were not routinely available. Instead, data on number of (pre)eclampsia cases and availability of magnesium sulfate were used as proxies. Overall, 9% of facilities (86/963) did not hold records on numbers of patients with (pre)eclampsia, with missing data most pronounced at country level in Bangladesh, Ghana and Nigeria (Table 4). Data on magnesium sulfate availability show that 3% of facilities (27/963) were not able to provide the information.
M3: Proportion of women receiving oxytocin within 1 min of birth of infant. Data on availability of oxytocin were widely available and missing in only 3% of facilities (28/963). However, the use of oxytocin as part of Active Management of the Third Stage of Labour (AMTSL) and/or whether AMTSL was practised was not routinely recorded in birth registers.
M4: Proportion of women with prolonged labour. Routine use of the partograph and number of cases of ruptured uterus were used as proxy measures for this indicator. All except 1% of facilities (10/830) were unable to provide data on partograph use. Recorded cases of ruptured uterus were missing in 9% of facilities (78/830), although there was variability between countries, with facilities in Bangladesh, Ghana and Nigeria facing challenges in reporting data (Table 4).
M5: Intrapartum stillbirth rate. Fresh stillbirth may be used as a surrogate measure for intrapartum stillbirths, although information on fetus weight at admission or whether fetal heart rate was heard was not generally available in facility registers. Data on stillbirths were widely available (2%, 16/963 facilities overall had missing data), but data with stillbirths disaggregated into fresh and macerated were missing for over a quarter of reported stillbirths (27%, 1580/5930). Except in South Africa, which does not report these data, disaggregation of stillbirths forms part of routine record-keeping; however, data were more commonly missing in Bangladesh, Nigeria and Pakistan (Table 4).
M6: Proportion of women with severe systemic infection or sepsis in postnatal period, including readmissions. Data on the number of postnatal sepsis cases were missing in 9% of facilities (73/830), but data on readmissions were not available in any health-facility registers.

Newborn health indicators
N1: Proportion of health facilities with functional bags and masks (two neonatal mask sizes) in the delivery areas of maternity services. Data on bag and mask availability were largely accessible, with missing data at very few facilities (< 1%, 4/963) (Table 3). However, data on specific sizes of bag and masks were not available.
N2: Proportion of newborns receiving all four elements of essential care. This was not documented as part of any routine register in the surveyed health facilities.
N3: Proportion of health facilities in which kangaroo mother care is operational. Although, in principle, information on whether kangaroo mother care was provided was anecdotally available in facilities, this indicator was not assessed routinely or recorded in any existing register at facility level.
N4: Facility neonatal death rate disaggregated by birthweight. Based on the assessment, health facilities mostly lacked neonatal discharge and death registers. Moreover, no data on deaths by birth weight categories were available. Neonatal death rates could be estimated in 540/825 (66%) of all facilities surveyed, based on the difference between the numbers of babies discharged alive and number of live births in the facility. However, babies discharged alive comprised both babies born in the facility and those referred from outside, thus potentially limiting the usability of the data. It was, nevertheless, the only proxy measure available.
N5: Proportion of health facilities offering maternity services that are certified as baby-friendly under the Baby-Friendly Hospital Initiative. 25 In many countries this information was only available at health facilities which had been part of a programme to specifically introduce this accreditation, and the information was not available from facility records.

General indicators
G1: Proportion of facilities which had stock-outs of essential lifesaving medicines for mothers, newborns and children in a specified period. Medicines assessed in the health facility surveys covered only part of the WHO essential drugs list. Nevertheless, information was readily accessible on the availability (i.e. whether a particular drug was available at all times, with stock-outs or not at all in the 3 months covered by the survey) of selected antibiotics (penicillin, metronidazole, gentamicin, cephalosporin), oxytocics (oxytocin, misoprostol) and an anticonvulsant and antihypertensive drug (magnesium sulfate, nifedipine). Among the groups of drugs, the highest percentage of missing data for availability of individual medicines among antibiotics was at 4% (37/963), 3% (28/963) for oxytocics and 4% (31/830) for anticonvulsants (Table 3).
G2: Proportion of maternal, perinatal and child deaths occurring in a facility that were reviewed. Data on the proportion of deaths reviewed were not collected as part of the health-facility assessments. However, for maternal and perinatal deaths, availability of review committees and whether or not action was taken could be used as proxy indicators. Data on these quality improvement activities were largely available. The existence of a quality improvement committee was reported by all except 3% of facilities (22/830), while information on the existence of maternal death reviews and perinatal/stillbirth review was missing in 2% (16/830) and 1% (12/830) of facilities respectively. However, data on actions taken were not necessarily informative and lacked detail of what the action entailed, and no standardized system for reporting the information was identified.
G3: Proportion of health facilities with soap and running water or alcohol-based rub. The feasibility of this indicator was assessed only in Sierra Leone. Data on water availability were generally accessible, with data missing for 3% of facilities (2/76), both offering basic emergency obstetric care services (Table 4). Information on availability of hand-washing facilities was missing at just 1% of facilities (1/76), with details of products for hand-hygiene widely available.
G4: Proportion of health facilities with safe, uninterrupted oxygen supply in childbirth, neonatal and paediatric wards. Data necessary to inform the indicator were not routinely collected at facility level.

Discussion
Our assessment used existing facility data from a large and broad selection of health-care facilities specializing in maternity services in 10 countries in Africa and Asia to assess the availability of data for each indicator (and the variability in data availability). Our work demonstrated that, while some of the proposed indicators can already be applied, other indicators need to be refined or will need additional sources and methods of data collection.
WHO indicators for quality of maternal health care (M1-M6), are related to clinical process, and require observation or special recording, and are unlikely to be captured in full as part of a standard facility survey. Sampling of case records and registers could be used to make the indicators more appropriate for measurement of the quality of the services provided. The indicators of quality of newborn care (N1-N5) include composite indicators (e.g. essential care at birth) which are in practice challenging to define and capture. The denominators for some of the indicators vary, and encompass mothers and babies as well as facilities, which allows for capturing a wide range of information. In practice, however, a variety of denominators may complicate any attempt to collect data in a standardized manner that allow for comparison across healthcare facilities or geographical settings. 13 Indicators that measure coverage of care and policy or guideline adherence require additional information to be useful for monitoring of quality of care. For example, whether women with pre-eclampsia and eclampsia are treated with magnesium sulfate (indicator M2) is not routinely recorded in registers, and may require analysis of case notes. Likewise, information on time oxytocin was administered (indicator M3) is not routinely recorded, and the 1-minute timeframe may not be realistic. For the indicators on essential newborn care (N2) and kangaroo care (N3), no standardized data are currently collected in registers or case notes, and so monitoring of these indicators would require new or modified data collection tools.
In terms of further specifications required, some indicators need further work to operationalize them. For example, standards are needed to clarify the meaning of terms such as operational (for indicator N3), prolonged labour (for indicator M4) and severe systemic infection (for indicator M6) and to agree clear definitions and criteria for terminology to ensure that they can be effectively utilized for comparison across countries. Other indicators require specifying so that it is clear what needs to be captured in a way that would be measurable. For example, if both early and late neonatal mortality are to be included in health facilities' recording of the neonatal mortality rate by birth weight (indicator N4), then there is a need to collect community data or for functioning vital registration systems to be in place. 27,28 In general, the proposed indicators also need to include a specified timeframe for evaluation, e.g. per quarter, in line with UN facility survey standards. 23,29 This is probably particularly pertinent for measuring stock-outs of essential drugs (indicator G1), but would be helpful for standardizing data collection for other indicators. For stock-outs of drugs, it would be helpful to differentiate between time-bound and permanent lack of availability of products or services. This could mean recording whether the drug was only temporarily unavailable and defining the number of days before a temporary lack of drugs is classified as a stock-out. Additionally, the list of essential drugs needs to take into account regional or local guidelines and practices, and could include a tracer drug or drugs (at least one of which needs to be available) to allow for standard monitoring.
Better data are needed with regard to both the availability of maternal and newborn care and the equality of that care. For the proposed indicators to provide an assessment of quality, not just coverage, of care they will need to reflect all components of care provision including input, process and outcome measures. 30 Moreover, quality of care can mean different things to the provider and the consumer of care. 31,32 The current set of proposed indicators does include input, process and outcome measures and is therefore a useful basis for assessing care. Nevertheless, the list will need further refinement and possibly expansion to ensure that the indicators used are representative of all aspects of quality.
This study had some limitations. First, the data used in the analysis were not collected for the purpose of this project and therefore some aspects of the assessment could not be performed. Second, the findings may not be generalizable as the results may not necessarily reflect the situation nationally in the countries from which the data originated or may not be immediately applicable to other settings. On the other hand, the data covered 10 countries in Africa and Asia and the national data recording systems within these countries are uniform. We argue that facility records can be a source of robust evidence when indicators are clearly defined and specified in existing registers.
Overall, the WHO proposed global core indicators focus on important elements of quality of care around the time of birth, and of care of the small or sick newborn, and include a balance of intervention coverage, process of care and impact indicators. However, several of the proposed indicators require some revision to be applied in real-world settings for measuring care in health facilities. In addition, for the indicators that measure coverage of care or availability of services or products, there is a need to further strengthen measurement of care quality. Collecting additional information which is not captured routinely at facilities is challenging in large-scale surveys. ■ Funding: WHO funded the assessment.
Conclusión Mientras que algunos de los indicadores propuestos ya pueden aplicarse, otros necesitan perfeccionarse o requerirán fuentes y métodos de recopilación de datos adicionales para poderse aplicar en entornos reales. was generally reported; data missing in 0-27% of facilities across countries. Nigeria, Bangladesh and Ghana were most affected by missing data Data on obstructed labour cases in facilities were available but were deemed unreliable due to poor data recording and lack of an agreed standard for classifying the complication; therefore, availability of data was not included in this assessment Clearer definition of prolonged labour is needed for the assessment of proportion of cases to be calculated Classification of prolonged labour may very between countries; a standardized definition is needed Assessing the quality of partograph completion (via retrospective review of partographs), not only frequency of use, would provide additional information on correct use In facilities without partographs, retrospective review of patient notes to assess the diagnosis could be applied Alternative indicator could be the proportion of deliveries monitored with partograph, among women delivering at the facility (the data need to take account of women being referred to a facility having started labour elsewhere; for referral cases complications may not necessarily reflect the standard of care in the referral facility) Consideration should be given to measuring the number of cases of ruptured uterus among women delivering at (but not referred to) the facility as a measure of quality of care offered at the facility Information on deaths occurring in facilities and corresponding reviews is collected in facilities, especially those with quality improvement activities. However, the information was not collected in facility surveys, and therefore a proxy was used in this assessment Availability of committees was widely reported; missing data in 0-15% of facilities across countries (missing data calculations were based on maximum number of facilities with missing data within each of three categories: quality improvement committee; maternal death reviews; and perinatal death reviews) Information on action taken following the reviews was less reliable and not captured in a standardized way in facilities; therefore, it was not presented in this assessment This assessment did not include child death reviews Definition of indicator requires clarification, as child (i.e. under 5 years old) death review is not a standard facility-based audit (perinatal and maternal death audits are more common) Collecting facility-level data on deaths reviewed can only be done in facilities with relevant committees established Information on individual deaths and corresponding reviews requires a clear time reference (i.e. deaths which occurred in the current review period that were reviewed in that period, or deaths from the previous review period that were not reviewed at the time, or deaths that occurred in the current review period but not reviewed) Frequency of reviews is important to assess, but such information is not available in most countries. Until a higher level system is available, individual facility data could serve as a useful proxy, but requires a reporting tool for committee activities linked to registry data on recoded deaths Definition of running water is required Indicator is measurable through WASH questions (some of which are already included in facility assessment survey tools) 26 Alternative indicator could be used to assess availability of steady supply of clean water and soap or alcohol-based rub Observation and review of policies or guidelines at facility level are additional ways to measure hand-washing standards (. . .continued) (continues. . .) Table 4. 15 (31) 18 (17) 12 (4) 0 (0) 20 (24) 2 (2) 1 (2) 14 (