Introduction

The Millennium Development Goals have catalyzed attention and action to address key health and development issues, while improving the evidence base for holding countries accountable for development progress by tracking results. Attention to maternal and child health commitments has continued, with new pledges to maintain engagement in reaching maternal and child survival goals [1,2]. High-quality and timely data are needed to track progress toward these goals. The ideal source of mortality data is comprehensive vital registration systems with at least 90% coverage of births and deaths [3]. However, only 30% of the world’s population lives in countries with complete birth and death registration [3]. Calls for strengthening vital registration processes have been made simultaneously with maternal and child health commitments, but have only recently been presented with investment and scale up strategies for guidance [4]. The creation or strengthening of a vital registration system is not an easy feat given complex health system weaknesses, bureaucratic inertia, and the range of health issues vying for program support and funding. Moreover, some socio-cultural beliefs at the community level act as barriers to registration [5–8].

In Malawi, efforts began in 1983 to change the 1904 Births and Deaths Registration Act, requiring only the registration of non-Africans, and making registration of Africans voluntary [9]. In 2009, Malawi ratified a compulsory and free birth and death registration act which incorporates health system activities and support into birth registration. Parents must submit the birth report they receive at birth or during a postnatal check-up to the district registrar within six weeks of birth for free registration and to avoid a fine [10]. Considerable work remains to fully implement this system and improve the completeness and quality of birth and death registration data [11]. In the meantime, Malawi is using other demographic and health data sources to inform policy and program decisions. The Malawi National Statistical Office (NSO) relies on the population census and household surveys such as the Demographic and Health Survey and Multiple Indicator Cluster Survey to estimate population and health statistics but these high-quality surveys with national and district level results are only conducted every three to five years [11]. Modeling is used in interim periods but does not generate subnational estimates. A Health and Demographic Surveillance Site was established in 2002 in the Karonga district of northern Malawi that provides complete and high-quality vital events surveillance of its catchment population, but was not designed to produce estimates representative at the national level [12].

An ideal interim mortality data source should generate national and district level mortality estimates with cause of death [13]. With an estimated 11,000 Health Surveillance Assistants (HSAs) located in every catchment area throughout the country, Malawi has a workforce within the health system to capture births and deaths. In fact, HSAs are a de facto vital event documentation system. Their scope of work includes tracking pregnancies and documenting births and deaths in their Ministry of Health-issued Village Health Register, referred to as register in this publication [14]. HSAs have the potential to be a source of district-level vital events data, but there has not yet been any systematic evaluation of the timeliness, completeness, and accuracy of vital events data reported by HSAs.

From January 2010 to December 2013, the NSO and the Institute for International Programs at Johns Hopkins University implemented the Real-Time Monitoring of Under-Five Mortality (RMM) project among a random sample of HSAs in the Balaka and Salima districts of Malawi. These districts were selected due to their high fertility, high under-five mortality, HSA placement in all catchment areas, average district population size, and easy access for the study team based at the NSO in the Southern Region. Details on the sampling procedures are described in detail elsewhere [15]. The main objective of RMM was to assess the accuracy and reliability of HSA routine reports as a source of data on district-level under-five mortality in a high mortality and fertility setting. If successful, the government planned to expand the approach to other districts in Malawi as an interim source for district-level estimates of under-five mortality rates (U5MR), an important indicator reflecting socio-economic development. As a collaborator in the project, the Ministry of Health was interested in assessing whether HSAs were able to perform the part of their job description that requires reporting on vital events to potentially support the strengthening of the civil registration and vital statistics system.

A first phase of RMM covered the period of time between 2010 and 2012, and validation analyses were conducted and published elsewhere [15]. A second phase with enhanced support was implemented between September 2012 and December 2013. The current research is an extension of the earlier phase one validation analysis and covers the full phase one and phase two periods. Here we describe the context, methods, results, and lessons learned from the community-based RMM method implemented from January 2010 through December 2013 in the two districts.

Methods

Program Design

Malawi is a landlocked country in east-central Africa with an estimated population of 13 million inhabitants, 85% of whom live in rural areas [16]. Administratively, Malawi is divided into 28 districts, which are further divided into traditional authorities ruled by chiefs. Health care is provided at district health centers and central hospitals with primary outreach care provided by HSAs, of which 50% should be female. HSAs are required to have a minimum of ten years of education and to complete a 12-week training course, during which they receive a register issued by the Ministry of Health in which to document their work [17]. HSAs are responsible for health promotion, treatment, and health surveillance in catchment areas that cover approximately 1,000 inhabitants [18]. They are expected to reside in their catchment area and are linked to a nearby health facility. Table 1 presents HSA and catchment area characteristics collected in two cross-sectional HSA surveys conducted in the final year of the project.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. HSA and catchment area characteristics.

https://doi.org/10.1371/journal.pone.0138406.t001

HSAs track pregnancies, births, and deaths in their community through active surveillance conducted as household visits and regular meetings with Village Health Committees in their catchment area. They also have many other tasks, including health education, disease surveillance, prevention and treatment of specific childhood diseases, and monitoring of the water supply and community sanitation, that demand varying amounts of HSA work time [14,19]. The RMM community-based approach relies on HSAs’ active surveillance of births and deaths in their register to calculate yearly estimates of under-five mortality.

The RMM method was developed and implemented in two phases in two districts, Balaka and Salima. Phase one covered the period from January 2010 through August 2012, and included a midline validation survey with a full birth history conducted between October 2011 and February 2012. Details of the initial formative research, study design, phase one implementation, and midline validation survey results are presented elsewhere [20]. Due to underreporting of births and under-five deaths revealed in the midline validation, a situation analysis was conducted in collaboration with the Ministry of Health to identify adjustments needed to the approach for increased reporting and completeness in the mortality results (S1 File). Results from phase one and the situation analysis informed the implementation of phase two, with an enhanced strategy and strengthened incentive structure for HSAs and supervisors. Phase two ran from September 2012 to December 2013. An endline validation survey with a full pregnancy history was conducted between November 2013 and January 2014 to validate data from the complete project timeframe and to provide a basis for comparing results from phases one and two. Fig 1 shows the timeline for project activities in Malawi.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. RMM Timeline.

Timeline of RMM implementation, validations, and meetings from January 2010 through December 2013.

https://doi.org/10.1371/journal.pone.0138406.g001

Data Collection Process

Within the scope of work set by the Ministry of Health, HSAs are expected to identify pregnancies, births, and deaths within their catchment areas during monthly visits to households and Village Health Committees in their catchment area. Through this active surveillance they identify events and record the information in their register. RMM reinforces these tasks and adds only an additional step of extracting pregnancies, births, and deaths each month from the register onto a monthly extraction form. Fig 2 shows the routine flow of information and monitoring/feedback processes associated with RMM. Details of the data management process and supervision standards are discussed in S2 File. Double data entry was conducted in CSPro and was followed by reconciliation conducted by the data editor and cleaning conducted by the NSO principal investigator [21].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Organizational chart.

Organizational chart of RMM data flow and supervision among HSAs, Assistant Environmental Health Officers (AEHO), Environmental Health Officers (EHO), supervisors, district RMM coordinators, and team members at the NSO and the Institute for International Programs at Johns Hopkins University.

https://doi.org/10.1371/journal.pone.0138406.g002

Training

Data review meetings were held bi-annually throughout phase one in both districts to review data and re-train HSAs and their supervisors on data quality guidelines. All HSAs and supervisors were invited to the review meetings, along with Ministry of Health officials, the District Health Officer, the Health Management & Information Systems officer, and other partners such as UNICEF and WHO. During the review meetings, Ministry of Health representatives and NSO data management team members reviewed the HSA scope of work, project results by district since the start of RMM, and data quality guidelines to reinforce the importance of ensuring data quality in terms of accuracy, reliability, and completeness.

The findings of the situation analysis indicated that the data review meetings were an important incentive for both HSAs and their supervisors. In phase two, which was formally introduced at the September 2012 data review meetings, we increased the frequency of these meetings and they were planned for every quarter. Strengthening community ties and the data management process became a priority in phase two. The RMM data management team at the NSO led the trainings of the Village Health Committee to orient them to RMM and to promote collaboration with HSAs. We implemented an mHealth intervention to improve data quality with a focus on improving the documentation of pregnancy outcomes. Phase two incentives, activities, and training details are described in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Description of phase two improvements and rationale.

https://doi.org/10.1371/journal.pone.0138406.t002

A final data review meeting was held in June 2014, during which complete project and validation results were presented and discussed. Participants also provided recommendations for improvement if RMM is continued or scaled-up. The advisory group reviewed this feedback at its final meeting in June 2014 to incorporate with key findings at the dissemination meeting in November 2014.

Results

Validation Surveys

The level of submission of monthly extraction data by HSAs was high. For most months, more than 70 HSAs per district (87.5%) consistently provided extracted data on pregnancies, births, and deaths to the NSO (S2 File).

Fig 3 compares the U5MR estimates based on HSA data with those estimated using the midline and endline validation surveys. There are some notable differences between the U5MR validation results for Balaka and Salima. The HSA-based U5MR estimates for the first two validation periods (January to December 2010 and April 2010 to March 2011) are within 20% of both the associated endline and midline survey estimates. However, for the subsequent validation periods up to and including January 2012 to December 2012, the HSA-based estimates systematically underestimate U5MR compared to those calculated using the endline validation survey. For the last four validation periods, the U5MR estimates based on HSA data appear to be more consistent with those estimated from the endline survey. However, the last four estimates from the endline survey suggest a consistent decline in U5MR by approximately 45% within 15 months that looks implausible. In Balaka, aside from the first validation periods, the HSA data systematically underestimated U5MR relative to the endline survey.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Under-five mortality estimates.

Under-five mortality rate estimates, RMM and midline and endline survey data, for annual periods from January 2010 through December 2013 (per 1,000).

https://doi.org/10.1371/journal.pone.0138406.g003

In Salima, the HSA data, on average, systematically underestimated U5MR relative to the endline validation survey. Also in Salima, in contrast to Balaka, there are inconsistent validation results for the periods for which the endline and midline survey reference periods overlapped. Validation against the midline survey for the first four annual periods (i.e., January 2010 to December 2010 through October 2010 to September 2011) suggests that the HSA data considerably underestimated U5MR for these four annual periods. Validation against the endline survey suggests that the HSA data estimated U5MR within 10% of those estimated by the endline survey for the first three annual validation periods. Given that these endline survey estimates rely on a longer recall period than the midline survey, and thus may be affected by recall lapse (i.e., underreporting) of child deaths by survey respondents, we consider the midline validation results more accurate. The preliminary results of the combined midline and endline survey validation for Salima suggest that the HSA data systematically underestimated U5MR by 25–50%.

Fig 4 shows the ratio of U5MR generated from the HSA data relative to those generated from the midline and endline surveys. For the last three 12-month annualized periods of the study period, the ratio of HSA-based U5MR estimates to those based on the endline validation survey exceeded 80%. However, this apparent improvement was driven by an implausibly large and rapid decline in U5MR estimated by the endline validation survey in Balaka. Hence, despite some initial promise, the HSA-based vital events reporting generally systematically underestimated annualized U5MRs in both Balaka and Salima.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Ratio of under-five mortality rates.

Ratio of under-five mortality rates, RMM data to midline and endline surveys, for annual periods from January 2010 through December 2013 (%).

https://doi.org/10.1371/journal.pone.0138406.g004

Village Health Register Verification Surveys

We verified the content of HSA vital events reported to the NSO with corresponding entries in their registers, and found high levels of consistency between the two sources both in phase one and in phase two (Table 3). In phase one, 92.3% of births reported to the NSO were found in their registers, but this declined to 83.5% in phase two. Death reporting also declined between verification assessments from 90.5% in phase one to 87% in phase two. The only slight improvement between assessments was in the documentation of the date of birth. The date of birth reported to the NSO matched register documentation in 85.9% of cases in phase one and 89.8% in phase two.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Comparison of select VHR Verification Survey indicators from phase one, January 2012, and phase two, August 2013.

https://doi.org/10.1371/journal.pone.0138406.t003

We were able to confirm more than 90% of births and around 90% of under-five deaths reported by HSAs by visiting the household where the event occurred (Table 3). This suggests that the RMM community-based method can produce accurate data when events are fully documented. Verification results from phase two produced slightly lower consistency than those reported in phase one, with associated differences in reporting consistency between the two phases.

Discussion

These results present an overview of process, data quality, and outcome indicators associated with implementation of RMM community-based methods in Malawi. We have shown that HSAs are capable of documenting and reporting vital events, but they consistently produce underestimates relative to household surveys with women of reproductive age. HSAs generally documented information on births and deaths with a high level of accuracy when they identified and recorded an event. On average, however, over the course of the entire RMM study in Malawi, HSAs underreported births by 44% and under-five deaths by 49%, resulting in an underestimate of the annualized U5MR of about 41%. The completeness of vital events reporting also varied over time. The range of underestimation of the U5MR by HSAs varied between 11% and 59%. Thus, the community-based RMM method did not accurately capture either the levels of, or trends in, childhood mortality. Even after the introduction of additional performance-based incentives for HSAs and increased supervision of the routine vital events reporting procedures in phase two of the study, HSA-based vital events reporting continued to be substantially incomplete and therefore the accuracy of HSA-based U5MR estimates did not improve.

RMM would be an attractive approach to consider as an interim data source to estimate short-term change in U5MR, if accurate results could be produced. RMM reinforces an existing government structure for community-based monitoring of vital events, and avoids creating separate, parallel structures. Additionally, RMM also reinforces the documentation process and the data quality of vital event documentation set by the Ministry of Health, enhancing its potential for sustainability [22,23]. This potential is further strengthened with the engagement of actors at all levels of the community and health system, especially those at the national and community level. Participation by Ministry of Health representatives in RMM review meetings motivated HSAs and improved the HSAs’ understanding of their scope of work, which is constantly evolving with the inclusion of new tasks from the Ministry of Health and local NGOs [22]. With the evolution of the role of community-based health workers and the competing demands on their time that often result in task shifting, the review of work responsibilities has been shown to improve motivation and satisfaction among community health workers [17,22,25,26]. The Ministry of Health should disseminate an HSA scope of work description annually to promote regular assessment of HSA responsibilities at the ministry level and provide job clarity among HSAs, supervisors, and health facility staff at the community level.

Although the approach of using HSAs for monitoring child mortality appears feasible and attractive, it requires a high level of input to ensure HSAs and their supervisors are well-trained and have support, and that incentives and supervision are provided continuously to maintain high levels of performance. Research has found that these inputs are important for job satisfaction and completion of tasks, but alone do not guarantee motivation and engagement [17,24,25]. We did not foresee a need for intensive training of HSAs, because the documentation of vital events was already part of the job description of the HSAs prior to the implementation of RMM. We learned after phase one that considerable additional investments were needed to strengthen the data management process, yet we did not find improvements in under-five mortality reporting in phase two. Full implementation of phase two incentives and activities was not accomplished until July 2013, which may have contributed to the lack of improvement in reporting during phase two.

Both HSAs and supervisors reported that data review meetings were an important incentive. Plans to hold these meetings quarterly in Phase two were not able to be fully realized, and this may have contributed to the lack of greater improvement in reporting accuracy between phases one and two. Research has shown that satisfaction with training is indicative of community health worker capacity and motivation, so the RMM training process would need to be improved if it is scaled up [26,27].

The data management process also required continuous assessment and support. For this reason, phase two included data management changes to improve data quality and the efficiency of the data management process. Despite these reinforcements, results did not fall within the acceptable range. Communication between the NSO and HSAs was also strengthened early on in the project, when the data editor was given the role and materials to communicate with HSAs by phone in order to notify them of field visits and to follow up on data quality issues. This improved the timeliness with which data quality issues were resolved, but cell phone network limitations and limited access to monthly airtime vouchers affected the data editor’s ability to follow up and confirm some events in a timely fashion.

The advantage of building RMM within an existing government structure also resulted in some important challenges. The integrated design hampered the efficiency of the data management process, because the shortcomings of the existing system affected RMM project operations. We worked primarily with Ministry of Health employees at the district level. As a result, there was limited control over these staff members, whose first responsibility was to their government duties. The scheduling of review meetings, submission of extraction forms, and completion of supervision were sometimes delayed or cancelled due to HSA and supervisor task shifting when activities such as indoor residual spraying or in-service training took priority over RMM. Surveillance of vital events is best conducted without interruptions, which cannot occur when HSAs are required to temporarily modify their work routine.

RMM also faced frequent staff turnover at all levels of the project and had little control over these personnel changes. The local NSO principal investigator for the project and district coordinators changed multiple times, with negative effects on coordination, management, and communication. At the community level, 28% of catchment areas experienced at least one turnover of the HSA during the project period. Research has shown that program continuity is hampered by turnover [27].

Full implementation of community-based reporting of vital events would not mean that other methods of collecting mortality data could be abandoned. High-quality survey data with sufficient power to validate results would be needed to evaluate the completeness of estimates. The RMM data management team at the NSO found it challenging to manage both the continuous RMM data management tasks and simultaneously manage the midline and endline surveys, which may have affected data quality of the latter.

The comparison of survey results for the first four annual periods suggests that endline validation survey interviewers did not capture all events. Also, the endline survey estimate of U5MR of 60 per 1,000 live births in 2013 suggests a dramatic halving of child mortality over the last few years in Balaka. Such a notable decline seems unlikely. This suggests that the ratio results comparing the community-based RMM method and the endline validation survey are overestimated, particularly for the later annual periods, and an adjustment of ratios would result in even more modest validation results for the HSA community-based method. We did not conduct a record linkage exercise between the HSA vital event reports and the validation surveys, so it is not possible to quantify the magnitude, or describe the characteristics of, events that were missed by the HSA and the current best practice surveys. We attempted to use results from the Millennium Development Goal survey conducted from November 2013 to April 2014 by the NSO with support from UNICEF and other partners, but the sample sizes of the survey did not provide sufficient power for it to be used to validate the phase two results.

The operational realities that resulted in poor data quality pose important sustainability challenges to community-based health workers and data management staff at the NSO. Further, pregnancy histories are not easy to implement as part of a validation survey and can be hampered by substantial non-sampling errors, as was the case in Balaka. However, even if the pregnancy histories underestimated annualized U5MRs in 2013, our results still show that the vital events data collected by HSAs consistently and significantly underestimated annualized U5MRs.

Conclusions

The RMM experience in Balaka and Salima districts demonstrates that HSAs have the potential to generate data to support estimates of under-five mortality to provide timely data to policy makers while the recently enacted vital event registration system is being fully implemented. However, our findings indicate that the current system needs substantial strengthening to ensure that birth and death reporting by HSAs is sufficiently complete over time to ensure reasonably accurate estimates of annualized U5MR. The optimum level of incentives and support to make the system work while ensuring sustainability is extremely difficult to find. We invested additional resources in phase two to identify and implement these supports and incentives but did not find the optimum level. Policy makers and program managers in Malawi can use lessons learned from RMM implementation to improve the HSA system and supports for improved outcomes.

Supporting Information

Acknowledgments

We are grateful to the members of the RMM Malawi working group: Gunther Baugh (World Health Organization); Elizabeth Hazel (Department of International Health, Johns Hopkins Bloomberg School of Public Health); and Martina Chimzimu, Kingsley Lajia, Roberta Makoko, and Tiope Mleme (Malawi National Statistics Office). Their contributions to the design and implementation of this research reported in this article were critical. We thank the Health Surveillance Assistants, HSA supervisors, RMM district coordinators, district health officers, and environmental health officers in Balaka and Salima; the leadership and staff of the Malawi National Statistics Office (in particular, Frank Matumula, Raphael Talipu, and Willie Kachaka); Ministry of Health personnel (namely Humphreys Masuku, Edwin Nkono, Humphreys Nsona, and Fannie Kachale); IIP colleagues who assisted with technical support (namely Emily Wilson, Andrew Marsh, Emily Carter, and Alain Koffi); and our collaborators (Alister Munthali and Peter Mvula) who worked with us on the formative research that informed this study. We are also grateful to the Department of Foreign Affairs, Trade and Development Canada for their generous financial support of the Real-Time Monitoring of Under-Five Mortality project.