Surveys of measles vaccination coverage in eastern and southern Africa: a review of quality and methods used

Abstract Objective To assess the methods used in the evaluation of measles vaccination coverage, identify quality concerns and provide recommendations for improvement. Methods We reviewed surveys that were conducted to evaluate supplementary measles immunization activities in eastern and southern Africa during 2012 and 2013. We investigated the organization(s) undertaking each survey, survey design, sample size, the numbers of study clusters and children per study cluster, recording of immunizations and methods of analysis. We documented sampling methods at the level of clusters, households and individual children. We also assessed the length of training for field teams at national and regional levels, the composition of teams and the supervision provided. Findings The surveys were conducted in Comoros, Eritrea, Ethiopia, Kenya, Lesotho, Malawi, Mozambique, Namibia, Rwanda, Swaziland, Uganda, Zambia and Zimbabwe. Of the 13 reports we reviewed, there were weaknesses in 10 of them for ethical clearance, 9 for sample size calculation, 6 for sampling methods, 12 for training structures, 13 for supervision structures and 11 for data analysis. Conclusion We recommend improvements in the documentation of routine and supplementary immunization, via home-based vaccination cards or other records. For surveys conducted after supplementary immunization, a standard protocol is required. Finally, we recommend that standards be developed for report templates and for the technical review of protocols and reports. This would ensure that the results of vaccination coverage surveys are accurate, comparable, reliable and valuable for programme improvement.


Introduction
Four types of surveys are commonly used to estimate vaccination coverage in developing countries: demographic and health surveys, multiple indicator cluster surveys, Expanded Programme on Immunization cluster surveys and surveys based on lot quality assurance sampling. 1,2 Expanded Programme on Immunization cluster surveys have been used to assess coverage in supplementary measles or measles-rubella immunization activities. These surveys often include some evaluation of routine immunization, the communication strategies that have been used and the reasons for non-vaccination. 3,4 Supplementary measles immunization activities are designed to ensure high population immunity against measles in areas that have yet to reach high levels of routine coverage with two doses of vaccine. 5 Typically, nationwide catch-up activities, designed to eliminate susceptibility to measles in the general population, precede follow-up rounds of supplementary immunization. The follow-up rounds are generally conducted nationwide every 2-4 years and generally target children aged 9-59 months. They are designed to eliminate any measles susceptibility -especially in children born since the last round of supplementary immunization activities -and to protect those children who remain susceptible after receiving one dose of measles vaccine. Countries are encouraged to continue supplementary measles immunization until they reach and sustain 93-95% coverage with two doses of vaccine via routine immunization.
In monitoring the progress being made towards measles elimination, population-based surveys that incorporate prob-ability sampling in all stages of sample selection and apply strict measures to minimize bias 1 can provide accurate and reliable estimates of immunization coverage. 4 Surveys of the coverage achieved after supplementary immunization activities may be hampered by problems in (i) obtaining the necessary funding far enough in advance, (ii) managing a logistical operation of considerable size that may be designed to measure the coverage achieved by the supplementary activities, by the routine immunization services, or by both the routine and supplementary interventions, and (iii) obtaining reasonably accurate subnational coverage estimates. The results of such surveys may be perceived as biased if the surveys are not conducted by organizations that are considered to be independent of the immunization programmes that are under scrutiny. Here we assess the methods used to evaluate and report coverage, identify quality concerns and provide recommendations for improvement.

Methods
We reviewed the results of 13 coverage surveys conducted in 2012-2013, following 16 programmes of supplementary measles immunization. Either monovalent measles vaccine or measles-rubella vaccine had been administered. At least one other child health intervention -e.g. the distribution of oral poliomyelitis vaccine, tetanus toxoid vaccine, vitamin A or an anthelminthic drug -was delivered in conjunction with 15 (94%) of the programmes. We investigated the organization(s) undertaking each survey, survey design, sample size, the numbers of study clusters and children per study cluster, recording of immunizations and methods of analysis. We documented Objective To assess the methods used in the evaluation of measles vaccination coverage, identify quality concerns and provide recommendations for improvement. Methods We reviewed surveys that were conducted to evaluate supplementary measles immunization activities in eastern and southern Africa during 2012 and 2013. We investigated the organization(s) undertaking each survey, survey design, sample size, the numbers of study clusters and children per study cluster, recording of immunizations and methods of analysis. We documented sampling methods at the level of clusters, households and individual children. We also assessed the length of training for field teams at national and regional levels, the composition of teams and the supervision provided. Findings The surveys were conducted in Comoros, Eritrea, Ethiopia, Kenya, Lesotho, Malawi, Mozambique, Namibia, Rwanda, Swaziland, Uganda, Zambia and Zimbabwe. Of the 13 reports we reviewed, there were weaknesses in 10 of them for ethical clearance, 9 for sample size calculation, 6 for sampling methods, 12 for training structures, 13 for supervision structures and 11 for data analysis. Conclusion We recommend improvements in the documentation of routine and supplementary immunization, via home-based vaccination cards or other records. For surveys conducted after supplementary immunization, a standard protocol is required. Finally, we recommend that standards be developed for report templates and for the technical review of protocols and reports. This would ensure that the results of vaccination coverage surveys are accurate, comparable, reliable and valuable for programme improvement. a Immunization, Vaccines and Emergencies, WHO Regional Office for Africa, Inter-country Support Team for East and Southern Africa, 86 Enterprise Road, Highlands, Harare, Zimbabwe. b Immunization, Vaccines and Emergencies, WHO Regional Office for Africa, Brazzaville, Congo. sampling methods at the level of clusters, households and individual children. We also assessed the length of training for field teams at national and regional levels, the composition of teams and the supervision provided.
The survey methods were compared with those recommended by WHO in 2005. 6

Results
All six rounds of supplementary measles immunization conducted in 2012 and nine of the 10 conducted in 2013 were followed by a coverage survey. At the time of our review, reports on only 13 of the 15 coverage surveys were available, since reports on two of the coverage surveys conducted in 2013 were pending. The surveys included in this review were performed in Comoros, Eritrea, Ethiopia, Kenya, Lesotho, Malawi, Mozambique, Namibia, Rwanda, Swaziland, Uganda, Zambia and Zimbabwe.
According to unpublished documents submitted to WHO, five (38%) of the 13 surveys were led by a local consultant or consultant firm, one (8%) by a national research institute -with technical support from consultants from several international organizations, four (31%) by WHO's office in Harare, Zimbabwe, and two (15%) by international WHO consultants. Eleven (85%) of the 13 reported surveys had used either finger marks or immunization cards to identify children who had been vaccinated during the preceding round of supplementary measles immunization activities. Where cards had been used (in four of the surveys), data were available for a median of 69% (range: 45-100%) of the eligible children. Finger marks were used in seven surveys. In these surveys, data were available for a median of 48% (range: 3-100%) of the eligible children.
In nine (69%) of the 13 surveys reported, the supplementary immunization-coverage estimate based on the survey results was lower than that based on the corresponding, routinely collected administrative data. Based on finger marks, immunization cards or the recall of members of the study households, the survey results indicated a median coverage of 93% (range: 81-98%). However, the corresponding value based only on finger marks or cards was only 62% (range: 3-91%).
Eight (62%) of the 13 surveys reported had measured the achievement of full immunization with all routine vaccines by 11 months of age. In these eight surveys, an estimated median of 78% (range: 64-95%) of children aged 12-23 months were found to be fully immunized. The reports on six (75%) of these eight surveys described the method that had been used to document routine child immunization. The estimated median full routine vaccination coverage by 11 months of age -based only on the data held on immunization cards -was 69% (range: 63-77%). One country measured the achievement of full routine measles immunization by 23 months of age -as assessed in children aged 24-35 months. Here, routine vaccination coverage by 23 months of age -based only on the data held on immunization cards -was 86%. Seven of nine surveys that assessed routine immunization services also included coverage of tetanus vaccinations in women of childbearing age. Table 1 shows selected results from our review. Only four of the reports we investigated provided details of the minimum sample size required per sampling area, as derived from sample size calculations. Most sample sizes were variations of a traditional design that involves 30 clusters and seven children per cluster. Although six of the reports provided information about the sampling frame, the originating year of the sampling frame and the methods used to adjust the data to the estimated population size at the time of the survey were not discussed in any of these reports. According to the reports we investigated, only one of the surveys reported any training at subnational level. Smaller size countries may not need training at subnational level (i.e. all survey teams are trained at national level); however, this should be discussed as part of survey protocols and reports. None of the reports included any information about the quality or consistency of supervision at subnational level. Although WHO recommends that inexperienced interviewers and supervisors be trained for at least 2-3 days, 6 five (50%) of the surveys for which the relevant information was available reportedly included training for no more than 2 days. Another concern was the lack of detail in the reports on how the initial or index study household and subsequent households were selected within the study clusters. For example, it was generally unclear how random starting points were selected, how and if clusters were subdivided, how the population size per cluster subdivision was kept similar, how many household lists had to be obtained or generated and how such lists were obtained or generated. At least two of the surveys that we investigated had spun a bottle or pen, then selected households among 10 houses according to the direction indicated. This practice is not consistent with WHO guidelines.
Seven of the surveys we investigated had used finger marks to verify vaccination. This practice only makes sense if the survey takes place soon after the vaccination round, since finger marks soon fade. Only four surveys (31%) had been conducted within 1 month of vaccination and eight (62%) failed to report survey dates.
Of the 13 reports, six (46%) provided information that allowed the number of field teams per supervisor to be calculated. Of those, three (50%) two (33%) and one (17%) described the use of one, two and four teams per supervisor, respectively. There were two interviewers per team in all but one survey, which used four interviewers per team. Just seven (54%) of the 13 reports we investigated provided details about the selection of clusters; six described cluster selection that was proportional to population size and one described cluster selection by square root allocation. Although nine (69%) of the 13 reports provided information about the software used for data management and analysis, only two provided more specific information about the analysis process, including the adjustment for cluster design and weighting.

Discussion
Our review showed that coverage surveys have become a regular component of supplementary measles immunization activities in the countries studied. Most of the reviewed surveys included some investigation of routine immunization services. All of them used some variation of the two-stage cluster survey design recommended by WHO in guidelines published in 2005. 6 However, sample size calculations were provided in less than one third of the survey reports we investigated. We found the potential for sampling bias related to uncertainty about the adequate updating of population data for sampling frames, the selection of index households in study clusters and the measures used to ensure Two surveys selected an index household among 10 houses counted in the selected direction; one reported that the first household was selected in the direction in which a spun pen pointed Other 6 (46) Six survey reports described use of a random starting point in a clusterwithout mentioning use of pen or bottle. One report described how clusters with large numbers of houses were subdivided before one subdivision was used at random. Two surveys reportedly selected an index household randomly from a household list. One of these two surveys had generated a household list where none was already available Not given in report 2 (15) None Method of selecting households within cluster after index household Next household 4 (31) None Random selection 4 (31) One survey reportedly used systematic random sampling and two surveys reportedly used a sampling interval Not given in report 5  the random selection of subsequent study households. The recommended generation and use of household lists in clusters, 1 was only described in a small number of the survey reports and, even then, more detail about how such lists -and how many -were obtained or generated would have been useful. The timing of surveys and the methods of documenting the immunizations were other areas of concern. Delays in conducting coverage surveys might have allowed finger marks to fade, resulting in an increased dependence on recall and an increased risk of information bias. A culture of providing, using, updating and retaining high-quality home-based records, such as immunization cards, could ensure that coverage estimates -for both routine and supplementary immunizations -are based more on documented doses and less on recall. Another potential problem was the frequent lack of independence between those who conducted the coverage surveys and those who were responsible for implementing the immunizations. Training and supervision of the field teams often appeared inadequate. WHO recommends that a supervisor should not be responsible for more than two interview teams. 6 Although most of the surveys we investigated used no more than two teams per supervisor, the corresponding survey reports provided too little detail about the length and contents of training for the teams -including the capacity needed at subnational level. None of the surveys described supervision that covered all levels of the immunization programme -i.e. from national level to cluster level. Ideally, any data analysis should follow a detailed plan that forms part of the survey protocol. Survey reports should include information not only about the software used for data analysis but also on how the software was used to adjust for cluster design, and whether weighting was used.
Weighting becomes particularly important if all the eligible children in a fixed number of households per cluster are investigated, rather than a fixed number of children per cluster. 6 Survey reports should also include a standard set of tables and figures representing a fixed minimum number of data elements, and full data sets should be available to other researchers for additional analyses, such as between-country comparisons. Finally, as recommended by WHO guidelines, 6 ethical approval for coverage surveys should be obtained in each country.
Our review has some limitations. A retrospective assessment cannot determine whether, during a reported survey, certain methods were applied but not reported, reported but not applied or neither reported or applied. We made no attempt to seek clarification or further information from the authors of the reports we studied. We focused on surveys conducted after rounds of supplementary immunization activities and ignored other types of coverage survey -e.g. Expanded Programme on Immunization surveys of routine immunization activities or evaluations of outbreak response immunization activities. Our main limitation was the lack of detail or absence of relevant information in many of the reports. This often limited our ability to assess the design and implementation of a survey fully. For example, it was often impossible to determine whether the recommended standards for probability sampling 1,6 had been met.
The participants at the consultative meeting in Zimbabwe that followed this review made recommendations on coverage survey methods. They called for further advocacy around appropriate advanced planning of coverage surveys -concurrent with campaign planning itself -and for the development of a standard protocol and report templates with the required level of detail. They recommended that full data sets from coverage surveys be made available to other researchers -to allow further analyses and comparisons across countries. A standardized technical review for protocols and reports, the development of criteria for selecting independent groups to conduct coverage surveys and further assessment of the appropriate duration and content of field training and supervision were also recommended.
WHO's guidelines on survey design, conduct and reporting are currently being updated, with the main aim of improving the quality of all vaccination coverage surveys -including those conducted after supplementary immunization activities. Organizations that are active in immunization programmes should ensure adherence to the current guidelines now and to the updated guidelines as soon as they become available. High-quality cluster surveys are considered a routine component in monitoring the progress of immunization systems, within the context of the Global Vaccine Action Plan. 7 All supplementary measles or measles-rubella immunization activities should include a plan and budget for a coverage survey, as part of a general monitoring and evaluation plan. 8 Although WHO will provide technical assistance -via its own staff or the engagement of external consultants -and sometimes financial support, it leaves the decision on whether or not to conduct a coverage survey and the organization of any such survey to its Member States.
In addition to WHO's guidelines, implementation of the recommendations from this review and the consultative meeting that followed it should help to ensure that the results of immunization coverage surveys are accurate, reliable and useful for programme improvement. ■

Comments
Method used to verify immunization Finger marks 7 (54) None Immunization cards 4 (31) None Not given in report 2 (15) None Confidence intervals provided 7 (54) None SIA: supplementary immunization activities; WCBA: women of childbearing age. a Training length varied from 1 day to 5 days (median: 2). The relevant information was missing from the reports of three surveys. None of the reports gave the criteria used for deciding length of training or the training agendas to be followed.

(. . .continued)
Measles vaccination coverage in eastern and southern Africa Reinhard Kaiser et al.

Research
Measles vaccination coverage in eastern and southern Africa Reinhard Kaiser et al.