Compliance of WHO and UNICEF estimates of national immunization coverage (WUENIC) with Guidelines for Accurate and Transparent Health Estimates Reporting (GATHER) criteria

Background: The objective of the study was to assess compliance of the WHO and UNICEF estimates of national immunization coverage (WUENIC) against the 18 criteria of the Guidelines for Accurate and Transparent Health Estimates Reporting (GATHER) that define and promote good practice in reporting of global health estimates. Methods: We conducted a desk review of the WUENIC estimation and reporting process vis-à-vis each of the 18 GATHER criteria to complete a self-assessment of compliance with GATHER. Results: Overall, WUENIC estimates are fully compliant with 17 of the GATHER criteria and partially compliant with one criterion—criterion 11, which is related to candidate model evaluation and final model selection. Conclusion: The GATHER criteria provide a useful framework for documenting WUENIC’s compliance with contemporary reporting requirements. Given the role of vaccination coverage estimates in global monitoring and guiding disease control efforts, WHO and UNICEF strive to produce and publish robust estimates of vaccination coverage through a transparent process that emphasizes country involvement.


Introduction
In 2016, an expert working group, convened by the World Health Organization (WHO) to define and promote good practice in reporting of global health estimates, published the Guidelines for Accurate and Transparent Health Estimates Reporting (GATHER) 1 . GATHER consists of a checklist of 18 essential items for reporting health estimates (Table 1) and reflects a set of guidelines or best reporting practices for studies that calculate health estimates for multiple populations by defining minimum reporting requirements. Additional details are available on the GATHER website. The GATHER criteria aim to define and promote good practice in reporting health estimates. Compliance with GATHER is not an indicator of quality of the health estimates; rather, compliance implies that decision makers and researchers have the key pieces of information necessary for making informed judgements about the quality of the estimates. Statements of GATHER compliance have accompanied publications of global health estimates produced by the Joint United Nations Programme on HIV/AIDS (UNAIDS) 2 and the Malaria Atlas Project 3 . In addition, compliance has also been noted in peer-reviewed papers 4 .
Since 2001, the World Health Organization (WHO) and United Nations Children's Fund (UNICEF) have jointly produced and published annual estimates of national vaccination coverage for all Member States of the World Health Assembly 5 and the State of Palestine. In addition to serving as a measure of an immunization programme's ability to effectively deliver immunization services and protect against vaccine-preventable diseases, vaccination coverage for the recommended infant immunization series has been used to monitor the progress of global and regional immunization initiatives, including the Universal Childhood Immunization (UCI) by 1990 6 , the Decade of Vaccines and its Global Vaccine Action Plan 7 and the Immunization Agenda 2030 8 , endorsed by the World Health Assembly in August 2020. In addition, vaccination coverage is frequently used as an indicator of a health system's ability to ensure access to primary care services and is currently included as one of many indicators in the Sustainable Development Goals 9 .
During the 1980s and 1990s, WHO and UNICEF collected reports on vaccination coverage from Member States through separate, annual exercises. Country-specific coverage levelsbased solely on the data reports from Member States-were published and used to produce regional and global estimates. In most instances, national coverage estimates were an aggregation of 1) vaccinations given in the public sector and 2) national-level population estimates or projections from the most recent census.
An internal review of global and regional coverage trends conducted in 1998 noted an apparent decline in global vaccination coverage based on national reports. Follow-up revealed that coverage decreases were not real but resulted from changes in the data sources being used by three countries, each with large annual birth cohorts. Specifically, coverage estimates from household surveys, rather than national administrative reports, were reported by India, Indonesia, and Bangladesh. Regional and global coverage appeared to decline because the survey reports consistently suggested meaningfully lower coverage as compared to administrative reports that had been previously used as the basis for reporting coverage.
Further examination of the consistency and reliability of national immunization coverage estimates reported to WHO from 1991 through 1996 10 found that 25% of expected reports were missing; 20% of reported data points reflected year-to-year changes greater than 10 percentage points; roughly 15% of reports had a meaningful inconsistency between reported coverage for the third dose of diphtheria-tetanus-pertussis containing vaccine and the third dose of polio containing vaccine (vaccine doses that are generally recommended at the same age); and 17% of reports had a meaningful difference between coverage levels reported to WHO and coverage levels from other sources, namely population-based household surveys. The results were presented to WHO's Strategic Advisory Group of Experts on Immunization (SAGE) 11 , which recommended that WHO invest resources to improve the completeness, accuracy and precision of coverage estimates. Subsequently, methods were developed, data were identified and acquired, and an initial time series of WUENIC coverage estimates was produced in July 2001 for the period from 1980 through 1999. WUENIC has continued evolving since its inception 20 years ago.

Methods
Using the GATHER checklist as a reference, we conducted a desk review of all relevant documents and datasets maintained and produced by the WHO and UNICEF working group responsible for WUENIC to complete a self-assessment of compliance with the GATHER criteria. In the absence of structured guidance for conducting such an assessment of GATHER compliance, each author reviewed available material and made a self-assessment against three qualitative compliance categories (fully compliant, partially compliant, not compliant). Final determinations were exchanged within the group, disagreements discussed, and final revisions to methodological and process descriptions made and final compliance determined by group consensus.

Results
This report summarizes WUENIC compliance with each of the 18 GATHER criteria. Detailed supporting descriptions are provided elsewhere 12 . Overall, WUENIC estimates are fully compliant with 17 of the GATHER criteria and partially compliant with one criterion-criterion 11, which is related to candidate model evaluation and final model selection. GATHER is focused on estimates of health status and determinants rather than service coverage or health systems indicators 13 and inclined towards mathematical or statistical model-based approaches that synthesize data from multiple sources. Nonetheless, the GATHER criteria provide a useful framework for documenting WUENIC's compliance with contemporary reporting requirements.

WUENIC compliance with GATHER explained
The GATHER compliance checklist items and rationale for WUENIC compliance are summarized briefly below and are comprehensively described elsewhere 12 . GATHER 3: Describe how the data were identified and accessed: Immunization system performance reports, including vaccination coverage reports from national authorities, and survey data from published and grey literature are reviewed annually for 13 antigens (Table A1 in 12). Official coverage reflects a national authority's assessment of coverage based on any combination of administrative coverage, survey-based estimates or other data sources or adjustments. Administrative coverage data are derived from routine administrative health service delivery reports. Vaccination coverage reports and other immunization system data are most often reported to WHO and UNICEF using the WHO and UNICEF Joint Reporting Form on Immunization (JRF). Survey-based coverage estimates are obtained from reports submitted by national authorities and from searches of national and survey agency websites (e.g., DHS; MICS).
Age-specific population estimates, which are utilized for producing global and regional averages and are an input to WUENIC's measure of uncertainty, are obtained from the United Nations Population Division (UNPD) World Population Prospects site. Finally, WHO and UNICEF obtain additional otherwise unreported information, such as changes in immunization policies and insight into the functioning of the immunization system and quality of reported data, through consultation with regional and national immunization and monitoring experts. Survey-based, vaccination coverage data must originate from finalized reports that include a sufficient description of the methods, including sampling methodology, in order to be included. Crude, but not valid, vaccination coverage estimates are considered. Survey results are excluded if they are not nationally representative; are derived from a survey of small size (i.e., <300 observations) or do not report sample size; or are not for single year cohorts (e.g., 12-23 months, 24-35 months).
GATHER 5: Provide information on all included data sources and their main characteristics: Countries report immunization system performance data annually using the WHO and UNICEF Joint Reporting Form on Immunization (JRF). These data include administrative and official vaccination coverage data and national vaccination schedule information. These data are complemented by population survey-based coverage data and by population data from the UNPD.
Official coverage reflects a national authority's assessment of their most likely coverage based on any combination of administrative coverage, survey-based estimates or other data sources or adjustments. Administrative coverage data are derived from routine administrative health service delivery reports aggregated across service providers in the country. Survey coverage reflects an assessment of coverage during a specified time period from a defined sample population that is independent of the biases in numerator and denominator data described below. National vaccination schedules provide standardized information regarding the vaccines and number of doses of vaccine that a population should receive and the recommended ages, minimum ages and intervals between doses. Vaccination schedules also provide information on whether a vaccine is recommended for an entire population or targeted to specific risk groups. In addition, population data from UNPD provide a consistent reference population time series for computing global and regional average coverage levels.

GATHER 6: Identify and describe any categories of input data that have potentially important biases:
Both the reported number of administered vaccine doses and the reported target population used to compute administrative coverage may be biased, which may lead to biased coverage estimates. Biases affecting numerator data include, but are not limited to, incomplete or untimely reporting and data recording or reporting errors. Biases affecting denominator data include, but are not limited to, population estimates based on dated or poorly implemented census estimates or poorly implemented census projections. The methods and data sources used to produce official estimates are not always described or available, and methods may change over time or without documentation or notice. When a country bases its official coverage on administrative coverage, biases in the administrative coverage carry over to official coverage. Survey-based coverage estimates are subject to sampling and non-sampling error. Official country reported coverage estimates time series", "4.2 Administrative data time series", and "4.7 Coverage Survey Data" with abstracted results from population-based survey reports.
GATHER 9: Provide a conceptual overview of the data analysis method: WHO and UNICEF distinguish between when data reported by national authorities accurately reflects immunization system performance and when the data are likely compromised or misleading. Unless challenged, the nationally reported estimate constitutes the WUENIC. If reported data are supported by independent survey results, then WUENIC will reflect the reported data. When reported data are challenged by survey results, WUENIC may diverge from the reported data.
WUENIC are based on a country-by-country, vaccine-by-vaccine, year-by-year assessment of available data. The estimates are not the product of a formal modelling exercise; no statistical or mathematical models are used (with one exception related to estimates for the first dose of diphtheria-tetanus-pertussis containing vaccine to ensure that estimated coverage for the third dose does not exceed that for the first dose). Data from different sources are not averaged to arrive at a final estimate, and the approach does not borrow information from data rich countries to fill in gaps among data poor countries. A detailed description is available in Burton et al. 14 and Burton et al. 15 .

GATHER 10: Provide a detailed description of all steps in the analysis:
Briefly, the aim of the WUENIC process is to produce a time series of annual vaccination coverage estimates for selected vaccines given data reported by national authorities and given available nationally representative survey data and any other available information. The estimation process is comprised of four general steps: 1) accepting, adjusting or ignoring available reported and survey coverage data; 2) generating coverage estimates for years when both reported and survey data are available (so-called "anchor" years); 3) generating coverage estimates for non-anchor years in which there is no survey data; and 4) examining estimates for year-to-year and vaccine-to-vaccine consistency and reconciling any discrepancies. Stepwise visual descriptions of the WUENIC estimation process are provided in Brown and colleagues 12 . Inaccuracies may exist within the reported numerator and denominator of administrative or official coverage estimates. Vaccination coverage estimates from population-based household surveys may be limited by the quality of the planning and implementation of the field work and the availability of documented evidence of vaccination history in home-based or acility-based records. Additionally, coverage indicators from survey reports do not always conform to standard definitions. WUENIC is also constrained by the underlying rules and heuristics utilized in the estimation approach.

Conclusions
Boerma and colleagues 22 described the role of health estimates such as WUENIC in monitoring progress towards global and regional goals and targets and guiding resource allocation while emphasizing the importance of transparency in the production of these health statistics. This report summarizes compliance of the WHO and UNICEF estimates of national immunization coverage with each of the 18 GATHER criteria for population health estimates. Although the GATHER guidelines were not originally designed for health service delivery indicators, such as vaccination coverage, they provide a useful framework for documenting WUENIC's compliance with contemporary reporting requirements. Given the role of vaccination coverage estimates in monitoring immunization system performance, guiding disease control efforts, and informing assessments of high risk areas that might require additional resources, WHO and UNICEF strive to produce and publish robust estimates of vaccination coverage through a transparent process that emphasizes country involvement. We acknowledge the strategic advantages of transparency and provide this summary of WUENIC compliance with GATHER in commitment to promoting openness so that health estimates such as WUENIC vaccination coverage estimates are translated in a meaningful way to promote global vaccination goals.

Amalie Dyda
School of Public Health, Faculty of Medicine, The University of Queensland, Herston, Qld, Australia This is an extremely well written paper and an important addition to the literature. It outlines a comparison of the WHO and UNICEF estimates of national immunization coverage (WUENIC) against the Guidelines for Accurate and Transparent Health Estimates Reporting (GATHER) to assess agreement between the two. This is important given the importance of, but many challenges of, reporting global annual national vaccination coverage.
Of serious note is the possible error/bias in how each coverage estimate is collected. I think the reason for the importance of using GATHER has been highlighted in the introduction ("Compliance with GATHER is not an indicator of quality of the health estimates; rather, compliance implies that decision makers and researchers have the key pieces of information necessary for making informed judgements about the quality of the estimates."), and bias addressed well in Gather 6.
I have provided a few specific comments for consideration below:

Methods:
"In the absence of structured guidance for conducting such an assessment of GATHER compliance, each author reviewed available material and made a self-assessment against three qualitative compliance categories (fully compliant, partially compliant, not compliant)." I believe this authorship group has the required experience and knowledge to provide this expert assessment. I think this would be strengthened by either adding a sentence to highlight this, or providing some more detailed information about what authors took into consideration in their reviews.
Results: "GATHER 10: Provide a detailed description of all steps in the analysis:" This description of the steps was quite brief, but given the context of what is needed in this comparison it is sufficient, especially given more detailed methods are referenced.

Conclusion:
"Boerma and colleagues described the role of health estimates such as WUENIC in monitoring progress towards global and regional goals and targets and guiding resource allocation while emphasizing the importance of transparency in the production of these health statistics." Perhaps better placed in the introduction.

Is the work clearly and accurately presented and does it cite the current literature? Yes
Is the study design appropriate and is the work technically sound? Yes

Are sufficient details of methods and analysis provided to allow replication by others? Partly
If applicable, is the statistical analysis and its interpretation appropriate? Not applicable