Coverage and timeliness of vaccination and the validity of routine estimates: Insights from a vaccine registry in Kenya

Highlights • A high coverage is essential if the full benefits of vaccines are to be enjoyed.• A vaccine registry can help quantify the errors in coverage estimates from surveys.• Vaccination coverage obtained using a survey approach overestimates coverage by 2%.• Survey and administrative methods underestimate fully immunised children by ≥10%.• Non-hospital delivery and stock-outs were associated with failure to vaccinate.


Introduction
Vaccines are the most powerful and cost effective interventions in public health; they prevent $3 million childhood death annually, foster health equity and yield a US$44 return on investment for every US$1 spent [1][2][3]. However, the impact of vaccination is highly dependent on coverage [4]. programmes both nationally and globally [5][6][7]. The benchmark of vaccination programme performance is coverage of the 3rd dose of a vaccine containing diphtheria-tetanus-pertussis (DTP3) at 12 months of age [8].
Following introduction of the Expanded Program on Immunization in 1974, global coverage of DTP3 rose to 21% by 1980. The WHO programme 'Universal Child Immunization by 1990' advanced this to 75% in 1990 but it remained stagnant for a further 15 years. Following the drive by WHO, UNICEF, Gavi, The Vaccine Alliance and other partners, global DTP3 coverage increased to 85% in 2015 but has stagnated again. In addition, 1-in-7 children remain unvaccinated and considerable geographical variation in coverage exists at both national and sub-national levels [9]. These factors have driven a focus on equitable access to vaccines [6,10].
Ideally, coverage should be measured continuously using electronic immunisation registry that records vaccinations received by all individuals in birth cohorts, or by the administrative method (vaccine doses given) [5,11]. Electronic vaccine registries are not routinely used in the low and middle income countries. So, the two principal methods supporting national and global coverage estimates are: administrative coverage and random cluster surveys. Administrative coverage is calculated using aggregate reported data (tallied from monthly reporting forms) on the number of vaccine doses administered to children in the target population in a given period of time and target population estimates. Because population denominators are, on average, 5 years out of date these frequently produce estimates in excess of 100% [12][13][14][15]. Demographic and Health Survey (DHS), Multiple Indicator Cluster Survey (MICS) and EPI Cluster Surveys produce more reliable estimates because they are not dependent on census data [16][17][18]. However, survey methods are susceptible to selection, recall and coverage biases; for example, they fail to capture unregistered, migrant populations [5,19].
Coverage is typically estimated for children aged 12-23 months and referenced to their vaccination status at 12 months of age. This focuses on survivors of infancy and, if vaccination is associated with survival, there is scope for survivor bias. It also discards information on timeliness of vaccination yet it is possible that timeliness is a more sensitive indicator of health equity than a static coverage percentage [20,21]. Finally, in a modern vaccine programme with a wide range of antigens, focus on delivery of individual vaccines does not take account of correlation between coverage of different vaccines and estimates the proportion of children who are fully immunised with difficulty.
Beyond being a performance metric, coverage is also a proxy measure of population immunity with relevance to disease control. Since coverage is closely related to disease incidence, monitoring coverage can identify likely gaps in immunity before increases in disease incidence are observed [22]. For diseases of infancy like invasive infections caused by Streptococcus pneumoniae, coverage at 12 months of age is a poor estimate of protection during infancy and alternative measures, incorporating the timeliness of vaccination, are likely to be more useful.
Here we use a vaccine registry in Kenya [23] to quantify errors in routine coverage methods, to estimate the magnitude of survivor bias, contrast coverage with timeliness and use both measures to estimate population immunity. Finally, we examine the risk factors for delayed immunisation and illustrate the breadth of inequality in both coverage and timeliness across different birth cohorts and different locations.

Methods
The study is an analysis of all vaccinations recorded in an electronic registry, established within the Kilifi Health and Demographic Surveillance System (KHDSS) in Kenya, between 2010 and 2016.

Study setting and population
The KHDSS which is located on the Indian Ocean had a census in 2000 to define the resident population and following that, all subsequent births, deaths and migration events are monitored by enumeration visits by fieldworkers to every participating household at approximately 4-monthly intervals [24]. This population registry has informed assessments of vaccine effectiveness and disease incidence among children and older residents of the Kilifi County area covered by the KHDSS [25][26][27].
The Kilifi Vaccine Monitoring System was established in 2009 in all 21 vaccine clinics in and around the KHDSS area. As the vaccine service expanded, we incorporated a further 15 clinics between 2011 and 2016 [23]. As previously described, children who are residents of KHDSS, attending clinics for vaccination, are matched electronically to the KHDSS population register and all the vaccinations they received are recorded at the clinic in real time. This also provides an avenue for updating the KHDSS population register for the new-borns [23,24]. Data are entered using laptop computers at the vaccine clinics and are synchronised to a bespoke MySQL v5.6.19 relational database at the main facility through weekly hard-copy transfers on laptops. Data management procedures are described elsewhere [23,24].

Statistical analysis
Demographic event data (births, deaths and migrations) were combined with vaccination data from the vaccine clinics to create individual life histories of a rolling cohort of children. The data were analysed using survival analysis tools and presented as inverse survival curves with age as analysis time. We focused on two time points for estimating coverage: (a) Vaccine coverage, also referred to as 'Up-to-date vaccination coverage' was defined as the proportion of children vaccinated by their first birthday; (b) Ageappropriate vaccination was defined as the proportion of children vaccinated within 4 weeks of the age of vaccine eligibility in the Kenya routine childhood immunisation schedule (see Table 1) [28]. According to this programme, a Fully Immunized Child was defined as one who had received a dose of Bacille Calmette-Guérin vaccine (BCG), a 3-dose course of each of Pentavalent vaccine (targeting Diphtheria, Pertussis and Tetanus [DPT], Hepatitis B and Haemophilus influenzae b), Oral Polio Vaccine (OPV) and Pneumococcal Conjugate Vaccine (PCV), and a first dose of measles-containing vaccine (MCV1) before his or her first birthday [29].
We sampled the event data in different ways to simulate different field approaches: (i) Birth cohort analyses. For these analyses, the denominator was the number of children who survived to the age of vaccine eligibility; the numerator was the number of these children who were then vaccinated within four weeks (age appropriate vaccination) or before their first birthday (vaccine coverage), regardless of whether they survived to these age milestones. (ii) Cross-sectional coverage surveys. We sampled the vaccine coverage status of all resident children aged 12-23 months on 1st July each year. Although we used a total population sample, this still mimics the approach of a cluster sample survey. To ensure the populations in the survey analyses were linked to those in birth cohort analyses, we offset the annual birth cohorts by six months (Fig. 1A); for example the 2010-11 birth cohort consisted of all children born between 1st July of 2010 to 30th June of 2011 and the crosssectional survey of coverage of 12-23 month-olds, corresponding to this cohort, was undertaken on 1st July 2012.
We estimated the median age at vaccination from inverse survival curves for each vaccination type. We estimated the timeliness of vaccination as the proportion of all children vaccinated by the age of 12 months who had received their vaccine within 4 weeks of becoming age-eligible. As an indicator of population immunity, we also estimated mean vaccine coverage among eligible infants as the area under the inverse survival curves (AUC) for vaccination between the age-eligible thresholds and 12 months of age ( Fig. 1B) [30]. Age-eligible thresholds for Penta 1, 2 and 3 were 6, 10 and 14 weeks, respectively. Similar thresholds were applied to OPV and PCV. The threshold for MCV1 was 36 weeks. We also explored health equity in timeliness and final coverage over time and place by plotting inverse survival curves for BCG, Penta3 (DPT3) and MCV1 for each birth cohort and in each of the 15 administrative locations in KHDSS.
We used Cox regression models to estimate the risk factors for vaccination. The risk factors examined were drawn from variables available within the KHDSS, including place of birth, sex, birth  order, maternal age and distance of the child's home to the nearest vaccine clinic. To understand the differences between populationbased and administrative coverage estimates we compared the KHDSS survey estimates for the 12-23 month old survey population in 2014 against the coverage estimates from Kilifi County in the 2014 DHS survey [31] and routinely reported administrative estimates from Kilifi County in 2014 (Kilifi County Reports). The administrative estimates are obtained from monthly aggregates of daily tally sheets filled out in each vaccine clinic and from monthly review of the paper vaccine registers/books that are then aggregated at the level of each sub-County within Kilifi before collation to obtain overall County estimates. All analyses presented here are confined to children born into, and continuously resident in, the KHDSS. Coverage are presented with 95% confidence intervals.
All statistical analyses were undertaken in Stata/IC TM 13.1 (Stata-Corp, College Station, Texas, USA).

Ethics statement
This study was approved by the Kenya Medical Research Institute's (KEMRI) Scientific and Ethics Review Unit (SSC 1433).

Vaccination coverage by birth cohort and survey
For each vaccine, coverage at age 12 months increased steadily with each advancing birth cohort but then declined in 2015-16, the last year ( Table 2). The timeliness of vaccination also improved over time and then declined, though the patterns were not as consistent from vaccine to vaccine. The greatest disparity between timely coverage (4 weeks after the child became age-eligible) and vaccine coverage (at 12 months) was seen for measles vaccine, which is also the vaccine that is scheduled closest to 12 months of age. In the last 3 years of the study the estimates for coverage of the Fully Immunised Child were within 1% of the coverage estimates for measles, suggesting that the great majority of children presenting for MCV1 are already up-to-date on all other vaccines.
The estimate of vaccine coverage derived from sampling children aged 12-23 months was greater than that derived from the birth cohort analysis in all birth cohorts and for all vaccines, except for measles vaccine in the 2015-16 birth cohort where the two coverage estimates were the same (77.5%, Table 2). The mean difference between survey and birth cohort coverage estimates at 12 months of age were 2.6%, 2.2%, 2.2%, 2.2%, 1.4% and 1.6% for BCG, Penta3, OPV3, PCV3, MCV1 and the fully immunised child.
The patterns of coverage of other vaccine doses (OPV1, OPV2, Penta1, Penta2, PCV1, PCV2) over time, and the disparity between coverage estimation methods were similar (Supplementary Table 1).

Comparing published coverage estimates
Vaccination coverage estimates, referenced to 2014, differed by source (Table 3). KHDSS survey coverage estimates appeared higher than coverage estimates for Kilifi County in the 2014 Kenya DHS report for all vaccines except BCG. However, the confidence intervals around the coverage estimates overlapped for all of the vaccines. Compared to the KHDSS survey estimates, administrative estimates for Kilifi County in 2014 were very similar for all vaccines except for BCG, where the administrative coverage estimate was 5% higher than the KHDSS survey estimate. Measures of the proportion of children who were fully immunised were considerably lower using both the DHS survey (71.5%) and the administrative method (67.2%) than using the survey approach in the KHDSS data (81.4%).

Timeliness of vaccination
The proportion of children vaccinated by 12 months of age who had received the vaccine within 4 weeks of becoming eligible ('Timely vaccination') is presented in Table 2 and Supplementary Table 1. The proportion of vaccinated infants receiving a timely vaccination increased over time for BCG, Penta3 and PCV3 but declined for OPV3 and MCV1.
Predictably, the mean coverage among eligible infants i.e. the area under the curve (AUC) was always lower than the final vaccine coverage at 12 months. The difference between these two estimates was 1.1% for BCG, 7.9% for OPV3, 7.2% for PCV3 and Penta 3 and 24.2% for MCV1. These differences are reflected in the spread of the inverse survival curves in Fig. 3.
Apart from 2010 to 11, which was the year of introduction of PCV3, delivery of PCV3 and Penta3 was timely, as was BCG. However, there is marked variation in the timeliness of OPV3 by birth cohort and timeliness of MCV1 is consistently poor across all birth cohorts.

Inequality in timeliness and coverage by location
Variation in the inverse survival curves for vaccination with BCG, Penta3 and MCV1 by administrative location is illustrated in Fig. 4. The dispersion of the curves is greatest for MCV1 (Fig. 4E) and least for BCG. For BCG, the AUCs varied from 86.3% to 93.8%; for Penta3, AUCs varied from 68.1% to 88.7%; and for MCV1 AUCs varied from 52.9% to 59.8%. Some inequality in timeliness of Penta3 vaccination is apparent in Fig. 4C and D. The take-off in the curve for MCV1, indicating the first age at vaccination, varied from location to location, with some locations starting at 33 weeks and others not beginning until 36 weeks (Fig. 4E and F). In these figures, age-appropriate vaccination is estimated by the y-axis value as each curve traverses the second vertical line (4 weeks after vaccination). These ranges are 68-82% (BCG), 58-78% (Penta3) and 19-49% (MCV1). 'Up-to-date coverage' at age 12 months varied within the ranges 84-92% (BCG), 81-91% (Penta3) and 70-82% (MCV1).

Predictors of coverage and timeliness
The univariate and multivariable hazard ratios for risk factors for timely coverage are shown in Table 4. After accounting for the secular trend in improved coverage (BCG and Penta3), the factors associated with more timely uptake are delivery in a health facility and increasing birth order. Factors associated with delayed coverage are vaccine stock outs, increasing maternal age and increasing distance of the home from the vaccine clinic. The risk and beneficial factors were broadly similar for the three vaccines except that for measles maternal age and birth order were not associated with vaccine timeliness and the impact of stock outs was less remarkable. The results for PCV3 and OPV3 (Table S2) reflect those of Penta3.

Discussion
By studying an entire population sample in a Vaccine registry, we have been able to characterise the accuracy of different methods and metrics of vaccination coverage in the setting of a vaccination programme in an LMIC. The principal findings of this analysis are: (i) vaccination coverage estimates using a survey approach in the second year of life overestimate coverage by approximately 2%; (ii) compared to a total population survey in KHDSS, the clustersurvey based DHS approach in 2014 underestimated coverage of Penta3, OPV3 and PCV3 by 4-7% but the results for BCG and MCV1 were equivalent in both methods; (iii) against the same standard, the administrative methods overestimate BCG coverage by 5% but were otherwise accurate; (iv) DHS and administrative methods considerably underestimate the proportion of children who are fully immunised by 10-14%; (v) the timeliness of vaccination in this population exhibited variation both in time and location; (vi) Factors affecting coverage ('risk factors') were similar for each of the antigens -the largest associations with failure to vaccinate were a vaccine stock-out at the time of presentation and birth of the child outside of a hospital; (vii) Survey estimates of vaccination coverage are a poor guide to population immunity, even though they are frequently used to populate models of vaccine impact; for all vaccines except BCG, the AUC was substantially lower (7-24%) than the survey coverage.
Overall, coverage for most vaccines in Kilifi is good and has been improving throughout 2011-2016, a period of rapid introduction of new vaccines (PCV, Rotavirus, Inactivated Polio Vaccine and   the combined Measles-Rubella vaccine). Samples of our registry that mimic widely utilized household surveys (MICS or DHS) tend to overestimate coverage by approximately 2%. This is inevitable if infant survival is associated with the probability of being vaccinated -either as a function of vaccine protection or as a manifestation of the 'healthy vaccinee' effect [32]. 6790 children in that year so it is possible this disparity is the consequence of a sampling error. Available evidence from household coverage surveys show estimates based on recall alone exaggerate coverage, combined/mixed vaccine card and recall result in large errors in coverage estimates that trend in all directions while restricting coverage estimates to data from cards alone significantly underestimates coverage [34]. Despite the reliance on coverage estimates from household coverage surveys by national and global actors in the childhood immunisation field, it would appear the results should be interpreted with an abundance of caution [34].
Both sampling methods fail to identify mobile, transitory or unregistered populations: in our analysis we excluded children migrating into the area because we did not have a verifiable record of their prior vaccination history. Population movements are associated with vaccination coverage [35], and it is likely that survey methods overestimate vaccination coverage by failing to sample transitory sub-populations. Unfortunately, we could not quantify the impact of migration on coverage estimates.
The 2014 administrative coverage estimates were remarkably close to the survey estimates, except for BCG. The positive impression this gives of administrative methods must be tempered by the fact that the ranges for administrative coverage in individual vaccine clinics in Kilifi were 20-384%, 52-144%, 52-145% and 51-144% for BCG, OPV3, Penta3 and PCV3, respectively. It may be that simple models of population growth have made accurate predictions in this area as a whole, but that parents do not always take their children to the nearest clinic; or it may be that the administrative methods were fortuitously accurate in this relatively simple 1-year comparison.
Our study shows that the prevalence of FIC is considerably underestimated by DHS-type survey methods and is grossly underestimated by administrative methods. The similarity between the coverage for MCV1 and FIC may be attributable to clinic staff following extant policy and using any vaccine visit to catch-up on missed vaccinations. Immunisation program managers and others like Gavi have advanced the FIC as a better measure of programme performance at national level than single antigen metrics like DTP3 or MCV1, and this is now the hallmark analysis for equity of access to vaccines [29,36]. If the FIC which is better aligned with the full benefits of vaccination is to be adopted as a performance indicator, then we will need new or improved monitoring systems that are capable of linking the identity of the child across multiple records of vaccines given. Although this is theoretically possible in vaccination cards and in clinic log books, as these vaccine registry results show, it is more effectively accomplished electronically.
The inverse Kaplan Meier analyses were useful in assessing uptake and timeliness of vaccination [30,37]. Many studies have shown marked differences in age-appropriate and 'up-to-date coverage' by socioeconomic status [38,39]. In this relatively homogenous study setting, we also find significant location-specific and year-to-year variation in timeliness. Essentially, this means unless coverage is very high (>95%), it is very likely that the district or national coverage figures will conceal significant local variation. And as seen with MCV1, this becomes more relevant the later in life a vaccine is given. This could lead to substantial islands of spatiotemporal susceptibility which may ignite as, for example, unexpected measles outbreaks.
Factors associated with high coverage were younger maternal age, more previous children, and delivery in hospital. The strongest detrimental effect was vaccine stock outs, an operational challenge that immunisation programmes should be able to tackle. Place of birth and clinic visits post-delivery have been reported by us and others as important for vaccination [40,41]. Increased contact with healthcare services is believed to increase the likelihood of vaccination via an intermediate step of health education. An alternative explanation may be differential health-seeking behaviour related to 'healthy vaccinee' bias. If this is true, interventions to improve vaccination coverage by increasing deliveries at health facilities may only benefit those already likely accept vaccines. Socioeconomic factors also contribute to inequities in coverage and time to vaccination [38,39]. though these are hard to characterise reliably in a relatively homogenous rural population.
In any setting with delays in vaccination, 'up-to-date coverage' will be a biased measure of vaccination-induced population immunity [42] and this bias, always over-estimating protection, can be as great as 24% in our survey [42][43][44]. AUC is a better measure because it provides an estimate of mean vaccination coverage throughout the period of risk. However, true population immunity will be lower than the AUC because a small proportion of vaccinated children will not develop an appropriate antibody response either because of operational factors (ineffective administration, inactive vaccine, heat destroyed vaccine, etc.) or host factors (immunodeficiencies) and even those who do respond adequately will not become immune until 2 weeks after a primary vaccine or 1 week after a booster. Seroepidemiological surveys provide more accurate estimates of the population fraction protected by vaccination. They can identify at-risk groups via population immunity profiles and help inform strategies to increase or sustain population immunity such as revisions of the vaccination schedule or mass campaigns [45]. Whilst the AUC may be the best epidemiological approximation of population immunity, more reliable estimates of the vaccine induced immunity can only be obtained with serological surveys. In LMICs, vaccines account for a rapidly increasing fraction of health expenditure. Vaccine programmes are expensive but highly effective, yet their impact is critically dependent on vaccination coverage. This analysis of vaccine registry data, in a setting typical of much of sub-Saharan Africa, illustrates that methods and measures for estimating coverage are suboptimal; biased by survival to sampling date and by recall of vaccinations, incapable of revealing small area heterogeneity, unlinked and therefore unable to estimate the prevalence of the Fully Immunised Child, blind to the timeliness of vaccination and therefore to signal the gaps this produces in population immunity. Our study can be replicated across the LMICs in Africa and as part of the Comprehensive Health and Epidemiological Surveillance System (CHESS) proposed by the INDEPTH network [46].
The methodology of measuring vaccination coverage needs to be improved on and with modern electronic record systems and serological sampling, we have the opportunity to refine our tools and make better use of the tremendous power of vaccination.

Availability of data and material
The datasets analysed during the current study are not publicly available because this is the subject on ongoing work. However, reasonable requests can be submitted (to Ms. Marianne Munene, Research Governance Office, KEMRI-Wellcome Trust Research Programme MMunene@kemri-wellcome.org) for consideration by the Data and Governance Committee.