Methodological changes implemented over time to support accurate and timely COVID-19 vaccine coverage estimates: Ontario, Canada

Abstract The COVID-19 vaccination program implementation in Ontario, Canada has spanned multiple years and is ongoing. To meet the challenges of the program, Ontario developed and implemented a new electronic COVID-19 immunization registry, COVaxON, which captures individual-level data on all doses administered in the province enabling comprehensive coverage assessment. However, the need for ongoing COVID-19 vaccine coverage assessments over a multi-year vaccination program posed challenges necessitating methodological changes. This paper describes Ontario’s COVID-19 immunization registry, the methods implemented over time to allow for the ongoing assessment of vaccine coverage by age, and the impact of those methodological changes. Throughout the course of the vaccination program, four different methodological approaches were used to calculate age-specific coverage estimates using vaccination data (numerator) obtained from COVaxON. Age-specific numerators were initially calculated using age at time of first dose (method A), but were updated to the age at coverage assessment (method B). Database enhancements allowed for the exclusion of deceased individuals from the numerator (method C). Population data (denominator) was updated to 2022 projections from the 2021 national census following their availability (method D). The impact was most evident in older age groups where vaccine uptake was high. For example, coverage estimates for individuals aged 70–79 years of age for at least one dose decreased from 104.9 % (method B) to 95.0 % (method D). Thus, methodological changes improved estimates such that none exceeded 100 %. Ontario’s COVID-19 immunization registry has been transformational for vaccine program surveillance. The implementation of a single registry for COVID-19 vaccines was essential for comprehensive near real-time coverage assessment, and enabled new uses of the data to support additional components of vaccine program surveillance. The province is well positioned to build on what has been achieved as a result of the COVID-19 pandemic and expand the registry to other routine vaccination programs.


Introduction
The coronavirus disease 2019 (COVID-19) pandemic has had unprecedented global impacts. In Ontario, Canada (population 15.1 million), as of October 2022, COVID-19 has infected an estimated 1.4 million individuals and resulted in an estimate of over 14,000 deaths [1]. Vaccination has played a critical role in reducing the burden of COVID-19. Similar to other jurisdictions, the scale of the COVID-19 vaccination program in Ontario, Canada has been unlike other programs, with implementation spanning multiple years and still ongoing [2]. Initial program eligibility was highly phased with the prioritization of groups most at risk of severe outcomes following COVID-19 infection, and expanded over time with additional vaccine supply, formulations for pediatric age groups, and recommendations for booster doses [3]. Timely and high quality COVID-19 vaccination data has been essential for monitoring trends in vaccine uptake over the course of the program to determine if coverage goals were being met, as well as to identify sub-populations where vaccine uptake was suboptimal [4]. Vaccination data has also been used to support a range of immunization program surveillance activities that have been conducted in close to real time. These include passive vaccine safety surveillance, vaccine effectiveness, and other assessments of vaccine program impact [5][6][7][8][9][10][11].
For most other immunization programs in Ontario, information is incomplete and scattered across a variety of administrative datasets that are not available for routine, real-time program surveillance. The one exception is a provincial database that local public health departments use to assess student compliance with school-entry immunization requirements and to document the delivery of three school-based vaccine programs [12]. To meet the urgency and challenges of the COVID-19 vaccine program, Ontario developed and implemented a new immunization registry specific to COVID-19 vaccines. The registry, referred to as COVaxON, was implemented in December 2020 in time to capture the first doses of COVID-19 vaccine administered in the province. The implementation of an individual-level, real-time provincial COVID-19 immunization registry has been transformational in Ontario and has allowed for comprehensive, robust vaccine program surveillance. However, the ongoing work of regular COVID-19 vaccine coverage assessments over a multi-year vaccination program has posed new challenges in understanding what proportion of the population has been vaccinated at a given point in time. These challenges have necessitated methodological changes in order to improve the accuracy of estimates of vaccine uptake over time, and in particular estimates for vaccine uptake by age. The objectives of this paper are to describe Ontario's COVID-19 immunization registry, the methods implemented over time to allow for the ongoing assessment of vaccine coverage by age, and to illustrate the impact of those methodological changes.

COVID-19 vaccine program in Ontario
Ontario began administering COVID-19 vaccines to eligible individuals in December 2020. Due to initially limited vaccine supply, the COVID-19 vaccination program was implemented with a three-phased distribution plan targeting priority populations (e.g. long-term care home residents, health care workers) before expanding to the general population at first by descending age and then incorporating age and other risk factors (Fig. A1 [3]. In the very early stages of the vaccine program when older adults were prioritized for the primary series initiation, age eligibility (e.g. 75 years of age and older) for vaccination was satisfied if individuals were turning the relevant age by the end of the 2021 calendar year. As the vaccine program matured, age at time of vaccination was used to determine age eligibility. As of August 2022, all individuals aged 6 months and older were eligible to receive a primary series; individuals 12 years of age and older were eligible for first booster doses; and individuals aged 18 years and older were eligible for second booster doses [13]. The most recent program changes included expanding primary series eligibility to children 6 months to 5 years of age and second booster dose eligibility for individuals 18 to 59 years of age (July 2022). Provincial guidance has also included recommendations for individuals that received COVID-19 vaccine products that were not authorized for use in Canada [14].

COVaxON: The COVID-19 immunization registry
The Ontario Ministry of Health's (MOH) COVID-19 electronic provincial immunization registry, COVaxON, captures individuallevel data on all COVID-19 vaccines administered in Ontario. All immunizers, regardless of clinic setting or provider type, are required to document COVID-19 vaccines administered in COV-axON and with rare exceptions, vaccinations are captured at the point of service delivery (i.e. at the time the individual presents for vaccination). The system captures data elements related to vaccine dose administration (i.e. dose administration date, vaccine product name, and lot number); client demographic information (i.e date of birth, sex, and residential address); and the individual's Ontario health card number (HCN), a unique identifier assigned at the time of enrollment in Ontario's publicly-funded health insurance plan (OHIP), which allows for linkage across other health administrative data sources. In addition to its use as a data source for COVID-19 vaccine coverage assessment and other aspects of vaccine program surveillance, COVaxON is also used to manage vaccine inventory and to record the consent of individuals who are interested in participating in COVID-19 vaccine research studies. Individuals are also able to access and download their COVID-19 vaccination information as proof of immunization via a web-based platform.
Since its initial release, COVaxON has undergone various enhancements including functionality updates to allow retrospective entry of COVID-19 vaccine doses administered outside of Ontario, as well as integration with MOH's Registered Persons Database (RPDB). The RPDB is a population database that captures basic demographic information, including mortality, for all individuals with an assigned HCN [15]. COVaxON-RPDB integration enabled the population of client details from RPDB, such as health card number and mortality information, for client records in COV-axON. Unlike other registries where population databases feed into immunization registries to create clients following a new birth or immigration, information from RPDB is integrated at the time of client record creation in COVaxON. RPDB integration is therefore limited to individuals with an existing client vaccination record in COVaxON and individuals that are in RPDB but who did not present for vaccination are not captured in COVaxON [16][17][18][19]. For example, when a client record is created in COVaxON, RPDB integration enables population of health card number using personal information, such as name and address. Further, once a client record is created, information from RPDB feeds into existing client records in COVaxON on a daily basis to reflect any changes to the information (e.g, mortality information) captured in RPDB. COV-axON also feeds into the COVID-19 Case and Contact Management (CCM) system, the provincial reportable disease database, to provide client vaccination details (e.g. dose administration date and vaccine product name) for the purposes of case and contact, as well as outbreak, management.

Numerators for coverage calculations
COVaxON registry data are used to determine the numerator for coverage calculations. The complete history of doses administered since the start of the vaccination program on December 14, 2020 is extracted and processed for each routine (e.g. biweekly) coverage assessment. Processing logic for COVaxON data involves excluding invalid doses by applying product-specific minimum intervals and de-duplication [4]. In response to real-time vaccine program changes (e.g. expanding eligibility based on priority population and/or age at the time of vaccination, recommendations for booster doses), iterative changes were applied to methods to calculate an individual's age to obtain age-specific numerators for coverage estimates. At the beginning of the program when only a primary series with a 3 or 4 week interval was recommended (i.e. no recommendations for a booster dose), method A was used to calculate age ( Fig. 1). Age was determined using the date of birth and date of dose one administration. At an individual level, numerators calculated using method A therefore reflected the age at the time of the first dose and did not change over time. As the program expanded to include additional recommendations with longer intervals between doses (e.g. six month intervals between booster doses), age calculations were revised as age at dose one no longer accurately reflected an individual's age at the time of the coverage assessment. For method B, an individual's date of birth and the date of data extraction were instead used to estimate age-specific numerators. Numerators therefore described an individual's age at the time of the coverage assessment and changed over time as an individual aged. For method C, following integration of mortality information from Ontario's RPDB with COVaxON, numerator calculations were further revised to exclude individuals reported as deceased in RPDB as these individuals no longer contributed to vaccination coverage for the province.

Denominators for coverage calculations
Ontario-specific population data from Canada's national census were the source of denominators for coverage calculations. National censuses are carried out every-five years by Statistics Canada and provide counts of the size of the provincial population, as well as future population projections, based on the information provided by an individual at the time of census completion [15]. Population estimates are later adjusted following post-censal studies carried out two years after enumeration [20]. Future population projections take into account recent trends in population growth, fertility, mortality, immigration, emigration, and interprovincial migration [21].
At the start of the vaccination program in Ontario, the most recent population data available was from the national census carried out in 2016. Ontario population estimates for 2020 from the 2016 census were used for age-specific denominators for methods A, B, and C ( Fig. 1) [22]. Following the availability of updated population data from the 2021 national census, denominator data was updated to Ontario population projections for 2022 in method D [23]. The use of census data for denominators is in contrast to a record-linkage approach to coverage assessment where vaccination data (numerator) is linked to a dynamic population denominator, such as a registered person's database, and the vaccination status of each individuals within a linked cohort is determined.

Methodological evolution of COVID-19 vaccine coverage estimates
To illustrate the evolution and impact of the changes in methods on COVID-19 vaccine coverage estimates, a single COVaxON data extract was used to facilitate comparison. Numerator data were extracted from COVaxON on August 29, 2022 and included doses administered up to August 28, 2022. For methods A to D, coverage estimates by vaccination status (i.e. at least one dose, completion of primary series, completion of primary series and receipt of one booster dose, completion of primary series and receipt of two booster doses) were determined by dividing the aggregated number of vaccinated individuals (numerator) in a particular age group (in years) by the estimated number of individuals in the Ontario population (denominator) for that age group to obtain the percentage of the population vaccinated [24]. As census data are not readily available by age in months, denominators for coverage estimates for children under 5 years of age include children 0 to under 6 months of age. However, this age group is not included in the numerator as children 0 to under 6 months of age are not eligible for COVID-19 vaccination. Age-based program eligibility was considered when calculating coverage estimates. For example, coverage estimates for a complete series and first booster dose are not shown for children under 5 to 11 years of age as this group became eligible for booster doses in September 2022.

Results
Early in the COVID-19 vaccination program roll-out, overall uptake was high. From December 14, 2020 to August 28, 2022 an estimated 88.7 % (10,937,784 individuals) of the Ontario population 18 years of age and older competed their primary series (Table 1, method D). As the program expanded to include first and second booster doses, uptake was lower. Coverage estimates for series completion and one booster (56.2 %) and series completion and two boosters (17.0 %) were much lower for age-eligible populations (Fig. 2). Despite lower overall booster uptake, coverage including booster doses was higher among older adults, a population at high risk of severe outcomes following a COVID-19 infection. Approximately 79.9 % and 42.9 % of adults aged 60 years and older received their first and second booster doses, respectively (Fig. 3).
The four methodological approaches applied over the course of the vaccination program to calculate age-specific COVID-19 vaccination coverage estimates are shown in Table 1. Regardless of the specific method applied, COVID-19 vaccine coverage tended to increase with increasing age across all vaccination status definitions and tended to decrease with increasing number of doses included in the coverage definition. However, the impact of the methodological changes varied with age and were most apparent in coverage estimates for at least one dose. For method A, where an individual's age at the time of their first dose is used for the numerator and 2020 population estimates for the denominator, coverage estimates increased with age with the highest estimates among older adults. Coverage estimates for at least one dose ranged from 5.  (Table 1). For method B, when an individual's age was calculated at the time of data extraction, vaccinated individuals shifted into older age groups given that months or years had passed since their first dose. While this methodological update better reflected the age distribution of vaccinated individuals in the province, coverage estimates generally decreased in younger age groups and increased in older age groups; in some cases coverage estimates increased to 100 % or higher. For example, among younger age groups the impact was largest among 5 to 11 years olds where the coverage estimate for at least one dose decreased by 9.1 percentage points, from 62.8 % (677,116 vaccinated individuals) using age at dose one to 53.7 % (528,091 vaccinated individuals) using age at data extract, as some individuals at the upper end of the age band shifted into the 12-17 year age group (Fig. 4). Similar trends were observed among 12 to 17 year olds. Among individuals aged 80 years and older the estimate for at least one dose increased by 16.7 percentage points, from 94.8 % (621,532 vaccinated indi-viduals) using age at dose one to 111.5 % (731,328 vaccinated individuals) using age at extract, as some individuals from the 70 to 79 year age group shifted into the 80 year and older group. Further, the coverage estimate for series completion among individuals 80 years of age and older increased over 100 % as well to 109.0 %.
For method C, removing deceased individuals from the numerator reduced the number of vaccinated individuals overall. The provincial coverage estimate for at least one dose decreased from 86.6 % (12,758,544 individuals) to 85.7 % (12,631,395 individuals). The impact of this update was minimal in coverage estimates for younger age groups, however, larger decreases were observed in estimates for the oldest age groups. For example, among individuals 60 to 69 years of age, the coverage estimate for at least one dose decreased by 0.9 percentage points from 100.0 % (1,765,384 vaccinated individuals) to 99.1 % (1,750,042 vaccinated individuals). For those aged 70-79 years and 80 years of age or older, the coverage estimates for at least one dose decreased by 2.4 percentage points (from 104.9 % to 102.5 %) and by 11.3 percentage points (from    111.5 % to 100.2 %), respectively. However, both estimates remained over 100 %. Finally, for method D, data from the most recent census, conducted in 2021, projected an increase in the Ontario population from approximately 14,734,014 individuals (from the 2016 census) to 15,128,543 individuals. Population projections for each age group increased, with the exception of the 50-59 year olds where projections decreased. Following the denominator update, coverage estimates for all age groups were less than 100 %. For example, the coverage estimate for at least one dose for individuals aged 70-79 decreased by 7.5 percentage points (from 102.5 % to 95.0 %) as the denominator increased from approximately 1,134,561 individuals to 1,223,821 individuals. A similar decrease of 5.1 percentage points (from 100.2 % to 95.1 %) was observed in the coverage estimate for at least one dose for individuals 80 years of age and older as the denominator increased from 655,835 individuals to 691,070 individuals.

Discussion
Similar to experiences elsewhere, the implementation of a COVID-19 immunization registry in Ontario, Canada was essential to support close to real-time vaccine program surveillance [2]. Over time various enhancements were made to the registry, including integration with Ontario's RPDB, which allowed for the advancement of methodological approaches to provincial COVID-19 vaccine coverage assessment. Although the COVID-19 vaccine program has been unique in many ways, the dynamic work of ongoing vaccine coverage monitoring has identified several methodological considerations that have relevancy to other vaccine programs and systems. Age at the date of the coverage assessment better reflected the current age distribution of the vaccinated population rather than a static age at the first dose measure, but resulted in coverage estimates exceeding 100 % in the oldest age groups where vaccine uptake was highest. Removing deceased individuals from the numerator improved estimates, but some remained over 100 %. Finally, updating population denominators to more recent census data further adjusted coverage estimates such that none exceeded 100 %.
Despite methodological changes to improve coverage estimates, certain limitations remained. A two to three month delay in mortality information reported in RPDB, as well as not accounting for migration of individuals in and out of the province, may have impacted numerator estimates. In Ontario, ascertainment of residents vaccinated outside of the province is likely incomplete due to delayed data entry and/or under-reporting, as doses administered out of province are reported by the individual to local public health departments, further impacting numerator estimates. With the removal of government mandates requiring proof of vaccination for a complete primary series there is likely little incentive for individuals to report out of province doses. Further, mandates only required reporting a complete primary series and never expanded to reflect recommendations for booster doses. Conversely, individuals that are not residents of Ontario such as interprovincial workers (e.g. residents of neighbouring provinces who work in Ontario) who were vaccinated in Ontario are captured in COVaxON, but would not be included in denominator estimates (regardless of the source) for Ontario's population. However, the impact of these aspects (out of province doses and immunizations administered to non-residents) on coverage estimates is expected to be small, but may vary by age. Finally, some aspects of COVID-19 coverage assessment, and opportunities for targeted follow-up based on coverage data, could not be accomplished with the use of COVaxON data alone such as estimating vaccine coverage for non-age based priority groups (e.g. long-term care home residents, health care workers), differentiating a first booster dose from an extended primary series, and the identification of eligible but unvaccinated individuals.
While electronic immunization registries provide more complete records than other systems, strategies are still needed to improve data quality [25]. Duplicate client records and incomplete or incorrect client or dose administration details may have resulted in numerator over or underestimates [25]. Data cleaning logic was developed to address these issues, however, given the scale of the vaccination program and the numerous types of immunization providers, ongoing efforts in regards to data quality are required. Further, due to the complexity of the program, such as eligibility requirements and recommended dose schedules, significant data cleaning and processing is required in order to most accurately describe coverage [25].
Population denominators from Statistics Canada census data are typically used for reportable disease surveillance in Ontario, including COVID-19. The Statistics Canada methodology for describing the population is well developed and estimates are refined over time following the availability of additional data [20]. However, census population estimates and projections are not frequently updated, and consequently their accuracy decreases with time [26][27][28]. At the beginning of the COVID-19 vaccination program in Ontario in 2020, available population estimates were from the 2016 census. The impact of the 2016 denominator estimates were most apparent in older age groups where high vaccine uptake resulted in some coverage estimates exceeding 100 %. Data from the 2021 census projected an increase in the Ontario population compared to estimates from the 2016 census, resulting coverage estimates that did not exceed 100 %. While the 2022 projections currently provide a more accurate description of the Ontario population, their accuracy will similarly decrease over time as the Ontario population changes. Linkage to a dynamic population database, such as RPBD, to create an individual-level cohort for the province (where all vaccinated individuals in the numerator are accounted for in the denominator) would mitigate this limitation but raises other potential challenges, such as the exclusion of individuals in the province who are eligible for COVID-19 vaccination but do not have a HCN.

Conclusions
Ontario's COVID-19 immunization registry is a powerful tool for vaccine program surveillance. The unprecedented impacts of the COVID-19 pandemic and scale of the vaccination program led to the development and implementation of COVaxON in Ontario. A single registry for COVID-19 vaccines encompassing all providers and all vaccinated individuals was essential for comprehensive near real-time coverage assessment, and further enabled new uses of vaccination data to support additional components of vaccine program surveillance [4][5][6][7][8][9][10][11]. Given the complexities and unique aspects of the COVID-19 vaccination program, various methodological changes were needed to maintain ongoing surveillance over a multi-year vaccination program. Further methodological changes will likely be needed as the vaccination program continues to expand. The province is well positioned to build on what has been achieved as a result of the COVID-19 pandemic and expand the registry to other routine vaccination programs in order to strengthen non-COVID-19 vaccine program surveillance.

Ethics Approval and Consent to Participate
This project did not require research ethics committee approval as the activities described in this manuscript were conducted in fulfillment of Public Health Ontario's legislated mandate ''to provide scientific and technical advice and support to the health care system and the Government of Ontario in order to protect and promote the health of Ontarians" (Ontario Agency for Health Protection and Promotion Act, SO 2007, c 10) and are therefore considered public health practice, not research.

Role of the funding source
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Contributors
JLH completed the original draft of the manuscript, as well as data interpretation and analysis. LP completed data interpretation and analysis. SW conceived of the report. All authors contributed to the methodological changes described in the manuscript, as well as edited and critically reviewed the manuscript. All authors have approved the final manuscript for submission.

Data availability
Public Health Ontario cannot disclose underlying data. Information on the data request process is available at https://www.publichealthontario.ca/en/data-and-analysis/using-data/data-requests.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.