Protocol for a living evidence synthesis on variants of concern and COVID-19 vaccine effectiveness

Background: It is evident that COVID-19 will remain a public health concern in the coming years, largely driven by variants of concern (VOC). It is critical to continuously monitor vaccine effectiveness as new variants emerge and new vaccines and/or boosters are developed. Systematic surveillance of the scientific evidence base is necessary to inform public health action and identify key uncertainties. Evidence syntheses may also be used to populate models to fill in research gaps and help to prepare for future public health crises. This protocol outlines the rationale and methods for a living evidence synthesis of the effectiveness of COVID-19 vaccines in reducing the morbidity and mortality associated with, and transmission of, VOC of SARS-CoV-2. Methods: Living evidence syntheses of vaccine effectiveness will be carried out over one year for (1) a range of potential outcomes in the index individual associated with VOC (pathogenesis); and (2) transmission of VOC. The literature search will be conducted up to May 2023. Observational and database-linkage primary studies will be included, as well as RCTs. Information sources include electronic databases (MEDLINE; Embase; Cochrane, L*OVE; the CNKI and Wangfang platforms), pre-print servers (medRxiv, BiorXiv), and online repositories of grey literature. Title and abstract and full-text screening will be performed by two reviewers using a liberal accelerated method. Data extraction and risk of bias assessment will be completed by one reviewer with verification of the assessment by a second reviewer. Results from included studies will be pooled via random effects meta - analysis when appropriate, or otherwise summarized narratively. Discussion: Evidence generated from our living evidence synthesis will be used to inform policy making, modelling, and prioritization of future research on the effectiveness of COVID-19 vaccines against VOC.


Introduction
The rapid development of safe and effective vaccines for COVID-19 [1,2] has been a tremendous achievement, however, research on variants of concern (VOC) is also a fast-moving field.Internationally, based on whole genome sequencing, successive VOC have dominated the epidemiology of SARS-CoV-2.Since the emergence of the Omicron variant in November 2021, there has been continued rapid spread and declines of the other VOC in all six World Health Organization (WHO) regions [3].Beyond the emergence of new variants, there is also continuous development of new vaccination products available to combat the spread of variants and the severity of disease caused by variants.To this end, there is an urgent and unmet need to continuously monitor whether and how much vaccination is effective against current and emerging VOC [4][5][6][7][8].
The rate of emergence of potential VOC [9][10][11] and the rapidly evolving and heterogeneous scientific evidence make it difficult to keep up to date with current, best evidence.Living evidence syntheses (LES) use rigorous scientific methods to identify, appraise and summarize the body of evidence on a particular question and offer an approach for continuous, ongoing literature surveillance [12,13].LES are particularly helpful when the evidence base is rapidly developing and has substantial policy, public health, and clinical practice implications.
Since April 2021, our research teams at McMaster University and the University of Ottawa have deployed 41 editions of a living review to determine the effects of VOC on vaccine effectiveness, which is regularly communicated to the Public Health Agency of Canada (PHAC) among others [14].While other groups conduct living evidence syntheses of RCTs of vaccine effectiveness, none focus on observational studies of VOC and vaccine effectiveness; given that most of the relevant and policy-informing evidence will stem from observational and database linkage studies, our living review has addressed this knowledge gap for a limited set of outcomes.
We are now extending this review [14] to determine vaccine effectiveness on pathogenesis and transmission of VOC for a range of outcomes, including immunological outcomes and patient-reported outcomes relating to post-acute sequelae of COVID-19 (PASC), also known as long COVID or post COVID-19 condition.The evidence synthesis will be regularly updated to account for VOC emergence, new vaccines, and availability of new information about how VOC affect vaccine response.In addition, results from the living evidence synthesis will be used to populate models of pathogenesis and transmission and to identify key uncertainties which would inform policy making and prioritization of research [15].Evidence generated from our living evidence synthesis and models will also help to prepare for future public health crises.

Objectives
Our aim is to further develop a living evidence synthesis of the effectiveness of COVID-19 vaccines in reducing the morbidity and mortality associated with, and transmission of, variants of concern of SARS-CoV-2.We will achieve this objective by (a) conducting a living evidence synthesis with a particular focus on observational and database-linkage primary studies, which will (b) feed into mathematical modelling of pathogenesis and transmission.This protocol outlines the rationale and methods for the living evidence synthesis.

Table 1
Eligibility criteria for living evidence synthesis on vaccine effectiveness and variants of concern.Studies in which participants were assigned to receive a standard vaccine dose or reduced vaccine dose are also eligible.
For immunological outcomes, we will distinguish between those (a) vaccinated and not exposed to SARS-CoV-2, (b) vaccinated and have been infected (hybrid immunity).We will further note if individuals with hybrid immunity were vaccinated and then developed COVID-19 (breakthrough infections) or were exposed to SARS-CoV-2 and then vaccinated.
For PASC outcomes, we will distinguish between those (a) vaccinated before SARS-CoV-2 infection and (b) vaccinated after SARS-CoV-2 infection.
Partial vaccination (an incomplete primary vaccine series) Vaccination administered through routes that are not intramuscular (e.g., intradermal, aerosolized)

Comparators
For outcomes in index individuals and unvaccinated members of their households, unvaccinated individuals from the same or a similar setting.For outcomes in populations, comparisons between regions or within regions over time, by vaccination uptake or adjusting for it.
For the relative effectiveness relating to booster doses, the comparator should consist of a similar population who has received a fewer number of doses of the same vaccine.
For the relative effectiveness of different vaccine brands, the comparator should consist of a similar population who has received a primary vaccine series and an equivalent number of booster doses.
For immunological outcomes, we will consider (a) healthy controls (unvaccinated and no history of infection

Methods
Protocol Development: This evidence synthesis is based on the framework of Crowcroft and Klein for research on vaccine effectiveness [16].The protocol follows Cochrane living systematic review guidance [13] and PRISMA-P reporting guidelines [17].The final protocol will be registered on PROSPERO (CRD42022359790) and the Open Science Framework (https://osf.io/qacw4/).Any amendments to the protocol will be documented in the PROSPERO registration and the final published report.
Eligibility Criteria: A summary of the eligibility criteria can be found in Table 1., with a more detailed description provided below.

Participants
For pathogenesis, index individuals will be classified by country/ region of residence, age, sex, ethnicity (including Indigenous status), resident in a long-term care facility, residential status, pregnancy status, whether immunocompromised or not, whether or not a health care worker, and neighbourhood characteristics (e.g., prioritized for vaccination on basis of racialization or concentration of essential workers).
For transmission, we will consider (i) contact(s) of index individuals; (ii) index individuals; (iii) populations in which COVID-19 vaccinations were implemented; (iv) viral characterization studies.

Interventions
COVID-19 vaccines approved by, or under consideration in, the WHO EUL/PQ evaluation process [18].We will consider number of doses received when relevant (fully vaccinated or those who have received booster doses), and dose interval.Studies that examine the effectiveness of partial vaccination (an incomplete primary vaccine series) and vaccination administered through routes that are not intramuscular (e. g., intradermal, aerosolized) will be excluded.We will consider studies in which participants were assigned to receive a standard vaccine dose or reduced vaccine dose, and will capture the dose amount used in these studies.We will consider studies on bivalent vaccines if this data becomes available.
For immunological outcomes we will distinguish between those (a) vaccinated and not exposed to SARS-CoV-2, (b) vaccinated and then infected (breakthrough infections) and (c) exposed to SARS-CoV-2 and then vaccinated.For PASC outcomes, we will distinguish between those (a) vaccinated before SARS-CoV-2 infection and (b) vaccinated after SARS-CoV-2 infection.

Comparators
For outcomes in index individuals and unvaccinated members of their households, unvaccinated individuals from the same or a similar setting.For outcomes in populations, comparisons between regions or within regions over time, by vaccination uptake or adjusting for it.
For relative effectiveness relating to booster doses, the comparator should consist of a similar population who has received a fewer number of doses of the same vaccine.
For the relative effectiveness of different vaccine brands, the comparator should consist of a similar population who has received a primary vaccine series and an equivalent number of booster doses.
For immunological outcomes, we will consider (a) healthy controls (unvaccinated and no confirmed history of infection), (b) a convalescent group (unvaccinated and confirmed prior infection), and c) a vaccinated control group.Eligible vaccinated control groups include (i) those given a standard dose (compared to reduced); (ii) those given a different vaccine brand; (iii) those given a fewer number of doses of the same vaccine; or (iv) those vaccinated and not exposed to SARS-CoV-2 (compared to vaccinated and infected, i.e., hybrid immunity).
For PASC outcomes, the comparator should consist of an unvaccinated population who has been infected with SARS-CoV-2.

Outcomes
Only studies that examine Omicron and/or Delta variants are eligible for inclusion.However, for PASC outcomes, we may choose to also include studies that examine Alpha, Beta, and Gamma, depending on the extent of literature available.If a new VOC emerges, then this eligibility criteria may be modified to include studies that examine the new VOC.
We will include PASC outcomes within a subset of outcome domains from the core outcome set for PASC [19][20][21]: respiratory functioning, symptoms and conditions; fatigue or exhaustion; pain; nervous system functioning, symptoms and conditions; cognitive functioning, symptoms and conditions; mental functioning, symptoms and conditions; and cardiovascular functioning, symptoms and conditions.Domains were selected based on the outcomes with the highest reported prevalence from systematic reviews on PASC outcomes [22][23][24][25][26][27][28][29].We will also include the overall prevalence of PASC as an outcome, which will be defined as having one or more symptoms at least 12 weeks after COVID-19 diagnosis [24].In addition, we will include the overall number of PASC symptoms, quality of life, overall functional impairment (ability to perform daily living activities), and ability to work as patient-reported outcomes [30].
PASC for adults will be defined through the WHO definition of symptoms in individuals with a history of probable or confirmed COVID-19 infection that occur 3 months from the onset of COVID-19 and that last for at least 2 months, and cannot be explained by an alternative diagnosis [31].If the duration of symptoms is not reported, we will rely on the criteria of post-onset symptom timing for eligibility.For children and youth, we will use the similar definition proposed of illness in individuals with a history of confirmed infection, with at least one persisting physical symptom for a minimum duration of 12 weeks after initial testing [32].
Hospitalization may be included as an outcome in the first weeks of emergence of a new VOC, if hospitalization data are the only early data available.However, we will not typically include hospitalization as an outcome due to inconsistent reporting and the decision to hospitalize can depend on many factors (including capacity of the healthcare system).Hospitalization will be considered as a binary outcome (yes/no admission to intensive care unit or other hospitalization, including emergency admissions and use of invasive mechanical ventilation).
(c) Secondary effectiveness relating to transmission: incidence of COVID-19 in unvaccinated contacts (such as household members) of vaccinated index individuals.(d) Immunological outcomes: We are interested in capturing outcomes where there is a linkage between immunological measures and clinical outcomes, such as anti-SARS antibody titers [33] and neutralization activity [34].Viral load is also an outcome of interest given that data on viral shedding kinetics can be used to inform models of variant transmission [35].If a study reports PCR cycle threshold (proxy for viral load) but no other eligible outcomes are reported, then it will be excluded.
Eligible outcomes include (i) PCR cycle threshold; (ii) humoral/ antibody mediated immunity: detection (seropositivity) for anti-SARS antibody titers (overall/anti-spike and RBD subtypes of IgG of main interest, but other immunoglobin types will be extracted, as available -IgA will be considered as a marker of mucosal immunity [36]), total neutralizing antibody titers (iii) cell-mediated immunity including anti-SARS-CoV-2-specific CD4 + and CD8 + T cell responses, T cell release of interferon-gamma, B-cell immunity.Additionally, results must be reported by vaccine brand and by the variant of concern.For example, the results must be separated by calendar time or by genome sequence to show the variant to which they apply.
We will exclude safety outcomes, which are beyond the scope of this living review focused on vaccine effectiveness for VOCs.Since our initial grant proposal, approved vaccines have very low rates of adverse events and systematic reviews have synthesized evidence on safety-related outcomes, such as VITT [37] and myocarditis [38,39].Additionally, there are currently specific international initiatives on the safety of COVID-19 vaccines, [40,41], as well as other reviews that have synthesized information on safety-related outcome [42,43].

Period Over Which Outcome(s) Will Be Assessed:
To avoid duplication of effort with an ongoing review [44], we will examine outcomes up to 4 months for two dose effectiveness and outcomes up to 3 months for three/booster dose effectiveness for data on primary vaccine effectiveness.
For primary/secondary vaccine effectiveness against PASC-related outcomes, outcomes beyond 12 weeks of acute infection will be assessed.No a priori restriction will be imposed on other outcomes.
Depending on the study design, we will document (a) the length of follow-up and (b) how the immediate post-vaccine window is handled (for example to exclude confounding by oversampling tested symptomatic vaccinated individuals, resulting in apparent early vaccine protection [45]).
Included Designs: Cohort studies, case-control studies [including test-negative case-control studies], surveillance studies, RCTs (for completeness).We will include in vitro studies with (human) participant data on immunological outcomes.
Editorials, commentaries, letters to the editor without novel relevant data, conference proceedings, government reports, case series, case report, narrative reviews, modelling studies will be excluded, as well as animal or cell culture studies.In vitro studies that are not linked to a defined human population with demographic data will also be excluded.
Studies will not be excluded based on their risk of bias (quality) assessment, given the lack of evidence on vaccination and PASC in the context of VOC, and that there is no standardized risk of bias assessment for immunological studies.
Data Sources: This LES is informed by the COVID-19 Evidence Alerts from McMaster PLUS TM [46] and our rapid scoping review of transmission characteristics of VOC [47].We will search electronic databases (MEDLINE; Embase; Cochrane, L*OVE [48], the CNKI and Wangfang platforms), pre-print servers (medRxiv, BiorXiv), Google, Twitter, and online repositories of grey literature (e.g., WHO, Canadian and other government agencies, the Canadian Agency for Drugs and Technologies in Health [CADTH] COVID-19 Evidence Portal).Building on our collaboration with PHAC, we will continue to incorporate a bi-weekly compilation of vaccine studies from PHAC.
Search Strategy: A detailed search strategy will be developed with the help of an experienced information specialist.We will use text-word searching of title and abstracts by combining content terms and validated methods terms to overcome delay in MeSH indexing [49][50][51].Syntax will be adjusted for each database.Another information specialist will peer review the MEDLINE strategy using the PRESS checklist [52].The completed PRESS checklist can be found in Appendix 1.No language restrictions will be applied.Data sources will be searched from January 2021 for immunological outcomes, and from March 2022 for PASC outcomes, as a previous review on vaccination and PASC has captured studies prior to this time [53].Search results will be priority ranked using a prototypal machine learning algorithm with > 50% specificity and screened by priority to maximize time efficiency [54].We will continue to update the literature search to May 2023 by re-running the search strategy with one day overlap and automatic deduplication of records.Monthly searches will be run to capture data on vaccine effectiveness, immunological studies and PASC outcomes.Grey literature sources will also be searched monthly.Search strategies may also be revised to ensure capture of emerging variants; any changes will be documented.Specific details regarding the draft search strategies are found in Appendix 2.
Study Selection: Duplicates across searches will be identified and removed, and the final list of articles will be uploaded using our reference management software, Covidence [55] for title/abstract and fulltext screening.We will pilot test the title and abstract screening and full-text article review forms on a random sample of 50 titles and abstracts and 25 full-text articles until reviewer agreement is high (>95%).Two reviewers will independently screen all retrieved records (Level 1) using a liberal accelerated process (one reviewer is required to include a study, but two to exclude).Any discrepancies among reviewers will be resolved by discussion or consulting with a third reviewer and adjustments to the form will be completed as needed.The full text of all records passing Level 1 screening will be retrieved for Level 2 dual reviewer screening to confirm final eligibility.Discrepancies will be resolved by consensus or by third-party adjudication.The screening cycle will be completed on a monthly basis to keep the process flowing in real time and synced with the result updates.
We will request articles that are not available electronically through the university (University of Ottawa or McMaster) interlibrary loan service.Corresponding authors will be contacted by email if a potentially relevant study reports information that is unclear for us to decide on eligibility.
Data Extraction: Standardized electronic data extraction forms will be developed, and pilot tested a priori.Full data extraction will initially be done in duplicate with assessment of disagreements for each outcome.Once agreement by outcome has reached kappa =>0.8, we will then maximize efficiency by single-person extraction with a second reviewer audit, with the two extractors alternating their role to limit fatigue induced errors.Disagreements will be resolved by consensus or third-party adjudication.For studies with missing outcome data or unclear information, we will contact the corresponding author by email with a maximum of three attempts.
Data Items: Complete list of data items to be extracted can be found in Appendix 3. Key elements for data extraction include publication characteristics, study design, details on the population for potential subgroup analyses, intervention/comparators (vaccine brand administered, number of doses, control group used), and outcomes.For immunological outcomes, we will extract the cut-off for seropositivity reported by the manufacturer of the assay used in each study, in order to standardize across studies that used varying thresholds for seropositivity [56].
. We will extract data on equity factors described by PROGRESS+ [57,58], to assess whether vaccine effectiveness differs across subgroups of individuals who experience different health inequities.We will also extract data on special populations, including health care/frontline workers, prison populations, long-term care residents, and immunocompromised populations.If this information is not reported by studies, we will document the lack of reporting of such information.
Risk of Bias (Quality) Assessment: Prior to study appraisal, reviewers will pilot the criteria of each tool on a random sample of five included studies, and conflicts will be resolved by discussion or the involvement of a third reviewer.One reviewer will independently appraise risk of bias using the appropriate tool for the included studies.A second reviewer will perform verification of the assessment.Any disagreements in the assessments will be resolved by consensus or by consulting with a third team member.
We will assess the risk of bias in individual studies as follows: (a) Cochrane risk of bias tool (RoB 2) [59] for primary RCTs of vaccines, (b) an adapted version of ROBINS-I for cohort, case-control and surveillance studies with PASC outcomes [60,61], and c) an adapted version of the Newcastle-Ottawa Quality Assessment Scale for studies with immunological outcomes [62].The purpose of the adaptation is to focus on study characteristics that introduce bias as reported in the vaccine literature [63].We will also assess the risk of bias in the body of evidence for each dimension of causality and the overall body of evidence for causality, upgrading or downgrading our confidence by considering mechanistic evidence along the lines of IARC Monographs [64].We plan to use the GRADE approach [65] to evaluate the certainty in the body of evidence for each outcome, when there is a sufficient volume of evidence.Additionally, if there is sufficient data (e.g., at least 10 studies), we will investigate meta-biases.For assessing small-study effects (e.g., publication bias), we will use funnel plots and statistical tests (e.g., Egger regression test, Hedges-Olkin method, trim-and-fill method [66][67][68].
Data Synthesis: We will describe the study characteristics, participant characteristics, intervention and comparator details, outcome results, and quality appraisals for the included studies in tables.
We anticipate that substantial differences in study design will impede meaningful meta-analysis for several combinations of VOC, vaccine type and outcome.We will consider clinical (e.g., patient characteristics) and methodological (e.g., study design) heterogeneity of included studies prior to performing a meta-analysis.If considerable heterogeneity (defined as I2 statistic above 75%) is detected, we may decide not to combine data in a meta-analysis.Instead, we will try to explain reasons for the heterogeneity via sensitivity analysis, and metaregression.We will describe the findings and present the range of effects.We may also classify associations into five categories (convincing, probable, limited-suggestive, limited-not conclusive or unlikely, following [69]), to conclude whether the evidence for a given outcome may be considered robust.Reporting will follow the SWiM guideline [70].
When meta-analysis is appropriate (results from > 1 study and heterogeneity isn't considerable), we will estimate the summary effect size and its confidence interval by using random effects models [71].We will pool results from randomized controlled trials and observational studies separately.For particularly sparse data, Bayesian meta-analysis with an appropriate selection of different priors as sensitivity analyses will be used.Cochrane's Q and the I2 statistic will be used to assess the statistical heterogeneity of effect estimates amongst included studies [72].Noting the limitations of the I2 statistic [73], we will also estimate the 95% prediction interval, which further accounts for between-study heterogeneity and provides an estimate of the range of magnitude of effect that would be expected in a new study [74,75].We will use the regression asymmetry test to test for small study effects [76].We will also apply the excess significance test [77,78].We will deem excess significance at a p < 0.10 threshold.In case of missing or particularly sparse data, we will use the ROB-ME tool to assess the risk of bias due to missing evidence [79].
Equity-related factors will be considered in our evidence synthesis, primarily explored through planned subgroup analyses.Our consideration of equity will be based on the reporting guidelines for health equity in COVID-19 observational studies [80].Equity is particularly important to analyze in the context of the COVID-19 pandemic, as systemic health and social inequities have led to a disproportionately higher burden of COVID-19 infection and adverse outcomes for certain populations [81][82][83], which can impact vaccine effectiveness.We will perform separate subgroup analyses according to factors described by PROGRESS+ [57,58].This includes, but is not limited to, health status, sex, gender, age, ethnicity, occupation, household structure (e.g., intergenerational or not) and socioeconomic status (individual income/ occupation, neighborhood measures of deprivation/income).
Additionally, we may perform subgroup analyses according to risk of bias (e.g., restricting to only include studies with low overall risk of bias), by publication type (e.g., removing abstracts only or preprints), or based on study design issues as considered in the risk of bias tool.For PASC outcomes, we may also perform subgroup analyses according to severity of initial COVID-19 infection (e.g., whether participants had been hospitalized for acute COVID-19), if sufficient data is available.However, other issues that we may want to examine through subgroup/ sensitivity analyses may only be identified during the systematic review.These analyses are deemed exploratory in nature and should not be construed as a priori with definitive hypothesis.
Quality Assurance: The living evidence syntheses will be reported according to the PRISMA 2020 reporting statement (Appendix 4) [47] and associated extensions [84].Any deviations from the protocol will be reported in the PROSPERO registration and in the final report.
Integrated Knowledge Translation: This protocol has been reviewed by co-applicants on the CIHR Operating grant supporting this project, as well as citizen partner co-investigators and knowledge users.A citizen engagement panel will be established to advise on communication about the ongoing research and results of this living evidence synthesis, including tailoring of plain language summaries, so that findings are easily accessible and understandable to the public amid an overwhelming amount of information on COVID-19.The operating grant co-applicant team will meet on a quarterly basis to discuss ongoing findings of the living evidence syntheses and interpretation of the data; refine the development of planned publications; and update on relevant stakeholders and their needs.They also may be contacted at key decision points, e.g., to inform eligibility criteria/retention of questionable studies or to refine the data extraction strategy to tailor to the available data.Towards the end of the project, the team will: (1) discuss results and implications for public health, policy and research; (2) determine the key messages to relevant stakeholders; and (3) discuss sustainability plans.
Outputs: Results will be communicated to PHAC monthly, and if there is a demand for more frequent updates, then results will be communicated on a bi-weekly basis.Additionally, results will be communicated to the Royal College of Physicians and Surgeons of Canada monthly and to the wider team at quarterly meetings.Other stakeholders (e.g., provincial bodies) who are interested may sign-up for updates at regular intervals.Results will be sent to the modelling team monthly and discussed at quarterly meetings.Results will be disseminated through publication of the results of the living evidence synthesis in peer-reviewed journals and on the Open Science Framework (htt ps://osf.io/qacw4/).Plain language summaries of results in both English and French will be posted on the COVID-END website (https://www.mcmasterforum.org/networks/covid-end)and the Knowledge Synthesis and Application Unit website (https://www.ksau.ca).
Discussion: Results from this living evidence synthesis will be used to inform the parameters of models on SARS-CoV-2 pathogenesis and transmission.Models will be regularly updated from the updated living evidence synthesis results.Modelling results will then be provided to the Coronavirus Variants Rapid Response Network (CoVaRR-Net) and the COVID-19 Evidence Network to support Decision-making (COVID-END) to guide public health policy development and research priorities in Canada.
One challenge is that there is no standardized risk of bias assessment for immunological studies.Therefore, we adapted the Newcastle-Ottawa Quality Assessment Scale to evaluate the risk of bias in immunological studies [62], and utilized the CONSISE-ROSES I reporting checklist [85] to extract whether these studies reported key elements related to laboratory methodology.There will be substantive heterogeneity across studies due to the varied public health measures by jurisdiction and time, and variable implementation and observance of each specific public health measure (or set of measures) across different regions and countries.While it will be challenging to make necessary adjustments, we will assess the robustness of associations across different public health jurisdictions and times.If consistent associations are observed across these factors, it will strengthen our overall certainty in the evidence.
An anticipated challenge is the limited amount of literature on vaccination and PASC, and particularly the lack of data on Omicron and N. Shaver et al.