Standardizing case definitions for monitoring the safety of maternal vaccines globally: GAIA definitions, a review of progress to date

In 2014, the Global Alignment on Immunization safety Assessment in pregnancy consortium (GAIA) was formed, with the goal of developing a harmonized, globally‐concerted approach to actively monitor the safety of vaccines in pregnancy. A total of 26 standardized definitions for the classification of adverse events have been developed. The aim of this review was to identify and describe studies undertaken to assess the performance of these definitions. A literature search was undertaken to identify published studies assessing the performance of the definitions, and reference lists were snowballed. Data were abstracted by two investigators and a narrative review of the results is presented. Four studies that have evaluated 13 GAIA case definitions (50%) were identified. Five case definitions have been assessed in high‐income settings only. Recommendations have been made by the investigators to improve the performance of the definitions. These include ensuring consistency across definitions, removal of the potential for ambiguity or variations in interpretation and ensuring that higher‐level criteria are acceptable at lower levels of confidence. Future research should prioritize the key case definitions that have not been assessed in low‐ and middle‐income settings, as well as the 13 that have not undergone any validation.


| INTRODUC TI ON
Despite significant gains in reducing under-5 mortality under the Millennium Development Goals (MDGs), neonatal mortality remains unacceptably high, now accounting for 47% of all under-5 deaths globally. 1 Implementing interventions to reduce this burden has therefore become an important part of the Sustainable Development Goal targets for 2030. 2 Maternal vaccination is an important intervention that has the potential to improve both maternal and infant health. It has been highlighted by the WHO in its lifecourse approach to immunization and is an area where the full bene- fits have yet to be realized. 3 The maternal-neonatal tetanus program has demonstrated the potential of this approach. First implemented in 1989, by 2020, all but 12 targeted countries had reached elimination status. This has contributed to a resurgent interest in maternal vaccination as a means of protecting infants through their vulnerable first months. Promising new maternal vaccines for group B Streptococcus 4 and respiratory syncytial virus 5 are under development, 6 alongside those that may be of particular benefit for pregnant women such as SARS-CoV-2, ebola virus, lassa fever and hepatitis E. 7,8 Implementation of these vaccines requires robust safety monitoring, including in low-and middle-income countries (LMICs).
Safety data generated to monitor maternal vaccine safety must be accurate and comparable globally, necessitating consistency in the terms and case definitions (CDs) used to quantify adverse events following immunization (AEFI).
The Brighton Collaboration founded in the year 2000 has developed standardized CDs and guidelines for vaccine adverse event data collection, analysis, and presentation via participation of more than 500 experts from 57 countries from public health, clinical care, academia, regulatory organizations and industry. 9 For each health outcome that could be an AEFI, a document is developed, including a background preamble to highlight the current knowledge on the outcome, the rationale and essential decisions made to develop and utilize the CD. The CD itself is structured in a format with three levels of diagnostic certainty, considering current scientific evidence and resources available in different research and geographic settings. Guidelines for data collection, analysis, and presentation of a given AEFI are provided, along with references for selected points discussed in the preamble (https://brigh tonco llabo ration.us/about -old/the-brigh ton-metho d/).
In 2014, the Global Alignment on Immunization safety Assessment in pregnancy (GAIA) consortium was formed, with the goal of developing a harmonized, globally-concerted approach to actively monitor the safety of vaccines and immunization programs in pregnancy. This consortium, part of the Brighton Collaboration, has developed 26 standardized definitions for the classification of adverse obstetric and infant events. These CDs were developed with the aim of achieving sufficient applicability for monitoring immunization safety in pregnancy globally. 10 The selection of these outcomes was prioritized based on recommendations from global experts convened by the WHO in 2014. 10 The definitions categorize the outcomes into levels of diagnostic certainty (LOC) 1-3, with greatest specificity at the highest level (level 1) and increasing sensitivity through the lower levels, while still maintaining an acceptable specificity. The CDs have been developed in this way to accommodate the resources and diagnostic capabilities available in different locations (Table S1).
The definitions were designed primarily for use in maternal vaccine trials whereby information could be prospectively collected to classify and report important maternal, fetal, and infant outcomes.
The definitions also incorporated clinical assessment methods commonly used in LMICs to optimize the ability to classify cases from these settings. 9 Given that health systems in many LMICs are overburdened and access to diagnostic tools is limited, the extent to which routine care and documentation will need to be strengthened to diagnose these outcomes with certainty in these settings is not clear.
Given that pregnant women are routinely excluded from clinical vaccine trials (unless a vaccine is specifically designed for use in pregnancy), there is a marked reliance on post-implementation safety studies. An understanding, therefore, of whether the definitions can be applied in retrospective datasets is an important consideration. Since their publication, a significant amount of work has been undertaken to assess the utility of the GAIA definitions, particularly their ability to classify outcomes in resource-limited settings.
This review brings together this research, highlighting the progress that has been made in field-testing these definitions in clinical trial and observational research contexts, prospective and retrospective datasets and identifies areas that require further research.

| ME THODS
A literature search was conducted in Ovid Medline using a combination of MeSH terms and keywords on the topics of immunization, safety, maternal or pregnancy and CDs (Table S2). The results were reviewed by a single reviewer (HGD) and assessed for inclusion. Studies that assessed performance of GAIA definitions using real-world data (routine or research) were included in the review and studies were included regardless of whether they used retrospective or prospectively collected data. Studies that applied the definitions, for example, in vaccine trials or safety studies, but did not assess performance or utility were not included. Reference lists of all relevant studies were also scrutinized. Studies that met these criteria were reviewed and data abstracted into an Excel workbook; they were checked by a second reviewer (CB) to ensure they were correctly abstracted. If data were not available with sufficient detail in the published manuscripts, then the authors were contacted to request further information.
Abstracted data were used to create a series of tables and figures.
The study setting, number of sites, country income status, and data types used were identified. The number of CDs assessed as well as the number of individual cases included were abstracted. The numbers and proportions falling into the LOC (1-3) as well as reported cases with insufficient data to classify (level 4), those determined not to be a case (level 5), and those with insufficient documentation to distinguish between levels 4 and 5 (unclassifiable cases) across all studies were presented (classification in Table S3). A narrative summary of challenges identified with the CDs was compiled as well as recommendations made by the authors for modifications to the CDs. Differences in the interpretation or application of the definitions in the individual studies or sites were also described.

| RE SULTS
Members of the GAIA working group have developed 26 pregnancy and neonatal outcomes to monitor safety of maternal vaccines. Ten were published in 2016, 12 in 2017, and a further 4 in 2019 ( Table 1).
A total of 110 results were returned from the database search (Table S2). Three studies were identified from this source and a fourth from reference list review. Four published studies have assessed the performance of the CDs using clinical data, assessing a total of 10 061 outcomes ( Table 2). The first, a retrospective feasibility assessment conducted by the WHO and published in 2018, 36 led to a prospective multi-country collaboration project that assessed seven CDs, published in 2021. 37 A study from South Africa and The Gambia assessed three CDs and one enabling term -gestational age (GA) in retrospective data from two randomized controlled trials. 38 The most recently published study focused on applicability of 10 definitions in retrospective data from highincome settings. 39 Two studies assessed the definitions using routine clinical data, one using research data and one using a mix of both. A combination of data from both high-income (four countries) and LMIC settings (eight countries) have been used to test the definitions, with a total of 12 countries contributing data. Three studies assessed medical records retrospectively and one study recruited participants prospectively. The majority of participants were recruited prospectively TA B L E 1 Published Global Alignment on Immunization safety Assessment in pregnancy consortium (GAIA) case definitions, date of publication, target of the definition, inclusion in published studies, and number of cases assessed. (95%) and information on outcomes was collected from their routine clinical data sources ( Investigators from the two WHO-led projects developed algorithms for classifying the cases entered into the electronic case report forms. 36,37 The retrospective randomized controlled trial (RCT)-based study utilized medical personnel from the sites to manually review the medical notes and assign a LOC. 38 Finally, the study from high-income settings used a combination of a Brighton Collaboration-developed automated tool (ABC Case-logic) and review by medical abstractors to assign LOCs. 39

| Performance
Performance of the definitions is summarized in Table 3, Figure 1, and

| Interpretation, challenges, modifications
The authors of several publications described the specific challenges they faced in using the CDs in their respective settings and, in some cases, proposed modifications to the CDs. Stuurman et al. 37 proposed a modification to allow more flexibility in assigning LOCs. Criteria that were acceptable at higher levels of diagnostic certainty were also de facto acceptable at lower levels of diagnostic certainty in their analyses. For example, for the low birth weight (LBW) outcome ( Figure 2; This level of flexibility, although also recommended as a modification to the stillbirth definition by the investigators in the earlier Stuurman et al. 36 was not applied whilst categorizing the cases as part of this study, leading to several cases falling between two LOCs, and therefore assigned LOC4 (Table S6). Kochhar et al. 38 found that all the antepartum and intrapartum stillbirths The investigators also highlighted some ambiguity within the LOCs the criteria were more lenient and for others more stringent; for example, LOC1A allows for second-trimester ultrasound scan, which is a GA LOC2 criteria and is not acceptable at LOC1 in the other definitions. They highlighted that the birth weight requirements for the LBW and SGA outcomes were also inconsistent, noting that SGA LOC3A required more stringent scale specifications (a scale with <50 g resolution, tared to zero and calibrated) than LBW LOC3 (weight measured using dial/spring/color-coded scale). 37 A full description of these inconsistencies is provided in their manuscript.  and for some require a significant amount of detailed clinical data to support classification (Tables S7 and S8). The retrospective nature of the data used in this study was likely to have contributed to this challenge. Performance of these definitions needs to be assessed using prospectively collected data to determine whether the classification challenges are due to the CDs being too specific, or due to to missing or inadequate data in these retrospective studies. These assessments need to be made in both high-, middle-and low-income settings as it is possible that the level of clinical detail required to classify these outcomes is lacking in lower-resourced settings, even in prospectively collected datasets. A feasibility assessment conducted in Uganda sought to establish whether 25 health centers of varying levels had the physical, laboratory, and human resources necessary to fulfill the criteria for each of 10 outcomes. The results were encouraging, with most facilities, in theory, able to classify to LOC3; however, they did not assess the five outcomes listed earlier in this study. 40  or ICD-10 diagnostic codes into the definition, which will enhance use with retrospective electronic health records data. 15 Moll et al. 41 successfully developed a claims-based algorithm for determining pregnancy outcomes (live birth, preterm birth, stillbirth, and spontaneous abortions), achieving high-percentage agreement between the algorithm and clinician adjudication of GAIA LOCs (62.4%-100% depending on the outcome). This study supports the notion that it might be possible to translate the GAIA CDs into the language of electronic health records.

| Future studies (upcoming research)
The authors are aware of a number of upcoming studies that will undertake further work in validating the CDs. These include a large retrospective cohort study from South Africa that aims to describe the incidence of adverse pregnancy and birth outcomes. A planned sub-analysis will describe the data required to improve diagnostic certainty in the setting. Optimizing documentation and implementation of key diagnostics such as urine dipsticks and blood sugar monitors will be key to improving diagnostic certainty in these settings. Another cohort study, designed to describe background rates The Brighton Collaboration CDs and guidelines are meant to undergo review on a regular basis. It is important therefore that further validation work is undertaken to enable this, and that planning of these reviews is now considered.

| CON CLUS IONS
Harmonization of adverse event terminologies, definitions, and methods of assessment is of critical importance to allow comparability and timely assessment of vaccine safety through pooling and meta-analyses of globally generated data. Further work is required to ensure that the GAIA definitions are suitable for undertaking these assessments in both clinical trials and post-implementation studies. The 13 CDs that have not undergone any field assessment should be prioritized as well as those that have only been assessed in high-income settings. Simple modifications, such as ensuring consistency across definitions, removal of the potential for ambiguity, or variations in interpretation and modifications to ensure that higherlevel criteria are de facto accepted at lowers levels of confidence, would improve performance of the CDs.

AUTH O R CO NTR I B UTI O N S
HGD: conceptualization, methodology, visualization, data curation, analysis and writing the original draft of the manuscript. CB: data curation, analysis, manuscript review and editing. CD and GW: data curation, manuscript review and editing. KLD and CLC: conceptualization, visualization, manuscript review and editing. PH, FM and CJ: manuscript review and editing.

FU N D I N G I N FO R M ATI O N
No funding was received for the completion of this work.

CO N FLI C T O F I NTE R E S T S TATE M E NT
HGD is an investigator on the cited Uganda study; she has no other conflicts to report. CB has no conflicts of interest to report.
PTH was a member of the GAIA working group. CLC is principal investigator for mentioned study in South Africa; she was a member of the GAIA working group and has no other COIs. KLD is principal investigator for the Ugandan study; she has no other conflicts to report. CEJ was a member of the GAIA working group and author of a GAIA case definitions; she was an author of one of the studies assessed in this manuscript (Watson et al. 39 ). GW was author of one of the studies assessed in this manuscript (Watson et al. 39 ).

DATA AVA I L A B I L I T Y S TAT E M E N T
Data sharing is not applicable to this article as no new data were created or analyzed in this study.