WHO global vaccine safety multi-country collaboration project on safety in pregnancy: Assessing the level of diagnostic certainty using standardized case definitions for perinatal and neonatal outcomes and maternal immunization

Highlights • Applicability of GAIA definitions were assessed using medical records in LMICs.• LBW, preterm birth and neonatal death definitions were applicable in the field.• SGA, stillbirth, neonatal infection, congenital microcephaly definitions were less applicable.• Suggestions to the improvement in GAIA definitions were made.• Better documentation of maternal immunization is suggested for vaccine-safety studies.


Introduction
Immunization during pregnancy can protect the pregnant woman and her child, both in the womb and in early life by increasing the antibody titers against vaccine-preventable diseases [1,2]. Diseases like pertussis, influenza, group B streptococcus (GBS) infection, respiratory syncytial virus (RSV) infection and tetanus have a disproportionate impact on the newborn and young infant. Immunization of pregnant women with Tetanus Toxoid Containing Vaccine through the Maternal and Neonatal Tetanus Elimination (MNTE) initiative, achieved elimination of maternal and neonatal tetanus in 47 countries within twenty years [3]. For tetanus, immunization in pregnancy is a key strategy to prevent significant morbidity and mortality amongst young infants globally. Implementation or strengthening of pertussis and influenza immunization of pregnant women holds great promise as a strategy to protect infants from these infections [4][5][6]. This is of specific interest for Low and Middle-Income Countries (LMICs), where access to basic health services may be limited and the burden of vaccine-preventable diseases is large [7].
No evidence of adverse pregnancy outcomes from the vaccination of pregnant women with inactivated virus, bacterial vaccine, or toxoid has been found [8]. Live attenuated vaccines pose a theoretical risk to the fetus, therefore, these vaccines are generally contraindicated during pregnancy [9]. However, potential risks of the vaccine must be balanced against the risk of infection, and in a recent systematic review no evidence of harm related to live attenuated vaccines, other than the smallpox vaccine, was found [10]. Vaccine safety surveillance is complex, especially during pregnancy, when the risk of adverse outcomes may change due to underlying maternal health conditions, quality of obstetric care and the exposure to infection or vaccination during pregnancy [8]. In LMICs, safety monitoring is further challenged by limited pharmacovigilance infrastructure [11]. Post-marketing safety surveillance of new vaccines used during pregnancy is important for the detection of any Adverse Events Following Immunization (AEFIs) in pregnant women and their infants and also to provide information on the potential risks to the fetus [8]. This has become very important now in the context of COVID-19 pandemic, where more safety data are needed for immunizing this vulnerable group.
Enhancing pharmacovigilance capacity is a strategic goal of the WHO's Global Vaccine Safety Blueprint [12]. In this context, the Global Vaccine Safety Initiative tested the establishment and operationalization of a global network of hospital-based sentinel sites for vaccine safety signal verification in the general population [13]. The present study of safety in pregnancy builds upon lessons learnt from the earlier proof of concept project [14].
The use of standardized case definitions can strengthen programs of immunization in pregnancy, by improving generated data and facilitating data interpretation and comparability across surveillance systems [15]. To this end, the Global Alignment of Immunization Safety Assessment in Pregnancy (GAIA) project, launched in 2015 and managed by the Brighton Collaboration [16], developed standardized case definitions for 21 key obstetric and neonatal terms. Within each case definition, multiple 'Levels of diagnostic certainty' are recognized, which take into account current scientific evidence and different levels of diagnostic capacity available in different research and geographic settings [17].
The primary objective of this prospective study was to assess the applicability of GAIA definitions for maternal exposure to vaccination [18] and for seven perinatal and neonatal outcomes (low birth weight (LBW) [19], preterm birth [20], small for gestational age (SGA) [21], stillbirth [22], neonatal death [18], neonatal infection [23], and congenital microcephaly [24]) in LMICs, in order to inform future vaccine safety studies. Specifically, we assessed the proportion of cases or exposures identified that met the GAIA def-inition at any level of diagnostic certainty and the proportion that could be classified to each level. In addition, we identified which missing data elements prevented identified cases and exposures from meeting the definition at the lowest level or a higher level of diagnostic certainty.

Methods
This was a prospective, descriptive, cohort study using a common protocol (S1) and routinely collected data at 21 sites in six LMICs (Ghana, Tanzania, Zimbabwe, Iran, India, Nepal) and one high-income country (HIC) (Spain), consisting of one primary care center, five secondary hospitals and fifteen tertiary hospitals. Each site had a maternity ward. Sites were selected using a 2017 study that employed site selection criteria and acceptable performance in a simulation exercise that tested capacity to access sufficient data of acceptable quality at the site level [25]. Table 1 lists characteristics of the participating sites. The first site started data collection on May 6, 2019, and the last site completed data collection on August 18, 2020. Detailed methods are available in the report [26].

Case identification, recruitment and data collection
All births at the sites were prospectively recorded during a oneyear period, and the following study outcomes occurring in the 28 days following birth were identified as part of routine care by the sites: LBW, preterm birth, SGA, stillbirth (antepartum or intrapartum), in-hospital neonatal death, neonatal infection (invasive bloodstream infection (BSI), respiratory infection or meningitis) and postnatally diagnosed congenital microcephaly. The outcomes were selected based on relevance in vaccine safety research and perceived ability to collect data on the outcome of interest. Cases were first identified by screening relevant data sources from the maternity and neonatal wards at the sites (e.g. labor room register, admission register, patient records; see full list in S2). Only study outcomes at the site were considered; no follow-up outside of the site was performed. Estimated rates of occurrence will be reported in a separate paper and are also accessible in the study report [26]. At each site, up to 100 cases of each study outcome were systematically recruited into the study (the first two cases per week, or all consecutive cases); informed consent was obtained from the mother. One hundred cases per outcome per site enabled the calculation of 20% relative precision around estimates of the proportion of cases meeting the GAIA definition, under the assumption that 50% of all cases met at least the lowest level definition. Exhaustive case report forms, including details on any vaccines received during pregnancy, were completed for recruited cases, based on existing routine medical records. Data sources included the mother's antenatal care records, the antenatal care card, and inpatient records (full list in S2).
Study site staff were trained on the study procedures. All the data were captured through an app-based electronic data capture system, SOMAARTH III [27] using tablets. Data quality was monitored centrally, and on-site monitoring visits and regular teleconferences with sites were conducted.

Statistical analysis
We developed algorithms for the GAIA definitions for LBW [19], preterm birth [20], SGA [21], stillbirth (antepartum and intrapartum stillbirth) [22], neonatal death [18], neonatal infection (bloodstream infection (BSI), respiratory infection, meningitis) [23], postnatally diagnosed congenital microcephaly [24] and maternal immunization [18] to assess the level of diagnostic certainty of the GAIA definition met by recruited cases, if any. Cases classified as Level 1, 2 or 3 were said to meet the GAIA definition. Level 1 represented the highest levels of diagnostic certainty (most specific, least sensitive), and Level 3 the lowest (least specific, most sensitive). Levels 4 and 5, if present, were not considered as those events did not meet the case definition. The GAIA definitions have been summarized in S3a. First, it assessed whether Level 1 criteria were met. If yes, then the case was considered classified to Level 1. If no, it assessed whether Level 2 criteria were met, and so on. For each definition, the applicability was assessed by calculating the proportion of cases or maternal immunization exposures meeting the GAIA definition and the proportion classified to each level, by site. The most common reasons for not meeting GAIA definitions or, for non-classification of level 3 cases to levels 1-2 were summarized (or described) for each outcome. We modified the GAIA definition so that criteria accepted at higher levels of diagnostic certainty ('higher levels of evidence') were also de facto acceptable at lower levels of diagnostic certainty. For example, in the case of LBW, we considered electronic scales (sufficient for levels 1 and 2) appropriate for a level 3 classification as well (S3b for further details). Several aspects of the maternal immunization definition were open to interpretation. For level 1, we interpreted 'date/time' as 'date AND time' and 'details of vaccine' as 'lot number AND EITHER name of disease OR name of vaccine'. For level 2, we interpreted 'details of disease' as 'name of the disease OR name of vaccine OR lot number'. For levels 1-2, primary sources such as the antenatal care card, vaccine card or vaccine register were required, and for level 3, secondary sources were accepted such as the patient case sheet or birth register.
For each outcome, the proportion of recruited cases that met the GAIA definition was stratified by country, health facility level (primary, secondary, tertiary/referral), and health facility ownership (public/private). A Chi-square test was used to assess whether there were any significant differences between the categories.
Double independent programming of all analyses was performed using R version 3.6.0 [28] by the company, P95 Epidemiology and Pharmacovigilance and Stata version 15.1 [29] by INCLEN Trust International. Output was compared and the differences were resolved.

Ethics approval
The study was approved by the WHO Ethics Review Committee (protocol ID: ERC.0003114), and by local and national committees as appropriate [26]. Table 2 shows the number of cases recruited for each outcome by site. The number of cases recruited was highest for LBW and preterm birth, and lowest for meningitis, respiratory infection and congenital microcephaly. Table 3 shows exposure to (any) vaccination during pregnancy for each site. The median percentage of mothers identified by the sites as vaccinated during pregnancy was 82.4% (interquartile range (IQR): 61.2-88%); the exposure status was reported as unknown for 16.2% (median) of mothers (IQR: 5.6-31.7%).

Results
The site-specific percentage of cases or exposures that met the GAIA definition and, among those, the percentage that were classified to levels 1-3 have been summarized for all study outcomes in Fig. 1. Further, median and site-specific results are also available in S4. Differences in percentage of cases meeting the case definition by country, health facility level and public vs private ownership are available in the report [26].
LBW: Nearly 100% of recruited cases with LBW across the sites met the case definition (median: 100%; IQR: 100-100%). At most   sites, all the cases within one site were classified to a single level of diagnostic certainty due to the standard operating procedures for birthweight measurements at the sites. Preterm birth: All recruited preterm birth cases met the GAIA definition. At most sites, classified cases were spread over the three levels of diagnostic certainty. At least 10% of the cases at each site were classified at level 3.
SGA: The percentage of SGA cases that met the GAIA definition varied widely among the sites (median: 63.5%, IQR: 0-79%). At six sites (out of 20, one site did not routinely diagnose SGA) no cases met the definition. The majority of the classified cases were classified to either level 2 or 3.
Stillbirth: Stillbirths were recruited at all sites, except for one site in Spain where no stillbirths were observed during the study period. At all but two sites part of the recruited stillbirth cases met the GAIA definition; all sites had cases that did not meet the GAIA definition. The median percentage of stillbirth cases that met the GAIA definition across sites was 66.7% (IQR: 45-89%) for antepartum and 37.3% (IQR: 25-58%) for intrapartum stillbirths. Most classified antepartum stillbirths were classified as level 1 or level 3 and the majority of intrapartum stillbirths were classified as level 3.
Neonatal Death: Nearly all recruited neonatal death cases met the GAIA definition (median: 100%; IQR: 100-100%). All but one of the classified cases were classified as Level 1. A single case was classified as level 3.
Neonatal Infection: At all but one site, part of the neonatal infection cases met the GAIA definition. All sites recruited BSI cases. The median percentage of recruited BSI cases that met the GAIA definition across sites was 93% (IQR: 57-98%). At one site no cases met the GAIA definition. Level 1 was reached for some of the cases at 11 sites. At 16 sites, most classified cases were of level 2. Thirteen sites recruited meningitis cases, less than a quarter of all cases met the GAIA definition (median across sites 0%; IQR: 0-33%). Two cases were classified to level 1, four to level 2 and nine cases to level 3; 49 cases were not classified. Thirteen sites recruited cases of neonatal respiratory infection, the median percentage of cases that met the GAIA definition across sites was 100 % (IQR: 83-100%). Most classified cases were classified to level 3.
Congenital Microcephaly: Seventeen sites (out of 20, one site did not routinely diagnose congenital microcephaly) recruited cases of congenital microcephaly. At five of these sites none of the cases met the GAIA definition. The median percentage of recruited congenital microcephaly cases that met the case definition was 67% (IQR: 0-93%). Most cases were classified to level 2, with fewer cases meeting level 1 or 3.
Maternal immunization: The percentage of identified maternal immunization exposures that were classified using the GAIA definition was close to 100%. The majority of exposures were classified at levels 2 and 3, level 1 was reached by two sites.

Reasons for not meeting levels of diagnostic certainty
The reasons for not meeting the GAIA definition to at least level 3 (least specific) were definition-specific (the most frequent reasons are summarized in Table 4, full details are available in S4). A recurrent limiting factor for level 3 cases not meeting levels 1-2 was the GA assessment, particularly that no or insufficient information on 1st or 2nd trimester ultrasound was available. For maternal immunization, the most important reason for not meeting levels 1-2 was the absence of a primary source documenting vaccine exposure during pregnancy, such as the antenatal card or a vaccine card.

Applicability of GAIA definitions
In this study, the applicability of GAIA definitions for seven perinatal and neonatal outcomes and for maternal immunization were assessed using routinely collected data, primarily at tertiary level referral hospitals. The definitions for LBW [19], preterm birth [20], and neonatal death [18] were applicable at the study sites, as nearly all cases recruited by the sites met the GAIA definitions. For neonatal death, nearly all cases met the criteria for level 1 of diagnostic certainty. Birth weight (BW), a key parameter for all the three outcomes, was routinely measured and recorded at all the sites, contributing to the applicability of these case definitions. Kochar et al. also successfully applied the preterm birth case definition using both prospective and retrospective data [30]. A significant proportion of cases of stillbirth [22], SGA [21], congenital microcephaly [24] and neonatal infection [23] (particularly meningitis) did not meet the GAIA definitions, hence these definitions were less applicable at the study sites. This study also gave insight into site-specific levels of diagnostic certainty and pinpointed aspects preventing higher levels of diagnostic certainty from being met, both of which will help inform the design of future vaccine safety studies in pregnancy. The main barrier to obtaining higher levels of diagnostic certainty (for all outcomes except low birth weight, neonatal infection and maternal immunization) was the lack of/ no access to sonographic documentation in first and second trimester and its impact on GA assessment. Better clinical record keeping or documentation of GA in the early stages of pregnancy as part of routine care across caregivers would improve assessment of outcomes requiring GA. In this study, we evaluated how certain we were that a case diagnosed by a site was truly a case; one limitation is that we did not assess how certain we were an outcome was truly absent if not diagnosed. This was not possible as the study was records-based.
The definition for maternal immunization [18] was applicable at the study sites. However, at 5 out of 21 sites, most exposures were classified to level 3, for which no formal documentation of the vaccination is required. The usefulness of level 3 may be limited in vaccine safety surveillance and studies due to the sparsity of information on vaccination. Level 1 was achieved at only two sites, which routinely collected the batch number for vaccines administered as part of antenatal care in their facility. Not all elements of level 1 are necessarily required when studying neonatal outcomes in the context of maternal immunization safety, such as time (hour) of vaccination and batch number. Whereas level 2 does not require the exact date of vaccination (only month and year), however, this information is key to understanding vaccine safety at different stages of fetal development. In the study, due to a mistake in the design of the case report form, 0.6% of identified exposures could not be classified. If the site indicated that the number of doses was not known, the question on data sources was not prompted and therefore not completed, preventing classification to any level. If the number of doses was not known it is likely the source was secondary, which would have enabled classification to Level 3.
Information on maternal immunization was collected from documents available at the sites at the time of case recruitment, frequently the antenatal care card (often allowing to reach level 2 of certainty), or at some sites, the patient record (e.g., whether vaccination was done as per recommendation, often enabling classification only to level 3); no interview with the mother was conducted as part of the study. Furthermore, the antenatal care card belonged to the mother and was normally not available post-discharge. Vaccine exposure status for some mothers was unknown, which might have resulted in missed exposures. Had interviews been conducted, the number of exposures classified to level 3 (instead of unknown) would likely have been higher. More intensive efforts and outreach may be critical for retrieval of information on vaccination status and details of the vaccination. As different COVID19 vaccines are being deployed worldwide and may be used alternatively or simultaneously in each country, the rigorous recording of vaccination exposure in pregnant women is of paramount importance to enable safety monitoring. Advocacy with health program managers and policy makers is needed for improved documentation and inclusion of additional fields vital for maternal immunization pharmacovigilance in the antenatal care card, and increased digitization could enhance reporting and utilizing immunization data in the context of vaccine safety.
We assessed whether the percentage of cases meeting the GAIA definition differed by country, health facility level and public vs private ownership. However, as sites were not selected to be representative of these categories and due to low number of sites in each stratification, we do not feel confident that any differences observed can be meaningfully extrapolated beyond this study.

Challenges in using the GAIA definitions
Based on our experiences in field-testing the GAIA definitions, we have suggested several improvements to the GAIA definitions that may lead to their easier implementation and increased standardization across studies.
Accepting higher levels of evidence at lower levels of diagnostic certainty: In the study, we accepted higher levels of evidence at lower levels of diagnostic certainty to prevent cases from ''falling in between" different levels and consequently not being classified. This has been previously documented for the stillbirth definition [25,30] but we observed the similar issues in the LBW, SGA, neonatal meningitis and maternal immunization definitions (details in S3b).
Consistency of GA and BW information: The requirements for GA or BW are not identical across all case definitions. The GA assess- *Unresolved queries sent to the site (n = 2 for low birth weight, n = 2 for neonatal death). **Exposure did not meet the GAIA definition due to a mistake in the design of the case report form (n = 24; 0.6% of exposures assessed).
A.L. Stuurman, A. Sharan, S. Jahagirdar et al. Vaccine: X 9 (2021) 100123 ment criteria are the same for preterm birth, stillbirth, neonatal death and SGA. However, the GA criteria for congenital microcephaly differ, and are at times more stringent (level 1a necessitates LMP, level 2a does not allow for 1st trimester physical exam, level 3a necessitates LMP and does not allow for other measures such as BW) and sometimes, more lenient (level 1a allows for second trimester ultrasound scan). In addition, for neonatal death level 1, either GA level 1 or BW is necessary for classification, however if GA were assessed using BW it would result in GA level 3. BW requirements in the LBW case definition were based on the details regarding timing of the measurement and scale specifications. The criteria listed in the SGA case definition differs for SGA level 3, where scale specifications for SGA level 3a are more stringent (resolution of less than 50 g, tared to zero, calibrated) than in the LBW case definition. Furthermore, for the case definitions for neonatal death and preterm birth, no requirements are attached to the BW assessment.
Several other areas of improvement were identified in the maternal immunization (interpretation; see methods section), neonatal death (viability/maturity), stillbirth (GA, signs of life, combination of criteria), congenital microcephaly (chart use) and SGA definitions (S3c).
More countries are adopting policy recommendations for COVID-19 vaccination in pregnant women, based on WHO's interim recommendations to use COVID-19 vaccines during pregnancy, when the benefits outweigh potential risks. As pregnant women were excluded from clinical trials, safety profile of COVID-19 vaccines will need to be evaluated from post marketing safety data. In this context, it is of critical importance to ensure harmonisation of terminologies, definitions, and methods of assessment to allow comparability and timely assessment through meta-analysis of data collected worldwide.

Conclusion
This prospective study showed that the GAIA definitions for LBW, preterm birth, neonatal death and maternal immunization when vaccination was noted were applicable at the study sites, however, the ones for stillbirth, SGA, neonatal infections and congenital microcephaly were less so. The level of diagnostic certainty of GA was identified as a limiting factor to attaining higher levels of diagnostic certainty for multiple outcomes and several areas of improvement have been suggested following field-testing of the definitions. The introduction of COVID-19 vaccines, and their possible use in pregnant women, reinforces the urgency for improved documentation of vaccination and outcomes vital for maternal immunization pharmacovigilance.