Active Surveillance of Adverse Events After COVID- 19 and Other Vaccinations: a Feasibility Study as Part of the CoVaKo Project

Nikoletta Lippert (  nikoletta.lippert@uk-erlangen.de ) Friedrich-Alexander-Universität Erlangen-Nürnberg: Friedrich-Alexander-Universitat Erlangen-Nurnberg https://orcid.org/0000-0001-8773-3618 Lisette Warkentin Friedrich-Alexander-Universität Erlangen-Nürnberg: Friedrich-Alexander-Universitat Erlangen-Nurnberg Thomas Kühlein Friedrich-Alexander-Universität Erlangen-Nürnberg: Friedrich-Alexander-Universitat Erlangen-Nurnberg Philipp Steininger Friedrich-Alexander-Universität Erlangen-Nürnberg: Friedrich-Alexander-Universitat Erlangen-Nurnberg Klaus Überla Friedrich-Alexander-Universität Erlangen-Nürnberg: Friedrich-Alexander-Universitat Erlangen-Nurnberg Susann Hueber Friedrich-Alexander-Universität Erlangen-Nürnberg: Friedrich-Alexander-Universitat Erlangen-Nurnberg Maria Sebastiao Friedrich-Alexander-Universität Erlangen-Nürnberg: Friedrich-Alexander-Universitat Erlangen-Nurnberg


Introduction
The COVID-19 pandemic is affecting everyone, forcing people into lockdown and to socially distance themselves. Vaccines are a promising remedy against this. Up to now COVID-19 vaccines from BioNTech/P zer, Moderna, AstraZeneca and Johnson & Johnson have been authorized by the European Medicines Agency (EMA). The pivotal studies have shown high e cacy and safety [1][2][3]. As e cacy and safety of a vaccine has a signi cant impact on its acceptance in the population [4,5], effective monitoring systems are important to ensure a safe and successful vaccination campaign. A reliable estimation of adverse events, especially of severe events affording medical attendance, is required. Given the rapid implementation of the COVID-19 vaccine, an active surveillance of side effects of COVID-19 vaccines is needed, including comparison with other commonly used vaccines to reduce the risk of bias. In Germany, as of Oct. 19, 2021 54,7 million people are fully vaccinated against COVID- 19 [6].
The Corona-Vakzin-Konsortium project (CoVaKo) analyses the e cacy and safety of COVID-19 vaccines and breakthrough infections in Bavaria, Germany [7,8]. One part of the project, the CoVaKo safety study, aims to capture adverse events in a real-world setting using an online survey. The observation group is planned to include participants after COVID-19 vaccination. The control group is planned to include participants after common other vaccinations, e.g. vaccinations against in uenza, shingles or pneumococcus. Posters, yers and lea ets will be used to recruit participants at vaccination centres and primary care practices. People aged 18 and older who have been vaccinated recently will be asked to register for the study on a secure website using REDCap (Research Electronic Data Capture), hosted at Universitätsklinikum Erlangen [9,10]. When registering, participants will be asked about sociodemographic characteristics, for information about the vaccination as well as about comorbidities using a modi ed German version of the Self-Administered Comorbidity Questionnaire (SCQ-D) [11][12][13]. Up to ve times in 18 weeks after vaccination, participants will be asked to complete an online survey about perceived adverse events and side effects, especially events leading to (in-or outpatient) medical consultation, medication intake and/or sick leave.
With a preceding feasibility study, we aimed to determine whether an online survey to capture adverse events after vaccination is practicable. The aim was, to (1) test the recruiting material for comprehensibility, visual design and motivation to participate, (2) to evaluate the online survey for comprehensibility, visual design, relevance of questions and technical implementation and (3) to assess practical implementation and realization in vaccination centres and primary care practices.

Study design and procedure
A mixed-methods design was used to combine elements of qualitative and quantitative research approaches in order to analyse the study's strengths and weaknesses, opportunities and threats (SWOT). Initially, the recruitment materials and the online survey were evaluated qualitatively. During test-roll-out the online survey was evaluated quantitatively and the feasibility both quantitatively and qualitatively.
The reporting of the study is based on the STROBE (Strengthening the Reporting of Observational studies in Epidemiology) recommendations [14], the Consolidated Standards of Reporting Trials (CONSORT) statement [15] and on the COREQ-Checklist (COnsolidated criteria for REporting Qualitative research) [16].
The procedure of the different project steps are explained in the following (Table 1). Qualitative evaluation of recruiting materials and online survey Before the test-roll-out, the recruitment materials and the online survey were qualitatively evaluated. In March 2021, three online focus groups (each 120 min) were performed by NL and MS, two female researchers at the Institute of General Practice who have profound knowledge in qualitative research. The only inclusion criterion was de ned as a minimum age of 18 years. Inclusion was independent of vaccination status, as this was not relevant for the evaluation. Exclusion criteria were lack of written informed consent and/or mental or cognitive impairments. Participants were recruited in the MVZ Eckental, a primary care practice associated with our institute (Institute of General Practice, Universitätsklinikum Erlangen). Snowball sampling was used to reach further participants (convenience sample). NL and MS did not know the participants before data collection. The participants received all versions of yers, lea ets and posters as well as the study information before the focus groups. In the focus groups, materials were discussed and questions were asked about comprehensibility and readability, attractiveness of the visual design and whether materials triggered interest and motivation to participate. The participants were given the opportunity to complete the online survey using different case vignettes (Supplementary Material). They evaluated the survey for technical errors, visual design, and readability on different devices (mobile phones, tablets and laptops) and different common internet browsers. We also discussed whether the study design and procedure as well as the inclusion criteria speci ed on the materials were understandably described. The focus groups were audio recorded, additionally eld notes were taken. After each focus group, the results were summarised. Based on the ndings, adjustments were made to the materials and the online survey, which were then presented to the third focus group.

Quantitative evaluation of the online survey
The study was tested across primary care practices and vaccination centres and was evaluated quantitatively. The online survey on adverse events was followed by a number of evaluation questions.
Questions were asked about the registration process, the structure of the survey, comprehensibility, completeness and importance of the questions. Also, the technical implementation and way of recruitment were assessed. Items were answered using a 5-point Likert Scale from 1 = strongly disagree to 5 = strongly agree. The evaluation form included a textbox for further comments. The test-roll-out and data collection for the evaluation form started on 17 April 2021 and ended on 14 July 2021.
Quantitative data were also collected on the participation rates of vaccinated individuals. The targeted minimum recruitment rate for the feasibility study was 50 participants/group. If we can reach this target in four weeks, we estimate that 3.000 participants/group can potentially be recruited within 9 months of the large-scale main study.
Qualitative evaluation of the practicability of the implementation Feasibility was qualitatively evaluated via interviews with General Practitioners (GPs) and heads of COVID-19 vaccination centres (May 2021). Vaccination centres in Central Franconia, a part of Bavaria, were recruited via direct email contact. Primary care practices were contacted through BayFoNet, a Bavarian Practice Based Research Network (https://bayfonet.de/en/). The centres and practices did not receive any monetary or non-monetary compensation. The centres' and practices' staff were asked to hand out the lea ets to patients after a vaccination. No further explanation by the staff was necessary.
Posters and yers could be used to raise more attention. A qualitative evaluation was conducted with the GPs and heads of the vaccination centres four weeks after the test-roll-out via telephone interviews (30 min) by NL and LW (female doctor in training/ researcher at the Institute of General Practice). Field notes were taken during the interviews. Questions were asked on the distribution of the material (e.g. who distributed and when, time consumption) and on patients' reaction (e.g. curiosity, aversion, lack of interest, etc., to adjust material if necessary). We also discussed study organisation, contact frequency to study centre, their motivation to participate in the main study and suggestions for recruitment of other practices/vaccination centres.

Data analysis
Qualitative evaluation. Focus groups were summarized in writing. NL and MS then used the method of mapping to structure and visualise the statements. Categories of the focus groups were comprehensibility and readability, visual design, technical errors and motivation to participate. Changes for the recruitment material and the surveys were then derived.
The eld notes of the interviews were categorised into distribution of the material, patients' reaction, organisation, motivation to participate, recruitment of other practices/vaccination centres. The results were discussed in the project team, not with participants.
Quantitative evaluation. The email addresses were checked for duplicates. If the same email address was entered in two different registrations, it was reviewed whether one person registered twice for the same vaccination, or whether the person and/or vaccination differed. In the rst case, the datasets were synthesised. In case of different vaccinations or different persons with the same email address, both datasets were considered individually. Various plausibility tests were used to check whether the participants correctly understood the questions and the given answer format. Data were checked for implausible answers, e.g. only participants with an interval between rst and second dose of COVID-19 vaccination of 14 to 91 days were included due to the vaccination strategy recommendations. In order to detect possible errors in sending out the invitation links for the subsequent surveys, it was checked whether the time corridor between vaccination and answer time was correct and whether the interval between two surveys was at least seven days. Sociodemographic characteristics, information about the vaccines and comorbidities are reported as proportion of all patients with a valid registration form. A drop-out rate over the observation period was estimated by the amount of fully completed surveys in relation to the sent email invitations to the participants remaining after data selection and preparation process. Comorbidity in form of a modi ed SCQ-D was calculated [12]. Data preparation, analyses and gures were performed using R Statistical Software (version 4.1.0, R Foundation for Statistical Computing, Vienna, Austria).
The rst evaluation form of each participant was considered for the analysis. The comments in the textbox were categorised in MAXQDA Plus 2020 by NL and MS. Categories were content, organisation and technical implementation. The results were discussed in the project team, not with participants.

Results
Qualitative evaluation of recruiting materials and online survey Study population. Fourteen participants took part in three focus groups (Table 2), two participants dropped-out due to personal reasons that were not related with the study. About equal numbers of women and men participated. Almost all of them were employed and had a university degree. One participant did not speak German as her native language. Not speci ed 00 000 Note. Focus group participants were divided into three groups.
Evaluation of recruiting materials. A lea et in size of the vaccination certi cate was rated most positively and practical as compared to posters and yers. Fig. 1 shows two (out of ve) different lea ets that were demonstrated to the participants and the nal adjusted lea et for the primary care practices.
In general, headlines were misleading because we only referred to COVID-19 vaccination but not the control vaccinations. Participants wanted headings to be more reader friendly, e.g. larger font. They were confused by the study acronym, CoVaKo, and preferred not to mention it in the title. Some participants also preferred the masculine form of language compared to gender-neutral language that uses the asterisk (e.g. Teilnehmer instead of Teilnehmer*innen; eng. participants). They also favoured the wording vaccine side effects instead of vaccine reaction or vaccine damage since it is most commonly known and/or perceived as more objective.
Study information should be listed as bullet points one below the other as opposed to alternating between left and right columns. Some participants preferred one image over two images on a page.
Participants appreciated the information on data protection, but found it too long. Some would want to know what will happen to the results and how data will be analysed. The options to register and access more information on the study via QR-Code or URL were evaluated practical by most participants. However, older participants stated they would not know how to use the QR-Code.
Final versions of the posters, lea ets and yers were created based on the feedback. We opted for two different versions: One version for vaccination centres that pointed to COVID-19 vaccination only and one version for primary care practices that targeted a variety of vaccinations. We included most of the feedback but decided to still use gender-neutral language. COVID-19 is now written in lowercase and placed at the bottom of the list to avoid too much focus on that vaccination. Information on data protection is now shortened with reference to a more detailed version on our study website. The adjusted version was used for the study test-roll-out.
Evaluation of online survey. Parts of the study information and registration form proved to be incomprehensible. More speci c information was requested regarding procedure and participating time, e.g. survey completion time in minutes. When asked about the vaccination received, they tended to report all the vaccinations received in the past (Fig. 2). The response options "Several/None" have been separated as this simpli ed the response decision. Originally, participants should only be recruited in the last two weeks after vaccination. As it later proved more practical to include participants at a later time as well, the time reference in the question about vaccination was adjusted accordingly.
Some buttons of the survey were in English (e.g. next page, submit) due to the software we used; here the participants expressed the wish that they were in German. When lling out the survey, it became apparent that not all termination criteria were correctly stored; no other technical problems occurred. We incorporated the majority of the feedback. We were not yet able to implement the German system language.
Also, before the test-roll-out we adapted the study procedure strategy. Initially, participants should register within 14 days after their rst dose in order to collect data at all survey times. In the test-roll-out participants could register up to 124 days after their rst or single dose, in order to receive at least one follow-up survey and to not lose too much information due to the vaccination pace.

Quantitative evaluation of the survey and feasibility
Study population. In 31 days, 2199 participants registered and received either a COVID-19 vaccination (99 %) or at least one of the control vaccinations (1 %) in the last 124 days. Few participants were born before 1951 (7 %) and 10.1 % were from medium sized towns. The distribution between rural areas, small towns and cities was balanced. The majority was employed (69 %) and had a university degree (48 %).
Most participants received BNT162b2 (BioNTech/P zer) or ChadOx1 (AstraZeneca) as rst COVID-19 vaccination (60 %, 28 %). For the second COVID-19 vaccination, BNT162b2 (BioNTech/P zer) and mRNA-1273 (Moderna) were administered most (68 %, 21 %). In ve cases, the second dose was cancelled. Named reasons were pregnancy, SARS-CoV-2-infection after rst dose, intolerance of rst dose or illness at date of second dose. Information on the second dose is missing in 446 cases due to missing information from the follow-up surveys. The mean body mass index was 26.2 (SD = 5.8).
Morbidity assessed with the mSCQ-D was in mean 1.6 (SD = 2.4) ( Table 3). Most participants (58 %) registered in the week of their rst dose or single dose, respectively. In the sixth week after rst/single dose registered nine percent and in the ninth week twenty percent.
The estimated recruitment rate in the test-roll-out was 7 % (2199 participants with 303050 distributed lea ets). Overall response rate of the surveys was very high with 84 %.
Evaluation. About 92 % of 1993 participants became aware of the study at a vaccination centre. The others heard about it at a physician's practice, their friends/family or on the internet (3 %, 3 % and 1 %, respectively). About 10 % did not ll out an evaluation form.
Most participants (strongly) agreed that the registration process was well structured, easy to understand that all relevant information was asked and all questions were important. Regarding the study process, most participants (strongly) agreed that the registration process was easy, the completion time was reasonable and the technical framework was manageable (Fig. 3).
Note. If a participant lled out several evaluation forms, rst answer was considered. Counts of observations in the bars are reported if n was > 50.
Optional feedback in the comment section. Suggestions in terms of organisation, technical implementation and content were made. After the registration, participants expected to receive an invitation link or a con rmation email immediately or shortly after their vaccination. For con dentiality and technical reasons, we did not send con rmation emails, but an information was added during the feasibility study at the end of the registration. The Redcap system language being English resulted in some participants getting an automatic and poor German translation of the survey in certain Internet browsers, although the survey was developed in German. A software update after the test-roll-out could solve this problem.
Concerning the content, some participants wished for a more differentiated list of symptoms and to specify the duration and onset of their symptoms. The item asking for personal perception of the link between vaccination and the complaints was di cult for some. Few participants suggested to add the answer option "I don't know".
As some participants shared an email address or registered twice, we implemented a disclaimer at the beginning of each survey with their personal information on type and date of vaccine, year of birth and gender. With this information participants could also give us feedback if their entered information was incorrect and/or could register again.
Qualitative Evaluation of the feasibility Study population. Eleven primary care practices and two large-scale vaccination centres, were recruited to distribute lea ets after vaccinations. In total three interviews with GPs and two interviews with vaccination centres staff were performed.
Evaluation. A total of 1600 lea ets were sent out to primary care practices and 28750 lea ets to the vaccination centres. The lea ets were distributed at different times, e.g. directly after the vaccination by physicians or in the waiting area. Participating partners said that the study is easy to integrate into the regular work ow. All preferred the small lea ets over yers and posters. Posters were not desired due to lack of space, especially from GPs. One GP reported a greater interest when lea ets were distributed by physicians instead of medical assistants. Physicians reported mostly positive reactions from vaccinated persons. Some reported little additional work due to questions from patients, but overall the questions did not intervene with their tight schedule. According to participants, almost all lea ets were distributed. Physicians suggested to recruit more GPs through mailing lists and by offering compensation.
Nine GP practices and both vaccination centres said they were willing to participate in the main project.

Discussion
Our study showed that capturing adverse events after vaccinations with an online survey is feasible. The qualitative evaluation of the recruiting materials led to some adjustments, e.g. adjusting the wording or designing two different recruiting materials for vaccination centres and primary care practices. After evaluation of the online survey self-checks (vaccination, gender, age) and feedback were built, so participants for example are informed that the survey will not be send out immediately after registration. The greatest challenge laid in recruiting individuals after a control vaccination.
The focus groups showed the relevance of testing the recruitment material to reduce misunderstandings. Despite adaptation of the study material following the focus groups, misinterpretation still occurred during the test-roll-out. Public attention is currently focused on COVID-19.
This made it easy recruiting patients after COVID-19 vaccination but potentially led to di culties in reaching the targets of participants for the control group. Additionally, few other vaccines were administered at that time. The evaluation of the recruiting materials and the survey showed that it is important to relocate the current general focus on COVID-19 to all vaccinations of the target group.
Reading instructions carefully is often problematic for participants in studies [17]. Therefore, to accomplish the focus relocation using unambiguous and clear formulations is even more important.
We aimed at 50 participants in each group for the feasibility study. We overachieved this for the COVID-19 vaccination group but barely anyone participated in the control group. Since recruiting participants for the control group proved to be more di cult, greater emphasis will need to be placed on the recruitment strategy for this group during the main study. Possible solutions for the main study are recruiting more primary care practices and company physicians as well as announcements through newspapers and radio for example right before the in uenza vaccination season.
The administrative burden of any study is a barrier to participation [18]. A successful implementation and integration of a study into existing processes and procedures of health care institutions might succeed by imposing as little effort as possible for the participants. The interviews with the participating GPs and heads of the vaccination centres showed that the study easily ts into the work ow. Although GP practices participated without compensation in the feasibility study we would tend to monetarily acknowledge their cooperation in the main study. The reason is that in the main study the duration of the data collection will be longer and nancial reward might motivate to recruit for the control group. Another way to motivate physicians might be to better explain them why the control group is methodologically necessary for the main question of the study for which recruiting participants turned out to be easy.
Noticeable in the recruiting was also that only a small proportion of the participants in the test-roll-out were older than 70 years; probably due to the vaccine recommendations and prioritisation [19]. Using an online survey might also be another reason, as older people are less con dent in their use of emails and the internet [20,21]. However, other studies show no group differences in age or gender when comparing different recruitment strategies [22]. In these studies, the online option was more likely to be chosen by better educated participants.
Due to the dynamic of COVID-19 vaccination recommendations, exibility in the study procedure proved to be important. Recommendations like the heterologous vaccination regime or changes in intervals between the two doses had to be considered in the survey [1-3, 19, 23-26]. The survey-question on the COVID-19 vaccination brand name and lot number was therefore adapted and repeatedly asked in the follow-up surveys. Automatically sent invitations and the survey itself had to be customizable. Initially it was planned to assess data from vaccinated persons after the rst or single dose of COVID-19, respectively. However, with this procedure a lot of information would have been lost, especially as our study started when 3,436,080 doses had already been administered to about 19.5 % of the Bavarian population [27]. We therefore adapted the strategy before the test-roll-out so that participants could register up to 124 days after their rst or single dose, in order to receive at least one follow-up survey.
Data of adverse events even if only assessed after the second dose are a valuable complement to the real-world evidence. For example, several weeks after recommendation of the heterologous vaccination regime in Germany there is still only little real-world evidence on safety of this regime [28][29][30].

Limitations
Due to the urgent implementation and limited nancial resources of the study, other data collection methods such as postal or telephone survey were not tested. Because of the time constraints caused by COVID-19 vaccinations already starting in practices, we could not lead more interviews with GPs. Data saturation could not be achieved because regular adjustments would have required continuous focus groups and interviews. COVID-19 vaccination prioritization of certain population groups, the study population of the test-roll-out might not be representative of the entire population in age, sex or educational background. In the progress of the vaccination campaign, quite different groups of people will be targeted over time. During the feasibility study, participants probably had higher grades of multimorbidity and were older or were predominantly medical personnel. These biases cannot be diminished whether now nor in the main study.

Conclusion
Testing of materials and surveys before implementation of a study is essential. Participating with health professionals, vaccination centres and physicians' practices, and incorporating their suggestions not only helps to reach many patients quickly, but also to ensure the practicality of the study procedures. For a successful implementation of the main study during the pandemic, exible adaptation of the study organization to dynamic changing structures and requirements proved to be essential. Recruitment of the control group will be the greatest challenge for the main study.  Sample question about vaccination received (before and after conducting focus groups)