Development and validation of a multi-dimensional COVID-19 vaccine hesitancy questionnaire

Background Since the emergence of the COVID-19 pandemic, vaccines have been developed to tackle the disease. However, many people worldwide were not confident enough to take the vaccines. Developing a questionnaire to measure COVID-19 vaccine hesitancy will give the health authorities and policymakers a clear picture to establish appropriate interventions addressing vaccine hesitancy among the community. Methods In this study, we used a mixed-method design over two phases. Phase 1 entailed a qualitative approach to developing the questionnaire, including a literature search, expert panel review, and focus group discussion. Phase 2 used a quantitative method for establishing the content and construct validity of the questionnaire via exploratory and confirmatory factor analysis (EFA & CFA). Internal consistency was checked using Cronbach's Alpha and intraclass correlation coefficient. Results We developed a 50-item instrument designed to measure COVID-19 vaccine hesitancy among adults in the state of Qatar. The study involved 545 adult participants. In terms of content validity, our study showed a value of 0.92 for the scale-level content validity index based on the average and a value of 0.76 for the scale-level content validity index - universal agreement. In the EFA, the Kaiser-Meyer-Olkin measure of sampling adequacy was calculated at 0.78, with statistical significance (P = 0.001). Regarding model fit indices of the seven-factor model, our findings showed an acceptable model-data-fit, with a relative chi-square: 1.7 (<3), Root mean square error of approximation: 0.05 (<0.08), PCLOSE = 0.41, Comparative fit index: 0.909, Tucker-Lewis index: 0.902, Incremental Fit Index: 0.910 and, Standardized Root mean square residual: 0.067 (<0.08). The seven-factor model of the questionnaire met the criterion of good internal consistency (Cronbach's alpha = 0.73). Conclusion This tool is deemed of methodological merits in terms of validity, reliability, and determining the underlying conceptual structure of COVID-19 vaccine hesitancy and its associating factors.


Introduction
There is now compelling evidence that the Coronavirus disease (COVID-19) pandemic has become one of the most devastating public health crises over the past years.It is a highly contagious and fatal disease that claimed millions of deaths globally.[1] It can spread rapidly from one person to another through respiratory droplets and causes various symptoms that differ in severity affect-ing nearly all human body systems requiring medical attention.[2] Such infection with this colossal burden required rapid action on developing evidence-based interventions, such as developing a vaccine.COVID-19 vaccine development started rapidly, and the first mass vaccination program took place in December 2020.Whereas the WHO has validated several COVID-19 vaccines, and all underwent quality control processes and clinical trials to ensure the vaccine's quality, safety, and efficacy.[3].
Despite all these procedures that have been done to ensure that vaccines meet safety standards, many people worldwide were not confident enough to take the vaccine.They were very hesitant about their ingredients, safety, and future side effect.Vaccine hesitancy is one of the significant issues that threaten public health globally.[4] It was described as a complex behavioral phenomenon with varying reluctance and refusal among populations despite the availability of the services.[5] It has often been associated with any new vaccine and poses a danger not only to hesitant individuals but also to their dependents and the whole society.It reduces the cover rate, prevents achieving 'herd immunity,' and enhances the occurrence of outbreaks from many preventable infections.Therefore, developing and implementing tools for measuring and addressing vaccine hesitancy is essential to detect concerns about vaccination in the population and monitor changes in vaccination behaviors.Furthermore, these tools should be context-specific, valid and provide a basis for data comparison with and possible integration with those generated in other parts of the world to fill any existing knowledge gap.[6] During the past few years, several tools were developed to measure vaccine hesitancy.These tools include the Parent Attitudes about Childhood Vaccines Survey, [7] Vaccine Confidence Scale, [8] Global Vaccine Confidence Index, [9] Vaccine Hesitancy Scale (VHS), [10] and recently, the Caregiver Vaccine Acceptance Scale.[11] These existing scales do not specifically pertain to vaccination against COVID-19.However, few studies adopted some items from existing scales, such as Huang's, who developed the COVID-19 Hesitancy Scale (HS) based on the Parent Attitudes about Childhood Vaccines Survey.[12] In contrast, another study in the United States developed novel items to measure adults' COVID-19 hesitancy.[13] On the other hand, Cvjetkovic ´S and his team used the 3C's model in developing COVID-19 (HS).[14].
Despite the wide availability of COVID-19 vaccines worldwide, the required reasonable vaccine coverage rate is still not achieved due to vaccine hesitancy.Improving the uptake rate of the COVID-19 vaccine requires an essential evaluation of the rate of vaccine hesitancy and identifying relevant factors among different populations.Therefore, a validated scale for measuring COVID-19 vaccine hesitancy is necessary.
Recently, few studies have developed and validated different scales to measure COVID-19 vaccine hesitancy.To our knowledge, there is a considerable research gap in the Middle East region on adapted or validated scales capable of assessing COVID-19 vaccine hesitancy among different populations.The state of Qatar has many nationalities with different cultures, beliefs, and backgrounds; therefore, developing and validating the COVID-19 vaccine hesitancy scale among such diverse people was necessary.Such validation will add unique value to the literature and future research in the Middle-East region and internationally.Consequently, our study aims to develop and validate the COVID-19 vaccine hesitancy scale for diverse populations residing in Qatar with satisfactory validity and reliability.Such a tool will give the health authorities and policymakers a clear picture to establish appropriate interventions addressing vaccine hesitancy among the community.

Study design and setting
We utilized a cross-sectional study design over a multi-phase process to develop and validate our questionnaire.This study is part of a large national project to assess COVID-19 vaccine hesitancy and its associated factors among the adult population of Qatar.It was conducted in the department of Community Medicine at Hamad Medical Corporation (HMC) in collaboration with the Health Protection and Communicable Disease Control (HP-CDC) Department at the Ministry of Public Health (MoPH) in the state of Qatar.The study was approved by the institutional research committee at the MoPH.

Questionnaire's items development
Initially, construct dimensions of the COVID-19 vaccine hesitancy phenomenon were addressed through an extensive literature review relevant to the topic.A pool of potential items (57 items) was used from previously validated questionnaires addressing vaccine hesitancy and its associated factors.[7,9,[15][16][17][18][19][20][21][22] Data obtained from the focus group were sorted and analyzed to check whether each item reflected the concept, which resulted in adding 25 items to the main pool.The authors then refined items based on the main domains without omitting necessary concepts.A list of 79 draft items was produced.Fig. 1 shows the process used for validating the questionnaire.

Coding of items in the questionnaire
The questionnaire comprised eight domains about COVID-19 vaccine hesitancy, including; socio-demographic factors, disease risk perception, conspiracy beliefs, vaccine literacy, attitudes regarding vaccination in general, and public trust in the health system organizations.Each domain was measured by a scale with multiple items and subitems.For Yes/No items, a coding of (1 & 0) was used respectively.Items having categories such as yes, no, and I do not know in the questionnaire were coded as 1, 0, and 0. Five and four-level Likert scales were both used and were coded as ''1," ''2," ''3," ''4," and ''5 00 and "1,'' "2,'' "3,'' and "4,'' respectively.Negatively keyed items were given reverse code, and the total score of each scale was calculated by summing up all items within the domain.Scales and subscales were retrieved through a literature search and cultural adaptation of similar studies addressing these scales in different populations.Table 2 shows a detailed description of each scale within the questionnaire.

Content validity
First, the questionnaire draft was sent to nine public health and vaccination experts to critically review each item and assess its relevance to the studied domain.Content validity indices (CVI) were calculated based on experts' ratings of each item's relevance as 'not relevant' (score of 1), 'somehow relevant' (score of 2), 'quite relevant' (score of 3), and 'highly relevant' (score of 4).I-CVI (individual level content validity), CVR (content validity ratio), S-CVI/Average (scale level content validity index with average method), and S-CVI/UA (scale level content validity with the universal average method) were calculated.Judgment on each item was made based on CVR & I-CVI.For the CVR, a decision was made based on Lawshe's critical values according to the number of panelists.[23] Regarding I-CVI, If It is more than 78%, the item is considered valid.If it is between 70 and 79%, it requires revision.If it is<70%, it is excluded.[24][25] ) .

Face validity
A focus group of nine experts in vaccination and public health was conducted and interviewed by the authors to explore their insight regarding the content of the questionnaires (English and Arabic versions).Experts evaluated whether the included domains, subdomains, and items successfully covered the addressed topic of the questionnaire.They also checked the questions' structure and common errors such as confusing, leading, or multi-barreled questions.The questionnaire was assessed for coherence, simplicity, accuracy, and cultural appropriateness as a whole and for each subsection to assess factors affecting COVID-19 vaccine hesitancy.

Translational validity
The questionnaire was written in English and then translated into Arabic, the main language in the state of Qatar.A bilingual clinician who was fluent in both English and Arabic, and who had expertise in the subject matter, conducted a forward translation of the questionnaire.This was followed by a panel review consisting of a vaccination expert, a field researcher, and the translator, who scrutinized the expressions and concepts in the Arabic version for any inconsistencies compared to the original English version.Any differences were resolved through consensus, resulting in the final Arabic version.Subsequently, a pilot study was conducted on a sample of 20 individuals between the ages of 18 and 65 to test the understandability of the Arabic version among the participants and interviewers.The pilot study checked if the Arabic version was understandable among the subjects as well as interviewers.The piloted sample was excluded from the analysis.

Psychometric validity of the questionnaire
In this phase, data collection was conducted through a telephone interview questionnaire from January 1st, 2021, to March 1st, 2021.

Construct validity
Exploratory factor Analysis (EFA) for assessing factor loading in the main domains of the questionnaire was used to cluster items under common themes, which interpret each factor according to the items loading on it, with a factor loading scale running between À1.0 and 1.0.Kaiser-Meyer-Olkin (KMO) sampling adequacy measure and Bartlett's sphericity test were performed before EFA.The KMO statistics range from 0 to 1, with KMO !0.7 and a P value of Bartlett's test is < 0.05, considered adequate for factorial analysis.The number of factors extracted is based on Eigenvalues (>1), parallel analysis, scree plot, and explainability of the factors.[26] To determine the rotation type, exploratory factor analysis with the direct Oblimin rotation was run first to calculate the inter-factor correlation.If the correlation coefficients were found to be less than 0.5, factors derived from EFA show an adequate discriminant validity, and Orthogonal rotation (Varimax) will be used.The final EFA was done using the orthogonal Varimax rotation.[27] A factor loading cut-off value of 0.40 was chosen to decide which items were highly associated with a given factor.[28].
Confirmatory factor analysis (CFA) was conducted based on a different sample of 300 participants, aiming to test the identified domains in the EFA.Following Sharif and Nia's recommendations for a good model fit, v2/df < 3 (P value of < 0.05), CFI, IFI, and TLI !0.9, RMSEA 0.08, and SRMR 0.08 were interpreted to indicate a good model fit.[29] Convergent validity of the questionnaire constructs was assessed using bivariate correlations, where a correlation coefficient !0.50 indicated a strong correlation and a coefficient of 0.30-0.49demonstrated a moderate correlation.[30] The discriminant validity was confirmed using cross-loading of indicator, Fornell & Larcker criterion, Heterotrait-monotrait (HTMT) ratio of correlation.Factor loading indicators on the assigned construct have to be higher than all loading of other constructs with the condition that the cut-off value of factor loading is higher than 0.7.[31] We used the statistical package of social science SPSS version 26 and SPSS AMOS 26 to run all the analyses.

Reliability
Cronbach's a coefficient, an index for internal consistency, was calculated for the sub-scales of the questionnaire to see the homogeneity of questionnaire items in each domain.A coefficient of 0.7 and above was considered internally consistent for the questionnaire.[32].

Demographic profile of the study subjects
Table 1 describes the socio-demographic and general characteristics of the study subjects.The mean age of the study participants was 41 years ± (SD: 10.7).The sample had a higher proportion of female respondents (55%) compared to males (45%).Most of the study subjects, 404 (74%), had a university degree or higher educational level.With regard to employment status, the majority of the participants were employed 86% (469), with almost half of them working in the healthcare sector (48%).In our study, we had the privilege of including participants from a diverse range of nearly 17 Arabic nationalities.Of the total participants, 48% (260 individuals) were Qatari nationals, while the remaining 52% (285 individuals) represented non-Qatari nationalities.Regarding income, 136 (25%) of the subjects had income less than 10 K QAR, 238 (44%) had income between 10 k and 30 K QAR, and 96 (18%) had income more than 30 K QAR.When it comes to the past history of chronic diseases, 37% (202) of the participants had a history of chronic diseases.In addition, about 11% (59) of the participants were diagnosed with COVID-19 at least once.The coding and description of the questionnaire scales and items are provided in Table 2.

Content validity
Content validity was measured for relevance, comprehensiveness, and representativeness of the content of the questionnaire and each of its subscales.The content validity index (CVI) was calculated from S-CVI/Average and S-CVI/UA indices through responses from 9 experts in the field of public health, vaccination, and psychology.Our study showed a value of 0.92 for S-CVI/ Average and a value of 0.76 for S-CVI/UA.Content validity indices and ratios of the questionnaire items are provided in (Appendix Table-7).

Exploratory factor analysis (EFA)
In order to conduct the EFA, the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was calculated at 0.78, and Bartlett's test of sphericity reached statistical significance (P = 0.0001), which indicated that these data were suitable for factor analysis.The total variance explained for the whole questionnaire was 58 %.In light of the above, we ran the analysis in the form of the principal axis factoring with the direct Oblimin rotation to assess for discriminant validity by checking the factor correlation matrix.The largest negative correlation was found between CCBS and GVCI (-0.222), while the largest positive correlation was found between MMI and CCBS (0.239).No correlation coefficients were found to be larger than 0.5; therefore, the factors derived from EFA revealed an adequate discriminant validity, and Orthogonal rotation (Varimax) is now indicated.[27] The next step of the factor analysis using varimax rotation.[33] An eigenvalue of 1 was used as a cut-off for determining the number of factors, though the scree plot also gave an estimate for the number of factors.Only eight factors were retained in the final questionnaire with 52 items, as shown in Table 3. (Rotated factor loadingsthrough EFA-for Questionnaire domains are provided in Appendix table 8.).

Confirmatory factor analysis (CFA)
Confirmatory factor analysis was run over the questionnaire items (52) yielded from the EFA.All the factor loadings ranged from 0.40 to 0.98.Results of the final model with the SEM are shown in Fig. 2, where the latent factors were presented in circles, and indicator items were described in rectangles.With regard to Raykov's composite reliability, seven factors showed values more than the acceptable level of 0.7.In contrast, one factor with two items showed factor loading less than 0.4, and items were removed, resulting in a seven-factor model.For convergent validity, the average variance extracted (AVE) values of adult vaccine literacy, whether functional or interactive, critical and GVCI factors, were above 0.5.On the other hand, AVE values of CPRS, CCBS, CVHS, and MMI were below 0.5, but items loading for these factors were acceptable for convergent validity >0.4.Hence, we preserved these factors in the model.(Table 5).Regarding the discriminant validity, the inter-correlation values between the seven latent variables were less than the squared root of AVE for each factor (Table 6).Regarding model fit indices of the seven-factor model, findings showed an acceptable model-data-fit, with a relative chi-square: 1.7 (<3), RMSEA: 0.05 (<0.08),PCLOSE = 0.41, CFI: 0.909, TLI: 0.902, IFI: 0.910 and, SRMR: 0.067 (<0.08).Consequently, the 50-item questionnaire evaluating COVID-19 vaccine hesitancy and its associated factors has good psychometric properties and model fit to observed data.

Reliability testing
We calculated Cronbach's a for the entire Questionnaire (a = 0.73) as well as for each subscale or domain to test for the reliability of the resultant factors and obtain a coefficient that can be used individually.Table 4 shows Cronbach's a for each scale in the questionnaire.All were above 0.80, which can be considered solid when it comes to internal consistency.

Discussion
Over the past three years since the emergence of COVID-19, the number of studies addressing hesitancy toward the proposed vaccines has been enormous.Despite this large number of studies, a discrepancy emerged between the study results regarding the extent and the determinants of such phenomenon.[34][35][36][37] This study aimed to develop and perform a detailed validation of a questionnaire that can evaluate an individual's COVID-19 vaccine hesitancy and its psychological antecedents among the adult population of Qatar.Our questionnaire consisted of 50 items, organized into seven subscales, including the vaccine hesitancy scale, adult vaccine literacy scale, COVID-19 perceived risk scale, COVID-19 conspiracy scale, and trust in the health system scale.Items were generated from both a literature search of already existing tools and the perspective of public health and vaccination It is a 14-item scale divided into three scales: functional vaccine literacy (VL) (items number 1 to 5); interactive VL (items number 6 to 10); critical VL (items number 11 to 14).Functional VL questions are more about language capabilities while the interactive/critical questions are more about the problem-solving and decision making.As vaccinations in adults are not mandatory, the VL scales are preceded by two filter questions.Before completing the functional VL section, participants have to reply to the following question: ''Have you ever read vaccine materials, such as leaflets or posters in doctor's or public health offices, recommending vaccinations?".If the reply to this question is affirmative, the person will be allowed to fill in the following section of the questionnaire.Also, before completing the interactive/critical section, participants have to reply affirmatively to the question, ''Have you ever been advised to vaccinate yourself against one or more diseases?Like Flu, Tdap, and Pneumococcal vaccines".( Source of information regarding vaccines 1 It's one question that askes about the main and reliable source of information regarding vaccination in general, whether from the healthcare organization like pediatricians, family physicians, general practitioner, other healthcare workers, scientific literature, and from non-health sector, such as mass media (TV & radio), the Internet (Wikipedia, Facebook, Twitter, and YouTube), social networks, relatives, friends or colleagues or others.9-Medical Mistrust Index (MMI) It's a 4-point Likert scale measuring mistrust in health care professionals and organizations with four responses, ranging from strongly disagree to strongly agree.(Coded: 1, 2, 3, 4).experts, and this approach ensured the validity of the tool at the very beginning of the study.
Regarding the socio-demographic factors, the mean age of the participants was higher (40 years) than that reported in similar studies.[13,38] There was no gender difference in term of the responses to the questionnaire, and the educational level of the study participants was a university level and higher, which comes in concordance with results from similar studies.[13,38] In addition, the prevalence of the comorbidities, including COVID-19 infection among the study participants, was almost identical to the reported national figures, indicating an accurate representation of the participants to the whole population of Qatar.Regarding content validity, we calculated (I-CVI), (S-CVI), and CVR to measure the content of the questionnaire items and as a whole.[23][24][25][26][27][28][29][30][31][32][33][34] The number of experts (n = 9) was considered sufficient for content validation.[25] An I-CVI of 0.78 or higher is considered excellent.The I-CVIs of all items in the questionnaire ranged from 0.67 to 1.00, with only two items having an I-CVI of<0.78.As for the CVR, a ratio between 0.60 and 1.00 is considered excellent.The CVR of all items ranged between 0.11 and 1.00, with nine items<0.6 that were eligible for deletion or modification.
The EFA was employed to adequately measure a construct involving the grouping of variances under different factors.In our study, EFA detected consistency in eight grouped factors that explained 58% of the overall variability in the data.This process allowed us to exclude three items from the initial 55-item questionnaire; one related to COVID-19 risk perception (CPRS3) and two related to conspiracy beliefs associated with the pandemic and the vaccine (CCBS4 & CCBS8).However, in the principal constructs yielded through exploratory factor analysis, the questionnaire had a sufficient psychometric property in terms of construct validity.Factor loadings of the items related to each of the eight factors were also satisfactorily ranging from 0.469 to 0.874, with a variance greater than 50% for the majority of items, which was less than other validated instruments described in the literature.The relatively small sample can explain this compared to a large number of items and variables.[38][39].
Confirmatory factor analysis was carried out to establish the factorial validity and structure obtained by the EFA.Our findings indicated that the seven-factor model provides an adequate fit to the data after the deletion of the two items representing the eights factor (CVHS5 & CVHS6), reducing the scale from 52 to 50 items.It has to be noted that the confirmatory factor analysis shows some items (CVHS3,4,9, CPRS4,5,7,8, CCBS3,6, MMI7) with a coefficient under the 0.50 value.However, this is the first psychometric evaluation of the measure, which can be a good guideline for future analyses among different populations, considering the fact that model fit indices of the 50-item model are acceptable.Moreover, Brown et al. stated that confirmatory factor analysis of factor loading of 0.40 or more is often interpreted as salient.[40] The reasons for the inadequate internal stability of the aforementioned deleted items or those with lower factor loading in both EFA and CFA may be due to difficulty in understanding the items' statements by the participants.In addition, we conducted the study in the early phases of vaccine distribution.The COVID-19 pandemic was rapidly evolving with various vaccine manufacturers, and governmental policies and mandates were set to, directly and indirectly, guide people toward vaccine uptake.
Keeping in mind the complex and context-specific nature of vaccine hesitancy, the results of the present study highlight the multi-dimensionality of the COVID-19 vaccine hesitancy phenomenon, especially in a multicultural population, including healthcare workers, public health professionals, and the general population.Additionally, this tool focused on different variables (''Disease risk perception," ''Conspiracy beliefs," ''Trust in the health system," ''Adult vaccine literacy," and ''General Attitudes toward vaccination") that are distinct from other scales that only focus on hesitancy alone.One of the limitations of the current study is the use of a cross-sectional study design that does not permit establishing temporality and causality between variables.Another limitation is the small samples used in the steps of analysis, especially in the CFA.

Conclusion
This tool is deemed of methodological merits in terms of validity, reliability, and determining the underlying conceptual structure of COVID-19 vaccine hesitancy and its associating factors.The validity of the COVID-19 vaccine hesitancy scale would need to be further evaluated across different populations and time to increase the validity and generalizability of the tool.

Fig. 1 .
Fig. 1.A flowchart depicting the process used for developing and validating the COVID-19 vaccine hesitancy questionnaire.

Table 1
Sociodemographic characteristics of the study participants.

Table 2
Description of questionnaire constructs.It is the first part of the questionnaire.It was developed through an extensive literature review.It includes (age, gender, nationality, religion, employment status, occupation, income, level of education, co-morbidities, smoking, pregnancy status for females, and history of a previous diagnosis of COVID-19) -item scale designed to capture the endorsement of some COVID-19 conspiracy theories and beliefs currently popular among the public.The scale items are obtained through extensive literature review and adaptation of different scales in the literature to suit the local situation in Qatar.(Coded: 1, 2, 3, 4, 5).

Table 3
Eigenvalues and variance for the addressed variables among study population.

Table 4
Internal consistency of the questionnaire domains indices.