Measuring vaccine hesitancy: The development of a survey tool

is to better inform the development of appropriate strategies and policies to address the concerns expressed, and to sustain conﬁdence in vaccination. The Working Group developed a matrix of the determinants of vaccine hesitancy informed by a systematic review of peer reviewed and grey literature, and by the expertise of the working group. The matrix mapped the key factors inﬂuencing the decision to accept, delay or reject some or all vaccines under three categories: contextual, individual and group, and vaccine-speciﬁc. These categories framed the menu of survey questions presented in this paper to help diagnose and address vaccine hesitancy. open access article the BY license (http://


Introduction
Despite compelling evidence of the value of vaccines in preventing disease and disability and in saving the lives of millions of children every year, vaccine hesitancy has become a growing focus of attention and concern [1-3], given its potential to lead to vaccine delays and refusals, and to risk the public health consequences of vaccine preventable disease outbreaks.
Research to date has shown that the reasons for and expressions of vaccine hesitancy are highly varied [4][5][6][7] and need to be better understood in order to appropriately address emerging concerns. are -i.e. which of the various possible reasons outlined above are driving their hesitancy, and where the hesitant individuals are located, i.e. in which geographic, socio-cultural or political context that may be contributing to the hesitancy. Not all vaccine hesitancy is specific to a vaccine or vaccination, and some influences are well beyond the scope of an immunization programme, however they must be understood in order to know how to best minimize the hesitancy.
This paper presents a set of survey questions which were developed following an initial review of existing vaccine hesitancy surveys, and consultations within the SAGE Vaccine Hesitancy Working Group as well as with SAGE members [9]. Additionally, proposed vaccine hesitancy questions for inclusion in the WHO-UNICEF Joint Reporting Form (JRF) [10] on immunization, which are completed by national immunization programme managers annually, were explored as another opportunity to capture an indication of the nature and scope of vaccine hesitancy at the national level.
In 2003, in order to begin to understand the scale of emerging vaccine concerns, one pilot question was included in the JRF, asking national immunization managers whether they had to manage negative media coverage about vaccines in the previous year. The 2004 JRF report [10] showed that negative media coverage was reported by countries in all WHO regions except in South-East Asia. The Americas and the European regions reported the highest levels of negative publicity about vaccines (30% of the countries in those regions). This initial survey question pointed to the scale of vaccine hesitancy already emerging over a decade ago. In the context of the Global Vaccine Action Plan (2011-2020) [11] and the SAGE work on vaccine hesitancy, new questions were considered and piloted for inclusion in the JRF in 2012, 2013 and 2014. While these JRF questions are limited in scope, they can serve as a valuable routine indicator of vaccine hesitancy in the population and point to areas where more in-depth research is needed.
In the feedback from the regions and countries concerning the 2012 JRF Indicators and the Immunization Managers' Survey [12] many countries called for a list of survey questions to help them assess vaccine hesitancy. More detailed survey questions identified through the SAGE process, discussed below, are an additional resource to more specifically understand who is hesitating and what the nature of the concerns are, in order to inform an appropriate response.

Methods
As part of the national JRFs completed annually, two indicators for vaccine hesitancy were pilot-tested in the Americas and the European WHO regions. The two indicators were also tested in a self-administered questionnaire distributed at the East, South and Central African Regional Immunization Managers' meetings in 2013. Pilot-testing the feasibility of these indicators within these various regions allowed for input from a broad range of high, middle and low-income countries. The questions were accompanied by the definition of vaccine confidence and prepared in English, French, Russian, and Spanish versions.
In the 2012 JRF pilot test, the focus was on vaccine confidence, rather than on the broader issues driving vaccine hesitancy, in order to be in harmony with the Global Vaccine Action Plan indicators. Specifically, the indicators for Strategic Objective 2: "Individuals and communities understand the value of vaccines and demand immunization both as a right and a responsibility" are: (1) "Percentage of countries that have assessed (or measured) confidence in vaccination at subnational level", and (2) "Percentage of un-and under-vaccinated in whom lack of confidence was a factor that influenced their decision." Modifications to the 2013 JRF included refining the indicators to encompass the wider scope of vaccine hesitancy reasons going beyond confidence to also include convenience and complacency, and to link with the revised Working Group definition of vaccine hesitancy. Question 2: Is this response based on or supported by some type of assessment, or is it an opinion based on your knowledge and expertise?

Process indicator
Question 1: Has there been some assessment of vaccine hesitancy or refusal among the public at the national or sub-national level?
Question 2: If yes, please provide assessment title(s) and reference(s) to any publication/report.
Parallel to the piloting of questions in the JRF, additional survey questions were collected to inform a more detailed survey tool.
A review was undertaken to identify all survey questions and tools mentioned in the articles collected in a systematic review of literature conducted on vaccine hesitancy [3]. A selection of articles (n = 2933) were re-screened for this purpose.
Keyword searches included: Search 1: 'Survey' and any one of 'accept/barrier/refus/trust/missed/delay/confidence/ partial/unsure/timeliness/hesitan' Search 2: 'Scale/Index/measurement/instrument/Questionnaire' and any one of 'Trust/Confidence/Hesitan' These keywords were chosen to identify either complete surveys or survey items that reflect the concern of vaccine hesitancy.

The JRF
The response rate to the first JRF pilot in 2012 was sub-optimal with only 14% (13/94) of countries completing the questions. The analysis revealed that 19% of all participating countries had done an assessment of the level of confidence in their country, indicating that vaccine confidence was an issue of concern in their country. In the assessments reported, lack of vaccine confidence ranged from 0% in Cuba, Dominica, Botswana and Sao Tomé & Principe, 1% in Germany and Brazil, 4% in Guatemala and Jamaica, 5% in Burundi, 8% in the Democratic Republic of Congo, and 10% in Romania, to 18% in Czech Republic and 19% in Uganda. These results demonstrate that the lack of confidence can be a significant problem, even in lowincome settings, such as Uganda, where suboptimal availability of services might be presumed to be the reason for under-vaccination rather than lack of confidence.
For the 2013 revision, 69% of countries (31/45) reported on indicator 1, which is a higher response rate to the indicator as compared to 52% (25/48) in the JRF 2012. In the 2013 JRF, 10 countries indicated having undertaken an assessment. This may be due to an increased number of assessments among the countries in the WHO European region, better understanding of the question due to the inclusion of a revised narrative, and/or the inclusion of both a national and a sub-national assessment in the indicator question in comparison to only a sub-national assessment in 2012. For those countries not responding to indicator 1, it remains unclear if nonresponse was a proxy for no assessment, lack of understanding, or lack of willingness to answer the question.
With regard to Indicator 2, in the 2013 pilot test, 36% (16/45) of the countries responded to the question and provided reasons for vaccine hesitancy. The response rate to this newly revised indicator was higher compared to the 2012 indicator where only 6% (3 out of 48) of the European region countries in 2012 had provided a measured or estimated percentage of un-or undervaccinated in whom a lack of confidence in vaccination was a factor.
The top three reasons for vaccine hesitancy reported in the 2013 JRF were (1) beliefs, attitudes, motivation about health and prevention, (2) risk/benefit of vaccines (perceived risks, experiences (heuristics)), and (3) communication and media environment. Major issues were fear of side effects of vaccination and distrust in the vaccine, lack of perceived risk of vaccine-preventable diseases and the influence anti-vaccination reports in the media.

Survey tool
One hundred and eight articles were reviewed for survey questions and 10 articles included complete survey tools on vaccine hesitancy, confidence or trust:
• Series of surveys with strong focus on trust in the influenza vaccine using the Trust and Confidence Model and Protection Motivation Theory [17]. • Nine questions developed to assess patient-health provider trust in post-partum mothers' relationship [18]. • The National (US) Network for Immunization Information (NNii) Survey Instrument [19].

Confidence -Healthcare and other sectors
Healthcare Confidence Index [20] All vaccines • The Vaccine Safety, Attitudes, Training and communication Project (VACSATC) present a list of core questions included in multiple surveys implemented across several European countries [21].
The work of Opel et al. was among the first to develop and validate a survey tool specific to vaccine hesitancy, the Parent Attitudes About Childhood Vaccines (PACV) survey. The PACV was originally developed by adapting items from previous surveys on health beliefs, conducting focus groups to produce additional items, submitting these items to a panel of immunization experts to remove items unlikely to be useful, and pre-testing the product on a group of parents. The result of this process was an 18-item survey encompassing the domains: immunization behaviour, beliefs about vaccine safety and efficacy, attitudes and trust [22]. This was then refined using a cross-sectional survey of parents of 19-35 month old children in a Washington State HMO 1 to assess the metric's construct validity and reliability [23]. Below is the set of questions adapted by the Working Group to have more global relevance, given that the tool was developed and validated for a high-income setting ( Table 1). Across all years, Search 1 identified 77 articles containing the keywords 'trust', 'confidence', or 'hesitan' (Fig. 1). Search 2 identified an additional 31 relevant articles. All 108 articles [13,15,[17][18][19]21, were reviewed for survey items pertaining to the components of the SAGE Vaccine Hesitancy Model of determinants.
The review of vaccine hesitancy research to identify specific survey questions revealed that much of the research on this topic is largely on cognition-based features such as knowledge, attitudes and beliefs and draws on two commonly used health behaviour models: the Health belief Model and the Theory of Planned Behaviour. Neither of these models adequately investigate the significance of social, economic and/or environmental factors as determinants of health behaviour, in this case vaccine hesitancy.
Questions that were identified through the literature search were then mapped against the key determinants identified (Tables 2-4) based on feedback from the Working Group, and were presented at a meeting on December 2013 for further discussion and feedback. It was clear that a number of determinants of vaccine hesitancy were not adequately addressed in the identified survey questions documented in the literature review. It was also agreed that while the matrix was a useful approach to identifying gaps in the questions, a more survey-ready format was needed for implementation.
The Working Group developed a compendium of three different types of survey questions (See Appendices): Core Closed Questions (Appendix A); Likert Scale Questions (Appendix B); and a set of Open Ended Questions (Appendix C). Some were derived from previously validated questionnaires, albeit in high-income countries only, some came from experts in the field, and others are newly proposed.

Discussion
Routine monitoring of vaccine hesitancy, such as through the JRF, could play a valuable role in identifying vaccine concerns early, in order to address them. Strategies to address any identified concerns, though, need more in-depth understanding of the nature of hesitancy, as well as who is hesitating. This more detailed understanding will need further research which could be initially guided by the menu of survey questions presented here, followed by additional qualitative research in areas where surveys do not provide adequate information to inform appropriate interventions. A standardized compendium of both survey and determinants questions could also enable intra-and inter-country comparison of the prevalence of, and the major determinants leading to, vaccine hesitancy, and support global assessment.
When countries use questions presented in this paper to monitor hesitancy, several points must be considered in question selection: (1) Refusal is not the same as hesitancy, i.e. counting only vaccine refusers will not capture hesitancy; (2) Vaccine hesitancy is defined as delay in acceptance or refusal of vaccines despite availability of vaccine services [125] and it is therefore essential to assess the different reasons why people are under-or unvaccinated in a particular setting to be able tackle vaccine hesitancy as distinct from access issues; and (3) Vaccine hesitancy may be specific to one or some, but not all, vaccines, so interpretation of surveys should not generalize findings to all vaccines unless that is stated in the survey response. All of the survey questions, whether from the general survey or from the determinant examples, need to be pilot tested and validated in all settings and then refined.
Given the dynamic and changing nature of vaccine hesitancy, the importance of ongoing monitoring cannot be overstated. A survey which reveals little hesitancy this year may have a different result next year. These trends need monitoring. Additionally, qualitative research to better understand the contextual and socio-cultural influences that may be contributing to vaccine hesitancy will be important in informing the most relevant strategies to engage hesitant publics, health providers and policy makers.

Limitations
As the term "vaccine hesitancy" is relatively new, the availability of existing survey questions is limited. Most are predominately designed for high-income settings, and few have been validated. Furthermore, the questions identified do not address all the determinants in the Vaccine Hesitancy Matrix and additional questions will need to be developed and validated.

Conclusion
In the final report on the work on vaccine hesitancy, SAGE "encouraged validation of the developed compendium of survey questions on vaccine hesitancy, which have been assessed and validated only in some high-income countries or not at all." SAGE also urged: WHO to develop capacity "for gaining behavioural insights that can be applied in an integrated fashion for prevention of many communicable and non-communicable diseases, as well as vaccine hesitancy" and stressed that "this will require a multi-disciplinary approach, involving sociologists, psychologists, anthropologists, experts in social marketing, communication experts, and specific disease and vaccine experts." The recommendation for bringing multiple disciplines together to understand and address the complex, context specific and dynamic nature of vaccine hesitancy -and its varied drivers of complacency, convenience and confidence -needs to extend to all immunization stakeholders from national immunization programmes and community based organizations to private sector partners and research entities.
Publics are changing, embracing their rights to information, and their rights to choice. A certain amount of questioning by new parents, or parents faced with new vaccines or combinations of vaccines, would be considered responsible. Keeping an open dialogue is fundamental to building trust. These survey tools are primarily aimed to better understand the public concerns and thereby better respond to the issues that are relevant to them.

Conflict of interest statements
The LSHTM research group "Project to monitor public confidence in Immunization Programs" has received research funding from Novartis as well as funding from GSK to host a meeting on vaccine confidence. Heidi Larson has done consulting on vaccine confidence with GSK.
None of the other authors had any potential conflict of interest. Some of the authors are World Health Organization staff members. The opinions expressed in this article are those of the authors and do not necessarily represent the decisions, official policy or opinions of the World Health Organization.