Aim of the evaluation
The aim of this study is to evaluate the effectiveness (on under-two immunization coverage) and the extent of implementation (reach, adoption, implementation and maintenance) of the Project in Mozambique and Malawi.
A hybrid evaluation focuses on both effectiveness and implementation (27). We chose to use a hybrid evaluation embedded within the project, as it provides an opportunity to work to together with the implementing partners; simultaneously evaluate the effectiveness of the intervention on specific outcomes of interest (impact) and the extent of implementation of an intervention in reaching specific implementation outcomes (e.g., reach, fidelity, acceptability, sustainability, etc.) (27, 28) (See additional file 1: Populated checklist of study design).
Objectives
We specifically aim to provide evidence over a three-year period guided by the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE AIM) framework as follows:
Objective 1: Evaluation of the Effectiveness of the Project: The study will evaluate the effectiveness of the Project on under- two routine immunization outcomes (see Table 2) over a one-year period at baseline, midline and end line using interrupted time series analysis and other relevant analysis.
Objective 2: Evaluation of the implementation of the Project: Process evaluation will be applied to evaluate the implementation of the participatory solution development and solution implementation using mixed methods to obtain evidence of implementation success (reach, adoption, implementation and maintenance). A document and literature review, in-depth interviews, observations and surveys will be employed.
Objective 3
Gather evidence and develop a comprehensive framework to inform community responsive programming in routine immunization and guide scalability of community-based approaches on childhood immunization uptake and access into similar LMIC contexts.
Study Design
Guided by the RE-AIM framework, a prospective quasi-experimental and process evaluation of outcomes and processes will be conducted using mixed methods to prospectively evaluate the effectiveness (on under-two immunization coverage) of the intervention (participatory solution development and solution implementation) and processes of implementation (reach, adoption, implementation and maintenance). The RE-AIM framework is used in planning and evaluation, operating according to five dimensions; Reach, Effectiveness, Adoption, Implementation and Maintenance. RE-AIM as an evaluation framework helps to provide an overall descriptive account of the intervention elements by defining implementation outcomes for sustainable and effective implementation (29, 30).
The study will also draw on selected implementation science constructs from the consolidated framework for implementation research (CFIR) to understand the factors influencing implementation. The CFIR framework is an explanatory framework outlining effective implementation consisting of 39 constructs organized into five domains: Intervention Characteristic; Outer Setting; Inner Setting; Characteristics of Individuals; and Process (31, 32). A theory of change (TOC) will articulate the “path” to outcome generation for the identified implementation strategies of the intervention (see Fig. 1: sequence of evaluation and Fig. 2: Theory of Change).
Study setting
The study will be conducted at the Project’s intervention sites (health facilities and catchment areas) in four selected districts, two in Malawi and two in Mozambique located in Southern Africa. Intervention districts were selected by VillageReach to participate in the participatory solution development and implementation of the solution. In Mozambique, the evaluation will be conducted in Namarroi and Gile (both rural districts) located in Zambezia Province. In Malawi, the study will be conducted in 1 rural setting in Mzimba North and 1 urban informal setting in Lilongwe. Control sites for comparison will be selected in coordination with VillageReach staff and national, provincial and district Expanded Immunization Programme (EPI) leadership. To select control sites, we will consider variables such as comparable routine immunization rates, average walking distance to health facilities, geography, population size, other Non-Governmental Organizations (NGOs) operating and other research studies and/or similar programs.
Study population
For the evaluation of the effectiveness of the Project, we will use de-identified aggregate data on under- two routine vaccination outcomes. This population includes all children under the age of two eligible for vaccination within the intervention and control sites.
For the evaluation of the implementation, the study population includes participants in the Project in the four selected districts based on VillageReach’s inclusion and exclusion criteria of the participants (see Table 1). This includes caregivers, caregiver researchers, health workers, Expanded Immunization (EPI) officials at the district, provincial and national levels, community leaders, the VillageReach project, national and global staff and other relevant stakeholders involved in immunization programming or who can influence immunization access and uptake.
Table 1
Inclusion and exclusion criteria for the participatory solution development
Inclusion criteria
● Caregiver Researchers: these are women from the intervention districts who are caregivers of young children, who were hired and trained by VillageReach to conduct the participatory data collection activities, participate in data analysis and solution design.
● Participants who were part of the CBPR study and HCD and/or solution implementation as follows:
○ Health Care Workers specifically responsible for vaccination education and administration.
○ Caregivers of young children. Caregivers’ include parents or other family members or guardians who take the primary responsibility of bringing the child to access health services. Caregivers must have a child aged 25–34 months with health center documentation of at least one of the routine under two vaccinations completed by age 24 months as well as contact information (phone number, house directions). Eligible participants must be the primary caregiver of the child in question, meaning that they are the guardian who takes the primary responsibility of bringing the child to access health services.
○ National, Provincial and District Health Officials from the selected districts in Mozambique and Malawi (Gile and Namarroi; Mzimba North and Lilongwe).
○ Community Leaders who participated in the Project from the intervention sites.
● VillageReach project staff who were historically involved or currently involved in the study and solution implementation.
Exclusion criteria
● Caregivers who were not employed by VillageReach as a Caregiver Researcher.
● Health care workers who do not provide vaccines and have no experience of vaccinating children.
● Caregivers with no documentation of any vaccinations for their child; if their child is under the age of 25 months or over the age of 34 months and caregivers who are below the age of 18 years.
● National, Provincial and District Health Officials who have no responsibilities related to routine immunization and have not participated in the Project.
● Community leaders who were not involved in the Project and if they are not from the intervention sites.
● VillageReach staff who have not been involved in the conceptualization, design or implementation of the project.
|
Table 2 Sample size according to category and districts/location
|
HCW
|
Caregivers
|
Community leaders
|
District
|
Key Informants
|
Sum
|
Mzimba district
|
5
|
10
|
5
|
3
|
|
23
|
Lilongwe district
|
5
|
10
|
5
|
3
|
|
23
|
Key Informants
|
|
|
|
|
|
|
Provincial health staff
|
|
|
|
|
|
4
|
VillageReach Staff
|
|
|
|
|
|
3
|
Caregiver Researchers
|
|
|
|
|
|
4
|
Gile district
|
5
|
10
|
5
|
3
|
|
23
|
Namarroi district
|
5
|
10
|
5
|
3
|
|
23
|
Key Informants
|
|
|
|
|
|
|
Provincial health staff
|
|
|
|
|
|
4
|
VillageReach Staff
|
|
|
|
|
|
3
|
Caregiver Researchers
|
|
|
|
|
|
4
|
VillageReach staff
|
|
|
|
|
|
6
|
Total
|
|
|
|
|
|
120
|
Sampling
For the evaluation of the effectiveness of the Project, all de-identified aggregate records on under- two childhood immunization, from 2022 to 2024 will be sampled for the quantitative data (impact indicators). To compare outcomes of interest, the records will be considered from selected intervention and control sites in the districts at baseline, midline and end line of the pilot solution implementation.
For the evaluation of the implementation, purposive sampling in consultation with VillageReach will be used to select caregiver researchers, caregivers, health care workers (HCWs), VillageReach project staff, community leaders and other key stakeholders involved in the Project. The researchers will seek documents from VillageReach, search databases and use a snowballing approach to identify articles and other relevant documents.
Sample size
The following sample size will be considered:
For the evaluation of the effectiveness of the Project, to be able to conduct an impact model (as described in the data analysis), a hypothesis about the dropout rates will be proposed at the start, depicting the possible effectiveness of the intervention. We thus present here a rough sample size calculation subject to revisions once a decision has been made on “proxy” vaccines to be used to develop the matrix (specific under two childhood vaccines) will be considered for monitoring the dropout rates as some vaccines have full coverage in the different study sites) for the ‘dropout rates’ outcome.
Our quantitative sample size will constitute all available data under two immunization, which will be bounded by time (per month) and geographical location as specified for the intervention and control sites. Based on a recent study in Malawi, the average dropout rates for 2015–2016 using DPT1-Penta1 and DPT3-Penta3 and DPT1-Penta1 and MCV1-MR1 were 4.5 and 6.4, respectively, giving an average of 5.5% ~6% (33). For a 2-sided test with a 5% significance level, power of 80% and a 50% change in effect size (dropout rate reduction from 6 to 3%), a sample size of 88 is needed. A 50% reduction in dropout rates is a conservative estimate to avoid a type II error. A total sample size of 106 observations will be used for the impact model analysis to allow for any data quality issues and other threats to obtaining a well-powered sample size. The same calculation will be hypothesized for study sites in Mozambique, as accurate statistics on the actual dropout rates for the same vaccines used here could not be established.
For the evaluation of implementation, we plan to recruit a total of 120 participants for the interviews inclusive of both countries (10 caregivers, 10 HCWs and 10 key informants (KIs) (national, provincial, district and local/community stakeholders and VR project staff) in each of the four intervention districts). To recruit participants for interviews, we will contact potential participants via phone, email or personal contact where possible and explain the purpose of the study and interview before asking if they would be willing to participate. Participation in the study will be voluntary, and all participants who agree to participate will be asked to sign written consent. Data saturation will be considered when an adequate sample size is reached (34). Table 2 below shows a breakdown of the sample size according to the selected districts:
Data Collection
Data will be collected concurrently to evaluate the effectiveness on under- two immunization outcomes and the extent of the Project reaching specific process and implementation outcomes (e.g., adherence/fidelity to the public engagement approach, drivers and solutions identified, perceived quality of engagement, etc.) as follows:
1. Documents and literature review
As an ongoing process, we will conduct an extensive review of the literature and documents on CBPR and HCD practice, under- two immunization coverage and health systems resilience in line with the evaluation. Existing documents in Malawi and Mozambique, including grey literature from VillageReach, such as project reports that include attendance records, CBPR results, meeting notes from prototyping activities, emails, presentations, records from health facilities and Demographic and Health Survey (DHS) reports, will be sourced from VillageReach staff, database searches and snowballing of articles. Documents will be stored and organized in Mendeley, and a matrix will be developed to summarize information gleaned from the reviewed documents (35).
2. Semi structured interviews
We will conduct interviews using a semi structured guide with ten (10) caregiver researchers, twenty (20) health care workers, other key stakeholders and VillageReach project staff (country-based and global) in each setting involved in the Project. Interviews and knowledge, attitude and practice surveys (KAPs) with caregivers, HCWs, project staff, and other key stakeholders will be conducted as ongoing research on implementation outcomes across contexts at midline and end line of the implementation. The study will be explained to the participants and they will be provided with an information sheet. Informed consent will be obtained from participants by each signing a consent form.
3. Non-participant observations
An observation guide will be used during the HCD workshops to capture the depth and fast pace of the process (engagement, identification of drivers and solutions, etc.) (12). The purpose of the observations is to observe the engagement with all the participants involved in these workshops. As such, no names will be put down, but we will merely note the level of participation by categories of stakeholders.
4. HCD participant feedback survey
We will also obtain immediate feedback at the end of each HCD workshop using a feedback survey form. The survey form was adapted from Bartlett et al. (14) and adjusted to suit the objectives of this evaluation (13).
5. Collection of health and demographic data on under- two immunization
The evaluation team will obtain health and demographic data on under- two immunization dropout rates and other outcome measures from various sources at different time points (pre-intervention, mid-way and end of intervention) in both countries in the intervention and control sites. Data sources will include administrative data extracted from the Malawi Demographic and Health Survey (MDHS) and the Health Information System of Mozambique for Monitoring and Evaluation (SIS-MA) through district reports, health facility records, online and any other available data sources. We will conduct a data quality assessment and develop an appropriate mitigation plan prior to beginning the full evaluation. To address potential data quality challenges with administrative data, the primary outcome of interest will be dropout rates, which does not require having to rely on (often contested) population data. Dropout rates give an indication of the success of vaccination throughout the required stages. The WHO defines dropout rates as “the percentage of children that started their immunization series, but didn’t finish it for some reason” (36). It is calculated as the “[(coverage of initial vaccine dose minus coverage of ending vaccine) divided by (coverage of initial dose) x 100] (37). Proxy vaccines will be agreed upon for measuring the outcome depending on the under- two immunization schedules for Mozambique and Malawi.
Data Analysis
Document analysis will identify, sort and organize data from the documents related to HCD practice, CBPR, under- two immunization and the Project into specific themes (35).
For the evaluation of implementation, interviews will be tape recorded and transcribed verbatim for analysis. Atlas.ti will be used to capture the qualitative data and thematic analysis applied to analyze the data inductively and deductively with the RE-AIM domains, CBPR/HCD principles, relevant CFIR constructs and TOC guiding the analysis. The data will be coded and categories and themes developed to make sense of the data. Data from observations and surveys will be used to add to the thick description of the analysis. Complex systems thinking will be employed to assess the embeddedness of the intervention within a wider immunization ecosystem, as well as relationships and mechanisms that explain emerging implementation- and impact/effectiveness-related outcomes (intended and unintended consequences). Health care is seen as a complex adaptive system (CAS) that has interactions and interconnectedness among its different parts (professionals, patients, equipment) leading to health care delivery. This complexity can influence the impact of interventions as a reaction to internal and external catalysts (38). Additionally, a CAS lens will guide the interpretation of the analysis with the use of constructs embedded in CAS, such as phase transitions, feedback loops, historical path dependence, self-organization, existing norms and attitudes, in response to the identified solutions (39). Analyzed qualitative data will be used to form a holistic understanding of effectiveness and inform the development of appropriate quantitative indicators. Qualitative data will include caregivers’ acceptance of the solutions as well as insights into the implementation processes, adaptability and sustainability of solutions.
For evaluation of the effectiveness of the Project, an interrupted time series analysis will be used to analyze the under- two immunization dropout rates and other outcomes of interest. A time series analysis is a Quasi experimental design that involves the collection of data by creating a time series of an outcome at equally sized time points and statistically testing for changes in the outcome pre- and post-intervention (40, 41). The exact time of interception of the intervention should be known. In this study, information will be obtained from VillageReach when the pilot solution implementation will commence in both countries. Under- two immunization outcomes will be analyzed in the study settings in Malawi and Mozambique from the period prior to implementation (baseline), midway of the intervention and at the end line of intervention. Data collected from the control sites will be analyzed to depict predicted trends (counterfactual scenarios of change without the intervention) for comparison with data where the intervention took place (42). This will reduce bias from any time-varying confounders.
Overall, with the interrupted time series analysis, an impact model will be proposed as a hypothesis at the start, depicting possible effectiveness of the intervention on the outcome (e.g., immediate level change or a lag period before effect) based on the data (42). For seasonal adjustment, the model will be stratified according to calendar months. Segmented regression will be used to statistically analyze the level (immediate changes in the dropout rates) and slope (changes in trend) compared pre-intervention and post- intervention (41). The results will be presented in the form of scatter plots and line graphs to show the trends.
The above data collection process and the data analysis will assist in documenting communities’ perceptions and experiences of their involvement in participatory solution development and whether the solutions identified align with their needs and opinions. A CAS lens will also help uncover important drivers of change for a reduction in immunization dropout rates and immunization uptake and how innovative context-based solutions developed include health system strengthening activities for sustainability and health system resilience. Illustrative feedback loop diagrams will be used to show the interactions and connections among activities and partners that result in changes in immunization coverage (39, 43). This analysis will set a basis for developing a comprehensive framework to guide the application of the community-based approach to childhood immunization uptake in similar LMIC contexts.
Trustworthiness and credibility
For credibility, respondent validation will be used to acquire reflections from participants on preliminary findings, interpretations and consistency with their experiences. Triangulation of data sources (interviews, participant observations, surveys, etc.) will also be applied for a detailed set of results (44) and a thick description of findings.
Outcomes- effectiveness and implementation
Outcomes will be assessed at baseline prior to the pilot and at midline and end line. The outcomes collectively represent various elements of analysis for effectiveness and implementation (see Table 3 and Table 4). Studying outcomes in each context enables comparison and cross-contextual learning. Outcome indicators across different contexts within countries (intervention and non-intervention) and across country sites will be compared.
Table 3
Immunization outcomes of interest for effectiveness evaluation
Immunization outcome
|
Outcome description and interpretation (3,33,36,37)
|
DTP1-DTP3 under two immunization dropout [EL1] rates
|
DTP1–DTP3 immunization dropout rates at the selected intervention and control health facilities. Dropout rate is calculated as follows: (# of children who received initial vaccine dose minus # of children who received 3rd dose) divided by (# of children who received initial dose) x 100]. A decrease in dropout rate implies improved utilization and increase implies poor utilization.
|
Number of zero dose children
|
# of children who did not receive any routine vaccine/Children who have not received a single dose of DTP. Decrease in zero dose children implies reach and improved coverage.
|
% under two immunization coverage rates by antigen
|
# of children immunized during the last 12 months by antigen divided by number of children eligible (x 100). Increased % implies availability, access to and continuity of use.
|
% stock out rates per health facility
|
Product/antigen absence over a given period (# of facilities that experienced a stockout of a specific vaccine divided total # of facilities expected to provide vaccine x 100). High stock out rates implies problems in the supply chain and disruption of services.
|
Number of children vaccinated at the selected intervention and control health facilities and from the outreach sessions/mobile brigades
|
Absolute numbers of children vaccinated at the selected intervention and control health facilities and from the outreach sessions/mobile brigades
|
% of fully vaccinated children
|
Full coverage indicator (not for a specific antigen) to designate a child who completes the vaccination calendar during the 11 months of life.
|
Number of mobile brigades/outreach sessions planned and implemented;
|
Indicator depicting number of actual mobile brigades executed (difference between number of mobile brigades/outreach sessions planned versus executed)
|
Penta 3-MCV1 dropout rate
|
Measles vaccination dropout. Penta1 to MCV1 should be < 10%.
|
Dropout rate for first to second dose of measles containing vaccine
|
MCV1 to MCV2 dropout = 100x (MCV1-MCV2)/MCV1. The MCV1 to MCV2 dropout rate assesses the ability of the program to vaccinate beyond the first year of life
|
Table 4
Implementation outcomes for the evaluation of implementation
RE AIM Domain
|
Evaluation Focus
|
Reach: The absolute number, proportion, and representativeness of individuals who are willing to participate in a given initiative, intervention, or program (30).
|
It is important to have a defined target population to ensure that the implementation strategies improve access to and uptake of under two immunizations.
Furthermore, the intervention should have elements that are appropriate to meet the target audience needs. Appropriateness – the perceived relevance and fit of the solutions, from the perspective of caregivers and service providers. This is particularly important for solutions that for, instance, may have service implications, such as changing how service providers give immunization service.
|
Effectiveness: understanding comprehensive effects of a program, including unintended consequences
|
In this study, the effect of the intervention on immunization coverage and other measures will be assessed (see outcomes of interest in Table 3).
|
Adoption: the measurement of the uptake of solutions in each context, from the perspective of the intended target populations, such as caregivers and health service providers (Phase 2).
|
To be adopted, the intervention has to be acceptable, i.e., whether the approach for community engagement was considered agreeable/satisfactory. Acceptability will be explored from the perspectives of the intended beneficiaries and stakeholders, particularly the caregivers, health service providers and others involved in the HCD process.
Furthermore, adoption also speaks to the feasibility of the suggested solution in a given context. Strictly defined, feasibility is a retrospective look at the organizational requirements for implementing the solution successfully within a defined setting, in order to inform implementation strategy in other contexts.
|
Implementation: focuses on fidelity to an intervention: the extent to which the program is implemented consistently across different settings, staff, and patients.
|
Fidelity in this study refers to the use and adherence to protocol, where the HCD process and the solution proposed, are implemented as designed (how well VR adhered to the CBPR and HCD process and the solution as designed).
It also includes adaptations made.
|
Maintenance: has indices at the individual- (long-term effectiveness - and extent to which behavior is sustained 6 months or more after the intervention) and setting-level (sustainability beyond external funding). Maintenance also considers the “diffusion” of an innovation, whether the solution(s) has been integrated at the service/system level and how daily practice is influenced.
|
Sustainability (maintenance) requires a longer time frame and may not be realized in the given project time frame.
However, there are some aspects of sustainability that might be observed as the implementing partner aims to involve local ministries of health when creating solutions. There is thus a reasonable possibility of Let's talk about vaccines! approach and solution adoption and integration at the system/service level.
|
Theory of Change
The theory of change maps the “path” to outcome generation of the Project and will be refined as more evidence is generated during the evaluation. As a result of the participatory solution development, short-term outcomes include awareness of the implications for the lives of children under two; adoption of the proposed solutions; solutions tailored to the needs and aspirations of both the individuals who use the public immunization system (caregivers of children under- two) and those who deliver the care (health care workers and Agentes Polivalentes Elementares (APEs) in Mozambique or Health Surveillance Assistants in Malawi); continuous feedback; stakeholders’ buy in and an increased sense of value for the CBPR and HCD approach. Furthermore, medium-term outcomes foreseen as solution implementation commences include increased community involvement, investment in childhood immunization programs and recognized priorities, improvement in the number of fully vaccinated children and coverage in hard-to-reach populations. Closely interrelated is the strengthening of collaboration between different community institutions and users of the health system that directly or indirectly impact EPI programs, which facilitates feedback loops between communities and systems such as health care, education, NGOs and the strengthening of data systems for improved reporting of childhood immunization coverage. In an intertwined process, there will be equitable access to vaccines and improvement in under- two immunization uptake and access in specific settings. This results in the effective delivery of under- two childhood immunization which in turn leads to a reduction in dropout rates, an indicator of the success of immunization interventions (45). When children are fully vaccinated, long-term impacts, such as a state of healthy children and the absence of childhood disease outbreaks, will be observed.
Context
Adoption and replication of the Project will also consider the contextual factors prevalent within the healthcare system, such as norms and attitudes and historical path dependence. Phase transitions, self-organization and emergent behavior and health systems issues such as infrastructure, health information systems, financing, health workforce, leadership and governance, service delivery and supply chain issues will play a role in the adoption of the Project. Other contextual factors include the following:
-
Inequalities in childhood vaccination coverage within and among countries in spite of advances in childhood immunization programmes globally such as EPI.
-
Childhood immunization uptake and access is low, especially in hard-to-reach populations.
-
New solutions driven by international stakeholders with limited caregiver and health worker engagement in identifying barriers and solutions to improve vaccination coverage.
Assumptions
To improve childhood immunization uptake and access and reduce childhood disease outbreaks, we assume the following:
-
A participatory co-creation process in identifying barriers to childhood immunization and solutions will reduce vaccination drop outs, and by using this tailored participatory approach to solution identification and design, the solution would have high adoption by users.
-
Tailored solutions that meet the needs and aspirations of caregivers will improve childhood immunization uptake and access.
Dissemination
The results of the evaluation will be shared regularly with VillageReach and relevant stakeholders in the Project. The findings will be disseminated to local, provincial and national governments, implementers, funders and relevant organizations to communicate lessons learned from the process of identifying barriers and potential solutions to under-two immunization. Knowledge/analysis products will be developed in the form of reports, policy and technical briefs, and scientific articles and distributed through scientific and media platforms (such as the MESH community engagement network and Boost community), community engagement, stakeholder forums, academic conferences and local and national governments.