Leveraging Community Health Workers and a Responsive Digital Health System to Improve Vaccination Coverage and Timeliness in Resource-Limited Settings: Protocol for a Cluster Randomized Type 1 Effectiveness-Implementation Hybrid Study

Background Tanzania is 1 of 20 countries where the majority of unvaccinated and undervaccinated children reside. Prior research identified substantial rural-urban disparities in the coverage and timeliness of childhood vaccinations in Tanzania, with children in rural settings being more likely to receive delayed or no vaccinations. Further research is necessary to identify effective and scalable interventions that can bridge rural-urban gaps in childhood vaccination while accounting for multifaceted barriers to vaccination. Objective This protocol describes a type 1 effectiveness-implementation hybrid study to evaluate Chanjo Kwa Wakati (timely vaccination in Kiswahili), a community-based digital health intervention to improve vaccination timeliness. The intervention combines human resources (community health workers), low-cost digital strategies (electronic communication, digital case management, and task automation), a vaccination knowledge intervention, and insights from behavioral economics (reminders and incentives) to promote timely childhood vaccinations. Methods The study will be conducted in 2 predominantly rural regions in Tanzania with large numbers of unvaccinated or undervaccinated children: Shinyanga and Mwanza. Forty rural health facilities and their catchment areas (clusters) will be randomized to an early or delayed onset study arm. From each cluster, 3 cohorts of mother-child dyads (1 retrospective cohort and 2 prospective cohorts) will be enrolled in the study. The timeliness and coverage of all vaccinations recommended during the first year of life will be observed for 1200 children (n=600, 50% intervention group children and n=600, 50% nonintervention group children). The primary effectiveness outcome will be the timeliness of the third dose of the pentavalent vaccine (Penta3). Quantitative surveys, vaccination records, study logs, fidelity checklists, and qualitative interviews with mothers and key informants will inform the 5 constructs of the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework. The results will be used to develop an implementation blueprint to guide future adaptations and scale-up of Chanjo Kwa Wakati. Results The study was funded in August 2022. Data collection is expected to last from February 2024 to July 2027. Conclusions This study will address the lack of rigorous evidence on the effectiveness of community-based digital health interventions for promoting vaccination coverage and timeliness among children from sub-Saharan Africa and identify potential implementation strategies to facilitate the deployment of vaccination promotion interventions in low- and middle-income countries. Trial Registration ClinicalTrials.gov NCT06024317; https://www.clinicaltrials.gov/study/NCT06024317 International Registered Report Identifier (IRRID) PRR1-10.2196/52523

and potential efficacy of a machine learning approach for proactively identifying children at risk of nonor delayed vaccinations and validate predictive models using vaccination data gathered in Aim 1. Study findings will inform future implementations and scale up of Chanjo Kwa Wakati, including potential interventions to improve vaccination equity for children living in rural, resource-limited, or underserved communities in the United States.
PUBLIC HEALTH RELEVANCE: Many children living in rural and resource-limited settings receive delayed or no vaccinations, placing them at risk for potentially dangerous vaccine-preventable diseases.The goal of the proposed research is to evaluate whether an integrated community-based digital health intervention can promote equitable routine vaccinations for children.The study will be conducted in rural Tanzania; however, findings may be broadly relevant for identifying digital health strategies that can be implemented by nursing or other non-physician cadres of health professionals, to bridge vaccination inequities among rural and underserved populations in the United States.

CRITIQUE 1
Significance: 1 Investigator(s): 2 Innovation: 2 Approach: 2 Environment: 1 Overall Impact: The proposed project is designed to increase vaccine effectiveness by expanding access to populations with low vaccination rates while also increasing compliance within the population.The goal is the design of a digital health intervention for new mothers to improve vaccination timeliness/rates, with a focus on rural LMIC communities.The aims focus on the development and evaluation of the intervention, and an exploratory aim focused on developing predictive models for children at risk for delayed vaccinations.The work is built on a strong theoretical foundation and prior work by the investigative team on the use of SMS and vaccination (albeit focused on a different health condition).The research team has some prior collaboration, and complementary expertise in the necessary domains of digital health, cultural competency, health economics, community based medicine, and qualitative/quantitative analyses.The research is also novel, and adapts methodologies developed for a different SES population to a rural, LMIC setting by combining the assets of community health workers and mHealth devices.The environment is also quite strong; there are domestic (US) and local (Tanzania) resources that maximize the effectiveness of international collaboration, and include the necessary computational, health, and administrative resources to coordinate the study.There is, however, concern about the use of a no-intervention control group for the comparative study; not providing any information to this group may not be ethically justified.Overall, the strong theoretical framework combined with an excellent team, novel approach, and research environment, suggest a high probability of success for the proposed work.

Strengths
• PI presents rigorous prior research regarding the use of SMS with vaccine appointments.
• If successful, the work may lead to increased vaccine equity (e.g., rural, low income areas), increased vax timeliness, and the extension of health care to rural settings using dig.Technologies/mhealth.  • Some members of the research team have prior research overlap and funded collaborations.
• There is appropriate infrastructure for the clinical trial, with administrative and organizational experience in the research team.

Weaknesses
• None noted by reviewer.

Strengths
• The proposed approach mimics existing interventions designed for high income using combination of human + digital tools; this is a novel approach to adapting interventions designed for populations of different demographics.
• The potential generation of LMIC specific ML algorithms (and data that can be used by others for model training) is a novel contribution to science regarding vaccination responses in similar populations.
• The novel research design is optimized to simultaneously measure effectiveness, combine qual/quant information, and to use a validated implementation framework (RE-AIM) to maximize eventual translation to practice.

Weaknesses
• None noted by reviewer.

Strengths
• The proposed intervention is built on a validated RE-AIM framework, increasing probability of success.Additionally, the software framework uses validated, open technologies.
• The to be developed software increases health worker efficiency by automating some tasks, minimizing CHW burden.
• The qualitative/self-report instrument has already been validated with target population CIDH OSTERMANN, J • Inclusion of an advisory group maximizes cultural appropriateness and reduces risk of failure due to misunderstanding between the research team and participants.

Weaknesses
• The use of a no intervention control group may not be appropriate.Not providing even a minimum of materials or information may not be ethically justifiable.

Strengths
• The research environment is excellent.The domestic (US) environment includes the necessary analytical resources, as well as global/international research resources.The LMIC partner includes a research institute with well aligned mission and strong organizational structure.
• Given this is a clinical trial, the presented research infrastructure (administrative, testing centers, etc.) are appropriate.This is based on prior experience of the research team and partner organizations in the LMIC.

Weaknesses
• None noted by reviewer.

Strengths
• Timeline allots sufficient time for human subjects recruitment and intervention components.

Weaknesses
• Model evaluation period (Aim 3) may not be sufficient (only 2 months per site).

Protections for Human Subjects:
Acceptable Risks and/or Adequate Protections • Risks are appropriately communicated, perhaps reflecting prior experience of investigators with target population.

Data and Safety Monitoring Plan (Applicable for Clinical Trials Only):
Acceptable o Plan is acceptable, though no DSMB or named study monitor is included.

Applications from Foreign Organizations:
None noted by reviewer.

Resource Sharing Plans:
Acceptable Budget and Period of Support:

CRITIQUE 2
Significance: 2 Investigator(s): 2 Innovation: 2 Approach: 3 Environment: 1 Overall Impact: This is a multi-PI application.The application proposes to increase vaccination in lowincome settings (Tanzania) through low-cost digital strategies.The application is a clinical trial.The study team has extensive experience working with investigators and the population in Tanzania.The proposed project is seen as significant and the rigor of the prior research is strong.Successful completion of the proposed project could broadly impact rural vaccination rates.The team is well-suited to perform the proposed research and has a rich collaborative and multinational experience.There are some minor concerns with not including a biostatistician on the design of the statistical analysis.The project is seen as innovative although new methods are not being applied, they are implemented in novel ways.The approach is robust and aims 1 and 2 are well-described and sound.There are some concerns with the machine learning in the exploratory aim 3. It does not take into account potential generalizability to other populations outside the one provided.It is not clear what data elements will be included and while stakeholders will be queried, there is no planned time for this and no design for including them in model development.There are some minor concerns about the resulting models being overfit to the provided population.Overall, there is high enthusiasm for the proposal and it is likely to result in a successful intervention, increased vaccinations, and an extensible method to improve outreach in rural areas.

Strengths
• The rigor of the prior research is strong and well described.
1 R01 AI170760-01 7 CIDH OSTERMANN, J • The rationale for a clinical trial is well-supported and the study team has the necessary expertise and experience.
• Increasing vaccination rates in children and in low-income settings is seen as significance and important.
• Digital health interventions have been shown to increase vaccination rates and applying these in low income and rural settings is seen as important.
• Successful completion of the project could help to improve the vaccination rates and health of children in all rural settings with parental access to cellular phones

Weaknesses
• A clinical trial is not required to test the safety of vaccinations but the proposed study is appropriate in testing the efficacy of the intervention to increase vaccinations.

Strengths
• The study team has the necessary expertise in health policy, immunization demand, epidemiology, implementation science, data mining and data integration, and economics.
• This is a mPI application.
• The study team has extensive experience working with and in Tanzania.The team has previously collaborated.

Weaknesses
• The lack of a biostatistician is a concern as it relates to the proposed analysis of the intervention.

Strengths
• The proposed interventions are not novel, but are being applied in a novel setting and new ways.
• The integration of machine learning for prediction in low income settings could potentially shift clinical practice for targeting vaccinations.

Weaknesses
• The implementation framework proposed is not seen as novel, but is seen as important and a robust study design.

Strengths
• The team has already successfully completed an R21 Fogarty award in Tanzania.
• The preliminary work is appropriate and robust.
• The aim 1 clinical trial to increase vaccination is well-described and has appropriate strategies and analysis in place.
• The scientific rigor of aims 1 and 2 is strong and well-described.
• The project will establish feasibility of a large-scale, mobile-phone based vaccination strategy.
• Aim 3 is well-described and the machine learning on existing data is sound.

Weaknesses
• Sex as a biological variable is taken into account with mothers and but not handled with the children and should be taken into account in the resulting model.
• It is not clear who will be performing the cost-effectiveness analysis in aim 2.
• Aims 2 and 3 are dependent on the successful clinical trial in aim 1 but this is seen as a minor concern given the experience of the study team.
• Aim 3 does not take into account potential generalizability to other populations outside the one provided.It is not clear what data elements will be included and while stakeholders will be queried, there is no planned time for this and no design for including them in model development.There are some minor concerns about the resulting models being overfit to the provided population.

Strengths
• The University of South Carolina has experience with international research and appropriate resources for computing.
• Duke University is outstanding and has participated and collaborated in the past.
• UNC Chapel Hill school of social work is excellent.
• National Institute for medical Research has the necessary medical and research capabilities for the proposed project.

Weaknesses
• None noted by reviewer.

Acceptable
• It would be better to see the mParis platform shared in a way that is more easily accessible.

Budget and Period of Support:
Recommend as Requested

CRITIQUE 3
Significance: 1 Investigator(s): 1 Innovation: 3 Approach: 3 Environment: 1 Overall Impact: Dr. Ostermann and colleagues propose to evaluate a multi-faceted intervention, ("Chanjo Kwa Wakati", including community health worker (CHW) outreach and variety of automated informatics interventions) aimed at increasing utilization of vaccinations among mothers of infants (<1yr) in Tanzania.The project would include a cluster-randomized hybrid (CHW + informatics) implementation trial of 1200 mother-child dyads across 40 sites, and then evaluate effectiveness and cost-effectiveness of the intervention as compared to standard practice.Later in the project the team would then use machine learning techniques to develop a model predicting risk of non-or delayed vaccinations, based on the premise that risk-tailored approaches could optimize effectiveness and costeffectiveness in resource-constrained environments.In general, the intervention design is interesting and creative, the study design is rigorous, and the work would lead to generalizable knowledge on how to effectively deploy clinical informatics to address socioeconomic and other health disparities (not just within vaccination).High enthusiasm for these aspects was attenuated by lack of clarity on details of the analysis approaches in Aims 1 and 3.

Strengths
• Globally, only 86% of children under age 5 are immunized in timely fashion, and lack of immunization disproportionately impacts low-resource populations.
• Vaccination against disease is one of the most effective ways to increase life expectancy in lowand middle-income countries.
• Progress on improving vaccination coverage has stalled over the last 10 years.
• Implementation of pediatric vaccination is complicated by disparities in preventive health care access/utilization (e.g., across the metro/rural divide and or digital divide) as well as differences in parental attitudes and perspectives toward vaccination.
• Vaccination delays are more pronounced in rural settings; as technology access and use steadily increases, the work would take advantage of new opportunities for reaching this challenging demographic via a combination of HCW outreach and digital interventions.
• Health care delivery-based efforts to address vaccine hesitancy in LMICs are scant, despite evidence of their effectiveness in high-income countries.

Weaknesses
• None noted by reviewer.

Strengths
• Team has extensive experience and prior collaborations on NIH-funded and other federally funded research.
• The international research team and the proposal present a transdisciplinary approach to the problem of vaccination coverage in LIMCs.
• PI has successfully conducted research in Tanzania, which is the setting of the proposed intervention.

Weaknesses
• None noted by reviewer.

Strengths
• Focused approach to addressing barriers and inequities in digital health technology (DHT) access/use through proactive DHT development.
• Focusing on new mothers (whose children are <1 year of age) aligns with vaccination guidelines and could maximize effectiveness of interventions to increase vaccination utilization.
• Combination of HCW involvement and DHT may particularly maximize effectiveness in lowsocioeconomic position populations.

Weaknesses
• Proposed machine learning approaches are routine ("off-the-shelf") and not innovative.

Strengths
• Choice of primary outcome for the cluster-randomized hybrid implementation trial (Aim 1), i.e., time from vaccination due date to receipt, is desirable and appropriate.
• Organization of the prospective data collection to facilitate validation of predictive models derived externally (Aim 3) is clever.

Weaknesses
• Details on the difference-in-differences analysis approach as contextualized to the survival analysis framework are unclear.Specifically, the explicit quantities derived from the model to characterize treatment effect are in the form of regression coefficients.While relative assessments of treatment outcomes are not irrelevant, this fails to capture absolute assessments of vaccination rates, which are the basis of the proposal.
• The selected measure of benefit in the proposed cost-benefit analysis of Aim 2 -namely, the time after vaccination due date at which 80% of a population has been vaccinated -may be sub-optimal for measuring cost-benefit of this public health intervention, since this measure would not sufficiently capture longer-term effects of overall increases in vaccination coverage.(This is not to say that the expressed measure is unimportant.) • Aim 3 modeling methods are vague and inadequately contextualized to the specific modeling task.For example, the approach lists "off-the-shelf" methods without considering their appropriateness for the task of predicting vaccination uptake or delayed vaccination or their relative benefits over alternative approaches for the task.
• It is unclear how the long short-term memory neural network model would apply to the proposed cross-sectional prediction tasks.
• The decision problem to be informed by prediction modeling/machine learning is not formally defined, nor are the aspects of predictive performance that would be most relevant to the decision problem.

Strengths
• Inter-institutional collaboration (U South Carolina, Duke University, Tanzania National Institute for Medical Research) is supported through previous long-term collaboration.
• Project plan includes regular site visits of key personnel to Tanzania.
• Scientific resources at participating institutions are suitable.

Weaknesses
• None noted by reviewer.

Strengths
• Detailed and organized.Preparatory activities will be completed within the first year, with the prospective study being carried out between late Year 1 and early Year 5.
Weaknesses CIDH OSTERMANN, J • Cost effectiveness analyses in Aim 2 planned to begin in year 5; it is unclear whether components of this analysis should be initiated prior to the completion of the study intervention, with the concern being that cost-effectiveness analyses are time-and resource-intensive and 6-9 months in Year 5 may be insufficient time to complete the aim.
• Cyclic approach to Aim 3 (annual sets of activities) does not clearly align with the described approach in the research plan.

Protections for Human Subjects:
Acceptable Risks and/or Adequate Protections Acceptable o Data will be de-identified and coded to unique study identifiers.Data will be transmitted only under established Data Transfer Agreements.Reporting of study results will only be made in aggregate.Any adverse events would be reported to the institutions' IRBs.

Inclusion Plans:
• Sex/Gender: Distribution justified scientifically  • The study will be conducted among mother-infant dyads in Tanzania.It is expected that the study participants will be of African descent.The team has carefully planned for inclusion of mothers aged 16 years and above if deemed to be ethically acceptable (otherwise, 18 years and above) in order to be inclusive of mothers of child-bearing age.

Vertebrate Animals:
Not Applicable (No Vertebrate Animals)

Applications from Foreign Organizations:
None noted by reviewer.

Acceptable
• Study data will be made publicly available upon request to the PI after publication of findings.

Budget and Period of Support:
Recommend as Requested Footnotes for 1 R01 AI170760-01; PI Name: Ostermann, Jan NIH has modified its policy regarding the receipt of resubmissions (amended applications).See Guide Notice NOT-OD-18-197 at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-197.html.The impact/priority score is calculated after discussion of an application by averaging the overall scores (1-9) given by all voting reviewers on the committee and multiplying by 10.The criterion scores are submitted prior to the meeting by the individual reviewers assigned to an application, and are not discussed specifically at the review meeting or calculated into the overall impact score.Some applications also receive a percentile ranking.For details on the review process, see http://grants.nih.gov/grants/peer_review_process.htm#scoring.

•
Prior work: experience with SMS based intervention (different population: HIV) in target country; some knowledge of factors limiting rural vaccination; validation of current intervention with a small subset of women (GOOD); validation of phone use in the target population; validation of digital reminders in target population.GOOD FEASIBILITY Timeline is acceptable for proposed study

•
Special consideration is given to children aged 16-17 years, who, in Tanzania, are legally permitted to make medical decisions relating to reproductive health.Owing to the 25% rate of pregnancy/delivery among females aged <18 years in Tanzania, the team would request a waiver of parental consent from IRBs at USC and Duke and in Tanzania to include these children if they are otherwise eligible to participate in the study.This is on the basis that the planned interventions are not outside risks encountered during ordinary life.Steps are taken to protect against breaches of sensitive data per HIPAA requirements, recognize signs of emotional distress that are inconsistent with the ability to provide informed consent, and adverse reactions that might be the result of receiving unwanted SMS messages.The team would consult an ethicist to review and help the team adapt SMS and other technological interventions to minimize risks of these adverse outcomes and experiences.
Data and Safety Monitoring Plan (Applicable for Clinical Trials Only):