Article Text
PDF
Abstract
Introduction To improve healthcare provider knowledge of Tanzanian newborn care guidelines, we developed adaptive Essential and Sick Newborn Care (aESNC), an adaptive e-learning environment. The objectives of this study were to (1) assess implementation success with use of in-person support and nudging strategy and (2) describe baseline provider knowledge and metacognition.
Methods 6-month observational study at one zonal hospital and three health centres in Mwanza, Tanzania. To assess implementation success, we used the Reach, Efficacy, Adoption, Implementation and Maintenance framework and to describe baseline provider knowledge and metacognition we used Howell’s conscious-competence model. Additionally, we explored provider characteristics associated with initial learning completion or persistent activity.
Results aESNC reached 85% (195/231) of providers: 75 medical, 53 nursing and 21 clinical officers; 110 (56%) were at the zonal hospital and 85 (44%) at health centres. Median clinical experience was 4 years (IQR 1–9) and 45 (23%) had previous in-service training for both newborn essential and sick newborn care. Efficacy was 42% (SD ±17%). Providers averaged 78% (SD ±31%) completion of initial learning and 7% (SD ±11%) of refresher assignments. 130 (67%) providers had ≥1 episode of inactivity >30 day, no episodes were due to lack of internet access. Baseline conscious-competence was 53% (IQR: 38%–63%), unconscious-incompetence 32% (IQR: 23%–42%), conscious-incompetence 7% (IQR: 2%–15%), and unconscious-competence 2% (IQR: 0%–3%). Higher baseline conscious-competence (OR 31.6 (95% CI 5.8 to 183.5)) and being a nursing officer (aOR: 5.6 (95% CI 1.8 to 18.1)), compared with medical officer, were associated with initial learning completion or persistent activity.
Conclusion aESNC reach was high in a population of frontline providers across diverse levels of care in Tanzania. Use of in-person support and nudging increased reach, initial learning and refresher assignment completion, but refresher assignment completion remains low. Providers were often unaware of knowledge gaps, and lower baseline knowledge may decrease initial learning completion or activity. Further study to identify barriers to adaptive e-learning normalisation is needed.
- PAEDIATRICS
- Protocols & guidelines
- Human resource management
- MEDICAL EDUCATION & TRAINING
Data availability statement
Data are available upon reasonable request. Deidentified participant data from this study are available upon reasonable request. Interested researchers may obtain the data by contacting the corresponding author, Dr. Peter Meaney, at meaneypa@stanford.edu. Access to the data will be granted following approval by an independent review committee, established to evaluate the scientific validity and ethical justification of the proposed use. Note that only the deidentified participant data are available, and no additional supporting information, such as study protocols or statistical analysis plans, will be provided. This process ensures that the data are used responsibly and in accordance with ethical research standards.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
Used the Reach, Efficacy, Adoption, Implementation and Maintenance framework and Howell’s conscious-competence model, providing a structured approach to assess implementation success and baseline provider knowledge.
Employed a multicentre approach, enhancing the generalisability of findings across different healthcare settings in Tanzania.
Inclusion of a diverse provider cohort in terms of cadre and experience, contributing to a comprehensive understanding of the intervention’s impact.
The study’s observational design limits the ability to establish causality between the intervention and observed outcomes.
Reliance on self-reported data for assessing provider knowledge and metacognition may introduce response bias.
Introduction
The Government of Tanzania is committed to reducing the neonatal mortality rate from 20 per 1000 live births to the SDG target of 12 per 1000 live births by 2030.1 Addressing gaps in quality of essential and emergency care is a key strategy for achieving this ambitious target.2–5 In Mwanza, Tanzania, correct diagnosis estimates at health centres and district hospitals range from 61% to 87%, while correct treatments are administered only 21%–86% of the time.6 7
Provider knowledge is a key factor influencing the quality of newborn care. Conventional in-service training programmes, such as Helping Babies Breathe and Essential Newborn Care, have shown effectiveness in improving care when sustained over time.8–10 These courses vary in their focus, ranging from immediate newborn care in the first hour to care throughout the first 28 days of life. They also differ in the depth and type of cognitive skills they target, from basic (remembering and understanding) to complex (evaluating and creating). The duration of these trainings varies from 2 to 8 days, and their content and methodologies have undergone updates to enhance their effectiveness.11 12
Unfortunately, conventional in-service education methods are often inadequate in coverage and difficult to sustain.2 13 14 Conventional education methods do not systematically adapt to individual providers’ knowledge or convenience,15–18 have time-limited education and target minimal competency, which limits education effectiveness.17 19–23 Our systematic review highlighted that current educational content and educational design often have limited adaptability to facility needs, which also decreases education effectiveness.22 The limited effectiveness and accuracy of current educational methods widens the ‘know-do’ gap, and this gap is greater in rural, under-resourced areas where in-service education is limited.24 A key research gap of the WHO is to identify effective provider education that extends across health systems.25 26
Adaptive eLearning, a subdomain of e-learning, pulls from computer science and artificial intelligence principles to create a cognitive model to adapt education to each provider.27 Usage data include metacognition categorised using Howell’s conscious-competence model: (1) conscious competence (correct and confident), (2) unconscious incompetence (confident in knowledge but incorrect), (3) conscious incompetence (not confident in knowledge) and (4) unconscious competence (not confident in knowledge but correct).28 Usage data are processed to create a cognitive model for each student and adjusts the sequencing of content and ratio of learning resources based on the formative assessment. Knowledge acquisition during initial learning has been both higher and faster compared with conventional education.27 29
In addition to optimising initial learning, adaptive eLearning can address forgetfulness through generating refresher learning assignments. Forgetfulness, first described by Ebbinghaus, is an exponential decay of knowledge with knowledge returning to baseline days or weeks after initial learning.30 31 We have seen this decay in our previous work in low-/middle-income countries (LMICs) with paediatric acute care knowledge and CPR (Cardiopulmonary Resuscitation) skills.17 20 Forgetfulness can be addressed with refresher assignments, spacing learning over time.32–34 Subsequent refresher assignments start at a higher baseline than previous, taking less time to achieve mastery. Over time, yields a more significant percentage of data remembered. In high-income settings, we have demonstrated that refresher assignment completion improves paediatric resuscitation skills and patient outcomes.35–37 In Tanzania, the use of refresher assignments over time is a key implementation strategy of Helping Babies Breathe, which has significantly increased adherence to guidelines and reduced neonatal deaths by almost 50%.10
Adaptive e-learning has the potential to be rapidly scalable with existing infrastructure: it does not require significant dedicated resources to implement and maintain and may close the training gap that exists for rural, under-resourced areas and allow increased dissemination of up-to date evidence based guidelines and reduce the knowledge deficit, decreasing the need for face-to-face education when instructors are limited.38–40 While adaptive e-learning is as effective as conventional education when examining patient outcomes for dyslipidaemia screening, monitoring of diabetes, drug dose calculation, and pressure ulcer classification,41–44 there is limited evidence examining newborn and paediatric acute care.45 46 Further, the optimal implementation strategy of adaptive e-learning for in-service provider education in LMICs is unknown.44 47
Our programme, Pediatric Acute Care Education (PACE), is an adaptive e-learning environment (AEE) codeveloped with the Pediatric Association of Tanzania (PAT), Catholic University of Health and Allied Sciences, Stanford University and Area9 Lyceum to improve facility-based adherence to national newborn and paediatric guidelines.48 PACE is designed for all cadres (provider types) that may be responsible for caring for newborns and sick children. Initially piloted in 2019, PACE has expanded in size to meet the training needs identified by PAT and PACE providers.
Our initial pilot demonstrated a 30% change in conscious competence during initial learning from 66% (57%–75%) to 94% (92%–98%).49 The average initial learning completion was only 37% and refresher assignment completion was not assessed. There were four barriers identified to initial learning completion: (1) use of pre–post assessments for efficacy, (2) lack of in-person technical support, (3) ineffective email nudging strategy, and (4) we observed a significant drop off with disability and exposure assessment using a structured order of modules.49 Based on these results, we revised our implementation strategy in four ways: (1) use of change in conscious competence from baseline for efficacy, (2) a full time in-person programme manager to provide support PACE providers, (3) incorporation of an escalating nudge strategy that included emails, WhatsApp messaging and in-person support, and (4) in addition, we allowed providers to complete modules in any order.
Tanzania’s Ministry of Health contextualised both WHO’s Essential Newborn Care and Helping Babies Breathe to Tanzania, publishing ‘Guidelines for Neonatal Care and Establishment of Neonatal Care Unit in 2019.50 We applied our content development methodology using these guidelines to develop our adaptive Essential Newborn and Sick Care (aESNC), and deployed modules within PACE in May 2022. aESNC currently contains two assignments with total of nine modules: preparing for delivery, first hour of life, neonatal resuscitation, introduction to sick newborn care, birth asphyxia+pain, convulsions and meningitis, glucose and electrolytes, haemorrhage and jaundice, pneumonia, sepsis and shock. Learning objectives are restricted to Bloom’s Taxonomy levels 1 and 2 (remembering, understanding), and 100% conscious competence is required to complete the module. Providers completing all PACE modules are awarded continuing professional development credit through PAT towards maintaining professional certification.
The objectives of this study are to assess the implementation success of adaptive Essential and Sick Newborn Care (aESNC) using the revised implementation strategy and describe baseline provider knowledge and metacognition in Mwanza, Tanzania. We evaluated implementation success using the RE-AIM (Reach, Efficacy, Adoption, Implementation and Maintenance) framework adapted by Soicher et al for the implementation of education interventions for higher education and describe provider knowledge and metacognition using Howell’s consciousness competence model.28 51 52
Methods
Study design
This was a prospective single-arm, multicentre pilot implementation study conducted in Mwanza, Tanzania from May 2022 to January 2023. This manuscript was formatted in accordance with Strengthening the Reporting of Observational Studies in Epidemiology guidelines.53
Setting
Study sites included all sites currently participating in PACE that had at least one provider in the study cohort. Facility characteristics are listed in table 1. Due to limited study personnel and this being the initial study, PACE was initially deployed at the zonal hospital in May 2022, and then extended to health centres within Mwanza Region that refer to the zonal hospital in stepwise fashion. Duration of facility participation at time of data extraction is listed in table 1.
Facility characteristics
Participants
The cohort consisted of a convenience sample of consenting healthcare providers (HCPs) who had participated in PACE >30 days. Providers were identified by the facility medical officer in-charge, facility head matron/patron or head of department as well as during sensitisation meetings at morning report and recruited to participate by the PACE programme manager. Informed electronic consent was obtained through REDCap from all providers who participated in PACE.54 Eligible PACE providers are facility-based HCPs responsible for providing clinical care to newborns, infants and/or children. In Tanzania, training duration varies by cadre: medical officers require 5 years of training and 1 year of internship; advanced degree nursing 3 years and 1 year internship; nurses and clinical officers 3 years of training, and assistant medical officers 2 years of training. We excluded providers who declined to consent or who withdrew from study; these numbers are given in the Consolidated Standards of Reporting Trials (CONSORT) diagram, figure 1.
Consolidated Standards of Reporting Trials diagram.
Implementation strategy
aESNC was implemented through the PACE programme. PACE is an AEE with locally derived content, a steering committee, a programme manager to provide in-person technical support and series of motivators to increase completion. It is designed to increase provider proficiency in neonatal and paediatric evidence-based guidelines in Tanzania and has been previously described.48
In the PACE platform, we leveraged the principles of AEEs to tailor the educational experience to each HCP.48 AEEs, a specialised subset of e-learning, use a combination of computer science and artificial intelligence to create a dynamic cognitive model for each learner. This model adapts the content delivery based on three core components: modular curriculum and learning resources, aggregate student data, and cognitive modelling. The PACE platform continuously assessed not only the correctness of HCP responses but also their metacognition and resilience in handling challenging topics. This iterative process allowed for the optimisation of content pacing and sequencing, enhancing HCP proficiency and confidence over time. Furthermore, the platform’s AEEs analysed performance and knowledge gaps to provide customised refresher packages, ensuring that learning is reinforced at optimal intervals and tailored to the individual learning needs of each HCP.
Adaptive e-learning with locally derived content
aESNC, consists of 2 assignments: essential newborn care (3 modules, 47 learning objectives) and sick newborn care (6 modules, 91 learning objectives). aESNC content was collaboratively developed with subject matter experts (SME) and content designers using chapters 1 and 3 of Tanzania’s ‘Guidelines for Neonatal Care and Establishment of Neonatal Care Unit’.50 Content creation was supervised by SME from the PAT’s Continuing Professional Development Committee (NM, NC, CM), the Tanzanian Society for Pediatric Nursing (ED), and Area9 Senior Learning Architects (BR, MB) (online supplemental materials).
Supplemental material
PACE steering committee
The Steering Committee (AH, PAM, AA, HN) provides oversight and coordination of PACE management, research administration, publications and data sharing, and integration of all resources needed for the project. The chair of the steering committee is responsible for communication among committee members, including meeting schedules and agendas, and rotates among the members on a yearly basis. The PACE steering committee consisted of experts in newborn and paediatric care, provider education and implementation research.
The programme manager
The programme manager is a medical (MD) or nursing (RN) officer with experience working in the Tanzanian health system, human subjects research training, effective communication skills and, either formal or informal health education and/or IT skills. The programme manager conducts sensitisation meetings at facilities and generates contact lists. These lists are reviewed and augmented by the medical officer in charge, head of department and/or chief nursing medical officer as appropriate, and current PACE providers. The programme manager meets with consented providers in-person individually to set up PACE on their mobile device, ensure proper functioning and provide initial data bundle.
Motivational strategies
Nudges. Weekly, providers who had not completed all PACE initial learning assignments were given reminders or ‘nudges’ to complete their learning. Our escalating nudge strategy was as follows: no activity for 2 days: auto email reminder from Rhapsode; >5 days: WhatsApp using standardised statements; >30 days: a virtual or face-to-face meeting with programme manager.
Internet support. Providers were reimbursed up to 10 gb of data (~US$4) monthly on their mobile carrier during the study period.
Maintaining certification. Continuing Professional Development credit was awarded via PAT for completion of all PACE initial learning assignments.
Passive feedback to health system leadership. Programme manager would send PACE Facility Progress Report (PDF) to facility stakeholders weekly via email. Stakeholders to this process may include hospital administration, regional and council health management teams, and thePAT leadership. Reports included aggregate activity, metacognition and median time to proficiency at a facility level, but not does not report proficiency or metacognition at an individual level.
Hypothesis
Implementation of aESNC with addition of in-person coordinator and nudging strategy to our implementation strategy would increase reach and improve average completion of both initial learning and refresher assignments compared with our initial PACE pilot.
Variables
Implementation outcomes. We used the established RE-AIM for the educational intervention implementation framework to define our feasibility outcomes of Reach, Efficacy and Implementation.52 55 Adoption and maintenance were not assessed. We assessed Reach using individual participating providers as a proportion of providers identified as eligible.
Efficacy was assessed using change in provider conscious competence (current–baseline). Baseline provider proficiency was determined using providers initial responses to knowledge probes, and current provider proficiency by last response to knowledge probes. Responses were categorised using Howell’s conscious-competence model: (1) conscious competence (correct and confident), (2) unconscious incompetence (confident in knowledge but incorrect), (3) conscious incompetence (not confident in knowledge), and (4) unconscious competence (not confident in knowledge but correct).28
We used six metrics to assess implementation: persistent activity, average progress of initial learning completion, average progress of refresher assignment completion, time to enrolment, nudging utilisation and loss to follow-up. Persistent activity was defined as actively using PACE within last 2 weeks of the study (Jan 2023). Average progress of initial learning was percentage of achieving 100% conscious competence of all aESNC learning objectives. Average progress of refresher assignments was percentage of achieving 100% of all content assigned by Rhapsode for refresher learning. Time to enrolment was days from consent to enrolment interview. Nudging utilisation included % of providers with at least one nudge, median number of nudges and distribution of nudge types. Loss to follow-up was defined as those who were inactive for >30 days and were not able to be contacted.
Predictors, potential confounders, and effect modifiers. Several variables were collected to describe the cohort. These included facility, cadre (profession), years of clinical experience, previous newborn training, job satisfaction, motivation and baseline knowledge and metacognition. Facilities were defined by government designation as a zonal hospital or health centre. Cadre categories include specialists, medical officers, assistant medical officers, clinical officers, assistant clinical officers, nursing officers, assistant nurse officers, nurse midwives, medical attendants, laboratory scientist/technologists and health assistants. The continuous variable ‘years clinical experience was collapsed to a categorical variable of ≤1 year, 2–3 years, 4–9 years and 10+ years based on quartiles rather than visual inspection given small numbers. Previous newborn training was defined as ever having taken either Essentials of Newborn Care, Helping Babies Breathe, both or neither. Job satisfaction and motivation were measured using previous questionnaires validated for HCPs in Ghana by Bonenberger et al.56
Data sources/measurement
At enrolment, providers completed electronic surveys regarding demographics, previous clinical training, job satisfaction and motivation via REDCap. Provider response data including knowledge competence, metacognition, average progress on initial learning or refresher assignments was collected within the Area9 Rhapsode platform. Data from Rhapsode and REDCap was extracted on 14 January 2023, linked with study ID numbers by name and deidentified prior to analysis.
Bias
The cohort was identified through sensitisation meetings at facilities by the programme manager, as well as by facility administrators using staff rosters and study providers and at same points of care over the same time frame for each facility. Outcome data used provider usage data from Rhapsode. In addition, enrolment interview and reminders were tracked in the study database. These modules were newly developed, and no study provider had piloted these modules before the study. In addition, baseline competencies were obtained as part of the exposure, allowing to account for an effect of previous training. Our multivariable analyses on factors associated with new provider retention adjusted for plausible prognostic factors. Providers completed all surveys and outcome data electronically using their mobile phones. Potential data entry errors were followed up by the programme manager with the provider for clarification where needed and logged within the database. The outcome assessments blinded to the study team and were linked by name and email addresses to the study database and deidentified prior to analysis.
Study size
As this was a feasibility study, no power calculation was done. Study size was determined by number of providers enrolled during the study period. Based on an estimate of 20% of providers at the zonal hospital and 40% of health centre providers caring for newborns and sick children, our initial enrolment target was 50 medical officers, 30 clinical officers and 50 nursing officers over 6 months.
Statistical methods
Descriptive, univariate and multivariate analyses were performed. Summary results are presented as means and SD for normally distributed variables and medians with IQRs for variables that were not normally distributed. For our multivariable logistic regression exploring completion of initial learning or persistent activity, we selected provider external characteristics (facility, cadre, experience, previous training) a-priori and did not incorporate response data (job engagement, motivation, or knowledge proficiency data) given the limited size to the cohort. Significance was set at p≤0.05 and models were fit using Python V.3.8.5.
Patient and public involvement
This research was done without patient involvement. Patients were not invited to comment on the study design and were not consulted to develop patient-relevant outcomes or interpret the results. Patients were not invited to contribute to the writing or editing of this document for readability or accuracy.
Results
Reach
All four facilities invited to participate adopted PACE. Of 246 providers, 90% (221) were eligible, with 17 of 25 ineligible due to not completing screening survey. Consent rate of eligible providers was 90% (n=195) with ‘no time to participate’ being the greatest reason for declining participation (figure 1).
Provider characteristics
One hundred ten (56%) of providers in this cohort were based at the zonal hospital, followed by HC#2 at 42 (22%), HC #1 at 27 (14%) and #3 at 16 (8%) (table 2). Providers were most commonly medical officers (75, 39%), nursing officers (53, 27%) and clinical officers (21, 11%). Median years of clinical experience was 4 (IQR 1–9). The reported prevalence of previous newborn training of Essential Newborn Care, Helping Babies Breathe, and both ENC and HBB were 23% was 31%, 41% and 23%, respectively. Overall job satisfaction was 3.6/5 (SD 0.8) with highest subscores of morale 4.1/5 (SD 0.8) and supervision 3.8/5 (SD 1.0) and lowest subscores of renumeration 3.9 (SD 1.2), followed by in-service training 3.5/5 (SD 1.2), management 3.5/5 (SD 1.0) and work environment at 3.5/5 (SD 1.0). Overall motivation scores were 3.7/5 (SD 0.6).
Provider characteristics
Efficacy
For providers who completed initial learning (n=110), average conscious competence increased 42±1 percentage points, while average unconscious incompetence decreased 31±1 percentage points. Of the efficacy, the improvement of conscious competence, 76% was due to reduction in unconscious incompetence. Average conscious incompetence decreased by 10±1 percentage points, while provider unconscious competence was extremely low both before and after PACE (online supplemental data A).
Implementation
The average progress of initial learning for all providers was 78% (SD=31%) (online supplemental data C1). Fifty-six (110) completing all initial learning, 13% (25) incomplete but active within 2 weeks of the end of the study, and 31% (60) becoming inactive during the study period. Providers who completed the initial learning took on average 4.5 (IQR 2.4–6.1) hours (online supplemental data B1-3). Inactive providers dropped out after on average 1.9 (IQR 0.8–3.2) hours, and the active providers who are still finishing initial learning spent 4.0 (IQR 2.3–5.4) hours. Median duration to complete all modules was 61 days (IQR 34–111). Reduction in probability of provider remaining completing initial learning or remaining active appeared to be evenly distributed over % completion and days of PACE participation (online supplemental data C1,B4).
Average progress of refresher assignments was 7%, with none completing all refresher assignments, 27% (37) incomplete but active in PACE within last 2 weeks of the study period, and 73% (98) becoming inactive during the study period. Average refresher progress appears to be heavily skewed to lower completion with median completion 2% (IQR: 0%–8%) (online supplemental data C2). Total hours spent using modules was 0.3±0.7 for active providers and 1.0±1.3 for inactive providers.
Days from consent to enrollment interview was 2 (IQR: 0–4). Sixty-seven per cent (130) of providers needed at least 1 nudge and 30% (59) needed at least one follow-up with the programme manager. A total of 1011 nudges were sent: 311 (31%) email nudges (2–5 days inactivity), 564 (56%) WhatsApp nudges (5–30 days inactivity), and 136 (13%) nudges by the programme manager (> 30 days inactivity) (online supplemental data D1,2). The programme manager conducted 106 nudges by phone and 30 in-person. Most frequent reasons reported for >30 days inactivity was ‘no time’ (93, 69%), followed by ‘forgot to use’ (68, 50%). Technical barriers (phone not working, cannot access PACE, not sure how to use PACE) represented only 15 (11%) and no provider reported inactivity due to lack of mobile data connectivity. No providers wished to terminate from study when interviewed with programme manager at 30 days of inactivity and none were lost to follow-up.
Baseline knowledge and metacognition of essential and sick newborn care
Overall median baseline conscious competence for aESNC was 53% (IQR: 38%–63%), with unconscious incompetence 33% (IQR: 25%–45%) and 32% (IQR: 23%–42%), respectively (figure 2). aESNC had conscious incompetence of 7% (IQR: 2%–15%), and unconscious competence 2% (IQR: 0%–3%).
Baseline metacognition heatmap.
For individual aESNC modules, conscious competence was highest in neonatal resuscitation (63% (IQR: 34%–75%)), first hour of life (58% (IQR: 33%–75%)), and glucose and electrolytes (57% (IQR: 29%–71%)), and lowest in birth asphyxia and pain (38% (IQR: 0%–54%)), preparing for delivery (38% (IQR: 25%–56%)), and pneumonia, sepsis and shock (43% (IQR: 14%–71%)). Unconscious incompetence was highest in preparing for delivery (38% (IQR: 25%–56%)), introduction to the sick newborn (33% (IQR: 22%–50%)), and convulsions and meningitis (33% (IQR: 17%–47%)), and lowest in first hour of life (17% (IQR: 8%–33%)), haemorrhage and jaundice (20% (IQR: 0%–33%)), glucose and electrolytes (21% (IQR: 7%–36%), and neonatal resuscitation (21% (IQR: 5%–32%)). Conscious incompetence was highest in birth asphyxia and pain (8% (IQR: 0%–19%)), haemorrhage and jaundice (7% (IQR: 0%–27%)) and preparing for delivery (6% (IQR: 0%–13%)), and zero in all others (IQR: 0%–13%). Unconscious competence was zero in all modules (IQR: 0%–8%).
Factors associated with initial learning completion or persistent activity
On univariate analysis, being a nursing officer, 2–3 years of experience, and having higher baseline conscious competence was associated with initial learning completion or persistent activity. On multivariate modelling, nursing officers, clinical officers and ‘other personnel were significantly associated with better initial learning completion or persistent activity when compared with medical officers adjusting for facility, clinical experience and previous newborn training (table 3). Facility, previous training with either Essential Newborn Care or Helping Babies Breathe were not associated with initial learning completion or persistent activity.
Factors associated with initial learning completion or persistent activity
Discussion
In this study, we have explored the implementation and impact of the aESNC modules within the PACE programme in Tanzania, offering critical insights into both the reach and efficacy of this innovative approach to in-service provider education. Our investigation delves into the nuances of enhancing provider education in a challenging healthcare context, highlighting the successes, limitations and future directions of our implementation strategy.
Increased reach to providers
The data demonstrate a significant enhancement in the reach of in-service provider education through aESNC, particularly targeting providers who were not previously engaged by conventional in-service training methods. This finding is crucial in contexts like Tanzania, where access to quality in-service training can be limited. The reach of aESNC was notable, with 85% of identified providers at participating facilities engaged. This level of engagement is critical for fostering collective behaviour change among providers. Drawing from social behavioural science, a critical mass of 10%–40% of providers engaged is often necessary to drive such change.57–59 Our ability to engage providers who had not received conventional newborn training (30% had received Essential Newborn Care training, 40% had received Helping Babies Breathe, but only 23% had received both) highlights the programme’s potential to fill crucial training gaps.
Efficacy and completion rates
The initial efficacy rate of 41% is consistent with our previous studies in paediatric acute care training, which demonstrated knowledge efficacy rates of 10%–25%,17–20 and other newborn and paediatric conventional in-service education studies.5 21 The improvement in completion rates in this study (78% for initial learning and 56% for completing all assignments) compared with our initial pilot may be attributed to the in-person support and the escalating nudging strategy. These rates are comparable to longitudinal e-learning studies in high-income countries (44%–50%),60 61 and significantly higher than massive open online courses rates (7%–15%).62 63 Beyond our revised implementation strategy, other factors such as increased peer support and a higher percentage of providers participating in PACE at each facility may have contributed to improved completion rates. However, it remains unclear whether this improvement is attributable to our implementation efforts or environmental factors like unmeasured peer support.
Barriers to completion, baseline knowledge and alternative factors influencing completion rates
A significant barrier to completion identified in our study was not technical issues or lack of internet access, but rather the inadequate allocation of time for education. Our initial time estimate for course completion was 5 hours over 1 month, yet the actual median time was 61 days, substantially longer than anticipated. This discrepancy underscores the necessity for more realistic time estimates in future programme planning. Notably, our observed level of unconscious incompetence exceeded the best practice guidelines suggested by Area9, which recommend maintaining it between 20% and 30%. This rate was also higher than in our initial pilot, indicating a need for focused support strategies for those at risk of being overwhelmed by the learning material. Unconscious incompetence is a significant concern, as it can mask workforce knowledge gaps and lead to clinical errors. Adverse events occur in 18% of inpatient admission in Africa and increasing provider knowledge is needed to improve patient safety.64 65 Our study found that providers often do not realise their knowledge gaps, a finding consistent with previous studies.2 66 Our exploratory analysis suggests a correlation between lower baseline knowledge and inactivity or non-completion in aESNC. This finding may be causal, due to cognitive overload, or may confounded by other provider or work environment characteristics.
Challenges in sustaining refresher learning
Challenges in Sustaining Refresher Learning and Assignment Completion: while our revised implementation strategy, involving in-person support and an escalating nudging strategy, successfully drove initial learning completion, sustaining engagement in refresher learning proved challenging. This aspect is critical for long-term knowledge retention.67 The challenge was further exacerbated by providers' low awareness of their knowledge gaps, especially among those with lower baseline knowledge, who were less likely to engage in ongoing education. This is evident from the completion rate for refresher assignments being less than a third, and the average progress through these assignments remaining below 10%. These findings highlight an urgent need to modify our implementation strategy to boost engagement in refresher training, which is essential for the development of long-term memory. Our initial focus on nudging strategies for completing initial learning assignments may have inadvertently conveyed to providers that no further learning was necessary. Therefore, adjustments in this strategy, coupled with sensitisation meetings to emphasise the importance of continuous learning, could be instrumental in normalising adaptive e-learning among providers.
Knowledge’s role in developing competence
The aESNC programme effectively boosts provider knowledge, yet it is important to understand that acquiring knowledge is only one aspect of enhancing provider competence and behavioural change. The criticality of retaining knowledge and skills long-term is evident, but the best methods for developing these remain uncertain. Rowe et al’s research presents inconclusive evidence on whether protocol training or clinical acumen training more effectively improves provider competence.66 Similarly, Clark et al’s study demonstrates improved healthcare outcomes from neonatal resuscitation training combined with ongoing mentorship and skill reinforcement.68 However, it remains to be determined whether skill practice directly strengthens provider competence or mainly supports continuous education and refresher training. Ayeiko et al’s work, showing significant care quality improvements in young hospitalised children through active feedback and mentorship, suggests an alternative approach to intensive skill maintenance.69 Future research should focus on detailed, rigorous studies to better comprehend and optimise these elements in diverse healthcare environments.
Limitations
Contextual validation of key constructs
The relationship between job satisfaction, motivation and turnover intentions has not been validated within the Tanzanian context. This lack of validation may impact how we interpret our findings, particularly concerning provider retention.
Challenges in monitoring implementation metrics
We recognise that this observational study may be subject to sampling bias. The unexpectedly high job engagement scores and enrolment rates observed in our study population, and the discordance between eligible providers identified by facility leadership and staffing estimates from district health system leadership (refer to table 1) could indicate a sample bias towards early adopters and not represent the true target population. These elements warrant further exploration in future studies.
Focus on knowledge over skills
The study primarily concentrated on knowledge acquisition, with less emphasis on skills training. Future iterations of this programme should more thoroughly incorporate skills training to offer a more holistic educational experience for HCPs.
Limitations in analytical scope due to sample size
The breadth of our univariate and multivariable analyses was limited by the sample size. This constraint may have hindered the identification of critical provider characteristics that affect retention in the programme.
Future directions
To advance the aESNC programme, our future directions encompass a comprehensive approach for PACE, integrating qualitative research, programme enhancement, clinical auditing, and effectiveness analysis.
Qualitative research with NPT
We aim to conduct in-depth qualitative research using a mixed-methods approach, guided by Normalisation Process Theory (NPT). This research will focus on evaluating the ongoing implementation of PACE, particularly how it integrates into daily healthcare practices. A key aspect of this research will be to provide a comprehensive summary of the results, emphasising the feasibility, acceptability and scalability of PACE in diverse healthcare environments. This will involve a detailed analysis of PACE’s reception among HCPs and its potential for broader application.
PACE programme enhancement
The PACE programme will undergo significant revisions to incorporate a blended learning approach, combining adaptive e-learning with tailored skills practice. This revision aims to enhance both long-term knowledge and practical skills among HCPs. Additionally, we plan to implement facility-based clinical champions. These champions will serve as on-site mentors and motivators, fostering continuous learning and skill enhancement within healthcare settings.
Clinical auditing and content development
Clinical auditing will be employed to assess the impact of this novel in-service education programme on clinical practices and patient outcomes. This assessment will guide future content development and refinement. Once a robust implementation strategy for refresher learning and skills practice is established, we will tailor the content to meet the specific clinical needs of different healthcare facilities, ensuring relevance and direct applicability.
Effectiveness and cost-effectiveness analysis
A critical future step involves a rigorous examination of the effectiveness of the PACE programme. This will include evaluating how well the programme improves provider knowledge, skills, and patient care outcomes. Furthermore, a comprehensive cost-effectiveness analysis will be conducted. This analysis will compare the costs of implementing and maintaining PACE against its benefits in terms of improved healthcare delivery and patient outcomes. This step is essential for understanding the economic viability of PACE and its potential for scalability in various healthcare settings.
These future directions are designed to not only enhance the PACE programme but also to provide a thorough understanding of its impact, feasibility and sustainability in improving HCP education and patient care across diverse settings.
Conclusion
Our investigation into the aESNC modules within Tanzania’s PACE programme reveals a notable increase in the reach and efficacy of in-service provider education. This approach successfully engaged providers who had been overlooked by conventional training methods. However, challenges in maintaining engagement in refresher learning, particularly among providers with lower baseline knowledge and lower awareness of their knowledge gaps, were evident. Future directions for the PACE programme are focused on enhancing refresher education and evaluating its implementation. This includes conducting in-depth qualitative research guided by NPT to assess ongoing implementation and scalability, revising the programme to integrate adaptive e-learning with practical skills training, and employing clinical auditing to measure the programme’s impact on patient care. These steps are crucial for ensuring the programme’s effectiveness, sustainability and broader applicability in improving HCP education and patient outcomes in diverse settings.
REDCap database
Study data were collected and managed using REDCap electronic data capture tools hosted at Stanford University.70 71 REDCap (Research Electronic Data Capture) is a secure, web-based software platform designed to support data capture for research studies, providing (1) an intuitive interface for validated data capture; (2) audit trails for tracking data manipulation and export procedures; (3) automated export procedures for seamless data downloads to common statistical packages; and (4) procedures for data integration and interoperability with external sources. The Stanford REDCap platform (http://redcap.stanford.edu) is developed and operated by Stanford Medicine Research IT team. The REDCap platform services at Stanford are subsidised by (1) the Stanford School of Medicine Research Office, and (2) the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through grant UL1 TR001085. Area9 Rhapsode meets the requirements for full GDPR compliance including encryption, data security, and ‘forget me’.
IRB language
The Institutional Review Board of the Tanzania National Institute of Medical Research (NIMR/HO/R.8a/Vol.IX/3990), Stanford University (60379), the ethics committee of the Catholic University of Health and Allied Science (no ID number given), and the Mwanza Regional Medical Officer (Ref. No. AG.52/290/01A/115) approved the study protocol including consent procedures. Data collection procedures were completed in compliance with the guidelines of the Health Insurance Portability and Accountability Act to ensure subject confidentiality. Informed electronic consent was obtained through REDCap from all providers who participated in PACE.54 All providers who completed consent were included. All surveys and questionnaires were entered directly by providers into REDCap. This study is reported according to the CONSORT 2010 extension to randomised pilot and feasibility trials.
Data availability statement
Data are available upon reasonable request. Deidentified participant data from this study are available upon reasonable request. Interested researchers may obtain the data by contacting the corresponding author, Dr. Peter Meaney, at meaneypa@stanford.edu. Access to the data will be granted following approval by an independent review committee, established to evaluate the scientific validity and ethical justification of the proposed use. Note that only the deidentified participant data are available, and no additional supporting information, such as study protocols or statistical analysis plans, will be provided. This process ensures that the data are used responsibly and in accordance with ethical research standards.
Ethics statements
Patient consent for publication
Ethics approval
The Institutional Review Board of the Tanzania National Institute of Medical Research (NIMR/HO/R.8a/Vol.IX/3990), Stanford University(60379), the ethics committee of the Catholic University of Health and Allied Science (no ID number given), and the Mwanza Regional Medical Officer (Ref. No. AG.52/290/01A/115) approved the study protocol including consent procedures. Participants gave informed consent to participate in the study before taking part.
Acknowledgments
The authors would like to thank the Pediatric Association of Tanzania; the Tanzanian Ministry Of Health, Regional and Council Health Management Teams for participating in stakeholder meetings, Géraldine Jossellin-Duval for her leadership and mentorship of the PACE Learning Engineering Team; Denis Albert and Agnes Hassan for their organising of participants and site work in Mwanza; Michael Alfonzo, Jose 'Jojo' Ferrer, Segolame Setlhare, CLN mentors and facilitators for their valuable feedback on the initial versions of aESNC and its refinements.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @0000-0002-2983-6632
Contributors The manuscript was shaped through the dedicated contributions of each author. PAM accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish. PAM, AH, HN, AD and TJ were pivotal in the study’s conception and design. PAM, AH and HN further extended their contributions to include the acquisition, analysis and interpretation of data. AD and TJ also contributed to the analysis and interpretation of the data. BR, MB, ZHS and AA shared their expertise in the study’s conception and design, while AA additionally took part in the data analysis. JRM, FSK, CLJ, RM, NC, NM, CM and ED were instrumental in interpreting the data. Each author participated in drafting or critically revising the manuscript for key intellectual content. All have approved the final version to be published and have agreed to be accountable for ensuring the integrity and accuracy of the work, comprehensively addressing any questions related to it.
Funding This study was funded by the Laerdal Foundation for Acute Medicine, Stanford University School of Medicine Maternal and Child Health Research Institute, Stanford Center for Innovation in Global Health, and the Stanford University School of Medicine Division of Pediatric Critical Care Medicine. Funding sources had no role in project design, data collection, analysis, or interpretation; reporting, or the decision to submit results for publication. Stanford CTSA award number UL1 TR001085 from NIH/NCRR.
Competing interests BR and MB are compensated by Area 9 Lyceum as Senior Learning Architect and Medical Director, respectively.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.