Tracking and debriefing birth data at scale: A mobile phone application to improve obstetric and neonatal care in Bihar, India

Abstract Aim This analysis assessed changes over time in skill and knowledge related to the use of evidence‐based practices associated with quality of maternal and neonatal care during a nurse midwife mentoring intervention at primary health clinics (PHCs) in Bihar, India. Design Nurse midwife mentors (NMMs) entered live birth observation data into a mobile App from 320 PHCs. Methods The NMMs completed prompted questions in the App after every live birth witnessed. The App consisted of questions around three main themes, “What went well?”, “What needed improvement?” and “What can be done differently next time?”. Results Observational data from 5,799 births was recorded by 120 NMMs in 320 PHCs. Knowledge and skill during normal spontaneous vaginal deliveries and complicated deliveries with either a postpartum haemorrhage or non‐vigorous infant all showed statistically significant improvement (p < .001) over time using a Chi‐squared test for trend with a mean increase of 41% across all indicators.

To address this high level of maternal and neonatal mortality, a large quality of care improvement initiative was implemented in Bihar by the non-governmental organization, CARE India, in close collaboration with the State Government of Bihar (Das et al., 2016). The statewide initiative aimed to improve the quality of obstetric and neonatal clinical care provided at every primary health clinic (PHC) in the state of Bihar through a variety of interventions including infrastructure improvements, increased supply procurement and the implementation of a mobile nurse-mentoring programme. The mobile nurse-mentoring programme aimed to improve auxiliary nurse midwife (ANM) and general nurse midwife (GNM) clinical skill and management practices during birth. To accomplish this, PRONTO International (PRONTO) partnered with CARE India to integrate simulation, team training and postevent debriefing after live births into the nurse-mentoring programme. Postevent debriefing, is understood to be an effective aspect of clinical education, quality improvement and systems learning (Agency for Healthcare Research and Quality, 2016) as it provides a space for self-reflection by providers including their role and behaviour, knowledge and skills, and team operation. In this intervention, postevent debriefing of live births was defined as a "structured and guided reflection process where students actively appraised their cognitive, affective and psychomotor performance within the context of their clinical judgment skill" (Al Sabei & Lasater, 2016). Mobile Apps have been previously shown in pilot studies to have potential benefits to debriefing in advanced life support simulations such as enabling the debriefer to have a more intuitive visual summary of the skills and techniques used during practice (Chang, Su, Lin, & Huang, 2015). Debriefers who used the App, were found to provide more and richer feedback than those not using an App (Chang et al., 2015). Additionally, studies evaluating programme outcomes of community health workers in low-resource settings have provided some evidence that mobile tools can help to improve the quality of care provided (Braun, Catalani, Wimbush, & Israelski, 2013). This paper discusses the results from the implementation of a postevent debriefing intervention by looking at the feasibility of a mobile App as both a job aid for postevent debriefing as well as to track changes in provider skill, teamwork and supply availability overtime.
In this study of a cross-sectional programmatic intervention, we examined the changes over time in ANM/GNM clinical knowledge and skill during deliveries at PHCs in Bihar, India. The findings were based on data collected during postevent debriefs during the nursementoring intervention. This primary aim of our study was to assess the effectiveness of the nurse-mentoring programme in improving quality of ANM/GNM clinical skill and knowledge during birth. The secondary aim of our study was to assess the acceptability of integrating a mobile App as a job aid during deliveries at PHCs in Bihar.

| Design
CARE India, working closely with the Government of Bihar, implemented the Integrated Family Health Initiative project in 2011 aimed at the reduction in maternal and childhood mortality in eight priority districts in Bihar (CARE India, 2013). The initiative was subsequently scaled up to all 38 districts in the state of Bihar beginning in January 2015 and ending in January 2017. As a part of the subsequent state-wide implementation of the programme, mobile nurse mentoring was conducted using a phased approach to roll-out over the course of 8 months in four phases. Each phase targeted 80 PHC facilities which provided basic emergency obstetric and neonatal care. PHCs were assigned to phases based on delivery load, facility readiness and support among leadership. By the end of the fourth phase, 320 PHC facilities in Bihar had received the nurse-mentoring programme. In the PHCs, mentorship was provided to six to eight ANM/GNMs working at each facility. ANM/GNMs were selected for participation in the programme based on having labour room duties as well as their interest in participation.
During implementation of the four phases, 120 nurse midwife mentors (NMMs) worked in pairs spending 1 week per month at each of their four assigned facilities, constituting eight total weeks of onsite mentorship per PHC state wide. On-site mentoring included an 8-week curriculum. The activities and inputs of this curriculum included simulation training, teamwork training, skills building and postevent debriefing after live births. Each of the four phases of implementation had the same curriculum. Overall, the NMMs were younger and less clinically experienced but had higher levels of clinical education than the ANM/GNM site staff or "mentees" they were working with at the PHCs. Most NMMs were not from the state of Bihar and had relocated temporarily for the job.

| Method
Use of postevent debriefing of live births as a mentoring strategy was implemented in PHCs. In the first phase of the programme, the debriefing tool was piloted as a paper-based tool in 80 PHCs. For the next three phases the debriefing tool was implemented using a mo- SBAR, check back, call out, mother spoken to directly, mother spoken to kindly, call for help, help arrived and transfer plans initiated). Supply availability based on visual verification was also collected through questions in the mobile App (i.e. antiseptic solution, sterile gloves, clean towel, sterile ties/clamps, sterile blade, Oxytocin, intravenous (IV) supplies, ambu-bag and mask and suction device). Questions around "what went well" vs. "what needed improvement" were presented in a dichotomous format requiring the NMM to provide a subjective assessment based on their observation of how the mentored providers performed during the observed birth. A response to each question was required to move on to the next question and there was also an option to mark "Not applicable" if a question was not relevant in a specific case and "Don't know." These responses were excluded from the analysis.
The mobile App asked if a complication occurred during the observed birth and if so a series of topic-specific questions were then generated to supplement those that appeared for every birth around the additional agreed on evidence-based practices. The four complication types that generated follow-up questions in the mobile App were postpartum haemorrhage, non-vigorous infant, pre-eclampsia and preterm birth. Postpartum haemorrhage was defined as abnormal blood loss estimated above 500 ml. A nonvigorous infant was defined as a baby born without crying and asphyxia referred to a baby born without crying after drying and stimulation were performed. Pre-eclampsia was defined as onset blood pressure of greater than or equal to 140/90 after 20 weeks of gestation, with or without features of severe pre-eclampsia.
Preterm birth was defined as the birth of a baby at gestational age of less than 37 weeks 0 days. The indicators collected for these complications included: pre-eclampsia (identification of hypertension, identification of hyperreflexia, identification of pre-eclampsia, reflexes checked, magnesium given), preterm (diagnosed as preterm labour before delivery, identified as preterm baby, IV placed, steroids given, tocolytics given), non-vigorous infant (immediate newborn care provided, identification of non-vigorous infant, identification of asphyxia, identification of meconium, bag and mask ventilation, suctioning), postpartum haemorrhage (identification of abnormal bleeding, 20 IU of oxytocin given and uterotonics given).
NMM were also able to select "Other" for a complication not listed and instructed to input the type into a text field. This response, however, did not generate additional questions in the mobile App.
When the NMM stepped in to either assist or conduct the delivery themselves, based on their ethical obligation to the patient, they were still instructed to answer the mobile App questions based on their opinion of how the "mentees" performed or would have performed, not on how they actually performed with assistance. NMM's were also asked in the mobile App about the level of support they provided during the delivery to the mentees. This was included to have an indication of level of assistance that was received during each birth.
The final page of the mobile App was a customized screen that provided the NMM with a tailored debriefing guide specific to the birth they had just observed based on the data they had entered.
The feedback guide was designed to aid the NMMs in facilitating a debrief with the participating providers.
Additionally, to better understand both acceptability and use patterns of the mobile App, in August 2016, the NMMs from Phase 4 were asked to complete a brief five-question online anonymous survey about the mobile App.

| Analysis
Aggregate level data collected on the ODK mobile App platform was available in the format of pre-specified tables and figures that could

| Ethics
The Institutional Review Board committee of the Indian Institute of Health Management Research in Jaipur and the Committee for Human Research at the University of California San Francisco reviewed and approved the study (Study ID# 14-15446).

| RE SULTS
In Phases 2-4, live birth observations were recorded in the mobile App for 4,923 deliveries at the 240 PHC facilities mentored by 120 NMMs (Table 1). Of the observed births, 22% were identified as having one or more complication (N = 1,088).
Among the births recorded as having one or more complication, having a non-vigorous infant was the most common type of complication which occurred in 7% of births, followed by postpartum haemorrhage in 6% of births, preterm birth in 2% and pre-eclampsia in 1%. "Other" complications including, for example, low birth weight, cervical and perineal tears, breech presentation, still-birth, retained placenta and multiple births were recorded in 6% of births. Based on ethical obligations at times when NMMs observed a complication and assistance was required they stepped in to assist with the delivery. Their level of participation was assessed on a Likert scale from "1" meaning no assistance to "10" indicating the NMM conducted the entire delivery. During complicated births the score averaged 4.9 compared with 4.2 in normal births. Additionally, a question at the end of the mobile App ensured that NMMs could select who "conducted" and who "participated" in each delivery with "NMM" as one of the job classifications. In 77% of complicated cases the NMM reported that they "conducted" the delivery themselves, whereas in normal deliveries the NMM reported that they "conducted" the delivery in 72% of births. Since these high levels of intervention by the NMM's in the deliveries bias the results of whether a clinical practice occurred or not, our analysis focuses only on data collected through the questions where NMM's were asked to subjectively assess the mentees performance by determining "what went well" and "what needed improvement" during the delivery.  Seventy-four per cent of respondents reported that they used the mobile App during or after deliveries "always" or "most of the time." Respondents were asked the question, "What do you like about using the mobile App?" Respondents most commonly stated positive characteristics including the mobile App being easy to use, offline and contributing to debriefs. Two NMMs stated: "It's easy, quick and helps us cover all that we need to [in the] debrief." "It is easy to use and doesn't take much time. It is offline." Respondents also appreciated that the mobile App helped them to see the areas where the mentees needed more practice. One respondent stated: "It helps to know in which part mentees need more practice." Lastly, one respondent appreciated how the data from the

| D ISCUSS I ON
We found that the nurse mentorship model is an effective method of transferring basic emergency obstetric and neonatal care knowledge and skills in low-resource settings. Results show that the mobile App is an acceptable job aid in a nurse-mentoring programme in Bihar and that it can be used to monitor changes in medical management, teamwork and supply availability over time. Additionally, these data support larger study results, presented elsewhere (Das et al., 2017), that suggest that the larger state-wide program increases these indicators over time. Other implementation programmes using skills training in India have also pointed to improved provider skill and knowledge (Bellad et al., 2016;Goudar et al., 2013   We also found that, communication with the mother during delivery was another indicator that showed strong improvement from beginning of the intervention to the end (Table 2) Center have found that training using a patient-actor may be better at improving perception of safety and communication than training with a computerized manikin simulator (Crofts et al., 2008).
Interestingly, significant improvements around supply availability were also reported during the span of the nurse midwife mentoring programme, although the overarching intervention had no specific arm that directly addressed the issue. This is believed to be one of the unintended, yet beneficial, consequences of this intervention.
These improvements were likely due to NMMs and mentees gaining a better understanding through the programmatic curriculum of the inherent importance and need for adequate supplies, including medications, to ensure safe and healthy outcomes for both the mother and baby. It is possible that supplies in a facility were made more readily accessible to facility staff by removing physical barriers to acquiring them (i.e. opening locks on cabinets, etc.) and that PHC patients were increasingly instructed on how to acquire supplies and medications independently as there were asserted, yet unconfirmed, reports of increased medication purchasing by community members.
The routine monitoring of these data from the mobile App was also an effective method for data-driven monthly programmatic feedback to the NMMs themselves.  in Phase 2 were a potential hindrance to data collection and therefore also may have an unknown impact on the results. The mobile App was initially designed in such a fashion that it could store data offline and whenever connectivity was restored it could push the data off the phone into a web-based cloud. Although this worked nicely during a small initial pilot with a sample of ten NMMs, technical issues were continually reported by Phase 2 mentors and thus a platform migration to ODK was implemented after 7 months. However, since two mentors were assigned to each facility, both of whom had a smartphone with the mobile App loaded this issue is not perceived to have prevented entry of a significant amount of data since if errors were encountered on one phone they could try on the other. Finally, although actual neonatal and maternal clinical outcomes would be the most rigorous way to assess the impact of the mobile debrief App, the unreliable birth and discharge registry data made this impossible in this context.

| E THI C S APPROVAL AND CON S ENT TO PARTI CIPATE
All participants provided written consent for the use of video simulation data in an aggregated analysis. Ethics approval was granted from the institutional review boards of the University of California San Francisco (14-15446) and the Indian Institute of Health Management Research.

| CON CLUS ION
In conclusion, we found that a mobile App is an acceptable job aid in a nurse-mentoring programme and that it can be used to monitor changes in medical management, teamwork and supply availability over time. Further research could help determine if use of a mobile App improves debriefing skill and if debriefing itself has a positive impact on maternal and newborn clinical outcomes.

ACK N OWLED G EM ENTS
The authors thank the Database Management System development team of CARE India including Sourav Tanti for their efforts in adapting the application onto ODK and developing the ONA based dashboard. We also thank all of the mentees, facility in charges

Dilys Walker and Susanna Cohen are founding members of PRONTO
International and sit on its board of directors. None of the other authors have any conflicts of interest to declare.