Evaluation of a web-based portal to improve resident education by neonatology fellows

Background Integration of web-based educational tools into medical training has been shown to increase accessibility of resources and optimize teaching. We developed a web-based educational portal (WBEP) to support teaching of pediatric residents about newborn medicine by neonatology fellows. Objectives 1) To compare residents’ attitudes about their fellow-led education in the NICU pre- and post-WBEP; including assessment of factors that impact their education and usefulness of teaching tools. 2) To compare fellow utilization of various teaching modalities pre- and post-WBEP. Design/methods We queried residents about their attitudes regarding fellow-led education efforts and various teaching modalities in the NICU and logistics potentially impacting effectiveness. Based on these data, we introduced the WBEP – a repository of teaching tools (e.g., mock code cases, board review questions, journal articles, case-based discussion scenarios) for use by fellows to supplement didactic sessions in a faculty-based curriculum. We surveyed residents about the effectiveness of fellow teaching pre- and post-WBEP implementation and the type of fellow-led teaching modalities that were used. Results After analysis of survey responses, we identified that residents cited fellow level of interest as the most important factor impacting their education. Post-implementation, residents described greater utilization of various teaching modalities by fellows, including an increase in use of mock codes (14% to 76%, p<0.0001) and journal articles (33% to 59%, p=0.02). Conclusions A web-based resource that supplements traditional curricula led to greater utilization of various teaching modalities by fellows and may encourage fellow involvement in resident teaching.

W eb-based tools are becoming critical supplements for training medical students, residents, and fellows. Benefits include the ability for standardization, flexibility for the studentÁteacher pair to select the time and place for learning, and ease of accessibility. These alternative teaching options are especially beneficial in the current environment of restricted resident work hours and decreased resident training time in Newborn Medicine per Accreditation Council for Graduate Medical Education (ACGME) guidelines. Studies have identified key subject areas best suited for Internet-based learning, including the critical care setting where learners can become familiar with rare events (1).
Although web-based tools for critical care learning are available, quality varies considerably. Prior to developing Internet-based learning tools, educators need to consider availability, relevancy, redundancy, and an approach for system modification (2). One study identified 135 web-based educational resources that utilized multiple $ Co-Authors: Drs. Lakshmanan and Leeman contributed equally to the work.  modalities varying in utility and accuracy (2). A systematic review examined the effectiveness of web-based learning in health professionals' education and found that interactivity, practice exercises, repetition, and feedback were associated with improved learning outcomes (3). In the absence of customized assessment tools, monitoring impact is challenging. Without proper guidance, the wide range of teaching options can be daunting for educators.
The goals of subspecialty fellowship training include development of skills as clinical leaders, researchers, and teachers. The teaching role of fellows requires mastery of subjects and problem-solving skills when challenged by engaged learners. However, fellows have limited formal teaching instruction and rarely receive feedback about their teaching skills. In addition, successful resident learning is dependent on both the residents' and fellows' level of interest. Several studies have cited that trainees benefit specifically from teaching by fellows (4). However, studies have also cited that fellows often lack a formalized curriculum or tools for teaching (5,6). To help engage fellows in resident education, one program encouraged NICU fellows to develop and implement a new daily didactic curriculum (7). In this study, pediatric resident attitudes toward fellow teaching and the program after implementation were found to be more positive. Webbased modalities are an ideal adjunct to provide fellows with tools for teaching, foster fellow teaching skills, and enhance resident learning.
We created a web-based newborn medicine educational portal to enhance utilization of existing and newly created educational tools, and enhance the neonatology fellow teaching role. The specific aims of our project were 1) to compare residents' attitudes about their fellow-led education in the NICU pre-and post-web-based educational portal (WBEP), including assessment of factors that impact their education and usefulness of teaching tools; 2) to compare fellow utilization of various teaching modalities pre-and post-WBEP.

Setting
We recruited pediatric residents from a large neonatal intensive care unit (NICU) at the Brigham and Women's Hospital (BWH) in Boston, Massachusetts. Residents and fellows rotate through this NICU from an affiliated tertiary care children's hospital (Boston Children's Hospital) from the Boston Children's Residency Program (BCRP) and the Harvard NeonatalÁPerinatal Medicine Fellowship Program (HNPMFP).
The BCRP is ACGME accredited and comprised 150 pediatric residents, most of whom are required to complete 2 four-week NICU rotations (one at the BWH). During pediatric internship [i.e., post-graduate year one (PGY-1)], each PGY-1 rotated through this NICU as part of a team comprised of 1 neonatologist, 1 neonatology fellow, and 2Á4 PGY-1s. This team provided clinical care to 16 Level 2 or 3 neonatal patients. Since 2011, the residency program has followed the ACGME 2011 Duty Hour Standard, limiting PGY-1 to a maximum of 16 continuous duty hours (8).
The ACGME-accredited HNPMFP trained 18 neonatology fellows per year. The first year of fellowship, included four one-month BWH NICU rotations including daytime shifts and overnight call. The second and third fellowship years consisted of 2Á3 months of BWH night call coverage. During the fellow-led daily work rounds and at the bedside, neonatology fellows spent significant time interacting directly with residents and played a major role in resident education. Fellow responsibilities included direct patient care and oversight of the pediatric residents' training. No formal scheduled teaching time or teaching tools had been delegated for fellow-led resident education prior to our intervention. The contribution of the fellowresident teaching interactions had not been previously formalized within the curriculum.

Baseline assessments
Given the coupling of decreasing exposure to clinical time in the NICU and an increasing role of the neonatology fellow as the supervising physician who spends the most time with the resident, resident education in the BWH NICU was becoming increasingly dependent on neonatology fellow-led teaching.
To assess both resident attitudes and utilization of teaching modalities, prior to implementation of our fellowbased teaching curriculum, we surveyed 49 residents who had rotated as PGY-1 residents during the 2009Á 2010 and 2010Á2011 academic years. After characterizing resident teaching modality preferences during the 6 months prior to the intervention, a need was suggested for process improvement to develop a structured curriculum for fellow-led resident education.

Study design
In the setting of the 2011 ACGME Duty Hour Standards altered duty hours for PGY-1 residents, we utilized a quality improvement Plan-Do-Study-Act (PDSA) tactical framework ( Fig. 1) to improve resident education. In the planning component, we assessed resident attitudes about fellow teaching and identified barriers to teaching prior to the intervention. We realized that the residents' educational program lacked both a formal structure for fellow-led resident education and also lacked easily accessible, comprehensive teaching tools.
We designed, piloted, and revised a cross-sectional quality improvement survey to evaluate resident attitudes about current education and teaching practices and then developed content and populated the WBEP with material guided by ACGME-recommended goals. We specifically addressed our first objective by querying residents about factors that both positively and negatively impact fellow-led education as well as usefulness of various teaching modalities. For our second objective, we also asked residents to identify the teaching modalities used by fellows. Because a validated assessment instrument does not exist, we created our survey by establishing face validity. In this approach, authors used a consensusbuilding approach to establish key questions that were relevant to NICU education. Subsequently, we determined process measures that addressed the utility of the WBEP. We developed the WBEP to provide educational prompts for fellows to utilize during impromptu resident teaching and scheduled weekly fellow-run mock codes. Usually, one fellow was paired with 2Á3 residents for these teaching opportunities. The WBEP served as an easily accessible platform for the fellows to utilize different types of teaching modalities to educate the residents. For example, the portal provided sentinel neonatology articles with discussion questions, scripted simulation exercises, and case vignettes for the fellows to describe and discuss with the residents. The teacher and learner team could choose which teaching modality they preferred for each session. We promoted the WBEP in bi-weekly fellow meetings including introduction to the portal, review of the available resources, and encouragement of use for resident teaching.

Intervention
The WBEP, a repository of resources, contained more than 110 resource pages including questions, case discussions, articles, mock codes, and knowledge tests ( Table 1). The resources were designed for a teacher and learner team to review together with a focus on discussion. For the fellow-run mock code scenarios, scripted simulations were available on the WBEP. Scenarios included apnea in the well-baby nursery, acute hypovolemic shock, unplanned extubation in the NICU, and attendance of the delivery of a depressed newborn with meconium stained amniotic fluid. A regular, once weekly assigned time block was scheduled and advertised for implementing the mock codes, which the fellow could choose to utilize. We also included 'virtual patient' cases of classic NICU clinical vignettes with both question prompts for discussion and laboratory and radiographic findings, enhancing the interactivity of the educational content.
We launched the WBEP in October 2011, 3 months after the 2011 ACGME Duty Hour Standards were implemented. The program was updated weekly and accessible to all BWH fellows and residents. The pre-implementation historical cohort was surveyed prior to the intervention in October 2011 and the post-implementation cohort was surveyed between October 2011 and June 2012.

Participants
Our pre-implementation cohort included 49 pediatric residents who rotated in the BWH NICU during the 2009Á10 or 2010Á11 year as PGY-1 residents. The postimplementation cohort included 29 residents who were also PGY-1 residents who rotated in the BWH NICU after  the implementation of the portal. All participants before and after the WBEP implementation were invited by email to complete a cross-sectional quality improvement survey. Both samples were obtained by convenience sampling.

Outcome measures
The primary outcomes assessed were resident attitudes about the impact of various factors on their neonatology education and usefulness of various teaching modalities in their education and the PGY-1 resident report of fellow teaching modality utilization.
We measured resident attitudes and teaching tool utilization using a 10-item cross-sectional quality improvement survey. The survey contained questions about participant characteristics, resident attitudes about factors impacting education, usefulness of teaching modalities, and rate of utilization of teaching modalities. After implementation of the WBEP, we surveyed the first 10 residents who had exposure to fellow-led use of the WBEP to complete our PDSA cycle 1. After examining resident feedback, we revised the WBEP by including additional PowerPoint presentations, journal articles, and clinical vignettes. We then recruited 19 additional residents who completed the cross-sectional quality improvement survey. We completed PDSA cycle 2 in June 2012.
The same survey instrument was administered to the residents pre-and post-implementation to address both objective 1 and 2 of the study.

Analysis
We conducted bivariate analyses to examine associations of predictors with outcomes using Chi-squared test for categorical and t-tests for continuous variables. We used SAS version 9.3 (SAS Institute Inc., Cary, NC) for analyses.
The study was deemed exempt from review by the Institutional Review Board because it was considered a quality improvement project.

Results
A total of 78 residents responded to the quality improvement education survey (51% response rate) Á 49 (54%) pre-implementation and 29 (64%) post-implementation. Resident participant characteristics are presented in Table 2. There was no significant difference between the pre-and post-intervention group in either the number of rotations as an intern, number of months since last rotation, training year of fellow, or attitude about effectiveness of fellow teaching.
To address our first objective of comparing residents' attitudes about their fellow-led education in the NICU pre-and post-WBEP, we surveyed residents about factors that positively and negatively affected their education during the NICU rotation including: patient acuity, fellow responsibilities, fellow level of interest, resident schedule, resident level of interest, attending presence on rounds, and lack of access to teaching tools. We specifically addressed attitudes toward both 'fellow level of interest' and 'resident level of interest' to assess impact of both members of the teacher/learner pair, as both are critical components of a successful educational experience. Interestingly in the pre-implementation survey, as demonstrated in Table 3, fellow level of interest was cited as the factor that most positively (60% of all respondents) and most negatively (60% of all respondents) affected their NICU education. In addition, resident reflections showed that resident level of interest positively impacted their education (56%), while fellow responsibilities (51%) and resident schedules (46%) had a negative effect. The majority of residents (80%) rated fellow teaching as effective. Post-implementation, similar results were obtained as pre-implementation survey results, including a continued highlight on fellow level of interest as a critical component (see Table 3). Comparing attitudes pre-and post-intervention, the negative impact of lack of access to teaching tools was more evident in the pre-implementation compared with the post-implementation group (25% vs. 7%, p 00.06).
As a second component of our first objective to assess resident attitudes toward fellow-led education, we asked residents' about the usefulness of various teaching modalities. After PDSA cycle 1, we found that residents (n010) surveyed post-WBEP implementation identified the following modalities as useful: mock codes (100%), journal articles (70%), case-based discussions (100%), and bedside teaching (100%) ( Table 4). Given these findings, we increased the number of teaching tools available for mock code/simulations, journal article discussions, and clinical case presentations on the WBEP. Additionally, we updated the user interface to improve our presentation of the case vignettes and continued encouragement of fellow use of the WBEP via reminders at biweekly fellow meetings. Subsequently, we completed PDSA cycle 2 (n019) in our outcome analysis. Combining our results from PDSA cycle 1 and 2, we found that residents rated usefulness of journal articles more highly after the WBEP implementation (59% vs. 83%, p00.01).
To measure our second objective of comparing fellow utilization of various teaching modalities, we queried residents about the utilization of various teaching modalities both pre-and post-implementation. Interestingly, although the residents cited that each tool was useful, they also reported that they were not being utilized frequently (range between 12 and 33%) prior to WBEP implementation (Table 5). Using our pooled results from PDSA cycle 1 and 2 (pre-intervention n049, postintervention n029), we found a significantly greater fellow-led utilization of several queried teaching modalities (Table 5). Specifically, there was a reported increase in use of mock codes (14Á76%, pB0.0001) and journal articles (33Á59%, p00.02) by fellows.

Discussion
We describe a successful educational quality improvement intervention, which created a structured teaching role for fellows to supplement an existing curriculum that led to increased utilization of various teaching modalities. We propose that the success of the intervention was the result of the following factors: 1. Accessibility of the WBEP. The WBEP was available online from any computer in the NICU to all fellows and residents. 2. Straightforward user interface. The WBEP user interface was very easy to navigate for the fellow as teacher. The toolbars were highlighted in a different color making it simple to find various resources. 3. Interactivity. Addition of 'virtual patient' cases provided a platform for interactive PGY-1 patient assessment and management with the fellow taking the role of facilitator. 4. Accuracy and quality. The WBEP included neonatology focused high quality resources and tools. The computer-based program permitted weekly modifications, allowing incorporation of the most up-todate evidence-based materials. 5. Encouragement and recognition of the fellow teaching role. WBEP development and focus on the NICU fellow teaching role led to an increased emphasis on fellow teaching. This increased awareness and recognition alone likely contributed to the success of fellow-led resident teaching.
Our first objective, comparing resident's attitudes toward fellow-led education pre-and post-intervention, identified fellow level of interest as a key factor affecting the Our second objective compared the utilization of different teaching modalities by the fellows pre-and postintervention and found an increased use of mock codes and journal articles post-intervention. The increased mockcode utilization may have been stimulated by providing a scheduled weekly blocked time for a potential mockcode session. The increased use of journal articles for teaching occurred at the discretion of the fellow/ resident pair. We can hypothesize that the increased use of journal articles may have been due to ease of access to sentinel and state-of-the-art neonatology articles on the WBEP.
One limitation of this intervention is that this is a single site study. However, the material provided in the WBEP could be generalizable and made accessible to other academic institutions to supplement their neonatology fellow teaching tools for resident education. Another limitation was that the survey measure was not piloted or administered to a focus group. However, it did have face validity as it was reviewed multiple times by the authors. The WBEP was specifically designed to focus on fellow-led teaching tools to enhance interactive teaching, rather than replacing faculty teaching or resident self-learning. While this set-up was by design, it could be seen as a limitation because of the missed opportunity for resident self-learning. However, the WBEP could be easily amended to include an additional self-learning focused section to address this issue.
Each fellow/resident pair had different levels of exposure to the WBEP. The fellow teaching curriculum implemented with the WBEP included only one scheduled weekly session. The remainder of the fellow teaching time was dependent upon fellow and resident level of interest, acuity of patient responsibilities and time constraints. While a strict curriculum was not enforced, we believe that our intervention is generalizable to a realistic NICU teaching environment with opportunities for impromptu teaching sessions arising at different periods of time throughout a resident's rotation.
Only a few publications address quality improvement initiatives in pediatric residency education (9Á11). Baron et al. examined the impact of fellow teaching on resident education after implementation of a locally designed webbased resource (9). Our results similarly suggest that using online resources may have a significant role in supplementing traditional curricula during an era of restricted duty hours.
We demonstrated that a web-based resource that supplemented traditional curricula encouraged fellow-led utilization of various teaching modalities. Utilization of web-based teaching tools presents the opportunity for continued improvement in both fellow teaching skills and resident education. In our current age of technology, teaching using web-based resources will likely become a critical component of medical training. Ongoing work is needed to develop accurate, accessible resources to test the quality of these resources and to assess their impact on resident learning.