Introduction

Intensive surgical boot camps are effective educational experiences to promote technical skills and fill knowledge gaps as learners transition into residency [1]. Boot camps have also proven effective in onboarding residents into subspecialty positions [2, 3]. A Pediatric Surgery Transition to Fellowship Course was launched after a needs analysis concluded that a significant knowledge gap existed among graduating Surgery residents before embarking on their Pediatric Surgery residency [4]. The first Pediatric Surgery boot camp (PS-BC) in North America was held at the Montreal Children’s Hospital/Shriners Hospital for Children in July 2017. This inaugural PS-BC included only Canadian programs (eight PS residents and two research fellows) as well as teaching faculty. Leadership from the Association of Pediatric Surgery Training Program Directors (APSTPD) also attended. Each hosted at a single site, this first and subsequent year PS-BC provided in-person content through a mix of didactic sessions, simulation and animal lab experiences, and demonstrated improved knowledge and learner confidence upon completion of the curriculum [5].

In 2019, the PS-BC was delivered in-person at two sites concurrently (one in USA and one in Canada), with increased attendance and a congruent curricular design. Although detailed plans were nearly finalized for in-person curricular delivery at an Eastern and Western children’s hospital, the COVID-19 pandemic necessitated that the 2020 PS-BC rapidly transition to an entirely online, virtual format to mitigate viral exposure. While video conferencing has existed for several decades, the impact of virtual content delivery on educational value, especially among health care providers overall and surgeons more specifically, is poorly described. In-person curricula and learning for surgeons relies heavily on simulation and animal labs to promote technical skills, involving active demonstration and repetition. But these resources, while undoubtedly helpful to the surgeon-in-training, are resource intensive, and necessitate in-person attendance. Technology that replaces in-person surgical simulation or animal laboratories is unavailable or of unproven efficacy on large scale, as for a PS-BC. A further challenge of digital content delivery to maintain comparable knowledge transmission centers around participant attentiveness. Alternative modes of learner engagement during virtual delivery appear necessary for optimal knowledge uptake and retention. Virtual platforms to facilitate live interaction between faculty and learner are critical features in mimicking in-person experiences. While the near future will see virtual educational delivery as much more sophisticated and realistic than available in 2020, transition to a virtual platform 3 months before the event posed unique obstacles and opportunities as well. We report the process and outcome of a virtual PS-BC, its impact on resource utilization, learner engagement, knowledge acquisition, and stakeholder satisfaction.

Methodology

Course design

Institutional Review Board approval was obtained for this work from IWK Health Centre (REB #1022572). A virtual curriculum was developed in general accordance with the Accreditation Council for Graduate Medical Education and the American Board of Surgery (ACGME/ABS) Pediatric Surgery Milestones and adapted from prior PS-BC. Input was sought from past camp directors as well as members of the APSTPD Boot Camp Committee with ultimate responsibility for the curriculum content shared between two course co-directors (HL, RB). Course content was condensed into a one-day virtual meeting consisting of material addressing communication, leadership, citizenship, self-study and academic productivity, surviving fellowship, and core didactic content. Diverse content experts were invited to pre-record their lectures for digital delivery. A ‘family experience’ of a child with a pelvic rhadomyosarcoma was provided, and a social component bookended the day. To replace simulation and animal laboratories, technical videos (n = 7) of procedures unique to Pediatric Surgery were provided. To enhance learner participation and engagement, the chat board was used frequently for salutations, commentary during lectures, delivery of emphasized teaching points, providing links to additional learning materials (e.g., PubMed and textbooks), and for faculty to entertain questions from learners. Additionally, polling questions were formulated around lecture topics and served as the foundation for multiple post-test questions to assess learner retention. Upon completion of the curriculum, a 1-h social reception was hosted to provide networking opportunities between learners and faculty. Appendix 1 is an overview of the 2020 PS-BC schedule.

Course budget

Budgetary considerations to host the content professionally were addressed through a combination of association stakeholder donations (the APSTPD, the American Association of Pediatrics Section on Surgery [AAP-SoSu] and the Canadian Association of Pediatric Surgeons [CAPS]), industry sponsorship (STORZ), and a single registration fee ($150.00 USD) per participating institution, which allowed as many Pediatric Surgery focused learners as possible. This nominal fee was designed to incentivize and anchor participation as well as defray the cost of memorabilia apparel.

Course delivery

GlobalCastMD (Cleveland, OH) facilitated recording, editing, organization, and live streaming via Zoom (Zoom Video Communications, Inc., San Jose, CA) of all course content. All North American Pediatric Surgery Training Programs (through both Program Directors and Managers) were invited to participate. Target learners were onboarding first-year accredited PS residents, but second-year PS residents and senior fellows in Pediatric Surgery sub-specialty programs (e.g., Fetal Surgery, Critical Care, Trauma, ECMO, Colorectal, Oncology, etc.) and PS research fellows were also welcomed. The PS-BC was delivered on a single 10-h day, which GlobalCastMD hosted through a website landing page [6]. Video recordings were batched and delivered by theme, with live panel discussions periodically held between faculty and facilitated by the course co-directors. The chat function was encouraged throughout the boot camp and influenced real-time discussion. The polling function was employed to deliver pre-planned knowledge assessment questions, as well as to seek immediate feedback from learners using Likert Scale surveys. Upon completion of the course, GlobalCastMD provided a transcript of all activities, including all chat board entries, polling function results, and responses to pre- and post-test questions.

Course appraisal

Registration information was collected from participants, including home institution and level of training. PS-BC finances were itemized for 2020, and comparisons were made to data accrued from the 2019 ‘in-person’ boot camp. Learner engagement and faculty interaction were assessed through parsing chat function data, polling question response rates, and social media activity (e.g., Twitter). Knowledge transmission was assessed through responses to polling questions as well as performance on a knowledge-assessment pre-test and knowledge-retention post-test. These questions were identical to prior PS-BCs and were administered to the learners 2 days before and 8 weeks after the event. Satisfaction data were collected via post hoc surveys from learners, as well as PDs (including participating and non-participating sites), with all surveys being offered online only. Survey questions were primarily Likert-style, although multiple free-text questions were posed, including perceived strengths and weakness of the course. Reminders to complete surveys were provided but not incentivized.

Statistical analysis

Data were aggregated with inferential statistics performed where appropriate. Categorical variables were analyzed using Chi-square tests on SPSS software. Statistical significance was set a priori at p < 0.05.

Results

Participants

Of 56 PS residencies listed on the American Pediatric Surgical Association website, 31 registered (55.4%) for the 2020 virtual BC. All 8 Canadian programs registered (100%), whereas only 23 of 48 ACGME-accredited American programs registered (48%; p = 0.006). A total of 42 learners were registered and participated, 33 of whom were first-year PS residents transitioning from categorical Surgery training. In 2019, 28 learners participated in the in-person boot camp across the two hosting sites, representing an increase in attendance in 2020 of 18% from onboarding PS residents and a 50% increase in overall attendance.

Course financials

The total budget to produce and host the 2020 PS-BC virtually was $15,500 (USD), 54% of the pre-pandemic estimate for the two-site in-person courses. The combined budget of the two concurrent 2019 ‘in-person’ PS-BC was $27,279. The 2020 PS-BC allocated nearly all of its budget towards meeting the costs of the virtual platform, whereas the 2019 PS-BC allocated funds almost entirely to simulation and animal lab costs, which also accounted for the bulk of the planned 2020 curriculum if held in-person. The direct calculated cost/learner in 2020 was $369, compared to $974.25 in 2019. Importantly, these estimates omit costs incurred by home institutions for learner travel and accommodations in 2019. Faculty and directors were not remunerated for their participation in 2020, but those costs were clearly mitigated as well through virtual delivery.

Learner engagement

A total of 308 unique chat board entries occurred across the virtual PS-BC. Of these, 188 were categorized as non-educational in nature (e.g., cordial greetings, acknowledgment of a presentation, questions about a technical issue during one session, etc.), whereas the remainder were direct questions from learners or teaching points from faculty concerning presented content (n = 109) or direct links to further educational materials (e.g., PubMed citations, YouTube video clips, textbooks). Analysis of social media engagement yielded 60 Tweets/Retweets, including 50 during the PS-BC; 41 instances of #PedSurgBootCamp2020 were noted, and 8 Twitter handles were circulated on the chat board.

Course impact

The response rate of learners completing the pre-test knowledge appraisal was 69%, while 17% completed the post-test. A significant knowledge gain was detected, albeit with a significant sample size reduction in the post-test: pre-test correct responses (286 of 589; 48.6%) versus post-test correct responses (89 of 133; 66.9%; p < 0.0002). Included in the post-test were 7 identical questions from the polling engagement function presented during the PS-BC. When correct responses were compared among learners from the day of the PS-BC to the post-test, an increase in score from 49.6 to 57.1% was observed. Construct validity of our testing was established through the differential outcomes observed between experts (faculty) and non-experts (fellows) on correct responses to all polling questions (n = 17) on the day of the PS-BC (77.1% v 49.6%; p < 0.002). An insufficient number of post-test respondents rendered a paired analysis with their pre-test performance statistically unfeasible.

Stakeholder satisfaction

The response rate of learners completing the PS-BC satisfaction survey was 33% (n = 14), while 39% (n = 22) of PDs completed the satisfaction survey, including seven whose trainees were not registered. Learners indicated that the virtual PS-BC generally assisted in transitioning to fellowship and built camaraderie amongst attendees, although most (77%) would have preferred an in-person event (Fig. 1).

Fig. 1
figure 1

Select Boot Camp participant survey reponses (n = 14)

Full size image

Identified strengths of the PS-BC included the incorporation of procedural and technical videos, live chat and polling functions, the inclusion of practical ‘ward survival tips’, and obviating the need for travel. Perceived weaknesses included screen fatigue during a long virtual day, some lectures being overly advanced for incoming PS residents, and a preference for in-person learning.

Program directors (PDs) were generally supportive of the PS-BC, as well as the online format (Fig. 2).

Fig. 2
figure 2

Select Program Director survey responses (n = 22)

Full size image

Some PD respondents expressed concerns about the redundancy of the course with established fellowship transition activities in many programs, the dilution of the collegial environment with the expansion of invitees to non-first year trainees and the necessary pivot to a virtual format, as well as the financial sustainability of the course, despite the clear cost savings. Although the majority (67%) of PDs completing the survey favored an in-person PS-BC, this number is lower than how the learners responded.

Discussion

The global SARS-CoV-2 pandemic accelerated the adoption of new technologies to provide remote patient care and also to address the curricular and educational needs of learners while sheltering from hospital exposure. Virtual learning has been a major issue in North America, particularly on procedural training for various surgical residencies [7,8,9]. These changes have threatened many of our existing patterns of knowledge delivery but have also provided novel opportunities to reduce financial and geographic barriers and potentially uncouple medical education from a fixed time-based paradigm [10]. The 2020 virtual PS-BC reviewed herein exemplifies many of these challenges and opportunities, albeit in the context of subspecialty training after residency—a niche educational category but one with significant concerns given decreasing case counts and exposure to ‘routine surgical practice’ in residency [11].

The virtual version of the PS-BC afforded several tangible advantages to incoming pediatric surgical trainees, including eliminating the need for travel, greatly decreasing institutional costs and reducing time away from service. Importantly, our data identified an increase in number of participants for the 2020 camp from prior years, both first year PS residents as well as other categories of PS learners who were only invited as a result of the virtual platform. This observation may have been a direct result of reducing said barriers to attendance but could also simply represent momentum and enthusiasm in the PS-BC regardless of delivery format. The virtual format also significantly reduced the financial burden of hosting the course for the Course Director institutions, and as well for the learners and participating faculty. Notably, the overall budget of the virtual Boot Camp was nearly halved, and the cost per learner was reduced by two-thirds. These cost-savings were substantial, especially given the economic challenges and uncertainties experienced during the peak pandemic months in the summer of 2020. Prior camps were able to maintain cost neutrality through industry sponsorship; it remains unclear whether future camps will be able to generate financial support from industry as has been traditional in the past. Notably, we did encounter less philanthropic contributions to the PS-BC likely due to economic challenges during the peak months of shelter at home mandates across North America. However, this 2020 PS-BC was generously sponsored through several Pediatric Surgical Associations, industry, and a nominal registration fee.

Certainly, the virtual experience has major limitations with screen fatigue and only short intermissions between sessions as content had been condensed into one lengthy day. Consideration had been given to hosting the event across several days with shorter screen time/day, but course directors were concerned about the audience presence at other major meetings that had transitioned to virtual format; these meetings appeared to experience diminishing attendance over time. We believed it would be most reliable and consistent with the ‘knowledge bolus’ of past camps to host a single day event to ensure the greatest attendance; we did not detect any erosion of learner participation throughout the day.

Another major advantage with the virtual delivery of the PS-BC was the recording of educational content for later viewing, courtesy of GlobalCastMD. Digital archiving of this invaluable material has the potential to reduce the need for future content creation and time expended. Importantly, this material could also be made available to practicing and training Pediatric Surgeons from resource-constrained regions around the globe, who might otherwise not have the opportunity to experience such a boot camp. However, concerns were raised regarding content ownership, copyright laws, and the preferred storage platform for high yield educational material. These issues are just beginning to be framed by medical educators [12]: should content be stored by third party providers that assist in its creation (e.g., GlobalCastMD)? Should specialty organizations like the Canadian Association of Pediatric Surgeons (CAPS) or the APSTPD act as stewards of archived content; if so, how do they regulate access? Would content be available to learners and faculty alike, and at the cost of membership or at further cost to individuals or institutions? Is there a risk of further alienating the disadvantaged already? Is it fair to offer open access if others have paid a registration fee some time in advance?

Importantly, several program directors remain unconvinced of the merits of the PS-BC. Evidence in favor of the positive effects of short, intense educational boot camps exist across several specialties [1, 13, 14], although sceptics will reasonably point out that the durability, and practical applicability of educational content is largely unproven [15]. A small sampling of boot camp participants who completed the post-test showed that learners retained educational content and even enhanced their fund of knowledge. Notably, learner responses were pooled for both the pre- and post-tests, which revealed significantly improved correct answers and indicated a positive impact on learner fund of knowledge. The relatively low post-test completion rate of participants is an acknowledged limitation of the presented data. Indeed, the seven responders to the post-test were arguably more motivated to complete the post-test and therefore could have biased this improved response rate. To evaluate greatest efficacy of educational content, a pre-test should be administered immediately before the intervention and then a post-test upon its completion. The delay of 8 weeks was intentional to assess knowledge retention but certainly impacted lower participation. Furthermore, the satisfaction survey data were incompletely anonymized (respondent identity was available only to the course co-directors), which may have reduced the potential to provide negative answers from junior learners. Given the abundance of constructive feedback from PDs, this lack on anonymity appeared to have not influenced their opinions.

We did not expect to observe such disparate participation between PS training programs from Canada (100%) and the United States (48%). These differences may be explained by the recent origins of the PS-BC in Canada as described above. More concrete evidence of the PS-BC benefits to learner education and transition to a pediatric hospital environment needs to be gathered to convince those PDs who have ongoing reluctance to register their learners for future PS-BCs. Nevertheless, we did observe an encouraging increase in participation of all level learners, which was novel to the virtual format, but also specifically first-year PS residents as well.

While opinions varied between learners and directors about the merits and weakness of the 2020 virtual PS-BC, relative unanimity emerged that all participants craved an in-person experience. The ability to participate in hands-on technical teaching during animal labs was lost, simulation activities could not be adequately transitioned to a virtual platform on short notice, and establishing social connections was clearly inferior in the virtual world than in person. And yet, nearly everyone reading this article will be doing so online. Seeing patients through HIPAA-approved virtual formats and holding high level administrative meetings without leaving an office is now the norm rather than the exception. Indeed, some authors of this manuscript have essentially only met virtually. Change is the only constant. Future course directors of the PS-BC, much like all medical educators, will need to carefully consider learner needs when weighing the potential of in-person versus virtual learning activities. Hybrid approaches that promote the relative advantages of in-person learning while leveraging the lessons of a virtual platform may prove to be the most appealing future option; as may case-based learning [16, 17]. While emerging sophisticated simulation opportunities in a virtual format hold significant promise, these tools are currently fledgling and were not available on short notice in the summer of 2020. It is ultimately hoped that these data will inform future curricular development for surgical bootcamps, whether hosted in-person, virtually or using a hybrid model depending on our capacity to gather.