Procedural Competency for Pediatric Residents in the Contemporary Training Environment: An Unachievable Goal?

Background: Pediatric residents do not get adequate experience with the procedural skills required by the Accreditation Council for Graduate Medical Education (ACGME). This lack of procedural experience can impair procedural competency prior to entering unsupervised practice. Objective: Describe procedural experience of pediatric residents at diverse tertiary care academic centers in United States, and how procedural experience was impacted by year of training, program size and career choice. Methods: Anonymous survey was emailed between September 2017-July 2018 to current pediatric residents (2 nd /3 rd year) and recent graduates of nine demographically and geographically diverse programs. Subjective recall of procedural experience, self-perceived proﬁciency in procedural skills and preferred training methods were explored. Results: Response rate was 28% (173/622). Only 6% (11/173) had completed >/= 1 of each procedure recommended by the ACGME. Residents who were more advanced in their training, from smaller and medium sized programs and with deﬁned plans post-residency completed more procedures. Most (90%) desired additional procedural exposure. Conclusions: Majority of recently graduated pediatric residents had not attempted all ACGME required procedures, and were


Introduction
The Accreditation Council for Graduate Medical Education (ACGME) requires that pediatric residents achieve competency in 13 procedures (Supplemental File 1), (Accreditation Council for Graduate Medical Education, 2017). Discussions about which procedures remain within the scope of pediatric residency training and general pediatric practice are ongoing (Ben-Isaac et al., 2013). It is unclear if pediatric residents are meeting this requirement. Individual residents' procedural experience is influenced by program schedule, presence of fellows, hospital size/acuity, location, seasonal variations in patient illness (Hicks et al., 2000;Chaytors, Szafran and Crutcher, 2001;Goertzen, 2006;Lee et al., 2010;Schumacher, Frintner and Cull, 2016;Schumacher, Frintner and Cull, 2018), and clinical practice changes (Moote et al., 2011). These effects may be exacerbated by dedicated procedure teams and increased use of advanced practice providers (Moote et al., 2011). Previous studies reported that pediatric residents failed to meet ACGME's procedural requirements, but most were limited surveys of residents prior to graduation, and do not include pediatric residents' assessment of procedural training adequacy (Gaies et al., 2007;Schumacher, Frintner and Cull, 2018).
The current study assessed procedural experience of pediatric residents and recent graduates, and examined their perceptions regarding adequacy of procedural training. We hypothesize that procedural experience will be influenced by post-graduate year of training, program size and career choice.

Study overview
The study followed a cross-sectional observational study design. Subjects included pediatric residents and recent graduates of ACGME-accredited programs between September 2017-July 2018. The primary outcome was to quantify the number of procedures performed by pediatric residents, while assessing impact of program size, location and career choice. Secondary outcomes include perceived sufficiency of procedural volume.

Study subjects
Current 2 nd and 3 rd year pediatric residents in nine ACGME-accredited programs in the United States, and recent graduates, were invited to participate. Programs varied in size and location ( Figure 1). Survey tool An electronic survey, developed using online software (Qualtrics, Provo, UT), was distributed via email. A questionnaire previously utilized to assess procedural experience in neonatology fellows (Sawyer et al., 2016) guided survey development. Questions on demographics, numbers of specific procedures performed, cumulative procedural performance, and use/perceived accuracy of procedure logs were included (Supplemental File 2). Feedback was solicited from pediatric faculty/trainees (three fellowship program directors, three faculty educators, one fellow, and three residents). Residents from one program completed pilot testing to optimize clarity and functionality of the 32item tool prior to distribution. An email survey link was distributed to eligible subjects in July 2018, with a reminder one month later.

Variables of Interest
Post-graduate year of training was classified as PGY2, PGY3 or resident graduate. Program size was either small (<15 residents/class), medium (16-25 residents/class), or large (>25 residents/class). Career choice was either fellowship, hospitalist or primary care. Procedural proficiency was defined as >5 performed procedures (DeMeo et al., 2015).

Statistical Analysis
We aimed at reaching at least 30% of 622 eligible residents, to get a representative number of participants. Each of our variables of interest had 3 levels, therefore, with 30-50 subjects per level, we were powered at 80% with twosided alpha of 0.05 to find effect sizes in the range of 0.3-1.0, using a two-sided Mann-Whitney test. Descriptive statistics summarized continuous and categorical variables. Between-group comparisons in the number of procedures were performed using Student's t-test (data summarized as means with standard deviations) and Wilcoxon Rank-sum test (data summarized as medians with range). Since the primary study focus was to provide associative information for future studies, we did not perform adjusted comparisons and did not adjust for multiple comparisons. Statistical significance was established at alpha=0.05. Analyses were performed using SAS 9.4 (Cary, NC). Yale's Institutional

Competency Assessment
Existing programmatic requirements for achievement of "competency" were reported by 67% (116/173). Ten percent (17/173) reported a standardized process for determining procedural competency prior to performing a procedure independently. The most cited practice was completing >5 supervised procedures, which was reported by 47% (8/17).

Program Size
Significant differences in procedural experience were noted in programs of varying size. Residents in large programs performed fewer procedures than those in medium and small programs(p=0.04). Residents in small programs performed more bladder catheterizations (p=0.004) and neonatal endotracheal intubations (p=0.026) ( Table 3). Medium and small programs had greater exposure to BVM, lumbar puncture, and umbilical catheter placement.  Suggestions to improve exposure were procedural electives (143/173, 83%) and simulation-based training (87/173, 50%) (Table 1).

Discussion
This cross-sectional cohort study investigated procedural training experience amongst a diverse group of pediatric residents and graduates of ACGME-accredited training programs, and affirmed concerns that present-day pediatric residents have fewer opportunities to attain competency in ACGME-required procedures (Gaies et al., 2007;Schumacher, Frintner and Cull, 2018). Although it was anticipated that future career choice and geographic location may influence procedural exposure, only program size influenced overall procedural numbers. A majority of residents deemed their current procedural training as insufficient, and desired supplemental training in at least one procedure.
Residents reported increasing procedural experience throughout training. Deficiencies exist with regards to specific procedures, as noted by cumulative numbers performed and residents' reported self-perception of inadequate procedural experience. This is a significant decline from previous eras (Leone, Rich and Finer, 2005;Gabrani et al., 2018) with numerous contributing factors, including changes in medical management (Finer et al., 2004;Wyckoff et al., 2015), staffing (Moote et al., 2011;Smith and Hall, 2011), supervision (Gozzo et al., 2011), and workflow (Accreditation Council for Graduate Medical Education, 2017). Previous authors have reported decreased volume for specific procedures (i.e., intubation), but this study demonstrates a decline across all procedures (Gozzo et al., 2011;Engorn et al., 2018). Residents are now infrequently performing procedures on the floor and intensive care units, which may be due to presence of subspecialty fellows or other staff who need to develop proficiency.
Similar to previous reports, (Schumacher, Frintner and Cull, 2018) program size affected procedural volume. Smaller programs may afford opportunities for procedures given fewer additional providers (fellows, practitioners) or absence of competing centers regionally. However, it is unclear if these differences were clinically meaningful. Different from previous descriptions of procedural experience amongst pediatric residents entering primary care positions (Walton and Edwards, 2002;Ben-Isaac et al., 2013;Schumacher, Frintner and Cull, 2018), we assessed for differences in residents entering subspecialties that require technical proficiency. Despite assumptions that trainees would advocate for additional procedural opportunities, their cumulative procedure numbers did not differ, suggesting influence of other factors (competition for procedures by other providers, limited procedural numbers overall, and work-hours restrictions). These residents must rely on fellowship training to master skills, compounding concerns regarding declining procedural opportunities for subspecialty fellows (Gozzo et al., 2011;Mittiga et al., 2013;Engorn et al., 2018).
In our study, residents reported procedural numbers by referencing their procedure log. Although most residents are required to use procedure logs, concern exists regarding accuracy of the reports, and inconsistency about what should be included (i.e., recording unsuccessful attempts). Currently, the ACGME does not require that pediatric residents maintain a procedure log as it does for surgical residents, where concerns about compliance and accuracy have been raised (Accreditation Council for Graduate Medical Education, 2020; Wolf et al., 2017). It would be beneficial for the ACGME to monitor pediatric residents' procedural experience across programs to inform expectations on feasibility of current procedural training requirements, potential modifications necessary, or ability to supplement procedural training with simulation. Local/ national data collection would allow PDs to identify trainees who are falling behind their peers with regards to procedural volume/performance using training specific metrics or a dashboard (Accreditation Council for Graduate Medical Education, 2017).
A major challenge of assessing resident procedural performance is that valid, standardized methods to define and assess procedural competency have not been established. Of programs having a defined system, most used an arbitrary numeric threshold for procedures completed under direct observation which the majority of residents are not reaching. This method does not evaluate cognitive procedural aspects or performance of discrete psychomotor steps, both of which contribute significantly to procedural success and safety. Further research is needed to define competency in all ACGME-required procedures, similar to previous work on neonatal intubation (Leone, Rich and Finer, 2005;DeMeo et al., 2015). In a retrospective single center observational study, using statistical modeling to estimate outcomes of resident intubations, pediatric residents needed to perform >4 successful intubations to be competent (DeMeo et al., 2015). In our study, most residents did not perform 4 intubations during residency. Standardized expectations for procedural training should exist across programs to protect patient safety, and include assessment of procedural sequence/technique based upon assessment tools with documented evidence of validity. Identification of minimum procedural thresholds for graduation might lead to procedural goal achievement. In addition, if these numbers are unattainable clinically, simulation or other modalities may serve as substitutes.
As it is unlikely that the majority of pediatric residents are meeting ACGME's existing expectations for procedural training, it may be time to re-evaluate what procedures are necessary for pediatric residents (Gilhooly, Redden and Leonard, 2015). Previous research has focused on skills required by primary care physicians, and was limited to graduates from a single pediatric residency program (Ben-Isaac et al., 2013) or general pediatricians in a single state (Iyer et al., 2019). Recently, general pediatricians noted 11 of the ACGME-recommended procedures were never performed in practice, yet were perceived to be important (Iyer et al., 2019). National surveys of primary care pediatricians and subspecialty PDs could inform discussions on procedures necessary for current practice, including which should be mastered during residency. Individual residents may benefit from specialized pathways based on career goals, with differing procedural expectations.
Novel strategies are needed to supplement procedural training for pediatric residents if the current 13 procedures remain "essential," presenting a challenge for pediatric educators. Residents desire alternative procedural training  (Augustine and Kahana, 2012;Barsuk et al., 2012;McMillan et al., 2016). Programs should consider implementing structured experiences combining simulation and clinical attempts using an established framework (Sawyer et al., 2015). Other strategies include electives which may improve exposure to particular procedures (ex., intubation during ENT/anesthesia rotation).
Our study has some limitations. It is unclear if residents utilized procedural logs to answer survey questions, which may lead to recall and response bias. Logs may contain inaccurate information. Due to the cross-sectional nature of the study, we were unable to ascertain individual subject's procedural volume over years of training. We surveyed a selection of institutions across a spectrum of size and geographical locations, but study enrollment was below target. Therefore, findings may not be generalizable to the population of residents as a whole.

Conclusion
In a cross-sectional sample of residents from programs of varying sizes and locations, we demonstrated low procedural volumes that likely challenge program's ability to effectively establish trainee procedural competency during training. ACGME may need to reassess what procedures are necessary in current pediatric practice and how to feasibly establish competency within training constraints, including supplementation with simulation-based training and development of individual career-based learning pathways.

Take Home Messages
The Accreditation Council for Graduate Medical Education (ACGME) requires that pediatric residents achieve competency in 13 procedures. Approximately four out of five recent graduates have not performed at least one each of the required procedures during residency. Significant differences in procedural experience were noted in programs of varying size. Majority of graduates desired additional procedural training.