Educational Handoffs between Medical School and Residency: A National Survey of Residency Program Directors

Background: Educational handoffs between medical schools and residency may facilitate the transition into graduate medical education. Attitudes of residency program directors about handoffs are largely unknown. Objective: This study assessed the opinions of residency program directors from four specialties, regarding information sharing between medical schools and residencies, and whether they desire student performance data post-match, including milestones, competency and entrustable professional activity (EPA) achievement, and capstone or boot camp-based performance data. Methods: In 2017 a thirteen question survey was sent to program directors from internal medicine, pediatrics, emergency medicine, and obstetrics/gynecology. Responses were anonymous and data was analyzed using descriptive statistics and thematic analysis. Results: Twenty percent of program directors surveyed submitted a survey (187 out of 950). Fifty-nine percent reported that unexpected remedial issues often arise within the first 6 months of internship. Twenty-four percent stated that information currently received is adequate to structure the learning environment of individual residents. Seventy-seven percent responded that they desire access to EPA-based information; seventy-four percent to milestone or competency-based data; eighty-two percent to performance data from fourth-year capstone courses or boot camps. No significant differences were found between specialties. Concerns were raised by respondents about the validity and reliability of performance metrics that would be shared with program directors. Cooper A, Walker C, Splinter A, Khan M, Schaffernocker T, Verbeck N, Kman N, McCallister J MedEdPublish https://doi.org/10.15694/mep.2018.0000047.1 Page | 2 Conclusions: The majority of the program directors who responded to this survey indicated that they desire access to and would utilize medical student performance data post-match, though concerns were raised about the validity and reliability of proposed metrics.


Introduction
In 2016 the Association of American Medical Colleges (AAMC) issued updated recommendations as to the content of the MSPE. 1 Traditionally, communication between medical schools and residency programs regarding medical student performance and abilities has consisted of pre-clinical and clinical transcripts and the Dean's letter, or Medical Student Performance Evaluation (MSPE), shared during the application to residency. Concerns have been raised in the medical education community about the utility of the MSPE to adequately convey students' abilities and training needs anticipated during residency. 2 With the goal of providing information about anticipated resident learning needs in addition to the evaluative and summative information provided by the MSPE, many have continued to call for an augmentation in communication between medical schools and residency programs prior to interns' starting clinical training. One proposed mechanism is the institution of educational handoffs. 34 An educational handoff entails the sharing of known performance data about medical students with residency programs, post-match and prior to the start of internship. The Accreditation Council for Graduate Medical Education (ACGME) has set a precedent for this type of information sharing, requiring what amounts to an educational handoff between residency and fellowship, and the ACGME now mandates that fellowship programs receive residency milestone performance data on incoming fellows.
Achievement regarding entrustable professional activities (EPAs), tasks and skills that medical students should be able to perform unsupervised, as well as performance data from capstone courses or boot camps at the end of medical school, are other forms of learner performance data that could potentially be handed off to residencies. 5,6 There are many educational outcomes that can be incorporated into an educational handoff. Pilot studies at the University of Michigan have focused on handing off ACGME educational milestone data of medical students entering emergency medicine and general surgery residencies, with positive feasibility results. 2 The revised AAMC MSPE recommendations include the inclusion of ACGME competency information about medical students when available. 7 We conducted an anonymous, national survey of residency program directors assessing attitudes regarding the current state of information sharing between medical schools and residency programs, and about the prospect of using competency, milestone (being related to competency achievement), EPA, and capstone/boot camp-based data for educational handoffs in the future.

Methods
In 2017 a thirteen question survey was developed based upon author consensus and review of the available literature regarding the current state of performance evaluation and information sharing between medical schools and residency programs. The questions were assessed for clarity and relevance by members of the research group, several of whom are content experts in medical student and post-graduate assessment. The survey consisted of items rated on a 5-point Likert-type agreement scale, yes/no questions, and a free text response space for comments. It consisted of two parts, the first querying the current status of information sharing between medical schools and residency programs and the second assessing program directors' opinions about post-match educational handoffs.
These specialties were chosen as a representative sample of residency types across the United States. The survey was completed anonymously with respondents indicating their specialty in the form. Two reminder emails with a link to the survey were sent five and twelve days after the initial email. One thousand nine-hundred and ninety-nine emails were sent initially. After accounting for bounce-backs and duplicate addresses, 950 emails were sent in the second reminder email. Institutional Review Board exemption was obtained from The Ohio State University, Wexner Medical Center's Institutional Review Board. 8 The survey was sent electronically to internal medicine, pediatrics, emergency medicine, and obstetrics/gynecology programs and program directors via contact information available from the American Medical Association FRIEDA database.
Responses were analyzed using descriptive statistics and thematic analysis. Two authors (A.C. and N.V.) coded and performed an iterative thematic analysis on free-text comments from respondents, and consensus was reached.

Results
One-hundred and eighty-seven program directors submitted a partial or complete survey, for a 20% response rate (187 out of 950). Respondent specialty was equally represented with no significant differences shown between specialties. Few program directors reported currently receiving competency, milestone, or EPA-based performance data from medical schools after the match (11% receive competency, 1% receive milestone or EPA data). Similarly, only 1.2% of respondents routinely receive information about the strengths and weaknesses of incoming interns (2 out of 164). When asked about unexpected issues requiring remediation that arise within the first 6 months of residency, 59% of respondents stated that this occurs frequently or often enough to be noticeable ( Figure 1). Twenty-four percent of respondents strongly agreed or agreed that information they currently receive from medical schools is adequate to structure their residents' learning environment, and 26% strongly agreed or agreed that they use such information in that manner (Table 1).
Seventy-seven percent of respondents strongly agreed or agreed that they wanted access to post-match information about medical student EPA performance, and 59% strongly agreed or agreed that they would use this performance data if available to structure residents' learning experiences. Similarly, 74% of respondents strongly agreed or agreed that they would want access to milestone and other competency-based performance data, and 61% strongly agreed or agreed that they would use this data if available to structure residents' learning experiences. Eighty-two percent of program directors strongly agreed or agreed that they would like to have access to information about student performance at the end of medical school, such as from capstone courses or boot-camps (Table 2).
Survey responses were analyzed for potential differences according to specialty [ Table 2], and no significant difference was found. Free-texted comments were solicited in the survey, and 15% of responding program directors provided comments (26 out of 165). Thematic analysis identified core concepts including (1) concerns about accuracy of the Dean's Letter, (2) questions about the validity and reliability of performance metrics, particularly medical student educational milestones, (3) a desire for information on procedural skills, (4) desire for forewarning about issues such as professionalism, behavioral concerns, and special learning needs, and (5) concerns about the lack of uniformity of assessment at different medical schools. A representative sample of comments is included in Table 3.

Discussion
The majority of program directors who responded to this survey felt that there are gaps in the information currently received about incoming interns. These gaps include students' strengths and weaknesses, and the usability of information received to structure the learning experience for a given learner. Unexpected issues requiring resident remediation were noted to occur frequently. Most respondents felt that they would utilize milestone, competency, and EPA-based assessment data from medical schools if it was accessible to them. Access to performance data from capstone courses and boot-camps from the end of medical school was similarly desired. Several concerns about the handoff process were raised by respondents, including the validity and reliability of the performance metrics being shared and the lack of uniformity of assessment at different medical schools.
Performance assessments in capstone courses and boot camps are the most proximal to the start of residency and represent the most up-to-date information available on an incoming intern. Transferring this information to residency programs in the form of an educational handoff could be instrumental in guiding the educational needs of medical students as they start their residency training. [9][10][11] To our knowledge, this is the first multispecialty, national survey of residency program directors' attitudes regarding educational handoffs of medical student performance information. Assessment of performance via milestone, competency and EPA instruments by medical schools is becoming more prevalent.
The results of this survey suggest that many residency program directors from the specialties sampled desire more information about incoming interns, and would use that information to enhance the training of their learners. Access to medical school milestone, competency, EPA, and capstone/boot camp-based, in some combination, could eventually comprise a standardized handoff between medical schools and residency, though concerns were raised about the validity and reliability of these metrics.
This study has several limitations. Despite multiple communications, the response rate was low at 20%, with the associated likelihood of participation bias. Not all participants responded to every item. The survey itself was generated by the authors and has not been otherwise validated. This survey did not assess the attitudes of medical school deans and clerkship directors or medical students. Only program directors from four specialties were surveyed. Not all medical schools collect data on student performance from each of the 4 types of instruments queried in the survey, potentially limiting its generalizability.
The introduction of educational handoffs would involve a paradigm shift in the aims of communication between medical schools and residency, from mostly summative (i.e. should a student be selected for matching into a residency) to formative as well (i.e. what are the learning needs for the intern going forward). Future research should assess what type of medical school performance data reliably predicts performance in residency, as well as which handoff formats minimize the administrative burden for medical schools. Different specialties likely will have different handoff needs (e.g. procedural skills of residents entering surgical programs), and the need for piloting and studying of potential handoff instruments across multiple specialties is anticipated. The attitudes of medical students and medical school faculty about educational handoffs should be studied.

Figure 1
How often are issues requiring active remediation identified within the first 6 months of internship that were not anticipated or known about? The frequency with which unanticipated issues requiring remediation are identified within the first 6 months of internship, stratified by percentage of respondents.   Table 3 Representative Comments from Survey Respondents I think information regarding an intern's technical skills, or lack thereof, would be most helpful for determining initial educational experiences and the best way to improve those skills. Milestones would be nice, but given the vagary of some of the milestones, I don't think it would necessarily change the educational rotations or duties. It would be most helpful (post-match) to know of any behavioral or mental health issues that the medical schools addressed while the resident was in medical school. This would really help when "red flags" start to appear early on in the first postgraduate year. We routinely make our own assessment of an intern's knowledge and skills and adjust educational activities to their needs. I would not rely on [handoff] information without verification, so I do not believe [it would be] helpful after match day. Medical school experiences seem to have become more disparate over time, increasing the importance of a good handoff. Milestones have been disappointing and highly variable. They do not represent an accurate view of performance. I don't feel the Dean's Letter is that helpful when I know it's only ever going to tell me positive things -it is not a true assessor of strengths and weaknesses. Wouldn't it be nice for an honest assessment for once?

Take Home Messages
Many of the program directors who responded to this survey felt that they do not receive adequate information from medical schools about the strengths, weaknesses, and learning needs of incoming residents, and desired access to additional medical school performance information, such as milestone, competency, EPA, and capstone/boot campbased data. With further study, some combination of these data types could represent the backbone of a standardized educational handoff as trainees transition from medical schools to residency programs.