Resident Led Teaching Sessions Improve Medical Student NBME Shelf Scores and Clerkship Satisfaction

Purpose: To evaluate the beneﬁts of a resident directed review session on medical student clerkship exam scores and satisfaction. Background: The role of residents as mentors and teachers has been well documented. It has not been examined whether a resident directed review session results in improved NBME exam scores. Methods: Following the introduction of a resident led review session, medical student NBME exam scores of those who did and did not participate in the review were compared. In addition, exam scores for the academic years following the introduction of the review were compared to the year prior to the intervention to look for overall changes. Medical student clerkship evaluations were also investigated for changes in reported satisfaction before and after implementation of the review session. Results: There was no change in overall exam scores (p=0.75) between academic years prior to and after the implementation of the review sessions, however, students who participated in the review session scored higher on their NBME shelf exam compared to those who did not


Background
Medical education has entered a period of flux as a growing variety of teaching modalities have been introduced to medical schools throughout the country. The majority of these changes have occurred in the preclinical setting with the most obvious example being the widespread adoption of small group and problem based learning as well as a strong emphasis on self-directed learning (Edmunds & Brown, 2010;Kilroy, 2004). While these preclinical changes have been met with primarily positive reviews and outcomes, medical schools have struggled to adapt the clinical clerkship curriculum to the changing healthcare environment (Warrier, Schiller, Frei, Haftel, & Christner, 2013;Yew, Chng, & Schmidt, 2011).
Over the past several years, there has been a growing discussion within the medical education field regarding the utility of clinical clerkships. Residency programs have struggled with the discrepancy in knowledge and clinical skills that incoming residents have accumulated during their undergraduate medical education (Langdale et al., 2003). Some have proposed that the clerkship curriculum is so broken that it should be reduced to a single year (Abramson & Jacob, 2013).
Despite these shortcomings, clinical rotations provide an invaluable experience to medical students. The selection of a specialty is a multifactorial process with clinical exposure to the specialty playing one of the largest roles (Cochran, Paukert, & Neumayer, 2003). Residents are the primary point of interaction with medical students on clinical rotations and spend a significantly larger amount of time interacting with students then attending physicians (Busari, Prince, Scherpbier, Van Der Vleuten, & Essed, 2002). As a result, residents have the opportunity to take on an important mentoring and educational role during medical clerkships (De, Henke, Ailawadi, Dimick, & Colletti, 2004;Nguyen & Divino, 2007). Traditionally, this has entailed residents instructing medical students on presentations, documentation and basic technical skills (De et al., 2004). Certain specialties have experimented with expanding this hidden curriculum into a more formal setting outside of the wards, with residents leading formalized medical student education sessions. The introduction of resident led formal small group problem based learning sessions has been demonstrated to improve medical student clerkship NBME shelf score (McKean & Palmer, 2015).
To our knowledge, no one has examined the effect of a resident led review session on NBME shelf scores. In this paper we examine the effects of a voluntary review session directed by two junior residents on Obstetrics and Gynecology NBME subject exam (shelf) scores, and overall medical student satisfaction with the clerkship.

Methods
Starting in the academic year of 2014-2015, a voluntary resident led review session was instituted at The Pennsylvania State University College of Medicine, a tertiary care academic medical center. All medical students completing their required third-year Obstetrics and Gynecology clerkship at this hospital and affiliated sites were invited to participate. The review was independently established outside of the formal curriculum by two junior level residents who volunteered their time, with the intention of addressing any knowledge deficiencies the clerkship students perceived over the rotation. Two residents led these sessions with the students without attending physician involvement in an effort to promote a more open conversation and facilitate learning and review. Structurally, the review sessions occurred 2-7 days prior to the NBME shelf exam as scheduling allowed. In order to facilitate student participation all sessions were held in a lecture room within the hospital after the conclusion of all medical student Wagner S, Waldman I, Maines J, D'Agata C, Kunselman A, Haidet P, MacNeill C MedEdPublish https://doi.org/10.15694/mep.2016.000077 Page | 3 clinical obligations for the day. The review itself was broken down into two segments, each of which lasted approximately one hour. In the first segment, medical students were invited to ask questions about topics or concepts that they felt were difficult to understand. For the second segment, images related to high yield topics were used as prompts for a brief review of the subject matter.
At the conclusion of the '14-15 academic year, all students who took the Obstetrics and Gynecology shelf over the previous 12 months received an online survey to determine who participated in the review session. The responses were then forward to the clerkship coordinator who deidentified student scores and created a spreadsheet comparing the scores between those who did and did not participate in the resident led review session. Shelf scores were then compared between the groups. The '15-16 academic year was incomplete at the time of this submission with 75% of medical students having rotated through their Obstetrics and Gynecology clerkship. Only those students who had rotated through the clerkship were included in the study. Beginning in the '15-16 academic year attendance was taken at the beginning of every review session. By identifying all students who participated over academic year '15-16 a more complete comparison between participants and nonparticipants was possible.
The primary outcome of this study was to examine if students who participated in a resident directed review session achieved higher shelf exam scores. Exam scores were compared between those who did and did not participate in the available resident led review session. In addition, scores for the entire clerkship student population for academic year '14-15 and those scores available for academic year '15-16, were compared to scores from the year prior to the introduction of the review session (academic year '13-14).
It is standard practice at The Pennsylvania State University College of Medicine to collect surveys at the conclusion of every clerkship to assess student satisfaction with the rotation. A secondary outcome of this research was to ascertain if a resident directed review resulted in improved medical student satisfaction. These surveys were compared between academic year '13-14 and '14-15.
Statistical analysis was performed using Microsoft Excel. The student t-test and difference in means calculations were utilized where appropriate. This study received IRB exemption from The Pennsylvania State University Hershey Medical Center office of human protections.

Results
The implementation of a resident directed review session did not result in a statistically significant improvement in overall Obstetrics and Gynecology NBME shelf exam scores. Between academic year '13-14 (pre-review session implementation) and '14-15 (post-review session implementation), the mean exam score increased by 0.5% from 77.1 to 77.6(p=0.74). In academic year '15-16, the mean exam score difference remained stable at 0.5% with a mean shelf score of 77.1 (p=0.75).
In academic year '14-15, comparison of scores between those who participated in the review and those who did not was based on a voluntary survey that was distributed at the conclusion of the academic year. 43 students responded to two survey requests, resulting in a 32% response rate. 17 of the 43 had participated in the review session (40% review session participation rate). Of those who responded, there was no difference in exam scores by participation in the review session (p=0.85). For the following academic year '15-16 attendance was recorded at all review sessions. The scores of 87 individuals were available for evaluation, 51 of whom had participated in the review session (58% review session participation rate). Again, the scores were deidentified and compared based on attendance at the review sessions. In the '15-16 academic year, participants in the review session were more likely Wagner S, Waldman I, Maines J, D'Agata C, Kunselman A, Haidet P, MacNeill C MedEdPublish https://doi.org/10.15694/mep.2016.000077 Page | 4 to have an improved shelf exam score with an average improvement of 3.4% (p=0.03) (figure 1).
A resident review session was correlated with a statistically significant improvement in medical student evaluations of the Obstetrics and Gynecology clerkship rotation. Medical students rated the overall rotation more highly and residents as more effective teachers following the introduction of the review session (Table 1).

Discussion
This study examined the effect of a resident directed exam review session on NBME subject exam (shelf) scores for the Obstetrics and Gynecology clerkship. The timing of the review was designed to maximize student benefit prior to their exam without interfering with either student or resident clinical duties.
Due to the limited number of survey responses from the '14-15 academic year, it was anticipated that there would not be a statistically significant difference in shelf scores between students who did or did not take advantage of the review session. The low survey response rate is likely due to its voluntary nature. Students who failed to participate in the review would also constitute a population less inclined to participate in this survey. For the '15-16 academic year, students who participated in the review session scored significantly higher than those who did not participate. This suggests that the review session does provide benefit to the medical students who elect to participate. The nature of the review, with its focus on high-yield topics, likely contributed to this significant increase in scores. The ability of medical students to further explore topics that they perceived as difficult to understand with the basic information provided during their clerkship would be another explanation for the improvement seen. An additional explanation would be the professional growth of the residents directing the session over the two years examined. These contributing factors would all support the hypothesis that as the review session progresses, future students will continue to receive an increasing benefit.
As the review was not mandatory, only an estimated 40% of medical students participated in academic year '14-15 and 58% in academic year '15-16. The initially low participation rate is likely a result of medical students being reluctant to devote time to a review session of unknown quality. However, the increase in participation between the two years points to a growing perception among the medical students at this institution that this review session is a valuable educational experience. The fact that the overall difference in scores when compared between academic years '13-14, '14-15, and '15-16 was not statistically significant is based on the fact that a large percentage of students did not participate in the review sessions. Our finding of a significant improvement in test scores during academic year '15-16 amongst those who attended the review sessions may provide a motive for students to participate in these review sessions in the future in order to better their own scores.
It was anticipated that the creation of a review session would enhance medical student perceptions of residents as teachers. This was based on previously documented desires of medical students for clearly delineated teaching from residents (Dyrbye & Shanafelt, 2011). As expected, medical student perceptions of residents improved significantly once the review sessions began. These improvements carried over to the clerkship as a whole, and suggests this would be an effective modality for clerkship directors to improve the clerkship experience.
Limitations to this study include the incomplete participation data for the '14-15 academic year as attendance was not taken during review sessions as it was done during the '15-16 academic year. We also did not control for changes in the preclinical curriculum, general intellect or interest in Obstetrics and Gynecology as a specialty. The differences in individual faculty and residents who interacted with medical students could also not be accounted for. Strengths of this study include its multiyear evaluation of scores and relatively large sample size. This is also the first Wagner S, Waldman I, Maines J, D'Agata C, Kunselman A, Haidet P, MacNeill C MedEdPublish https://doi.org/10.15694/mep.2016.000077 Page | 5 study to examine whether a resident directed review influences NBME shelf scores.
Further research regarding the incorporation of residents into a formal clerkship curriculum is needed. This study provides evidence that residents can serve an important role as educators outside the clinical setting, and that NBME shelf scores, and medical student satisfaction, can be improved by resident directed learning in a classroom setting.

Take Home Messages
Residents play a vital role in medical student education and strongly influence quantitative and qualitative 1.
outcomes on clerkship rotations. While often seen as leaders in the "hidden" curriculum on the wards, resident involvement in more formal 2.
settings benefit medical student learning.

Notes On Contributors
Stephen