Evaluation of a course to prepare international students for the United States Medical Licensing Examination step 2 clinical skills exam

Purpose United States (US) and Canadian citizens attending medical school abroad often desire to return to the US for residency, and therefore must pass US licensing exams. We describe a 2-day United States Medical Licensing Examination (USMLE) step 2 clinical skills (CS) preparation course for students in the Technion American Medical School program (Haifa, Israel) between 2012 and 2016. Methods Students completed pre- and post-course questionnaires. The paired t-test was used to measure students’ perceptions of knowledge, preparation, confidence, and competence in CS pre- and post-course. To test for differences by gender or country of birth, analysis of variance was used. We compared USMLE step 2 CS pass rates between the 5 years prior to the course and the 5 years during which the course was offered. Results Ninety students took the course between 2012 and 2016. Course evaluations began in 2013. Seventy-three students agreed to participate in the evaluation, and 64 completed the pre- and post-course surveys. Of the 64 students, 58% were US-born and 53% were male. Students reported statistically significant improvements in confidence and competence in all areas. No differences were found by gender or country of origin. The average pass rate for the 5 years prior to the course was 82%, and the average pass rate for the 5 years of the course was 89%. Conclusion A CS course delivered at an international medical school may help to close the gap between the pass rates of US and international medical graduates on a high-stakes licensing exam. More experience is needed to determine if this model is replicable.


Introduction
International medical graduates (IMGs) make up approximately 25% of current trainees in United States (US) residency programs. A substantial subset (about 38%) of IMGs are US and Canadian citi zens (US IMGs) who attend medical school abroad [1]. Most of these students desire to return to the US or Canada to complete resi dency training and practice medicine. US IMGs represent about 14% of all US residency applicants [1]. To enter graduate training in the US, IMGs must be certified by the Educational Commission for Foreign Medical Graduates. In 2015, US citizens made up 26% of all IMGs seeking certification in the US [2]. A critical step for certi fication is passing the US Medical Licensing Examinations (USM LEs). A 2006 report demonstrated that US medical graduates (US MGs) received higher scores on the USMLE steps 1 and 2 than IMGs and that US IMGs scored below US MGs and nonUS IMGs [3]. One exception was the USMLE clinical skills (CS) examination, on which US IMGs had a higher pass rate than nonUS IMGs [2]. Re gardless, the USMLE step 2 CS remains a highstakes endeavor for US IMGs, with an overall firsttime pass rate of 80%, compared to 96% for US MGs [4]. Taking the exam is associated with consider able cost and anxiety. Although the value of the USMLE step 2 CS exam to residency programs and learners has been questioned [5], its value to the public of ensuring that individuals seeking to train and practice in the US meet minimum standards of competence in com munication and physical examination skills, clinical reasoning, and spoken English proficiency may have more importance when ap plied to IMGs.
US medical schools have adjusted CS training in response to the USMLE step 2 CS examination, including increased use of standard ized patients (SPs) and simulations [6]. US and Canadian students matriculating at foreign medical schools may be at a disadvantage for the USMLE step 2 CS examination due to less exposure and train ing with SPs and objective structured clinical exam experiences; a decreased emphasis on structured patientcentered communication skills training, in particular around complex skills such as shared de cisionmaking, delivering bad news, and handling emotions; fewer opportunities to be directly observed in the clinical setting [3]; and less role modeling of patientcentered communication skills.
Multiple USMLE step 2 CS preparation courses exist in the US, but these courses are expensive, require students to travel from their existing learning environment, and may be more focused on 'teach ing to the exam' rather than developing competence in patientcen tered communication skills. We describe the development and evalu ation of a USMLE step 2 CS preparation course provided at the Tech nion Israel Institute of Technology School of Medicine for US and Canadian medical students in the Technion American Medical School (TEAMS) program between 2012 and 2016.

Course development
The goals of the course were to improve participants' (1) knowl edge of the format and content of the USMLE step 2 CS examina tion and comfort with SP encounters; (2) patientcentered commu nication skills; (3) ability to perform a focused history with a SP; and (4) ability to complete a USMLE step 2 CS examination posten counter note. The 2day course employed didactic and experiential learning methods to promote deliberate practice. Patientcentered communication instruction was modeled on the elements of com munication for which there is broad consensus [7]. Other course components were based on published information on the exam, evi dencebased medical education, and our own extensive experience teaching CS in the US [8,9]. On day 1, students received an over view of the exam format and strategies for patientcentered inter viewing, performing a focused physical exam, and completing the postencounter note. Students engaged in role play and received peer and faculty feedback using a structured observation guide. On day 2, students completed 3 videorecorded timed mock exam stations in which they performed a focused history and completed a posten counter note. Fourthyear TEAMS medical students served as SPs. Cases were developed by the course faculty based on likely exam sce narios. Following each encounter, students received structured feed back from the SP on their interpersonal and communication skills. Students then reviewed 2 of the 3 videos in two 35minute, oneon one sessions with course faculty (RBL, DC). During these sessions, students used a structured template to guide selfassessment, record feedback, and develop an individualized learning plan for indepen dent examination preparation.

Setting and course participants
The course participants were thirdyear students at TEAMS, an international 4year medical program located at the Technion Medi cal School in Haifa, Israel. The 4year curriculum is taught in Eng lish. The program accepts applicants who are US and Canadian citi zens or permanent residents who have spent at least 8 years out of the last 10 years residing in North America. TEAMS students have all completed a 4year college premedical curriculum in the US or Canada. The program offers an opportunity for students to pursue an MD (doctor of medicine) educational program with a curricu lum and course of study patterned after US medical schools. The application requirements are similar to those of North American medical schools and include MCAT (Medical College Admission Test) scores, academic transcripts, and letters of recommendation. The average class size is 30 students per year. The preclinical curricu lum consists of basic science courses and an introduction to CS. The final 2 years consist of clinical rotations in both Israel and the US. Participation in the course was voluntary, and students paid course tuition.

Course evaluation
We hypothesized that students' confidence and competence would increase and that the overall pass rate for the TEAMS program would improve. Students were asked to complete pre and postcourse on line surveys addressing their knowledge of the USMLE step 2 CS examination format, comfort with performing a timed SP encoun ter, and overall confidence and competence with the CS tested. We collected demographic information including age, gender, and coun try of birth. The survey included a Likert scale and openended re sponses. Emails inviting students to complete the surveys stated that responses would be confidential and asked students to select an 'opt out' response if they did not want their responses used for research. An administrative assistant with no other role in the course or TEAMS program had access to deidentified data. Students were contacted by email up to 3 times in order to increase the response rate. The course evaluation plan and surveys were reviewed through the Tech nion Internal Review Board and deemed exempt from further re view after informed consent was received from the participants.

Statistical analysis
All items measuring overall knowledge and confidence in taking and passing the USMLE step 2 CS examination, comfort with the SP encounter, and confidence and competence in specific CS were measured on a 5point Likert scale (0, poor; 1, below average; 2, av erage; 3, above average; 4, outstanding). A mean score was comput ed for each student at pre and postcourse time points. All single item and summary score measures fell within acceptable limits of skew and kurtosis.
To test whether students' selfreported measures of knowledge, preparation, confidence, and competence in CS increased between http://jeehp.org the pre and postcourse responses, we computed the paired ttest for each of the 22 singleitem measures. To test whether pre to post course changes in confidence and competence in skills varied accord ing to gender or country of birth, we used repeatedmeasures analy sis of variance (ANOVA) with summed confidence and competence scores, respectively, as the dependent variable. Time (pre versus post) by gender and country of birth interaction terms tested hypotheses related to demographic differences. For country of birth, a categori cal variable was created to compare the total scores for students born in the US, Canada, and other countries. We compared the overall pass rates provided by the TEAMS program with data publicly avail able on the USMLE website [4].
We began evaluation of the course in 2013, using our 2012 course as a pilot. We completed an analysis of survey items for course par ticipants during the years 2013-2016. We did include the pass rate from 2012. All analyses were performed in IBM SPSS ver. 24.0 (IBM Corp., Armonk, NY, USA). Values are presented as number (%) or mean ± standard deviation (range).

Results
Ninety students have participated since 2012, with 76 participat ing during the evaluation period between 2013 and 2016. There were no significant demographic differences between participants and nonparticipants. The raw data are available in Supplement 1. Of the 76 participants, 73 gave their permission to use their course eval uation for research. Nine of the consenting students did not com plete the postcourse survey. Table 1 presents the demographic char acteristics of the remaining sample of 64 students who completed pre and postcourse surveys. A slight majority were male, and most were USborn.
Among the 64 respondents who completed pre and postcourse surveys, most items were missing a negligible number of responses (0 to 1). All available data for each pair of prepost measures were in cluded. The results of paired ttests comparing pre to postcourse changes on USMLE selfratings are compiled for each item in Table 2.
Students reported significant increases in confidence and compe tence scores across all clinical skill items and averaged scores (Table  3). Repeatedmeasures ANOVA indicated that the increase in mean CS confidence scores did not differ according to gender (interaction of gender× time F(1, 62)= 0.042, P= 0.84), or country of birth (in teraction of country of birth × time F(2,59) = 1.05, P = 0.36). The mean competence scores did not vary by gender (interaction of gen der× time F(1,57)= 1.08, P= 0.30) or country of birth (interaction of country of birth× time F(2,54)= 0.26, P= 0.77).
The average TEAMS USMLE step 2 CS pass rate from 2007 to 2011 was 82%, while the average pass rate of the 2012-2016 course participants was 89%. The pass rate for students in the course was 95% and 92% in 2015 and 2016, respectively. Fig. 1  The ratings of teaching methods were compiled, and mean scores for the overall ratings of teaching quality and specific teaching meth ods are presented in Fig. 2. Interactive, experiential teaching meth ods received the highest ratings.

Discussion
This is the first study to describe a successful USMLE step 2 CS preparation course conducted within the local learning environment of an international medical school. This course significantly improved students' confidence and perceived competence in their ability to pass and perform specific CS related to the exam. Over 5 years, there was a trend towards overall improved pass rates for TEAMS students; the pass rate approached that of US MGs and was higher than IMGs overall. There is currently a dearth of published descriptions or evi dence regarding commercial and noncommercial USMLE step 2 CS preparation courses. A limited number of published studies eval uating the impact of commercial preparation courses have focused on the USMLE step 1 exam; these studies are methodologically lim ited, and have demonstrated little to no impact on test scores [10].
IMGs continue to make up a significant proportion of practicing US physicians and should have access to CS training that not only prepares them for the USMLE exams, but also for the care of pati ents in the US. Additionally, with the availability of the Accredita tion Council for Graduate Medical Education's international accred itation program, an opportunity has emerged for strengthening CS training across the international medical education continuum [11]. Overal teaching quality An individual's performance on the data interpretation and commu nication and interpersonal skills sections of the USMLE step 2 CS examination is positively correlated with ratings of history taking and physical examination during internship [9]. Ideally, CS training in preparation for the USMLE step 2 examination would also improve longterm practice and meaningful health outcomes. Another important outcome of this program that may have helped to improve pass rates is improved student confidence and comfort, which may decrease test anxiety. Testtaking anxiety is modestly in versely correlated with USMLE step 1 performance and can be re duced [12]. We incorporated methods shown to reduce stress and potentially improve performance, such as mental rehearsal [13]. As students in the TEAMS program do not have the opportunity to participate in many formative or summative SP encounters, the abil ity to increase familiarity with that format may increase their com fort and reduce their cognitive load during the exam. In addition, using peers as SPs may reduce stress and improve learning [14].
Conducting this course at the local institution with input from students and program directors allowed us to better understand stu dents' local educational environment and tailor our teaching to stu dents' needs. A positive, supportive environment that invites learners to share their strengths and areas for improvement promotes learn ing. Similarly, a learning community in which students are intention ally engaged in learning from each other activates the social aspects of learning and encourages students to challenge themselves and take risks with their learning. TEAMS classes are typically small and stu dents spend a significant amount of time together. We leveraged this asset and emphasized a team learning approach while promoting a supportive learning environment. This study has limitations. First, this was a singleinstitution study that has not been replicated elsewhere. Second, while this course cost less to participate in than commercially available courses, the cost was still significant, and this factor may limit the generalizability of our findings. However, the course allowed students to remain within their existing learning environment where they may have benefited from collaborative learning with peers, as discussed above. Third, this course primarily focused on teaching communication skills, but the physical exam and the patient note tended to be lowscoring com ponents for many examinees [15]. Lastly, while we were able to dem onstrate an overall increase in the pass rate, we cannot prove causali ty. There may have been other factors, both internal and external, that impacted pass rates. In 2013, TEAMS was undergoing a curric ular change, which might explain the low pass rate for that year. Sim ilarly, the USMLE step 2 CS grading scale has changed over time, potentially confounding our comparison of pass rates.
This study is the first to describe a USMLE step 2 CS preparation course specifically designed for US IMGs that was delivered within the students' local learning environment. The use of experiential learn ing and instruction focusing on patientcentered communication skills improved students' confidence and perceived competence, and a trend toward improved pass rates on the USMLE step 2 CS exam ination was observed. This type of course may help to close the gap between US MG and IMG pass rates on a highstakes licensing exam and potentially promote the longterm retention of CS.

Conflicts of interest
Drs. Cayea and Levine receive an honorarium for developing and teaching the course.

Funding
No funding support was received for the evaluation of this course.