Moving the Needle on our Med-Peds Residency Board Pass Rate

Introduction The national pass rates for the American Board of Internal Medicine (ABIM) and the American Board of Pediatrics (ABP) Certification Exams (CE) has declined for combined internal medicine-pediatric (Med-Peds) residents. The lower board pass rate is especially evident in smaller institutions like our own. Methods We instituted a board preparation assistance program where each resident is stratified into a color zone according to their in-training exam (ITE) scores. Results The ITE results of 67 residents from 12 classes were analyzed. 65.7% were graduates of medical schools within the United States (US) and 85.1% were allopathic graduates. Mean scores for USMLE step 1 and step 2 were 213 and 217 respectively. Mean ITE scores increased in each post-graduate year. Our first time ABIMCE board pass remained constant at 73% before the intervention and 75% after the intervention with 84% of residents ever passing the ABIMCE prior to the intervention and 81% after. Our first time ABPCE board pass rate is much improved from 46% to 75% and our overall pass rate improved from 57% to 93% (p=0.0121). On multivariate analyses there was no association between passing either the ABIMCE or ABPCE and any of our measured variables. Conclusions O'Keefe S, Turner M, Brinn N, Newton D MedEdPublish https://doi.org/10.15694/mep.2018.0000075.1 Page | 2 It is encouraging that our ITE scores and ABPCE and ABIMCE pass rates are improving although this seems to be due to factors other than our intervention. Other institutions that struggle with poor board pass rates may be interested in our approach.


Introduction
In 2002 it was reported that 88% of med-peds residents who graduated from the med-peds residency programs between 1994 and 1998 took the American Board of Internal Medicine certifying exam (ABIMCE) and 86% had taken the American Board of Pediatrics certifying exam (ABPCE). Of these residents the pass rate for the ABIMCE was 97% and the pass rate for the ABPCE was 96%. A reported total of 79% of all Med-Peds residents from the programs in the survey had passed both examinations and were dual certified 1 . Over the past decade or so there has been gradual decline in the national pass rates of both the ABIMCE and the ABPCE for all residents. Unfortunately, the national pass rate of combined medicine-pediatrics residents of both exams is lower than that of categorical residents nationwide. This disparity seems to have disproportionately affect smaller underserved institutions like our own [2][3] .
Prior to this study the combined medicine-pediatrics program at our institution had a worrisome low pass rate for both boards. Between the years of 2007 and 2011 the national first time pass rate of the ABIMCE was 88% 2 . Our categorical IM residency program has a pass rate similar to this national standard at 87% whereas for the same time period our combined Med-Peds residency had a pass rate of 69% for the ABIMCE. 2 The national pass rate for the ABPCE had been on average 77% between 2007 and 2011. 4 The overall first time pass rate of the ABPCE at our institution was 66% 5 . The 5 year pass rate for our combined medicine-pediatrics graduates seemed higher at 80% but this pass rate needs adjustment to reflect a very low "sit" rate. Unfortunately, only 60% of our eligible combined medicine-pediatrics residents actually took the ABPCE. Therefore, overall only 48% of all of our graduating board eligible combined Med-Peds residents actually pass the ABPCE.
In 2007 Freed et al reported that approximately 11% of physicians who claim to be pediatricians in the US have not ever achieved board certification with the American Board of Pediatrics (ABP) 6 . As more institutions and practices and ultimately patients and payers use board certification as a benchmark for qualification in a certain subspecialty our graduates would be at a significant disadvantage.
The faculty and program directors of our combined medicine pediatrics program felt very strongly that in order to consider ourselves an educationally sound and successful residency program we needed to attain a board pass rate of at least above the national average and preferably as close to 100% as possible. Therefore, we decided to create a new approach to the board preparation plan for our combined Med-Peds residents.
There are several factors that are associated with success in national board exams. Ranking within the top one third of one's medical student class, Alpha Omega Alpha status and mandatory research year in a surgical residency was associated with an increased likelihood of passing the American Board of surgery certification exam whereas a delay in taking the exam was associated with failure. 7-9 Self-directed reading of an electronic knowledge resource, being younger, and international medical school graduation were significantly associated with Internal Medicine In-training examination (ITE) performance in some studies but no evaluation of effect on success in the ABIMCE was studied [10][11] . Attendance at teaching conferences was found to be associated with IM-ITE performance in some studies but this was not corroborated by other studies [11][12]  United States Medical Licensing exam (USMLE) scores have been shown to be predictive of passing national board exams in multiple specialties including emergency medicine, surgery, orthopedics, internal medicine 7,8,[13][14][15][16] . USMLE scores <200 may be the most predictive of failure in national board exams and in some cases the score in the step 2 exam seemed more predictive than step 1 scores 7,8,12,16 .
Residency ITEs have also been found to be highly correlated with passing national board examination in multiple specialties including surgery, ophthalmology, orthopedics, internal medicine and pediatrics 8,14,[17][18][19][20][21] . One study found that those residents who scored below 35 percentile had an 83% probability of failing the ABIMCE 20 . Another study showed that a raw score of 49 or below predicted failure in the ABIMCE 21 . In pediatrics one study showed that residents scoring 410 or above in their 3 rd year pediatrics ITE have a 96% chance of passing the boards on first attempt whereas those scoring 250 or less during the PGY3 year have less than 50% chance of passing the boards on first attempt 19 .
Several residency programs have instituted formal programs in order to help increase their board pass rate. One surgical residency instituted weekly assigned reading followed by weekly examinations and another administered brief practice exams periodically throughout the year. Both interventions had positive results on their national board exam pass rates [22][23] . An internal medicine residency program instituted a 12-month intervention of pre and post tests for each clinical rotation. This was coupled with board review closed book testing. After exposure to this intervention the study group had a statistically significant increase in their percentile scores on the ITE exams. That residency program was participating in internal medicine educational innovation project (EIP) which gave them the flexibility to conduct that study during a 12-month long block of ambulatory months, inpatient and outpatient electives without night call 24 . A pediatrics residency program in Brooklyn a weekly email based board review course and found that total number of questions answered had a significant positive correlation with standard board scores 25 .
In the same residency program, a board review course was found to increase the standard board examination scores and slightly increase passing rate in at risk residents (i.e. residents with ITE scores <200) 26 .
Having reviewed what approaches have been tried at other institutions we created a board preparation assistance program specific for our med-peds residents.

Methods
We instituted a board preparation assistance program in 2012. Each senior resident was stratified into a color zone depending on their Internal Medicine ITE (IM-ITE) and Pediatrics ITE (Ped-ITE) scores.
For internal medicine each resident receives a score with a percentile that compares them to other combined medicine-pediatrics residents nationally. Therefore, the plans assigned to each resident for internal medicine were decided by their percentile scores. For pediatrics there are no percentile scores assigned and the research describing the predictive value of ITE scores is based on the raw scores. Therefore, the designation of color zone for pediatrics was according to their raw score in the Ped-ITE. For pediatrics the ABP changed the ITE reporting score in 2012 from a 0-800 standardized score to a 0-200 standardized score. In order to compare the performance of our current residents to residents in years past we converted all of the old standardized scores to the new standardized score according to a conversion supplied to us by the ABP. (PREP). The residents were encouraged to read the chapter relevant to the rotation from their review books and practice questions during their rotation. We organized some of the questions from their MKSAP and PREP resources into mini-tests that were related with every clinical rotation that our combined medicine-pediatrics residents rotate through each year. The residents were required to score 75% on the mini tests to satisfactorily complete the rotation. They were allowed to take the test up to 5 times if needed. It was an open-book exam. There was a time limit of 30 minutes on each test. The residents in yellow, orange and red zones were emailed once every 1-2 months with a reminder of their color zone and results on their rotation specific mini-tests.
Those residents at high risk of failure of their boards based on their ITE score were also heavily encouraged to follow a board remediation plan. This remediation plan involved developing and maintaining an active study plan that is supervised by the program directors and participating in all categorical board preparation events available. As part of their study plan we requested that the residents email us an update regarding their progress with their study plan every 1-2 months.

Analyses
We compared the ITE scores and final ABIMCE and ABPCE sit and pass rate in the years prior to implementation of our board assistance program to the ITE scores and ABIM and ABP sit and pass rate after the implementation of the program. ITE scores were compared using two-sample t-tests and pass rates were compared using Chi-square tests. Logistic regression models were used to identify possible factors that may affect the likelihood of passing boards. We also used logistic regression models to compare the likelihood of passing board before and after the intervention controlling for possible confounding factors such as USMLE step 1 score, USMLE step 2 score, US or international medical school graduation, and first year ITE score.  intervention and similarly our ever pass ABIMCE remained at best stagnant with 84% passing prior to the intervention and 81% after. Our first time ABPCE board pass rate is much improved from 46% to 75% and our overall pass rate improved from 57% to 93% and this improvement was statistically significant (p=0.0121). On univariate analysis first year ITE scores and USMLE-1 and USMLE-2 scores was associated with the first time pass rate of the ABIMCE and ABPCE and overall pass rate of the ABPCE. Only ITE-1 scores and USMLE-2 scores were associated with the overall pass rate of the ABIMCE. None of these associations remained when variables were controlled on multivariate analysis. There was no association between being a US graduate and board pass rates. Our intervention appeared to be associated with likelihood of passing the ABPCE based on univariate analysis but again this association disappeared once controlled for other variables on multivariate analyses.

Discussion
It is encouraging that our ITE scores and board pass rates have improved compared to before the intervention although this seems to be due to factors not related to our intervention such as prior academic success of the residents in the intervention group. Other interventions in other residency programs have also demonstrated improved board pass rates. [22][23][24][25][26] However to our knowledge this is the only board preparation assistance program described in the literature that specifically targets medicine-pediatrics residents. Our intervention is also somewhat unique in that we focused the energy of the intervention on "at risk" residents. Given limited resources in underserved institutions such as our own, and limited available faculty time we feel that focusing on "at risk" residents is the most sensible use of time and energy.
While other authors have shown a strong association between ITE scores and USMLE results and board pass rates the association found in our study did not remain when controlled for other variables. 7,8,14,[12][13][14][15][16][17][18][19][20][21] We did not demonstrate any relationship between board pass rate and US versus international graduate status or allopathic versus osteopathic status which may be reassuring to international and osteopathic graduates and those programs that match residents from these groups.
Limitations to our study included the following: Our sample size was small which can be improved by many more years of implementation and study. The ABP changed the standardized scoring system during our study. The ABP was able to provide us a chart so that the old standardized scores could be converted to the new scale. The old standardized score of zero converted to a wide range in the new standardized score so we used the upper quartile of this range to be consistent in the conversion. We chose the upper quartile because we felt this would be less likely to skew the data than choosing the median or upper limit of the range. For the analysis looking on effect of USMLE scores we had to exclude the osteopathic residents if they only had COMLEX scores.
In the pre-intervention group, we assumed that a resident did not pass if they are not listed as certified on the ABP or ABIM website. There may be some residents who chose not to sit a particular board because of their area of subspecialty. There were also some residents in the post intervention group that we know did not attempt one of the boards because they were pursuing subspecialty fellowship but in order to be consistent with our data recording we recorded these results as if the resident failed the board. This may have negatively skewed the data but we wanted to ensure consistency between the pre and post intervention groups. We also felt this approach was appropriate because our goal as a program is to have every resident sit and pass both the ABIMCE and ABPCE after graduation from our combined residency regardless of their career choice. We did exclude one recent graduate from the analyses because of board exam deferral for one year due to extenuating family reasons.
Over the course of the intervention we observed that ultimately those who were successful in their boards were typically those who applied themselves early on during residency to their studies. Our Board Preparation Assistance Program seemed to help those who would typically be self-motivated but perhaps needed a reminder of the urgency of the need to study. Many residents came to us with study plans and started emailing us monthly with their progress in those plans. Those who did not benefit were often residents who under-performed in other areas and could not seem to self-motivate. We were hoping that just drawing attention to their position in the red or yellow zone would be enough on its own but it seemed that if a resident procrastinated over their study it continued regardless.
Although there was considerable passive motivation applied by the program directors in the form of regular emails reminding them of their scores on the online tests and one on one meetings emphasizing the importance of studying, ultimately there was no hard stop for those who opted out of meaningful participation. After considering this obstacle in autumn of 2015 we revised our board preparation policy. We met with our categorical pediatric colleagues and tried to standardize the approach between the two programs and then we updated the policy to simplify it. The updated board policy can be seen in appendix 1 and includes such items as mandating an academic assessment for those who are demonstrating stagnation in their ITE scores, mandating a study plan be emailed to the program directors by every resident in the red zone, mandating a minimum number of PREP and MKSAP questions be completed on their online account by end of the academic year etc. Tied to this we now have decided that any resident who is in the red zone and not meaningfully engaged in the board preparation assistance program will receive a formal letter of notification. A sample of this letter is in Appendices. The letter notes that if they do not demonstrate meaningfully engagement in the program by their next semi-annual evaluation they will then be placed on academic remediation without extension [DN2] of length of residency. If they continue to fail to meaningfully participate in a board preparation assistance program, then academic remediation with extension of residency would be considered. Of note the decision to issue a letter of notification is based solely on the participation or meaningful engagement of the resident in the board preparation assistance program not on their ITE scores. If a resident has been actively participating in the preparation program but still remains in the red zone they would not receive a letter. Since introducing this policy we have noticed an increase in participation among those few residents who were having difficulty motivating to improve their study habits.
Future steps include continuing to collect and analyze the data annually especially after the implementation of the new policy in 2015. During the course of the study it also became apparent that personal/family circumstances seem to anecdotally have an influence on board success rates. Further areas of study could include analyses of the effect of major life events during residency such as pregnancy on the likelihood of passing boards.

Conclusion
We successfully implemented a board preparation assistance program specifically tailored for medicine-pediatrics residents. The ITE results and the likelihood of passing the boards are improving. As we continue to implement the more robust board preparation assistance program we hope to see continued improvement every year. Other institutions that struggle with poor board pass rates may be interested in our approach. Dale Newton was a clinical professor and a associate program director of the MedPeds residency program at East Carolina University at the time of the study. He is currently retired and enjoying time with his family.