A Modified Team Based Learning Approach to Enhance Resident Education in Outpatient Internal Medicine

Objective: To implement a modified team-based learning (TBL) approach in ambulatory care educationfor a large Internal Medicine residency program with limited resourcesand to evaluate its effectiveness. Methods: 91 medicine residents were exposed to a modified TBL curriculum, composed of high-yield topics for ambulatory medicine. All residents participated in 10 TBL sessions per academic year. One faculty and one chief resident developed the pre-session reading requirements, readiness assurance tests, and clinical case scenarios for each session and facilitated the TBL exercises. At the end of the academic year, residents were asked to complete an anonymous survey online. Results: 72.5% of residents completed the survey. 96% reported being actively engaged in the sessions and contributing substantially to the discussions. A majority expressed preference for future TBL sessions. Educational effectiveness of TBL was at least as good or better compared to traditional lectures. Both faculty and residents expressed very high satisfaction with our TBL format. Conclusion: We successfully implemented a one year modified TBL curriculum to teach high yield topics in outpatient internal medicine in a large, multi-site residency program. TBL resulted in high resident engagement in the classroom and high satisfaction with the format by both residents and faculty.


Introduction
Medical training aims to instill the principles of self-directed learning, critical thinking, and teamwork in future physicians throughout their undergraduate and graduate medical education. Traditional lectures are passive teaching methods which impart a great deal of information in a short span of time but do not necessarily teach the application of the information to real-life patient scenarios.
Team based learning (TBL) is an active learning pedagogy stemming from the principles of adult learning and constructivist theories (Hrynchak and Batty 2012). It enables learners to apply learned content through intra-and inter-group class discussions that are moderated by faculty. Initially developed by and implemented in business schools, TBL has gained growing popularity in healthcare education over the past decade and is now employed by many medical schools in the USA and abroad. It has been shown that TBL improves students' engagement in their learning and enhances critical thinking, content retention, and teamwork (Thomas and Bowen 2011), while academic outcomes have been as good or better compared to traditional lectures (Thomas and Bowen 2011;Fatmi et al. 2013).
TBL use in residency education remains sparse, with most published data in the fields of pathology and psychiatry. The first study regarding TBL implementation in an internal medicine residency curriculum was published recently (Balwan et al. 2015). While principles of TBL are well described, the gold standard of its implementation is not clear. There is considerable variation in implementation, as applied to the local realities of each institution and learner group (i.e. medical students versus residents, specialty topic, learner group size, occasional sessions versus full TBL curriculum, program culture, etc.), and these variations may impact upon the measured outcomes and successful replication in other programs (Burgess et al. 2014). For this reason, active scholarship detailing description of TBL implementation in residencies, according to the guidelines published by Haidet's group (2012), is necessary to grow a body of evidence in support of a gold standard.
We describe our experience with the use of a structured TBL curriculum, which was used to teach outpatient internal medicine to a large university residency program.

Setting and Participants
All categorical internal medicine residents (PGY-1 to PGY-3, N= 91) enrolled in the Rutgers New Jersey Medical School (NJMS) Internal Medicine Program as of July 2015 were included in our study. Our program uses an 8+2 block schedule (Tentler et al. 2016). In this model, each 8-week period (inpatient/outpatient rotations, electives, vacation, etc.) is followed by a 2 week block dedicated to continuity clinic. As such, residents are divided in 5 cohorts, each with 18-19 residents across all years of training, that always rotate together through outpatient ambulatory blocks. This allows for 5-6 ambulatory blocks per resident per year.
During each ambulatory block, residents are assigned mandatory online learning modules from Johns Hopkins Online Ambulatory Care Curriculum, and their completion is tracked. In addition, there is protected time on Friday mornings for all residents on the ambulatory block to attend an Academic Half Day (AHD) dedicated to education in ambulatory internal medicine. The AHD is a 2.5-hour, weekly, mandatory session, and is run by the Associate Program Director and the Ambulatory Chief Resident. Prior to July 2015, the AHD consisted of resident-driven PowerPoint presentations of pertinent ambulatory topics, Evidence-Based Medicine (EBM) sessions, journal clubs, Feurdean M, Matassa D, Kang M, Matthews G, Kothari N MedEdPublish https://doi.org/10.15694/mep.2017.000176 Page | 3 and various workshops.
Focus groups were conducted at the end of the 2014 -2015 academic year and resulted in a list of high-yield topics, with which residents reported having a difficult time applying their knowledge in the clinic. It was decided that the most effective method to deliver this information could be TBL, as residents reported strong dissatisfaction with the then-current lecture program. The Associate Program Director subsequently reviewed literature pertaining to effective TBL implementation in healthcare education and attended a faculty development session dedicated to active learning pedagogies (e.g. "flipped classroom" methodology) in medical education.

Program Description
In July 2015, we introduced TBL within AHD. This resulted in having 2 TBL sessions per ambulatory block, for a total of 10 TBL sessions per resident per academic year. We replaced the resident-driven PowerPoint presentations with TBL activities, using high-yield topics derived from the aforementioned focus groups. After the 1.5-hour TBL session, we filled the remaining hour with various activities, such as: EBM sessions, journal clubs, and various workshops.
The two TBL sessions for each two-week ambulatory block were generally organized thematically. Topics were chosen to supplement the online Johns Hopkins modules, which residents were assigned to complete during each block. Each set of TBL topics was repeated for five consecutive blocks, allowing all residents to receive the same modules over a ten-week span. The residents were informed of the TBL topics and the required pre-class readings at the beginning of each ambulatory block, resulting in at least a one-week lead time before the first TBL session.
At the beginning of the TBL session, all residents took an individual readiness assurance test (IRAT), composed of 10-15 short questions to be completed over 10-15 minutes. Questions included both multiple-choice and open-ended varieties. There was no formal grading of IRAT, and there was no subsequent group readiness assurance test (GRAT), all of which was entirely contrary to the norm of TBL application. Correct answers, instead, were discussed immediately within the larger group, with moderation from the faculty who emphasized important teaching points. This modified IRAT process provided each resident the opportunity to assess his/her individual knowledge and receive immediate feedback from the group and faculty within a friendly learning environment. The IRAT and immediate feedback encompassed approximately 25-30 minutes, in total.
For the TBL application exercise, residents were then divided into small groups of four or five, with at least one member of each PGY level represented. The group members were chosen ad-hoc by the Chief Resident, such that member pairings varied from one session to the next. Complex, case-based, multiple-choice questions were posed via PowerPoint, and ample time was given for each group to discuss internally. Team leaders were encouraged to involve interns in the discussion. Questions were purposely designed to have more than one possible answer, with learning points that emphasized guidelines of practice and their application to our urban healthcare environment. All teams worked on the same problem simultaneously, and when the time for internal debate expired, the team leader raised the answer placard(s) and each team's answers were written on a whiteboard. Each answer choice and its reasoning were debated out loud and points were given for correct answers and for well-formed arguments. The faculty moderated the discussion and exposed the deeper learning value of the case. The tone of the process was intentionally designed to be fun and competitive to engender engagement and active participation, but the focus was maintained on knowledge application. Winners were given token prizes based on the topic at hand (e.g. Pre-Operative Evaluation TBL winners received EKG pins for their white coats). Each TBL session, including both the IRAT and application exercise, was designed to run an average of 90 minutes. The pre-assigned reading materials, IRAT, clinical vignettes, and associated learning points were created entirely by the Associate Program Director and Ambulatory Chief Resident, who also served as the moderators for the sessions. Compared to our previous ambulatory care curriculum, we did not change the time allotted to AHD or the core facilitators. However, the time spent by faculty to prepare the TBL was far greater, estimated at 8-12 hours per topic (16-24 hours for 2 TBL sessions, every 10 weeks). Evidently, no additional time had to be spent on grading IRAT/GRAT.

Program Evaluation
At the end of the academic year, all participating residents were asked to complete an online anonymous survey (www.surveymonkey.com). A total of 8 out of 32 items, using a 5 point Likert scale, were adapted from an existing and validated classroom engagement survey (Baylor Medical College, 2001). The other questions delved into the specifics of our TBL sessions (e.g. atmosphere of soft competition, quiz feedback) and perceived effectiveness with regard to specific topics. Many of the questions asked the residents to compare their TBL experience against a standard lecture in that topic. The free text section of the questionnaire contained three questions: "What did you like about TBL?", "What did you NOT like about TBL?" and "Any other comments about TBL?" The survey results were collected separately per year of training. A separate survey asked about residents' prior exposure to TBL before its implementation in our curriculum.
The Institutional Review Board at Rutgers New Jersey Medical School declared this study exempt. (Pro20160000359)

Participants
Survey results were analyzed separately per year of training. 66 out of 91 residents completed the survey (72.52%). Of the 66 responders, 24 were PGY-1, 20 were PGY-2 and 22 were PGY-3. Not everyone answered all questions. Free text sections were analyzed thematically.

Quantitative Responses
56 of 91 (61.5%) residents answered the survey regarding their prior exposure to TBL format. Overall, 31 of 56 (55.3%) respondents reported exposure to TBL prior to its implementation in our ambulatory curricu1um (71% of PGY-I, 36.8% of PGY-II, and 56.25% of PGY-III).
Across all PGY levels, 63 % (54.5-70%) agreed or strongly agreed that they would like to have more class sessions in TBL format, in addition to what was provided during AHD. 7.5% agreed or strongly agreed that they did not enjoy TBL, while 87.8% (86-90%) disagreed or strongly disagreed with this statement. 20.3 % agreed or strongly agreed that they learned better from lectures than TBL, while 50% disagreed or strongly disagreed with this statement, and 29.6% were neutral (neither agreed nor disagreed). Further comparison of educational effectiveness in teaching specific topics can be found in the following table. Likert scale: 1=strongly disagree (SA), 2: disagree (D), 3: neither agree nor disagree (NAND), 4: agree (A), 5: strongly agree (SA) Overall satisfaction with our TBL format was very high. The majority of residents agreed or strongly agreed that the competitive nature of TBL promoted learning engagement (86.3%), that pre-TBL reading was helpful (86.3%), that the pre-TBL quizzes (i.e. IRAT) followed by immediate feedback were educational (80%), and that intra-and intergroup discussions were useful for learning (84.6% and 86.3%, respectively) and offered supplemental information to the required reading (90.9%).
Resident's self-assessed knowledge at the end of the academic year varied depending on the topic and the PGY level. Not surprisingly, senior residents were more likely to report being better prepared compared to interns in the same academic year, likely because knowledge was impacted not only by TBL but also by other didactic and experiential activities during training.

Thematic Analysis of Resident Comments
Strengths and weaknesses of the TBL format and content were identified by residents in the free comment section of the survey. The following themes were identified in their comments: self-assessment of knowledge (i.e. how much they retained from pre-TBL assigned reading), quality and quantity of pre-TBL reading material, resident engagement in TBL, perception of learning effectiveness (i.e. how much residents learned from TBL compared to lecture format), team competition (i.e. competition as a motivator to complete assigned reading before TBL), and choice of TBL topics (i.e. relevance of chosen TBL topics for daily practice). Across all PGY levels, most residents enjoyed the interactive format and sustained engagement in learning, the practical application of knowledge to common clinical scenarios, and the lighthearted competition. Several commented that TBL sessions were "educational" and "forced [residents] to read ahead of time," while the IRAT allowed for critical self-assessment of one's knowledge. It was also repeatedly noted that "the interactive format reinforced concepts and helped [them] to retain information." Several issues were identified: fragmentation of information compared to an organized lecture, inconsistent participation in discussion (with some residents dominating the discussion at times), and environment described as "too competitive at times." Suggestions for improvement included: additional topics, careful calibration of required reading material and quizzes in terms of length and content, and dissemination of correct quiz answers in written format for future reference. Several residents recommended expanding the TBL format to other didactics in the program. Some suggested a combination of TBL and lecture format (i.e. following intra-and inter-group discussions of clinical vignettes, there could be a short literature review by faculty in support of correct answers).

Themes identified in resident comments about TBL and associated examples
Self-assessment of knowledge "Allows one to truly see which areas you retained and understood from the reading and apply them together in a group setting" ( PGY-II) "Reading is just passive learning as well as noon lectures. But this was reading and then testing what I knew with critical thinking so if there were gap or things that I thought I knew I was able evaluate my true knowledge on a topic." ( PGY-III) Quality and quantity of pre-TBL assigned reading "Sometimes the reading that we are required to do before is not as inclusive or helpful -some of the articles contain too little information and some too much. Overall the articles have been good, it's just the few here and there."(PGY-I) "The amount of reading on certain weeks can be dense or overwhelming." (PGY-I) "Amount of Pre-TBL reading material needs to be cut down." (PGY-II) "TBL reading […] each week to prepare for the quiz was slightly overwhelming." (PGY-III)

Team Competition
"The teaching is great, important information is highlighted and having a fun lighthearted competition between teams helps solidify the information in my memory." (PGY-I) "I like the fact that we have to prepare to compete"(PGY-I) "I liked the playful nature of TBL, the "competition… the discussion that arises about management of particular conditions" (PGY-I) "TBL seemed overly competitive at times." (PGY-I) "Sometimes the environment got too competitive and some members were too vocal, not letting anyone participate."(PGY-I)

Choice of TBL topics
"I liked the fact that we dealt with common illnesses and clinical scenarios that we will see in practice daily. " (PGY-I) "Allowed us to read on common topics" (PGY-I) "All the topics this year were very relevant and high yield for application clinically and in terms of board relevance." (PGY-II)

Discussion
We successfully implemented a TBL methodology in our large, university-based categorical Internal Medicine residency program. We adapted the classic TBL framework to our unique program needs (i.e. 8+2 block schedule, TBL integration within academic half days, 18-19 learners per group, high yield topics as suggested by residents), and implemented it successfully in a longitudinal one-year curriculum in ambulatory internal medicine.
Our TBL exercises deviated from established guidelines in two significant ways. First, the IRAT was self-scored and was not collected by the faculty; it was not used as a tool for resident evaluation. Secondly, the GRAT was completely eliminated due to time constraints and was instead replaced by large group discussion of correct answers with faculty moderation. Despite our modified approach, our model follows closely the conceptual model of TBL as Page | 8 it links to learning outcomes, including the essential three-step process: pre-class preparation, assurance of readiness through IRAT, and application of content to common clinical scenarios encountered in the clinic (Haidet et al. 2012;Parmelee et al. 2012). We also worked to include the 7 core TBL design elements: team formation, readiness assurance, immediate feedback, in-class problem solving, the 4 S's of application (significant problem, same problem, specific choice, and simultaneous report), incentive structure, and peer review (Haidet et al. 2012;Parmelee et al. 2012). Our survey results point to the fact that residents welcomed the lack of formal grading of IRAT, and they acknowledged it as a useful tool for self-assessment of knowledge retention and as a motivator for pre-session learning.
We described our experience in detail, following the published guidelines for reporting on TBL (Haidet et al. 2012), to allow readers to understand the minutiae of our workflow and to successfully replicate and draw meaningful conclusions about our particular experience compared to others published in the literature.
Our study confirmed previously published outcomes of enhanced engagement and satisfaction with learning activities, increased self-directed learning, and high acceptance rate of TBL compared to traditional lectures (Sisk 2011;Rezaee and Mosalanejad 2015). Prior resident exposure to TBL was highly variable and did not seem to affect the outcomes and acceptance rate of TBL.
One attending faculty and one chief resident implemented the TBL curriculum. A single teaching faculty, who is a content expert, is sufficient to moderate the group, and previous experience with TBL is not mandatory for a successful outcome. The chief resident can effectively replace faculty during didactics, but faculty supervision is critical for development of session goals and objectives, selection of reading materials, and writing of the IRAT and case-based vignettes. Faculty time used for session preparation increased by an estimated 8-12 hours per topic (16-24 hours for 2 TBL sessions, every 10 weeks); no time was spent on grading quizzes, and the classroom time for academic half day remained the same. While the level of engagement and preparation time involved in TBL could be prohibitive compared to the delivery of traditional lectures, both our faculty moderator and the Ambulatory Chief Resident reported that TBL sessions were interactive, engaging, and fun. They reported increased satisfaction with teaching compared to delivery of traditional lectures. They confirmed that residents had remained engaged throughout the sessions, looked forward to the 'soft' competitive nature of the sessions, and were motivated to prepare well so that they could contribute to their teams' success.
Because time commitment from the faculty is an obvious barrier to TBL implementation, we have since implemented a few strategies to reduce the burden on any one faculty member. First, in our 8+2 block schedule, 5 different groups of residents rotate through the clinic every 2 weeks. Since our TBL sessions are weekly, we only have to create 2 TBL sessions every 10 weeks. Secondly, the work is shared amongst a TBL-experienced faculty member and a dedicated chief resident. Lastly, we plan to create a 3-year curriculum so that TBL modules can be reused and updated every 3 years.
Our study identified several areas for further improvement in our methodology: appropriate team balance and faculty moderation to give all residents a chance to voice their thoughts and opinions; selection of topics guided by their high relevance to practice as well as appropriateness for a TBL format; selection of reading materials that balance content with length, appropriate to allotted study time; and creation of a short literature review to be presented in support of correct answers.
One weakness of our study is that we did not report on academic outcomes with regard to short-and long-term knowledge retention. Numerous studies have previously established that test scores on GRATs are significantly higher than IRATs, supporting the fact that immediate retention of knowledge is improved by focused group discussions. Since we did not have a GRAT, we cannot report the same types of outcomes. Instead, our quiz served a double role: as an assessment of readiness and simultaneously as a motivator for class engagement and active participation. As shown by the survey, satisfaction with group discussion post-quiz was high, and hearing the proper reasoning process from the instructor enhanced residents' knowledge. In addition, at the end of the academic year, residents' self-assessed knowledge, with regard to topics taught through TBL, was very high. Maintaining a high level of trainee satisfaction with their education is very important, as satisfaction has been linked directly to academic progress (Kilgour et al. 2016); it also bears consequence on the ability to recruit and retain the best and brightest graduates into the program.
Our case-based application exercises allowed residents to practice with content under the watchful eye of a seasoned clinician who, through open group discussion, assisted in modeling clinical reasoning and critical thinking. We intentionally wrote our cases to include controversial answers or ambiguities that are common in clinical practice; controversy predictably arose and the intergroup discussion that followed helped foster respectful communication skills, deepen understanding of concepts and critical appraisal of one's knowledge, and acceptance of uncertainty that governs clinical practice in outpatient internal medicine, akin to a simulated environment. However, our study does not report on outcomes with regard to attitudes towards self-directed learning, improvement of communication skills, and teamwork as a result of exposure to TBL.

Conclusion
The TBL model is a successful model of teaching in the undergraduate world, but it is sparingly used in post graduate medical education. TBL offers a structured but flexible framework that is engaging and fun for both faculty and residents. TBL enhances residents' self-awareness of their knowledge and their engagement in, responsibility for, and satisfaction with learning. It can be successfully adapted to the realities and culture of a large, multi-site residency training program. However, it requires more faculty preparation compared to traditional models where learners present assignments. With the exception of session preparation time by faculty, there is minimal additional cost. TBL is an evolving pedagogy in graduate medical education, and active scholarship detailing varieties of implementation, according to published guidelines, is necessary to grow a body of evidence in support of a gold standard (Burgess et al. 2014).

Take Home Messages
TBL offers a structured framework that is flexible and adaptable to any residency program TBL enhances residents' engagement in and satisfaction for learning Preparation of TBL has only one cost: dedicated time by a faculty member A goal standard of TBL design is not yet available, and as such, there is room for experimentation