Introduction

Evidence-based medicine (EBM) is defined as “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” [1]. Practicing EBM requires a clinician to integrate clinical expertise with the best available evidence as well as the patient’s values in order to make medical decisions [2]. It is widely recognized that the practice of EBM requires clinicians to use the following five-step process: (i) create an answerable question from a clinical case, (ii) search the literature for the best available evidence, (iii) critically appraise the evidence, (iv) integrate and apply the results with clinical judgment and patient values, and (v) evaluate the effectiveness and efficiency of their methods [3, 4].

It is important for medical students to gain early and frequent practice with EBM as physicians rely on EBM on a daily basis in order to make informed decisions on patient care [5]. The Association of American Medical Colleges (AAMC) recognizes “basic skills in critiquing the quality of evidence and assessing the applicability to their patients and the clinical context” as a core competency for entering residency [6]. However, a 2007 study assessing residents’ ability to interpret basic evidence-based medicine concepts including study design, methods, and data analysis found that residents averaged a score of only 41% on a test focusing on statistical knowledge and interpretation of results [7]. All residents surveyed agreed that it is important to understand basic research methods in order to understand the literature and wanted to learn more about this topic [7].

Research has examined the most effective way to teach EBM to medical students. Khan and Coomarasamy (2006) developed a hierarchy of methods to teach evidence-based medicine, from most to least effective, with level 1 being interactive and clinically integrated activities, level 2(a) consisting of interactive but classroom-based activities, level 2(b) didactic but clinically integrated, and lastly level 3 being didactic or classroom teaching alone [8]. Pre-clerkship EBM curricula typically consist of lectures and classroom sessions; however, during the clinical years of medical school, the teaching of EBM typically occurs at the bedside, with opportunities for further discussions and case presentations based upon students’ real-time clinical experiences [2]. Therefore, pre-clerkship EBM curricula should attempt to teach EBM in a clinically relevant and interactive manner. Prior research has shown that a combination of lectures and interactive clinical tutorials is effective in preparing medical students to utilize EBM during clerkships [2, 9, 10].

Journal clubs can be a particularly effective way for medical students to gain practice applying evidence-based medicine. William Osler is credited with creating the first journal club at McGill University in 1875, and by the 1980s, 85% of medicine residencies had a journal club component [11]. Journal clubs can be helpful for students to gain practice searching for relevant literature and critically appraising research alongside peers [12, 13]. However, many journal clubs read expertly chosen, pre-selected articles, preventing learners from engaging in formulating a clinical question and searching the literature for the best available evidence [2, 3, 14].

The present study sought to assess the effectiveness of a student-run, faculty-mentored journal club elective for improving EBM skills and confidence in pre-clerkship medical students. Specifically, this journal club made use of theoretical patient cases to allow students to practice all steps of EBM including forming clinical questions, performing a literature search, appraising evidence, and applying literature to hypothetical patients.

Methods

Participants and Overview

We offered a virtual journal club elective during the 2020–2021 academic year to all first- and second-year medical students at Rutgers Robert Wood Johnson Medical School (RWJMS) (n = 330). First-year medical students had recently completed a 2-week introductory EBM course prior to beginning the elective, which the second-year students had also completed during their first year. The requirements to fulfill the elective were as follows:

  1. 1.

    Attend the 2-h journal club introductory session on clinical questions and literature searching.

  2. 2.

    Attend at least five of the eight monthly journal club sessions.

  3. 3.

    Lead and present for one of the eight journal club sessions (completed in groups of 3–4 students).

Each group was assigned one of eight pre-selected theoretical patient cases and was asked to complete the following for their assigned session, with faculty support:

  1. (a)

    Write a clinical question in patient, intervention, comparison, outcome (PICO) format based on the patient case. In order to use this method, a clinical question is written using the following format:

    1. (i)

      Patients: who are the relevant patients?

    2. (ii)

      Intervention: what is the main intervention, management strategy, or exposure?

    3. (iii)

      Comparison: what is the control or main alternative?

    4. (iv)

      Outcome: what is being measured?

      The PICO method was chosen because it is a standardized format utilized by EBM practitioners to write focused clinical questions.

  2. (b)

    Conduct a literature search to answer the clinical question.

  3. (c)

    Choose and disseminate an article for the journal club session.

  4. (d)

    Prepare a PowerPoint presentation in order to guide the journal club session.

Participation in a research study, approved by the Rutgers Institutional Review Board (Pro2020001135), was offered as part of this elective. Students enrolled in the research component took a pre-test before the introductory session and post-test at the conclusion of the journal club. All students in the elective completed a course evaluation at the conclusion of the journal club. Due to the COVID-19 pandemic, all elective events were conducted on Zoom. The overall format of the journal club is shown in Fig. 1.

Fig. 1
figure 1

Journal club format

Full size image

Evaluation

Students who volunteered for the research study were asked to take an online pre-test on Qualtrics prior to the first session in September 2020 [15]. The pre-test consisted of the Test of Evidence Based Medicine Knowledge and Skills and a shortened version of the Evidence-based Practice Confidence (EPIC) Scale [16,17,18]. The Test of Evidence Based Medicine Knowledge and Skills is a validated, 17-item multiple choice test which measures skills in the following domains of EBM: seeking appropriate resources, critical appraisal, data interpretation, and diagnostic accuracy [17, 18]. This test is scored from one to 17, with a higher score indicating greater proficiency in EBM. Based on scores on this test, respondents can be categorized as beginner (mean = 8.74), intermediate (mean = 12.20), or expert (14.75) in EBM. The EPIC Scale is a tool that evaluates healthcare provider confidence in the ability to implement EBM [16]. We assessed confidence using seven total Likert questions from the EPIC Scale asking respondents to rate their confidence in the following: identifying a knowledge gap, formulating a question, conducting a literature search, critically appraising the methods of a study, critically appraising the measurement properties of a study, interpreting results, and determining if evidence applies to a particular patient.

At the conclusion of the elective in April 2021 and after attending at least five of the eight journal club sessions, participants were asked to take a post-test on Qualtrics [15]. The post-test consisted of the Test of Evidence Based Medicine Knowledge and Skills and a modified version of the Evidence-based Practice Confidence (EPIC) Scale [16,17,18].

Additionally, a course evaluation was created and included at the end of the post-test. This evaluation was composed of 13 Likert-type items and three open-ended questions regarding their subjective experience in the journal club and suggestions for improvement.

Journal Club Sessions

Eight 1-h sessions were offered. All students were required to attend the introductory session, at least five monthly journal club sessions, and lead one session with their assigned group.

Journal Club Introductory Session

Part 1: Introduction

A faculty member from the RWJMS Committee on EBM delivered a 30-min, interactive presentation over Zoom on creating clinical questions and conducting literature searches (Appendix A). Using example cases, the faculty member explained to students how to formulate a clinical question, write a clinical question in PICO format, and search the literature for evidence.

Part 2: Group Assignment

Students were split up into groups of 4 in breakout rooms and assigned one of the 8 journal club cases, which were adapted from the New England Journal of Medicine Clinical Practice section (Appendix B). Student groups were given 1 h to create a PICO question. Faculty from the RWJMS Committee on EBM rotated throughout the breakout rooms to provide initial feedback on the PICO questions. At the conclusion of the session, student groups posted their PICO question onto a dedicated Canvas course site. Faculty provided written feedback over Canvas regarding the PICO question.

Journal Club Session Format

One of the eight patient cases was the focus of each 1-h journal club session. The student group assigned to that particular case served as the leaders of that session. They were tasked with utilizing their PICO question to conduct a literature search and find an article for the rest of the group to read. Student groups sent their chosen article to a faculty member for approval prior to the session, and all students were provided with the article 1 week ahead of the session.

The assigned student group prepared a presentation to be used as a framework for discussion of the article. To guide their preparation, the assigned group was advised to appraise their chosen article using either the Centre for Evidence-Based Medicine (CEBM) or Critical Appraisal Skills Programme (CASP) checklists, which are readily available online [19, 20]. The presentation format (Appendix C) included a summary of the case, any necessary background information needed to understand the case, the PICO question, literature search strategy, and a discussion of the main points of the article including the methods, results, limitations, and conclusions. The assigned student group was encouraged to make the session interactive by asking the group questions throughout. At the end of each session, the group discussed how they would apply the results of the particular study to the patient case. EBM faculty were present at each session. A sample presentation is provided in Appendix D.

Faculty Preparation

Faculty facilitators included members of the RWJMS Committee on EBM and were from the Departments of Internal Medicine and Neuroscience and Cell Biology. One to two faculty regularly attended each journal club. Visiting faculty also were invited to attend based on their area of expertise. For the introductory session, one faculty facilitator led the presentation on clinical questions and literature searching, while the remaining faculty assisted student groups with PICO questions. During the introductory session, faculty rotated through approximately three breakout rooms, during the 1-h PICO question section, spending at least 10 min in each room. Faculty facilitators were assigned several student groups and were asked to review and approve each group’s chosen journal club article prior to their scheduled session. At the journal club session, faculty in attendance helped provide clinical context to the patient case, offered assistance with interpretation of the article, and asked relevant questions to the group for discussion.

Analysis

We compared pre- and post-test scores using the student two-sample t test. A two by two analysis of variance (ANOVA) was utilized to evaluate the interaction between pre- and post-test scores and class year. We used SPSS Statistics version 28.0.0.0 (IBM) and GraphPad Prism 9.0.0 to analyze the data [21, 22]. We considered p < 0.05 to be a statistically significant change.

Results

Participants

Out of a total of 330 eligible students, 47 medical students (14.2%) completed the pre-test (24 MS1 and 23 MS2). Twenty-nine of the 47 medical students (17 MS1 and 12 MS2, 8.7% of the classes) completed both the pre- and post-tests and the elective requirements. Thirty-five of the 47 medical students (10.6% of the classes) completed all requirements of the elective. Thirty-five of the 47 medical students completed the post-elective course evaluation.

Modified EPIC Scale

Table 1 and Fig. 2 show the change in confidence from pre-test to post-test across the seven domains of EBM confidence using the modified EPIC scale. Confidence increased significantly across all seven domains of EBM confidence (p < 0.05). Using a 2 × 2 ANOVA with post-hoc Sidak’s multiple comparison test, it was found that MS1 students contributed most to improvements in self-assessed EBM confidence. For formulating a PICO question, both MS1 and MS2 students had significant improvements in confidence (p < 0.05).

Table 1 Mean (SD) confidence levels as measured by the modified EPIC test across seven domains of EBM
Full size table
Fig. 2
figure 2

Change in confidence levels across seven domains from pre-test to post-test using the modified EPIC scale

Full size image

EBM Test of Knowledge and Skills

No changes were observed from pre-test (mean = 12.6, SD = 1.84) to post-test (mean = 12.2, SD = 2.23) in the scores on the Test of EBM Knowledge and Skills (Table 2 and Fig. 3). MS2 students (mean = 13.3, SD = 1.87) performed significantly better than MS1 students (mean = 11.4, SD = 2.15) on the post-test (p < 0.05).

Table 2 Mean (SD) pre-test and post-test scores on the EBM Test of Knowledge and Skills
Full size table
Fig. 3
figure 3

Change in score pre-test to post-test using the modified EBM Test of Knowledge and Skills

Full size image

Course Evaluation

Table 3 shows median ratings on a scale from 1 “strongly disagree” to 5 “strongly agree” for each aspect of the journal club. Overall, median ratings for each statement in the course evaluation were 4, “agree.” On narrative feedback, participants emphasized practice with literature search, presenting to peers, and faculty guidance. The majority of students agreed or strongly agreed with implementing the journal clubs within the pre-clerkship curriculum.

Table 3 Median (IQR) ratings on course evaluation, from 1 “strongly disagree” to 5 “strongly agree”
Full size table

Discussion

In order to enhance the pre-clerkship medical student EBM curriculum at our institution, we developed a year-long journal club elective. Using hypothetical patient cases as the starting point, students gained practice with each step of EBM including (i) creating an answerable question from a clinical case, (ii) searching the literature for the best available evidence, (iii) critically appraising the evidence, (iv) integrating and applying the results with clinical judgment and patient values, and (v) evaluating the effectiveness and efficiency of their methods [3, 4]. Our aim was to determine the effectiveness of this journal club format in improving EBM skills and confidence in pre-clerkship medical students.

We found that EBM confidence significantly improved from pre-test to post-test across all seven areas of EBM, with MS1 students improving more than MS2 students. The larger improvement in self-confidence in the MS1 versus the MS2 cohort may have been because the MS1 group had less experience with EBM and thus started the study with lower confidence levels than the MS2 cohort. MS1 students may also have been more engaged in the journal club. Although the MS2 students completed more of the pre-clerkship EBM curriculum and thus had higher pre-test confidence scores, they still demonstrated an increase in self-confidence in EBM after this journal club. It was notable that confidence in creating a PICO was significantly improved in both the MS1 and MS2 cohort. Writing a PICO from a clinical case is an important skill that was emphasized in this journal club. Students received both instruction and faculty feedback on their ability to formulate a PICO to answer a clinical question.

Prior research has shown that self-efficacy beliefs are a primary driver in the decision to engage in an activity and are correlated with performance of the specific task [16, 23, 24]. Therefore, students who are more confident in their ability to use EBM in clinical practice may be more likely to utilize it effectively. Given that this journal club format allowed participants to actively engage in each step of EBM, it is not surprising that their confidence improved.

We did not find any significant difference in scores on a test of EBM skills after participation in our journal club. Overall, mean scores at both pre-test and post-test on the Test of EBM Knowledge and Skills were in the “intermediate” skill range. This test focused more on interpretation of statistical equations and data. While data interpretation was part of our journal clubs, this was not an explicit focus, which may partly explain the lack of improvement in our study. Journal clubs, however, can be effective activities to reinforce basic data interpretation and statistical concepts. Perhaps including a review of the statistical methods utilized in each journal club article would be helpful in teaching these aspects of EBM.

Our journal club format was rated favorably by participants, a majority of whom felt this format should be integrated into the EBM curriculum at our institution. The use of a similar journal club format in graduate medical education has shown positive results. Rodriguez and Hawley-Molloy revamped an internal medicine residency journal club in order to encourage participants to learn how to answer clinical questions [14]. Participants were divided into groups and tasked with searching the literature for an article that answered a clinical question. A majority of participants voted to continue the new format and respondents commented that they were more actively engaged using this new format. Results from both an obstetrics and gynecology residency program and an emergency medicine residency program revealed greater participation and engagement using a similar journal club format based on clinical questions [25]. By utilizing this case-based format in undergraduate medical education, trainees may be more confident in their ability to practice EBM in their daily patient encounters.

Journal club formats have also been utilized in nursing and pharmacy education [26,27,28]. A review of the use of journal clubs in pharmacy education noted that journal clubs can be utilized to each EBM or biostatistics [28]. Perhaps if we focused more time on interpreting the statistical components of each article, our students would have had improvements on the knowledge test. These articles also importantly note that the quality of a journal club is highly variable — it is important to have organization, leadership, and clearly defined goals in order for students to learn effectively from membership in a journal club.

This study is not without limitations. First, this journal club elective was voluntary and therefore prone to selection bias, as it is possible that only students who were already interested and/or proficient in EBM joined the elective. Additionally, MS1 students took the pre-test within weeks after completing a 2-week introductory EBM course, which likely influenced their pre-test score significantly. In addition, there was no control group of pre-clerkship students who did not participate in the EBM elective and took the pre-test and post-test to compare the results to. Despite these limitations, the results of our study add to a growing body of literature that suggests the use of journal clubs focused on clinical questions is effective in teaching EBM. Further research should be done to assess whether journal club experiences improve critical appraisal skills or whether journal club experiences influence the likelihood of practicing EBM.

Conclusions

The present study utilized a journal club centered on patient cases in order to provide pre-clerkship medical students the opportunity to engage in EBM. The use of patient cases as a starting point was a strength of this study, as it allowed students to engage in all aspects of EBM including generating a clinical question, searching the literature, appraising the literature, and applying the literature to a hypothetical patient. Additionally, students gained practice presenting in front of their peers. This format was effective in improving confidence in utilizing EBM and was rated positively by participants. This student-led, faculty-mentored EBM journal club can be used as a model for curricular integration into an existing EBM curriculum. Incorporating journal clubs into the pre-clerkship curriculum allows medical students to gain early practice with the use of EBM in patient care prior to the clerkship years. The results of our research add to the growing literature that journal clubs centered on patient cases and clinical questions are effective in teaching EBM to medical trainees. In the future, we plan to implement this journal club model in our curriculum for all pre-clerkship medical students and continue to assess its effectiveness using both the knowledge and confidence questionnaires.