Association for Academic Surgery
Correlating surgical clerkship evaluations with performance on the National Board of Medical Examiners examination

https://doi.org/10.1016/j.jss.2014.02.031Get rights and content

Abstract

Background

Evaluation of medical students during the surgical clerkship is controversial. Performance is often based on subjective scoring, whereas objective knowledge is based on written examinations. Whether these measures correspond or are relevant to assess student performance is unknown. We hypothesized that student evaluations correlate with performance on the National Board Of Medical Examiners (NBME) examination.

Methods

Data were collected from the 2011–2012 academic year. Medical students underwent a ward evaluation using a seven-point Likert scale assessing six educational competencies. Students also undertook the NBME examination, where performance was recorded as a percentile score adjusted to national standards.

Results

A total of 129 medical students were studied. Scores on the NBME ranged from the 52nd to the 96th percentile with an average in the 75th percentile (±9). Clerkship scores ranged from 3.2–7.0 with a mean of 5.7 (±0.8). There was a strong positive association between higher NBME scores and higher clerkship evaluations shown by a Pearson correlation coefficient of 0.47 (P < 0.001). Students clustered with below average ward evaluations (3.0–4.0) were in the 69.5th percentile of NBME scores, whereas students clustered with above average ward evaluations (6.0–7.0) were in the 79.2th percentile (P < 0.001).

Conclusions

A strong positive relationship exists between subjective ward evaluations and NBME performance. These data may afford some confidence to surgical faculty and surgical resident ability to accurately evaluate medical students during clinical clerkships. Understanding factors in student performance may help in improving the surgical clerkship experience.

Introduction

Appropriate evaluation of medical student performance during the third year is an issue of limited research. Recently, Epstein described three main goals of assessing medical students: (1) optimize capabilities by creating incentives and a structure for learning, (2) protect the public by identifying incompetent physicians, and (3) create a metric for stratifying applicants in selection for advanced training [1]. Medical students are heavily invested in their educational process both academically and financially. During both the undergraduate and preclinical years of a traditional medical student education, coursework and class performance are largely determined by written examinations. Although the quality of these examinations may vary, an objective criterion can be applied to the evaluative and grading process.

There is variability in the manner by which students are provided with a summative evaluation at the end of their surgical clerkship [2], [3]. At present the most common method used in the United States is to incorporate both a subjective component completed by surgeons based on students' clinical performance and an objective component to assess students' fund of knowledge. Variability or inconsistencies in the evaluation process have been demonstrated to result in most of the students not feeling satisfied with subjective faculty evaluations [4]. Previous studies demonstrate that the subjective evaluation process has questionable validity and reliability. It is useful only when used along with other methods of evaluation, and represents a composite of multiple evaluators [5], [6], [7], [8], [9]. It does have its advantages in being a low cost, flexible, and convenient tool [10]. Furthermore, such evaluations are important in identifying critical nontechnical qualities needed in a physician such as professionalism [11] and interpersonal communication.

Objective performance is almost uniformly measured using the National Board Of Medical Examiners (NBME), however the validity of this examination in regards to assessing surgical knowledge is controversial despite rigorous validation [2], [7], [12]. Although there is no “passing score,” the test is scaled to have a mean of 70% with standard deviation of 8. To date, studies examining the relationship between subjective faculty evaluations and the objective measurement of the NBME surgery subject examination have not shown a positive correlation [8], [9], [13], [14].

Our subjective evaluation process includes the feedback of all faculty surgeons and senior residents that interact with medical students during their clerkship. In addition, all students are required to undertake the NBME surgical examination. Given the rigor of our faculty evaluations, we hypothesized that subjective evaluation of surgical students correlate with their performance on the NBME surgery subject examination.

Section snippets

Methods

During the 2011–2012 academic year, we collected data from all rotating students on the surgical clerkship. Their clinical experience consisted of two 4-wk rotations on a surgical service, of which one was general surgery based. In addition, students completed an additional 4 wk of clinical experience, which was split into 2-wk blocks of elective rotations of the students choosing. Students completed the clerkship during one of four quarters during the year across four different health care

Results

A total of 129 medical students were included in the study. Subjective clerkship scores ranged from 3.2–7.0 with a mean of 5.7 (±0.8). There was no observed increasing scoring trend as the year progressed. Also, students who had completed previous clerkships before the surgical clerkship did not demonstrate higher subjective scores (Table). Scores on the NBME surgery subject examination ranged from the 52nd–96th percentile with an average in the 75th percentile (±9). As the year progressed and

Discussion

The correlation between the surgical NBME examination and subjective medical student evaluations has not previously, until now, been shown to correlate [8], [9], [13], [14]. The reason for this is unknown, however, variability in the subjective component of the evaluation process is most likely [5], [15]. Significant areas of bias and error are inherent in subjective evaluations of medical students. Furthermore, evaluators may have limited exposure to students, use differing standards or

Acknowledgment

Authors' contributions: C.M.R., D.Y.K., J.M., A.S., and V.B. contributed to conception and design. C.M.R., D.Y.K., J.M., and V.B. analyzed and interpreted the data. C.M.R., D.Y.K., and V.B wrote the article. C.M.R., D.Y.K., J.M., A.S., and V.B. made the critical revision of the article.

References (16)

There are more references available in the full text version of this article.

Cited by (0)

This work will be presented at the ninth Annual Academic Surgical Congress to be held February 4–6, 2014 in San Diego, CA, and was elected an AAS Outstanding Resident Research Award.

View full text