Elsevier

Journal of Surgical Education

Volume 77, Issue 3, May–June 2020, Pages 627-634
Journal of Surgical Education

ORIGINAL REPORTS
Concordance Between Expert and Nonexpert Ratings of Condensed Video-Based Trainee Operative Performance Assessment

https://doi.org/10.1016/j.jsurg.2019.12.016Get rights and content

Objective

We examined the impact of video editing and rater expertise in surgical resident evaluation on operative performance ratings of surgical trainees.

Design

Randomized independent review of intraoperative video.

Setting

Operative video was captured at a single, tertiary hospital in Boston, MA.

Participants

Six common general surgery procedures were video recorded of 6 attending-trainee dyads. Full-length and condensed versions (n = 12 videos) were then reviewed by 13 independent surgeon raters (5 evaluation experts, 8 nonexperts) using a crossed design. Trainee performance was rated using the Operative Performance Rating Scale, System for Improving and Measuring Procedural Learning (SIMPL) Performance scale, the Zwisch scale, and ten Cate scale. These ratings were then standardized before being compared using Bayesian mixed models with raters and videos treated as random effects.

Results

Editing had no effect on the Operative Performance Rating Scale Overall Performance (-0.10, p = 0.30), SIMPL Performance (0.13, p = 0.71), Zwisch (-0.12, p = 0.27), and ten Cate scale (-0.13, p = 0.29). Additionally, rater expertise (evaluation expert vs. nonexpert) had no effect on the same scales (-0.16 (p = 0.32), 0.18 (p = 0.74), 0.25 (p = 0.81), and 0.25 (p = 0.17).

Conclusions

There is little difference in operative performance assessment scores when raters use condensed videos or when raters who are not experts in surgical resident evaluation are used. Future validation studies of operative performance assessment scales may be facilitated by using nonexpert surgeon raters viewing videos condensed using a standardized protocol.

Section snippets

INTRODUCTION

Development as a surgeon requires the acquisition and refinement of operative skills. Technical competence is difficult to define and challenging to measure. Currently, general surgery trainees with the United States must complete at least 6 operative performance evaluations during their training,1 and this requirement may increase in the future. This, however, presents some implementation challenges. Specifically, this increased measurement requirement may not be feasible if the burden is

Audio/Video Capture

Intraoperative audio and video of 6 general surgery procedures (laparoscopic cholecystectomy, laparoscopic colectomy, laparoscopic inguinal hernia repair, open inguinal hernia repair, open ventral hernia repair, and thyroidectomy) (Table 1), were recorded (N = 6 full length videos, 6 condensed videos). All videos used in the current work were captured at a single institution. These specific procedures were chosen based on how common they were and the availability of widely used

RESULTS

A total of 78 assessments were collected across 6 procedures. For each procedure, 8 assessments were completed by evaluation experts and 5 assessments by nonexpert raters (Table 1). Expert raters generally scored performances lower than nonexpert raters using OPRS (-0.16 standard deviations [SD]). In contrast, compared to nonexperts, expert raters scored performances slightly higher with the SIMPL (0.18 SD), ten Cate (0.25 SD), and Zwisch (0.21 SD) tools. None of these differences was

DISCUSSION

In this study, we found no difference between raters with expertise in surgical resident evaluation and those without, and no difference for condensed compared to full-length videos. Our study demonstrated that assessments by surgeon raters who are nonexpert raters appear to be acceptable and not statistically significantly different than evaluations performed by raters who are evaluation experts. Furthermore, high residual variance suggests that the vast majority of the variability in

CONCLUSION

There is no difference in performance assessment scores between full-length and condensed videos. In addition, nonexpert surgeon raters are no different than surgeon raters with expertise in resident evaluation in the assessment of surgical performance. Future studies of operative performance may be facilitated by using nonexpert surgeon raters viewing condensed videos edited using a standardized protocol.

Cited by (0)

Funding: The project was supported by a grant from the Association of Surgical Education (ASE) and Association of Program Directors in Surgery (APDS).

View full text