2020 APDS SPRING MEETINGCrowd-Sourced and Attending Assessment of General Surgery Resident Operative Performance Using Global Ratings Scales
Introduction
While it is widely accepted that rapid, accurate assessment of intraoperative performance is essential to guiding resident feedback and self-improvement of technical skills, the provision of frequent, timely, and objective evaluation remains a challenge.1 Crowd sourcing may be one way to operationalize frequent objective feedback and has been shown to provide accurate assessment when raters use instruments focused on technical skills.2, 3, 4, 5 Meanwhile, there is emerging interest in using global assessments of performance.6
To address this issue, the system for improving and measuring procedural learning (SIMPL) application was developed. It is a smart-phone based mobile application for the evaluation of operative performance and autonomy. The application captures 3 metrics: an autonomy metric, or the Zwisch Scale, which has previously been validated, a difficulty scale, and a performance metric.7 While the SIMPL performance scale is designed to be intuitive, the performance metric has not yet been validated against existing, longer-form tools.
The correlation between crowd ratings from more detailed multi-item instruments versus global ratings scales is unknown. We sought to assess the extent to which both crowd and intraoperative attending ratings using objective structured assessment of technical skill (OSATS) or global objective assessment of laparoscopic skills (GOALS) would correlate with the SIMPL Zwisch and Performance scales.
Section snippets
Audio & Video Capture
Six core general surgery procedures, 3 open and 3 laparoscopic, were selected from the American Board of Surgery's Resident Assessments list including laparoscopic cholecystectomy, laparoscopic colectomy, laparoscopic inguinal hernia repair, open inguinal hernia repair, open ventral hernia repair, and thyroidectomy (Table 1).8 Intraoperative audio and video of 32 general surgery procedures were recorded. For laparoscopic procedures, video was captured using a single mounted GoPro Hero4 camera
Results
Crowd raters evaluated 32 procedures using GOALS/OSATS, Zwisch and Performance (35-50 ratings per video). Attendings also evaluated all 32 procedures using GOALS/OSATS and 26 of the procedures using SIMPL Zwisch and Performance. Six SIMPL requests were not complete by the attending surgeon within the requisite 72 hours after observed performance. Pearson correlation coefficients with 95% confidence intervals for crowd ratings were: GOALS and Zwisch −0.40 [−0.73 to 0.10], OSATS and Zwisch 0.11
Discussion
This study demonstrates that attending rater evaluation of global performance using the SIMPL performance scale correlates well with both GOALS and OSATS technical evaluation assessments. This provides additional evidence that when taken in aggregate, the technical performance of a resident tends to positively correlate with the observer's impression of their overall performance or readiness for graduated autonomy. We also sought to determine if crowd workers could be utilized to provide a
Conclusions
Overall, correlations between crowd-sourced ratings using GOALS or OSATS and SIMPL global operative performance ratings tools were weak, yet for attending raters, they were strong. Further studies are needed to see if more extensive crowd training would result in improved ability for global performance evaluation.
Conflict of Interest
The authors declare no conflicts of interest with respect to the authorship and/or publication of this article.
Funding Source
The project was supported by a grant from the Association of Surgical Education (ASE) and Association of Program Directors in Surgery (APDS). No grant number was provided.
References (12)
- et al.
Crowd-sourced assessment of technical skills: a novel method to evaluate surgical performance
J Surg Res
(2014) - et al.
Crowd-sourced assessment of technical skills: an opportunity for improvement in the assessment of laparoscopic surgical skills
Am J Surg
(2016) - et al.
Teaching and assessing operative skills: from theory to practice
Curr Probl Surg
(2017) - et al.
Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance
J Surg Educ
(2014) - et al.
A global assessment tool for evaluation of intraoperative laparoscopic skills
Am J Surg
(2005) - et al.
Video-based assessment in surgical education: a scoping review
J Surg Educ
(2019)