Deep Learning Analysis of Surgical Video Recordings to Assess Nontechnical Skills

Key Points Question Is it feasible to automatically assess an operating room (OR) team’s nontechnical skills through deep learning–based analysis of surgical videos? Findings This cross-sectional study of 30 cardiac surgical procedures found specific OR team members’ motion features, such as average trajectory and displacement acceleration, to positively correlate with higher nontechnical skills performance as assessed by the Non-Technical Skills for Surgeons (NOTSS) assessment tool, while displacement entropy was negatively correlated with NOTSS scores. These findings suggest that certain patterns of team motion in the OR are associated with a team’s nontechnical skills. Meaning This study suggests the feasibility of applying deep learning methods to analyze surgical videos and assess an OR team’s nontechnical skills, which could be used for surgical education and quality improvement initiatives.


OR Team Consent Process:
Eligible hospital staff provided an informed consent, which was distinct from that completed by patients, prior to beginning any A/V recordings in a case that they were assigned to (i.e., any staff member who had any reason to enter the OR while the recording was in progress).Research staff obtained their informed consent at one time point, which encompassed any subsequent cases that clinical staff may have been involved in.To obtain consent, research staff hosted meetings with each specialty, during which time they presented relevant background material, described the procedures that would be involved (e.g., setting up cameras and microphones, asking representative members to equip themselves with the ECG device, overviewing self-report surveys they would be asked to respond to, etc.), and welcomed questions from the providers.At the end of these meetings, research staff collected informed consent documents from all providers present.The informed consent documents outlined that researchers were interested in observing teamwork indicators, including communication patterns, leadership style, behaviors, etc.Given that an earlier grant had funded a very similar, multi-year project involving many of the same research staff and providers, providers had already been exposed to the premise of this work for an extended period of time, diminishing the likelihood of the Hawthorne Effect persisting throughout the data collection period.

Motion Data Pre-processing:
To deal with possible noise in the extracted data, we downsampled data into 1-second time window and applied a two-stage filtering process to ensure the integrity of the signal.Initially, a low-pass Butterworth filter was employed to attenuate high-frequency noise, which is common in motion capture data 39,40 .This filter was chosen for its flat frequency response in the passband, minimizing the distortion of the signal's amplitude.Subsequently, a Savitzky-Golay filter was utilized to smooth the data, which preserves the signal's higher momenta and is particularly effective in reducing noise when computing derivatives [40][41][42]

Motion features Formula Description
Displacement where is the person ID in each team and is the time index Displacement Speed where is the time between frames, is the person ID, and is the time index.

Displacement Speed Variability
Where Speed Variability(j) represents the speed variability of the team at time index j.Speed(i, j) denotes the speed of the person i at time index j.Average Speed(j) is the average speed for the team at time index j.n is the number of team members contributing to the calculation.

Displacement Acceleration
Here, is the time between frames, is the team person ID, and is the time index.

Displacement Entropy
The entropy of a given window is calculated using the formula, where is the proportion of observations in the window that belong to symbol .
presents a sample of team movement visualizations including average displacement and acceleration, and entropy over time.5.Correlation Between Team MotionFeatures and Individual NOTSS Scores: eFigure 2. presents a series of correlation analyses between four team motion features (Average Displacement, Average Trajectory, Speed Variability, and Displacement Entropy) and four NOTSS (Non-Technical Skills for Surgeons) categories (Decision Making, Situational Awareness, Communication & Teamwork, and Leadership).Each subplot displays a scatter plot with a linear regression line, along with the correlation coefficient and p-value for each analysis.The results show varying degrees of correlation across different motion features and NOTSS categories, with some combinations showing stronger relationships than others.Notably, Displacement Entropy demonstrates a significant negative correlation with Situational Awareness (r= -0.44) and Communication & Teamwork (r= -0.42) scores, suggesting that higher irregularity in displacement pattern in team movement may be associated with lower performance in these areas.Conversely, Avg.Team Trajectory shows positive correlation with Situational Awareness (r= 0.35) and Communication & Teamwork (r= 0.45) scores.These findings highlight the potential of motion analysis as a tool for assessing NTS in surgical environments.© 2024 Harari RE et al.JAMA Network Open.

eTable 1 .
Formulas used for calculating motion features for surgical team

eFigure 1 .
Movement visualization of surgical team members.First row plots the line chart of individual displacement over time.Second row plots the team's average displacement and acceleration over time.Third row shows team displacement entropy measured over time.© 2024 Harari RE et al.JAMA Network Open.

eFigure 2 .
Correlation Between Team Motion Features and NOTSS Scores © 2024 Harari RE et al.JAMA Network Open.

eTable 2 .
Linear regression models adjusted for preoperative and intraoperative variables.