Skip to main content
SpringerLink
Log in

Temporal variability of surgical technical skill perception in real robotic surgery

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Summary score metrics, either from crowds of non-experts, faculty surgeons or from automated performance metrics, have been trusted as the prevailing method of reporting surgeon technical skill. The aim of this paper is to learn whether there exist significant fluctuations in the technical skill assessments of a surgeon throughout long durations of surgical footage.

Methods

A set of 12 videos of robotic surgery cases from common human patient robotic surgeries were used to evaluate the perceived technical skill at each individual minute of the surgical videos, which were originally 12–15 min in length. A linear mixed-effects model for each video was used to compare the ratings of each minute to those from every other minute in order to learn whether a change in scores over time can be detected and reliably measured apart from inter- and intrarater variation.

Results

Modeling the change over time of the global evaluative assessment of robotic skills scores significantly contributed to the prediction models for 11 of the 12 surgeons. This demonstrates that measurable changes in technical skill occur over time during robotic surgery.

Conclusion

The findings from this research raise questions about the optimal duration of footage needed to be evaluated to arrive at an accurate rating of surgical technical skill for longer procedures. This may imply non-negligible label noise for supervised machine learning approaches. In the future, it may be necessary to report a surgeon’s skill variability in addition to their mean score to have proper knowledge of a surgeon’s overall skill level.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Kowalewski T, Comstock B, Sweet R, Schaffhausen C, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Landman J, Lee B, Schwartz BF, McDougall E, Lendvay TS (2015) Crowd-sourced assessment of technical skills for validation of basic laparoscopic urologic skills (BLUS) tasks. J Urol 195(6):1859–1865

    Article  Google Scholar 

  2. Hung A, Chen J, Che Z, Nilanon T, Jarc A, Titus M, Oh PJ, Gill IS, Liu Y (2018) Utilizing machine learning and automated performance metrics to evaluate robot-assisted radical prostatectomy performance and predict outcomes. J Endourol 32(5):438–444

    Article  Google Scholar 

  3. Makary M, Daniel M (2016) Medical error—the third leading cause of death in the US. BMJ 353:i2139

  4. Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJ (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369(15):1434–1442

    Article  CAS  Google Scholar 

  5. Kelly JD, Petersen A, Lendvay TS, Kowalewski TM (2020) The effect of video playback speed on surgeon technical skill perception. Int J Comput Assist Radiol Surg 15:739–747

  6. Chen C, White L, Kowalewski T, Aggarwal R, Lintott C, Comstock B, Kuksenok K, Aragon C, Holst D, Lendvay T (2013) Crowd-sourced assessment of technical skills: a novel method to evaluate surgical performance. J Surg Res 187(1):65–71

    Article  Google Scholar 

  7. Doh FC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187(1):247–252

    Article  Google Scholar 

  8. Geri N, Winer A, Zaks B (2017) Challenging the six-minute myth of online video lectures: can interactivity expand the attention of learners? Online J Appl Knowl Manag 5(1):101–111

    Article  Google Scholar 

  9. Schmitt M, Bulterman DCA, Cesar PS (2018) The contrast effect: QoE of mixed-video qualities at the same time. Qual User Exp 3(1):7

    Article  Google Scholar 

  10. French A, Lendvay TS, Sweet RM, Kowalewski TM (2017) Predicting surgical skill from the first n seconds of a task: value over task time using the isogony principle. Int J Comput Assist Radiol Surg 12(7):1161–1170

    Article  Google Scholar 

  11. Kelly JD, Heller N, Petersen A, Lendvay TS, and Kowalewski TM (2020) The effect of video playback speed on perception of technical skill in robotic surgery. Int J Comput Assist Radiol Surg (submitted)

  12. Dockter R, Lendvay TS, Sweet RM, Kowalewski TM (2017) The minimally acceptable classification criterion for surgical skill: intent vectors and separability of raw motion data. Int J Comput Assist Radiol Surg 12:1151–1159

    Article  Google Scholar 

  13. FFmpeg Developers (2016) ffmpeg tool (Version 4.1.3) [Software]. http://ffmpeg.org/

  14. Core Team R (2018) R: a language and environment for statistical computing. Austria, Vienna. http://www.R-project.org/

  15. Python Software Foundation. Python language reference, version 3.6. http://www.python.org

Download references

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, University of Minnesota, Minneapolis, MN, USA

    Jason D. Kelly & Timothy M. Kowalewski

  2. Department of Biostatistics, University of Washington, Seattle, WA, USA

    Michael Nash

  3. Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN, USA

    Nicholas Heller

  4. Department of Urology, University of Washington, Seattle, WA, USA

    Thomas S. Lendvay

Authors
  1. Jason D. Kelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Nash
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicholas Heller
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thomas S. Lendvay
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Timothy M. Kowalewski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jason D. Kelly.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical standard

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Funding

This work was supported, in part, by the Office of the Assistant Secretary of Defense for Health Affairs under Award No. W81XWH-15-2-0030, the National Science Foundation CAREER Grant under Award No. 1847610. Opinions, interpretations, conclusions and recommendations are those of the authors and are not necessarily endorsed by the Department of Defense, the National Science Foundation or the National Institutes of Health’s National Center for Advancing Translational Sciences.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kelly, J.D., Nash, M., Heller, N. et al. Temporal variability of surgical technical skill perception in real robotic surgery. Int J CARS 15, 2101–2107 (2020). https://doi.org/10.1007/s11548-020-02253-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02253-5

Keywords

Search

Navigation