Skip to main content
SpringerLink
Log in

Video-based assessment of intraoperative surgical skill

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Surgeons’ skill in the operating room is a major determinant of patient outcomes. Assessment of surgeons’ skill is necessary to improve patient outcomes and quality of care through surgical training and coaching. Methods for video-based assessment of surgical skill can provide objective and efficient tools for surgeons. Our work introduces a new method based on attention mechanisms and provides a comprehensive comparative analysis of state-of-the-art methods for video-based assessment of surgical skill in the operating room.

Methods

Using a dataset of 99 videos of capsulorhexis, a critical step in cataract surgery, we evaluated image feature-based methods and two deep learning methods to assess skill using RGB videos. In the first method, we predict instrument tips as keypoints and predict surgical skill using temporal convolutional neural networks. In the second method, we propose a frame-wise encoder (2D convolutional neural network) followed by a temporal model (recurrent neural network), both of which are augmented by visual attention mechanisms. We computed the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and predictive values through fivefold cross-validation.

Results

To classify a binary skill label (expert vs. novice), the range of AUC estimates was 0.49 (95% confidence interval; CI = 0.37 to 0.60) to 0.76 (95% CI = 0.66 to 0.85) for image feature-based methods. The sensitivity and specificity were consistently high for none of the methods. For the deep learning methods, the AUC was 0.79 (95% CI = 0.70 to 0.88) using keypoints alone, 0.78 (95% CI = 0.69 to 0.88) and 0.75 (95% CI = 0.65 to 0.85) with and without attention mechanisms, respectively.

Conclusion

Deep learning methods are necessary for video-based assessment of surgical skill in the operating room. Attention mechanisms improved discrimination ability of the network. Our findings should be evaluated for external validity in other datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Aghdasi N, Bly R, White LW, Hannaford B, Moe K, Lendvay TS (2015) Crowd-sourced assessment of surgical skills in cricothyrotomy procedure. J Surg Res 196(2):302–306

    Article  Google Scholar 

  2. Agresti A (2003) Categorical data analysis, vol 482. Wiley, Hoboken

    Google Scholar 

  3. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations

  4. Bettadapur V, Schindler G, Plötz T, Essa I (2013) Augmenting bag-of-words: data-driven discovery of temporal and structural information for activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2619–2626

  5. Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJO (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369(15):1434–1442

    Article  CAS  Google Scholar 

  6. Carroll N, Richardson I (2016). Software-as-a-medical device: demystifying connected health regulations. J Syst Inf Technol

  7. Dalal N, Triggs B, Schmid C (2006) Human detection using oriented histograms of flow and appearance. In: European conference on computer vision, pp 428–441. Springer

  8. Deal SB, Stefanidis D, Telem D, Fanelli RD, McDonald M, Michael Ujiki L, Brunt M, Alseidi AA (2017) Evaluation of crowd-sourced assessment of the critical view of safety in laparoscopic cholecystectomy. Surg Endosc 31(12):5094–5100

    Article  Google Scholar 

  9. Golnik C, Beaver H, Gauba V, Lee AG, Mayorga E, Palis G, Saleh GM (2013) Development of a new valid, reliable, and internationally applicable assessment tool of residents’ competence in ophthalmic surgery (an American ophthalmological society thesis). Transactions of the American Ophthalmological Society. 111:24

  10. Hand DJ, Till RJ (2001) A simple generalisation of the area under the roc curve for multiple class classification problems. Mach Learn 45(2):171–186

    Article  Google Scholar 

  11. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 770–778

  12. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  CAS  Google Scholar 

  13. Kim TS, O’Brien M, Zafar S, Hager GD, Sikder S, Swaroop Vedula S (2019) Objective assessment of intraoperative technical skill in capsulorhexis using videos of cataract surgery. Int J Comput Assist Radiol Surg 14(6):1097–1105

    Article  Google Scholar 

  14. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. International Conference on Learning Representations, 12

  15. Laptev I (2005) On space-time interest points. Int J Comput Vision 64(2–3):107–123

    Article  Google Scholar 

  16. Lendvay TS, White L, Kowalewski T (2015) Crowdsourcing to assess surgical skill. JAMA Surg 150(11):1086–1087

    Article  Google Scholar 

  17. Lin T, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp 2999–3007

  18. Maier-Hein L, Vedula SS, Speidel S, Navab N, Kikinis R, Park A, Eisenmann M, Feussner H, Forestier G, Giannarou S, Hashizume M, Katic D, Kenngott H, Kranzfelder M, Malpani A, März K, Neumuth T, Padoy N, Pugh C, Schoch N, Stoyanov D, Taylor R, Wagner M, Hager GD, Jannin P (2017) Surgical data science for next-generation interventions. Nat Biomed Eng 1(9):691–696

    Article  Google Scholar 

  19. Malpani SA, Vedula S, Chen CCG, Hager GD (2015) A study of crowdsourced segment-level surgical skill assessment using pairwise rankings. Int J Comput Assist Radiol Surg 10(9):1435–1447

    Article  Google Scholar 

  20. Marcano-Cedeno A, Quintanilla-Domínguez J, Cortina-Januchs MG, Andina D(2010) Feature selection using sequential forward selection and classification applying artificial metaplasticity neural network. In: IECON 2010-36th annual conference on IEEE industrial electronics society, pp 2845–2850. IEEE,

  21. Pandey VA, Wolfe JHN, Black SA, Cairols M, Liapis CD, Bergqvist D (2008) Self-assessment of technical skill in surgery: the need for expert feedback. Ann R Coll Surg England 90(4):286–290

    Article  CAS  Google Scholar 

  22. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Benoit Steiner L, Fang JB, Chintala S (2019) Pytorch: an imperative style high-performance deep learning library. In: Wallach H, Larochelle H, Beygelzimer A, d‘Alché-Buc F, Fox E, Garnett R et al (eds) Advances in neural information processing systems. Curran Associates Inc., RedHook

    Google Scholar 

  23. Powers MK, Boonjindasup A, Pinsky M, Dorsey P, Maddox M, Su LM, Gettman M, Sundaram CP, Castle EP, Lee JY, Lee BR (2016) Crowdsourcing assessment of surgeon dissection of renal artery and vein during robotic partial nephrectomy: a novel approach for quantitative assessment of surgical performance. J Endourol 30(4):447–452

    Article  Google Scholar 

  24. Pugh C, Hashimoto DA, Korndorffer Jr JR (2020) The what? how? and who? of video based assessment. Am J Surg

  25. Robertson S (2004) Understanding inverse document frequency: on theoretical arguments for idf. J Doc

  26. Sharma Y (2014) Surgical skill assessment using motion texture analysis. PhD thesis, Georgia Institute of Technology

  27. Sun K, Xiao B, Liu D, Wang J (2019) Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5693–5703

  28. Valanci-Aroesty S, Alhassan N, Feldman LS, Landry T, Mastropietro V, Fiore Jr J, Lee L, Fried GM, Mueller CL (2020) Implementation and effectiveness of coaching for surgeons in practice–a mixed studies systematic review. J Surg Educ

  29. Vedula SS, Ishii M, Hager GD (2017) Objective assessment of surgical technical skill and competency in the operating room. Ann Rev Biomed Eng 19:301–325

  30. Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R, Zemel R, Bengio Y (2015) Show, attend and tell: neural image caption generation with visual attention. In: International conference on machine learning, pp 2048–2057

  31. Zhou P, Shi W, Tian J, Qi Z, Li B, Hao H, Xu B (2016) Attention-based bidirectional long short-term memory networks for relation classification. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp 207–212

  32. Zia A, Essa I (2018) Automated surgical skill assessment in rmis training. Int J Comput Assist Radiol Surg 13(5):731–739

    Article  Google Scholar 

  33. Zia A, Sharma Y, Bettadapura V, Sarin EL, Clements MA, Essa I (2015) Automated assessment of surgical skills using frequency analysis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp 430–438. Springer

  34. Zia A, Sharma Y, Bettadapura V, Sarin EL, Essa I (2018) Video and accelerometer-based motion analysis for automated surgical skills assessment. Int J Comput Assist Radiol Surg 13(3):443–455

    Article  Google Scholar 

  35. Zia A, Sharma Y, Bettadapura V, Sarin EL, Ploetz T, Clements MA, Essa I (2016) Automated video-based assessment of surgical skills for training and evaluation in medical schools. Int J Comput Assist Radiol Surg 11(9):1623–1636

    Article  Google Scholar 

Download references

Acknowledgements

Dr. Austin Reiter mentored this work in its early stages.

Funding

Drs. Vedula, Sikder, and Hager are supported by a grant from the National Institutes of Health, USA; NIH 1R01EY033065. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author information

Author notes

  1. Sanchit Hira, Digvijay Singh and Tae Soo Kim have equally contributed to this work.

Authors and Affiliations

  1. Laboratory for Computational Sensing & Robotics, Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD, 21218, USA

    Sanchit Hira, Gregory Hager & Shameema Sikder

  2. Department of Computer Science, Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD, 21218, USA

    Digvijay Singh, Tae Soo Kim & Gregory Hager

  3. Malone Center for Engineering in Healthcare, Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD, 21218, USA

    Tae Soo Kim, Gregory Hager, Shameema Sikder & S. Swaroop Vedula

  4. Indian Institute of Technology, Hauz Khas, New Delhi, 110016, India

    Shobhit Gupta

  5. Wilmer Eye Institute, Johns Hopkins University School of Medicine, 615 N. Wolfe Street, Baltimore, MD, 21287, USA

    Shameema Sikder

Authors
  1. Sanchit Hira
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Digvijay Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tae Soo Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shobhit Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gregory Hager
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shameema Sikder
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. S. Swaroop Vedula
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. Swaroop Vedula.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (docx 36 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hira, S., Singh, D., Kim, T.S. et al. Video-based assessment of intraoperative surgical skill. Int J CARS 17, 1801–1811 (2022). https://doi.org/10.1007/s11548-022-02681-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-022-02681-5

Keywords

Search

Navigation