Skip to main content
SpringerLink
Log in

Current applications of artificial intelligence for intraoperative decision support in surgery

  • Review
  • Published:
Frontiers of Medicine Aims and scope Submit manuscript

Abstract

Research into medical artificial intelligence (AI) has made significant advances in recent years, including surgical applications. This scoping review investigated AI-based decision support systems targeted at the intraoperative phase of surgery and found a wide range of technological approaches applied across several surgical specialties. Within the twenty-one (n = 21) included papers, three main categories of motivations were identified for developing such technologies: (1) augmenting the information available to surgeons, (2) accelerating intraoperative pathology, and (3) recommending surgical steps. While many of the proposals hold promise for improving patient outcomes, important methodological shortcomings were observed in most of the reviewed papers that made it difficult to assess the clinical significance of the reported performance statistics. Despite limitations, the current state of this field suggests that a number of opportunities exist for future researchers and clinicians to work on AI for surgical decision support with exciting implications for improving surgical care.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Spencer F. Teaching and measuring surgical techniques: the technical evaluation of competence. Bull Am Coll Surg 1978; 63: 9–12

    Google Scholar 

  2. Suliburk JW, Buck QM, Pirko CJ, Massarweh NN, Barshes NR, Singh H, Rosengart TK. Analysis of human performance deficiencies associated with surgical adverse events. JAMA Netw Open 2019; 2(7): e198067

    Article  Google Scholar 

  3. Pugh CM, Santacaterina S, DaRosa DA, Clark RE. Intra-operative decision making: more than meets the eye. J Biomed Inform 2011; 44(3): 486–496

    Article  Google Scholar 

  4. Hashimoto DA, Axelsson CG, Jones CB, Phitayakorn R, Petrusa E, McKinley SK, Gee D, Pugh C. Surgical procedural map scoring for decision-making in laparoscopic cholecystectomy. Am J Surg 2019; 217(2): 356–361

    Article  Google Scholar 

  5. Pugh CM, DaRosa DA. Use of cognitive task analysis to guide the development of performance-based assessments for intraoperative decision making. Mil Med 2013; 178(10 Suppl): 22–27

    Article  Google Scholar 

  6. Flin R, Youngson G, Yule S. How do surgeons make intraoperative decisions? Qual Saf Health Care 2007; 16(3): 235–239

    Article  Google Scholar 

  7. Hashimoto DA, Rosman G, Witkowski ER, Stafford C, Navarette-Welton AJ, Rattner DW, Lillemoe KD, Rus DL, Meireles OR. Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy. Ann Surg 2019; 270(3): 414–421

    Article  Google Scholar 

  8. Hashimoto DA, Rosman G, Rus D, Meireles OR. Artificial intelligence in surgery: promises and perils. Ann Surg 2018; 268 (1): 70–76

    Article  Google Scholar 

  9. Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL. Artificial intelligence in radiology. Nat Rev Cancer 2018; 18(8): 500–510

    Article  CAS  Google Scholar 

  10. Bera K, Schalper KA, Rimm DL, Velcheti V, Madabhushi A. Artificial intelligence in digital pathology—new tools for diagnosis and precision oncology. Nat Rev Clin Oncol 2019; 16(11): 703–715

    Article  Google Scholar 

  11. Hogarty DT, Su JC, Phan K, Attia M, Hossny M, Nahavandi S, Lenane P, Moloney FJ, Yazdabadi A. Artificial intelligence in dermatology—where we are and the way to the future: a review. Am J Clin Dermatol 2020; 21(1): 41–47

    Article  Google Scholar 

  12. Maier-Hein L, Vedula SS, Speidel S, Navab N, Kikinis R, Park A, Eisenmann M, Feussner H, Forestier G, Giannarou S, Hashizume M, Katic D, Kenngott H, Kranzfelder M, Malpani A, März K, Neumuth T, Padoy N, Pugh C, Schoch N, Stoyanov D, Taylor R, Wagner M, Hager GD, Jannin P. Surgical data science for next-generation interventions. Nat Biomed Eng 2017; 1(9): 691–696

    Article  Google Scholar 

  13. Udelsman R, Donovan P, Shaw C. Cure predictability during parathyroidectomy. World J Surg 2014; 38(3): 525–533

    Article  Google Scholar 

  14. Harangi B, Hajdu A, Lampe R, Torok P. Recognizing ureter and uterine artery in endoscopic images using a convolutional neural network. In: 2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS). 2017. 726–727. doi: https://doi.org/10.1109/CBMS.2017.137

  15. André B, Vercauteren T, Buchner AM, Wallace MB, Ayache N. Endomicroscopic video retrieval using mosaicing and visualwords. In: 2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. 2010. doi: https://doi.org/10.1109/isbi.2010.5490265

  16. André B, Vercauteren T, Buchner AM, Wallace MB, Ayache N. Learning semantic and visual similarity for endomicroscopy video retrieval. IEEE Trans Med Imaging 2012; 31(6): 1276–1288

    Article  Google Scholar 

  17. André B, Vercauteren T, Perchant A, Buchner A, Wallace M, Ayache N. Endomicroscopic image retrieval and classification using invariant visual features. In: 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. 2009. doi: https://doi.org/10.1109/isbi.2009.5193055

  18. Kohandani Tafresh M, Linard N, André B, Ayache N, Vercauteren T. Semi-automated query construction for content-based endomicroscopy video retrieval. In: Medical Image Computing and Computer-Assisted Intervention — MICCAI 2014. Springer International Publishing, 2014. 89–96. doi: https://doi.org/10.1007/978-3-319-10404-1_12

  19. Gu Y, Yang J, Yang GZ. Multi-view multi-modal feature embedding for endomicroscopy mosaic classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2016. 11–19

  20. Gu Y, Vyas K, Yang J, Yang GZ. Unsupervised feature learning for endomicroscopy image retrieval. In: Medical Image Computing and Computer Assisted Intervention — MICCAI 2017. Springer International Publishing, 2017. 64–71 doi: https://doi.org/10.1007/978-3-319-66179-7_8

  21. Quellec G, Lamard M, Cazuguel G, Droueche Z, Roux C, Cochener B. Real-time retrieval of similar videos with application to computer-aided retinal surgery. Conf Proc IEEE Eng Med Biol Soc 2011; 2011: 4465–4468

    PubMed  Google Scholar 

  22. Ritschel K, Pechlivanis I, Winter S. Brain tumor classification on intraoperative contrast-enhanced ultrasound. Int J CARS 2015; 10 (5): 531–540

    Article  Google Scholar 

  23. Ilunga-Mbuyamba E, Lindner D, Avina-Cervantes J, Arlt F, Rostro-Gonzalez H, Cruz-Aceves I, Chalopin C. Fusion of intraoperative 3D B-mode and contrast-enhanced ultrasound data for automatic identification of residual brain tumors. Appl Sci (Basel) 2017; 7(4): 415

    Article  Google Scholar 

  24. Dollar P, Tu Z, Perona P, Belongie S. Integral channel features. In: Procedings of the British Machine Vision Conference. 2009. doi: https://doi.org/10.5244/c.23.91

  25. Wan S, Sun S, Bhattacharya S, Kluckner S, Gigler A, Simon E, Fleischer M, Charalampaki P, Chen T, Kamen A. Towards an efficient computational framework for guiding surgical resection through intra-operative endo-microscopic pathology. In: Medical Image Computing and Computer-Assisted Intervention — MICCAI 2015. Springer International Publishing, 2015. 421–429. doi: https://doi.org/10.1007/978-3-319-24553-9_52

  26. Kamen A, Sun S, Wan S, Kluckner S, Chen T, Gigler AM, Simon E, Fleischer M, Javed M, Daali S, Igressa A, Charalampaki P. Automatic tissue differentiation based on confocal endomicroscopic images for intraoperative guidance in neurosurgery. BioMed Res Int 2016; 2016: 6183218

    Article  Google Scholar 

  27. Li Y, Charalampaki P, Liu Y, Yang GZ, Giannarou S. Context aware decision support in neurosurgical oncology based on an efficient classification of endomicroscopic data. Int J CARS 2018; 13(8): 1187–1199

    Article  Google Scholar 

  28. Couceiro S, Barreto JP, Freire P, Figueiredo P. Description and classification of confocal endomicroscopic images for the automatic diagnosis of inflammatory bowel disease. In: Machine Learning in Medical Imaging. Springer Berlin Heidelberg, 2012. 144–151. doi: https://doi.org/10.1007/978-3-642-35428-1_18

  29. Halicek M, Lu G, Little JV, Wang X, Patel M, Griffith CC, El-Deiry MW, Chen AY, Fei B. Deep convolutional neural networks for classifying head and neck cancer using hyperspectral imaging. J Biomed Opt 2017; 22(6): 60503

    Article  Google Scholar 

  30. Halicek M, Little JV, Wang X, Patel M, Griffith CC, El-Deiry MW, Chen AY, Fei B. Optical biopsy of head and neck cancer using hyperspectral imaging and convolutional neural networks. Proc SPIE Int Soc Opt Eng 2018; 104690X doi: https://doi.org/10.1117/12.2289023

  31. Fabelo H, Halicek M, Ortega S, Shahedi M, Szolna A, Piñeiro JF, Sosa C, O’Shanahan AJ, Bisshopp S, Espino C, Márquez M, Hernández M, Carrera D, Morera J, Callico GM, Sarmiento R, Fei B. Deep learning-based framework for in vivo identification of glioblastoma tumor using hyperspectral images of human brain. Sensors (Basel) 2019; 19(4): 920

    Article  Google Scholar 

  32. Hou F, Liang Y, Yang Z, Gu W, Yu Y. Automatic identification of metastatic lymph nodes in OCT images. Proceedings Volume 10867, Optical Coherence Tomography and Coherence Domain Optical Methods in Biomedicine XXIII; 108673G. 2019. doi: https://doi.org/10.1117/12.2511588

  33. Tian S, Yin XC, Wang ZB, Zhou F, Hao HW. A VidEo-Based Intelligent Recognition and Decision System for the phacoemulsification cataract surgery. Comput Math Methods Med 2015; 2015: 202934

    Article  Google Scholar 

  34. Fan B, Li HX, Hu Y. An intelligent decision system for intraoperative somatosensory evoked potential monitoring. IEEE Trans Neural Syst Rehabil Eng 2016; 24(2): 300–307

    Article  Google Scholar 

  35. Gordon L, Grantcharov T, Rudzicz F. Explainable artificial intelligence for safe intraoperative decision support. JAMA Surg 2019; 154(11): 1064

    Article  Google Scholar 

  36. Lalys F, Jannin P. Surgical process modelling: a review. Int J CARS 2014; 9(3): 495–511

    Article  Google Scholar 

  37. Krause J, Gulshan V, Rahimy E, Karth P, Widner K, Corrado GS, Peng L, Webster DR. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 2018; 125(8): 1264–1272

    Article  Google Scholar 

  38. Lee H, Yune S, Mansouri M, Kim M, Tajmir SH, Guerrier CE, Ebert SA, Pomerantz SR, Romero JM, Kamalian S, Gonzalez RG, Lev MH, Do S. An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat Biomed Eng 2019; 3(3): 173–182

    Article  Google Scholar 

  39. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017; 542(7639): 115–118

    Article  CAS  Google Scholar 

  40. Safdar NM, Banja JD, Meltzer CC. Ethical considerations in artificial intelligence. Eur J Radiol 2020; 122: 108768

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Surgical Artificial Intelligence and Innovation Laboratory, Massachusetts General Hospital, Boston, MA, 02114, USA

    Allison J. Navarrete-Welton & Daniel A. Hashimoto

  2. Harvard Medical School, Boston, MA, 02114, USA

    Daniel A. Hashimoto

Authors
  1. Allison J. Navarrete-Welton
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel A. Hashimoto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel A. Hashimoto.

Ethics declarations

Allison J. Navarrete-Welton has received research support from Olympus Corporation for projects outside of this paper. Daniel A. Hashimoto is an independent consultant for Verily Life Sciences and the Johnson & Johnson Institute. He serves on the clinical advisory board of Worrell, Inc. He has received research support from Olympus Corporation for projects outside of this paper. This manuscript is a review article and does not involve a research protocol requiring approval by the relevant institutional review board or ethics committee.

Supplementary Material

11684_2020_784_MOESM1_ESM.pdf

List of Papers Found on the Topic of Artificial Intelligence-Based Decision Support for Pre-Operative & Post-Operative Phases of Surgery

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Navarrete-Welton, A.J., Hashimoto, D.A. Current applications of artificial intelligence for intraoperative decision support in surgery. Front. Med. 14, 369–381 (2020). https://doi.org/10.1007/s11684-020-0784-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11684-020-0784-7

Keywords

Search

Navigation