Skip to main content

Advertisement

SpringerLink
Log in

Augmented reality in open surgery

  • Review Article
  • Published:
Updates in Surgery Aims and scope Submit manuscript

Abstract

Augmented reality (AR) has been successfully providing surgeons an extensive visual information of surgical anatomy to assist them throughout the procedure. AR allows surgeons to view surgical field through the superimposed 3D virtual model of anatomical details. However, open surgery presents new challenges. This study provides a comprehensive overview of the available literature regarding the use of AR in open surgery, both in clinical and simulated settings. In this way, we aim to analyze the current trends and solutions to help developers and end/users discuss and understand benefits and shortcomings of these systems in open surgery. We performed a PubMed search of the available literature updated to January 2018 using the terms (1) “augmented reality” AND “open surgery”, (2) “augmented reality” AND “surgery” NOT “laparoscopic” NOT “laparoscope” NOT “robotic”, (3) “mixed reality” AND “open surgery”, (4) “mixed reality” AND “surgery” NOT “laparoscopic” NOT “laparoscope” NOT “robotic”. The aspects evaluated were the following: real data source, virtual data source, visualization processing modality, tracking modality, registration technique, and AR display type. The initial search yielded 502 studies. After removing the duplicates and by reading abstracts, a total of 13 relevant studies were chosen. In 1 out of 13 studies, in vitro experiments were performed, while the rest of the studies were carried out in a clinical setting including pancreatic, hepatobiliary, and urogenital surgeries. AR system in open surgery appears as a versatile and reliable tool in the operating room. However, some technological limitations need to be addressed before implementing it into the routine practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Navab N, Traub J, Sielhorst T, Feuerstein M, Bichlmeier C (2007) Action- and workflow-driven augmented reality for computer-aided medical procedures. IEEE Comput Graph 27(5):10–14. https://doi.org/10.1109/Mcg.2007.117

    Article  Google Scholar 

  2. Cutolo F (2017) Augmented reality in image-guided surgery. In: Lee N (ed) Encyclopedia of computer graphics and games. Springer, Cham, pp 1–11. https://doi.org/10.1007/978-3-319-08234-9_78-1

    Chapter  Google Scholar 

  3. Vavra P, Roman J, Zonca P, Ihnat P, Nemec M, Kumar J, Habib N, El-Gendi A (2017) Recent development of augmented reality in surgery: a review. J Healthc Eng. https://doi.org/10.1155/2017/4574172

    Article  PubMed  PubMed Central  Google Scholar 

  4. Cutolo F, Parchi PD, Ferrari V (2014) Video see through AR head-mounted display for medical procedures. In: 2014 IEEE international symposium on mixed and augmented reality (ISMAR), 10–12 Sept 2014, pp 393–396. https://doi.org/10.1109/ismar.2014.6948504

  5. Kersten-Oertel M, Jannin P, Collins DL (2012) DVV: a taxonomy for mixed reality visualization in image guided surgery. IEEE Trans Vis Comput Gr 18(2):332–352. https://doi.org/10.1109/Tvcg.2011.50

    Article  Google Scholar 

  6. Meola A, Cutolo F, Carbone M, Cagnazzo F, Ferrari M, Ferrari V (2017) Augmented reality in neurosurgery: a systematic review. Neurosurg Rev 40(4):537–548. https://doi.org/10.1007/s10143-016-0732-9

    Article  PubMed  Google Scholar 

  7. Azuma RT (1997) A survey of augmented reality. Presence Teleop Virt 6(4):355–385

    Article  Google Scholar 

  8. Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B (2001) Recent advances in augmented reality. IEEE Comput Graph 21(6):34–47. https://doi.org/10.1109/38.963459

    Article  Google Scholar 

  9. Bimber O, Raskar R (2006) Modern approaches to augmented reality. Paper presented at the ACM SIGGRAPH 2006 Courses, Boston, Massachusetts

  10. Billinghurst M, Clark A, Lee G (2015) A survey of augmented reality. Found Trends Hum Comput Interact 8(2–3):73–272. https://doi.org/10.1561/1100000049

    Article  Google Scholar 

  11. Borgmann H, Rodriguez Socarras M, Salem J, Tsaur I, Gomez Rivas J, Barret E, Tortolero L (2017) Feasibility and safety of augmented reality-assisted urological surgery using smartglass. World J Urol 35(6):967–972. https://doi.org/10.1007/s00345-016-1956-6

    Article  CAS  PubMed  Google Scholar 

  12. Sauer IM, Queisner M, Tang P, Moosburner S, Hoepfner O, Horner R, Lohmann R, Pratschke J (2017) Mixed reality in visceral surgery: development of a suitable workflow and evaluation of intraoperative use-cases. Ann Surg 266(5):706–712. https://doi.org/10.1097/SLA.0000000000002448

    Article  PubMed  Google Scholar 

  13. Ntourakis D, Memeo R, Soler L, Marescaux J, Mutter D, Pessaux P (2016) Augmented reality guidance for the resection of missing colorectal liver metastases: an initial experience. World J Surg 40(2):419–426. https://doi.org/10.1007/s00268-015-3229-8

    Article  PubMed  Google Scholar 

  14. Marzano E, Piardi T, Soler L, Diana M, Mutter D, Marescaux J, Pessaux P (2013) Augmented reality-guided artery-first pancreatico-duodenectomy. J Gastrointest Surg 17(11):1980–1983. https://doi.org/10.1007/s11605-013-2307-1

    Article  PubMed  Google Scholar 

  15. Okamoto T, Onda S, Yasuda J, Yanaga K, Suzuki N, Hattori A (2015) Navigation surgery using an augmented reality for pancreatectomy. Dig Surg 32(2):117–123. https://doi.org/10.1159/000371860

    Article  PubMed  Google Scholar 

  16. Onda S, Okamoto T, Kanehira M, Fujioka S, Suzuki N, Hattori A, Yanaga K (2013) Short rigid scope and stereo-scope designed specifically for open abdominal navigation surgery: clinical application for hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci 20(4):448–453. https://doi.org/10.1007/s00534-012-0582-y

    Article  PubMed  Google Scholar 

  17. Onda S, Okamoto T, Kanehira M, Suzuki F, Ito R, Fujioka S, Suzuki N, Hattori A, Yanaga K (2014) Identification of inferior pancreaticoduodenal artery during pancreaticoduodenectomy using augmented reality-based navigation system. J Hepatobiliary Pancreat Sci 21(4):281–287. https://doi.org/10.1002/jhbp.25

    Article  PubMed  Google Scholar 

  18. KleinJan GH, van den Berg NS, van Oosterom MN, Wendler T, Miwa M, Bex A, Hendricksen K, Horenblas S, van Leeuwen FW (2016) Toward (hybrid) navigation of a fluorescence camera in an open surgery setting. J Nucl Med 57(10):1650–1653. https://doi.org/10.2967/jnumed.115.171645

    Article  PubMed  Google Scholar 

  19. van Oosterom MN, Meershoek P, KleinJan GH, Hendricksen K, Navab N, van de Velde CJH, van der Poel HG, van Leeuwen FWB (2018) Navigation of fluorescence cameras during soft tissue surgery—is it possible to use a single navigation setup for various open and laparoscopic urological surgery applications? J Urol 199(4):1061–1068. https://doi.org/10.1016/j.juro.2017.09.160

    Article  PubMed  Google Scholar 

  20. Okamoto T, Onda S, Matsumoto M, Gocho T, Futagawa Y, Fujioka S, Yanaga K, Suzuki N, Hattori A (2013) Utility of augmented reality system in hepatobiliary surgery. J Hepatobiliary Pancreat Sci 20(2):249–253. https://doi.org/10.1007/s00534-012-0504-z

    Article  PubMed  Google Scholar 

  21. Tang R, Ma L, Xiang C, Wang X, Li A, Liao H, Dong J (2017) Augmented reality navigation in open surgery for hilar cholangiocarcinoma resection with hemihepatectomy using video-based in situ three-dimensional anatomical modeling: a case report. Medicine (Baltimore) 96(37):e8083. https://doi.org/10.1097/MD.0000000000008083

    Article  Google Scholar 

  22. Ferrari V, Megali G, Troia E, Pietrabissa A, Mosca F (2009) A 3-D mixed-reality system for stereoscopic visualization of medical dataset. IEEE T Bio Med Eng 56(11):2627–2633. https://doi.org/10.1109/Tbme.2009.2028013

    Article  Google Scholar 

  23. Gavaghan KA, Peterhans M, Oliveira-Santos T, Weber S (2011) A portable image overlay projection device for computer-aided open liver surgery. IEEE Trans Biomed Eng 58(6):1855–1864. https://doi.org/10.1109/TBME.2011.2126572

    Article  PubMed  Google Scholar 

  24. Kersten-Oertel M, Jannin P, Collins DL (2010) DVV: towards a taxonomy for mixed reality visualization in image guided surgery. Med Imaging Augmented Reality 6326:334–343

    Article  Google Scholar 

  25. Cutolo F, Freschi C, Mascioli S, Parchi P, Ferrari M, Ferrari V (2016) Robust and accurate algorithm for wearable stereoscopic augmented reality with three indistinguishable markers. Electronics 5(3):59

    Article  Google Scholar 

  26. Navab N, Heining SM, Traub J (2010) Camera augmented mobile C-Arm (CAMC): calibration, accuracy study, and clinical applications. IEEE T Med Imaging 29(7):1412–1423. https://doi.org/10.1109/Tmi.2009.2021947

    Article  Google Scholar 

  27. Marmulla R, Hoppe H, Muhling J, Eggers G (2005) An augmented reality system for image-guided surgery. Int J Oral Maxillofac Surg 34(6):594–596. https://doi.org/10.1016/j.ijom.2005.05.004

    Article  CAS  PubMed  Google Scholar 

  28. Haouchine N, Dequidt J, Berger MO, Cotin S (2013) Deformation-based augmented reality for hepatic surgery. Stud Health Technol Inf 184:182–188

    Google Scholar 

  29. Peterhans M, vom Berg A, Dagon B, Inderbitzin D, Baur C, Candinas D, Weber S (2011) A navigation system for open liver surgery: design, workflow and first clinical applications. Int J Med Robot Comput Assist Surg MRCAS 7(1):7–16. https://doi.org/10.1002/rcs.360

    Article  CAS  Google Scholar 

  30. Rolland JP, Holloway RL, Fuchs H (1994) A comparison of optical and video see-through head-mounted displays. P Soc Photo Opt Ins 2351:293–307

    Google Scholar 

  31. Cutolo F, Fontana U, Ferrari V (2018) Perspective Preserving Solution for Quasi-Orthoscopic Video See-Through HMDs. Technologies. https://doi.org/10.3390/technologies6010009

    Article  Google Scholar 

  32. Ferrari V, Viglialoro RM, Nicoli P, Cutolo F, Condino S, Carbone M, Siesto M, Ferrari M (2016) Augmented reality visualization of deformable tubular structures for surgical simulation. Int J Med Robot Comput Assist Surg MRCAS 12(2):231–240. https://doi.org/10.1002/rcs.1681

    Article  Google Scholar 

  33. Viglialoro RM, Condino S, Gesi M, Ferrari M, Ferrari V (2014) Augmented reality simulator for laparoscopic cholecystectomy training. In: De Paolis LT, Mongelli A (eds) Augmented and virtual reality. Springer, Cham, pp 428–433. https://doi.org/10.1007/978-3-319-13969-2_33

    Chapter  Google Scholar 

  34. Suenaga H, Tran HH, Liao H, Masamune K, Dohi T, Hoshi K, Takato T (2015) Vision-based markerless registration using stereo vision and an augmented reality surgical navigation system: a pilot study. BMC Med Imaging. https://doi.org/10.1186/s12880-015-0089-5

    Article  PubMed  PubMed Central  Google Scholar 

  35. Wang JC, Suenaga H, Yang LJ, Kobayashi E, Sakuma I (2017) Video see-through augmented reality for oral and maxillofacial surgery. Int J Med Robot Comp. https://doi.org/10.1002/Rcs.1754

    Article  Google Scholar 

  36. Kilgus T, Heim E, Haase S, Prufer S, Muller M, Seitel A, Fangerau M, Wiebe T, Iszatt J, Schlemmer HP, Hornegger J, Yen K, Maier-Hein L (2015) Mobile markerless augmented reality and its application in forensic medicine. Int J Comput Ass Rad 10(5):573–586. https://doi.org/10.1007/s11548-014-1106-9

    Article  Google Scholar 

Download references

Funding

This work was funded by the HORIZON2020 Project VOSTARS, Project ID: 731974. Call: ICT-29-2016—Photonics KET 2016.

Author information

Authors and Affiliations

  1. Department of Information Engineering, University of Pisa, Pisa, Italy

    Benish Fida, Fabrizio Cutolo & Vincenzo Ferrari

  2. Department of Translational Research and New Technologies in Medicine and Surgery, EndoCAS Center, University of Pisa, Pisa, Italy

    Benish Fida, Fabrizio Cutolo, Mauro Ferrari & Vincenzo Ferrari

  3. General Surgery Unit, Department of Surgery, Translational and New Technologies, University of Pisa, Pisa, Italy

    Gregorio di Franco

  4. Vascular Surgery Unit, Cisanello University Hospital AOUP, Pisa, Italy

    Mauro Ferrari

Authors
  1. Benish Fida
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fabrizio Cutolo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gregorio di Franco
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mauro Ferrari
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Vincenzo Ferrari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fabrizio Cutolo.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Research involving human participants and/or animals

The authors confirm that no animal nor human testing were performed in this study.

Informed consent

No informed consent is required.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fida, B., Cutolo, F., di Franco, G. et al. Augmented reality in open surgery. Updates Surg 70, 389–400 (2018). https://doi.org/10.1007/s13304-018-0567-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13304-018-0567-8

Keywords

Search

Navigation