Skip to main content

Advertisement

SpringerLink
Log in

Augmented reality navigation for liver surgery: an enhanced coherent point drift algorithm based hybrid optimization scheme

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Augmented reality (AR) based bowel or liver surgery still has not been implemented successfully due to limitations of accurate and proper image registration of uterus and gallbladder during surgery. This research aims to improve target registration error, which helps to navigate through hidden uterus and gallbladder during surgery. Therefore, it will reduce risk of cutting uterus or common bile duct during surgery, which can be fatal and cause devastating effects on the patient. The proposed system integrates the enhanced Coherent Point Drift (CPD) Algorithm with hybrid optimization scheme that incorporates Nelder-Mead simplex and genetic algorithm, to optimize the obtained weight parameter, which in turns improves the target image registration error and processing time of image registration. The system has minimized the target registration error by 0.31 mm in average. It provides a substantial accuracy in terms of target registration error, where the root mean square error is enhanced from 1.28 ± 0.68 mm to 0.97 ± 0.41 mm and improves processing time from 16 ~ 18 ms/frame to 11 ~ 12 ms/frame. The proposed system is focused on improving the accuracy of deformable image registration accuracy of soft tissues and hidden organs, which then helps in proper navigation and localization of the uterus hidden behind bowel and gallbladder hidden behind liver.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Abbreviations

APA:

American Psychological Association

AR:

Augmented Reality

ARGA:

Automatic Region Growing Algorithm

BMA:

Block Matching Algorithm

CPU:

Central Processing Unit

CPD:

Coherent Point Drift

CT:

Computed Tomography

CBCT:

Cone Beam Computer Tomography

GMM:

Gaussian Mixture Model

GUI:

Graphical User Interface

GPU:

Graphics Processing Unit

IG-NS:

Image guided Navigation System

IEEE:

Institute of Electrical and Electronics Engineers

ICP:

Iterative Closet Point

MRI:

Magnetic Resonance Imaging

OMS:

Oral and Maxillofacial Surgery

PQP:

Proximity Query Package

RISC:

Reduced Instruction Set Computer

ROI:

Region of Interest

R-CNN:

Region-Convolutional Neural Network

RMSD:

Root Mean Square Deviation

RMaTV:

Rotational Matrix and Translation Vector

SGBM:

Semi Global Block Matching

SLAM:

Simultaneous Localization and Mapping

SAD:

Sum of Absolute Difference

TLD:

Tracking Learning Detection

References

  1. Azer SA, Azer S (2016) 3D anatomy models and impact on learning: a review of the quality of the literature. Health Prof Educ 2(2):80–98. https://doi.org/10.1016/j.hpe.2016.05.002

    Article  Google Scholar 

  2. Basnet B, Alsadoon A, Withana C, Deva A, Paul M (2018) A novel noise filtered and occlusion removal: navigational accuracy in augmented reality-based constructive jaw surgery. Oral Maxillofac Surg 22(4):385–401. https://doi.org/10.1007/s10006-018-0719-5

    Article  Google Scholar 

  3. Bernhardt S, Nicolau SA, Agnus V, Soler L, Doignon C, Marescaux J (2016) Automatic localization of endoscope in intraoperative CT image: a simple approach to augmented reality guidance in laparoscopic surgery. Med Image Anal 30:130–143. https://doi.org/10.1016/j.media.2016.01.008

    Article  Google Scholar 

  4. Bernhardt S, Nicolau SA, Soler L, Doignon C (2017) The status of augmented reality in laparoscopic surgery as of 2016. Med Image Anal 37:66–90. https://doi.org/10.1016/j.media.2017.01.007

    Article  Google Scholar 

  5. Cash DM, Miga MI, Sinha TK, Galloway RL, Chapman WC (2005) Compensating for intraoperative soft-tissue deformations using incomplete surface data and finite elements. IEEE Trans Med Imaging 24(11):1479–1491. https://doi.org/10.1109/TMI.2005.855434

    Article  Google Scholar 

  6. Chen L, Tang W, John NW, Wan TR, Zhang JJ (2018) SLAM-based dense surface reconstruction in monocular minimally invasive surgery and its application to augmented reality. Comput Methods Prog Biomed 158:135–146. https://doi.org/10.1016/j.cmpb.2018.02.006

    Article  Google Scholar 

  7. Fan Y, Jiang D, Wang M, Song Z (2014) A new markerless patient-to-image registration method using a portable 3D scanner. Med Phys 41(10):101910. https://doi.org/10.1118/1.4895847

    Article  Google Scholar 

  8. Haouchine N, Cotin S, Peterlik I, Dequidt J, Lopez MS, Kerrien E, Berger M-O (2015) Impact of soft tissue heterogeneity on augmented reality for liver surgery. IEEE Trans Vis Comput Graph 21(5):584–597. https://doi.org/10.1109/TVCG.2014.2377772

    Article  Google Scholar 

  9. Kim JH, Bartoli A, Collins T, Hartley R (2012) Tracking by detection for interactive image augmentation in laparoscopy. Biomedical Image Registration:246–255. https://doi.org/10.1007/978-3-642-31340-0_26

  10. Kopelman Y, Lanzafame RJ, Kopelman D (2013) Trends in evolving technologies in the operating room of the future. JSLS : Journal of the Society of Laparoendoscopic Surgeons 17(2):171. https://doi.org/10.4293/108680813X13693422522196

    Article  Google Scholar 

  11. Lee CP, Xu Z, Burke RP, Baucom RB, Poulose BK, Abramson RG, Landman BA (2015) Evaluation of five image registration tools for abdominal CT: pitfalls and opportunities with soft anatomy. Proc SPIE Int Soc Opt Eng 9413. https://doi.org/10.1117/12.2081045

  12. Ma L, Jiang W, Zhang B, Qu X, Ning G, Zhang X, Liao H (2019) Augmented reality surgical navigation with accurate CBCT-patient registration for dental implant placement. Med Biol Eng Comput 57(1):47–57. https://doi.org/10.1007/s11517-018-1861-9

    Article  Google Scholar 

  13. Mahmoud N, Hostettle AR, Collins T, Soler L, Doignon C and Montiel JMM (2017) SLAM based Quasi Dense Reconstruction for Minimally Invasive Surgery Scenes, ICRA 2017 workshop C4 Surgical Robots: Compliant, Continuum, Cognitive, and Collaborativ, https://doi.org/10.1016/j.cmpb.2018.02.006.

  14. Mahmoud N, Collins T, Hostettler A, Soler L, Doignon C, Montiel JMM (2019) Live tracking and dense reconstruction for handheld monocular endoscopy. IEEE Trans Med Imaging 38(1):79–89. https://doi.org/10.1109/TMI.2018.2856109

    Article  Google Scholar 

  15. Meng F, Zhai F, Zeng B, Ding H, Wang G (2018) An automatic markerless registration method for neurosurgical robotics based on an optical camera. Int J Comput Assist Radiol Surg 13(2):253–265. https://doi.org/10.1007/s11548-017-1675-5

    Article  Google Scholar 

  16. Morrison MA, Tam F, Garavaglia MM, Golestanirad L, Hare GM, Cusimano MD, Schweizer TA, Das S, Graham SJA (2016) A novel tablet computer platform for advanced language mapping during awake craniotomy procedures. J Neurosurg 124(4):938–944. https://doi.org/10.3171/2015.4.JNS15312

    Article  Google Scholar 

  17. Mountney P, Fallert J, Nicolau S, Soler L, Mewes PW (2014) An augmented reality framework for soft tissue surgery, vol 8673. Springer Verlag, Berlin, pp 423–431

    Google Scholar 

  18. Murugesan YP, Alsadoon A, Manoranjan P, Prasad PWC (2018) A novel rotational matrix and translation vector algorithm: geometric accuracy for augmented reality in oral and maxillofacial surgeries. Int J Med Rob Comput Assisted Surg 14(3):e1889. https://doi.org/10.1002/rcs.1889

    Article  Google Scholar 

  19. Nicolau S, Soler L, Mutter D, Marescaux J (2011) Augmented reality in laparoscopic surgical oncology. Surg Oncol 20(3):189–201. https://doi.org/10.1016/j.suronc.2011.07.002

    Article  Google Scholar 

  20. Paolis LD, Luca VD (2019) Augmented visualization with depth perception cues to improve the surgeon’s performance in minimally invasive surgery. Med Biol Eng Comput 57(5):995–1013. https://doi.org/10.1007/s11517-018-1929-6

    Article  Google Scholar 

  21. Penza V, Ortiz J, Mattos LS, Forgione A, De ME (2016) Dense soft tissue 3D reconstruction refined with super-pixel segmentation for robotic abdominal surgery. Int J Comput Assist Radiol Surg 11(2):197–206. https://doi.org/10.1007/s11548-015-1276-0

    Article  Google Scholar 

  22. Peterlík I, Courtecuisse H, Rohling R, Abolmaesumi P, Nguanc C, Cotin S, Salcudean S (2018) Fast elastic registration of soft tissues under large deformations. Med Image Anal 45:24–40. https://doi.org/10.1016/j.media.2017.12.006

    Article  Google Scholar 

  23. Peters TM, Linte CA (2016) Image-guided interventions and computer-integrated therapy: quo vadis? Med Image Anal 33:56–63. https://doi.org/10.1016/j.media.2016.06.004

    Article  Google Scholar 

  24. Rusinkiewicz S, Levoy M (2001) Efficient variants of the ICP algorithm, presented at the Proceedings Third International Conference on 3-D Digital Imaging and Modeling, 2001.

  25. Singh P, Alsadoon A, Prasad PWC, Venkata HS, Ali RS, Haddad S, Alrubaie A (2020) A novel navigation algorithm of using the augmented reality to visualize the Hiden organs and internal structure in surgeries. Int J Med Rob Comput Assisted Surg 16(2):1. https://doi.org/10.1002/rcs.2055

    Article  Google Scholar 

  26. Somogyi-Ganss E, Holmes HI, Jokstad A (2015) Accuracy of a novel prototype dynamic computer-assisted surgery system. Clin Oral Implants Res 26(8):882–890. https://doi.org/10.1111/clr.12414

    Article  Google Scholar 

  27. Stoyanov D, Mylonas GP, Lerotic M, Chung AJ, Guang-Zhong Y (2008) Intra-operative visualizations: perceptual Fidelity and human factors. J Disp Technol 4(4):491–501. https://doi.org/10.1109/JDT.2008.926497

    Article  Google Scholar 

  28. Su LM, Vagvolgyi BP, Agarwal R, Reiley CE, Taylor RH, Hager GD (2009) Augmented Reality During Robot-assisted Laparoscopic Partial Nephrectomy: Toward Real-Time 3D-CT to Stereoscopic Video Registration. Urology 73(4):896–900. https://doi.org/10.1016/j.urology.2008.11.040

    Article  Google Scholar 

  29. Tang R, Ma L-F, Rong Z-X, Li M-D, Zeng J-P, Wang X-D, Liao H-E, Dong J-H (2018) Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: a review of current methods. Hepatobiliary Pancreat Dis Int 17(2):101–112. https://doi.org/10.1016/j.hbpd.2018.02.002

    Article  Google Scholar 

  30. Wang P, Qu Z, Gao Y, Shen Z (2011) A refined coherent point drift (CPD) algorithm for point set registration. SCIENCE CHINA Inf Sci 54(12):2639–2646. https://doi.org/10.1007/s11432-011-4465-7

    Article  MathSciNet  Google Scholar 

  31. Wang J, Suenaga H, Yang L, Kobayashi E, Sakuma I (2017) Video see-through augmented reality for oral and maxillofacial surgery. International Journal of Medical Robotics and Computer Assisted Surgery 13(2):n/a–n/a. https://doi.org/10.1002/rcs.1754

    Article  Google Scholar 

  32. Wang R, Geng Z, Zhang Z, Pei R, Meng X (2017) Autostereoscopic augmented reality visualization for depth perception in endoscopic surgery. Displays 48:50–60. https://doi.org/10.1016/j.displa.2017.03.003

    Article  Google Scholar 

  33. Wang J, Shen Y, Yang S (2019) A practical marker-less image registration method for augmented reality oral and maxillofacial surgery. Int J Comput Assist Radiol Surg 14(5):763–773. https://doi.org/10.1007/s11548-019-01921-5

    Article  Google Scholar 

  34. Yasuda J, Okamoto T, Onda S, Futagawa Y, Yanaga K, Suzuki N, Hattori A (2018) Novel navigation system by augmented reality technology using a tablet PC for hepatobiliary and pancreatic surgery. International Journal of Medical Robotics and Computer Assisted Surgery 14(5):n/a–n/a. https://doi.org/10.1002/rcs.1921

    Article  Google Scholar 

  35. Yu F, Song E, Liu H, Li Y, Zhu J, Hung C-C (2018) An augmented reality endoscope system for ureter position detection. J Med Syst 42(8):1–12. https://doi.org/10.1007/s10916-018-0992-8

    Article  Google Scholar 

  36. Zhang X, Wang J, Wang T, Ji X, Shen Y, Sun Z, Zhang X (2019) A markerless automatic deformable registration framework for augmented reality navigation of laparoscopy partial nephrectomy. Int J Comput Assist Radiol Surg 14(8):1285–1294. https://doi.org/10.1007/s11548-019-01974-6

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing and Mathematics, Charles Sturt University (CSU), Wagga Wagga, Australia

    Ramesh Dhoju, Abeer Alsadoon & P. W. C. Prasad

  2. School of Computer Data and Mathematical Sciences, University of Western Sydney (UWS), Sydney, Australia

    Abeer Alsadoon

  3. Monash University, Melbourne, Australia

    Abeer Alsadoon

  4. Information Technology Department, Kent Institute Australia, Sydney, Australia

    Abeer Alsadoon

  5. Information Technology Department, Asia Pacific International College (APIC), Sydney, Australia

    Abeer Alsadoon

  6. Faculty of Information Technology, Applied Science Private University, Amman, Jordan

    Nedhal A. Al-Saiyd

  7. Faculty of Medicine, University of New South Wales, Sydney, Australia

    Ahmad Alrubaie

Authors
  1. Ramesh Dhoju
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abeer Alsadoon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P. W. C. Prasad
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nedhal A. Al-Saiyd
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ahmad Alrubaie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abeer Alsadoon.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dhoju, R., Alsadoon, A., Prasad, P.W.C. et al. Augmented reality navigation for liver surgery: an enhanced coherent point drift algorithm based hybrid optimization scheme. Multimed Tools Appl 80, 28179–28200 (2021). https://doi.org/10.1007/s11042-021-11070-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11070-0

Keywords

Search

Navigation