Abstract
Purpose.
Augmented Reality (AR) in Laparoscopic Liver Resection requires anatomical landmarks and the silhouette to be found on the laparoscopic image. They are used to register the preoperative 3D model obtained from CT segmentation. The existing AR systems rely on the surgeon to 1) annotate the landmarks and silhouette and 2) provide an initial registration. These non-trivial tasks require surgeon attention which may perturb the procedure. We propose methods to solve both tasks, hence registration, automatically.
Methods.
The landmarks are the lower ridge and the falciform ligament. We solve 1) by training a U-Net from a new dataset of 1415 labelled images extracted from 68 procedures. We solve 2) by a novel automatic coarse-to-fine pose estimation method, including visibility-reasoning within an iterative robust process. In addition, we propose to divide the ridge into six anatomical sub-parts, making its annotation and use in registration more accurate.
Results.
Our method detects the silhouette with an error equivalent to an experienced surgeon. It detects the ridge and ligament with higher errors owing to under-detection. Nonetheless, our method successfully initialises the registration with tumour target registration errors of 22.4, 14.8 and 7.2 mm for 3 clinical procedures. In comparison, the errors from manual initialisation are 30.5, 15.1 and 16.3 mm.
Conclusion.
Our results are promising, suggesting that we have found an appropriate methodological approach.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bernhardt S, Nicolau SA, Soler L, Doignon C (2017) The status of augmented reality in laparoscopic surgery as of 2016. Med Image Anal 37:66–90
Plantefève R, Haouchine N, Radoux JP, Cotin S (2014) Automatic alignment of pre and intraoperative data using anatomical landmarks for augmented laparoscopic liver surgery. In: Bello F, Cotin S (eds) Biomedical simulation. ISBMS 2014. Lecture Notes in Computer Science, vol 8789. Springer, Cham. https://doi.org/10.1007/978-3-319-12057-7_7
Özgür E, Koo B, Le Roy B, Buc E, Bartoli A (2018) Preoperative liver registration for augmented monocular laparoscopy using backward-forward biomechanical simulation. IJCARS 13(10):1629–1640
Adagolodjo Y, Trivisonne R, Haouchine N, Cotin S, Courtecuisse H (2017) Silhouette-based pose estimation for deformable organs application to surgical augmented reality. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 539–544. https://doi.org/10.1109/IROS.2017.8202205
Espinel Y, Özgür E, Calvet L, Le Roy B, Buc E, Bartoli A (2020) Combining Visual Cues with Interactions for 3D–2D Registration in Liver Laparoscopy. Ann Biomed Eng 48(6):1712–1727
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells W, Frangi A (eds) Medical image computing and computer-assisted intervention–MICCAI 2015. Lecture Notes in Computer Science, vol 9351. Springer, Cham. https://doi.org/10.1007/978-3-319-24574-4_28
François T, Calvet L, Madad Zadeh S, Saboul D, Gasparini S, Samarakoon P, Bourdel N, Bartoli A (2020) Detecting the occluding contours of the uterus to automatise augmented laparoscopy: score, loss, dataset, evaluation and user study. IJCARS 15(7):1177–1186
Zhu J, Styler W, Calloway I (2019) A CNN-based tool for automatic tongue contour tracking in ultrasound images. Preprint at https://arxiv.org/abs/1907.10210
Salehi SSM, Erdogmus D, Gholipour A (2017) Tversky loss function for image segmentation using 3d fully convolutional deep networks. In: Machine learning in medical imaging
Yeung M, Sala E, Schönlieb C-B, Rundo L (2021) Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation. Preprint at https://arxiv.org/abs/2102.04525
Robu MR, Ramalhinho J, Thompson S, Gurusamy K, Davidson B, Hawkes D, Stoyanov D, Clarkson MJ (2018) Global rigid registration of CT to video in laparoscopic liver surgery. IJCARS 13(6):947–956
Koo B, Robu MR, Allam M, Pfeiffer M, Thompson S, Gurusamy K, Davidson B, Speidel S, Hawkes D, Stoyanov D, Clarkson MJ (2022) Automatic, global registration in laparoscopic liver surgery. IJCARS 17:167–176. https://doi.org/10.1007/s11548-021-02518-7
Min Z, Liu L, Meng MQH (2019) Generalized non-rigid point set registration with hybrid mixture models considering anisotropic positional uncertainties. In: Medical image computing and computer assisted intervention–MICCAI 2019. MICCAI 2019. Lecture Notes in Computer Science, vol 11768. Springer, Cham. https://doi.org/10.1007/978-3-030-32254-0_61
Li M, Kambhamettu C, Stone M (2005) Automatic contour tracking in ultrasound images. Clin Linguist Phon 19(6–7):545–554
Espinel Y, Calvet L, Botros K, Buc E, Tilmant C, Bartoli A (2021) Using multiple images and contours for deformable 3D-2D registration of a preoperative CT in laparoscopic liver Surgery. In: Medical image computing and computer assisted intervention–MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science, vol 12904. Springer, Cham. https://doi.org/10.1007/978-3-030-87202-1_63
Rabbani N, Calvet L, Espinel Y, Le Roy B, Ribeiro M, Buc E, Bartoli A (2021) A methodology and clinical dataset with ground-truth to evaluate registration accuracy quantitatively in computer-assisted laparoscopic liver resection. Comput Methods Biomech Biomed Eng Imaging Vis. https://doi.org/10.1080/21681163.2021.1997642
Collins T, Pizarro D, Gasparini S, Bourdel N, Chauvet P, Canis M, Calvet L, Bartoli A (2021) Augmented reality guided laparoscopic surgery of the uterus. IEEE Trans Med Imaging 40(1):371–380
Modrzejewski R (August 2020) Recalage déformable, jeux de données et protocoles d’évaluation pour la chirurgie mini-invasive abdominale augmentée. PhD thesis, Université Clermont Auvergne
Funding
M. Labrunie is supported by a CIFRE PhD fellowship (N\(^{\circ }\)2021/0184) from ANRT under partnership between EnCoV and SurgAR.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no conflicts of interest.
Ethical approval
Our study received ethical approval (IRB00008526-2019-CE58) issued by CPP Sud-Est VI in Clermont-Ferrand, France
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Labrunie, M., Ribeiro, M., Mourthadhoi, F. et al. Automatic preoperative 3d model registration in laparoscopic liver resection. Int J CARS 17, 1429–1436 (2022). https://doi.org/10.1007/s11548-022-02641-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-022-02641-z