Abstract
Minimally invasive surgery (MIS) has many documented advantages, but the surgeon’s limited visual contact with the scene can be problematic. Hence, systems that can help surgeons navigate, such as a method that can produce a 3D semantic map, can compensate for the limitation above. In theory, we can borrow 3D semantic mapping techniques developed for robotics, but this requires finding solutions to the following challenges in MIS: 1) semantic segmentation, 2) depth estimation, and 3) pose estimation. In this paper, we propose the first 3D semantic mapping system from knee arthroscopy that solves the three challenges above. Using out-of-distribution non-human datasets, where pose could be labeled, we jointly train depth+pose estimators using self-supervised and supervised losses. Using an in-distribution human knee dataset, we train a fully-supervised semantic segmentation system to label arthroscopic image pixels into femur, ACL, and meniscus. Taking testing images from human knees, we combine the results from these two systems to automatically create 3D semantic maps of the human knee. The result of this work opens the pathway to the generation of intra-operative 3D semantic mapping, registration with pre-operative data, and robotic-assisted arthroscopy. Source code: https://github.com/YJonmo/EndoMapNet.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ali, S., Jonmohamadi, Y., Roberts, J., Crawford, R., Carneiro, G., Pandey, A.K.: Arthroscopic multi-spectral scene segmentation using deep learning. arXiv preprint arXiv:2001.05566 (2021)
Ali, S., Jonmohamadi, Y., Takeda, Y., Roberts, J., Crawford, R., Pandey, A.K.: Supervised scene illumination control in stereo arthroscopes for robot assisted minimally invasive surgery. IEEE Sens. J. 4(10), 11577–11587 (2020)
Bae, G., Budvytis, I., Yeung, C.-K., Cipolla, R.: Deep multi-view stereo for dense 3D reconstruction from monocular endoscopic video. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12263, pp. 774–783. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59716-0_74
Chen, R.J., Bobrow, T.L., Athey, T., Mahmood, F., Durr, N.J.: Slam endoscopy enhanced by adversarial depth prediction. arXiv preprint arXiv:1907.00283 (2019)
Da, K.: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 270–279 (2017)
Godard, C., Mac Aodha, O., Firman, M., Brostow, G.J.: Digging into self-supervised monocular depth estimation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3828–3838 (2019)
Grasa, O.G., Bernal, E., Casado, S., Gil, I., Montiel, J.M.M.: Visual SLAM for handheld monocular endoscope. IEEE Trans. Med. Imag. 33(1), 135–146 (2013)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Jaiprakash, A., et al.: Orthopaedic surgeon attitudes towards current limitations and the potential for robotic and technological innovation in arthroscopic surgery. J. Orthop. Surg. 25(1), 2309499016684993 (2017)
Jonmohamadi, Y., et al.: Automatic segmentation of multiple structures in knee arthroscopy using deep learning. IEEE Access 8, 51853–51861 (2020)
Leonard, S., et al.: Evaluation and stability analysis of video-based navigation system for functional endoscopic sinus surgery onin vivoclinical data. IEEE Trans. Med. Imag. 37(10), 2185–2195 (2018)
Liu, F., Jonmohamadi, Y., Maicas, G., Pandey, A.K., Carneiro, G.: Self-supervised Depth Estimation to Regularise Semantic Segmentation in Knee Arthroscopy. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 594–603. Springer (2020). https://doi.org/10.1007/10704282
Liu, X., et al.: Dense depth estimation in monocular endoscopy with self-supervised learning methods. IEEE Trans. Med. Imag. 39(5), 1438–1447 (2019)
Liu, X., et al.: Self-supervised Learning for Dense Depth Estimation in Monocular Endoscopy. arXiv pp. arXiv-1806 (2018)
Liu, X., et al.: Reconstructing sinus anatomy from endoscopic video-towards a radiation-free approach for quantitative longitudinal assessment. arXiv preprint arXiv:2003.08502 (2020)
Ma, R., Wang, R., Pizer, S., Rosenman, J., McGill, S.K., Frahm, J.-M.: Real-time 3D reconstruction of colonoscopic surfaces for determining missing regions. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11768, pp. 573–582. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32254-0_64
Marmol, A., Banach, A., Peynot, T.: Dense-arthroSLAM: Dense intra-articular 3-D reconstruction with robust localization prior for arthroscopy. IEEE Robot. Autom. Lett. 4(2), 918–925 (2019)
Mayer, N., et al.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4040–4048. IEEE, Las Vegas, NV, USA, June 2016. /DOIurl 0.1109/CVPR.2016.438, http://ieeexplore.ieee.org/document/7780807/
Otsu, H., Yamamoto, M., Hachisuka, T.: Reproducing spectral reflectances from tristimulus colours. Comput. Graph. Forum (2018). https://doi.org/10.1111/cgf.13332
Paszke, A., et al.: Gross automatic differentiation in pytorch (2017)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Sharan, L., et al.: Domain gap in adapting self-supervised depth estimation methods for stereo-endoscopy. Curr. Direct. Biomed. Eng. 6(1) (2020)
Ullman, S.: The interpretation of structure from motion. Proc. R. Soc. Lond. Ser B Biol. Sci. 203(1153), 405–426 (1979)
Vijayanarasimhan, S., Ricco, S., Schmid, C., Sukthankar, R., Fragkiadaki, K.: Sfm-net: learning of structure and motion from video. arXiv preprint arXiv:1704.07804 (2017)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Wu, L., et al.: Robotic and image-guided knee arthroscopy. In: Handbook of Robotic and Image-Guided Surgery, pp. 493–514. Elsevier, Amesterdam (2020)
Yang, Z., Wang, P., Wang, Y., Xu, W., Nevatia, R.: Lego: learning edge with geometry all at once by watching videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 225–234 (2018)
Zach, C., Pock, T., Bischof, H.: A globally optimal algorithm for robust tv-l 1 range image integration. In: 2007 IEEE 11th International Conference on Computer Vision, pp. 1–8. IEEE (2007)
Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851–1858 (2017)
Acknowledgements
Supported by AISRF53820 and Australian Research Council through grants DP180103232 and FT190100525.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Jonmohamadi, Y. et al. (2021). 3D Semantic Mapping from Arthroscopy Using Out-of-Distribution Pose and Depth and In-Distribution Segmentation Training. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12902. Springer, Cham. https://doi.org/10.1007/978-3-030-87196-3_36
Download citation
DOI: https://doi.org/10.1007/978-3-030-87196-3_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87195-6
Online ISBN: 978-3-030-87196-3
eBook Packages: Computer ScienceComputer Science (R0)
