Skip to main content
SpringerLink
Log in

Less-than-One Shot 3D Segmentation Hijacking a Pre-trained Space-Time Memory Network

  • Conference paper
  • First Online:
Advanced Concepts for Intelligent Vision Systems (ACIVS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14124))

  • 200 Accesses

Abstract

In this paper, we propose a semi-supervised setting for semantic segmentation of a whole volume from only a tiny portion of one slice annotated using a memory-aware network pre-trained on video object segmentation without additional fine-tuning. The network is modified to transfer annotations of one partially annotated slice to the whole slice, then to the whole volume. This method discards the need for training the model. Applied to Electron Tomography, where manual annotations are time-consuming, it achieves good segmentation results considering a labeled area of only a few percent of a single slice. The source code is available at https://github.com/licyril1403/hijacked-STM.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Akers, S., et al.: Rapid and flexible segmentation of electron microscopy data using few-shot machine learning. NPJ Computat. Mater. 7(1), 1–9 (2021)

    Google Scholar 

  2. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: semantic image segmentation with deep convolutional nets, Atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2018)

    Article  Google Scholar 

  3. Cheng, H.K., Tai, Y.W., Tang, C.K.: Modular interactive video object segmentation: interaction-to-mask, propagation and difference-aware fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5559–5568 (2021)

    Google Scholar 

  4. Cheng, H.K., Tai, Y.W., Tang, C.K.: Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Adv. Neural. Inf. Process. Syst. 34, 11781–11794 (2021)

    Google Scholar 

  5. Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-net: learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 424–432. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_49

    Chapter  Google Scholar 

  6. Dosovitskiy, A., Springenberg, J.T., Riedmiller, M., Brox, T.: Discriminative unsupervised feature learning with convolutional neural networks. Adv. Neural Inf. Process. Syst. 27 (2014)

    Google Scholar 

  7. Ersen, O., et al.: 3D-TEM characterization of nanometric objects. Solid State Sci. 9(12), 1088–1098 (2007)

    Article  Google Scholar 

  8. Evin, B., et al.: 3D analysis of helium-3 nanobubbles in palladium aged under tritium by electron tomography. J. Phys. Chem. C 125(46), 25404–25409 (2021)

    Article  Google Scholar 

  9. Fernandez, J.J.: Computational methods for electron tomography. Micron 43(10), 1010–1030 (2012)

    Article  Google Scholar 

  10. Flores, C., et al.: Versatile roles of metal species in carbon nanotube templates for the synthesis of metal-zeolite nanocomposite catalysts. ACS Appl. Nano Mater. 2(7), 4507–4517 (2019)

    Article  Google Scholar 

  11. Genc, A., Kovarik, L., Fraser, H.L.: A deep learning approach for semantic segmentation of unbalanced data in electron tomography of catalytic materials. arXiv preprint arXiv:2201.07342 (2022)

  12. Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: 2006 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 2, pp. 1735–1742 (2006)

    Google Scholar 

  13. He, W., Ladinsky, M.S., Huey-Tubman, K.E., Jensen, G.J., McIntosh, J.R., Björkman, P.J.: FcRn-mediated antibody transport across epithelial cells revealed by electron tomography. Nature 455(7212), 542–546 (2008)

    Article  Google Scholar 

  14. Khadangi, A., Boudier, T., Rajagopal, V.: EM-net: deep learning for electron microscopy image segmentation. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 31–38 (2021)

    Google Scholar 

  15. Li, C., Ducottet, C., Desroziers, S., Moreaud, M.: Toward few pixel annotations for 3D segmentation of material from electron tomography. In: International Conference on Computer Vision Theory and Applications, VISAPP 2023 (2023)

    Google Scholar 

  16. Liu, Q., Xu, Z., Jiao, Y., Niethammer, M.: iSegFormer: interactive segmentation via transformers with application to 3D knee MR images. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022. LNCS, vol. 13435, pp. 464–474. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16443-9_45

    Chapter  Google Scholar 

  17. Mahadevan, S., Voigtlaender, P., Leibe, B.: Iteratively trained interactive segmentation. In: British Machine Vision Conference (BMVC) (2018)

    Google Scholar 

  18. Medeiros-Costa, I.C., Laroche, C., Pérez-Pellitero, J., Coasne, B.: Characterization of hierarchical zeolites: combining adsorption/intrusion, electron microscopy, diffraction and spectroscopic techniques. Microporous Mesoporous Mater. 287, 167–176 (2019)

    Article  Google Scholar 

  19. Milletari, F., Navab, N., Ahmadi, S.A.: V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571 (2016)

    Google Scholar 

  20. Oh, S.W., Lee, J.Y., Xu, N., Kim, S.J.: Video object segmentation using space-time memory networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9226–9235 (2019)

    Google Scholar 

  21. Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2009)

    Article  Google Scholar 

  22. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  23. Sukhbaatar, S., Weston, J., Fergus, R., et al.: End-to-end memory networks. Adv. Neural Inf. Process. Syst. 28 (2015)

    Google Scholar 

  24. Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703 (2019)

    Google Scholar 

  25. Tran, V.D., Moreaud, M., Thiébaut, É., Denis, L., Becker, J.M.: Inverse problem approach for the alignment of electron tomographic series. Oil Gas Sci. Technol.-Rev. d’IFP Energies Nouvelles 69(2), 279–291 (2014)

    Article  Google Scholar 

  26. Volkmann, N.: Methods for segmentation and interpretation of electron tomographic reconstructions. Methods Enzymol. 483, 31–46 (2010)

    Article  Google Scholar 

  27. Wang, H., Jiang, X., Ren, H., Hu, Y., Bai, S.: SwiftNet: real-time video object segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1296–1305 (2021)

    Google Scholar 

  28. Wurm, M., Stark, T., Zhu, X.X., Weigand, M., Taubenböck, H.: Semantic segmentation of slums in satellite images using transfer learning on fully convolutional neural networks. ISPRS J. Photogramm. Remote. Sens. 150, 59–69 (2019)

    Article  Google Scholar 

  29. Zhao, X., et al.: Contrastive learning for label efficient semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10623–10633 (2021)

    Google Scholar 

  30. Zhou, T., Li, L., Bredell, G., Li, J., Konukoglu, E.: Quality-aware memory network for interactive volumetric image segmentation. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12902, pp. 560–570. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87196-3_52

    Chapter  Google Scholar 

  31. Zhou, T., Li, L., Bredell, G., Li, J., Unkelbach, J., Konukoglu, E.: Volumetric memory network for interactive medical image segmentation. Med. Image Anal. 83, 102599 (2023)

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the LABEX MILYON (ANR-10-LABX-0070) of Université de Lyon, within the program “Investissements d’Avenir” (ANR-11-IDEX- 0007) operated by the French National Research Agency (ANR).

Author information

Authors and Affiliations

  1. Université Jean Monnet Saint-Etienne, CNRS, Institut d Optique Graduate School, Laboratoire Hubert Curien UMR 5516, 42023, Saint-Etienne, France

    Cyril Li & Christophe Ducottet

  2. Manufacture Française des Pneumatiques Michelin, 23 Place des Carmes Déchaux, 63000, Clermont-Ferrand, France

    Sylvain Desroziers

  3. IFP Energies nouvelles, Rond-point de l’échangeur de Solaize BP 3, 69360, Solaize, France

    Maxime Moreaud

Authors
  1. Cyril Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christophe Ducottet
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sylvain Desroziers
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maxime Moreaud
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cyril Li .

Editor information

Editors and Affiliations

  1. DGA TA, Toulouse, France

    Jaques Blanc-Talon

  2. University of Auckland, Auckland, New Zealand

    Patrice Delmas

  3. Ghent University, Ghent, Belgium

    Wilfried Philips

  4. University of Antwerp, Wilrijk, Belgium

    Paul Scheunders

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, C., Ducottet, C., Desroziers, S., Moreaud, M. (2023). Less-than-One Shot 3D Segmentation Hijacking a Pre-trained Space-Time Memory Network. In: Blanc-Talon, J., Delmas, P., Philips, W., Scheunders, P. (eds) Advanced Concepts for Intelligent Vision Systems. ACIVS 2023. Lecture Notes in Computer Science, vol 14124. Springer, Cham. https://doi.org/10.1007/978-3-031-45382-3_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-45382-3_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-45381-6

  • Online ISBN: 978-3-031-45382-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation