Skip to main content
SpringerLink
Log in

Attention adaptive instance normalization style transfer for vascular segmentation using deep learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Deep learning models have demonstrated substantial progress in medical image segmentation. However, these models require large datasets for training, which can prove to be clinically difficult. Medical imaging datasets exhibit domain shift problems due to different imaging techniques, scanners, and data privacy issues and, conventional deep neural networks lack generalization capabilities, as their effectiveness decreases with different data distributions. This paper presents a deep learning-based attention-adaptive instance normalization style transfer technique to address the challenges encountered when segmenting blood vessels. The proposed methodology combines adaptive instance normalization style transfer with a dense extreme inception network and convolution block attention module to achieve the best observed vessel segmentation performance. A simple yet effective method is proposed, and it improves the generalization performance of deep neural networks in vascular segmentation. The network is trained on natural images and tested on medical images, thereby overcoming the need for a large dataset or labelled ground truth to train for vessel segmentation. The proposed technique uses experimental results from five distinct medical datasets to demonstrate higher cross-domain generalization capabilities than the state-of-the-art baselines available in the current literature, and the segmentation performance is compared qualitatively and quantitatively with other models. The results demonstrate the feasibility of generalizing our approach to various datasets. This approach overcomes the constraints of traditional deep learning algorithms, which require enormous volumes of medical data along with manually-labelled ground truth. The predictions by the proposed approach are based on natural image training and can be reliably used to detect and identify cardiac and retinal abnormalities without prior medical imaging information.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability and access

All data relevant to this research are available from [26, 40, 41], and [42].

Notes

  1. https://github.com/naoto0804/pytorch-AdaIN

  2. https://github.com/Jongchan/attention-module

References

  1. Shen D, Wu G, Suk H-I (2017) Deep learning in medical image analysis. Annu Rev Biomed Eng 19. https://doi.org/10.1146/annurev-bioeng-071516-044442

  2. Sun L, Li C, Ding X, Huang Y, Chen Z, Wang G, Yu Y, Paisley J (2022) Few-shot medical image segmentation using a global correlation network with discriminative embedding. Comput Biol Med 140:105067. https://doi.org/10.1016/j.compbiomed.2021.105067

    Article  Google Scholar 

  3. Ding H, Sun C, Tang H, Cai D, Yan Y (2023) Few-shot medical image segmentation with cycle-resemblance attention. In: 2023 IEEE/CVF Winter conference on applications of computer vision (WACV), pp 2487–2496. https://doi.org/10.1109/WACV56688.2023.00252

  4. Ouyang C, Biffi C, Chen C, Kart T, Qiu H, Rueckert D (2022) Self-supervised learning for few-shot medical image segmentation. IEEE Trans Med Imaging 41(7):1837–1848. https://doi.org/10.1109/TMI.2022.3150682

    Article  Google Scholar 

  5. Khanal A, Estrada R (2020) Dynamic deep networks for retinal vessel segmentation. In: Frontiers in computer science

  6. Maninis KK, Pont-Tuset J, Arbeláez P, Gool LV (2016) Deep retinal image understanding. In: Medical image computing and computer-assisted intervention (MICCAI)

  7. Gao Z, Wang L, Soroushmehr SMR, Wood A, Gryak J, Nallamothu B, Najarian K (2022) Vessel segmentation for x-ray coronary angiography using ensemble methods with deep learning and filter-based features. BMC Med Imaging 22. https://doi.org/10.1186/s12880-022-00734-4

  8. Moccia S, De Momi E, El Hadji S, Mattos LS (2018) Blood vessel segmentation algorithms — review of methods, datasets and evaluation metrics. Comput Methods Prog Biomed 158:71–91. https://doi.org/10.1016/j.cmpb.2018.02.001

    Article  Google Scholar 

  9. Xu Y, Li Y, Shin B-S (2020) Medical image processing with contextual style transfer. HCIS 10(1):46. https://doi.org/10.1186/s13673-020-00251-9

    Article  Google Scholar 

  10. Tomar D, Bozorgtabar B, Vray MG, Rad MS, Thiran J (2022) Self-supervised generative style transfer for one-shot medical image segmentation. In: 2022 IEEE/CVF Winter conference on applications of computer vision (WACV), pp 1737–1747. https://doi.org/10.1109/WACV51458.2022.00180

  11. Liu Q, Chen C, Qin J, Dou Q, Heng P-A (2021) Feddg: federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. The IEEE/CVF Conference on computer vision and pattern recognition (CVPR)

  12. Li H, Wang Y, Wan R, Wang S, Li T-Q, Kot A (2020) Domain generalization for medical imaging classification with linear-dependency regularization. In: Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H (eds) Advances in neural information processing systems, vol 33, pp 3118–3129

  13. Li D, Yang Y, Song Y, Hospedales TM (2017) Learning to generalize: meta-learning for domain generalization. CoRR abs/1710.03463

  14. Balaji Y, Sankaranarayanan S, Chellappa R (2018) Metareg: towards domain generalization using meta-regularization. In: Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R (eds) Advances in neural information processing systems, vol 31

  15. Zhang L, Wang X, Yang D, Sanford T, Harmon S, Turkbey B, Wood BJ, Roth H, Myronenko A, Xu D, Xu Z (2020) Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation. IEEE Trans Med Imaging 39(7):2531–2540

    Article  Google Scholar 

  16. Yin B, Sun M, Zhang J, Liu W, Liu C, Wang Z (2022) Afa: adversarial frequency alignment for domain generalized lung nodule detection. Neural Comput & Applic 34(10):8039–8050. https://doi.org/10.1007/s00521-022-06928-9

    Article  Google Scholar 

  17. Huang X, Belongie SJ (2017) Arbitrary style transfer in real-time with adaptive instance normalization. 2017 IEEE International conference on computer vision (ICCV), 1510–1519

  18. Cheng M-M, Liu X-C, Wang J, Lu S-P, Lai Y-K, Rosin PL (2020) Structure-preserving neural style transfer. IEEE Trans Image Process 29:909–920. https://doi.org/10.1109/TIP.2019.2936746

    Article  MathSciNet  Google Scholar 

  19. Zhu T, Liu S (2020) Detail-preserving arbitrary style transfer. In: 2020 IEEE International conference on multimedia and expo (ICME), pp 1–6. https://doi.org/10.1109/ICME46284.2020.9102931

  20. Huo Z, Li X, Qiao Y, Zhou P, Wang J (2022) Efficient photorealistic style transfer with multi-order image statistics. Appl Intell 52(11):12533–12545. https://doi.org/10.1007/s10489-021-03154-z

    Article  Google Scholar 

  21. Mulay S, Ram K, Murugesan B, Sivaprakasam M (2021) Style transfer based coronary artery segmentation in x-ray angiogram. In: 2021 IEEE/CVF International conference on computer vision workshops (ICCVW), pp 3386–3394. https://doi.org/10.1109/ICCVW54120.2021.00378

  22. Zhang J, An C, Dai J, Amador M, Bartsch D-U, Borooah S, Freeman WR, Nguyen TQ (2019) Joint vessel segmentation and deformable registration on multi-modal retinal images based on style transfer. In: 2019 IEEE International conference on image processing (ICIP), pp 839–843. https://doi.org/10.1109/ICIP.2019.8802932

  23. Orlando JI, Prokofyeva E, Blaschko MB (2017) A discriminatively trained fully connected conditional random field model for blood vessel segmentation in fundus images. IEEE Trans Biomed Eng 64(1):16–27. https://doi.org/10.1109/TBME.2016.2535311

    Article  Google Scholar 

  24. Galdran A, Anjos A, Dolz J, Chakor H, Lombaert H, Ben Ayed I (2022) State-of-the-art retinal vessel segmentation with minimalistic models. Sci Rep 12:6174. https://doi.org/10.1038/s41598-022-09675-y

    Article  Google Scholar 

  25. Gu J, Tian F, Oh I-S (2022) Retinal vessel segmentation based on self-distillation and implicit neural representation. Appl Intell. https://doi.org/10.1007/s10489-022-04252-2

    Article  Google Scholar 

  26. Hao D, Ding S, Qiu L, Lv Y, Fei B, Zhu Y, Qin B (2020) Sequential vessel segmentation via deep channel attention network. Neural Dev 128:172–187. https://doi.org/10.1016/j.neunet.2020.05.005

    Article  Google Scholar 

  27. Qin B, Mao H, Liu Y, Zhao J, Lv Y, Zhu Y, Ding S, Chen X (2022) Robust pca unrolling network for super-resolution vessel extraction in x-ray coronary angiography. IEEE Trans Med Imaging

  28. Iyer K, Najarian CP, Fattah AA, Arthurs CJ, Soroushmehr SMR, Subban V, Sankardas MA, Nadakuditi RR, Nallamothu BK, Figueroa CA (2021) Angionet: a convolutional neural network for vessel segmentation in x-ray angiography. medRxiv. https://doi.org/10.1101/2021.01.25.21250488

  29. Tian F, Gao Y, Fang Z, Gu J (2021) Automatic coronary artery segmentation algorithm based on deep learning and digital image processing. Appl Intell 51(12):8881–8895. https://doi.org/10.1007/s10489-021-02197-6

    Article  Google Scholar 

  30. Yuan Q, Wei Y, Meng X, Shen H, Zhang L (2018) A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 11(3):978–989. https://doi.org/10.1109/JSTARS.2018.2794888

    Article  Google Scholar 

  31. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: 3rd International conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference track proceedings

  32. Woo S, Park J, Lee J-Y, Kweon I-S (2018) Cbam: convolutional block attention module. In: ECCV

  33. Soria Poma X, Riba E, Sappa A (2020) Dense extreme inception network: towards a robust cnn model for edge detection, pp 1912–1921. https://doi.org/10.1109/WACV45572.2020.9093290

  34. Zhang T, Gao Y, Gao F, Qi L, Dong J (2021) Arbitrary style transfer with parallel self-attention. In: 2020 25th International conference on pattern recognition (ICPR), pp 1406–1413. https://doi.org/10.1109/ICPR48806.2021.9412049

  35. Arbeláez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell 33:898–916

    Article  Google Scholar 

  36. WikiArt.org - Visual Art Encyclopedia (2023). https://www.wikiart.org/

  37. He K, Sun J (2015) Fast guided filter. CoRR

  38. Petro AB, Sbert C, Morel J-M (2014) Multiscale retinex. Image processing on line, 71–88. https://doi.org/10.5201/ipol.2014.107

  39. Hassanpour H, Samadiani N, Salehi S (2015) Using morphological transforms to enhance the contrast of medical images. The Egyptian Journal of Radiology and Nuclear medicine 46:481–489

    Article  Google Scholar 

  40. Budai A, Bock R, Maier A, Hornegger J, Michelson G (2013) Robust vessel segmentation in fundus images. Int J Biomed Imaging 2013:154860. https://doi.org/10.1155/2013/154860

    Article  Google Scholar 

  41. Staal J, Abramoff MD, Niemeijer M, Viergever MA, van Ginneken B (2004) Ridge-based vessel segmentation in color images of the retina. IEEE Trans Med Imaging 23(4):501–509. https://doi.org/10.1109/TMI.2004.825627

    Article  Google Scholar 

  42. Alipour SHM, Rabbani H, Akhlaghi MR (2012) Diabetic retinopathy grading by digital curvelet transform. Computational and Mathematical Methods in Medicine 2012

  43. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. International Conference on Learning Representations

  44. Qin B, Mao H, Liu Y, Zhao J, Lv Y, Zhu YY, Ding S, Chen X (2022) Robust pca unrolling network for super-resolution vessel extraction in x-ray coronary angiography. IEEE Trans Med Imaging 41:3087–3098

    Article  Google Scholar 

  45. Ding L, Bawany MH, Kuriyan AE, Ramchandran RS, Wykoff CC, Sharma G (2020) A novel deep learning pipeline for retinal vessel detection in fluorescein angiography. IEEE Trans Image Process 29:6561–6573. https://doi.org/10.1109/TIP.2020.2991530

    Article  Google Scholar 

  46. Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9(3):81–84. https://doi.org/10.1109/97.995823

    Article  Google Scholar 

  47. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  48. Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Process 15(2):430–444. https://doi.org/10.1109/TIP.2005.859378

    Article  Google Scholar 

  49. Zhou J, Civco DL, Silander JA (1998) A wavelet transform method to merge landsat tm and spot panchromatic data. Int J Remote Sens 19(4):743–757. https://doi.org/10.1080/014311698215973

    Article  Google Scholar 

  50. An J, Huang S, Song Y, Dou D, Liu W, Luo J (2021) Artflow: unbiased image style transfer via reversible neural flows. 2021 IEEE/CVF Conference on computer vision and pattern recognition (CVPR), 862–871

  51. Roy S, Mitra A, Roy S, Setua SK (2019) Blood vessel segmentation of retinal image using clifford matched filter and clifford convolution. Multimedia Tools and Applications 78:34839–34865. https://doi.org/10.1007/s11042-019-08111-0

    Article  Google Scholar 

  52. Mahapatra S, Agrawal S, Mishro PK, Pachori RB (2022) A novel framework for retinal vessel segmentation using optimal improved frangi filter and adaptive weighted spatial fcm. Comput Biol Med 147:105770. https://doi.org/10.1016/j.compbiomed.2022.105770

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank Prof. Shantanu Mulay for providing their valuable comments and suggestions.

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Indian Institute of Technology Madras, Chennai, India

    Supriti Mulay & Mohanasankar Sivaprakasam

  2. Healthcare Technology Innovation Centre, Chennai, India

    Supriti Mulay, Keerthi Ram & Mohanasankar Sivaprakasam

Authors
  1. Supriti Mulay
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Keerthi Ram
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohanasankar Sivaprakasam
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Supriti Mulay: Conceptualization, Methodology, Software, Validation, Writing - original draft. Keerthi Ram: Conceptualization, Writing - review & editing, Resources. Mohanasankar Sivaprakasam: Supervision, Project administration, Funding acquisition.

Corresponding author

Correspondence to Supriti Mulay.

Ethics declarations

Ethics declarations

This article does not contain any studies with human participants and/or animals performed by any of the authors.

Competing Interest

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or nonfinancial interest in the subject matter or materials discussed in this manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mulay, S., Ram, K. & Sivaprakasam, M. Attention adaptive instance normalization style transfer for vascular segmentation using deep learning. Appl Intell 53, 29638–29655 (2023). https://doi.org/10.1007/s10489-023-05033-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05033-1

Keywords

Search

Navigation