Skip to main content
SpringerLink
Log in

Analysis of multimodal fusion strategies in deep learning for ischemic stroke lesion segmentation on computed tomography perfusion data

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Stroke poses a significant risk to human life. Segmenting and immediately treating the stroke core stops its further development, therefore, enhancing the likelihood of survival. Convolutional neural networks (CNN) have been very successful in medical image segmentation, namely in the field of deep learning, and have produced the most advanced outcomes. Multi-modal images provide superior outcomes in the segmentation of stroke lesions compared to single-modal images. The integration of input from several modalities at various levels is crucial in determining performance and producing diverse outcomes in deep learning models that use multimodalities. Further investigation is required to explore the optimal methods for processing multimodal data in CNNs, the influence of fusion on CNN learning, and the effect of fusion strategies on lesions of varying sizes. To examine the impact of a multi-modal fusion method on lesion segmentation, we assessed four models using distinct fusion techniques, including early, late, bottleneck, and hierarchical fusions. This study discusses the various fusion procedures used in segmenting the lesion using computed tomography perfusion data. In addition, both quantitative and qualitative assessments, including deep feature analysis and feature similarity, were conducted to assess the impact of the fusion technique on the model’s performance. Furthermore, we examined the influence of fusion techniques on the size of the lesion. In addition, we analyzed the advantages and disadvantages of several multi-modal fusion systems. Our findings demonstrate that the bottleneck fusion technique got the highest dice score, 0.582, on the Ischemic Stroke Lesion Segmentation 2018 validation data as a result of its capacity to construct complex relationships across several modalities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

The ISLES 2018 dataset, which was used in the experiments, is available at https://www.isles-challenge.org/ISLES2018/

References

  1. Abulnaga SM, Rubin J (2019) Ischemic stroke lesion segmentation in ct perfusion scans using pyramid pooling and focal loss. In: Bakas S, Kuijf H, Keyvan F, Reyes M, van Walsum T (eds) Crimi A. Glioma multiple sclerosis stroke and traumatic brain injuries springer international publishing. Cham, Brainlesion, pp 352–363

  2. Al Jowair H, Alsulaiman M, Muhammad G (2023) Multi parallel u-net encoder network for effective polyp image segmentation. Image Vis Comput 137:104767

    Article  Google Scholar 

  3. Alshehri F, Muhammad G (2023) A few-shot learning-based ischemic stroke segmentation system using weighted mri fusion. Image Vis Comput 140:104865

    Article  Google Scholar 

  4. Anand V.K, Khened M, Alex V, Krishnamurthi G, (2019) Fully automatic segmentation for ischemic stroke using ct perfusion maps In: Crimi A, Bakas S, Kuijf H, Keyvan F, Reyes M, van Walsum T (Eds.) Brainlesion: Glioma Multiple Sclerosis stroke and traumatic brain injuries springer international publishing. Cham, pp 328–334

  5. Bertels J, Robben D, Vandermeulen D, Suetens P (2019) Contra-lateral information CNN for core lesion segmentation based on native CTP in acute stroke, vol 11383. Springer International Publishing, LNCS. https://doi.org/10.1007/978-3-030-11723-8_26

  6. Chen L, Bentley P, Rueckert D (2017) Fully automatic acute ischemic lesion segmentation in dwi using convolutional neural networks. Neuroimage Clin 15:633–643. https://doi.org/10.1016/j.nicl.2017.06.016

    Article  Google Scholar 

  7. Chen Y, Chen J, Wei D, Li Y, Zheng Y (2020) Octopusnet: A deep learning segmentation network for multi-modal medical images In: Li Q, Leahy R, Dong B, Li X (Eds.) Multiscale multimodal medical imaging springer international publishing. Cham. pp 17–25

  8. Clèrigues A, Valverde S, Bernal J, Freixenet J, Oliver A, Lladó X (2019) Acute ischemic stroke lesion core segmentation in CT perfusion images using fully convolutional neural networks. Comput Biol Med 115. https://doi.org/10.1016/j.compbiomed.2019.103487

  9. Dolz J, Ben Ayed I, Desrosiers C (2019) Dense multi-path u-net for ischemic stroke lesion segmentation in multiple image modalities. In: Bakas S, Kuijf H, Keyvan F, Reyes M, van Walsum T (eds) Crimi A. Glioma Multiple Sclerosis Stroke and Traumatic Brain Injuries Springer International Publishing. Cham, Brainlesion, pp 271–282

  10. Ghnemat R, Khalil A, Abu Al-Haija Q (2023) Ischemic Stroke Lesion Segmentation Using Mutation Model and Generative Adversarial Network. Electronics (Switzerland) 12. https://doi.org/10.3390/electronics12030590

  11. Huff DT, Weisman AJ, Jeraj R (2021) Interpretation and visualization techniques for deep learning models in medical imaging. Phys Med Biol 66. https://doi.org/10.1088/1361-6560/abcd17

  12. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift In: Proc. Int. Conf. Mach. Learn, JMLR.org, pp 448-456

  13. Islam M, Nooruddin S, Karray F, Muhammad G (2023) Internet of things: Device capabilities architectures protocols and smart applications in healthcare domain. IEEE Internet Things J 10(4):3611–3641

    Article  Google Scholar 

  14. Islam M, Nooruddin S, Karray F, Muhammad G (2023) Multi-level feature fusion for multimodal human activity recognition in internet of healthcare things. Inf Fusion 94:17–31

    Article  Google Scholar 

  15. Islam M, Vaidyanathan NR, Jose VJM, Ren H (2019) Ischemic stroke lesion segmentation using adversarial learning In: Crimi A, Bakas S, Kuijf H, Keyvan F, Reyes M, van Walsum T (Eds.) Brainlesion: glioma multiple sclerosis stroke and traumatic brain injuries springer international publishing. Cham, pp 292–300

  16. Kamnitsas K, Ledig C, Newcombe VF, Simpson JP, Kane AD, Menon DK, Rueckert D, Glocker B (2017) Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Med Image Anal 36:61–78. https://doi.org/10.1016/j.media.2016.10.004

    Article  Google Scholar 

  17. Khurana S, Gourie-Devi M, Sharma S, Kushwaha S (2021) Burden of stroke in India during 1960 to 2018: a systematic review and meta-analysis of community based surveys. Neurol India 69:547

    Article  Google Scholar 

  18. Kumar A, Ghosal P, Kundu SS, Mukherjee A, Nandi D (2022) A lightweight asymmetric u-net framework for acute ischemic stroke lesion segmentation in ct and ctp images. Computer Methods and Programs in Biomedicine 226. https://doi.org/10.1016/j.cmpb.2022.107157

  19. Li J, Yu ZL, Gu Z, Liu H, Li Y (2019) Mman: Multi-modality aggregation network for brain segmentation from mr images. Neurocomputing 358:10–19. https://doi.org/10.1016/j.neucom.2019.05.025

    Article  Google Scholar 

  20. Liu L, Yang S, Meng L, Li M, Wang J (2019) Multi-scale deep convolutional neural network for stroke lesions segmentation on ct images. In: Bakas S, Kuijf H, Keyvan F, Reyes M, van Walsum T (eds) Crimi A. Glioma Multiple Sclerosis Stroke and Traumatic Brain Injuries Springer International Publishing. Cham, Brainlesion, pp 283–291

  21. Liu R, Pu W, Zou Y, Jiang L, Ye Z (2022) Pool-unet: Ischemic stroke segmentation from ct perfusion scans using poolformer unet In: 2022 6th Asian conference on artificial intelligence technology (ACAIT) IEEE, pp 1–6. https://ieeexplore.ieee.org/document/10137834/. https://doi.org/10.1109/ACAIT56212.2022.10137834

  22. Liu Z, Cao C, Ding S, Liu Z, Han T, Liu S (2018) Towards clinical diagnosis: Automated stroke lesion segmentation on multi-spectral mr image using convolutional neural network. IEEE Access 6:57006–57016

    Article  Google Scholar 

  23. Lv W, Ashrafinia S, Ma J, Lu L, Rahmim A (2019) Multi-level multi-modality fusion radiomics: application to pet and ct imaging for prognostication of head and neck cancer. IEEE J Biomed Health Inform 24:2268–2277

    Article  Google Scholar 

  24. Maier O, Menze BH, von der Gablentz J, Häni L, Heinrich MP, Liebrand M, Winzeck S, Basit A, Bentley P, Chen L, Christiaens D, Dutil F, Egger K, Feng C, Glocker B, Götz M, Haeck T, Halme HL, Havaei M, Iftekharuddin KM, Jodoin PM, Kamnitsas K, Kellner E, Korvenoja A, Larochelle H, Ledig C, Lee JH, Maes F, Mahmood Q, Maier-Hein KH, McKinley R, Muschelli J, Pal C, Pei L, Rangarajan JR, Reza SM, Robben D, Rueckert D, Salli E, Suetens P, Wang CW, Wilms M, Kirschke JS, Krämer UM, Münte TF, Schramm P, Wiest R, Handels H, Reyes M (2017) Isles 2015 - a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri. Med Image Anal 35:250–269. https://doi.org/10.1016/j.media.2016.07.009

    Article  Google Scholar 

  25. Mair G, Wardlaw J (2014) Imaging of acute stroke prior to treatment: current practice and evolving techniques. Br J Radiol 87:20140216

    Article  Google Scholar 

  26. Mittermeier A, Reidler P, Fabritius MP, Schachtner B, Wesp P, Ertl-Wagner B, Dietrich O, Ricke J, Kellert L, Tiedt S, Kunz WG, Ingrisch M (2022) End-to-end deep learning approach for perfusion data: A proof-of-concept study to classify core volume in stroke ct. Diagnostics 12. https://doi.org/10.3390/diagnostics12051142

  27. Muhammad G, Alshehri F, Karray F et al (2021) A comprehensive survey on multimodal medical signals fusion for smart healthcare systems. Inf Fusion 76:355–375

    Article  Google Scholar 

  28. Omarov B, Tursynova A, Postolache O, Gamry K, Batyrbekov A, Aldeshov S, Azhibekova Z, Nurtas M, Aliyeva A, Shiyapov K (2022) Modified unet model for brain stroke lesion segmentation on computed tomography images. Comput Mater Contin 71:4701–4717https://doi.org/10.32604/cmc.2022.020998

  29. Pinheiro GR, Voltoline R, Bento M, Rittner L (2019) V-net and u-net for ischemic stroke lesion segmentation in a small dataset of perfusion data, vol 11383. Springer International Publishing, LNCS. https://doi.org/10.1007/978-3-030-11723-8_30

  30. Raju CSP, Kirupakaran AM, Neelapu BC, Laskar RH (2022) Ischemic stroke lesion segmentation in ct perfusion images using u-net with group convolutions In: International Conference on Computer Vision and Image Processing. Springer, pp 276–288

  31. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation In: Navab N, Hornegger J, Wells WM, Frangi AF (Eds.) Medical image computing and computer-assisted intervention – MICCAI 2015 Springer International Publishing. Cham, pp 234–241

  32. de la Rosa E, Sima DM, Menze B, Kirschke JS, Robben D (2021) AIFNet: Automatic vascular function estimation for perfusion analysis using deep learning. Med Image Anal 74:102211. https://doi.org/10.1016/j.media.2021.102211

    Article  Google Scholar 

  33. Shi T, Jiang H, Zheng B (2021) C2MA-Net: Cross-modal cross-attention network for acute ischemic stroke lesion segmentation based on CT perfusion scans. IEEE Trans Biomed Eng 69:108–118. https://doi.org/10.1109/TBME.2021.3087612

    Article  Google Scholar 

  34. Soltanpour M, Greiner R, Boulanger P, Buck B (2019) Ischemic stroke lesion prediction in ct perfusion scans using multiple parallel u-nets following by a pixel-level classifier In: In Proc. Int. Conf. BIBE IEEE Computer Society. Los Alamitos CA USA. pp 957–963. https://doi.org/10.1109/BIBE.2019.00179

  35. Soltanpour M, Greiner R, Boulanger P, Buck B (2021) Improvement of automatic ischemic stroke lesion segmentation in ct perfusion maps using a learned deep neural network. Comput Biol Med 137:104849. https://doi.org/10.1016/j.compbiomed.2021.104849

    Article  Google Scholar 

  36. Song T (2019) Generative model-based ischemic stroke lesion segmentation. https://doi.org/10.48550/ARXIV.1906.02392

  37. Song T, Huang N (2019) Integrated extractor generator and segmentor for ischemic stroke lesion segmentation. In: Bakas S, Kuijf H, Keyvan F, Reyes M, van Walsum T (eds) Crimi A. Glioma multiple sclerosis stroke and traumatic brain injuries springer international publishing. Cham, Brainlesion, pp 310–318

  38. Tureckova A, Rodríguez-Sánchez AJ (2019) Isles challenge: U-shaped convolution neural network with dilated convolution for 3d stroke lesion segmentation. In: Bakas S, Kuijf H, Keyvan F, Reyes M, van Walsum T (eds) Crimi A. Glioma multiple sclerosis stroke and traumatic brain injuries springer international publishing. Cham, Brainlesion, pp 319–327

  39. Vries LD, Emmer BJ, Majoie CB, Marquering HA, Gavves E (2023) Perfu-net: Baseline infarct estimation from ct perfusion source data for acute ischemic stroke. Med Image Anal 85. https://doi.org/10.1016/j.media.2023.102749

  40. Wang G, Song T, Dong Q, Cui M, Huang N, Zhang S (2020) Automatic ischemic stroke lesion segmentation from computed tomography perfusion images by image synthesis and attention-based deep neural networks. Med Image Anal 65:101787. https://doi.org/10.1016/j.media.2020.101787

    Article  Google Scholar 

  41. Wang Y, Katsaggelos AK, Wang X, Parrish TB (2016) A deep symmetry convnet for stroke lesion segmentation In: In Proc. Int. Conf. Image. Proc, pp 111–115. https://doi.org/10.1109/ICIP.2016.7532329

  42. Yang HY (2019) Volumetric adversarial training for ischemic stroke lesion segmentation. In: Bakas S, Kuijf H, Keyvan F, Reyes M, van Walsum T (eds) Crimi A. Glioma multiple sclerosis stroke and traumatic brain injuries springer international publishing. Cham, Brainlesion, pp 343–351

  43. Zhang Y, Sidibé D, Morel O, Mériaudeau F (2021) Deep multimodal fusion for semantic image segmentation: A survey. Image Vision Comput 105. https://doi.org/10.1016/j.imavis.2020.104042

  44. Zhou T, Ruan S, Canu S (2019) A review: Deep learning for medical image segmentation using multi-modality fusion. Array 3–4:100004. https://doi.org/10.1016/j.array.2019.100004

    Article  Google Scholar 

  45. Zhou T, Ruan S, Guo Y, Canu S (2020) A multi-modality fusion network based on attention mechanism for brain tumor segmentation In: In Proc. IEEE Int. Symp. Biomed. Imaging, pp 377–380. https://doi.org/10.1109/ISBI45749.2020.9098392

  46. Zhu H, Chen Y, Tang T, Ma G, Zhou J, Zhang J, Lu S, Wu F, Luo L, Liu S, Ju S, Shi H (2022) Isp-net: Fusing features to predict ischemic stroke infarct core on ct perfusion maps. Comput. Methods. Programs. Biomed 215:106630. https://doi.org/10.1016/j.cmpb.2022.106630

    Article  Google Scholar 

Download references

Funding

This work was supported by the Researchers Supporting Project number (RSP2024R34), King Saud University, Riyadh, Saudi Arabia.

Author information

Authors and Affiliations

  1. Speech and Image Processing Lab, Department of ECE, National Institute of Technology Silchar, Silchar, 788010, Assam, India

    Chintha Sri Pothu Raju & Rabul Hussain Laskar

  2. Department of Biotechnology and Medical Engineering, National Institute of Technology Rourkela, Rourkela, 769008, Odisha, India

    Bala Chakravarthy Neelapu

  3. Department of Computer Engineering, College of Computer and Information Sciences, King Saud University, Riyadh, 11543, Saudi Arabia

    Ghulam Muhammad

Authors
  1. Chintha Sri Pothu Raju
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bala Chakravarthy Neelapu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rabul Hussain Laskar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ghulam Muhammad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Chintha Sri Pothu Raju or Ghulam Muhammad.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Raju, C.S.P., Neelapu, B.C., Laskar, R.H. et al. Analysis of multimodal fusion strategies in deep learning for ischemic stroke lesion segmentation on computed tomography perfusion data. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-19252-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-024-19252-2

Keywords

Search

Navigation