Abstract
Multimodal medical image fusion helps in combining contrasting features from two or more input imaging modalities to represent fused information in a single image. One of the pivotal clinical applications of medical image fusion is the merging of anatomical and functional modalities for fast diagnosis of malign tissues. In this paper, we present a novel end-to-end unsupervised learning based Convolutional neural network (CNN) for fusing the high and low frequency components of MRI-PET grayscale image pairs publicly available at ADNI by exploiting Structural Similarity Index (SSIM) as the loss function during training. We then apply color coding for the visualization of the fused image by quantifying the contribution of each input image in terms of the partial derivatives of the fused image. We find that our fusion and visualization approach results in better visual perception of the fused image, while also comparing favorably to previous methods when applying various quantitative assessment metrics.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
James, A.P., Dasarathy, B.V.: Medical image fusion: a survey of the state of the art. Inf. Fusion 19, 4–19 (2014)
Nensa, F., Beiderwellen, K., Heusch, P., Wetter, A.: Clinical applications of PET/MRI: current status and future perspectives. Diagn. Interv. Radiol. 20(5), 438–447 (2014)
Bhatnagar, G., Wu, Q.M.J., Liu, Z.: Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Trans. Multimedia 15(5), 1014–1024 (2013)
Du, J., Li, W., Xiao, B., Nawaz, Q.: Union Laplacian pyramid with multiple features for medical image fusion. Neurocomputing 194, 326–339 (2016)
Du, J., Li, W., Xiao, B.: Anatomical-functional image fusion by information of interest in local Laplacian filtering domain. IEEE Trans. Image Process. 26(12), 5855–5866 (2017)
Yin, M., Liu, X., Liu, Y., Chen, X.: Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans. Instrum. Meas. 68(1), 49–64 (2019)
Li, S., Kang, X., Hu, J.: Image fusion with guided filtering. IEEE Trans. Image Process. 22(7), 2864–2875 (2013)
Li, H., He, X., Tao, D., Tang, Y., Wang, R.: Joint medical image fusion, denoising and enhancement via discriminative low-rank sparse dictionaries learning. Pattern Recogn. 79, 130–146 (2018)
Yang, B., Li, S.: Pixel-level image fusion with simultaneous orthogonal matching pursuit. Inf. Fusion 13(1), 10–19 (2012)
Liu, Y., Liu, S., Wang, Z.: A general framework for image fusion based on multi-scale transform and sparse representation. Inf. Fusion 24, 147–164 (2015)
Daneshvar, S., Ghassemian, H.: MRI and PET image fusion by combining IHS and retina-inspired models. Inf. Fusion 11(2), 114–123 (2010)
Hoffmann, N., Weidner, F., Urban, P., et al.: Framework for 2D–3D image fusion of infrared thermography with preoperative MRI. Biomed. Eng./Biomedizinische Technik 62(6), 599–607 (2017)
Liu, Y., Chen, X., Cheng, J., Peng, H.: A medical image fusion method based on convolutional neural networks. In: 20th International Conference on Information Fusion (Fusion), Xi’an, pp. 1–7 (2017). https://doi.org/10.23919/ICIF.2017.8009769
Liu, Y., Chen, X., Wang, Z., Wang, Z.J., Ward, R.K., Wang, X.: Deep learning for pixel-level image fusion: recent advances and future prospects. Inf. Fusion 42, 158–173 (2018)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Clifford, R.J., et al.: The Alzheimer’s Disease neuroimaging initiative (ADNI). J. Magn. Reson. Imaging 27(4), 685–691 (2008)
Johnson, K., Becker, J.: http://www.med.harvard.edu/AANLIB/home.html
Wang, Q., Shen, Y.: Performances evaluation of image fusion techniques based on nonlinear correlation measurement. In: 21st IEEE Instrumentation and Measurement Technology Conference (IEEE Cat. No. 04CH37510), vol. 1, pp. 472–475 (2004)
Xydeas, C., Petrovic, V.: Objective image fusion performance measure. Electron. Lett. 36, 308–309 (2000)
Haghighat, M.B.A., Aghagolzadeh, A., Seyedarabi, H.: A non-reference image fusion metric based on mutual information of image features. Comput. Electr. Eng. 37(5), 744–756 (2011)
Han, Y., Cai, Y., Cao, Y., Xu, X.: A new image fusion performance metric based on visual information fidelity. Inf. Fusion 14(2), 127–135 (2013)
Das, S., Kundu, M.K.: NSCT-based multimodal medical image fusion using pulse-coupled neural network and modified spatial frequency. Med. Biol. Engi. Comput. 50(10), 1105–1114 (2012)
Acknowledgements
This work was supported by the European Social Fund (project no. 100312752) and the Saxonian Ministry of Science and Art.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Kumar, N., Hoffmann, N., Oelschlägel, M., Koch, E., Kirsch, M., Gumhold, S. (2019). Structural Similarity Based Anatomical and Functional Brain Imaging Fusion. In: Zhu, D., et al. Multimodal Brain Image Analysis and Mathematical Foundations of Computational Anatomy. MBIA MFCA 2019 2019. Lecture Notes in Computer Science(), vol 11846. Springer, Cham. https://doi.org/10.1007/978-3-030-33226-6_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-33226-6_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33225-9
Online ISBN: 978-3-030-33226-6
eBook Packages: Computer ScienceComputer Science (R0)
