Skip to main content

Advertisement

SpringerLink
Log in

A general variation-driven network for medical image synthesis

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The significance of medical image synthesis has exponentially grown due to constrained medical resources, making it a critical component in numerous clinical applications. This process facilitates the generation of high-quality, multi-modal medical images, ultimately enhancing medical image diagnostics. Currently, prevailing medical image synthesis methodologies primarily rely on voxel-based or GAN-based strategies to address substantial challenges arising from disparities in various imaging principles and significant noise. However, these methodologies rarely consider the intensity distribution difference among multi-modal or multi-parameters medical images, which generates unstable and unexplainable results. In response to these limitations, we propose a novel approach-a general variation-driven neural network for medical image synthesis that considers explicit data distribution. Within this method, we introduce the concept of a variation-based distance metric, providing a quantitative framework for capturing distribution disparities between medical images originating from both the source and target domains. Subsequently, guided by this variation-based distance metric, we introduce a robust end-to-end neural network architecture carefully designed to synthesize target medical images. Our proposed method has undergone extensive experimentation across various medical image synthesis tasks, including cross-modality transformations between CT and MRI, high-dose CT synthesis from low-dose CT, and the conversion of multi-parameters in MRI, including T1, T2, T1ce, and Flair sequences. In comparative assessments against existing methods, our approach consistently outperforms them across three publicly available datasets: Gold Atlas, LDCT, and BraTS2018. Additionally, we have successfully applied our model to generate high-quality Micro-CT images in dental clinics from CBCT data, significantly enhancing diagnostic capabilities in this clinical setting.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and materials

The data and materials used during the current study is available from the corresponding author on resonable request.

References

  1. Wang Y, Zhou L, Yu B, Wang L, Zu C, Lalush DS, Lin W, Wu X, Zhou J, Shen D (2019) 3D Auto-Context-Based Locality Adaptive Multi-Modality GANs for PET Synthesis. IEEE Trans Med Imaging 38(6):1328–1339

    Article  Google Scholar 

  2. Chen Y, Yue X, Fujita H, Fu S (2017) Three-way decision support for diagnosis on focal liver lesions. Knowl-Based Syst 127:85–99

    Article  Google Scholar 

  3. Yang X, Chen Y, Yue X, Ma C, Yang P (2021) Local linear embedding based interpolation neural network in pancreatic tumor segmentation. Appl Intell

  4. Zhang Y, Li H, Du J, Qin J, Wang T, Chen Y, Liu B, Gao W, Ma G, Lei B (2021) 3D Multi-Attention Guided Multi-Task Learning Network for Automatic Gastric Tumor Segmentation and Lymph Node Classification. IEEE Trans Med Imaging 40(6):1618–1631

    Article  Google Scholar 

  5. Gao S, Zhuang X (2022) Bayesian Image Super-Resolution with Deep Modeling of Image Statistics. IEEE Trans Pattern Anal Mach Intell pp 1–1

  6. Huang Y, Shao L, Frangi AF (2018) Cross-Modality Image Synthesis via Weakly Coupled and Geometry Co-Regularized Joint Dictionary Learning. IEEE Trans Med Imaging 37(3):815–827

    Article  Google Scholar 

  7. Escobar M, Castillo A, Romero A, Arbeláez P (2020) UltraGAN: Ultrasound Enhancement Through Adversarial Generation. In: International Workshop on Simulation and Synthesis in Medical Imaging, pp. 120–130

  8. Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  9. Nie D, Trullo R, Lian J, Petitjean C, Ruan S, Wang Q, Shen D (2017) Medical image synthesis with context-aware generative adversarial networks. In: International conference on medical image computing and computer-assisted intervention, pp 417–425

  10. Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232

  11. Chen X, Lian C, Wang L, Deng H, Fung SH, Nie D, Thung K-H, Yap P-T, Gateno J, Xia JJ, Shen D (2020) One-Shot Generative Adversarial Learning for MRI Segmentation of Craniomaxillofacial Bony Structures. IEEE Trans Med Imaging 39(3):787–796

    Article  Google Scholar 

  12. Wang G, Gong E, Banerjee S, Martin D, Tong E, Choi J, Chen H, Wintermark M, Pauly JM, Zaharchuk G (2020) Synthesize high-quality multi-contrast magnetic resonance imaging from multi-echo acquisition using multi-task deep generative model. IEEE Trans Med Imaging 39(10):3089–3099

    Article  Google Scholar 

  13. Yang H, Sun J, Carass A, Zhao C, Lee J, Prince JL, Xu Z (2020) Unsupervised MR-to-CT Synthesis Using Structure-Constrained CycleGAN. IEEE Trans Med Imaging 39(12):4249–4261

    Article  Google Scholar 

  14. Yu B, Zhou L, Wang L, Shi Y, Fripp J, Bourgeat P (2020) Sample-Adaptive GANs: Linking Global and Local Mappings for Cross-Modality MR Image Synthesis. IEEE Trans Med Imaging 39(7):2339–2350

    Article  Google Scholar 

  15. Hoshen Y, Li K, Malik J (2019) Non-adversarial image synthesis with generative latent nearest neighbors. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5811–5819

  16. Zhou T, Fu H, Chen G, Shen J, Shao L (2020) Hi-Net: Hybrid-Fusion Network for Multi-Modal MR Image Synthesis. IEEE Trans Med Imaging 39(9):2772–2781

    Article  Google Scholar 

  17. Zhu J-Y, Zhang R, Pathak D, Darrell T, Efros AA, Wang O, Shechtman E (2017) Toward multimodal image-to-image translation. In: Proceedings of the international conference on neural information processing systems, pp 465–476

  18. Mao Q, Lee H-Y, Tseng H-Y, Ma S, Yang M- (2019) Mode seeking generative adversarial networks for diverse image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1429–1437

  19. Dar SU, Yurt M, Karacan L, Erdem A, Erdem E, Çukur T (2019) Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks. IEEE Trans Med Imaging 38(10):2375–2388

    Article  Google Scholar 

  20. Yue Z, Yong H, Zhao Q, Meng D, Zhang L (2019) Variational Denoising Network: Toward Blind Noise Modeling and Removal. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada., pp. 1688–1699

  21. Qu L, Zhang Y, Wang S, Yap P-T, Shen D (2020) Synthesized 7T MRI from 3T MRI via deep learning in spatial and wavelet domains. Med Image Anal 62:101663

    Article  Google Scholar 

  22. Meng M, Li S, Yao L, Li D, Zhu M, Gao Q, Xie Q, Zhao Q, Bian Z, Huang J (2020) Semi-supervised learned sinogram restoration network for low-dose CT image reconstruction. In: Medical Imaging 2020: Physics of Medical Imaging, vol. 11312, p 113120

  23. Xie Q, Zhou M, Zhao Q, Meng D, Zuo W, Xu Z (2019) Multispectral and hyperspectral image fusion by MS/HS fusion net. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1585–1594

  24. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R (2014) The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging 34(10):1993–2024

    Article  Google Scholar 

  25. Moen TR, Chen B, Holmes DR III, Duan X, Yu Z, Yu L, Leng S, Fletcher JG, McCollough CH (2021) Low-dose CT image and projection dataset. Med Phys 48(2):902–911

    Article  Google Scholar 

  26. Nyholm T, Svensson S, Andersson S, Jonsson J, Sohlin M, Gustafsson C, Kjellén E, Söderström K, Albertsson P, Blomqvist L (2018) MR and CT data with multiobserver delineations of organs in the pelvic area-Part of the Gold Atlas project. Medical Phys 45(3):1295–1300

    Article  Google Scholar 

  27. Balwant M (2022) A review on convolutional neural networks for brain tumor segmentation: Methods, datasets, libraries, and future directions. IRBM 43(6):521–537

    Article  Google Scholar 

  28. Dequidt P, Bourdon P, Tremblais B, Guillevin C, Gianelli B, Boutet C, Cottier J-P, Vallée J-N, Fernandez-Maloigne C, Guillevin R (2021) Exploring radiologic criteria for glioma grade classification on the brats dataset. IRBM 42(6):407–414

  29. Khairandish MO, Sharma M, Jain V, Chatterjee JM, Jhanjhi N (2022) A hybrid cnn-svm threshold segmentation approach for tumor detection and classification of mri brain images. Irbm 43(4):290–299

    Article  Google Scholar 

  30. Jiang L, Mao Y, Chen X, Wang X, Li C (2023) Cola-diff: Conditional latent diffusion model for multi-modal mri synthesis. arXiv preprint arXiv:2303.14081

  31. Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B (2022) High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10684–10695

  32. Chartsias A, Joyce T, Giuffrida MV, Tsaftaris SA (2017) Multimodal MR synthesis via modality-invariant latent representation. IEEE Trans Med Imaging 37(3):803–814

    Article  Google Scholar 

  33. Yurt M, Özbey M, Dar SU, Tinaz B, Oguz KK, Çukur T (2022) Progressively volumetrized deep generative models for data-efficient contextual learning of mr image recovery. Med Image Anal 78:102429

    Article  Google Scholar 

  34. Peng B, Liu B, Bin Y, Shen L, Lei J (2021) Multi-modality mr image synthesis via confidence-guided aggregation and cross-modality refinement. IEEE J Biomed Health Inform 26(1):27–35

    Article  Google Scholar 

  35. Sharma A, Hamarneh G (2019) Missing mri pulse sequence synthesis using multi-modal generative adversarial network. IEEE Trans Med Imaging 39(4):1170–1183

    Article  Google Scholar 

  36. Huang Z, Zhang J, Zhang Y, Shan H (2021) DU-GAN: Generative Adversarial Networks with Dual-Domain U-Net Based Discriminators for Low-Dose CT Denoising. IEEE Trans Instrum Meas pp 1–1

  37. Chen H, Zhang Y, Kalra MK, Lin F, Chen Y, Liao P, Zhou J, Wang G (2017) Low-Dose CT With a Residual Encoder-Decoder Convolutional Neural Network. IEEE Trans Med Imaging 36(12):2524–2535

    Article  Google Scholar 

  38. Fan F, Shan H, Kalra MK, Singh R, Qian G, Getzin M, Teng Y, Hahn J, Wang G (2020) Quadratic Autoencoder (Q-AE) for Low-Dose CT Denoising. IEEE Trans Med Imaging 39(6):2035–2050

    Article  Google Scholar 

  39. Yang Q, Yan P, Zhang Y, Yu H, Shi Y, Mou X, Kalra MK, Zhang Y, Sun L, Wang G (2018) Low-Dose CT Image Denoising Using a Generative Adversarial Network With Wasserstein Distance and Perceptual Loss. IEEE Trans Med Imaging 37(6):1348–1357

    Article  Google Scholar 

  40. Mäkinen Y, Azzari L, Foi A (2020) Collaborative filtering of correlated noise: Exact transform-domain variance for improved shrinkage and patch matching. IEEE Trans Image Process 29:8339–8354

    Article  Google Scholar 

  41. Shan H, Zhang Y, Yang Q, Kruger U, Kalra MK, Sun L, Cong W, Wang G (2018) 3-D Convolutional Encoder-Decoder Network for Low-Dose CT via Transfer Learning From a 2-D Trained Network. IEEE Trans Med Imaging 37(6):1522–1534

    Article  Google Scholar 

  42. Bera S, Biswas PK (2021) Noise conscious training of non local neural network powered by self attentive spectral normalized markovian patch gan for low dose ct denoising. IEEE Trans Med Imaging 40(12):3663–3673

    Article  Google Scholar 

  43. Geng M, Meng X, Yu J, Zhu L, Jin L, Jiang Z, Qiu B, Li H, Kong H, Yuan J, Yang K, Shan H, Han H, Yang Z, Ren Q, Lu Y (2021) Content-Noise Complementary Learning for Medical Image Denoising. IEEE Trans Med Imaging, pp 1–1

  44. Marcos L, Alirezaie J, Babyn P (2022) Low dose ct denoising by resnet with fused attention modules and integrated loss functions. Front Signal Process 1:812193

    Article  Google Scholar 

  45. Gholizadeh-Ansari M, Alirezaie J, Babyn P (2020) Deep learning for low-dose ct denoising using perceptual loss and edge detection layer. J Digit Imaging 33:504–515

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (No. 62173252, 61976134).

Author information

Authors and Affiliations

  1. College of Electronics and Information Engineering, Tongji University, Shanghai, 200092, China

    Yufei Chen & Xiaoyu Yang

  2. Artificial Intelligence Institute, Shanghai University, Shanghai, 200444, China

    Xiaodong Yue

  3. Department of Endodontics, School and Hospital of Stomatology, Tongji University, Shanghai, 200072, China

    Xiang Lin & Qi Zhang

  4. Malaysia-Japan International Institute of Technology (MJIIT), Universiti Teknologi Malaysia, Kuala Lumpur, 54100, Malaysia

    Hamido Fujita

  5. Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI), University of Granada, Granada, Spain

    Hamido Fujita

  6. Reginonal Research Center, Iwate Prefectural University, Takizawa, 020-0693, Japan

    Hamido Fujita

Authors
  1. Yufei Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoyu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaodong Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiang Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hamido Fujita
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yufei Chen or Qi Zhang.

Ethics declarations

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: The word “image” was accidentally removed.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Yang, X., Yue, X. et al. A general variation-driven network for medical image synthesis. Appl Intell 54, 3295–3307 (2024). https://doi.org/10.1007/s10489-023-05017-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05017-1

Keywords

Search

Navigation