Skip to main content
SpringerLink
Log in

Controllable fundus image generation based on conditional generative adversarial networks with mask guidance

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Fundus image generation can serve training data diversity as well as clinical and medical education. To solve the problem of the controllability and flexibility of fundus image generation with multiple lesion features, we propose a controllable fundus image generation model (CFIGGAN) based on conditional generative adversarial networks (GAN) for medical data augmentation. The least square loss and the perceptual loss term are added to the final loss to make the generated images more realistic, and the spectral normalization is used as the normalization method of the discriminator to make the training process more stable. In the two-stage training of the model, the vascular tree image concatenates the real and generated images as positive and negative samples to train the model. CFIGGAN can generate diseased fundus images by using the annotations of vascular tree, field of vision(FOV), DR-related lesions as input and controlling the morphology of four types of lesions. Qualitative experimental evaluation shows that the fundus images generated by our model are clear and realistic and close to the real image data distribution. Quantitative experimental evaluation shows that the combination of the spectral norm and the perceptual loss can improve visual observation and quantitative indices, and data augmentation by image generation can further increase the classification accuracy. More importantly, CFIGGAN achieve the controllability of fundus image generation corresponding to DR-related lesions, and the proposed method can be extended to medical images generation of other diseases for broader prospects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data Availibility Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. Fiorini S, Ballerini L, Trucco E, Ruggeri A (2014) Automatic generation of synthetic retinal fundus images. In: MIUA. p 7–12 https://diglib.eg.org/bitstream/handle/10.2312/stag.20141238.041-044/041-044.pdf?sequence=1 &isAllowed=yhttps://doi.org/10.1016/j.procs.2016.07.010

  2. Costa P, Galdran A, Meyer MI, Abramoff MD, Niemeijer M, Mendonça AM, Campilho A (2017) Towards adversarial retinal image synthesis. arXiv:1701.08974https://doi.org/10.48550/arXiv.1701.08974

  3. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention. Springer p 234–241. https://doi.org/10.1007/978-3-319-24574-4_28https://link.springer.com/chapter/10.1007/978-3-319-24574-4_28

  4. Costa P, Galdran A, Meyer MI, Niemeijer M, Abrámoff M, Mendonça AM, Campilho A (2017) End-to-end adversarial retinal image synthesis. IEEE Trans Med Imaging 37(3)781–791. https://doi.org/10.1109/TMI.2017.2759102https://ieeexplore.ieee.org/document/8055572

  5. Mao X, Li Q, Xie H, Lau RY, Wang Z, Paul Smolley S (2017) Least squares generative adversarial networks. In: Proceedings of the IEEE international conference on computer vision. p 2794–2802. https://doi.org/10.48550/arXiv.1611.04076arXiv:1611.04076

  6. Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. p 1125–1134. https://doi.org/10.48550/arXiv.1611.07004arXiv:1611.07004v3

  7. Miyato T, Kataoka T, Koyama M, Yoshida Y (2018) Spectral normalization for generative adversarial networks. arXiv:1802.05957https://doi.org/10.48550/arXiv.1802.05957

  8. Srivastava A, Aggarwal AK (2018) Medical image fusion in spatial and transform domain: a comparative analysis. In: Handbook of research on advanced concepts in real-time image and video processing. IGI global p 281–300. https://doi.org/10.4018/978-1-5225-2848-7.ch011https://www.igi-global.com/chapter/medical-image-fusion-in-spatial-and-transform-domain/186281

  9. Thukral R, Kumar A, Arora A, et al. (2020) Effects of different radiations of electromagnetic spectrum on human health. In: 2020 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS). IEEE p 1–6. https://doi.org/10.1109/SCEECS48394.2020.154https://ieeexplore.ieee.org/abstract/document/9086961

  10. Thukral R, Kumar A, Arora A, et al. (2019) Effect of different thresholding techniques for denoising of emg signals by using different wavelets. In: 2019 2nd International conference on intelligent communication and computational techniques (ICCT). IEEE p 161–165. https://doi.org/10.1109/ICCT46177.2019.8969036https://ieeexplore.ieee.org/document/8969036

  11. Srivastava A, Singhal V, Aggarawal AK (2017) Comparative analysis of multimodal medical image fusion using pca and wavelet transforms. Int J Latest Technol Eng Manag Appl Sci (IJLTEMAS) VI https://www.academia.edu/download/53002072/115-118.pdf

  12. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inf Process Sys 27 https://proceedings.neurips.cc/paper_files/paper/2014/hash/5ca3e9b122f61f8f06494c97b1afccf3-Abstract.html

  13. Singh NK, Raza K (2021) Medical image generation using generative adversarial networks: a review. Health Inf Comput Perspec Healthcare 77–96. https://doi.org/10.1007/978-981-15-9735-0_5https://link.springer.com/chapter/10.1007/978-981-15-9735-0_5

  14. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434https://doi.org/10.48550/arXiv.1511.06434

  15. Ring M, Schlör D, Landes D, Hotho A (2019) Flow-based network traffic generation using generative adversarial networks. Comput Secur 82:156–172. https://doi.org/10.1016/j.cose.2018.12.012https://www.sciencedirect.com/science/article/pii/S0167404818308393

  16. Karras T, Aila T, Laine S, Lehtinen J (2017) Progressive growing of gans for improved quality, stability, and variation. arXiv:1710.10196https://doi.org/10.48550/arXiv.1710.10196

  17. Chuquicusma MJ, Hussein S, Burt J, Bagci U (2018) How to fool radiologists with generative adversarial networks? a visual turing test for lung cancer diagnosis. In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018). IEEE p 240–244. https://doi.org/10.1109/ISBI.2018.8363564https://ieeexplore.ieee.org/document/8363564

  18. Frid-Adar M, Diamant I, Klang E, Amitai M, Goldberger J, Greenspan H (2018) Gan-based synthetic medical image augmentation for increased cnn performance in liver lesion classification. Neurocomputing 321:321–331. https://doi.org/10.1016/j.neucom.2018.09.013https://www.sciencedirect.com/science/article/pii/S0925231218310749

  19. Bermudez C, Plassard AJ, Davis LT, Newton AT, Resnick SM, Landman BA (2018) Learning implicit brain mri manifolds with deep learning. In: Medical Imaging 2018: Image Processing, Vol. 10574. International Society for Optics and Photonics p 105741L. https://doi.org/10.1117/12.2293515https://www.spiedigitallibrary.org/conference-proceedings-of-spie/10574/105741L/Learning-implicit-brain-MRI-manifolds-with-deep-learning/10.1117/12.2293515.short?SSO=1

  20. Jin D, Xu Z, Tang Y, Harrison AP, Mollura DJ (2018) Ct-realistic lung nodule simulation from 3d conditional generative adversarial networks for robust lung segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer p 732–740. https://doi.org/10.1007/978-3-030-00934-2_81https://link.springer.com/chapter/10.1007/978-3-030-00934-2_81

  21. Mok TC, Chung A (2018) Learning data augmentation for brain tumor segmentation with coarse-to-fine generative adversarial networks. In: International MICCAI brainlesion workshop. Springer p 70–80. https://doi.org/10.1007/978-3-030-11723-8_7https://link.springer.com/chapter/10.1007/978-3-030-11723-8_7

  22. Chartsias A, Joyce T, Dharmakumar R, Tsaftaris SA (2017) Adversarial image synthesis for unpaired multi-modal cardiac data. In: International workshop on simulation and synthesis in medical imaging. Springer p 3–13. https://doi.org/10.1007/978-3-319-68127-6_1https://link.springer.com/chapter/10.1007/978-3-319-68127-6_1

  23. Maspero M, Savenije MH, Dinkla AM, Seevinck PR, Intven MP, Jurgenliemk-Schulz IM, Kerkmeijer LG, van den Berg CA (2018) Dose evaluation of fast synthetic-ct generation using a generative adversarial network for general pelvis mr-only radiotherapy. Phys Med Biol 63(18):185001. https://doi.org/10.1088/1361-6560/aada6dhttps://iopscience.iop.org/article/10.1088/1361-6560/aada6d/meta

  24. Yang H, Sun J, Carass A, Zhao C, Lee J, Xu Z, Prince J (2018) Unpaired brain mr-to-ct synthesis using a structure-constrained cyclegan. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer p 174–182. https://doi.org/10.1007/978-3-030-00889-5_20https://link.springer.com/chapter/10.1007/978-3-030-00889-5_20

  25. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, et al. (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. p 4681–4690. https://doi.org/10.48550/arXiv.1609.04802https://arxiv.org/abs/1609.04802v5

  26. Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Qiao Y, Change Loy C (2018) Esrgan: Enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV) workshops. https://doi.org/10.48550/arXiv.1809.00219arXiv:1809.00219v2

  27. Guan J, Pan C, Li S, Yu D (2019) Srdgan: learning the noise prior for super resolution with dual generative adversarial networks. https://doi.org/10.48550/arXiv.1903.11821arXiv:1903.11821

  28. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. p 2223–2232. https://doi.org/10.48550/arXiv.1703.10593arXiv:1703.10593v6

  29. Yi Z, Zhang H, Tan P, Gong M (2017) Dualgan: Unsupervised dual learning for image-to-image translation. In: Proceedings of the IEEE international conference on computer vision. p 2849–2857. https://doi.org/10.48550/arXiv.1704.02510arXiv:1704.02510

  30. Lin K, Li D, He X, Zhang Z, Sun M-T (2017) Adversarial ranking for language generation. Adv Neural Inf Process Sys 30 https://proceedings.neurips.cc/paper_files/paper/2017/file/bf201d5407a6509fa536afc4b380577e-Paper.pdf

  31. Mogren O (2016) C-rnn-gan: Continuous recurrent neural networks with adversarial training. https://doi.org/10.48550/arXiv.1611.09904arXiv:1611.09904

  32. Vondrick C, Pirsiavash H, Torralba A (2016) Generating videos with scene dynamics. Adv Neural Inf Process Sys 29 https://proceedings.neurips.cc/paper/2016/file/04025959b191f8f9de3f924f0940515f-Paper.pdf

  33. Diaz-Pinto A, Colomer A, Naranjo V, Morales S, Xu Y, Frangi AF (2019) Retinal image synthesis and semi-supervised learning for glaucoma assessment. IEEE Trans Med Imaging 38(9)2211–2218. https://doi.org/10.1109/TMI.2019.2903434https://ieeexplore.ieee.org/document/8662628

  34. Deshmukh A, Sivaswamy J (2019) Synthesis of optical nerve head region of fundus image. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). IEEE p 583–586. https://doi.org/10.1109/ISBI.2019.8759414https://ieeexplore.ieee.org/document/8759414

  35. Wang X, Xu M, Li L, Wang Z, Guan Z (2019) Pathology-aware deep network visualization and its application in glaucoma image synthesis. In: International conference on medical image computing and computer-assisted intervention. Springer p 423–431. https://doi.org/10.1007/978-3-030-32239-7_47https://link.springer.com/chapter/10.1007/978-3-030-32239-7_47

  36. Zhao H, Li H, Maurer-Stroh S, Cheng L (2018) Synthesizing retinal and neuronal images with generative adversarial nets. Med Image Anal 49:14–26. https://doi.org/10.1016/j.media.2018.07.001https://www.sciencedirect.com/science/article/pii/S1361841518304596

  37. Zhou Y, He X, Cui S, Zhu F, Liu L, Shao L (2019) High-resolution diabetic retinopathy image synthesis manipulated by grading and lesions. In: International conference on medical image computing and computer-assisted intervention. Springer p 505–513. https://doi.org/10.1007/978-3-030-32239-7_56https://link.springer.com/chapter/10.1007/978-3-030-32239-7_56

  38. Burlina PM, Joshi N, Pacheco KD, Liu TA, Bressler NM (2019) Assessment of deep generative models for high-resolution synthetic retinal image generation of age-related macular degeneration. JAMA Ophthalmol 137(3)258–264 https://doi.org/10.1001/jamaophthalmol.2018.6156https://jamanetwork.com/journals/jamaophthalmology/article-abstract/2720489

  39. Altaf F, Islam SM, Akhtar N, Janjua NK (2019) Going deep in medical image analysis: concepts, methods, challenges, and future directions. IEEE Access 7:99540–99572. https://doi.org/10.1109/ACCESS.2019.2929365https://ieeexplore.ieee.org/document/8764525

  40. Mahapatra D, Bozorgtabar B, Hewavitharanage S, Garnavi R (2017) Image super resolution using generative adversarial networks and local saliency maps for retinal image analysis, in: International conference on medical image computing and computer-assisted intervention. Springer p 382–390. https://doi.org/10.1007/978-3-319-66179-7_44https://link.springer.com/chapter/10.1007/978-3-319-66179-7_44

  41. Staal J, Abràmoff MD, Niemeijer M, Viergever MA, Van Ginneken B (2004) Ridge-based vessel segmentation in color images of the retina. IEEE Trans Med Imaging 23(4):501–509. https://doi.org/10.1109/TMI.2004.825627https://ieeexplore.ieee.org/document/1282003

  42. J Z J (2018) Laddernet: Multi-path networks based on u-net for medical image segmentation. https://doi.org/10.48550/arXiv.1810.07810arXiv:1810.07810

  43. Li T, Gao Y, Wang K, Guo S, Liu H, Kang H (2019) Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Inf Sci 501:511–522. https://doi.org/10.1016/j.ins.2019.06.011https://www.sciencedirect.com/science/article/pii/S0020025519305377

  44. Hore A, Ziou D (2010) Image quality metrics: Psnr vs. ssim. In: 2010 20th international conference on pattern recognition. IEEE p 2366–2369. https://doi.org/10.1109/ICPR.2010.579https://ieeexplore.ieee.org/document/5596999

  45. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Adv Neural Inf Process Sys 30. https://proceedings.neurips.cc/paper_files/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf

  46. Bińkowski M, Sutherland DJ, Arbel M, Gretton A (2018) Demystifying mmd gans. https://doi.org/10.48550/arXiv.1801.01401arXiv:1801.01401

  47. Guibas JT, Virdi TS, Li PS (2017) Synthetic medical images from dual generative adversarial networks. https://doi.org/10.48550/arXiv.1709.01872arXiv:1709.01872

  48. Karras T, Laine S, Aittala M, Hellsten J, Lehtinen J, Aila T (2020) Analyzing and improving the image quality of stylegan. Proc IEEE/CVF Conf Comput Vis Pattern Recognit 2020:8110–8119

    Google Scholar 

  49. Dhariwal P, Nichol A (2021) Diffusion models beat gans on image synthesis. Adv Neural Inf Process Syst 34:8780–8794

    Google Scholar 

  50. Niu Y, Gu L, Zhao Y, Lu F (2021) Explainable diabetic retinopathy detection and retinal image generation. IEEE J Biomed Health Inform 26(1):44–55

    Article  Google Scholar 

  51. Yu Z, Xiang Q, Meng J, Kou C, Ren Q, Lu Y (2019) Retinal image synthesis from multiple-landmarks input with generative adversarial networks. Biomed Eng Online 18(1):1–15

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 82071995, the Key Research And Development Program of Jilin Province under Grant 20220201141GX and the Natural Science Foundation of Jilin Province under Grant 20200201292JC.

Author information

Authors and Affiliations

  1. Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education, Jilin University, 130012, Changchun, China

    Xiaoxin Guo, Xiang Li & Guangyu Li

  2. College of Computer Science and Technology, Jilin University, 130012, Changchun, China

    Xiaoxin Guo, Xiang Li & Guangyu Li

  3. College of Software, Jilin University, 130012, Changchun, China

    Qifeng Lin

  4. Ophthalmology Department, Bethune First Hospital of Jilin University, 130021, Changchun, China

    Xiaoying Hu

  5. Ophthalmology Department, Bethune Second Hospital of Jilin University, 130041, Changchun, China

    Songtian Che

Authors
  1. Xiaoxin Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qifeng Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guangyu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaoying Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Songtian Che
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoxin Guo.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, X., Li, X., Lin, Q. et al. Controllable fundus image generation based on conditional generative adversarial networks with mask guidance. Multimed Tools Appl 83, 46065–46085 (2024). https://doi.org/10.1007/s11042-023-17280-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-17280-y

Keywords

Search

Navigation