Generative adversarial network in medical imaging: A review
Graphical abstract
Introduction
With the resurgence of deep learning in computer vision starting from 2012 (Krizhevsky et al., 2012), the adoption of deep learning methods in medical imaging has increased dramatically. It is estimated that there were over 400 papers published in 2016 and 2017 in major medical imaging related conference venues and journals (Litjens et al., 2017). The wide adoption of deep learning in the medical imaging community is due to its demonstrated potential to complement image interpretation and augment image representation and classification. In this article, we focus on one of the most interesting recent breakthroughs in the field of deep learning - generative adversarial networks (GANs) - and their potential applications in the field of medical imaging.
GANs are a special type of neural network model where two networks are trained simultaneously, with one focused on image generation and the other centered on discrimination. The adversarial training scheme has gained attention in both academia and industry due to its usefulness in counteracting domain shift, and effectiveness in generating new image samples. This model has achieved state-of-the-art performance in many image generation tasks, including text-to-image synthesis (Xu et al., 2017), super-resolution (Ledig et al., 2017), and image-to-image translation (Zhu et al., 2017).
Unlike deep learning which has its roots traced back to the 1980s (Fukushima and Miyake, 1982), the concept of adversarial training is relatively new with significant recent progress (Goodfellow et al., 2014). This paper presents a general overview of GANs, describes their promising applications in medical imaging, and identifies some remaining challenges that need to be solved to enable their successful application in other medical imaging related tasks.
To present a comprehensive overview of all relevant works on GANs in medical imaging, we searched databases including PubMed, arXiv, proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), SPIE Medical Imaging, IEEE International Symposium on Biomedical Imaging (ISBI), and International conference on Medical Imaging with Deep Learning (MIDL). We also incorporated cross referenced works not identified in the above search process. Since there are research publications coming out every month, without losing generality, we set the cut off time of the search as January 1st, 2019. Works on arXiv that report only preliminary results are excluded from this review. Descriptive statistics of these papers based on task, imaging modality and year can be found in Fig. 1.
The remainder of the paper is structured as follows. We begin with a brief introduction of the principles of GANs and some of its structural variants in Section 2. It is followed by a comprehensive review of medical image analysis tasks using GANs in Section 3 including but not limited to the fields of radiology, histopathology and dermatology. We categorize all the works according to canonical tasks: reconstruction, image synthesis, segmentation, classification, detection, registration, and others. Section 4 summarizes the review and discusses prospective applications and identifies open challenges.
Section snippets
Vanilla GAN
The vanilla GAN (Goodfellow et al., 2014) is a generative model that was designed for directly drawing samples from the desired data distribution without the need to explicitly model the underlying probability density function. It consists of two neural networks: the generator G and the discriminator D. The input to G, z is pure random noise sampled from a prior distribution p(z), which is commonly chosen to be a Gaussian or a uniform distribution for simplicity. The output of G, xg is expected
Applications in medical imaging
There are generally two ways GANs are used in medical imaging. The first is focused on the generative aspect, which can help in exploring and discovering the underlying structure of training data and learning to generate new images. This property makes GANs very promising in coping with data scarcity and patient privacy. The second focuses on the discriminative aspect, where the discriminator D can be regarded as a learned prior for normal images so that it can be used as regularizer or
Discussion
In the years 2017 and 2018, the number of studies applying GANs has risen significantly. The list of these papers reviewed for our study can be found on our1 GitHub repository.
About 46% of these papers studied image synthesis, with cross modality image synthesis being the most important application of GANs. MR is ranked as the most common imaging modality explored in the GAN related literature. We believe one of the reasons for the
Declaration of Competing Interest
We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.
References (258)
- et al.
Generating highly realistic images of skin lesions with GANs
OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis
(2018) - et al.
Adversarial image synthesis for unpaired multi-modal cardiac data
International Workshop on Simulation and Synthesis in Medical Imaging
(2017) - et al.
Freehand ultrasound image simulation with spatially-conditioned generative adversarial networks
Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment
(2017) - Huang, H., Yu, P.S., Wang, C., 2018. An introduction to image synthesis with generative adversarial nets....
- et al.
Synseg-net: synthetic segmentation without target modality ground truth
IEEE Trans. Med. Imaging
(2018) - et al.
Refacing: reconstructing anonymized facial features using GANs
- et al.
Data from NSCLC-radiomics
Cancer Imaging Arch
(2015) - et al.
Generative adversarial networks for brain lesion detection
SPIE Medical Imaging
(2017) - et al.
Retinal image synthesis for cad development
International Conference Image Analysis and Recognition
(2018) - et al.
Wasserstein generative adversarial networks