Skip to main content
SpringerLink
Log in

A review of single image super-resolution reconstruction based on deep learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Single image super-resolution (SISR) is an important research field in computer vision, the purpose of which is to recover clear, high-resolution (HR) images from low-resolution (LR) images. With the rapid developments in deep learning theory and technology, deep learning has been introduced into the field of image super-resolution (SR), and has achieved results far beyond traditional methods in many domains. This paper summarizes current image SR algorithms based on deep learning. Firstly, the mainstream frameworks, loss functions, and datasets used for SISR are introduced in detail. Then, the SISR algorithm based on deep learning is explored using three models: a convolutional neural network (CNN), a generative adversarial network (GAN), and a transformer. Next, the evaluation indices used for SR are introduced, and the reconstruction results from various algorithms based on deep learning are compared. Finally, future trends in research on image SR algorithms based on deep learning are summarized.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32
Fig. 33
Fig. 34
Fig. 35
Fig. 36
Fig. 37

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available in the following repositories: Set5(https://www.kaggle.com/msahebi/super-resolution), Set14(https://www.kaggle.com/msahebi/super-resolution), BSD100(https://www.kaggle.com/msahebi/super-resolution), Urban100(https://drive.google.com/drive/folders/1pRmhEmmY-tPF7uH8DuVthfHoApZWJ1QU?usp=sharing), Manga109(http://www.manga109.org/en/index.html), DIV2K(https://data.vision.ee.ethz.ch/cvl/DIV2K/), Flickr2K(https://drive.google.com/drive/folders/1B-uaxvV9qeuQ-t7MFiN1oEdA6dKnj2vW), RealSR(https://drive.google.com/open?id=17ZMjo-zwFouxnm_aFM6CUHBwgRrLZqIM), DPED(https://drive.google.com/file/d/0BwOLOmqkYj-jeUJwQjRNUFkzOTA/view), OutdoorScene(https://drive.google.com/drive/u/1/folders/ 1iZfzAxAwOpeutz27HC56_y5RNqnsPPKr), PIRM(https://drive.google.com/drive/folders/ 17FmdXu5t8wlKwt8extb_nQAdjxUOrb1O?usp = sharing), T91(https://drive.google.com/drive/folders/1pRmhEmmY-tPF7uH8DuVthfHoApZWJ1QU?usp=sharing),ImageNet(https://image-net.org/challenges/LSVRC/), City100(https://github.com/ngchc/CameraSR), MSCOCO(https://cocodataset.org/#download), PIPAL(https://github.com/HaomingCai/PIPAL-dataset).

References

  1. Huang TJCV, Processing I (1984) Multi-frame image restoration and registration. Multiframe Image Restor Registration 1:317–339

    Google Scholar 

  2. Greenspan HJTCJ (2009) Super-resolution in medical imaging. Comput J 52:43–63

    Article  Google Scholar 

  3. Isaac JS, Kulkarni R (2015) Super resolution techniques for medical image processing. 2015 International Conference on Technologies for Sustainable Development (ICTSD). IEEE, pp 1–6

  4. Huang Y, Shao L, Frangi AF (2017) Simultaneous super-resolution and cross-modality synthesis of 3D medical images using weakly-supervised joint convolutional sparse coding. 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 5787–5796

  5. Thornton MW, Atkinson PM, Holland DJIJORS (2006) Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping. Int J Remote Sens 27:473–491

    Article  Google Scholar 

  6. Barzegar S, Sharifi A, Manthouri MJMT et al (2020) Super-resolution using lightweight detailnet network. Multimed Tools Appl 79:1119–1136

    Article  Google Scholar 

  7. Yang W, Zhou F, Zhu R et al (2019) Deep learning for image super-resolution. Neurocomputing 398:291–292

    Article  Google Scholar 

  8. Timofte R, Rothe R, Van Gool L (2016) Seven ways to improve example-based single image super resolution. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 1865–1873

  9. Zeyde R, Elad M, Protter M (2012) On single image scale-up using sparse-representations. Curves and Surfaces: 7th International Conference. Springer, pp 711–730

  10. Haris M, Shakhnarovich G, Ukita N (2018) Deep back-projection networks for super-resolution. 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 1664–1673

  11. Lai W-S, Huang J-B, Ahuja N et al (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 5835–5843

  12. Ledig C, Theis L, Huszár F et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 105–114

  13. Bevilacqua M, Roumy A, Guillemot C et al (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. 23rd British Machine Vision Conference, pp 1–10.

  14. Huang J-B, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 5197–5206

  15. Martin D, Fowlkes C, Tal D et al (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001. IEEE, pp 416–423

    Chapter  Google Scholar 

  16. Fujimoto A, Ogawa T, Yamamoto K et al (2016) Manga109 dataset and creation of metadata. 1st international workshop on comics analysis, processing and understanding (MANPU), pp 1–5

  17. Agustsson E, Timofte R (2017) Ntire 2017 challenge on single image super-resolution: Dataset and study. 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, pp 1122–1131

  18. Xia B, Hang Y, Tian Y et al (2022) Efficient non-local contrastive attention for image super-resolution. 36th AAAI Conference on Artificial Intelligence 36(3): 2759–2767

  19. Lee J, Jin KH (2022) Local texture estimator for implicit representation function. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 1929–1938

  20. Ma C, Zhang J, Zhou J et al (2022) Learning Series-Parallel Lookup Tables for Efficient Image Super-Resolution. 17th European Conference on Computer Vision (ECCV). Springer, 13677: 305–321

  21. Timofte R, Agustsson E, Van Gool L et al (2017) Ntire 2017 challenge on single image super-resolution: Methods and results. 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, pp 1110–1121

  22. Zhou L, Cai H, Gu J et al (2022) Efficient image super-resolution using vast-receptive-field attention. European Conference on Computer Vision (ECCV). Springer, pp 256–272

  23. Ji X, Cao Y, Tai Y et al (2020) Real-world super-resolution via kernel estimation and noise injection. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). IEEE, pp 1914–1923

  24. Liang J, Zeng H, Zhang L (2022) Efficient and degradation-adaptive network for real-world image super-resolution. 17th European Conference on Computer Vision (ECCV). Springer, 13867: 574–591

  25. Cai J, Zeng H, Yong H et al (2019) Toward real-world single image super-resolution: A new benchmark and a new model. IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, pp 3086–3095

  26. Deng J, Dong W, Socher R et al (2009) Imagenet: A large-scale hierarchical image database. IEEE-Computer-Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE, pp 248–255

  27. Ignatov A, Kobyshev N, Timofte R et al (2017) Dslr-quality photos on mobile devices with deep convolutional networks. 16th IEEE International Conference on Computer Vision (ICCV). IEEE, pp 3297–3305

  28. Wang X, Yu K, Dong C et al (2018) Recovering realistic texture in image super-resolution by deep spatial feature transform. 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 606–615

  29. Blau Y, Mechrez R, Timofte R et al (2018) The 2018 PIRM challenge on perceptual image super-resolution. 15th European Conference on Computer Vision (ECCV), vol 11133. Springer, pp 334–355

  30. Yang J, Wright J, Huang TS et al (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19:2861–2873

    Article  MathSciNet  MATH  Google Scholar 

  31. Yoo J, Kim T, Lee S et al (2022) Enrich CNN-transformer feature aggregation networks for super-resolution. 23rd IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, pp 4945–4954

  32. Lin T-Y, Maire M, Belongie S et al (2014) Microsoft coco: common objects in context. 13th European Conference on Computer Vision (ECCV), vol 8693. Springer, pp 740–755

  33. Jinjin G, Haoming C, Haoyu C et al (2020) Pipal: a large-scale image quality assessment dataset for perceptual image restoration. European Conference on Computer Vision (ECCV). Springer, pp 633–651

  34. Chen C, Xiong Z, Tian X et al (2019) Camera lens super-resolution. 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 1652–1660

  35. Dong C, Loy CC, He K et al (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38:295–307

    Article  Google Scholar 

  36. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. 14th European Conference on Computer Vision (ECCV), vol 9906. Springer, pp 391–407

  37. Shi W, Caballero J, Huszár F et al (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 1874–1883

  38. He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 770–778

  39. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 1646–1654

  40. Simonyan K, Zisserman AJaPA (2015) Very deep convolutional networks for large-scale image recognition. 3rd International Conference on Learning Representations (ICLR), pp 1–14

  41. Mao X-J, Shen C, Yang Y-BJaPA (2016) Image restoration using convolutional auto-encoders with symmetric skip connections. Neural Information Processing Systems (NIPS) 29

  42. Lim B, Son S, Kim H et al (2017) Enhanced deep residual networks for single image super-resolution. 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, pp 1132–1140

  43. Nah S, Hyun Kim T, Mu Lee K (2017) Deep multi-scale convolutional neural network for dynamic scene deblurring. 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 257–265

  44. Szegedy C, Ioffe S, Vanhoucke V et al (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. 31st AAAI Conference on Artificial Intelligence, pp 4278–4284

  45. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 1637–1645

  46. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 2790–2798

  47. Han W, Chang S, Liu D et al (2018) Image super-resolution via dual-state recurrent networks. 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 1654–1663

  48. Gilbert CD, Sigman MJN (2007) Brain states: top-down influences in sensory processing. Neuron 54:677–696

    Article  Google Scholar 

  49. Hupé J, James A, Payne B et al (1998) Cortical feedback improves discrimination between figure and background by V1, V2 and V3 neurons. Nature 394:784–787

    Article  Google Scholar 

  50. Li Z, Yang J, Liu Z et al (2019) Feedback network for image super-resolution. 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 3867–3871

  51. Huang G, Liu Z, Van Der Maaten L et al (2017) Densely connected convolutional networks. 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 2261–2269

  52. Tong T, Li G, Liu X et al (2017) Image super-resolution using dense skip connections. 16th IEEE International Conference on Computer Vision (ICCV). IEEE, pp 4809–4817

  53. Tai Y, Yang J, Liu X et al (2017) Memnet: A persistent memory network for image restoration. 16th IEEE International Conference on Computer Vision (ICCV). IEEE, pp 4549–4557

  54. Chaudhari S, Mithal V, Polatkan G et al (2021) An attentive survey of attention models. ACM Trans Intell Syst Technol (TIST) 12(5):1–32

    Article  Google Scholar 

  55. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 7132–7141

  56. Zhang Y, Li K, Li K et al (2018) Image super-resolution using very deep residual channel attention networks. 15th European Conference on Computer Vision (ECCV), vol 11211. Springer, pp 294–310

  57. Dai T, Cai J, Zhang Y et al (2019) Second-order attention network for single image super-resolution. 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 11057–11066

  58. Zhang Y, Wei D, Qin C et al (2021) Context reasoning attention network for image super-resolution. 18th IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, pp 4258–4267

  59. Zhao H, Kong X, He J et al  (2020) Efficient image super-resolution using pixel attention. European Conference on Computer Vision (ECCV) Workshops. Springer, pp 56–72

  60. Hui Z, Wang X, Gao X (2018) Fast and accurate single image super-resolution via information distillation network. 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 723–731

  61. Hui Z, Gao X, Yang Y et al (2019) Lightweight image super-resolution with information multi-distillation network. 27th ACM International Conference on Multimedia (MM), pp 2024–2032

  62. Luo X, Xie Y, Zhang Y et al (2020) Latticenet: towards lightweight image super-resolution with lattice block. European Conference on Computer Vision (ECCV). Springer, pp 272–289

  63. Zhang Y, Wang H, Qin C et al (2021) Learning efficient image super-resolution networks via structure-regularized pruning. International conference on learning representations 1–12

  64. Zhang Y, Wang H, Qin C et al (2021) Aligned structured sparsity learning for efficient image super-resolution. Adv Neural Inf Process Syst 34:2695–2706

    Google Scholar 

  65. Wang H, Zhang Y, Qin C et al (2023) Global aligned structured sparsity learning for efficient Image super-resolution. IEEE Transactions on pattern analysis and machine intelligence. IEEE 45:10974–10989

    Google Scholar 

  66. Goodfellow I, Pouget-Abadie J, Mirza M et al (2020) Generative adversarial networks. IEEE Signal Process Mag 63:139–144

    Google Scholar 

  67. Wang X, Yu K, Wu S et al (2018) Esrgan: Enhanced super-resolution generative adversarial networks. 15th European Conference on Computer Vision (ECCV) Workshops, vol 11133. Springer, pp 63–79

  68. Jolicoeur-Martineau AJaPA (2018) The relativistic discriminator: a key element missing from standard GAN. International Conference on Learning Representations (ICLR 2019)

  69. Lee O-Y, Shin Y-H, Kim J-OJIA (2019) Multi-perspective discriminators-based generative adversarial network for image super resolution. IEEE Access 7:136496–136510

    Article  Google Scholar 

  70. Rakotonirina NC, Rasoanaivo A (2020) ESRGAN+: Further improving enhanced super-resolution generative adversarial network. ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp 3637–3641

    Chapter  Google Scholar 

  71. Chen Y, Li J, Xiao H et al (2017) Dual path networks. 31st Annual Conference on Neural Information Processing Systems (NIPS) 30

  72. Karras T, Laine S, Aila T (2019) A style-based generator architecture for generative adversarial networks. IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, pp 4401–4410

  73. Shi W, Tao F, Wen YJITOI et al (2023) Structure-aware deep networks and pixel-level generative adversarial training for single image super-resolution. IEEE Trans Instrum Meas 72:1–14

    Google Scholar 

  74. Isola P, Zhu J-Y, Zhou T et al (2017) Image-to-image translation with conditional adversarial networks. 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 5967–5976

  75. Ma C, Rao Y, Lu J et al (2021) Structure-preserving image super-resolution. IEEE Trans Pattern Anal Mach Intell 44:7898–7911

    Article  Google Scholar 

  76. Ma C, Rao Y, Cheng Y et al (2020) Structure-preserving super resolution with gradient guidance. IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, pp 7769–7778

  77. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. 31st Annual Conference on Neural Information Processing Systems (NIPS) 30

  78. Arnab A, Dehghani M, Heigold G et al (2021) Vivit: A video vision transformer. 18th IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, pp 6816–6826

  79. Chen H, Wang Y, Guo T et al (2021) Pre-trained image processing transformer. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 12294–12305

  80. Liu Z, Lin Y, Cao Y et al (2021) Swin transformer: Hierarchical vision transformer using shifted windows. 18th IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, pp 9992–10002

  81. Liang J, Cao J, Sun G et al (2021) Swinir: Image restoration using swin transformer. 18th IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, pp 1833–1844

  82. Lu Z, Li J, Liu H et al (2022) Transformer for single image super-resolution. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). IEEE, pp 456–465

  83. Zhang X, Zeng H, Guo S et al (2022) Efficient long-range attention network for image super-resolution. 17th European Conference on Computer Vision (ECCV), vol 13677. Springer, pp 649–667

  84. Wang Z, Bovik AC, Sheikh HR et al (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Proc 13:600–612

    Article  Google Scholar 

  85. Ma C, Yang C-Y, Yang X et al (2017) Learning a no-reference quality metric for single-image super-resolution. Comput Vis Image Underst 158:1–16

    Article  Google Scholar 

  86. Zhang L, Zhang L, Mou X et al (2011) FSIM: A feature similarity index for image quality assessment. IEEE Trans Image Proc 20:2378–2386

    Article  MathSciNet  MATH  Google Scholar 

  87. Zhang R, Isola P, Efros AA et al (2018) The unreasonable effectiveness of deep features as a perceptual metric. 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 586–595

  88. Zhang Y, Tian Y, Kong Y et al (2018) Residual dense network for image super-resolution. 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 2472–2481

  89. Zareapoor M, Celebi ME, Yang JJSPIC (2019) Diverse adversarial network for image super-resolution. Signal Proc: Image Commun 74:191–200

    Google Scholar 

  90. Gao G, Wang Z, Li J et al (2022) Lightweight bimodal network for single-image super-resolution via symmetric cnn and recursive transformer. Thirty-First International Joint Conference on Artificial Intelligence (IJCAI), pp 913–919

Download references

Acknowledgements

This work was supported by the following grants: National Natural Science Foundation of China: 62276088, 62102129, Natural Science Foundation of Hebei Province: F2021202030.

Author information

Authors and Affiliations

  1. School of Electronics and Information Engineering, Hebei University of Technology, Tianjin, 300401, China

    Ming Yu & Jiecong Shi

  2. School of Artificial Intelligence, Hebei University of Technology, Tianjin, 300401, China

    Ming Yu, Xiaoke Hao & Gang Yan

  3. Technical College for the Deaf, Tianjin University of Technology, Tianjin, 300384, China

    Cuihong Xue

Authors
  1. Ming Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiecong Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cuihong Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoke Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gang Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cuihong Xue.

Ethics declarations

Conflicts of interests

All authors declared that we have no conflicts of interests to this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, M., Shi, J., Xue, C. et al. A review of single image super-resolution reconstruction based on deep learning. Multimed Tools Appl (2023). https://doi.org/10.1007/s11042-023-17660-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-023-17660-4

Keywords

Search

Navigation