Skip to main content
SpringerLink
Log in

Noise Robust Video Super-Resolution Without Training on Noisy Data

  • Conference paper
  • First Online:
Image and Graphics (ICIG 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12890))

Included in the following conference series:

  • 2235 Accesses

Abstract

Previous CNN-based video super-resolution (VSR) approaches can not be directly applied to noisy images, otherwise the noise will be enhanced after super-resolution (SR) reconstruction models. Some methods are robust to noise but all of them need to be trained on specific noisy training datasets. In this paper, we propose a noise-robust VSR network which only needs to be trained on the clean images. That is, in our deep network for VSR, the model can appropriately super-resolve noisy images without any training on noisy data. We put forward a non-local spatio-temporal module, which not only achieves motion estimation and compensation, but also improves the robustness of our VSR model to noise. A inter-frame fusion module is further presented to fuse the complementary information from different frames. The experiments conducted on both additive noise and multiplicative noise demonstrate that the proposed method can generate visually and quantitatively high-quality results, superior to state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Buades, A., Coll, B., Morel, J.M.: Nonlocal image and movie denoising. Int. J. Comput. Vis. 76(2), 123–139 (2008)

    Article  Google Scholar 

  2. Caballero, J., et al.: Real-time video super-resolution with spatio-temporal networks and motion compensation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4778–4787 (2017)

    Google Scholar 

  3. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13

    Chapter  Google Scholar 

  4. Haris, M., Shakhnarovich, G., Ukita, N.: Recurrent back-projection network for video super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3897–3906 (2019)

    Google Scholar 

  5. Huang, S., Sun, J., Yang, Y., Fang, Y., Lin, P., Que, Y.: Robust single-image super-resolution based on adaptive edge-preserving smoothing regularization. IEEE Trans. Image Process. 27(6), 2650–2663 (2018)

    Article  MathSciNet  Google Scholar 

  6. Kappeler, A., Yoo, S., Dai, Q., Katsaggelos, A.K.: Video super-resolution with convolutional neural networks. IEEE Trans. Comput. Imaging 2(2), 109–122 (2016)

    Article  MathSciNet  Google Scholar 

  7. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

    Google Scholar 

  8. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)

    Google Scholar 

  9. Lee, O.Y., Lee, J.W., Kim, J.O.: Combining self-learning based super-resolution with denoising for noisy images. J. Vis. Commun. Image Represent. 48, 66–76 (2017)

    Article  Google Scholar 

  10. Lee, T.B., Heo, Y.S.: Single image super resolution using convolutional neural networks for noisy images. In: 2020 International Conference on Information and Communication Technology Convergence (ICTC), pp. 195–199. IEEE (2020)

    Google Scholar 

  11. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

    Google Scholar 

  12. Liu, J., Zhang, W., Tang, Y., Tang, J., Wu, G.: Residual feature aggregation network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2359–2368 (2020)

    Google Scholar 

  13. Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3D residual networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5533–5541 (2017)

    Google Scholar 

  14. Tai, Y., Yang, J., Liu, X., Xu, C.: MemNet: a persistent memory network for image restoration. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4539–4547 (2017)

    Google Scholar 

  15. Tian, Y., Zhang, Y., Fu, Y., Xu, C.: TDAN: temporally-deformable alignment network for video super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3360–3369 (2020)

    Google Scholar 

  16. Vaswani, A., et al.: Attention is all you need. arXiv preprint arXiv:1706.03762 (2017)

  17. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)

    Google Scholar 

  18. Wang, X., Chan, K.C., Yu, K., Dong, C., Change Loy, C.: EDVR: video restoration with enhanced deformable convolutional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)

    Google Scholar 

  19. Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 63–79. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_5

    Chapter  Google Scholar 

  20. Yeganli, F., Nazzal, M., Ozkaramanli, H.: Super-resolution using multiple structured dictionaries based on the gradient operator and bicubic interpolation. In: 2016 24th Signal Processing and Communication Application Conference (SIU), pp. 941–944 (2016). https://doi.org/10.1109/SIU.2016.7495896

  21. Yoo, J.S., Kim, J.O.: Noise-robust iterative back-projection. IEEE Trans. Image Process. 29, 1219–1232 (2019)

    Article  MathSciNet  Google Scholar 

  22. Zhang, K., Gool, L.V., Timofte, R.: Deep unfolding network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3217–3226 (2020)

    Google Scholar 

  23. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 294–310. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_18

    Chapter  Google Scholar 

  24. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018)

    Google Scholar 

  25. Zhou, F., Xia, S.T., Liao, Q.: Nonlocal pixel selection for multisurface fitting-based super-resolution. IEEE Trans. Circ. Syst. Video Technol. 24(12), 2013–2017 (2014)

    Article  Google Scholar 

Download references

Acknowledgements

This work was in part by Guangdong Basic and Applied Basic Research Foundation with No. 2020A1515110884, and in part by Guangdong Basic and Applied Basic Research Foundation with No. 2021A1515011584. The authors would like to thank the editors and reviewers for their constructive suggestions on our work.

Author information

Authors and Affiliations

  1. College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China

    Fei Zhou, Zitao Lu, Hongming Luo, Cuixin Yang & Bozhi Liu

  2. Peng Cheng Laboratory, Shenzhen, China

    Fei Zhou, Zitao Lu, Hongming Luo, Cuixin Yang & Bozhi Liu

  3. Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen, China

    Fei Zhou, Zitao Lu, Hongming Luo, Cuixin Yang & Bozhi Liu

  4. Key Laboratory of Digital Creative Technology, Shenzhen, China

    Fei Zhou, Zitao Lu, Hongming Luo, Cuixin Yang & Bozhi Liu

Authors
  1. Fei Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zitao Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongming Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cuixin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bozhi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Peking University, Beijing, China

    Yuxin Peng

  2. Tsinghua University, Beijing, China

    Shi-Min Hu

  3. Tampere University, Tampere, Finland

    Moncef Gabbouj

  4. Zhejiang University, Hangzhou, China

    Kun Zhou

  5. Technion – Israel Institute of Technology, Haifa, Israel

    Michael Elad

  6. Tsinghua University, Beijing, China

    Kun Xu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhou, F., Lu, Z., Luo, H., Yang, C., Liu, B. (2021). Noise Robust Video Super-Resolution Without Training on Noisy Data. In: Peng, Y., Hu, SM., Gabbouj, M., Zhou, K., Elad, M., Xu, K. (eds) Image and Graphics. ICIG 2021. Lecture Notes in Computer Science(), vol 12890. Springer, Cham. https://doi.org/10.1007/978-3-030-87361-5_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87361-5_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87360-8

  • Online ISBN: 978-3-030-87361-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation