Skip to main content
SpringerLink
Log in
Book cover

German Conference on Pattern Recognition

GCPR 2017: Pattern Recognition pp 203–214Cite as

End-to-End Learning of Video Super-Resolution with Motion Compensation

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10496))

Abstract

Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    We use the implementation from [11]; it differs from [12] in that it performs bilinear interpolation instead of bicubic.

References

  1. Myanmar 60p, Harmonic Inc.: (2014). https://www.harmonicinc.com/resources/videos/4k-video-clip-center

  2. Babacan, S.D., Molina, R., Katsaggelos, A.K.: Variational Bayesian super resolution. IEEE Trans. Patt. Anal. Mach. Intell. (TPAMI) 20(4), 984–999 (2011)

    MathSciNet  MATH  Google Scholar 

  3. Caballero, J., Ledig, C., Aitken, A.P., Acosta, A., Totz, J., Wang, Z., Shi, W.: Real-time video super-resolution with spatio-temporal networks and motion compensation. CoRR abs/1611.05250 (2016). http://arxiv.org/abs/1611.05250

  4. Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2004

    Google Scholar 

  5. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). doi:10.1007/978-3-319-46475-6_25

    Chapter  Google Scholar 

  6. Cheng, M.H., Lin, N.W., Hwang, K.S., Jeng, J.H.: Fast video super-resolution using artificial neural networks. In: International Symposium on Communication Systems, Networks Digital Signal Processing (CSNDSP), pp. 1–4, July 2012

    Google Scholar 

  7. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 38(2), 295–307 (2016)

    Article  Google Scholar 

  8. Dosovitskiy, A., Fischer, P., Ilg, E., Häusser, P., Hazırbaş, C., Golkov, V., Smagt, P., Cremers, D., Brox, T.: FlowNet: learning optical flow with convolutional networks. In: IEEE International Conference on Computer Vision (ICCV) (2015)

    Google Scholar 

  9. Drulea, M., Nedevschi, S.: Total variation regularization of local-global optical flow. In: IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 318–323, October 2011

    Google Scholar 

  10. Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graph. Appl. (ICGA) 22(2), 56–65 (2002)

    Article  Google Scholar 

  11. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    Google Scholar 

  12. Kappeler, A., Yoo, S., Dai, Q., Katsaggelos, A.K.: Video super-resolution with convolutional neural networks. IEEE Trans. Comput. Imaging (TCI) 2(2), 109–122 (2016)

    Article  MathSciNet  Google Scholar 

  13. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654, June 2016

    Google Scholar 

  14. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) abs/1511.04491 (2016)

    Google Scholar 

  15. Ledig, C., Theis, L., Huszar, F., Caballero, J., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  16. Liu, C., Sun, D.: On Bayesian adaptive video super resolution. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 36(2), 346–360 (2014)

    Article  Google Scholar 

  17. Ma, Z., Liao, R., Tao, X., Xu, L., Jia, J., Wu, E.: Handling motion blur in multi-frame super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5224–5232, June 2015

    Google Scholar 

  18. Ranjan, A., Black, M.J.: Optical flow estimation using a spatial pyramid network. CoRR abs/1611.00850 (2016). http://arxiv.org/abs/1611.00850

  19. Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    Google Scholar 

  20. Song, B.C., Jeong, S.C., Choi, Y.: Video super-resolution algorithm using bi-directional overlapped block motion compensation and on-the-fly dictionary training. IEEE Trans. Circuits Syst. Video Technol. (TCSVT) 21(3), 274–285 (2011)

    Article  Google Scholar 

  21. Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2432–2439, June 2010

    Google Scholar 

  22. Takeda, H., Milanfar, P., Protter, M., Elad, M.: Super-resolution without explicit subpixel motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 18(9), 1958–1975 (2009)

    MathSciNet  MATH  Google Scholar 

  23. Tao, X., Gao, H., Liao, R., Wang, J., Jia, J.: Detail-revealing deep video super-resolution. CoRR abs/1704.02738 (2017). http://arxiv.org/abs/1704.02738

  24. Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution as sparse representation of raw image patches. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2008

    Google Scholar 

  25. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 19(11), 2861–2873 (2010)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

We acknowledge the DFG Grant BR-3815/7-1.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany

    Osama Makansi, Eddy Ilg & Thomas Brox

Authors
  1. Osama Makansi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eddy Ilg
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Brox
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eddy Ilg .

Editor information

Editors and Affiliations

  1. University of Basel, Basel, Switzerland

    Volker Roth

  2. University of Basel, Basel, Switzerland

    Thomas Vetter

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1628 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Makansi, O., Ilg, E., Brox, T. (2017). End-to-End Learning of Video Super-Resolution with Motion Compensation. In: Roth, V., Vetter, T. (eds) Pattern Recognition. GCPR 2017. Lecture Notes in Computer Science(), vol 10496. Springer, Cham. https://doi.org/10.1007/978-3-319-66709-6_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-66709-6_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-66708-9

  • Online ISBN: 978-3-319-66709-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation