Skip to main content
SpringerLink
Log in

Video super-resolution reconstruction based on correlation learning and spatio-temporal nonlocal similarity

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A novel video super-resolution reconstruction algorithm based on correlation learning and spatio-temporal nonlocal similarity is proposed in this paper. Objective high-resolution (HR) estimates of low-resolution (LR) video frames can be obtained by learning LR-HR correlation mapping and fusing the spatio-temporal nonlocal similarity information between video frames. First, the LR-HR correlation mapping between LR and HR patches is established based on semi-coupled dictionary learning. With the aim of improving algorithm efficiency while guaranteeing super-resolution quality, LR-HR correlation mapping is performed only for the salient object region, and then an improved visual saliency-based nonlocal fuzzy registration scheme using the pseudo-Zernike moment feature and structural similarity is proposed for spatio-temporal similarity matching and fusion. Visual saliency and self-adaptive regional correlation evaluation strategies are used in spatio-temporal similarity matching to improve algorithm efficiency further. Experimental results demonstrate that the proposed algorithm achieves competitive super-resolution quality compared to other state-of-the-art algorithms in terms of both subjective and objective evaluations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Chang H, Yeung D, Xiong Y (2004) Super-resolution through neighbor embedding. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 275–282

  2. Chen J (2012) Video super-resolution using generalized gaussian markov random fields. Signal Process Lett 19(2):63–66

    Article  Google Scholar 

  3. Chen J, Nunez-Yanez J, Achim A (2012) Video super-resolution using generalized gaussian Markov random fields. IEEE Signal Process Lett 19(2):63–66

    Article  Google Scholar 

  4. Dong C, Change Loy C, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. European Conference on Computer Vision (ECCV), pp. 184–199

  5. Dong W, Zhang L, Lukac R, Shi G (2013) Sparse representation based image interpolation with nonlocal autoregressive modeling. IEEE Trans Image Process 22(4):1382–1394

    Article  MathSciNet  Google Scholar 

  6. Dowson N, Salvado O (2011) Hash nonlocal means for rapid image filtering. IEEE Trans Pattern Anal Mach Intell 33(3):485–499

    Article  Google Scholar 

  7. Gao X, Wang Q, Li X (2011) Zernike-moment-based image super resolution. IEEE Trans Image Process 20(10):2738–2747

    Article  MathSciNet  Google Scholar 

  8. Gao X, Zhang K, Tao D et al (2012) Image super-resolution with sparse neighbor embedding. IEEE Trans Image Process 21(7):3194–3205

    Article  MathSciNet  Google Scholar 

  9. Giachetti A, Asuni N (2012) Real-time artifact-free image upscaling. IEEE Trans Image Process 21(4):2361–2369

    Article  MathSciNet  Google Scholar 

  10. He L, Qi H, Zaretzki R (2013) Beta process joint dictionary learning for coupled feature spaces with application to single image super resolution. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 345–352

  11. Heng S, Liang T, Wu Y et al (2012) Spatially adaptive block-based super-resolution. IEEE Trans Image Process 21(3):1031–1045

    Article  MathSciNet  Google Scholar 

  12. Jiang J, Hu R, Han Z, Lu T (2014) Efficient single image super-resolution via graph-constrained least squares regression. Multimed Tools Appl 72:2573–2596

    Article  Google Scholar 

  13. Jordi S, Axel K, Siegfried S (2013) Patch-based spatio-temporal super-resolution for video with non-rigid motion. Signal Process Image Commun 28(5):483–493

    Article  Google Scholar 

  14. Lin D, Tang X (2005) Coupled space learning of image style transformation. International Conference on Computer Vision (ICCV), pp. 1699–1706

  15. Lin Z, Zhang L, Mou X, Zhang D (2011) FSIM: a feature similarity index for image quality assessment. IEEE Trans Image Process 20(8):2378–2386

    Article  MathSciNet  Google Scholar 

  16. Liu C, Sun D (2014) On Bayesian adaptive video super resolution. IEEE Trans Pattern Anal Mach Intell 36(2):346–360

    Article  Google Scholar 

  17. Lu Z, Lu X, Yang L (2014) A local structure adaptive super-resolution reconstruction method based on BTV regularization. Multimed Tools Appl 71:1879–1892

    Article  Google Scholar 

  18. Maalouf A, Larabi M-C (2012) Colour image super-resolution using geometric grouplets. IET Image Process 6(2):168–180

    Article  MathSciNet  Google Scholar 

  19. Mudenagudi U, Banerjee S (2011) Space-time super-resolution using graph-cut optimization. IEEE Trans Pattern Anal Mach Intell 33(5):995–1008

    Article  Google Scholar 

  20. Protter M, Elad M, Takeda H (2009) Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Trans Image Process 18(1):349–366

    MathSciNet  Google Scholar 

  21. Su K, Tian Q, Xue Q, Sebe N, Ma J (2005) Neighborhood issue in single-frame image super-resolution. IEEE International Conference on Multimedia and Expo (ICME), pp. 1–4

  22. Timofte R, De V, Van Gool L (2013) Anchored neighborhood regression for fast example-based super-resolution. International Conference on Computer Vision (ICCV), pp. 1920–1927

  23. Timofte R, De V Smet, Van L Gool (2014) A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution. Asian Conference on Computer Vision (ACCV), pp. 111–126

  24. Wang S, Zhang L, Liang Y, Pan Q (2012) Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2216–2223

  25. Yang J, Lin Z, Cohen S (2013) Fast image super-resolution based on in-place example regression. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1059–1066

  26. Yang C-Y, Ma C, Yang M-H (2014) Single-image super-resolution: a benchmark. European Conference on Computer Vision (ECCV), pp. 372–386

  27. Yang M-C, Wang YCF (2013) A self-learning approach to single image super-resolution. IEEE Trans Multimed 15(3):498–508

    Article  Google Scholar 

  28. Yang J, Wang Z, Lin Z, Cohen S, Huang T (2012) Coupled dictionary training for image super-resolution. IEEE Trans Image Process 21(8):3467–3478

    Article  MathSciNet  Google Scholar 

  29. Yang J, Wang Z, Lin Z, Shu X, Huang T (2012) Bilevel sparse coding for coupled feature spaces. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2360–2367

  30. Yang J, Wright J, Huang T, Ma Y (2010) Image super resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873

    Article  MathSciNet  Google Scholar 

  31. Zhou F, Yang W, Liao Q (2012) Single image super-resolution using incoherent sub-dictionaries learning. IEEE Trans Consum Electron 58(3):891–897

    Article  Google Scholar 

  32. Zhu W, Liang S, Wei Y, Sun J (2014) Saliency optimization from robust background detection. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2814–2821

  33. Zhu Y, Zhang Y, Yuille AL (2014) Single image super-resolution using deformable patches. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2917–2924

Download references

Acknowledgments

This work was supported by National Basic Research Program of China (973 Program) 2012CB821200 (2012CB821206) and the National Natural Science Foundation of China (No. 61320106006, No.61532006, No. 61502042).

Author information

Authors and Affiliations

  1. Beijing Key Laboratory of Intelligent Telecommunication Software and Multimedia, School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Meiyu Liang, Junping Du & Linghui Li

Authors
  1. Meiyu Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junping Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linghui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junping Du.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liang, M., Du, J. & Li, L. Video super-resolution reconstruction based on correlation learning and spatio-temporal nonlocal similarity. Multimed Tools Appl 75, 10241–10269 (2016). https://doi.org/10.1007/s11042-015-2952-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-2952-3

Keywords

Search

Navigation