Skip to main content
SpringerLink
Log in

Enhanced multi-scale feature progressive network for image Deblurring

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper tackles the problem of single image motion blur removal. Recently methods have achieved state-of-the-art results owe to multi-scale, scale-recurrent and coarse-to-fine architecture, however, the problem of image feature information extraction and information transfer between different stages has not been well solved. In this paper, first, an efficient Enhanced Multi-scale Feature Progressive Network (EMFPNet) was proposed, in order to solve the above problem, a multi-scale feature extraction module is applied in each stage to enrich the spatial features of the maps. Second, introducing a Cross-stage Feature Fusion module to solve the problem of information transmission in different stages. Third, a cross-stage attention mechanism is used to monitor and help the transmission of information. Compared to SOTA method, our method achieve 0.6% and 0.2% improvement in PSNR respectively on GoPro and HIDE datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Aharon M, Elad M, Bruckstein A (2006) K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process 54(11):4311–4322

    Article  MATH  Google Scholar 

  2. Anwar S, Barnes N (2019) Real image denoising with feature attention. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 3155–3164

  3. Brooks T, Mildenhall B, Xue T et al (2019) Unprocessing images for learned raw denoising[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 11036–11045

  4. Chakrabarti A (2016) A neural approach to blind motion deblurring[C]//Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14. Springer International Publishing, pp 221–235

  5. Chan TF, Wong CK (1998) Total variation blind deconvolution[J]. IEEE Trans ImageProcess 7(3):370–375

  6. Chen C, Chen Q, Xu J et al (2018) Learning to see in the dark[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3291–3300.

  7. Dai T, Cai J, Zhang Y, et al. (2019) Second-order attention network for single image super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition : 11065–11074

  8. Dong J, Pan J (2021) Deep outlier handling for image deblurring[J]. IEEE Trans Image Process 30:1799–1811

  9. Fergus R, Singh B, Hertzmann A et al (2006) Removing camera shake from a single photograph[M]//Acm Siggraph 2006 Papers, pp 787–794

  10. Hu Z, Cho S, Wang J et al (2014) Deblurring low-light images with light streaks. Proc IEEE Conf Comput Vis Pattern Recognit:3382–3389

  11. Huang G, Liu Z, Van Der Maaten L et al (2017) Densely connected convolutional networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  12. Jiang Z, Zhang Y, Zou D et al (2020) Learning event-based motion deblurring. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition:3320–3329

  13. Koh J, Lee J, Yoon S (2021) Single-image deblurring with neural networks: a comparative survey[J]. Comput Vis Image Underst 203:103134

  14. Krishnan D, Fergus R (2009) Fast image deconvolution using hyper-Laplacian priors[J]. Adv Neural Inf Process Syst 22:1033–1041

  15. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classifification with deep convolutional neural networks. In Proc Adv Neural Inf Process Syst, pages 1097–1105

  16. Kupyn O, Budzan V, Mykhailych M et al (2018) Deblurgan: Blind motion deblurring using conditional adversarial networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8183–8192

  17. Kupyn O, Martyniuk T, Wu J et al (2019) Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better[C]//Proceedings of the IEEE/CVF international conference on computer vision, pp 8878–8887

  18. Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[C]//2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06). IEEE, 2:2169–2178

  19. Ledig C, Theis L, Huszár F et al (2017) Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690

  20. Levin A, Weiss Y, Durand F et al (2009) Understanding and evaluating blind deconvolution algorithms[C]//2009 IEEE conference on computer vision and pattern recognition. IEEE, pp 1964–1971

  21. Li X, Wu J, Lin Z et al (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining[C]//Proceedings of the European conference on computer vision (ECCV), pp 254–269

  22. Li J, Fang F, Mei K et al (2018) Multi-scale residual network for image super-resolution[C]//Proceedings of the European conference on computer vision (ECCV), pp 517–532

  23. Lin CY, Tao Z, Xu AS, Kang LW, Akhyar F (2020) Sequential dual attention network for rain streak removal in a single image [J]. IEEE Trans Image Process 29:9250–9265

    Article  MATH  Google Scholar 

  24. Lu X, Wang W, Ma C, et al. (2019) See more, know more: Unsupervised video object segmentation with co-attention siamese networks [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3623–3632

  25. Lu X, Wang W, Danelljan M et al (2020) Video object segmentation with episodic graph memory networks[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer International Publishing, pp 661–679

  26. Lu X, Wang W, Shen J et al (2020) Zero-shot video object segmentation with co-attention siamese networks[J]. IEEE Trans Pattern Anal Mach Intell 44(4):2228–2242

  27. Lu X, Ma C, Shen J et al (2020) Deep object tracking with shrinkage loss[J]. IEEE Trans Pattern Anal Mach Intell 44(5):2386–2401

  28. Lu X, Wang W, Shen J et al (2021) Segmenting objects from relational visual data[J]. IEEE Trans Pattern Anal Mach Intell 44(11):7885–7897

  29. Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. Proceedings of the IEEE International Conference on Computer Vision:3397–3405

  30. Mairal J, Elad M, Sapiro G (2007) Sparse representation for color image restoration. IEEE Trans Image Process 17(1):53–69

    Article  MathSciNet  Google Scholar 

  31. Nah S, Hyun Kim T, Mu Lee K (2017) Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3883–3891

  32. Qian R, Tan R T, Yang W et al (2018) Attentive generative adversarial network for raindrop removal from a single image[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2482–2491

  33. Ren D, Zuo W, Hu Q et al (2019) Progressive image deraining networks: A better and simpler baseline[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3937–3946

  34. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, pp 234–241

    Google Scholar 

  35. Rozumnyi D, Oswald M R, Ferrari V et al (2021) Defmo: Deblurring and shape recovery of fast moving objects[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3456–3465

  36. Rudin LI, Osher S, Fatemi E (1992) Nonlinear total variation based noise removal algorithms. Physica D: Nonlin Phenom 60(1–4):259–268

    Article  MathSciNet  MATH  Google Scholar 

  37. Shan Q, Jia J, Agarwala A (2008) High-quality motion deblurring from a single image[J]. ACM Transactions on Graphics (TOG) 27(3):1–10

  38. Shan Q, Jia J, Agarwala A (2008) High-quality motion deblurring from a single image. Acm Trans Graphics (tog) 27(3):1–10

    Article  Google Scholar 

  39. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv:1409.1556

  40. Song H, Xu W, Liu D et al (2021) Multi-stage feature fusion network for video super-resolution[J]. IEEE Trans Image Process 30:2923–2934

  41. Suin M, Purohit K, Rajagopalan AN (2020) Spatially-attentive patch-hierarchical network for adaptive motion deblurring[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3606–3615

  42. Sun J, Cao W, Xu Z et al (2015) Learning a convolutional neural network for non-uniform motion blur removal[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, pp 769–777

  43. Tao X, Gao H, Shen X et al (2018) Scale-recurrent network for deep image deblurring[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8174–8182

  44. Wang P, Sun W, Yan Q et al (2021) Non-uniform motion deblurring with blurry component divided guidance[J]. Pattern Recognit 120:108082

  45. Whyte O, Sivic J, Zisserman A et al (2012) Non-uniform deblurring for shaken images[J]. Int J Comput Vis 98:168–186

  46. Xu L, Zheng S, Jia J (2013) Unnatural l0 sparse representation for natural image deblurring[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1107–1114

  47. Xu L, Ren J S, Liu C et al (2014) Deep convolutional neural network for image deconvolution[J]. Adv Neural Inf Process Syst 27:1790–1798

  48. Yeh CH, Huang CH, Kang LW (2019) Multi-scale deep residual learning-based single image haze removal via image decomposition. IEEE Trans Image Process 29:3153–3167

  49. Yuan Y, Su W, Ma D (2020) Efficient dynamic scene deblurring using spatially variant deconvolution network with optical flow guided training[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3555–3564

  50. Zamir S W, Arora A, Khan S et al (2020) Cycleisp: Real image restoration via improved data synthesis[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2696–2705

  51. Zamir S W, Arora A, Khan S et al (2021) Multi-stage progressive image restoration[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14821–14831

  52. Zhang Y, Li K, Li K et al (2018) Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European conference on computer vision (ECCV), pp 286–301

  53. Zhang Y, Li K, Li K et al (2019) Residual non-local attention networks for image restoration[J]. arXiv preprint arXiv:1903.10082

  54. Zhang H, Dai Y, Li H et al (2019) Deep stacked hierarchical multi-patch network for image deblurring[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5978–5986

  55. Zhang T, Li J, Hua Z (2021) Iterative multi‐scale residual network for deblurring[J]. IET Image Process 15(8):1583–1595

Download references

Funding

This work was supported by the Natural Science Foundation of Shandong Province (No. ZR2019MF050) and the Shandong Province colleges and universities youth innovation technology plan innovation team project under Grant (No. 2020KJN011).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Qingdao University, Qingdao, People’s Republic of China, 266071

    Zhijun Yu, Guodong Wang, Xinyue Zhang & Ziying Wang

Authors
  1. Zhijun Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guodong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinyue Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ziying Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guodong Wang.

Ethics declarations

Conflict of interests

We declare that we have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, Z., Wang, G., Zhang, X. et al. Enhanced multi-scale feature progressive network for image Deblurring. Multimed Tools Appl 82, 21147–21159 (2023). https://doi.org/10.1007/s11042-023-14629-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14629-1

Keywords

Search

Navigation