Special Section on CAD & Graphics 2021Blind motion deblurring via sparse representation
Graphical abstract
Introduction
Motion blur is caused by the movement of objects in a scene or camera. In addition to a severe reduction in visual quality, the distortion caused by blur can also result in a decline in the performance of many visual tasks [1]. Although computational imaging has made great strides in the past few years, it is still a challenge to deal with motion blur in captured content.
Eliminating distortion from blurred images, that is, image deblurring, is a classical inverse problem with ill-posed nature. Most algorithms can model image blurring by shaking the camera, such as rotation and translation. However, the real blur is not only camera shaking, but also dynamic scene blur such as object motion and abrupt variations in depth. Therefore, the problem of motion blur is a difficult one in computer vision. When the blur kernel remains invariant in space, it is called the space-invariant model of the image blind deconvolution [2], and in that condition, the blurry image [3] can be easily expressed aswhere denotes the 2D convolution operator, is the latent clean image, is the blur kernel, and is the additive white Gaussian noise (AWGN) with noise level . It can be seen that blind deconvolution should estimate both and from a blurry image making it remain a very challenging problem after decades of studies.
Traditional methods usually apply various constraints to fuzzy model features (for example, uniformity/ unevenness/ depth perception), and use different natural image priors to regularize the solution space. Over the past decade, significant progress has been made in the area of deblurring, with the main effort being devoted to design priors to easy restore the underlying non-distorted images and camera trajectories [4], [5]. Some detailed study of the blind solution fuzzy algorithm can be found in [6]. Recently, several hybrid algorithms have been proposed which combined machine learning and optimization framework, some using a convolution neural network(CNN) to estimate the blur kernel and then apply it to alternative optimization framework to restore latent images [7], [8], some using parameter training method to refine optimization function [9]. Most of these methods involve heuristic parameter adjustments and expensive calculations. In addition, the simplified assumptions of the blur model usually hinder its performance in content examples. In actual situations, blurring is much more complicated than modeling and will be entangled with the image processing pipeline in the camera, which limits the ability to deblur in the scenarios.
Different from the aforementioned methods, end-to-end systems based on deep learning methods which is used to deblur are developed. A method of using fully convolutional CNN to directly estimate potential sharp images was proposed in [10], and was adopted by recent work to further advance. By bypassing the iterative optimization stage involving hand-designed motion model fitting, they provide the advantage of achieving universal dynamic scene deblurring at low latency. Multi-scale residual networks are proposed in [10], [11], [12], which aggregates features in a coarse-to-fine manner and simultaneously displays benefits of selective parameter sharing and cycle layers. Paper [13] proposed a multi-patch hierarchical network and stacked its copies along the depth to achieve the latest performance. Recently, leveraging the fact that motion blur is essentially an aggregation of various spatially varying transformations of the image, K. Purohit and A. N. Rajagopalan [14] proposed a network that implicitly adapts to the location and direction of such motion.
Inspired by Jiang et al. [15], we propose an edge extraction block to fix the edge ambiguity in solving blind deblurring problem. And we use an auto-encoder with asymmetric metric with skipped connections to capture statistical information of latent images. Compared with the existing technology, it has three advantages: First, it combines traditional methods and deep learning. We have improved the original multi-scale recurrent network (SRN) architecture, which still contains a 3-level encoder-decoder architecture, but we have added an edge extraction module after the output of each level. sparse expression is introduced in the edge extraction module due to its ability to maintain image features to better extract the edge; second, it incorporates the edge extraction module into the blind deblurring network for the first time, so that the network can better maintain the prominent structure of the image during the deblurred process. In addition, the network uses the attention mechanism of location and channel to better extract similar features (such as blur) in space and channel to reconstruct high-quality clear images.
A comprehensive comparison with the latest technology of image deblurring benchmarks proves the effectiveness of our architecture. The main contributions of our work are:
- •
We propose an updated scale-recurrent network(SRN) architecture by adding an edge extraction module to process the output intermediate results.
- •
We propose an edge extraction block, which embeds the norm into network architecture for the first time. The block uses the sparsity of the norm of the image to maintain the significant structure of the image.
- •
We propose a dual-attention mechanism to fully utilize the dependency in both spatial and channel domain of the image.
- •
Extensive experiments have been conducted on dynamic scene deblurring benchmarks which show that the edge extraction block realized by the L0 norm can help enhance the salient structure of the image, and has a certain interpretability for the edge preservation ability in final results.
Section snippets
Related works
In the past few decades, blind image deblurring has been the subject of extensive research. In this section, we briefly summarize related work, including optimization-based blind deconvolution and blind deblurring methods based on deep learning.
Deblurring method combining SRN and sparse representation
The proposed model is based on a scale-recurrent architecture that includes three levels to handle multi-scale image patches. We first crop the image into 256256 in random, and then downsample it to 128128 and 6464 to obtain the image pyramid, and use the blurred images of different scales as the input of each level of the network, then the output is the corresponding clear image. In order to make better use of the relationship among blur features, we propose spatial and channel attention
Experimental results
In this section, we mainly introduce the implementation details of our experiment, the comparison with SOTA methods on GOPRO datasets, and the performance of our methods in processing real blur image.
Conclusion
In this article, we propose an effective end-to-end network that can restore blurred images to clear images. The norm has sparse characteristics, which enables feature selection. Introducing the norm into the SRN network can help restore the edge information of the blurred image, that is, the salient structure, thereby reducing the problem of artifacts or unclear edges in the predicted image. Based on the original SRN, the end-to-end network proposed in this paper improves its ability to
CRediT authorship contribution statement
Menghang Li: Conceptualization, Methodology, Software, Formal analysis, Writing - original draft, Visualization. Shanshan Gao: Conceptualization, Writing - review & editing, Project administration, Funding acquisition, Supervision. Chenhao Zhang: Validation, Visualization, Supervision. Minfeng Xu: Writing - review & editing, Supervision. Caiming Zhang: Writing - review & editing, Supervision.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgment
This work was supported in part by National Natural Science Foundation of China (U1909210, 61772309, 61902217), Natural Science Foundation of Shandong Province (ZR2020MF037, ZR2019MF016, ZR2019MF051), Key Research and Development Project of Shandong Province (2019GGX101007), Planning Foundation of Education Ministry (20YJA870013), Introduction and Education Plan of Young Creative Talents in Colleges and Universities of Shandong Province.
References (47)
- Igor V, Ayan C, Gregory S. Examining the impact of blur on recognition by convolutional...
- et al.
Restoring images degraded by spatially variant blur
SIAM J Sci Comput
(1998) - et al.
Blind image deconvolution
Signal Process Mag IEEE
(2013) - et al.
Image deblurring via extreme channels prior
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) - et al.
Non-uniform blind deblurring by reblurring
IEEE international conference on computer vision
(2017) - et al.
A comparative study for single image blind deblurring
IEEE conference on computer vision and pattern recognition
(2016) A neural approach to blind motion deblurring
European conference on computer vision
(2016)- et al.
Learning to deblur
IEEE Trans Pattern Anal MachIntell
(2016) - Jiangxin D, Jinshan P, Deqing S, Zhixun S, Hsuan YM. Learning data terms for non-blind...
- et al.
Deep multi-scale convolutional neural network for dynamic scene deblurring
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017)
Scale-recurrent network for deep image deblurring
2018 IEEE/CVF conference on computer vision and pattern recognition
Dynamic scene deblurring with parameter selective sharing and nested skip connections
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Deep stacked hierarchical multi-patch network for image deblurring
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Region-adaptive dense network for efficient motion deblurring
Proc AAAI Conf ArtifIntell
A simple pooling-based design for real-time salient object detection
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Total variation blind deconvolution
IEEE Trans Image Process A Publ IEEE Signal Process Soc
Removing camera shake from a single photograph
ACM Trans Graph
High-quality motion deblurring from a single image
ACM Trans Graph
Blind image deblurring with local maximum gradient prior
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Graph-based blind image deblurring from a single photograph.
IEEE Trans Image Process A Publ IEEE Signal Process Soc
Fast motion deblurring
ACM Trans Graph
Unnatural l0 sparse representation for natural image deblurring
IEEE conference on computer vision and pattern recognition
Cited by (7)
Dynamic scene deblurring via receptive field attention generative adversarial network
2023, Computers and Graphics (Pergamon)Image motion deblurring via attention generative adversarial network
2023, Computers and Graphics (Pergamon)An Algorithm for the Recognition of Motion-Blurred QR Codes Based on Generative Adversarial Networks and Attention Mechanisms
2024, International Journal of Computational Intelligence SystemsVeracious Interpolated Measure of Angle and Length for Underwater Motion Blurred Images
2023, Fluctuation and Noise LettersImage Deblurring Based on Normalized-weighted Total Variation
2022, Proceedings - 2022 8th Annual International Conference on Network and Information Systems for Computers, ICNISC 2022