Blind motion deblurring via $L_0$ sparse representation

Highlights

• An updated scale-recurrent network(SRN) architecture is proposed to improve the ability to restore image structure and edge by adding an edge extraction module to process the output intermediate results.

• We propose an edge extraction block, which embeds the L0 norm into network architecture for the first time. The block uses the sparsity of the L0 norm of the image to maintain the significant structure of the image.

• Propose a dual-attention mechanism to fully utilize the dependency in both spatial and channel domain of the image.

• Extensive experiments have been conducted on dynamic scene deblurring benchmarks which show that the edge extraction block realized by the L0 norm can help enhance the salient structure of the image, and has a certain interpretability for the edge preservation ability in final results.

Abstract

The method of blind deblurring based on machine learning can effectively deal with the blurred images in the real world. However, existing multi-level architectures can lead to problems such as the inability to reserve edges, the expected introduction of artifacts and ghosts when deblurring. Multi methods have found that using $L_0$ norm to realize image sparse representation can help keeping main structure of images. In this paper, an edge extraction module based on $L_0$ sparse representation is proposed to preserve the edge of images, which is embedded in a multi-scale recurrent network(SRN). When the current scale transmits information to the next scale, edge enhancement is performed using the edge extraction module. Furthermore, considering the correlation among pixels and the correlation among channels, we introduce dual-attention mechanism into the encoder-decoder structure. The deblurring experiment was carried out on the GOPRO dataset. Comparing with 5 state-of-art methods qualitatively and quantitatively, the experimental results show that the proposed method can better preserve the image edges and effectively avoid the artifact of the image. And the peak signal-to-noise ratio of the proposed method are improved compared with other methods.

Introduction

Motion blur is caused by the movement of objects in a scene or camera. In addition to a severe reduction in visual quality, the distortion caused by blur can also result in a decline in the performance of many visual tasks [1]. Although computational imaging has made great strides in the past few years, it is still a challenge to deal with motion blur in captured content.

Eliminating distortion from blurred images, that is, image deblurring, is a classical inverse problem with ill-posed nature. Most algorithms can model image blurring by shaking the camera, such as rotation and translation. However, the real blur is not only camera shaking, but also dynamic scene blur such as object motion and abrupt variations in depth. Therefore, the problem of motion blur is a difficult one in computer vision. When the blur kernel remains invariant in space, it is called the space-invariant model of the image blind deconvolution [2], and in that condition, the blurry image $y$ [3] can be easily expressed as

$$y=x\otimes k+n$$

where $\otimes$ denotes the 2D convolution operator, $x$ is the latent clean image, $k$ is the blur kernel, and $n$ is the additive white Gaussian noise (AWGN) with noise level $\sigma$. It can be seen that blind deconvolution should estimate both $k$ and $x$ from a blurry image $y,$ making it remain a very challenging problem after decades of studies.

Traditional methods usually apply various constraints to fuzzy model features (for example, uniformity/ unevenness/ depth perception), and use different natural image priors to regularize the solution space. Over the past decade, significant progress has been made in the area of deblurring, with the main effort being devoted to design priors to easy restore the underlying non-distorted images and camera trajectories [4], [5]. Some detailed study of the blind solution fuzzy algorithm can be found in [6]. Recently, several hybrid algorithms have been proposed which combined machine learning and optimization framework, some using a convolution neural network(CNN) to estimate the blur kernel and then apply it to alternative optimization framework to restore latent images [7], [8], some using parameter training method to refine optimization function [9]. Most of these methods involve heuristic parameter adjustments and expensive calculations. In addition, the simplified assumptions of the blur model usually hinder its performance in content examples. In actual situations, blurring is much more complicated than modeling and will be entangled with the image processing pipeline in the camera, which limits the ability to deblur in the scenarios.

Different from the aforementioned methods, end-to-end systems based on deep learning methods which is used to deblur are developed. A method of using fully convolutional CNN to directly estimate potential sharp images was proposed in [10], and was adopted by recent work to further advance. By bypassing the iterative optimization stage involving hand-designed motion model fitting, they provide the advantage of achieving universal dynamic scene deblurring at low latency. Multi-scale residual networks are proposed in [10], [11], [12], which aggregates features in a coarse-to-fine manner and simultaneously displays benefits of selective parameter sharing and cycle layers. Paper [13] proposed a multi-patch hierarchical network and stacked its copies along the depth to achieve the latest performance. Recently, leveraging the fact that motion blur is essentially an aggregation of various spatially varying transformations of the image, K. Purohit and A. N. Rajagopalan [14] proposed a network that implicitly adapts to the location and direction of such motion.

Inspired by Jiang et al. [15], we propose an edge extraction block to fix the edge ambiguity in solving blind deblurring problem. And we use an auto-encoder with asymmetric metric with skipped connections to capture statistical information of latent images. Compared with the existing technology, it has three advantages: First, it combines traditional methods and deep learning. We have improved the original multi-scale recurrent network (SRN) architecture, which still contains a 3-level encoder-decoder architecture, but we have added an edge extraction module after the output of each level. $L_0$ sparse expression is introduced in the edge extraction module due to its ability to maintain image features to better extract the edge; second, it incorporates the edge extraction module into the blind deblurring network for the first time, so that the network can better maintain the prominent structure of the image during the deblurred process. In addition, the network uses the attention mechanism of location and channel to better extract similar features (such as blur) in space and channel to reconstruct high-quality clear images.

A comprehensive comparison with the latest technology of image deblurring benchmarks proves the effectiveness of our architecture. The main contributions of our work are:

• We propose an updated scale-recurrent network(SRN) architecture by adding an edge extraction module to process the output intermediate results.

• We propose an edge extraction block, which embeds the $L_0$ norm into network architecture for the first time. The block uses the sparsity of the $L_0$ norm of the image to maintain the significant structure of the image.

• We propose a dual-attention mechanism to fully utilize the dependency in both spatial and channel domain of the image.

• Extensive experiments have been conducted on dynamic scene deblurring benchmarks which show that the edge extraction block realized by the L0 norm can help enhance the salient structure of the image, and has a certain interpretability for the edge preservation ability in final results.