Depth Image Inpainting method based on sparse gradient prior

This paper solves the inpainting problem of single depth images. depth images are regarded as natural images without texture. Because of the sparsity property of natural images and the textureless property of depth images, we propose a similar group-based sparse model with sparse gradient regularization. For one thing, the similar group-based sparse model can better represent the local smooth and nonlocal self-similarity. For another, the sparse gradient regularization can better represent the textureless properties. The proposed algorithm takes advantage of the properties of depth images. The experimental results show the effect of the proposed algorithm.


Introduction
Depth images are a key research topic in the field of 2D-to-3D technology. However, the acquired depth images are prone to loss of information and require repairing.
At present, image inpainting technologies [1][2][3] are mostly focused on natural and medical images, such as inpainting of old photos and restoration of CT images. Most image inpainting methods can be applied to depth images directly, and relatively few scholars have paid attention to depth images inpainting.
Depth images can be viewed as natural images without texture. Due to the limitations of hardware and scenes, there exist black holes in depth images. With the extensive application of depth images, certain scholars have begun to focus on depth image inpainting. A weighted analysis representation model was proposed that advanced the conventional method in two aspects, namely, task-driven learning and dynamic guidance [3]. In [4], a modification to the morphological closing by a reconstruction algorithm was proposed without correlative color image information, which improved the depth image quality. In [5], Xue et al. proposed the integration of low gradient regularization with the low rank regularization method for depth image inpainting. the proposed method was effective.
The mathematical expression of the depth image inpainting problem is : Because the prior of depth images plays an important role in inpainting, we should take advantage of it. Depth images are approximated as natural images with textureless [5], so we apply a similar group-based sparse model assumption to obtain the rough inpainting result. Each similar group of depth images can be accurately represented based on an effective self-adaptive dictionary learning method. Due to the textureless property, we choose L 0 gradient regularization as the sparse gradient constrain, which is the most suitable measure for textureless [6][7][8]. In summary, we propose a similar group-based sparse model with L 0 gradient regularization for the inpainting of single depth images.

Similar Group-Based Sparse Model
In [9], the basic unit of sparse representation is a similar group. The construction of a similar group is shown below.

Extracting
Matching Stacking . Therefore, according to equation (1), depth image inpainting can be formulated as follows: where G D is the dictionary, G α is the sparse vector,  is the weight parameter and p G α represents the norm. In this paper, p is 0.
The similar group-based sparse model can be directly applied to depth image inpainting. Because of the textureless property, it is reasonable to define the L 0 gradient regularization as the regular term.

Similar Group-Based Sparse Model with L0 Gradient Regularization
As mentioned in section 2, this section introduces the L0 gradient regularization [11][12][13] in the similar group-based sparse model and describes the determination of a suitable solution. The proposed model is formulated as: where 1  and 2  are the weight parameters; 0 || || x  represents the norm of the gradient and can be used to control the number of non-zero gradients.
We use the split Bregman iteration (SBI) method to divide the complex problem into subproblems, which are easy to solve. The first subproblem is updated by: The second subproblem is updated by: The update of is as follows:

The x Subproblem
Given G α , the solution is as follows: The TV norm is usually a relaxation of the L 0 norm, but this operation will smooth the boundary of the regions [13,14]. In this paper, we introduce the region fusion criterion [5,10] , equation (7) can be written as: where  represents the missing area in depth images, L represents the number of pixels, i According to [10], the objective function is defined as follows: .
where  is a parameter and i  represents the number of pixels in th i region. Similarly, j  is obtained. Whether the two regions should be fused becomes: where: where ~i  represents the number of pixels in the th i region but not in the missing region  . Similarly, ~j  is obtained. We introduce the region fusion method to solve the L 0 gradient minimization. This method ensures that the solution converges rapidly and is accurate.

The G α
Subproblem Given x , the solution is as follows: . Then, Eqn. (11) becomes: (12) In each iteration, estimates k G r of similar groups k G x are expressed using SVD.
So the formula for solving each similar group is as follows: where ) hard( denotes hard thresholding. Each similar group is solved according to equation (15), which can be integrated to obtain k  G α .

Experiments and Results
In this paper, our experimental platform is MATLAB version 7.0. We use subjective visual effects and objective parameters to compare the results.
In experiment1,there are no perfect depth images for comparison. The objective metrics can be assessed on the basis of one objective parameters: Natural image quality evaluar(NIQE) .In experiment 2, the objective metrics can be assessed on the basis of two objective parameters: PSNR and FSIM .    Subjectively, as show in figure 4 and 5, all four algorithms can meet the visual requirements. The effect of the FOE algorithm is fuzzy, particularly in Experiment (2). The BPFA algorithm can repair the images smoothly, but the algorithm also easily blurs boundaries. The GSR algorithm can improve the effect on boundaries, but the algorithm readily causes blurring in areas where the grayscale value is not very different. To a certain extent, the proposed algorithm improves the inpainting effect.
Objectively, our proposed algorithm is superior to the other three algorithms, as shown in Tables 1  and 2. For different types of images, the objective parameters of our proposed algorithm result in improvements.

Conclusions
The main research problem is the inpainting of single depth images. Similar group-based sparse representation with an L 0 gradient regularization can repair depth images better due to the properties of the depth images. Then, we extend adaptive dictionary learning and region fusion to solve two subproblems. The experiments show that the proposed algorithm obtains improved inpainting effects in terms of subjective visual effects and objective parameters because the algorithm fully utilizes the properties of depth image.