Improved L0 Gradient Minimization with L1 Fidelity for Image Smoothing

Edge-preserving image smoothing is one of the fundamental tasks in the field of computer graphics and computer vision. Recently, L0 gradient minimization (LGM) has been proposed for this purpose. In contrast to the total variation (TV) model which employs the L1 norm of the image gradient, the LGM model adopts the L0 norm and yields much better results for the piecewise constant image. However, as an improvement of the total variation (TV) model, the LGM model also suffers, even more seriously, from the staircasing effect and is not robust to noise. In order to overcome these drawbacks, in this paper, we propose an improvement of the LGM model by prefiltering the image gradient and employing the L1 fidelity. The proposed improved LGM (ILGM) behaves robustly to noise and overcomes the staircasing artifact effectively. Experimental results show that the ILGM is promising as compared with the existing methods.


Introduction
Image smoothing aims at removing the insignificant details and preserving salient structure such as edges, there are many applications of image smoothing in computer graphics and image processing. However, there is always a dilemma for the smoothing algorithms to simultaneously remove details and preserve edges. For example, the linear Gaussian filter can smooth images effectively but blur the edges seriously. The objective of almost all methods focuses on how to get a tradeoff between flattening details and preserving sharp edges and there have been a plenty of works devoted to this task. Bilateral filter is one of such works and it is simple and effective in removing noise-like structures [1]. Soon after its debut in 1998, there has been a flurry of extensions such as [2][3][4][5][6] and its accelerated variants [7][8][9]. A comprehensive review of the bilateral filter was presented in [10]. Anisotropic diffusion [11] is another category of filter aiming at suppressing noise while preventing salient edges. Since its introduction, there has been a great deal of researches devoted to the theoretical and practical understanding of this model for image smoothing [12][13][14][15][16][17][18][19][20]. For example, Black et al. proposed the robust anisotropic diffusion model based on robust statistics [12], Weickert proposed the structure tensor model based diffusion [13] [14], Sochen et al. proposed the manifold model based diffusion [15], and modified Perona-Malik model based on directional Laplacian was proposed in [16]. The fourth-order partial differential equation models were also proposed, such as [17][18][19][20]. Total variation (TV) takes the form of an unconstrained optimization model [21], where the desired image is obtained as the minimizer to a certain functional that contains both regularization and fidelity terms. The TV model employs the L 1 norm of the image gradient as the regularization term and behaves more stably than the anisotropic diffusion [11]. Therefore, the TV model was studied and applied widely for image smoothing [22][23][24][25][26][27][28][29][30]. However, the TV-based approach suffers from staircasing effect. There are also other works aiming at image smoothing, such as the local extremes model by Subr et al. [31], the guided filter model by He et al. [32] and the weighted least squares method [33].
Recently, Xu et al. proposed an improvement of the TV model by replacing the L 1 norm of the image gradient with the L 0 norm, i.e., the L 0 gradient minimization (LGM) [34]. The LGM model performs more effectively than the TV model for piecewise constant images, however, just as the TV model, the LGM model also suffers, even more seriously, from the staircasing effect. Since the LGM model counts the number of non-zero gradients in the result, it is not robust to noise. In order to overcome these drawbacks of the LGM model, an improved LGM (ILGM) mode is proposed by prefiltering the image gradient during the iteration in the LGM model. The L 1 fidelity is also adopted since it is more robust than the L 2 one when erroneous measurements exist [35] [36]. Similar to LGM [34], the alternating minimization (AM) algorithm [37] is employed for the ILGM model by introducing auxiliary variables; the AM algorithm also yields global optimal result for the ILGM model. Experiment and comparison show that the ILGM model outperforms the LGM model on staircase suppression and noise robustness.
The rest of paper is organized as follows. Section 2 reviews the L 0 gradient minimization (LGM), Section 3 introduces the improved L 0 gradient minimization (ILGM) with L 1 fidelity. In section 4, some experimental results are reported and analyzed. Finally, we conclude our work in Section 5.
LGM: L 0 Gradient Minimization The LGM model [34] employs an L 0 penalty term to directly measure gradient sparsity and has a strong ability to preserve edges. Let f be the input image and U the smoothed one, the gradient of image U at pixel p is denoted by rU p = (@ x U p ,@ y U p ) T . L 0 gradient measure is expressed as: Where #{} denotes the number of pixels of which the gradient is not zero. The term E(U) serves as the regularization term and is combined with a general data constraint term that makes the result U be structurally similar to the input image f. The energy function of the LGM is defined as follows: where λ is a non-negative parameter directly controlling the weight of the regularization term. However, this regularization term is difficult to optimize because the value of E(U) may range from tens to thousands and it is non-convex and non-derivative. To solve the problem, the AM algorithm [37] is employed and two auxiliary variables h p and v p are introduced to replace the @ x U p and @ y U p respectively. The specific objective function is expressed as Where @ x U p and @ y U p are replaced by h p and v p respectively in E(U), and β is a parameter to control the similarity between the auxiliary variables and their corresponding gradients.

ILGM: Improved L 0 Gradient Minimization
Keeping in mind that the energy function for the LGM model is reformulated as Eq (3) by introducing two auxiliary variables h and v. h and v are then estimated from the image gradient (@ x U,@ y U) by exerting an L 0 norm constraint. However, when there is false gradient in the image caused by noise or inhomogeneity, h and v will deviate from the correct values. Based on this consideration, we propose to pre-filter the image gradient (@ x U,@ y U) by an edge-preserving filter when estimating h and v. Since the L 1 fidelity is more robust to outliers than the L 2 one, we also adopt the L 1 fidelity. As a result, the proposed ILGM model is formulated as follows, Where the (h M , v N ) is estimated from an optimized version (M,N) of the image gradient (@ x U,@ y U) when solving Eq (4). The LGM model is employed to calculate (M,N) from (@ x U,@ y U) as follows, In order to handle the L 1 fidelity, the variable W is introduced to denote the difference between U and f. Then, the functional in Eq (4) is rewritten as follows, The numerical implementation of Eqs (5) and (6) is identical to the LGM model, so the alternating minimization algorithm [37] is employed by introducing other auxiliary variables to approximate rM and rN respectively. As for Eq (7), it will be split into three subproblems to alternatively minimize W, (h M ,v N ) and U by using the AM algorithm [37], which lead to an global optimization procedure. (7) as Through the matrix calculus, we can get the unique solver, and the solution can be expressed as, where the convention 0Á(0/0) = 0 is followed, and (U p − f p )/|U p − f p | denotes the sign function. α is the parameter and set to 0.1 in our experiments. Input: image f, λ, β max , α, and κ Initialization: U = f, the current parameter β 0 = λ, α = 0.1, κ = 2, β max = 1.0e5, i = 0 With U (i) , solve for W (i) in Eq (9) With U (i) , solve for M (i) and N (i) in Eqs (5) and (6) With M (i) and N (i) , solve for h ðiÞ M , v ðiÞ N in Eq (11)   Given U and W, we first estimate M and N from (@ x U,@ y U) using Eqs (5) and (6) respectively. Then, the objective function for (h M ,v N ) reads where E(h M , v N ) denotes the number of non-zero elements in |h M | + |v N |. This problem is similar to Subproblem 2 in the numerical implementation of LGM model [34], and the strategy there is employed to obtain (h M ,v N ). After some manipulation, the estimate of (h M , v N ) is given by Eq (12) is quadratic and thus has a global minimum even by gradient decent. Alternatively, it can be efficiently solved using Fast Fourier transform (FFT), which yields solution where F(Á) is the FFT operator and F(Á) Ã denotes the complex conjugate. The whole algorithm is summarized in Algorithm 1. Parameter β is automatically adapted in iterations starting from a small value β 0 , it is multiplied by κ each time.

Experimental Results
In this section, we will demonstrate the performance of the proposed method and make a comparison with several state-of-the-art methods including Bilateral filtering(BLF) [1], Weighted least square method(WLS) [33] (http://www.cs.huji.ac.il/~danix/epd/wlsFilter.m), Total variation (TV) [21] (The minimization of TV is solved using Chambolle's method in Ref. [29], when a color image is encountered, a generalization of Chambolle's method in Ref. [30] is employed.), and the LGM model [34]. Several natural images are adopted as test images, and we focus on the performance of noise robustness, edge preservation, and staircasing effect. To get the best results, the parameters for these methods are manually tuned and presented in the figure captions.
The noisy color image coined by Farbman et al. [33] is employed to test the noise robustness of the proposed ILGM model. The results are presented in Fig 1. Since a prefiltering process is employed for the ILGM, the λ is smaller than that for the original LGM model. However, the proposed ILGM model is not sensitive to λ. For example, Fig 1F, 1G and 1H are the results with different λ, which are almost indistinguishable. Since the numerical implementation or parameters of the associated methods are not presented in [34], the results in Fig 1 are slightly different from those in [34], however, similar results are reported in [36]. Fig 2 shows another example. In this example, the ILGM method also yields a much better result. These observations demonstrate the noise robustness of ILGM method. LGM model smoothed out the texture clearly, but the edges are merged, see Fig 4E. On the other hand, the ILGM method preserved the sharp edges very well while smoothing out the texture clearly. Fig 5A shows an image of texture. In this image, we also aim at smoothing out the texture while preserving the edges. We first smoothed the image using LGM and ILGM respectively, then detected the edges in the results. The smoothed images and edge detection results are shown in Fig 5. One can conclude that the ILGM model performs much better than the LGM model from this example.
It is well-known that the TV model suffers from the staircasing effect [21] [23]. Although the LGM model replaces the L 1 norm in the TV model with the L 0 norm, it also suffers from this notorious effect [34]. However, the proposed ILGM model can conquer this staircasing effect effectively. Fig 6 shows an example of this case. In this example, the wood texture image suffers from inhomogeneity. When smoothing the texture, the LGM model yields staircases due to intensity inhomogeneity, see (b) and the close-up (e). However, the proposed ILGM model yields a much smoother result, see (c) and the close-up (f), and the wood texture is also removed. This example manifests that the ILGM model outperforms the LGM model on staircase suppression.

Conclusion
Recently, the L 0 gradient minimization (LGM) is proposed for image smoothing and performs effectively for piecewise constant images. However, it suffers from staircasing effect and is not robust to noise. In this paper, we presented an improved LGM (ILGM) model by prefiltering the image gradient and employing L 1 fidelity. The proposed ILGM model behaves robustly to noise and overcomes the staircase artifact effectively. Experimental results and comparison with several state-of-the-art methods show that the ILGM model is promising. In the future, we will expand our method to find more applications.