New Adaptive Regularization Parameter Selection Based on Expected Patch Log Likelihood

Digital images have been applied to various areas such as evidence in courts. However, it always suffers from noise by criminals. This type of computer network security has become a hot issue that can’t be ignored. In this paper, we focus on noise removal so as to provide guarantees for computer network security. Firstly, we introduce a well-known denoising method called Expected Patch Log Likelihood (EPLL) with Gaussian Mixture Model as its prior. This method achieves exciting results in noise removal. However, there remain problems to be solved such as preserving the edge and meaningful details in image denoising, cause it considers a constant as regularization parameter so that we denoise with the same strength on the whole image. This leads to a problem that edges and meaningful details may be oversmoothed. Under the consideration of preserving edges of the image, we introduce a new adaptive parameter selection based on EPLL by the use of the image gradient and variance, which varies with different regions of the image. Moreover, we add a gradient fidelity term to relieve staircase effect and preserve more details. The experiment shows that our proposed method proves the effectiveness not only in vision but also on quantitative evaluation.


Introduction
Computer network security is always a hot topic because it is closely related to our daily life. However, it often suffers from attack. For example, as we all know digital images can be used as a powerful evidence in juridical practice, but criminals may corrupt images to disrupt the line of sight. For solving this, many experts focus on the source of images because it plays an important role in image forensics practice. Investigators may track suspects form images that taken by particular cameras. The most popular image forensic approach is mainly based on sensor pattern noise [Lucas, Fridrich and Goljian (2013)]. The key to this approach is extracting sensor pattern noise by advanced denoising filters. Therefore noise removal becomes a hot issue in computer forensics, and then much effort has been paid to image denoising techniques to obtain the high quality image. A large number of methods have been proposed on image restoration in the past few decades. Sparse representation is rather well-known among them [Dong, Zhang, Shi et al. (2011); Elad and Aharon (2006)]. The main idea of this theory is to consider image patches as the linear combination of some atoms based on a dictionary. The overcomplete dictionary can be ready-made, or it can be trained from image patches. Then low-rank approximation methods show amazing results on denoising [Dong, Shi, Li et al. (2014); Dong, Shi and Li (2013)]. It can be divided into two categories: the nuclear norm minimization methods [Cai, Candes and Shen (2010)], and the low rank matrix factorization approaches. The total variation (TV) [Beck and Teboulle (2009) ;Zheng, Jeon, Zhang et al. (2015); Zhang, Yu, Zheng et al. (2016);Zheng, Ma, Yu et al. (2017)] has always been active in image denoising. Building a proper image model as prior is of great significance in image denoising. So mixture models attract attention in image denoising due to its robustness, especially Gaussian Mixture Model (GMM). Then many meaningful explorations were devoted to learn image priors by GMM [Wang and Morel (2012) ;Yu, Sapiro and Mallat (2012); Zoran and Weiss (2011);Zhang, Liu, Li et al. (2017)]. Inspired by this, some further studies have been proposed [Zheng, Zhou, Jeon et al. (2017)]. These methods have already successfully prove their reliability. As a major feature of the image, edges are full of meaningful structure information. Preserving edges occupies an important position in image denoising. Therefore, many researches on edge-preserving have been applied to noise removal. Anisotropic diffusion is one of the most efficient algorithm. In Perona et al. [Perona and Malik (2002)], the first pioneering model of anisotropic diffusion came into the public's view. The original Perona-Malik diffusion model mainly considered on the gradient information of the image. The basic idea behind the method is that gradient is an ideal indication of the edge. In detail, the classical P-M model smooths strongly at the homogeneous region which always has low gradient and stops diffusion at the inter-region edge that has large gradient. The number of iterations is the key to the success of the denoising result since too much iterations will lead to the loss of edges. Inspired by this, many further studies have been proposed due to its great success [Chao and Tsai (2010); Ma, Shen, Zhang et al. (2017); Masiseli and Gao (2016) ;Xu, Jia, Shi et al. (2016)], now the anisotropic diffusion approach has become a hot topic in image restoration [Wang, Guo, Chen et al. (2013)], image segmentation, texture segmentation, image enhancement [Giboa, Sochen and Zeevi (2004); Dang, Gao, Wang et al. (2015)], image smoothing [Lions, Morel and Coll (1992)] and defect detection [Malarvel, Sethumadhavan, Bhagi et al. (2017)]. However, there still remain tiny problems such as the huge cost of iteration and staircase effect. For solving the former, Tebini et al. [Tebini, Mbarki, Seddik et al. (2016)] proposed a fast and efficient method based on the tangent sigmoid model to obtain a higher speed of convergence. For solving the latter, Guo et al. [Guo, Sun, Zhang et al. (2012)] proposed an adaptive P-M diffusion which combined the original P-M model heat equation. Edgepreserving methods aim to remove noise effectively and preserve inter-region edges at the same time. In this paper, we focus on the edge-preserving, and propose a new method of adaptive regularization parameter selection. Moreover, we add a gradient-fidelity term to relieve the staircase effect and preserve more details of image. In this paper, we focus on the noise removal method used in computer forensics. Good quality denoising methods enable experts to extract sensor pattern noise better and keep the accuracy of camera forensics. The noise removal method aims to enhance the security of computer identification. This paper is organized as follows: Section 2 reviews the GMM model, and then introduces original EPLL algorithm briefly. Section 3 introduces the proposed method. In Section 4, we show experiment results as the proof of effectiveness of the proposed method. Section 5 makes the conclusion and summarizes the paper.

Original EPLL model 2.1 Normal denoising model
Assuming a noisy image 0 u u v = + corrupted by additive white Gaussian noise. Our purpose is to achieve a pure image from the observation. In this paper, we consider it as a Maximum a Posteriori (MAP) problem. MAP approach is based on Bayesian theory that maximizes the posteriori probability to obtain the clean image: The first term is a data fidelity term, and the second term is the log of the prior. First of all, we need to learn prior knowledge. We consider the distribution ( ) p u as the prior model which is independent of the data observation. Obviously, the distribution ( ) p u directly affects the result, so how to build the prior model has become a hot issue. As we all know, image is a type of high-dimensional data with complex distribution, so directly learning the whole image prior is a giant challenge. To solve this, we can extract overlapping image patches with low dimension from the whole image, and naturally our goal is to make every patch is likely under our prior in statistics.

Gaussian mixture model (GMM)
Usually we can assume that one patch can be described by a specific distribution function. However, as is mentioned above, image is consisted of numerous independent patches that we can hardly describe it just by a single distribution. As is shown in the Fig. 1, (b) represents gray distribution histogram of (a), abscissa represents the gray value, and ordinate represents the frequency of the gray value. Several peaks in (b) means one image can be considered as the superposition of several different Gaussian distributions, then naturally we call it Gaussian Mixture Model (GMM). GMM is always considered as the image prior instead of a single gaussian distribution.
In this paper, we train the GMM model by a set of clean natural image patches . For a given patch i a , we define the GMM distribution as the following: where K is the number of mixture components, k π is the mixing weight for each mixture component, k µ and k Σ denote the corresponding mean vector and covariance matrix.
Moreover, k π denotes the priori probability, and In the E-step, we calculate the posterior probability for the component k as the following: Then, for all patches, we get the likelihood function as the following: In the M-step, to obtain parameters Θ , let , then we get: Eq. (7) means the mean vector for each component can be estimated by weighted average method, the weight is the posterior probability.
Similarly, let , then we get: For mixing weights k π , considering the form of Lagrange, leading to the following: Let the derivative of the Eq. (9) be 0 Eq. (11) means the mixing weight for each component is determined by the averaging posterior probability. (2011)] is an image restoration method that builds a model from patches to the whole image. The fundamental idea of EPLL is to make every patch have the highest log likelihood probability under our prior. EPLL is defined as:

Expected patch log likelihood EPLL [Zoran and Weiss
where is an operator that extracts the patch from the whole image at the i-th position, obviously is the extracted patch we need, denotes the regularization parameter and log means the likelihood of the i-th patch under the prior. Our purpose is to obtain a pure image by minimizing the cost function as the following: Similar to Eq. (1), Eq. (13) has two terms: the data fidelity term and the regularization term. We should point out that EPLL (u) denotes the total log probabilities of all overlapping patches, rather than the whole image. Overlapping patches means each pixel will appear several times in different patches, so each patch will be denoised independently and each pixel will be average according to its frequency.

Proposed method
Regularization parameter affects the quality of the restored image directly. As is shown in Chao et al. [Chao and Tsai (2010)], when it comes to image denoising, the adaptive regularization parameter always plays a central role. How to maintain a balance between preserving edges of the image and denoising remains a big problem for us to solve. So in this paper, a method with adaptive selection of regularization parameter for a better restored image is proposed as the following: where the second term is the gradient fidelity term, u ∇ denotes the image gradient, G σ is a Gaussian filter operator, ( ) where 1 k and δ are constants, 2 k is the threshold value, is the gradient of pixel ( ) where denotes the time step and is the unit matrix. Repeat such process for few iterations, usually 4 or 5. At each iteration, we fix the value of . There are two normal ways for us to apply to the choice of values. One option is to optimize the value by a series of training sets, the other way is to try to estimate the intensity of noise from the current image, then set β .
Generally speaking, the proposed method can be implemented as the following:    As a comparison, our proposed method preserves edges better and keeps more details of the image. As in shown in Figs. 2 and 3, the original EPLL method may denoise well, but it also leads to a problem that the loss of edges and details in some regions. Fig. 2(c) shows that details such textures are fuzzy in EPLL result, while clearer textures can be seen in proposed method. The gradient fidelity term proves its validity in preserving image details. Fig. 3(c) shows that the edge of house is barely visible in original EPLL result, while in Fig. 3(d) the edge is smoother and more real. This probably due to the fact that the new adaptive parameter can take a small number at edge areas so as to preserve edges better. Tabs. 2 and 3 show numerical results between two methods on different images at different noise deviation. At both low and high noise deviation, various images can obtain higher PSNR by proposed method. We can see that the proposed method can achieve very competitive denoising performance. It is obvious that our method is superior to the original EPLL method.

Conclusions
Image is a carrier of abundant information, so directly learning image priors is a big challenge for us. GMM has proven its power in various areas due to its robustness, so image denoising based on GMM has become more and more active in the past decades. The EPLL algorithm is such a method that builds a model from the extracted patches to the whole image based on GMM. Classical P-M model has achieved satisfying results, but there is still weakness that it focuses only on the gradient. We should realize that details in image may involve variation information, therefore we should make full use of variation along with image gradient. In this paper, we construct an adaptive regularization parameter function based on the image gradient and variance. In addition, we add a gradient fidelity term to maintain the similar structure between the degraded image and the restored one. Our method performs well in image denoising, especially in edge and detail preserving, and shows an obvious advance compared with the original EPLL algorithm. There is still much room for improvement. In the process of actual application, data tends to be symmetrical distribution from a statistical standpoint. GMM can be treated as deviation special distribution of the gaussian mixture distribution. Using a skew gaussian distribution as prior obviously will have stronger generalization ability. Moreover, we can consider the geometric characteristics of the image, so that the prior information dictionary more perfect and more accurate.