Partial Differential Equation Inpainting Method Based on Image Characteristics

Abstract

Inpainting is an image processing method to automatically restore the lost information according to the existing image information. Inpainting has great application on restoration of the lost information for photographs, text removal of image, and recovery for the loss coding of image, etc. Image restoration based on partial differential equation (PDE) is an important repair technology. To overcome the shortcomings of the existing PDEs in repair process, such as false edge, incomplete interpolation information, a new PDE for image restoration based on image characteristics is proposed. The new PDE applies different diffusion mode for image pixels with the different characteristics, which can effectively protect the edges, angular points, and other important characteristics of the image during the repair process. The experimental results in both gray images and color images show that our method can obviously improve the image visual effect after inpainting compared with different traditional diffusion models.

1 Introduction

Digital image inpainting is one of the fundamental problems in the field of digital image processing, which can be used for repairing the loss or damage part of an image. The inpainting process is filling-in the missing part in accordance with the information. The key of inpainting is to make the repaired image close to the original image in terms of visual effects and ensure the rationality of the repaired image. Image inpainting was required from the restoration for some medieval artworks since Renaissance Period, and it is a valuable technique. With the rapid development of computer science and technology, majority of artworks such as paintings, sculptures and many cultural relics are scanned into the computer and become digital images. Therefore, the formerly manual repair is transformed to inpainting on computer via variety of models, algorithm and software.

PDE-based image inpainting has been paid much attention in recent years [1]. It looks the gray value of each pixel upon heat, and take the image inpainting processing as heat diffusion. Therefore the lost gray information of some pixels can be supplemented through gray diffusion, which is achieved by a PDE. PDE for digital inpainting was first introduced by Bertalmio etc [2], who introduced BSCB (Bertalmio-Sapiro-Caselles-Ballester) model based on a third-order PDE. However, the model’s speed is too slow, and it will blur the inpainted area, especially for the area with large loss information. TV (total variation) model [3] is better than BSCB in time, but its disadvantage is causing ladder-effect in the smooth regions. HCE (heat conduction equation) model was introduced by Adolf Fick, this model may obscure edge and lead poor visual effects after inpainting. PM (Perona-Malik) equation [4] is an improved model based on HCE, but it keeps traces of restoration due to excessive edge protection. To overcome the above shortcomings, a new PDE inpainting model is proposed based on image features in this paper, different interpolation strategies were adopted in the corresponding position according to the different characteristics of the image, which can protect the image features (such as edges and corners) better and have better visual effect.

2 Corner Detection

Corner is an important feature and is significant to various applications such as image matching, objective tracking, and image modeling. Corners must be protected in the process of inpainting in order to keep good visual effect and image quality.

The scatter matrix is used to detect corners, which is defined as:

$$ J_{\rho } = \left| {\begin{array}{*{20}c} {j_{{\text{11}}} } & {j_{{\text{12}}} } \\ {j_{{\text{21}}} } & {j_{{\text{22}}} } \\ \end{array} } \right| = \left| {\begin{array}{*{20}c} {\text{(}\frac{{\partial I_{\sigma } }}{\partial x}\text{)}^{\text{2}} \text{*}G_{\rho } } & {\text{(}\frac{{\partial I_{\sigma } }}{\partial x}\;\frac{{\partial I_{\sigma } }}{\partial y}\text{)*}G_{\rho } } \\ {\text{(}\frac{{\partial I_{\sigma } }}{\partial x}\;\frac{{\partial I_{\sigma } }}{\partial y}\text{)*}G_{\rho } } & {\text{(}\frac{{\partial I_{\sigma } }}{\partial y}\text{)}^{\text{2}} \text{*}G_{\rho } } \\ \end{array} } \right| $$

(1)

where $I_{\sigma }$ denotes the Gaussian smoothing image for the initial image I with the smoothing parameter $\sigma$, $G_{\rho }$ is a Gaussian kernel with the parameter $\rho$, * is the convolution operator.

$J_{\rho }$ has two eigenvectors $\omega^{ \bot }$ and $\omega$, which are calculated by:

$$ {\kern 1pt} \omega = \left( {\cos \theta ,\sin \theta } \right)^{\text{T}} \quad \omega^{ \bot } = \left( { - \sin \theta ,\cos \theta } \right)^{\text{T}} $$

(2)

where $\omega$ denotes the gradient direction of image characteristic (such as edge or one side of corner) which is perpendicular to the direction of image edge or corner side; $\omega^{ \bot }$ gives the coherence direction of the image edge or corner side, which is the tangent direction of the image edge or corner side; $\theta$ is the angle of $\omega$ with the horizontal axis, which can be calculated based on the scatter matrix:

$$ \theta \text{ = }\frac{\text{1}}{\text{2}}\text{arctan}\frac{{\text{2}j_{{\text{12}}} }}{{j_{{\text{11}}} \text{ - }j_{{\text{22}}} }} $$

(3)

Corner intensity $C_{I}$ of image I can be evaluated based on Eqs. (1), (2) and (3):

$$ C_{I} = \left| {\nabla (\omega^{ \bot } } \right.\omega^{ \bot T} )^{T} \cdot \nabla I = \left| {\nabla \left( {\begin{array}{*{20}c} {\sin^{2} \theta } & { - \sin \theta \cos \theta } \\ { - \sin \theta \cos \theta } & {\cos^{2} \theta } \\ \end{array} } \right) \cdot \nabla I} \right| $$

(4)

Figure 1 shows the corner detection results for the noiseless image, Gaussian fuzzy image and Gaussian noisy image, respectively. It can be seen that the corner detection based on the scattering matrix is robust for both Gaussian fuzzy and Gaussian noisy. This is because not only the gray value of current pixel is used, but also the gray information of the local neighborhood is applied when calculating Gaussian convolution of the initial image in Eq. (1) [6].

Fig. 1.

Corner detection results

3 PDE Image Inpainting Method Based on Image Characteristics

3.1 PDE Image Inpainting Model

PDE image processing method considers the gray distribution of image as heat distribution, and regards the process of image evolution as heat conduction. By establishing and solving the corresponding PDE model to accomplish heat diffusion, image gray will be redistributed. PDE image inpainting method interpolates the missing image gray values for the initial image through gray diffusion [5–7].

As indicated in Fig. 2, assuming Ω as the entire domain of the initial image I, D is the inpainting domain, D^C is the complementation of D, E is the neighborhood out of the inpainting domain, Γ is the boundary of D. Let I represent a gray-level image to be processed, $\left. I \right|_{{D^{C} }}$ denotes the known information, and u denote the repaired image [8].

Fig. 2.

Inpainting domain and its neighborhood

In order to protect the corner structure and not influence the visual effect of the repaired image, the following PDE is proposed for image inpainting.

$$ \left\{ \begin{aligned} \partial_{t} u = g\text{(}C_{u} \text{)} \cdot \text{(}g\text{(}|\nabla u|\text{)} \cdot D^{2} \text{(}\eta ,\eta \text{)} + \alpha \cdot D^{2} \text{(}\xi ,\xi \text{)),}\;u\left( {x,y,0} \right){ = }I\left( {x,y} \right)\text{, }\left( {x,y} \right) \in D\text{ } \hfill \\ \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \;\;\;\;u = I\quad \quad \quad \text{ }\;\;\text{ }\;\text{ }\left( {x,y} \right) \in D^{C} \hfill \\ \end{aligned} \right. $$

(5)

where $u\left( {x,y,t} \right)$ is the evolving image, whose initial values is $u\left( {x,y,0} \right){ = }I\left( {x,y} \right)$, $\nabla u$ is the gradient of u and $\text{|}\nabla u\text{|}$ denotes the module value of $\nabla u$, $C_{u}$ is the corner intensity of u; $g\text{(|}\nabla u\text{|) = 1/(1} + \text{|}\nabla u\text{|}^{\text{2}} \text{/}k^{\text{2}} \text{)}$ and $g(C_{u} )\text{ = 1/(1} + C_{u}^{\text{2}} \text{/}k^{\text{2}} \text{)}$ are edge stopping function and corner protection operator respectively; $D^{2} \text{(}\eta ,\eta \text{)}$ and $D^{2} \text{(}\xi ,\xi \text{)}$ are second-order directional derivatives in the gradient direction $\eta \text{ = }\nabla u\text{/|}\nabla u\text{|}$ and tangential direction $\xi \text{ = }\nabla^{ \bot } u\text{/|}\nabla u\text{|}$ respectively, $\alpha$ is a constant coefficient.

Equation (5) is an anisotropic diffusion equation, which has different diffusion degree along the image edge and through the image edge. By setting a big diffusion coefficient $\alpha$ for $D^{2} \text{(}\xi ,\xi \text{)}$, the diffusion is strong along the tangential direction of image edge. While the diffusion along the gradient direction $\eta$ is controlled by $g\text{(|}\nabla u\text{|)}$. The diffusion coefficient $g\text{(|}\nabla u\text{|)}$ is decided by the local image information, and is a reduction function of $\text{|}\nabla u\text{|}$. As $g\text{(|}\nabla u\text{|)}$ has small value on the edge of the image, diffusion is restricted along $\eta$ but not completely forbidden, and mainly spreads along $\xi$, which can effectively remove the zigzag boundaries and obtain the inpainted image with smooth edge. Similarly, the corner protection operator $g(C_{u} )$ is a decreasing function of corner intensity and its value is small in corner, but the diffusion is not completely forbidden also, which can effectively avoid sudden change of corner grayscale in the inpainted results. Therefore, the diffusion equation can protect corner and edge features when inpainting, so that the processing results seem more natural.

3.2 Discrete Difference Form of the PDE

In order to solve Eq. (5) numerically, the equation has to be discretized. The images are represented by M×N matrices of intensity values. So, for any function (i.e., image) u(x, y), we let u_i,j denote u(i, j) for 1<i<M, 1<j<N. The evolution equations give images at times $t_{n} = n\varDelta t$. We denote $u\left( {i,j,t_{n} } \right)$ by $u_{i,j}^{n}$. The time derivative at (i, j, t_n) is approximated by the forward difference:

$$ \partial_{t} u_{i,j}^{n} = \frac{{u_{i,j}^{n + 1} - u_{i,j}^{n} }}{\varDelta t} $$

(6)

where $\varDelta t$ is step-size in time.

The spatial derivatives are as follows:

$$ (u_{x} )^{n}_{{i},\,{j}} = u_{{i + 1},\,{j}}^{n} - u_{{i},\,{j}}^{n} $$

(7)

$$ (u_{y} )^{n}_{i,j} = u_{i,j + 1}^{n} - u_{i,j}^{n} $$

(8)

$$ (u_{xx} )^{n}_{ij} = u_{i + 1,j}^{n} + u_{i - 1,j}^{n}- 2u_{i,j}^{n} $$

(9)

$$ (u_{yy} )^{n}_{i,j} = u_{i,j + 1}^{n} + u_{i,j - 1}^{n} - 2u_{i,j}^{n} $$

(10)

$$ (u_{xy} )^{n}_{i,j} = \left( {u_{i + 1,j + 1}^{n} - u_{i,j + 1}^{n} } \right) - \left( {u_{i + 1,j}^{n} - u_{i,j}^{n} } \right) $$

(11)

$$ \text{(}u_{\eta \eta } \text{)}_{{i\text{,}j}}^{n} \text{ = }\frac{{\text{((}u_{x} \text{)}_{{i\text{,}j}}^{n} \text{)}^{\text{2}} \cdot \text{(}u_{xx} \text{)}_{{i\text{,}j}}^{n} \text{ + ((}u_{y} \text{)}_{{i\text{,}j}}^{n} \text{)}^{\text{2}} \cdot \text{(}u_{yy} \text{)}_{{i\text{,}j}}^{n} \text{ + 2(}u_{x} \text{)}_{{i\text{,}j}}^{n} \cdot \text{(}u_{y} \text{)}_{{i\text{,}j}}^{n} \cdot \text{(}u_{xy} \text{)}_{{i\text{,}j}}^{n} }}{{\text{((}u_{x} \text{)}_{{i\text{,}j}}^{n} \text{)}^{\text{2}} \text{ + ((}u_{y} \text{)}_{{i\text{,}j}}^{n} \text{)}^{\text{2}} }} $$

(12)

$$ \text{(}u_{\xi \xi } \text{)}_{{i\text{,}j}}^{n} \text{ = }\frac{{\text{((}u_{x} \text{)}_{{i\text{,}j}}^{n} \text{)}^{\text{2}} \cdot \text{(}u_{yy} \text{)}_{{i\text{,}j}}^{n} \text{ + ((}u_{y} \text{)}_{{i\text{,}j}}^{n} \text{)}^{\text{2}} \cdot \text{(}u_{xx} \text{)}_{{i\text{,}j}}^{n} - \text{2(}u_{x} \text{)}_{{i\text{,}j}}^{n} \cdot \text{(}u_{y} \text{)}_{{i\text{,}j}}^{n} \cdot \text{(}u_{xy} \text{)}_{{i\text{,}j}}^{n} }}{{\text{((}u_{x} \text{)}_{{i\text{,}j}}^{n} \text{)}^{\text{2}} \text{ + ((}u_{y} \text{)}_{{i\text{,}j}}^{n} \text{)}^{\text{2}} }} $$

(13)

where u_x, u_y, u_xx, u_yy, u_xy are the first and second order spatial derivatives in the x and y direction, respectively. $u_{\eta \eta }$ and $u_{\xi \xi }$ are second-order derivatives in the gradient direction $\eta$ and tangential direction $\xi$, respectively. Finally, we obtain PDE difference scheme based on the corner protect operator:

$$ u_{i,j}^{{n + \text{1}}} \text{ = }u_{i,j}^{n} \text{ + }\varDelta t \cdot \left( {g\left( {\left( {C_{u} } \right)_{i,j}^{n} } \right) \cdot \left( {g\left( {|\nabla u|_{i,j}^{n} } \right) \cdot \left( {u_{\eta \eta } } \right)_{i,j}^{n} \text{ + }\alpha \cdot \left( {u_{\xi \xi } } \right)_{i,j}^{n} } \right)} \right) $$

(14)

4 Image Inpainting Experiments and Discussion

In daily life, the inpainting images usually have no original undamaged image for comparison with the inpainted results. Inpainting effects evaluation cannot be determined with simple right or wrong, and should rely on visual perception, generally based on human visual habits and following four principles [5]:

• (1) The goal of inpainting is to ensure the integrity and continuity of the original image;

• (2) Important geometric features such as edge must be restored well, because human eyes have great interested in edge information;

• (3) Repaired image regions are harmonic with the known information in visualization performance, and there isn’t much unnatural things exist, including edge sharpness and zigzag effect, sharp degree of angle, and the ringing effect near the edge and the flat area;

• (4) The method should be robust.

We used the proposed PDE for image inpainting. In addition we compared some typical PDEs with our method to illustrate the advantages of the proposed PDE model based on the above four principles.

Experiment 1: Test image was inpainted by the PDE image restoration method according to formula (14) based on image characteristics. Three classic PDE models including the heat conduction equation (HCE), PM equation, and the mean curvature flow equation (MCF) [7] were used to compare. Figure 3(a) is the original image, which includes structure of edges and corners. Figure 3(b) is the contaminated image, namely the image to be repaired. Figure 3(c) is the artificial calibration region of the damaged area, and Fig. 3(d) is the mask image that extracted from Fig. 3(c). The parameters of the experiment are chosen as $k = 20$, $\varDelta t = 0.25$, $\alpha = 1$, n = 250. Figures 3(e) to (h) show the repaired results of the four methods respectively. HCE inpainted result given in Fig. 3(e) has obvious unrestored traces and blurred corners. Because PM equation and MCF equation focus on edge protection and ignore corner protection, when the difference of the gray values between the artificial calibration region and its surrounding pixel is great, these two PDEs will protect the artificial calibration region as image edge and lead to uncompleted repair (shown in Figs. 3(f) to (g)). Figure 3(h) shows inpainted result by the proposed method, which has the ability to protect the image characteristics (edges and corners), and repair the image more completely.

Fig. 3.

Comparison of different PDE inpainting methods for a test image

Experiment 2: A gray natural image was inpainted by the proposed PDE. The parameters in this experiment were used as: $k = 20$, $\varDelta t = 0.25$, $\alpha = 1.5$, n = 300. Figures 4(d) to (f) show the repaired results of HCE, PM and MCF respectively. It is apparent that the edge of the hair excessively diffused and hairline blurred, especially in the HCE and PM inpainted images. Figure 4(g) is the result carried out by our proposed method. One can see that edge stop function and corner protection operator can well protect the edge of the hairline’s border with background, and it avoids the loss of image detail.

Fig. 4.

Comparison of different PDE inpainting methods for a gray natural image (part)

Experiment 3: A color natural image was inpainted by the proposed PDE. Three channels of color image of R, G, B are repaired respectively first, and then color image is composited by the three repaired images of the three channels after painting. In this experiment, $k = 20$, $\varDelta t = 0.25$, $\alpha = 1.5$, $n = 300$. Figures 5(c) to (e) show the repaired results of HCE, PM, MCF models respectively. One can see that it exist different degrees of restoration traces in the three repaired results, especially the HCE and PM models, in which the marks of letters are very obvious. Figure 5(f) give the inpainted result used our method. There are no obvious traces of restoration in Fig. 5(f), and the method retains the integrity information of the original image.

Fig. 5.

Comparison of different PDE inpainting methods for a color natural image (part)

5 Conclusion

To solve the problems of the details missing or still have obvious unrestored traces in image inpainting, we present a new partial differential equation for image restoration based on image characteristics. The edge stopping function and corner protection operator can effectively control the velocity of diffusion for pixels with different characteristics, and effectively protect the important information such as image edges and corners. Experiment results show that our method can maintain the integrity of the image information and have great application on image inpainting.