Enhancement of damaged-image prediction through Cahn–Hilliard image inpainting

We assess the benefit of including an image inpainting filter before passing damaged images into a classification neural network. We employ an appropriately modified Cahn–Hilliard equation as an image inpainting filter which is solved numerically with a finite-volume scheme exhibiting reduced computational cost and the properties of energy stability and boundedness. The benchmark dataset employed is Modified National Institute of Standards and Technology (MNIST) dataset, which consists of binary images of handwritten digits and is a standard dataset to validate image-processing methodologies. We train a neural network based on dense layers with MNIST, and subsequently we contaminate the test set with damages of different types and intensities. We then compare the prediction accuracy of the neural network with and without applying the Cahn–Hilliard filter to the damaged images test. Our results quantify the significant improvement of damaged-image prediction by applying the Cahn–Hilliard filter, which for specific damages can increase up to 50% and is advantageous for low to moderate damage.


Introduction
Image inpainting consists in filling damaged or missing areas of an image, with the ultimate objective of restoring it and making it appear as the true and original image. There are multiple applications of image inpainting, ranging from restoration of the missing areas of oil paintings and removal scratches in photographs to noisy MRI scans and blurred satellite images of the earth. Manual image inpainting techniques have been employed for many centuries by art conservators and professional restorers, but it was not until the turn of the twenty-first century that digital image inpainting models based on partial-differential equations (PDEs) and variational methods were introduced [1][2][3]. These methods are usually referred to as non-texture, geometrical or structural inpainting since they focus on restoring the structural information in the inpainted domain such as edges, corners or curvatures. This is done by performing an image interpolation of the damaged areas based on the information collected from the surrounding environment only, leading to appealing images for the human vision system. On the contrary, texture inpainting is based on recovering global patterns of the image for the inpainted region [4], and a popular tool in this category is the exemplar-based inpainting methods [5,6]. Associated with these developments, a field that has gained a lot of traction in recent years is the so-called generative image inpainting, where deep-learning-based approaches have proven to be successful even for blind inpainting in which the inpainted region is not provided a priori [7][8][9]. In this work, we focus on non-texture image inpainting methods based on partial differential equations (PDEs), and we refer the reader to [10] for a general review of the topic.
There have been multiple PDE models for image inpainting proposed since the initial work of Bertalmio et al. [1] nearly 20 years ago. Their trail-blazing model is able to propagate isotopes, i.e. contours of uniform greyscale image intensity, through the inpainted region, a common technique employed by museum artists in restoration. It also turns out that the original model bears close connection to fluid dynamics through the Navier-Stokes equation with the image intensity function acting as the stream function [11]. Another fluid dynamic model that has played a pivotal role in image inpainting is the Cahn-Hilliard (CH) equation, initially proposed in [12] for phase separation in binary alloys. It has been employed in a wide spectrum of applications from wetting phenomena [13,14] and polymer science [15] to tumour growth [16]. This equation satisfies a gradient-flow structure employing a H −1 norm, where ϕ is an order parameter widely referred to as the phase field which for a binary system takes on the value ϕ = 1 in one of the phases and ϕ = −1 in the other, while varying smoothly in the interface region with a width of O(e ), with e being a positive parameter related to the interface thickness coming from the derivation of diffuse-interface models [17]. M(ϕ) is the mobility obtained here from the one-sided model M(ϕ) = 1 and F [f] is the free energy satisfying with the variation of the free energy denoted as ξ and given by and H(ϕ) taken here as the Ginzburg-Landau double-well potential with the two wells corresponding to the two phases, (1:4) The boundary conditions imposed for the CH equation in (1.1) are no-flux for both the phase field and for the variation of the free energy, e 2 rf Á n ¼ 0, M(f)rj Á n ¼ 0, (1:5) where n is defined in the normal direction to and into the wall.
There is a clear physical interpretation of the terms forming the free energy in (1.2): on the one hand, the potential H(ϕ) contains the equilibrium information of the system, and each of the minima of the choice (1.4) correspond to an equilibrium phase; on the other hand, the gradient term in (1.2) accounts for the cost of spatial inhomogeneities, resulting in a surface tension between different phases. As a result, the equilibrium state balances the tendency of separation from the hydrophobic potential H(ϕ) and the tendency of mixing from the hydrophilic gradient term. The mobility term impacts the rate of phase separation and the coarsening process, and some works employ non-constant degenerate mobilities that cancel when the phase field corresponds to one of the two wells in (1.4) (see e.g. [18][19][20] for further discussions of the physical significance of the various terms of the CH equation).
royalsocietypublishing.org/journal/rsos R. Soc. Open Sci. 8: 201294 The CH equation was firstly proposed in the context of image inpainting by Bertozzi et al. [21]. Specifically, the authors adopted a modified CH equation for binary images with inpainting quality as accurate as the state-of-the-art inpainting models but with a much faster computational speed taking advantage of the efficient computational techniques already available for the CH equation (DJ Eyre 1998, unpublished article) [22]. Since then several authors have extended the applicability of the CH equation in the field of image inpainting, for instance by taking into account grey-value images [23,24], non-smooth potentials instead of the double-well potential (1.4) [25], and considering colour image inpainting [26]. The modified CH equation in [21] introduces a fidelity term λ(x) to avoid modifying the original image outside of the inpainted region D, and the CH equation in (1.1) becomes and ϕ(x, t = 0) refers to the original damaged image. The parameter e plays a similar role as in the original CH equation, and here it is related to the interface between the two phases or colours presented in the image. The two parameters e and λ 0 are essential to achieve an adequate image inpainting outcome, and it is usually necessary to iterate until finding appropriate tunings for their values, which typically depend on the image specifications. Furthermore, the new modified CH equation (1.6) is not strictly a gradient flow: although the original CH equation satisfies a gradient-flow structure under an H −1 norm and the fidelity term in (1.6) can be derived from a gradient flow under an L 2 norm, the combined modified CH equation is neither a gradient flow in H −1 nor L 2 .
One of the main advantages of employing the CH equation for image inpainting is the myriad of fast and reliable numerical methods available for its solution. A pivotal contribution was the convex-splitting scheme developed by DJ Eyre (1998, unpublished article), which is unconditionally energy-stable, by treating as implicit the convex terms of the free energy in (1.2), while keeping the concave terms explicit. The design of energy-stable and maximum-principle satisfying schemes for the CH equation has been an active area of research [27], and several authors have proposed schemes based on finite differences [28][29][30], finite elements [31,32], finite volumes [33] or discontinuous Galerkin [34][35][36][37]. These schemes have also proven effective for degenerate mobilities or logarithmic potentials. Schemes satisfying the maximum principle condition for specific choices of free energy have also been constructed [38]. We refer the reader to [39] for a recent work discussing the state-of-the-art numerical techniques for nonlinear gradient flows.
In a recent effort [19], we constructed a robust semi-implicit finite-volume scheme for the CH equation that offers crucial advantages when applied to the field of image inpainting: -Firstly, finite volumes are a straightforward discretization when dealing with images, which often consist of rectangular-shaped pixel cells with an average colour intensity. This is exactly the starting point of finite-volume schemes, and as a result it is conceptually simpler to apply finite volumes in comparison with finite elements, finite differences or discontinuous Galerkin (which would be more suitable for other more complex and rare pixel shapes such as triangular ones). -Secondly, our scheme is based on a dimensional-splitting approach: instead of solving the full twodimensional image altogether, dimensional splitting initially solves row by row and then column by column. This has a massive benefit in computational cost, which is reduced from O(N dg ) for an image with N cells in d dimensions to O(dN dþgÀ1 ), with 2 < γ < 3. The reason for this is that the cost of inverting an N × N matrix is O(N dg ), with 2 < γ < 3 that slightly varies depending on the inversion algorithm and matrix structure (see [40] for details). For images with N cells per dimension, the solution of the full two-dimensional scheme involves inverting a N d × N d Jacobian matrix, with a subsequent cost of O(N dg ). By contrast, the dimensional-splitting technique requires inverting dN d−1 Jacobians of size N × N, amounting for a total computational cost of O(dN dþgÀ1 ). This is already advantageous for a two-dimensional image, and the computational cost is further reduced for highdimensional images, such as the ones for [41] or X-ray computed tomography [42] in medical image analysis. To add more, the dimensional-splitting technique allows for parallelization, and it is possible to halve the total computational cost by solving non-adjacent rows and columns in parallel. -Thirdly, our scheme has been extensively tested in [19] for challenging configurations of the original CH equation (1.1). In addition, in [19] we prove that the scheme unconditionally satisfies the discrete royalsocietypublishing.org/journal/rsos R. Soc. Open Sci. 8: 201294 decay of the free energy for different choices of potentials (1.4) [25], while at the same time we prove the phase-field boundedness for mobilities of the form M(ϕ) = 1 − ϕ 2 . Even though the modified CH equation (1.6) is not strictly a gradient flow due to the inclusion of the fidelity term, our scheme preserves its robustness for all the image-inpainting test cases presented in this work.
The combination of these properties and reduced computational cost, together with the versatility of finite volumes, make our scheme efficient and robust for the solution of the modified CH equation in (1.6) for a variety of applications in image inpainting. The objective of this work is to show precisely the applicability of our numerical framework in [19] for a benchmark dataset of images in need of restoration through image inpainting. For this task, we purposely add different types and intensities of damage to the popular Modified National Institute of Standards and Technology (MNIST) dataset [43], and then apply image inpainting by solving the modified CH equation (1.6) with our finite-volume scheme in [19]. The MNIST database is a standard dataset to validate image-processing methodologies, and it contains binary handwritten images of numbers from zero to nine. We choose this dataset since the modified CH equation (1.6) is applicable to binary images such as the ones in the MNIST dataset. The extension of this work to non-binary images is also relevant and will be explored elsewhere. We also assess the improvement in pattern recognition accuracy of the restored MNIST images, and for this we construct a neural network for the task of classification. A key objective of our study is to quantify the benefits of including a CH filter before introducing a damaged image into a neural network. Our results demonstrate that accuracies in classification can increase up to 50% for particular damages in the images, and, in general, applying the CH filter improves the accuracy prediction for a wide range of low to moderate image damage. This increase in accuracy is obtained by applying our image-inpainting methodology to the MNIST dataset exclusively, and as a disclaimer we remark that other datasets or types of damage may result in different increases of accuracy. The application of other types of filters, such as texture inpainting or generative inpainting, may also result in different increases of accuracy, but overall the accuracy should be higher compared to the case where no filter is applied to the damaged image. The code to reproduce the results of this work is available at [44].
In §2, we outline the methodology: in §2.1 we adapt our finite-volume scheme in [19] for the modified CH equation in (1.6); in §2.2, we recall the two-step method for image inpainting in [21]; in §2.3, we detail the neural network architecture for the classification task; and lastly in §2.4, we explain the structure of the integrated algorithm which takes a damaged image, applies a CH filter to it, and then classifies the image through a neural network. Subsequently in §3, we present the results of the integrated algorithm applied to the MNIST dataset: in §3.1, we begin by identifying appropriate tunings for the values of e and λ 0 ; in §3.2, we present the different types of damage introduced into the MNIST testset of images; and finally in §3.3, we quantify the improvement in accuracy of applying the CH filter to the damaged MNIST images before introducing them into the neural network. A discussion and final remarks are offered in §4.

Integrated algorithm with image inpainting and pattern recognition
We detail the construction of an integrated algorithm that firstly applies image inpainting and subsequently conducts pattern recognition for the restored image. In §2.1, we begin by presenting the finite-volume scheme employed to solve the modified CH equation, based on the work in [19]. Then in §2.2 we illustrate the two-step method for image inpainting, based on tuning the parameters e and λ 0 of the modified CH equation. In §2.3, we present the neural network employed for pattern recognition, detailing its architecture and training parameters. Finally, in §2.4, we gather all previous elements to formulate an integrated algorithm for prediction with an image-inpainting filter.

Two-dimensional finite-volume scheme for the modified CH equation
We summarize the two-dimensional finite-volume scheme constructed for the original CH equation in our previous work [19]. This scheme satisfies an unconditional decay of the discrete free energy of the original CH equation in (1.2), independently of the time step, and for specific choice of mobility M(ϕ) = 1 − ϕ 2 it ensures the unconditional boundedness of the phase field. The scheme can be straightforwardly extended to the modified CH equation in (1.6) proposed in [21] as we show here. Even though the modified CH equation does not possess some of the properties of the original CH royalsocietypublishing.org/journal/rsos R. Soc. Open Sci. 8: 201294 equation, such as the gradient-flow structure, our finite-volume scheme preserves its robustness for all the image-inpainting test cases presented in this work. In remark 2.1, we explain how to choose the time step and the mesh size, and in remark 2.2, we detail how to turn the scheme into a dimensionalsplitting one, with promising applicability in high-dimensional images such as medical ones. We refer the reader to [19,45] for further details about dimensional-splitting schemes.
For simplicity, let us rewrite (1.6) in two dimensions by introducing u = (v, w) as the physical velocity term and ξ as the variation of the free energy with respect to the density, For the finite-volume formulation, we begin by dividing the computational The time step is denoted as Δt. In each of the cells we define the cell average f n i,j at time t = nΔt as where f 0 i,j is the phase field at t = 0, which corresponds to the normalized pixel intensities of the initial damaged image to be inpainted.
The finite-volume scheme is then derived by integrating the modified CH equation (2.1) over each of the cells C i,j of the domain, leading to an implicit formulation satisfying with F nþ1 iþ1=2,j and G nþ1 i,jþ1=2 being flux approximations at the boundaries, and λ i,j the discrete version of l(x) in (1.7) satisfying with D being the inpainted domain where the image damage is located. The impainted domain has to be determined beforehand for each image, and is formed by those finite-volume cells to be repaired during the image inpainting. The flux terms F nþ1 iþ1=2,j and G nþ1 i,jþ1=2 are the discrete approximations of the velocity components v and w at the cell interfaces (x i+1/2 , y j ) and (x i , y j+1/2 ), respectively. Such approximations follow an upwind and implicit approach inspired by the works in [19,[45][46][47], satisfying where the velocities v nþ1 iþ(1=2),j and w nþ1 (2:8) The discretized variation of the free energy j nþ1 is computed with a semi-implicit scheme inspired by the ideas of DJ Eyre (1998, unpublished article) [22], where the so-called convexity splitting scheme is proposed to construct unconditional gradient-stable schemes (i.e. schemes that royalsocietypublishing.org/journal/rsos R. Soc. Open Sci. 8: 201294 ensure the decay of the discrete version of the free energy in (1.2)). In our recent effort [19], we show that our finite-volume scheme unconditionally decreases the discrete free energy of the CH equation if the contractive part of the potential, H c (ϕ), is taken as implicit; the expansive part of the potential, H e (ϕ), is taken as explicit; and the Laplacian is taken as an average between the explicit and the implicit second-order discretizations, so that (2:9) where the potential H(ϕ) in (1.4) is decomposed into two convex functions, H c (ϕ) and H e (ϕ), The discrete two-dimensional approximation of the Laplacian (Df) n i,j is chosen to satisfy the second-order form Dy 2 : (2:10) The modified CH equation in (1.6) employs the no-flux boundary conditions defined in (1.5). The numerical implementation of the boundary conditions follows from Remark 2.1 (Choice of time step Δt and mesh size Δx = Δy). Explicit finite-volume schemes are often stable under a Courant-Friedrichs-Lewy (CFL) condition imposed over the time step and mesh size. The implicit scheme presented in §2.1 does not need a CFL condition to decay the discrete free energy of the original Cahn-Hilliard equation in (1.1), as proved in [19]. However, this does not mean that any Δt works in practice: in each time step, the nonlinear system (2.4) has to be solved by iteration, and with a large Δt the convergence is likely to fail. In this work we have tested that with the choice of Δt = 0.1 and Δx = Δy = 1 our simulations converge to the right solution for the simulations presented in §3.
Remark 2.2 (Dimensional-splitting scheme and parallelization). The full two-dimensional finitevolume scheme presented in §2.1 can be reformulated by employing a dimensional-splitting methodology: instead of solving the full two-dimensional image altogether, this technique initially solves row by row and then column by column. The detailed construction of such a scheme is presented in [19] and is based on [45]; and here we briefly illustrate how it works.
For the dimensional-splitting approach, we firstly update the solution along the x-axis, for each index j corresponding to a fixed value of y j where j ∈ [1, N]. Subsequently, we proceed in the same way along the y-axis, for each index i corresponding to a fixed value of x i , where i ∈ [1, N]. The index r, where r ∈ [1, N], denotes the index j of the fixed y j value in every x-axis of the first loop, and the updated average density for each x-axis with j = r is f n,r i,j . Similarly, the index c ∈ [1, N] denotes the index i for every fixed value of x j in each y-axis of the second loop, and the updated density for each y-axis with i = c is f n,c i,j . In the first place, we go through each of the x-axes of the domain at a fixed y j with j = r. The initial conditions for the scheme are f n,0 i,j :¼ f n i,j . The scheme for each x along the loop r = 1, …, N satisfies i,j otherwise: :¼ f n,N i,j . This dimensional-splitting scheme can be fully parallelized in order to save computational time. This is because: (i) the scheme does not take notice of the order of updating the rows/columns, as long as all of them are updated; (ii) one row or column only depends on the values of the directly adjacent rows or columns, respectively. As a result, a strategy to parallelize the dimensional-splitting scheme consists of updating at the same time all the odd rows/columns, since they do not depend on one another. One can then proceed with all the even rows/columns at the same time.

Two-step method for the modified CH equation
The two-step method followed here was firstly proposed in [21]. It basically consists of applying the finite-volume scheme in §2.1 twice with different values of the parameter e. The first stage consists of taking a large e to execute a large-scale topological reconnection of shapes, leading to images with diffused edges. Subsequently, and in order to sharpen the edges after the first stage, e is substantially reduced, and the final outcome becomes less blurry and diffused. We denote the corresponding values of e as e 1 and e 2 .
Adequately tuning the two values of e in each stage, as well as λ, is vital to complete a successful image inpainting. Those values have to be chosen empirically and depend on the dataset and type of damage, and in §3.1 we conduct a study to select them (with a sensitivity analysis provided in appendix A). As explained there, the appropriate values for e are between 0.5 and 1.5 for MNIST-like images, while λ ∈ [1, 1000]. The cell sizes are Δx = Δy = 1. The reader can find the exact values employed after the analysis of §3.1 in table 2.

Neural network architecture for classification
The prediction of the label in the restored images is performed via a neural network constructed in TensorFlow [48]. Its architecture is defined taking into account that in this work we employ the MNIST dataset [43], which contains binary images of digits from 0 to 9 and has a resolution of 28 × 28 pixels. This is a benchmark dataset in the community and is the de facto 'hello world' dataset of computer vision. There are consequently plenty of neural network architectures attaining extremely high accuracies for the MNIST dataset, and we refer the reader to the Kaggle competition of Digit Recognizer in [49] for examples of such architectures.
Here, however, our overarching objective is to quantify how the prediction of damaged images is enhanced once the CH filter is applied to the images beforehand. Hence, we do not require a highly sophisticated neural network and a cutting-edge architecture as in computer vision: our images are not going to be exactly the same as in the training set due to the damage and the subsequent restoration. We then select a standard architecture for classification based on sequential dense layers. Such architecture is formed by: (1) A flatten layer that takes the 28 × 28 image input and turns it into an array with 784 elements. There are no weights to optimize in this layer.  For the training of this network, we initially divide the original MNIST dataset in 60 000 training images and 10 000 testing images. We then train the neural network for 10 epochs with the Adam optimizer, choosing as loss function the categorical cross-entropy defined as with w being the weights to optimize, y i each of the N true labels of the training dataset, andŷ i each of the N predicted labels. After 10 epochs, we get an accuracy for the training dataset of 99.02%, while for the test set the accuracy is 97.47%. Once the neural network is trained, we keep the weights fixed for the comparison of damaged and restored images in §3. A display of the neural network is depicted in figure 1.

Integrated algorithm
The integrated algorithm proposed and tested in this work takes as input a damaged image, applies the CH filter based on §2.1 and §2.2 to restore it, and finally applies the already-trained neural network in §2.3 to predict its label.
To show the applicability of this integrated algorithm, we initially create damage in the images of the test set in the MNIST dataset [43]. After we apply the image inpainting to the damaged test images, we introduce the restored images in the neural network. At that point, and since we have the true labels of the test set, we can assess the attained accuracy in comparison to directly introducing the damaged images or the original images into the neural network. This procedure is conducted for multiple types of damage in §3, and a schematic of all steps is depicted in figure 2.

Application of the integrated algorithm to the MNIST dataset
Our focus here is testing the applicability of the integrated algorithm in §2.4 to increase predictability in damaged images. In §3.1, we start by analysing the impact of the parameters λ 0 and e on the inpainting process, with the objective of calibrating them before employing the MNIST dataset. In §3.2, we detail the types of damage that we insert into the MNIST dataset, and we also show the restored outcomes of applying the CH equation as an image inpainting filter. Finally, in §3.3, we evaluate how the accuracy of the damaged images increases after applying the CH filter to them, for various types and degrees of damage.
The code to reproduce the results of this work is available at [44].

Inpainting of a cross-line
We employ the cross-line example in [21] to analyse the role of the parameters e and λ 0 in the finitevolume scheme of §2.1. These two parameters crucially determine the success of the image inpainting We first aim to determine λ 0 in (2.5) and we set e = 1 as an initial guess so that e = Δx = Δy. From (2.5), λ i,j is only non-zero for the predefined area of undamaged pixels. Indeed the term with λ i,j in the finitevolume scheme in (2.4) ensures that the undamaged pixels are not modified during the image inpainting, but for this λ 0 has to be sufficiently large to counterbalance the fluxes of the scheme. Bearing this in mind, we run the numerical scheme with a Δt = 0.1 and until the L 1 norm between successive states is lower than a certain tolerance fixed to be 10 −4 . Our simulation produces satisfactory results and does not break down for a range of λ ∈ [1, 1000]. Table 1 shows that the computational time to reach the required tolerance decreases when increasing the value of λ. In addition, the L 1 norm between the final and initial state is also lower for greater λ. It is worth mentioning that different choices for Δt yield different ranges of valid λ, given that ( figure 3a. The reason for that is explained in [50], where multiple steady-state solutions of the modified CH equation were shown to exist. As the information under the inpainting region has been destroyed, there is no way of knowing that the steady state we obtain is less accurate than other viable solutions, in comparison to figure 3a. For further details, we refer the reader to [50], where a bifurcation analysis is carried out to show that the steady state may vary depending on the choices for e and Δx, Δy.

Damage introduced in the MNIST dataset
Here we discuss the types of damage inserted into the MNIST test set, with the objective of subsequently applying the CH filter developed in §2.1 for image inpainting. The varied damage employed aims to represent a mock case of damage that may be encountered in an image in need for restoration. As a result, we decide to employ two kinds of damage with different intensities: customized damage affecting particular regions of the image, and random damage selecting arbitrary pixels or horizontal lines in the image. Specifically: (a) Customized damage: this type of damage is applied in four different fashions, as shown in figure 4. The basic idea is to turn vertical or horizontal lines of pixels into a uniform grey intensity between black and white colour. In figure 4, we show the outcome of applying the CH filter to the damaged images. It can be seen that our model is able to recover the images from the different types of damage, albeit with varying degrees of success. For instance, the damage introduced in figure 4c is a thick horizontal line implying a considerable loss of information from the original image, compared to the other types of damage. As a result, the inpainted image filter for this type of damage is not as effective as the other ones, as can be seen from the inpaintings in figure 4. (b) Random damage: this second type of damage is inserted in a random fashion and with different levels of intensity. Two ways of randomly creating damage are considered: one makes use of randomly selecting whole horizontal rows of pixels, while the other is obtained by randomly selecting individual pixels. In addition, for both types of random damage we employ different  figure 5. Similarly to the case of customized damage, the higher the intensity of damage the more information is lost in the inpainting, as we can see for example in the case of 80% pixel damage in figure 5. But despite this, our image inpainting algorithm renders recognizable images even with relative high amounts of damage.
The parameter values for λ, e 1 and e 2 are gathered in table 2, and follow the reasoning discussed in §3.2. This choice of parameters in our finite-volume scheme in §2.1 leads to an effective image-inpainting algorithm capable of restoring images with damage of varied nature as shown in figures 4 and 5.
Further details about the selection of λ, e 1 and e 2 are provided in appendix A, where a sensitivity analysis is provided in order to justify the parameter choices in table 2. The next step is to integrate this image-inpainting algorithm within a pattern recognition framework for the MNIST dataset.

Pattern recognition for inpainted images
We now apply the neural network described in §2.3 to predict labels of damaged images with and without image inpainting, with the aim of quantifying the improvement of accuracy following the application of the CH filter. This study is completed for the different types and intensities of the noise depicted in §3.2. We begin by adding the types of damage in §3.2 to the 10 000 samples of the MNIST test dataset. The next step is to apply the CH filter and the two-step method to each one of them, while also saving copies of the test images with the damage. Eventually, for each type of damage we get two batches of 10 000 images: one still with the damage, and another one with image inpainting applied. Given that the neural network of §2.3 is already trained with the 60 000 samples of the MNIST training dataset, we can directly compute the accuracy of each of the two batches. This way we are able to assess the improvement in accuracy thanks to applying image inpainting to restore the damage.
For the validation we just employ the accuracy metric, which is defined as follows: accuracy ; number of correct predictions number of total predictions : There are, however, many other metrics apart from the accuracy one that play a vital role in other classification problems: recall, precision, F1 score, true positive rate and so on. Here we believe that the accuracy metric is enough to draw conclusions about how the image inpainting is improving the royalsocietypublishing.org/journal/rsos R. Soc. Open Sci. 8: 201294 predictions with respect to the damage images. This is due to the fact that the MNIST dataset is a balanced dataset, where there is generally no preference between false positives and false negatives. The measure of improvement between the batch of samples with image inpainting and the damaged ones without it is computed as improvement ; accuracy with CH filter À accuracy without CH filter accuracy without CH filter , ( 3 :1) and it basically represents the percentage of improvement that results from adding the CH filter to the prediction process.
The results for all types of damage under consideration are displayed in tables 3-5. In table 3, we gather the prediction accuracies for the four customized damages displayed in figure 4, as well as the prediction accuracy for the unmodified MNIST test set, which for our neural network architecture is 0.97. We observe that for the types of more intense customized damage B and C the accuracy prediction for the damaged images without CH filter drops to 0.71 and 0.64, respectively. By applying   Figure 5. Examples of image inpainting for random damage in whole horizontal rows and in individual pixels. The column entitled 'damaged rows' marks the number of rows randomly selected for damage in the 28 × 28 images. The column entitled 'damaged pixels' marks the percentage of randomly damaged pixels over the whole image. For higher levels of damage intensity, the inpaintings lose more information.
royalsocietypublishing.org/journal/rsos R. Soc. Open Sci. 8: 201294 the filter we find that the accuracy predictions can significantly escalate to 0.93 and 0.82, leading to improvements of 31% and 28%, respectively. The other two types of customized damage, A and D, are not as pervasive as B and C, and as a result the accuracy predictions are high even without applying the CH filter. In tables 4 and 5, we test the accuracies for random damage with various levels of intensities. The objective here is to analyse how the CH filter responds when the damage occupies more and more Table 3. Accuracy for the test dataset of MNIST without and with the CH filter, for the customized damage in figure 4. The improvement is computed following [21]. space in the images, both for the case of rows or pixels, as displayed in figure 5. In table 4, we show the accuracies for a range of damaged rows between 6 and 26, bearing in mind that the dimensions of the MNIST images are 28 × 28. We observe that for low numbers of damaged rows, the accuracy prediction even without the CH filter is high and it does not improve significantly by adding the filter. But then the improvement surges until reaching a maximum value of 47% for 16 random damaged rows, where the prediction without CH filter is 0.55 and with CH filter 0.81. For larger numbers of damaged rows the accuracies drastically drop due to the large amount of information lost, and not even the image inpainting process is able to achieve decent accuracies. In the limit of the damaged number of rows tending to 28, we observe that the accuracies are close to the ones of a dummy classifier with one out of ten chances of rightly guessing the label. In this limit, there is no difference between adding the CH filter or not, and it turns out that the improvements are even negative. We observe a similar pattern for the case of random damaged pixels in table 5. For low percentages of damaged pixels, the improvement of adding the CH filter is negligible and the accuracies with and without the filter are quite high. As we increase the percentage of damaged pixels the improvement escalates until it reaches 45% for a scenario with 80% of the pixels randomly damaged. For this case, the accuracy prediction without the filter is just 0.55, but thanks to the filter it significantly increases to a decent value of 0.8. For larger percentage of pixels, the improvement and accuracies drop, and in the limit towards 100% of damaged pixels we get close to the accuracy of a dummy classifier. This is due to the large loss of information that the original images have suffered.
In figure 6, we depict some specific examples for which the label is only predicted correctly after applying the CH filter to the damaged image. These are just some particular samples out of the 10 000 images contained in the test dataset of MNIST, and for some of them the opposite effect can occur: that the label is correctly predicted for the damaged image but following the inpainting process it is predicted incorrectly. This only occurs for really severe damages, where there is no difference between adding the CH filter or not and the accuracies are close to the ones of a dummy classifier guessing randomly. However, we have shown in tables 4 and 5 that, overall, the CH filter increases the global accuracy for images with low to moderate damage, and consequently we expect that scenarios such as the ones displayed in figure 6 are much more common than the opposite ones.
In figure 6, we also find that some greyscale pixels remain in the inpainted image after applying the modified CH filter. This is due to the fact that the finite-volume scheme has not run for long enough to  Figure 6. Particular examples of label predictions for MNIST samples with various types of damage: customized damage C, random damage in 16 rows and random damage in 70% of pixels. The label of the damaged image is wrongly predicted, while the label for the inpainted image is correctly predicted. P D and P I represent the label predictions of the damaged and inpainted images, respectively.
royalsocietypublishing.org/journal/rsos R. Soc. Open Sci. 8: 201294 turn those intensities into the stable white (ϕ = −1) or black (ϕ = 1) phase field. This is prone to happen for images with high levels of damage, and one should try to let the simulation run for as long as possible to avoid possible misclassification issues due to those greyscale regions.

Discussion and conclusion
We have quantified the prediction improvement of employing a CH image inpainting filter to restore damaged images which are then passed into a neural network. We combined a finite-volume scheme with a neural network for pattern recognition to develop an integrated algorithm summing up the process of adding damage to the images and then predicting their label. Our results for the MNIST dataset suggest that, in general, the accuracy is improved for a wide range of low to moderate damages, while for some particular cases we reach improvements of up to 50%. We also provide the image inpainting outcome of multiple damage scenarios and the benefits of adding the CH filter to predict the label of the image are easily visible. We believe that our results employing the MNIST dataset lay the foundations towards the application of image inpainting in more complex datasets. We have demonstrated the benefit of combining the fields of image inpainting with machine learning, and we believe that many applications can take advantage of it. For instance, there are applications such as medical images from MRI or satellite observations where typically there is some inherent noise or damage involved and where there may be potential to employ tools from machine learning as was done here.
MNIST is one of the most well-known and convenient benchmark datasets. It is possible that applying our methodology to increasingly complex datasets might bring about new challenges. At the same time, we have relied on two main assumptions about the damage and the images: the first one is that the images are binary, leading to the standard CH potential in (1.4) that has only two wells (i.e. one for each of the two colours); the second one is that the damage is not blind, meaning that the location of the damage is known. Performing image inpainting without these two assumptions becomes substantially more involved as already pointed out in [7,23,26]. We will be exploring these and related questions in future works. On the other hand, the parameter λ in the finite-volume scheme in (2.4) ensures that the undamaged pixels are not modified during the image inpainting. The choice of λ has to be sufficiently large to counterbalance the fluxes of the scheme and act as a penalty term that keeps the undamaged pixels invariant. However, λ cannot be too large since otherwise the finite-volume scheme becomes a singularly perturbed problem, and the convergence of the implicit scheme is deteriorated. The value of λ is related to the time step Δt, and greater values of λ are possible if Δt is refined. In our case, with the choice of Δt = 0.1, our finite-volume scheme does not yield any result and breaks down during the simulation for values of l Ó [1, 10 000]. To evaluate the impact of λ we run three experiments: in the first one we choose the really small value λ = 0.01, in the second the small λ = 0.1 and in the third the large value λ = 10 000. The outcomes with the two small values are depicted in figure 7e,f, and we observe that the undamaged pixels have been altered and the overall appearance is not clean. The outcome with the large value is depicted in figure 7g, and the image is comparable to the optimal one in figure 7b. There is a threshold-λ value above which there is no convergence of the implicit finite-volume scheme. For the particular image, we found this threshold can be approximated by λ = 10 000. royalsocietypublishing.org/journal/rsos R. Soc. Open Sci. 8: 201294