Instant Edit Propagation on Images Based on Bilateral Grid

: The ability to quickly and intuitively edit digital content has become increasingly important in our everyday life. However, existing edit propagation methods for editing digital images are typically based on optimization with high computational cost for large inputs. Moreover, existing edit propagation methods are generally inefficient and highly time-consuming. Accordingly, to improve edit efficiency, this paper proposes a novel edit propagation method using a bilateral grid, which can achieve instant propagation of sparse image edits. Firstly, given an input image with user interactions, we resample each of its pixels into a regularly sampled bilateral grid, which facilitates efficient mapping from an image to the bilateral space. As a result, all pixels with the same feature information (color, coordinates) are clustered to the same grid, which can achieve the goal of reducing both the amount of image data processing and the cost of calculation. We then reformulate the propagation as a function of the interpolation problem in bilateral space, which is solved very efficiently using radial basis functions. Experimental results show that our method improves the efficiency of color editing, making it faster than existing edit approaches, and results in excellent edited images with high quality.


Introduction
With the development of information technology, digital media technology [Shen, Shen, Liu et al. (2018) ;Xiang, Shen, Qin et al. (2018), Shen, Shen, Sun et al. (2018)] has become widely used, and people accordingly have higher requirements for visual media quality. Thanks to powerful image editing tools, people without any knowledge about image processing can easily edit a photo in a visually plausible way. The edit processing of images/videos has become a research hotspot, and color processing technology is an important part of this field. Color processing technology mainly includes color transfer and color editing propagation [He, Gui and Li (2017)].
show that our method can greatly improve the speed of color editing and yields highquality editing results. The Section 2 mainly describes some works related to this paper. The Section 3 will present the framework of our method. The Section 4 is mainly concerned with the presentation of experimental results and drawing comparisons with existing editing propagation methods. Finally, the Section 5 presents the conclusion.

Related work
Our work is inspired by earlier work on color transfer, but mainly uses edit propagation. Color transfer. Color transfer involves mapping the color distribution of a reference image to the input image [Tai, Jia, Tang et al. (2005); Pitié, Kokaram and Dahyot (2007)]. Xiao et al. [Xiao, Wan, Leung et al. (2015)] proposed a color transfer method based on gradient mesh optimization that can effectively maintain the gradient network structure information of images in color transfer results. On this basis, Liu et al. [Liu and Song (2018)] proposed a multi-source image color migration method based on edit propagation. This method proposes an interactive, automatic correction method that uses the modified interactive mark as the control sample in the edit propagation, finds all regions in the image similar to the interactive mark, and combines color migration and apparent migration technology, such that the corresponding color or texture information is migrated to the target image in order to obtain the migration result. Accordingly, our work draws on the advantages of both approaches: We optimize the interpolation model to accurately propagate the interactive color information to the grid vertices. Edit propagation. User interaction-based methods can help users to propagate editing operations from one image region to another region with a similar appearance [An, and Pellacini (2008); Bie, Huang and Wang (2011)]. Li et al. [Li, Ju and Hu (2010)] proposed a fast image editing method based on the radial basis function interpolation model, which can edit other image regions that are not interactive but are similar to the sample points. To solve color mixing in Li's method, Chen et al. [Chen, Chen and Zhao (2012)] employed local linear embedding (LLE); however, when the value of the K-domain [Xiao, Wang and Liu (2018)] is large, the expected color conversion effect cannot be achieved. This method solves the data overflow presenting in the first two methods. Xu et al. [Xu, Yan and Jia (2013)] proposed an image editing method based on the sparse control model, which can achieve better color editing results with a small amount of user interaction. Combining the technical characteristics of Li et al. [Li, Ju and Hu (2010)], Chen et al. [Chen, Chen and Zhao (2012)] and Xu et al. [Xu, Yan and Jia (2013)], we propose an instant image editing method based on the bilateral grid. The method only requires a small amount of interaction to ensure the quality and instant editing of the image. Recently, Chen et al. [Chen, Li, Chen et al. (2016)] proposed learning sparse dictionaries for edit propagation. This method follows the principle of sparse representation to obtain a representative and compact dictionary and perform edit propagation on the dictionary instead; accordingly, it reduces memory consumption while still maintaining a high degree of visual fidelity.

Framework
In order to improve the speed of image color editing, we propose an interactive color editing method, a flowchart of which is shown in Fig. 1. The method includes two main stages: image preprocessing and color editing propagation.
(1) Image preprocessing. In this stage, the user needs to perform color line marking on the local area. However, when the grayscale channel of the color of interactive input is very different from the grayscale channel of the original image, the color will exhibit a large deviation after coloring is complete. Accordingly, we adjust the grayscale channels to accommodate the grayscale channels of interactive colors. Finally, the bilateral grid is used to resample the input image via user interaction.
(2) Color editing propagation. This stage consists of two parts: the construction of the interpolation model and the color editing propagation. When constructing the interpolation model, the grid vertex's feature vector of the original image grid is used as input, while the feature vector of the user edits grid is used for interpolation to construct the radial basis interpolation model. During color editing propagation, the color values of the marked grid vertex are propagated to the target grid, after which the color values of the grid vertex are inversely mapped to individual pixels within the grid vertex. Finally, all pixels in the input image grid are inversely mapped to the RGB color space to obtain the output image after re-coloring. The advantages of our method are that it uses the grid vertex to replaces the image pixel, significantly reducing the amount of image data to be edited, and can greatly improve the speed of color editing.

Image preprocessing
To improve time efficiency, we resample input images with interactions by using a bilateral grid [Chen, Paris and Durand (2007)]. The bilateral grid can be described as a five-dimensional array where the first three dimensions represent the Lab value and the last two dimensions correspond to the image space coordinates. As a result, the input image data can be described in the high-dimensional feature space.
It should be noted that all experiments in this paper were performed in the Lab space. The related parameters l s , c s and s s are respectively the sampling rate of the gray value axis, ab channel, and the spatial coordinate axis ( , ) x y . Intuitively, the size of the bilateral grid is inversely proportional to the sampling rate of each dimension. By setting the appropriate sampling rate, the number of bilateral grid vertices will be much smaller than the total number of pixels in the image, which can greatly reduce the amount of image data needing to be edited during color editing propagation. In this paper, given an input image I with user interaction, in a bilateral grid of a specified size, each image pixel will be mapped to the corresponding grid cell according to the following formula: where [.] is the closest-integer operator for calculating a grid vertex coordinate corresponding to each pixel. The homogeneous coordinates ( ( , ),1)

I x y
are used to accumulate the color values and the total number of pixels in each grid cell. In order to define the subsequent bilateral grid-based editing propagation, we also need to obtain the color values i a C of each grid vertex i v : N is the total number of unmarked pixels in the grid cell. We next need to calculate the coordinate values of the grid vertex. However, if we use the average coordinate value of the pixel points contained in the grid to define the grid vertex coordinates, a coordinate value overflow may occur. In order to solve this overflow, the coordinate value i X of the grid vertex i v is defined as the coordinate value of the last pixel that enters the grid. Since the color values of all marked pixel points of the same color are the same, the color value ' j a C of the marked grid vertex j v is defined herein as the color value of the marked pixel point. Thus, the input feature vector for each grid vertex i v is represented as ( , , , ) Since the feature vector of the grid vertex does not contain gray values, the above two feature vectors are four-dimensional. It should be noted that the grid cells containing black marker pixels do not require calculation of the color values and coordinate values of their vertex; these grid cells will also be ignored in the subsequent re-coloring of the grid vertex.

Color editing propagation
In the bilateral space, we transform the editing propagation into a discrete point interpolation problem. We define the total set of all marked grid vertices as G .
In this paper, we use the least squares method to define the energy function [Amiri-Simkooei, Mortazavi and Asgari (2015)]. Constructing the energy function in this way allows for minimization of the editing difference between the vertex of the re-coloring grid and the vertex of the marked grid after the energy function is minimized. The energy function is as follows: : where vi a is the weight coefficient of the proportion of the control grid vertex i v in h(f) , which is unknown. In the subsequent optimization edit, we will explain how to solve i v a .
f is the feature vector of the vertex of the grid to be edited, and the radial basis function is Minimizing the energy function is actually a problem of solving a series of linear equations. The classical method for solving a series of linear equations is to use the normal equation [Huang and Chen (2002)]. Substituting Eq. (3) into Eq. (4) obtains a complete energy function J : In Eq. (5), the only unknown is the weighting factor, which is obtained by substituting the matrix: Where X is the input matrix, a is the coefficient matrix to be solved, and Y is the The above energy formula is obtained by partial derivative of the coefficient matrix and calculated by matrix derivation: Let the derivative 0 J(a) a = ∇ be expressed as the normal equations: Therefore, the minimum value of a is: Once the relationship function h is determined, the grid vertex can be re-colored. After re-coloring the grid vertex, the next step is inverting the color of the grid vertex and the vertex of the marked grid to each pixel point it contains, so that the color value of the pixel in the grid is consistent with the color value of the grid vertex. However, the gray value remains unchanged. Finally, all pixels in the input image grid are inversely mapped to the RGB color space to obtain the output image after re-coloring.

Experimental settings
We test on the Windows 7 operating system, which runs on a PC equipped with a 2.80 GHz CPU and 8 GB of memory. Firstly, the optimal sampling rate parameters are obtained, after which the effectiveness of the proposed method is verified by the quality and efficiency of the experimental results.  Fig. 2(c), once the sampling rate is lower, the excessive bilateral grid cells will leave most pixels out of a grid. The number of bilateral grid cells is reduced as the sampling rate increases, which facilitates subsequent color editing and propagation. However, a high sampling rate will result in fewer grids. Once a grid contains a tremendous number of pixels, this will cause a numerical overflow, as shown in Fig. 2(b). After repeated adjustment of the parameters, we select the l s = c s =15 sampling rate as the subsequent experimental parameters. We selected four images randomly downloaded from the Internet as the experimental images, which have been named starfish, fruit, children and flower. All of these are natural images that can generally be divided into two classes: one class contains starfish and fruit, which are the images with repeated scene elements, while the other contains children and flower, which are the images with complex scene elements. To conduct the comparison, we employed four different color editing methods to be compared with our method [Li, Ju and Hu (2010); Chen, Chen and Zhao (2012); Xu, Yan and Jia (2013); Chen, Li, Chen et al. (2016)], which are referred to as Li et al. [Li, Ju and Hu (2010); Chen, Chen and Zhao (2012); Xu, Yan and Jia (2013); Chen, Li, Chen et al. (2016)] respectively (because Chen has proposed two different methods, we distinguish between them using the year of publication). In order to get the color edit output image, each method needs to interact with the input image before color editing takes place. Tab. 1 presents the user interaction statistics of the images for the different methods. The interacted images are then color edited, and the results of this process are presented in Fig. 3. Chen et al.'s [Chen, Chen and Zhao (2012)] requires fewer user interactions for a singlecolor image. However, when color editing is performed on an image with a large color difference, the number of user interactions increases. The advantage of Xu et al.'s [Xu, Yan and Jia (2013)] method is that the color editing propagation takes the spatial positional features into account, enabling variable color editing of the same area. However, its shortcoming is also obvious in that the number of interactions is slightly increased. For both Chen et al.'s [Chen, Li, Chen et al (2016)] method and our method require few user interactions, thus we make the same number of interactions on the same images using these two methods, while making more interactions by Chen et al.'s [ Chen, Chen and Zhao (2012); Xu, Yan and Jia (2013)] method.
In the subsequent method quality comparison, the quality of the color editing results was quantitatively evaluated using the image's structural similarity index, or SSIM [Wang and Bovik (2002); Wang, Bovik, Sheikh et al. (2004)]. For the original target image 1 I and the color edited image 2 I , the SSIM can be obtained using the following formula: c are the constants added to prevent the denominator from being zero. Fig. 3 shows the color editing results of our method, Li's method, Chen et al.'s [Chen, Chen and Zhao (2012)] method, Xu et al.'s [Xu, Yan and Jia (2013)] method and Chen et al.'s [Chen, Li, Chen et al. (2016)] method for images with repeated scene elements and complex scene elements. For the color editing images in Fig. 3, we calculated their corresponding SSIM, which are presented in Tab. 2.

Image color editing efficiency comparison
Color editing methods usually contain two stages: (1) importing image data and building a model; (2) color editing propagation. We calculated the running time of the two stages of each method in Fig. 3. The results are listed in Tab. 3. Tab. 3 compares the time taken to conduct image color editing for the images in Fig. 3. As can be seen from the Tab. 3, although the proposed method takes the most time in the first stage, it takes the least time by far in the second stage, with the result that the total running time of the proposed method is minimal. In the proposed method, the pixels are clustered into a grid in a bilateral space, which greatly reduces the selection of sample points, the computational complexity and the memory loss. Since the establishment of a bilateral grid is required in the first stage, our method requires more time in the first stage, but much less time in the second stage. Thus, our method requires only a small amount of user interaction to complete the color editing propagation task in a short time. The time efficiency of our method is thus highest than that of existing color editing methods. Li et al.'s [Li, Ju and Hu (2010); Chen, Chen and Zhao (2012)] method involve performing similarity judgment and color editing propagation on a single pixel; therefore, these methods take a long time. Xu et al.'s [Xu, Yan and Jia (2013)] method needs to calculate the spatial position feature, meaning that it takes an even longer time. Finally, Chen et al.'s [Chen, Li, Chen et al. (2016)] method uses a sparse dictionary for editing propagation; sparse dictionaries can reduce computations and significantly reduce time loss.

Conclusion
In this paper, we propose a fast image color processing method based on bilateral grid. By providing only a small amount of user interactions, different regions of the image can be edited simultaneously by combining the color features and coordinate features of the image. Using bilateral grids to cluster pixels greatly reduces the computational effort and memory loss, with the result that the time efficiency of the proposed method is improved. A grid with pixels similar to the color of the sample pixels and without marked pixels is defined as an edited grid, which can be used to accurately re-color the vertices of the edited grid. Accordingly, the proposed method realizes fast, accurate and high-quality color editing for images.