Abstract
When the art images are restored by the virtual restoration method, there are problems such as insufficient clarity and more noise in the reference image. An improved cartoon-texture decomposition method for art image fusion is proposed. The nonlinear local total variation component is used as the indicator function of image decomposition to obtain the image cartoon structure component and texture oscillation component. According to the oscillation component’s strong repetitiveness and structural directionality, the image texture part is filtered by combining the improved directional diffusion algorithm. Using the sparse coefficients of the fused cartoon component and the sparse coefficients of the texture component, the cartoon and texture of the image is inverse transformed and weighted and summed to obtain the recovered image after fusion. The experimental results show that this paper has a good effect after image fusion, and the recovered clarity is higher, which can better express the basic information of the source image; compared with several decomposition fusion methods commonly used at present, this paper has better recovery performance and detail processing ability and preserves the edge information of essential details in the image while filtering and denoising and is more excellent in objective performance evaluation indexes such as PSNR and SSIM. It can be used as a reference basis in the restoration process of art images.
1. Introduction
There is a potential correspondence between the restored artwork and the viewer’s appreciation. The performance before and after restoration can create a feeling that there is a huge difference between the viewers. Therefore, artworks need to be photographed and archived before restoration to witness future comparison [1, 2]. In addition, the process of restoring artworks often requires the original image of the work as an auxiliary basis to grasp the color, style, composition, and other important information of the work, which not only avoids the risk of “secondary damage” that may result from direct restoration but also provides the necessary data resources for digital display and dissemination [3]. However, in the process of art image formation, dissemination, and storage, due to the influence of imaging equipment or shooting environment and other factors, a certain degree of distortion will inevitably occur, resulting in blurred or severely noisy art images, where clarity cannot be guaranteed. Detailed information cannot be well presented, which brings greater trouble to the restoration of artworks and seriously affects the subsequent restoration work [4]. Therefore, it is of good research value and practical significance to restore the degraded art images to obtain clear and complete images.
Image restoration technology processes the image by the computer. It reconstructs the image without improving the accuracy of imaging equipment to obtain the important information lost in the image, improve the image quality, and has low application cost and good practicability [5]. The early research results of image restoration technology mainly introduced some technologies and methods used in digital signal processing, such as inverse filtering technology. In addition, it is found that the decline of image quality can be solved by the state-space model, autoregressive moving average model, nonlinear parameter identification, complex adaptive theory, and other methods. These methods use modern control technology and greatly promote the development of image restoration technology [6–8]. Therefore, more and more scholars are committed to finding methods with strong decomposition ability, fast operation speed, and conducive to the fusion of results. Afonso et al. [9] proposed an improved variable splitting method based on the augmented Lagrangian method (ALM) algorithm and studied the image restoration problems based on framework and constraints. The image restoration quality and processing speed are improved, and the restoration effect is relatively good. Li et al. [10] proposed a multicomponent decomposition method of learning images based on a discriminant dictionary. This method uses weighted kernel norm regularization and sparse constraints to characterize the coarse structure and fine components. It uses weighted Schatten sparse kernel norm regularization to extract rough structure from the separation model. The fusion result of this method can retain more detailed information and perform well in the objective evaluation index. Zhou et al. [11] proposed a learning method for medical image fusion dictionary, which uses multilayer details of the image to enhance weak information, introduces a multiscale sampling method to realize multiscale representation of patches, completes image patch clustering through neighbourhood energy index and multiscale spatial frequency index, and uses K-SVD combined with subdictionary to construct fusion dictionary. The calculation efficiency is high, and the objective evaluation standard is excellent. Niknejad and Figueiredo [12] used the ALM algorithm to restore the image polluted by Poisson noise and achieve a good restoration effect. Li et al. [13] proposed an image fusion method based on three-layer decomposition and sparse representation, which decomposes the source image into high-frequency and low-frequency components by using the high-pass characteristics of noise and realizes the fusion and denoising of high-frequency components by reconstructing error parameters. A structured texture decomposition model is designed according to the details and energy of low-frequency components. The corresponding fusion rules are used to complete the fusion of the two. Experimental results show that this method can effectively suppress the noise problem of images and has better performance in subjective vision and quantitative evaluation. At the same time, Li et al. [14] introduced the double-layer decomposition image fusion method through the joint bilateral filter, designed a local gradient energy operator based on structural tensor and neighbourhood energy to fuse the energy layer, and verified the reliability of the fusion method through multiple groups of images. The results show that this method has the excellent visual quality and quantitative evaluation performance, good real-time performance, and high fusion efficiency. Ono et al. [15] proposed a new matrix block kernel norm BNN to describe the prior information of globally dissimilar textures but highly repeated local texture patterns. The TV + BNN model uses the image cartoon total variational regularization method. After rotation and overlap, the image texture components have a low rank and can better decompose the image textures in different directions.
Most decomposition methods have more or less false boundaries in the smooth region, loss of details in the texture part, and ladder effect in the cartoon part after image processing, resulting in poor image definition after fusion. Therefore, this paper proposes an art image fusion method based on improved cartoon texture decomposition. The nonlinear local total variation is used as the indicator function of image decomposition to obtain the cartoon structure component and texture oscillation component. The improved directional diffusion algorithm filters the texture part of the image. Based on maintaining the structural integrity of the cartoon part and the fine structure of the texture part, the influence of noise on the fusion result is reduced. Finally, the cartoon component sparse coefficient and texture component sparse coefficient are used to transform the cartoon and texture of the image, and the clear art image is obtained after weighted fusion. This method is suitable for complex art images and can effectively solve many practical problems such as image denoising and restoration, edge detection and texture recognition. It is superior to the traditional fusion method in signal-to-noise ratio and recognition rate. It can be used as a reference in art image restoration.
2. Improved Cartoon-Texture Decomposition
2.1. Cartoon-Texture Decomposition
The art image after cartoon texture decomposition gives a texture part containing the source image oscillations and noise and a cartoon part containing the source image structure and geometry [16]. The texture part can represent the texture of the clear part of the source image containing the noise and oscillation in the image, and the cartoon part is the difference between the source image and the texture part, so the clear part of the cartoon part becomes blurred after the image cartoon texture decomposition [16, 17]. Compared with image decomposition methods such as wavelet decomposition and curvilinear transform, cartoon texture decomposition can preserve the information in the source image without producing a lack of description of the features in the image, and it has better results in image denoising, restoration, distortion processing, and detail reorganization. Figure 1 shows the source image and the decomposed cartoon and texture images.
By observing Figure 1, it can be seen that although the decomposed cartoon image and the textured image describe the same scene, the degree of blurring and the texture features differ greatly, and thus the clear part of the two images need to be fused to come to enhance the quality of the image. There are more methods of image fusion, such as, fusion algorithms based on sparse representation and nonlocal regularized image fusion based on variational framework. This type of algorithm generally treats the image as a single signal [18]. However, under realistic conditions, a single artistic image often consists of several different components, such as the cartoon component and the texture component, which contain information content with different characteristics from each other. If the image is decomposed into different parts and then different fusion rules are applied according to the characteristics of each part, the quality of the fused image will be further improved.
2.2. Local Full Variational Cartoon-Texture Decomposition
According to the theory of variation and scale space, the cartoon part of the image has a small total variation (TV), and the texture part has a small parametric number. The higher the frequency of the texture component oscillation, the smaller the magnitude. The essence of the total local variation (LTV) cartoon-texture decomposition is to use nonlinear low-pass and high-pass filters to solve the approximate solution of the original variational problem. By calculating the LTV value around each pixel in the graph and comparing it with the LTV after the low-pass filter, the LTV is used as a local indicator function of the pixel to determine the type and attribution of the pixel [19]. After low-pass filter processing, if the LTV of the pixel point changes weakly, it is a cartoon point type classified as a cartoon component; if the LTV of the pixel point decays rapidly after convolution operation, it is a texture point type. And then, keep the cartoon component U unchanged and classify the value of texture pixel point after low-pass filter as cartoon component U. In contrast, the texture component V takes the difference between cartoon component and source image H.
The LTV expression for any pixel p in the source image H iswhere Jσ denotes the Gaussian kernel with standard deviation σ and denotes the convolution operation. is the gradient operator.
Mapping p to , the relative rate of change of LTV of pixel p iswhere is the low-pass filter and determines the decomposition scale, if the LTV changes under the low-pass filter condition, its local oscillation can be derived from . When tends to 0, it indicates that the LTV of the pixel point does not change much, and the low-pass filter has little effect on the LTV of the point, which belongs to the cartoon component U; conversely, when tends to 1, it indicates that the LTV of the pixel point has a large relative rate of change and decays fast, which belongs to the texture component V. A set of nonlinear high-pass and low-pass filters can be obtained by solving the weighted average of and H according to the relative rate of change of LTV of the pixel point.where denotes a segmented, nondecreasing soft threshold function taking values in the range [0, 1]. To facilitate later calculations, the soft threshold function is set towhere the parameters n1 and n2 take 0.3 and 0.6, respectively. When is small, the oscillation of function H(p) near pixel p is small, indicating that the point belongs to the cartoon part, that is, U(p) = H(p). Conversely, when is large, indicating that the function H(p) has a large oscillation near pixel p, then U(p) is replaced by .
With this image decomposition method, it is impossible to obtain a cartoon texture separation that can simultaneously take into account the extraction of texture components and keep the edges of the structured part unblurred, which is suitable for contaminated noise-laden images. However, the information contained in the cartoon component U cannot meet the actual needs because some key details are still retained in the texture part. The decomposed texture must be filtered and then weighted and synthesized with the decomposed cartoon component U to obtain the recovered image H.
2.3. Improved Texture Filtering by Directional Diffusion Algorithm
For the filtering correction of the texture part after the decomposition of the art image, the traditional interpolation and amplification are used first. Then, the image is filtered and smoothed by the improved direction-based diffusion algorithm, and the edge information is retained. Finally, the low-frequency information of the acquired image is used as the basis for image fusion.
The directional diffusion model belongs to the partial differential equation of nonlinear diffusion along the tangential direction parallel to the edges, i.e., perpendicular to the direction of the gradient vector [20]. The diffusivity of an image is the rate of change of the image gradient, and the expression iswhere t is the time and denotes the unit vector perpendicular to the image gradient .
Since the art image is a two-dimensional plane, the image gradient component calculation can be expressed aswhere is the component of gradient and x and y represent low resolution image and source image, respectively.
According to the p-m diffusion equation [21], the selection of edge function significantly impacts the diffusion behavior of texture. When the value of the edge function is unreasonable, the image edge gradient becomes more extensive, resulting in the occurrence of reverse diffusion. Therefore, this paper solves this problem by introducing the local coordinate system . The unit vector is parallel to the gradient direction in the coordinate system, and is perpendicular to the gradient direction. Due to , there are
By substituting and deriving, we can get
After solving the texture part of the image in different coordinate systems, the final expression of directional diffusion can be obtained:
This method is used to filter the image texture. After several iterations, a smooth image can be obtained, ensuring that the texture part’s critical information is preserved. The image remains sensitive to the edges of the image while filtering.
3. Image Fusion
3.1. Cartoon Component Sparsity Factor Fusion
The cartoon component of an image mainly reflects some structure and color information, which is sensitive to the human eye. At the same time, the cartoon component of natural images often reflects a large amount of primary information in the original image, so the purpose of fusing cartoon components is to make such information more prominent [22]. Some scholars use the absolute maximum fusion rule to fuse the cartoon component sparse coefficient of the decomposed image. However, this method does not reflect the proportion of the original information in the image and will make the resulting image too sharp in the information representation. In Figure 2, the cartoon part of the water bucket is not ideal. The absolute proportion of the fused data cannot be reflected by simply using the “relative-absolute maximum” fusion rule. Therefore, this paper uses the “absolute maximum” and “relative-absolute maximum” fusion rules to fuse the sparse coefficients of the cartoon components of the fused image.
M and N is the row and column values of the sparse coefficient matrix, respectively. Then, according to the “relative-absolute maximum” fusion rule, the sparse coefficients of the combined cartoon components are
The “absolute maximum” fusion rule is used to obtain
Also, since the cartoon component of the image partly reflects the primary energy information of the original image, the information entropy is used to determine the weights of and . Finally, the sparsity coefficient of the cartoon component after fusion can be obtained aswhere IE is the information entropy of the cartoon component of the image to be fused.
3.2. Texture Component Sparse Coefficient Fusion
Since the texture component reflects the details such as edge information within the original image, in general, the weight of edge information in the final fused image is different between different fused images. So, it is crucial to calculate the weight of the texture component in the final fused image for each image to be fused. The same sparse coefficient fusion is used for texture component fusion. However, since the texture component of the image partially reflects the edge information of the original image, the edge intensity is used as the basis for weight judgment [23]. Then, the sparse coefficients of the fused texture components arewhere IE is the information entropy of the cartoon component of the image to be fused.
3.3. Decomposition and Fusion Process
The art image decomposition and fusion process is shown in Figure 3.
The process of art image fusion with improved cartoon texture decomposition is briefly described. First, the input image H is decomposed using the local full-variance cartoon texture algorithm to obtain the cartoon image Hc and the texture image Ht. The texture image is filtered using the improved directional diffusion algorithm to obtain the optimized image texture image . Then, the sparse coefficients of the fused cartoon component and the sparse coefficients of the texture component are inverted. Finally, the results of the inverted conversion are weighted and summed, and the fused image F is output.
4. Experiments
4.1. Experimental Preparation
The experimental environment of this paper is Intel Core i7 3.6 g CPU, 16 g SDRAM, and MATLAB 2019. In order to verify the feasibility and effectiveness of the proposed method, two standard art images (Vang. BMP, bri. BMP) are selected for experimental simulation. The texture of image Vang is apparently, while image bri is relatively fuzzy. Use a size of 7 × 7. The Gaussian check source image with the mean value of 0 and standard deviation of 0.6 is blurred, and the white noise with a standard deviation of 10−3 normal distribution is added to obtain the blurred high noise image. The source image and blurred image are shown in Figure 4.
(a)
(b)
(c)
(d)
This paper adopts the general image evaluation standard’s peak signal to noise ratio (PSNR). The structural similarity index measurement (SSIM) index is used to evaluate the main features of the image, such as comprehensive brightness, contrast, and structure. The closer the SSIM value is to 1, the closer the fused image is to the actual image, and the better the algorithm performance [24, 25]. Objective evaluation indicators are defined as follows:
In the formula, MSE is the mean square error between source image H and fused image F. and are the average values of image H and F, and and are the variances of image H and F.
4.2. Experimental Results and Analysis
The blurred and noisy images (Figures 4(b) and 4(d)) are decomposed into cartoon textures. The value of the decomposition scale parameter is 5, and the value of the soft threshold function is 0.3 and 0.6, respectively. The fused cartoon component sparse coefficient and texture component sparse coefficient are used for inverse transformation, and the weighted sum is used to output the result:
Among them, α and β are the weights of cartoon and texture parts, respectively. Considering the nature of the selected image, the image Vang BMP weight α = 0.7, β = 0.15; image bri weight of BMP α = 0.6, β = 0.25.
The image cartoon texture decomposition and fusion results are shown in Figures 5 and 6. Among them, (a) and (b) are the decomposed texture part and texture filtered image, respectively, (c) is the decomposed cartoon part, and (d) is the image fusion result.
(a)
(b)
(c)
(d)
(a)
(b)
(c)
(d)
From Figures 5 and 6, we can see that, after the cartoon texture decomposition of the blurred image by the local total variation method and the texture filtering by the improved directional diffusion algorithm, the effect of image fusion is relatively good, which can better represent the basic information of the source image, and the image performs well in detail and brightness, with high definition.
4.3. Performance Comparison of Different Algorithms
In order to further verify the performance of the method in this paper, the traditional cartoon-texture decomposition method, the nonlocal regularization iterative method, the global TV iterative method, and the method in this paper were used for comparison experiments taking image vang as an example. The image comparison results are shown in Figure 7.
(a)
(b)
(c)
(d)
As shown in Figure 7, the fused images from the traditional cartoon-texture decomposition method and the global TV iteration method contain more noise. There will be different degrees of fuzziness and poor definition in some parts. The nonlocal regularization iterative method has dramatically improved the brightness, noise, and clarity, but there is a problem of local block effect. Because the cartoon texture decomposition method is improved and the directional diffusion algorithm is used to filter the image texture, the fusion result has a high definition, less noise, richer texture, no local block effect, and relatively excellent performance. Table 1 shows the comparison results of performance indexes of different methods.
Comparing the related indexes in Table 1, we can see that this method is superior to the other three methods in PSNR and SSIM performance indicators, has better recovery performance and detail processing ability, and retains important edge details in the image while filtering. Compared with the current image fusion methods, this method is shorter, better in real-time, more efficient in calculation, and more suitable for complex artistic images. It can be used as a reference in the process of art image repair.
5. Conclusion
According to the different sensitivity of cartoon components and oscillation components to Gaussian noise, this paper proposes an improved cartoon-texture decomposition method for artistic image fusion. By analyzing the geometric structure characteristics of the image, the variational problem of the source image is quickly approached by using the nonlinear low-pass filter to avoid the generation of false boundary and step effect and ensure that the decomposed cartoon section maintains a high level of detailed information. The texture part is also filtered using an improved directional diffusion algorithm to avoid the destruction of the image structure by medium and large-scale noise and to preserve the segmental smoothness and small-scale detail information of the image structure. The cartoon and texture components are inverted, respectively, using the cartoon component sparsity coefficients and texture component sparsity coefficients. Finally, a clear recovered image is obtained after weighted fusion. The experimental results show that the image fusion performance of the proposed method is relatively reasonable and can express the basic information of the source image. The image performs well in detail and luminance, with high definition. Compared with the traditional cartoon-texture decomposition, nonlocal regularization iteration, and global TV iteration, the PSNR and SSIM indexes of the algorithm of this paper have been greatly improved. Therefore, the method in this paper has good adaptability and robustness and can be used as a reference basis and method for art image-assisted restoration.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The author declares no conflicts of interest.