A Variety of Choice Methods for Image-Based Artistic Rendering

Abstract

Neural style transfer (NST) is a technique based on the deep learning of a convolutional neural network (CNN) to create entertaining pictures by cleverly stylizing ordinary pictures with the predetermined visual art style. However, three issues must be carefully investigated during the generation of neural-stylized artwork: the color scheme, the strength of style of the strokes, and the adjustment of image contrast. To solve these problems and select image colorization based on personal preference, in this paper, we propose modified universal-style transfer (UST) method combined with the image fusion and color enhancement methods to design a good post-processing framework to tackle the three above-mentioned issues simultaneously. This work provides more visual effects for stylized images, and also can integrate into the UST method effectively. In addition, the proposed method is suitable for stylized images generated by any NST method, but it also works similarly to the Multi-Style Transfer (MST) method, which mixes two different stylized images. Finally, our proposed method successfully combined the modified UST method and post-processing method to meet personal preference.

1. Introduction

Automatic image colorization is one of the popular research topics in image processing. In general, image colorization is the process of adding colors to mono-chroma images to make them more aesthetically appealing and perceptually meaningful. However, the most crucial task in this paper is how to design a multiple-choice method depending on personal preference to define an appropriate color from the reference images for propagating to the target image. Among deep learning techniques, the convolutional neural networks (CNN) have been validated to be quite useful for image analysis and applications [1,2,3]. For example, CNN-based art style transfers have recently been employed to produce impressive artwork [4,5,6,7] to such an extent that a new market has been created for the mobile apps that can stylize user-provided images with a particular art style. The popularity of commercial apps such as Prisma [8] and Deepart.io [9] have been apparently increased. Neural style transfer (NST) is an optimization technique used to take two images—a content image C and a style reference image S (such as an artwork by a famous painter)—and blend them together so the output image looks like the content image, but “painted” in the style of the style reference image. In general, there are two main categories of style-transfer algorithms, namely Image-Optimization-Based Online Neural Methods (IOB-NST) and Model-Optimization-Based Offline Neural Methods (MOB-NST) [10]. According to the development process, there are two major approaches in the generative neural method, that is, Fast Style Transfer (FST) [4] and Multi-Style Transfer (MST) [4]. During the training phase in art education and industrial applications, the MST is advantageous in effectively and efficiently producing a variety of styles for the specified image. Further, Li et al. [11] proposed the seminal work that transfers an arbitrary visual style to content images, achieving so-called universal style transfer (UST). The main point of this paper is completely independent of the image style during the training phase and aims to make it “style learning-free”. Recent, Wang et al. [12] proposed a new knowledge distillation method to reduce the model size, which exploits the phenomenon that encoder–decoder pairs in UST construct an exclusive collaborative relationship, which is regarded as a new kind of knowledge for style transfer models.

Although the artistic effect of the NST-based methods is impressive, there are three issues worth noting. The first is the $S_C$ image copies the color distribution of the S image which might be undesirable for most artwork applications. In order to tackle this problem, one possible method is to use luminance-only transfer [13] by a 3×3 kernel to convert the C and S images from the RGB color space to the YIQ space, respectively. In the inverse conversion, the I and Q components of the C image are retained and the Y component of the C image is replaced by that of the $S_C$ image. Consequently, the color information of the C image can be preserved in the new image while the spatial details of the new image are inherited from the $S_C$ image. Second, in the control of the trade-off between content and style in the $S_C$ image, the predefined relative content-style weight plays a leading role. If the stylized effect is not as expected, one solution is to use the blending technique to adjust the mixing ratio between C and $S_C$ images [14]. Therefore, retraining is not required at all. The third concern is the use of batch or instance normalization in FST and MST. It is particularly useful for forwarding propagation when the number of CNN layers is too large. However, the normalization may significantly degrade the image contrast performance. To improve the visual effect, some effective contrast-enhancing methods have been proposed [15,16]. Gatys et al. [15] present intuitive ways to control NST. The authors proposed a method for transferring different styles to different spatial regions of the output image during stylization to substantially improve the quality and flexibility of the existing method. In addition, Jing et al. [16] propose a fine and flexible stroke size control approach for Fast Style Transfer. The core component of their algorithm is a Stroke Pyramid module, which learns different stroke sizes with adaptive receptive fields. By combining the proposed runtime control strategies, this work can achieve continuous changes in stroke sizes and produce distinct stroke sizes in different spatial regions within the same output image.

It is noted that almost all the implementations of the proposed algorithms are segmental. When the effect is not as expected, the style transfer procedures must be executed iteratively. In FST and MST, the style evaluation is very fast, and the most time-consuming operation is in the training phase, since what most of the users are concerned with is the style itself, not the style training. Thus, it will not be acceptable to spend too much time enhancing the $S_C$ image. In addition, to be effective, a good post-processing operation is also required to be fast and simple to use.

In this work, the image fusion algorithm can be applied to post-process the $S_C$ images. This work proposes a versatile post-processing framework that combines Brovery transform (BT) [17], blending technique, and intensity conservation direct decorrelation stretch (ICDDS) [18] methods to tackle the above-mentioned three issues simultaneously. This framework composed of six modules is implemented by a window-based program which is used to demonstrate the adjustment of the $S_C$ image. To validate the proposed techniques, several experiments are conducted with the stylized image produced from UST methods.

The remainder of the paper is organized as follows. Section 2 briefly introduces the related works. A detailed description of the proposed strategy for automatic colorization in neural stylized image is given in Section 3. The experimental results are discussed and described in Section 4, and Section 5 gives the conclusion.

2. Review of the Related Works

2.1. The Classical UST Method

A novel approach of image style transfer, namely UST, is proposed by Li et al. [11]. In this work, the style transfer is regarded as an image reconstruction process, accompanied by feature transform operations. The advantage of this method is that it can transform arbitrary visual style images without training on any predefined styles. As can be seen in Figure 1a, the UST employs the VGG-19 network as an encoder, and the decoder is designed as a symmetric network of VGG-19. This training is conducted by the Microsoft COCO dataset. UST extracts the image features from the five layers of the VGG-19 network and trains five corresponding decoders to achieve image reconstruction. The single UST module is shown in Figure 1b, the method is embedded with a pair of whitening and coloring transforms to an image reconstruction network. The steps of detail are shown as follows:

• The C and S images first employ the encoder network (VGG-19) to get the feature maps $f_C$ and $f_S$, and calculate their respective eigenvalues ($D_C$, $D_S$) and eigenvectors ($E_C$, $E_S$), respectively.
• The output ${\tilde{f}}_C$ is whitening transform by $f_C$, but the feature map ${\tilde{f}}_{CS}$ is coloring transform by the projection basis of $f_S$, $D_S$, and $E_S$.
• The new ${\tilde{f}}_{CS}$ is blending with ${\tilde{f}}_{CS}$ and $f_C$, and using the parameter α to control. A simple blending technique is applied to trade-off between content and style in the output image.
• The stylized image $S_C$($I_x$) of a single UST module is reconstructed by the decoder network, as shown in Figure 1b. To obtain a more refined effect, the multi-module superposition from coarse to fine has been shown in Figure 1c.
• The Figure 1c is the complete framework of UST. The X of the UST_X is the X levels being used in VGG19 features ($\mathrm{X}=1, 2,\dots , 5$). UST_5 means that it is only using the 5th modules. The UST_5,4 means that it is using the 5th and 4th modules without the 1st, 2nd, and 3rd module, and so on.

As a summary, the stylized image $S_C$ is controlled by parameter α and the number of modules in UST. Different from NST, FST, and MST, the UST method is the ideal combination between CNN networks and pattern recognition methods. A versatile implementation can be found in [11].

2.2. The Image Fusion and Color Enhancement for the Neural Stylized Imagery

The neural stylized effect is impressive, but it is limited in certain respects as described in Section 1. Although the problems can be solved separately, it may be quite time-consuming and cumbersome. Aimed at finding an effective solution, the image fusion and the color enhancement by ICDDS technique used in remote sensing community are applied. In the following, IHS, BT, and ICDDS methods will be briefly introduced.

2.2.1. Image Fusion by IHS and BT

Among various image fusion methods for remote sensing images, FIHS and BT are frequently used. They can be represented as [17,19]

$$\left[\begin{array}{c}{R}_{FIHS}\\ G_{FIHS}\\ B_{FIHS}\end{array}\right]=\left[\begin{array}{c}R+\left(Pan-I\right)\\ G+\left(Pan-I\right)\\ B+\left(Pan-I\right)\end{array}\right] and \left[\begin{array}{c}{R}_{BT}\\ G_{BT}\\ B_{BT}\end{array}\right]=\frac{Pan}{I}·\left[\begin{array}{c}R\\ G\\ B\end{array}\right],$$

(1)

where ${\left[R_{FIHS} G_{FIHS }{B}_{FIHS}\right]}^T$ and ${\left[R_{BT} G_{BT} B_{BT}\right]}^T$ are the fused image by the FIHS and BT, respectively, $I=\left(R+G+B\right)/3$ is the intensity component of the low-resolution RGB image, and $Pan$ is the high-resolution panchromatic image.

The fused images of various IHS-like and BT-like approaches can be evaluated in the IHS color space by the method reported in [19]. The performances for FIHS and BT methods are:

$$I_{FIHS}=I+\left(Pan-I\right)=Pan \mathrm{vs}. I_{BT}=\frac{Pan}{I}\times I=Pan$$

(2)

$${\phi }_{FIHS}=\phi \to H_{FIHS}=H \mathrm{vs}. H_{BT}=H$$

(3)

$$\frac{S_{FIHS}}{S}=\frac{I}{I_{FIHS}}=\frac{I}{Pan} \mathrm{vs}. \frac{S_{BT}}{S}=\frac{I_{BT}}{I}=\frac{Pan}{I}$$

(4)

where $S=1-\frac{3\min \left(R,G,B\right)}{R+G+B}=\frac{I-a}{I}$, and $a=\min \left(R,G,B\right)$, which is the definition of saturation in the nonlinear RGB-IHS conversion model [20]. From Equations (2)–(4), it can be seen that the hue value of the fused image is the same as the original RGB image, the intensity and saturation values will be altered. Moreover, it can be seen that the saturation value is the main difference in color distortion between FIHS and BT methods. The value is compressed in FIHS when $Pan$ is larger than I, but the value is stretched in BT. This means that the fused images by FIHS and BT suffer from the color distortion problems. Although, the saturation distortion is an important issue in image fusion, it seems negligible in NST because the saturation only affects the contrast of the stylized image without affecting the style in the image. From the color theory point of view, the replacement of the image will inevitably lead to color distortion. Using FIHS or BT to replace the luminance-only transfer is an effective method because they do not require coordinate conversion.

2.2.2. Color Enhance by ICDDS

As mentioned in Section 1, the $S_C$ images produced by the FST or MST are usually with lower contrast. The remote sensing images also have the similar problem. The multispectral bands are often highly correlated and thus unable to be adequately shown on RGB displays by independent contrast stretching, which is caused by the limit of sensor bandwidth. To solve this problem, the Intensity Conservation Direct Decorrelation Stretch (ICDDS) technique has been presented in [18]. ICDDS method is represented as

$$\left[\begin{array}{c}{R}_{ICDDS}\\ G_{ICDDS}\\ B_{ICDDS}\end{array}\right]=\frac{I}{I-k·a}\left[\begin{array}{c}R-k·a\\ G-k·a\\ B-k·a\end{array}\right]$$

(5)

Obviously, the ICDDS approach has a BT-like form, in which an IHS-like operation is included. The following properties can be derived from Equations (2)–(4),

$$I_{ICDDS}=I$$

(6)

$$H_{ICDDS}=H$$

(7)

$$\frac{S_{ICDDS}}{S}=\frac{S_{ICDDS}}{S^{\prime }}·\frac{S^{\prime }}{S}=\frac{I_{ICDDS}}{I^{\prime }}·\frac{I}{I^{\prime }}={\left(\frac{I}{I-k·a}\right)}^2$$

(8)

It is important to recognize that both intensity and hue are preserved in the process and that only the saturation component is changed by controlling the k parameter. Evidently, ICDDS is a suitable tool for color enhancement.

3. The Proposed Methods

3.1. The Modified UST Method

Figure 2a is the C image, and the upper left corner is the S image, the Figure 2b is the $S_C$ image using complete framework of UST with $\mathsf{\alpha }=0.5$, and the Figure 2d is the $S_C$ image using the UST_5,4 with $\mathsf{\alpha }=1$ (no blending). In comparison to Figure 2b,d, there is not much difference from the viewpoint of vision. However, Figure 2c is the $S_C$ image only using the single module (UST_5) with $\mathsf{\alpha }=1$. Although the stylized effect of Figure 2c is impressive, the image has lower contrast. To solve this problem, first of all, we would like to describe the whitening and coloring transforms of UST. Figure 3a is the Gaussian distribution of two-dimensional data. After principal component analysis (PCA), the distribution of Figure 3a is transformed to the distribution of Figure 3b, and this is then transformed to Figure 3c by whitening transform. Then, the coloring transform is an inverse transform module to transform Figure 3c to Figure 3b, and the inverse PCA transformation is used to transform Figure 3b to Figure 3a.

As shown in Figure 3c, when the coloring transform is used to transform to Figure 3b, the eigen matrix $\mathsf{\Lambda }$ is multiplied by the constant k (k > 1), and conduced to the Figure 3d distribution being extended. We refer to this phenomenon as enhance coloring transform. Then, the distribution of inverse PCA transformation is also extended, as shown in Figure 3e. As we can see in Figure 1b, the single UST module can be rewritten as follows:

$${\widehat{f}}_{CS}=E_S{\left(k{D}_S\right)}^{\frac{1}{2}}{E}_S^T{\widehat{f}}_C$$

(9)

$$\Rightarrow E\left({\widehat{f}}_{CS}{\widehat{f}}_{CS}^T\right)=E_S{k}^{\frac{1}{2}}{D}_S^{\frac{1}{2}}{E}_S^{T}E\left({\widehat{f}}_C{\widehat{f}}_C^T\right)E_S{k}^{\frac{1}{2}}{D}_S^{\frac{1}{2}}{E}_S^T$$

$$=k{E}_S{D}_S^{\frac{1}{2}}{E}_S^{T}I{E}_S{D}_S^{\frac{1}{2}}{E}_S^T=k\cdot E\left(f_S{f}_S^T\right)$$

Equation (9) states that the amount of variance in the original digital image becomes larger, which indicates the image has higher contrast. The $S_C$ image only used the single module (UST_5) with $\mathsf{\alpha }=1$ and $k=2$ to obtain the desired results, as shown in Figure 4. The original UST architecture used the five UST modules to generate the style image. However, each module cannot reserve the full features of VGG because of the blending method. The proposed method only used one module (UST_5) without blending, the style image contains the whole features of VGG and obtains the higher contrast. In addition, the proposed method can be practiced in bigger-size images due to the lower computation cost and GPU memory.

3.2. The Post-Processing Framework

For NST, users hope to improve three issues: the color scheme, the strength of the style of the strokes, and the adjustment of image contrast. To satisfy the demands of users, the $S_C$ image from modified UST method (UST_5) combined with the image fusion and color enhancement methods to design a good post-processing framework to tackle the above-mentioned three issues simultaneously. Equation (10) can use the blending technique to re-adjust the mixing ratio between the C and $S_C$ images. A possible approach is to integrate the blending technique as shown below

$$\left[\begin{array}{c}{R}^{\prime }\\ G^{\prime }\\ B^{\prime }\end{array}\right]=\frac{k_1·I_B+\left(1-k_1\right)·I_A}{I_B-k_2·a_B}·\left[\begin{array}{c}{R}_B-k_2·a_B\\ G_B-k_2·a_B\\ B_B-k_2·a_B\end{array}\right]$$

(10)

where $a_B=\min \left(R_B,{\mathrm{G}}_B,B_B\right)$, the parameter $k_1$ ranging from 0 to 1 is for adjusting the blend ratio between the C and $S_C$ images to adjust the stroke strength, $k_2$ ranging from −1 to 1 is for controlling the degree of saturation compression or stretch. $I_B$ and $I_A$ are the C or $S_C$ images to solve the color scheme. Based on Equation (10), six different modules are as follows

• The content-preserved ($C_P$) module when $k_1=k_2=0$,$I_A$ is the content image, and $I_B$ is the stylized image.
• The style-preserved ($S_P$) module when $k_1=k_2=0$, $I_A$ is the stylized image, and $I_B$ is the content image.
• The $S_P$ module with content/style blending when $0 \le k_1\le 1$, $k_2=0$, $I_A$ is the stylized image, and $I_B$ is the content image.
• The $C_P$ module with content/style blending when $0\le k_1\le 1$, $k_2=0$, $I_A$ is the content image, and $I_B$ is the stylized image.
• The $S_P$ module with content/style blending and the contrast adjusting when $0\le k_1\le 1$, $-1<k_2<1$, $I_A$ is the stylized image, and $I_A$ is the content image.
• The $C_P$ module with content/style blending and the contrast adjusting when $0\le k_1\le 1$, $-1<k_2<1$, $I_A$ is the content image, and $I_B$ is the stylized image.

The two most important modules of the proposed approach are (5) and (6). Modules (5) and (6) can not only combine the spatial details of the style/content image with the color information of the content/style image, but also control the compression or stretch of the image saturation. By making some adjustments the value of $k_2$, we can solve the problem of low contrast. Thus, they can provide multiple-choice methods for automatic colorization in the new stylized image which will be shown in the next section.

4. Experiment Results and Analysis

The proposed versatile framework has been implemented by a window-based program, as shown in Figure 5. Based on the concept of “What You See Is What You Get” (WYSIWYG), the window consists of five sub-windows which are called the main window and four Thumbnail Previews, respectively. Initially, the $S_C$ and C images are read and displayed on the upper left and upper right sides of the main window. Next, select the content- or the style-preserved module and execute. The resultant $C_P$ or $S_P$ images will be displayed at the bottom left or bottom right side of the main window. The default is module (1), however, the user can easily change the module and adjust the parameters $k_1$ and $k_2$ to find the most satisfactory module. The processing results are obtained and analyzed in real time. In the following, several experiments are conducted.

Effect of Different Modules of the Post-Processing Framework

The first experiment demonstrates the operations and corresponding results of the proposed versatile framework. The C image is a digital painting image, and its $S_C$ image is generated by the UST_5. The $S_C$, C, and the resultant $C_P$ and $S_P$ images are shown in Figure 5, respectively. Compared to the $S_C$ and C images, both the $C_P$ and $S_P$ images achieve their intended functionality. By the default settings in modules (1), we have $k_1=k_2=0$. Thus, the $C_P$ image can keep the spatial details of the C image and maintain the color information of the $S_C$ image while the $S_P$ image is the result of the opposite function provided by modules (2).

To evaluate modules (3) and (4), the parameter $k_1$ is set between 0 and 1 to obtain an image with different mixing effects. The $C_P$ images with $k_1=0.2, 0.5, \mathrm{and} 0.8$ from left to right in sequence are displayed on the upper row in Figure 6. The bottom row is the $S_P$ images with the same $k_1$ values as $C_P$. Compared to the original $S_C$ and C images in Figure 6, the style effect in the $C_P$ images are increased gradually by increasing the $k_1$ value. On the contrary, the style effects in the $S_P$ images are then decreased gradually. As expected, both modules (3) and (4) achieve the design goals, this allows the user to flexibly choose his favorite composition of the C and $S_C$ images.

In order to make the image contrast more variable, the modules (5) and (6), extended from the modules (3) and (4), can perform contrast reduction or enhancement. First, set the value of $k_1$ by choosing a favorite type of mixing, and then adjust the $k_2$ value to control the required saturation. When $k_2<0$, the contrast is reduced, the smaller the $k_2$ is, the lower the contrast of the image, but when $k_2>0$, the effect is just the opposite, that is, the contrast is enhanced. The resultant images with different $k_2$ values are demonstrated in Figure 7. The first row is displayed the reduced-contrast $C_P$ images with a fixed $k_1=0.5$ and $k_2=-0.8,-0.5, \mathrm{and}-0.2$, from left to right in sequence. The second row shows the enhanced-contrast $C_P$ with the same $k_1$ value, but $k_2=0.2, 0.5, \mathrm{and} 0.8$, respectively. The third and fourth rows are the contrast-reduced and -enhanced $S_P$ images with the same $k_1$ and $k_2$ values as the first and second rows, respectively. As can be seen in Figure 7, the saturation of each image has reached the desired effect. This demonstrates that Equation (3) can execute the functions of replacement, mixing, and contrast adjustment to improve the neural stylized image.

Artists often need to render different colors for sketches or monotonous textures which is time-consuming and tedious. In the second experiment, we will show that the $S_P$ module (5) is able to provide the color re-rending effect. In addition, we used modified UST features (UST_5) and shown the corresponding stylized results in the third row of Figure 8. In this case, it clearly shows that the UST_5 features capture more complicated local structure. However, the $C_P$ images not only generate better style quality but also preserve content shape. Because the parameters are fixed, i.e., $k_1$= 0 and $k_2$ = 0.8, module (6) can be executed by a batch operation so that this re-coloring work will be efficient.

The third experiment will show the results of a real digital painting processed by the UST_5 and the proposed modules (5) and (6). Figure 9 shows the results of C, $S_C$, $C_P$, and $S_P$ images. The parameters are $k_1$ = 0.4, $k_2$ = 0.2 for module (5), and $k_1$ = 0.5, $k_2$ = −0.4 for module (6). As can be seen in Figure 9, the original C image is simple and elegant, showing a relaxed and pleasant style. In contrast, the $S_C$ image processed by the UST_5 has a larger contrast in color and a clearly visible texture, showing a cool style. By using modules (5) and (6), the $C_P$, and $S_P$ images are created by balancing the visual effects. They provide better style quality in terms of preserving content shape and keeping long style coherence.

In order to prove the validity of general landscape photos, we used the stylized image produced by UST_5,4 (described in Section 3.1) with Equation (10) in the following examples. As it can be seen in Figure 10a,e are two pairs of images including the C and S images. Figure 10b,f are the $S_C$ images produced by the UST_5,4 method. The two stylized images have extracted enough features from the original C and S images and the effects are very abstract. By using Equation (6), the corresponding $C_P$ and $S_P$ images (both of $k_1$ = 0.5, $k_2$ = 0) are shown in Figure 10c,d,g,h, respectively. Finally, those stylized images have a more visual-pleasing effect. Generally speaking, the quality of the artistic style image varies from person to person. For the stylized image, some people prefer more content-preserved images, and some others prefer more style-preserved images. Herein, our proposed method successfully combined the modified UST method and post-processing method to meet personal preference.

In addition, most NST methods allow us to recompose the content of general natural image in another style, but this is dissatisfied with the portrait images. Figure 11a is the original portrait image, and Figure 11b–d show the different styles of the original portrait image by using the different values of $k_1$ and $k_2$. In this case, we used the stylized image produced by UST_3,2,1 (UST_5,4 are unused). As we can see in Figure 11b–d, the three portrait images have outstanding performance in the color rendering, the style strokes, and the image contrast. For NST application, modified UST is a superior method. Based on VGG architecture, different applications should use different encoder/decoder pairs rather than using the original UST method.

5. Conclusions

More recently, NST-based style transfer algorithms have demonstrated very exciting results that have attracted academic and industrial attention. To obtain more visual effects, researchers have been using more complex algorithms to improve CNN’s learning or network architecture. However, the effect of these methods seems not satisfactory. Since the qualitative assessment is highly subjective, there is no metric to evaluate artistic style transfer effects. To give users more choices, a versatile post-processing framework has been proposed. In this framework, the blending technique, BT, and ICDDS methods have been fully integrated and implemented by a window-based program. It is easy to use and able to provide more visual effects for the stylized images, and also can integrate into the UST method effectively. Especially in the second experiment, we provide art workers a good solution for processing massive color rendering. When the deep learning technology is limited inherently, traditional methods such as this work may sometimes be able to solve the problem. Our proposed method is suitable for stylized images generated by any NST method, but it also works similarly to the MST method, which mixes two different stylized images.

This paper has investigated MST, and UST, and proposes a modified UST method to achieve more visual effects for the stylized images, which makes our method more applicable to real-world image generation scenarios where style needs to be defined by the user. However, the proposed method would fall short due to massive memory cost when processing ultra-high-resolution images. Therefore, how to avoid the memory problem caused by ultra-high-resolution images will be the subject of our future research work.