Nonlinear Activation Functions in CNN Based on Fluid Dynamics and Its Applications

The nonlinear activation functions in the deep CNN (Convolutional Neural Network) based on fluid dynamics are presented. We propose two types of activation functions by applying the so-called parametric softsign to the negative region. We use significantly the well-known TensorFlow as the deep learning framework. The CNN architecture consists of three convolutional layers with the max-pooling and one fullyconnected softmax layer. The CNN approaches are applied to three benchmark datasets, namely, MNIST, CIFAR-10, and CIFAR-100. Numerical results demonstrate the workability and the validity of the present approach through comparison with other numerical performances.

neural networks. The activation function of ReLU has been widely used by many researchers for visual recognition tasks [Glorot, Bordes and Bengio (2011); Krizhevsky, Sutskever and Hinton (2012); Srivastava, Hinton, Krizhevsky et al. (2014); LeCun, Bengio and Hinton (2015); Kuo (2016); Agarap (2018)]. The ReLU activation function leads to better recognition performances than conventional sigmoid/tanh units involving the vanishing gradient problem, while has parameter-free, and zero-gradients in the negative part. In order to provide meaningful such negative values, there have been presented some activation functions, such as leaky rectified linear unit (LReLU) [Maas, Hannun and Ng (2013)], parametric rectified linear unit (PReLU) [He, Zhang, Ren et al. (2015)], exponential linear unit (ELU) [Clevert, Unterthiner and Hochreiter (2016)], and so forth. The LReLU has been slightly improved for ReLU by replacing the negative part of the ReLU with a linear function involving small constant gradient. The PReLU has been generalized by adaptively learning the parameters introduced in the negative part of LReLU. They have improved significantly the learning performances on large image datasets called ImageNet. Clevert et al. [Clevert, Unterthiner and Hochreiter (2016)] have proposed an activation function, ELU, and shown applicability and validity for various benchmark datasets. As another approach, Goodfellow et al. [Goodfellow, Warde-Farley, Mirza et al. (2013)] have also proposed an activation function called maxout that has features both for optimization and model averaging with dropout [Hinton, Srivastava, Krizhevsky et al. (2012)]. In our previous work, we have presented newly the characteristic function (i.e., activation function) as an optimum function which is derived on the advection-diffusion system in fluid dynamics framework [Kakuda (2002)]. The purpose of this paper is to propose the activation functions based on the concept of fluid dynamics framework. We present two types of activation functions by applying the so-called parametric softsign [Glorot and Bengio (2010)] to the negative part of ReLU. By using the well-known TensorFlow [Abadi, Agarwal, Barham et al. (2015)] as the deep learning framework, we utilize the CNN architecture that consists of three convolutional layers with the max-pooling and one fully-connected softmax layer. The workability and the validity of the present approach are demonstrated on three benchmark datasets, namely, MNIST [LeCun, Bottou, Bengio et al. (1998)], CIFAR-10 and CIFAR-100 [Krizhevsky and Hinton (2009)], through comparison with other numerical performances.

Construction of nonlinear activation functions 2.1 Neural network model
In the field of neural networks, the input-output (I/O) relationship known as the backpropagation is represented by inputs , output and the characteristic function h (i.e., activation function) as follows: where are j-th input values as shown in Fig. 1, are the connection weights, is the bias value, and denotes threshold.
The sigmoid function (see Fig. 2(a)) has been mainly used as the following continuous function.
where k is ad hoc parameters.

Nonlinear activation functions based on fluid dynamics
Heretofore, we have presented the following activation function as an optimum function which is derived on the steady advection-diffusion system in fluid dynamics framework [Kakuda (2002)] (see Eq. (25) in next Subsection 2.3).
Therefore, we obtain the functions by substituting Eq. (6) into Eq. (4) and considering the sign of v as follows (see Fig. 2 In this stage, we adjust the functions, h(v), so that (0) = 0.
As a result, we obtain the following form by taking into account that ̂= 2 .
In order to avoid zero-gradients in the negative part of v, by applying Eq. (9) to the negative region, we propose two types of activation function involving parameter, a, as follows (see Fig. 3): (a) Sigmoid functions with k (b) Characteristic functions of Eq. (7)

Figure 2: Characteristic functions
Rational-type activation function and its derivatives Exponential-type activation function and its derivatives The corresponding derivatives of the activation functions are also shown in Fig. 4.
(a) Rational-type activation function (b) Exponential-type activation function

Steady advection-diffusion equation 2.3.1 Problem statement
Let us briefly consider the one-dimensional advection-diffusion equation in spatial coordinate, x, given by with the adequate boundary conditions, where = φ, u and k are the given velocity and diffusivity, respectively.

Finite element formulation
In order to solve the flux, = φ, in a stable manner, we shall adopt the Petrov-Galerkin finite element formulation using exponential weighting function [Kakuda and Tosaka (1992)]. On the other hand, the conventional Galerkin finite element formulation can be applied to solve numerically Eq. (14).
First of all, we start with the following weighted integral expression in a subdomain Ω = [ −1 , ] with respect to weighting function ̃: The weighting function ̃ can be chosen as a general solution which satisfies where Δ = − −1 , and σ( ) denotes some function described by Yee et al. [Yee, Warming and Harten (1985)], which is sometimes referred to as the coefficient of numerical viscosity. The solution of Eq. (16) is as follows: ̃= −̂ (17) where A is a constant, and ̂= Δ ( ) ⁄ . By applying the piecewise linear function to the flux f and φ, we obtain the following integral form Here, applying an element-wise mass lumping to the first term of the left-hand side of Eq. (18), and carrying out exactly those integrals in Eq. (18), we can obtain the following numerical fluxes −1/2 and +1/2 in the subdomains Ω and Ω +1 , respectively where = 2 ( ) ⁄ , and ( ) denotes the signum function.
Let us next derive the Galerkin finite element model for Eq. (14). The weighted residual equation in Ω is given as follows: In this stage, we assume a uniform mesh Δ = Δ for simplicity of the formulation. Taking into consideration the continuity of , at nodal point i, we can obtain the following discrete form where for any velocity u Using the element Peclet number (≡ ∆ /2 ) as , we reduce Eq. (24) to the following form { ( ) − ℎ| |} +1 + 2 ℎ| | − { ( ) + ℎ| |} −1 = 0 (26) This equation has the same structure as the SUPG scheme developed by Brooks et al. [Brooks and Hughes (1982)], and it leads to nodally exact solutions for all values of [Christie, Griffiths, Mitchell et al. (1976)].

CNN architecture
We adopt the similar approach as the PReLU [He, Zhang, Ren et al. (2015)] which can be trained using back-propagation and optimized simultaneously with other layers. As the variables, v and a, in Eq. (10) through Eq. (13), we define and with respect to the input and the coefficient, respectively, on the j-th channel. The momentum approach when updating is given as follows: where E represents the objective function, is the momentum to accelerate learning, is the learning rate. The parameters, , are optimally obtained by using back-propagation analysis. Fig. 5 shows the CNN architecture consisting of three convolutional (i.e., conv) layers with some max-pooling and one fully-connected (i.e., fc) softmax layer.

Numerical experiments
In this section, we use the well-known TensorFlow [Abadi, Agarwal, Barham et al. (2015)] as the deep learning framework, and present numerical performances obtained from applications of the above-mentioned CNN approach to three typical datasets, namely, MNIST [LeCun, Bottou, Bengio et al. (1998)], CIFAR-10 and CIFAR-100 [Krizhevsky and Hinton (2009)]. We utilize the Adam [Kingma and Ba (2015)] as the learning algorithm for the stochastic gradient-based optimization.
The model is previously trained for some epochs on mini-batches of size 100 with the learning rate, = 10 −3 , and the momentum, = 0. The specification of CPU and GPU using CUDA is summarized in Tab. 1.

MNIST
Let us first consider the MNIST dataset which consists of 28×28 pixel gray-scale handwritten digit images with 50,000 for training and 10,000 for testing images. Fig. 6 shows the behaviors of training accuracy and loss (i.e., cross-entropy) obtained by using various activation functions for the MNIST. The corresponding validation accuracy and loss behaviors are shown in Fig. 7. We can see from Fig. 6 and Fig. 7 that our approaches are similar to the ones using other activation functions. Tab. 2 summarizes the transitions of the learned parameter, a, at each layer of CNN architecture (see Fig. 5) for the MNIST. The validation accuracy rate and loss for the MNIST are given in Tab. 3. In this case, the quantitative agreement between our results and other ones appears also satisfactory.

CIFAR-10
As the second benchmark dataset, we consider the CIFAR-10 which consists of 32×32 color images drawn from 10 classes with 50,000 for training and 10,000 for testing images. Fig. 8 shows the behaviors of training accuracy and loss (i.e., cross-entropy) obtained by using various activation functions for the CIFAR-10. The corresponding validation accuracy and loss behaviors are shown in Fig. 9. We can see from Fig. 8 and Fig. 9 that our approaches outperform entirely the ones using other activation functions. Tab. 4 summarizes the transitions of the learned parameter, a, at each layer of CNN architecture (see Fig. 5) for the CIFAR-10. The validation accuracy rate and loss for the CIFAR-10 are given in Tab. 5. For the accuracy rate on the CIFAR-10, we obtain best result of 80.76% using the exponential-type activation function. On the other hand, our approaches for the loss outperform other ones.

CIFAR-100
As the third benchmark dataset, the CIFAR-100 is the same size and format as the CIFAR-10 one, while contains 100 classes for consisting of 20 super-classes with five classes each.     Fig. 11. We can see from Fig. 10 and Fig. 11 that our approaches outperform the ones using other activation functions. Tab. 6 summarizes the transitions of the learned parameter, a, at each layer of CNN architecture for the CIFAR-100. The validation accuracy rate and loss for the CIFAR-100 are given in Tab. 7. For the accuracy rate on the CIFAR-100, we obtain best result of 56.91% using the rational-type activation function. On the other hand, our approaches for the loss outperform also other ones.

Conclusions
We have proposed new activation functions which were based on the steady advectiondiffusion system in fluid dynamics framework. In our formulation, two types of activation functions have been reasonably presented by applying the so-called parametric softsign to the negative part of ReLU. By using the TensorFlow as the deep learning framework, we have utilized the CNN architecture that consists of three convolutional layers with some max-pooling and one fully-connected softmax layer. The performances of our approaches were carried out on three benchmark datasets, namely, MNIST, CIFAR-10 and CIFAR-100, through comparison with the ones using other activation functions. The learning performances demonstrated that our approaches were capable of recognizing somewhat accurately and in less loss (i.e., cross-entropy) the object images in comparison with other ones.