An improved LogNNet classifier for IoT application

In the age of neural networks and Internet of Things (IoT), the search for new neural network architectures capable of operating on devices with limited computing power and small memory size is becoming an urgent agenda. Designing suitable algorithms for IoT applications is an important task. The paper proposes a feed forward LogNNet neural network, which uses a semi-linear Henon type discrete chaotic map to classify MNIST-10 dataset. The model is composed of reservoir part and trainable classifier. The aim of the reservoir part is transforming the inputs to maximize the classification accuracy using a special matrix filing method and a time series generated by the chaotic map. The parameters of the chaotic map are optimized using particle swarm optimization with random immigrants. As a result, the proposed LogNNet/Henon classifier has higher accuracy and the same RAM usage, compared to the original version of LogNNet, and offers promising opportunities for implementation in IoT devices. In addition, a direct relation between the value of entropy and accuracy of the classification is demonstrated.


Introduction
The fast development of smartphones, tablet PCs and wearable devices lead to an exponential increase in the volume of transferred images through networks. Image classification is an important tool for extracting information from digital images. This method is widely used in science and engineering. Murugappan proposed the method for facial emotional recognition [1]. Zhang et al introduced the synergic deep learning to classify medical diagnosis images [2]. Peng et al conducted the security assessment of deep learning for image classification [3].
The fourth industrial revolution (Industry 4.0) is the automation and data exchange in the official and industrial processes to improve various aspects of our lives. Industry 4.0 enables organizations and industries to digitize their systems, improve their performance and help its customers make better decisions regarding waste management and environmental sustainability. The internet of things is one of the building blocks of Industry 4.0. IoT focuses on the leveraging interconnection between the physical objects and users, which collect and share data through the internet. In recent years, IoT received wide attention from the scientists and manufactures due to its wide applications in industries. For example, IoT is used in health care, fitness, education, entertainment, social life, energy conservation, environment monitoring, home automation and transport systems [4]. In 2020, the global IoT industry market was 119 Billion dollars. The annual Compound Annual Growth Rate of 20% is predicted for IoT market [5].
In recent years, the IoT image classification attracts a great attention due to the huge number of IoT users and wide applications of image classification. However, the small memory size and high computational efforts are two important problems in the designing of IoT devices. Sun [8]. Bhardwaj introduced network of neural networks to overcome the low-memory bottleneck [9]. Penkovsky et al used the hybrid CMOS -hafnium oxide resistive memory technology to reduce the memory requirement [10]. Velichko designed LogNNet model to reduce the total used memory in an IoT device. LogNNet is feed forward neural network with the reservoir property which uses chaotic logistic map to classify MNIST-10 dataset [11]. This paper proposes modifications to Velichko study [11] to improve the accuracy, while the required memory is approximately unchanged. The rest of the paper is organized as follows. In Section 2, the preliminaries are given. The proposed method is discussed in Section 3. The numerical results are given in Section 4. Section 5 is devoted to the conclusions and future studies.

The concept of RAM saving on the LogNNet classifier
A new neural network architecture, named LogNNet, was proposed in [11], which allows efficient use of small amounts of RAM and opens up a new potential for designing low-power IoT devices. LogNNet is a feed forward neural network, where signals are directed only from input to output. The main feature of this network is using deterministic chaotic filters, which mix the input information in incoming signals. The network uses the discrete logistic map to generate a chaotic sequence and mixed input to extract initially invisible information.
The model saves RAM because the array of weights is not stored in the chaotic filter, and only three parameters are required to calculate the weights. LogNNet was tested on the MNIST-10 database and demonstrated the recognition accuracy of about 96.3%. The developed architecture uses no more than 29 kB of RAM. In addition, LogNNet was simulated on devices with very small RAM and it took only 1-2 kB of RAM. The results showed good accuracy of 80.3%. A miniature controller Atmega328, which fits on a fingertip and can be used in "smart" devices of small dimensions, has about the same amount of memory. Therefore, the developed network with a digital reservoir is a promising network architecture in comparison with the known algorithms. This paper proposes a feed forward LogNNet neural network, which uses a semi-linear Henon type discrete chaotic map to classify MNIST-10 dataset. The parameters of the chaotic map are optimized using particle swarm optimization (PSO) with random immigrants. The proposed LogNNet/Henon classifier has higher accuracy and same RAM usage, compared to the original version of LogNNet, and has broad opportunities for implementation in IoT devices.

The concept of chaos on the LogNNet classifier
Chaotic systems are deterministic nonlinear dynamical systems, which behavior is sensitive to the initial conditions. A little change in the initial conditions may lead to big difference in their trajectories. In recent years, chaotic systems attracted wide attention from theoretical and practical point of view. Fan et al studied long-term prediction of chaotic systems with machine learning [12]. Velichko applied chaotic logistic map for handwriting digits classification [11]. Luo et al introduced a four dimensional chaotic system and used it for image encryption [13].
The degree of chaos in the reservoir part of LogNNet affects the efficiency of the model [11]. The classification accuracy depends on the parameter of logistic maps. The form of the dependence correlates with the dependence of the Lyapunov exponent λ on the parameter r, and indicates that the chaotic dynamics of the logistic map plays an important role in the recognition process by the neural network.
Lyapunov exponent is a parameter for investigating the stability of dynamical system. Entropy is another important concept that measures the rate of the complexity in dynamical system [14]. In this study, we show that entropy is a better indicator correlating with classification accuracy of LogNNet network.

Modification of LogNNet classifier with Henon type chaotic map
Researchers believe that designing a neural network is an art. This paper presents an optimized feedforward LogNNet neural network. The model consists two parts: reservoir part and trainable classifier. A schematic representation of the LogNNet architecture is shown in Ошибка! Источник ссылки не найден.. An image of a handwritten digit from the MNIST-10 database is the network input. Based on [11], the pattern T3 is selected to convert the 28×28 pixel image into a one-dimensional array The first part is the reservoir of the model. It transfers the input array Y into a more suitable vector Sh which is used in the training process of the neural network. The function of the reservoir part, denoted by fh , is following where W1 is the given matrix of input weights. The dimension of the matrix W1 is P×785, where P is the number of neurons in the reservoir.
The reservoir part has two aims. The first aim is to transform the vector Y into vector Sh with the dimension P in a way that the best classification accuracy is achieved. The second aim is to reduce the required RAM, which is critical in IoT applications [11].
The training process and the final output are done in the second part of the model, denoted by Sout, as follows 2 ( ).
The output layer Sout contains N = 10 neurons and determines the result of image classification from the MNIST-10 database. The W2 matrix is trained by the back propagation method. If the output classifier contains one layer of neurons, then the layer architecture of the neural network is denoted as LogNNet 784:P:N. If the output classifier contains two layers with an additional hidden layer with the number of neurons H, then the architecture of the neural network is denoted as LogNNet 784:P:H:N. In the latter case, an additional matrix of weights (W3) is required.
We construct the weight matrix W1 using the following discrete chaotic map [15]

Particle swarm optimization
Particle swarm optimization is a metaheuristics optimization technique, proposed by Kennedy and Eberhart [16]. This algorithm is inspired by swarm behavior, such as bird flocking and schooling fish in food searching. The swarm members are sharing information to achieve the best result. From the mathematical point of view, the algorithm is simple and easy to implement. Therefore, it is suitable to use in the IoT devices with memory limitations. In PSO, each particle i is characterized by its location xi and the velocity vi. To improve the current solution, the location and velocity of the particles are changed in each iteration in the following way where xBest and gBest denote the best position of particle i and the global best position of all particles until the current iteration, respectively. The parameters r1 and r2 are random numbers between 0 and 1.
The parameter  is inertia weight and the parameters c1 and c2 are individual-cognition parameters, which determine the effects of the particle's own previous experiences on the global optimal solution [16,17]. These parameters play an important role in the convergence of PSO algorithm. Our experiments demonstrate that c1= c2=2 and  =0.5 are suitable values for the proposed problem. Recently, Ünal and Kayakutlu used random immigration method to improve the accuracy of PSO in multi-objective optimization problem [17]. In this paper, we use a similar method, where 70% of particles are chosen randomly in each iteration. This method is denoted as RPSO.

Optimization of parameters
Training, validation and test are the main processes in every machine-learning algorithm.  The aim of validation process is fine-tuning the performance of the training process in neural network. The test process determines the accuracy and margin of error of the neural network. This paper uses a two-stage algorithm to construct LogNNet model. In the first stage, the PSO method is applied to find The matrix W1 is constructed in the reservoir part to transfer the input of the model in a way that the accuracy is maximized. The elements of the matrix W1 depend on the values of {ai, i=1,…,4} and the initial conditions (x0, y0) in the Henon type map (Eq. (3)). We use RPSO method for determining the best values of these parameters to maximize the accuracy of the model. To speed up the training process during optimizing the parameters in the reservoir part (the first stage), twenty percent of the training dataset is considered (12000 objects) (see Ошибка! Источник ссылки не найден.). All elements of the training dataset (60000 objects) are used in validation process. After optimization, a re-training process was carried out on a full base (60,000 objects), followed by final testing using a test base of 10,000 objects (see figure 3). As a result, the classification accuracy was optimal for the given configuration of the LogNNet / Henon network.

Classification and matrix filling methods
Based on Section 3, the initial conditions (x0, y0) are needed to construct the weight matrix W1. We consider two forms of the initial conditions. The first form is initial condition of the sine function as in [11]. The first row of the matrix W1 is constructed in the following way where A and B are given constant numbers. The next rows are constructed by using the previous rows as the input of Eq. (3). The second form is the constant initial conditions (x0, y0)= (A, B), and the matrix W1 is filled in a snake manner (W1[p,i]=yn), as depicted in figure 4(a). Its pseudo code is presented in figure 4(b).
To avoid the influence of transient period at constant initial conditions [18], the preliminary 10000 iterations are required.    represent xn and yn, xp1 and yp1 represent xn+1 and yn+1 in (3)).
In addition, the following upper limit condition for yn is applied to investigate its effect on the accuracy of the classification y 10 then y 1. The hypothesis (7) is required, as the time series may tend to an unstable mode of abnormally large numbers sequence, generated by certain values of the parameters (A, B, a1, a2, a3, a4).
The methods numbered from 1 to 6 are explained in Table 1.   figure 5 shows the bifurcation diagram of the Henon type map and the accuracy of LogNNet (method 6) for different values of a1. The variation of a1 affects the classification accuracy over a wide range, while the other parameters are fixed. To find the parameters with the highest classification accuracy, we used the RPSO method.
A detailed comparison between LogNNet and other classification methods is done in [11], and here, we compare the proposed method with LogNNet. The activation function and the layers architecture of the feedforward neural model of the proposed method are selected the same as in [11]. The results for the different proposed models are given in the Table . RPSO runs with 150 particles in 100 iterations, and the lower and upper bounds for the variables [a1,a2,a3,a4] are set [0.01,0.1,0,0,0,0] and [1.5,10,1.5,1.5,1.5,1.5], respectively. The neural network model is trained with the maximum 20 epochs in optimization mode (see. figure 2).
The differences between the accuracy of the proposed methods and [11] are given in table 3. The proposed methods have significantly better accuracy than [11]. From a practical point of view, we suggest to use LogNNet/Henon Method 4 or LogNNet/Henon Method 6, as they have better performance in all considered layer structures.

Relation between the entropy and accuracy of the classification
Another interesting result is the direct relation between the approximate entropy (ApEn) [19]  Let us consider the points p1, p2, p3, p4, p5, which are depicted in this figure. A comparison between the subfigures of approximate entropy shows that the subfigure with r=0.025 and m=2 match with the trend of classification accuracy, except the point p5, where the maximum accuracy is achieved. The reason is that ApEn is sensitive to the input parameters (m and r) and is accurate in measuring the complexity of time series [20]. The Poincare plots for the points p3, p4 and p5 are depicted in figure 7. The complexity of the time series is highest in the point p5, where the classification accuracy is highest too. Therefore, applying the LogNNet method on MNIST-10 dataset is as an alternative way to measure the regularity of the time series, and it provides more reliable results than the ApEn calculation method.

Conclusion and future studies
The paper proposes a new structure for the reservoir part of feedforward neural network LogNNet. Results demonstrate that the accuracy of the model depends on the initial conditions and the chosen discrete chaotic map used to construct the weight matrix W1. A Henon type chaotic map is examined, and RPSO method is used for determining the best parameters of the map to maximize the accuracy of the classification. The memory requirements for IoT devices remain the same. The proposed method uses the discrete chaotic map (3), which has four parameters and two parameters for initial conditions. The reference method [11] used the discrete chaotic logistic map, which required one parameter and two parameters for initial condition. The difference of required memory between the proposed method and [11] is the allocated memory for three real variables that is a minor difference. The accuracy of IoT devices can be improved by modifying the chaotic map and PSO method, while the training process and hardware used are unchanged. This method modification would help manufacturers to improve their products without additional hardware costs.
The results show that the accuracy is significantly improved in the layer structure 784:25:10. The layer structure 784:25:10 is the simplest structure, which requires small memory and is easy to implement in various IoT devices. The simple structure and low memory usage make the proposed LogNNet model suitable for implementation in edge computing in IoT devices, which will be a hot topic  [21]. The relation between the approximate entropy and the accuracy of the classification is investigated. Therefore, images classification by LogNNet can be considered as an alternative way to measure the regularity of the time series.
For the future studies, the different discrete chaotic maps with higher entropy can be examined to improve the accuracy. Applying the proposed method to different datasets for generalizing the results is another topic for the further research.