BatTS: a hybrid method for optimizing deep feedforward neural network

Bat algorithm

The bat algorithm (Yang, 2010) is a metaheuristic, swarm intelligence-based method. The algorithm is inspired by the echolocation behavior of microbats when it helps them find distances between themselves and their prey. In this algorithm, each bat has specific properties, i.e., frequency (f), velocity (v), position (x), loudness (A), and pulse rate (r). Initially, bats start randomly flying with a thunderous sound pulse and focus on the sound that bounced back from the object. The bounced-backed echo will tell about the distance, size, etc. The benefit of the bat algorithm is that it combines a swarm-based approach with a local search. The progressive iterations update each bat’s frequency, velocity, and position. In contrast, pulse and loudness adjusted only after accepting the new solution. The adjustment in frequency, velocity, and position can be made based on the following equations:

(1) $$f_i=f_{min}+\left(f_{max}-f_{min}\right)\beta$$

(2) $$v_i^t=v_i^{t-1}+\left(x_i^{t-1}-x_{gbest}^t\right)f_i$$

(3) $$x_i^t=x_i^{t-1}+v_i^t$$where $\beta \in \left[0,1\right]$ is a random number, $f_i$ represents the frequency of the i^th bat, velocity and positions of the i^th bat is denoted by $v_i$ and $x_i$ correspondingly. Also, $x_{gbest}^t$ is global best during iteration ‘t’.

Here a local search approach is applied to improve the variety of possible solutions based on certain conditions in the bat algorithm. Once the solution satisfies the condition, the random walk (Eq. (4)) strategy is used to populate a new solution:

(4) $$x_{new}=x_{old}+\epsilon A^t$$

In Eq. (4), $\epsilon$ represents a random number between the range of [−1, −1], and ‘A^t’ represents the mean loudness of the population in t^th iteration. As the bat moves toward its prey, typically, its loudness reduces, and its pulse rate rises. Loudness and pulse rate can be updated by using the following equations:

(5) $$A_i^{t+1}=\propto A_i^t$$

(6) $$r_i^{t+1}=r_i^0\left[1-\exp \left(-\gamma t\right)\right]$$

In Eqs. (5) and (6), $\propto$, and $\gamma$ are the constant where $\propto \in \left[0,1\right]$ and $\gamma >0$.

Tabu search algorithm

Tabu search is also a metaheuristic algorithm for finding the optimal global solution. The Tabu search method helps evaluate a group of solutions in each iteration, so it can reduce the cost of computation and can rapidly converge to an optimal solution. In Tabu search, if a solution is not improving, it can accept the lousy solution. This accepting bad solution property in Tabu search helps not get stuck in local minima/maxima. Tabu search algorithm, at each iteration, finds a local optimal s’ and then compares this s’ with s_global. If s’ < s_global, then s_global is updated by s’. To minimize repetition, the technique preserves a list (tabu list) of previously visited solutions.

Optimization methodology

This article integrates the Bat algorithm with the Tabu search and GDM algorithm (Fig. 1). This work aims to find the best solution $s_{Best}$ from the given set of solutions S, where f( $s_{Best}$) ≤ f(s’), f is the objective function ∀s’∈ S. The proposed methodology starts with Algorithm 1 having a population of size ‘popSize.’ Then, the population in the said algorithm is initialized randomly, keeping in mind that ‘Bat[p]’ (actually the position of bat ‘p’) represents the neural network architecture. Bat[p] is a single-dimensional list, where the index of the list means hidden layer and the element represents respective processing units. Bat[p] initially has one hidden layer with randomly initialized processing units. The proposed algorithm works for maximum ‘Iter’ iterations and uses the tabu search to generate the local best solution rather than random generation. If the new best f(Bat[p]) is better than the current best f(s’) and has high loudness, then accept the new best, update this to the current best and increase pulse and decrease the loudness. Finally, update the current s’ best with $s_{Best}$ if f(s’) < f( $s_{Best}$).

Algorithm 1 shows the proposed methodology with its pseudocode. For proper implementation, there is a particular condition that must be fulfilled first, like (a) representation of the solution, (b) fitness function, (c) generation of population, and (d) stopping conditions.

Representation of solution

In this article, FNN with various hidden layers is taken for evaluation. All the connections are in the forward direction with no loop. Each neural network combines an Input layer with ‘I’ processing units, various hidden layers where H_i represents hidden units at the i^th layer, and an output layer with ‘O’ processing units. Basically, ‘I’ and ‘O’ are problem-dependent; the main task is to estimate the ideal number of hidden layers with their corresponding neurons. Mathematically, the structure of a neural network is represented as:

(7) $$N_{net}\equiv \left(I\times H_1+B\times H_1\right)+\left(H_1\times H_2+B\times H_2\right)+\dots +\left(H_{max}\times O+B\times O\right)$$where B belongs to bias, in the proposed algorithm, every solution is represented by an array of four sections. The first section of the solution represents the number of hidden layers, H_L, the second section of the solution represents a list of hidden neurons layer-wise, H_N, the third section of the solution represents training error, T_tr, and the fourth section or the last section represents T_te testing error. Mathematically, the solution can be shown as:

(8) $$S\equiv \left(H_L,H_N,T_{tr},T_{te}\right)$$

(9) $$H_N\equiv \left(H_1,H_2,H_3,\dots ,H_{max}\right),H_i\in \mathbb{N}$$

$H_L\equiv \left(1,2,3,\dots ,max\right)$ and

(10) $$T_{tr},T_{te}\in \mathrm{\Re }$$where $\mathrm{\Re }$ represents the real numbers set and $\mathbb{N}$ represents the natural numbers set. In the representation, input and output units can be estimated by a given problem. In Contrast, at hidden levels, processing units are initialized by choosing randomly between the range of two thumb rules i.e., $\left[{\displaystyle \frac{\left(I+O\right)}{2},\left(I+O\right)\times {\displaystyle \frac{2}{3}}}\right]$. The weights are initialized using a uniform distribution [−1.0, +1.0].

Fitness function

The accuracy of a populated model can be evaluated by using a fitness function. Recording the model’s fitness in all iterations is mandatory in any evaluation. Finally, it will return the model that optimizes that objective function. For example, considering the data is of ‘E_N’ classes, let the true class of instance ‘d’ from training set ‘T’ can be defined as:

(11) $$\lambda \left(d\right)\in \left\{1,2,3,\dots ,E_N\right\}\mathrm{\forall }d\in T$$

In this article, the proposed methodology implemented the “winner takes all” way, meaning the number of classes C_N, and neurons at output layers are equal.

If O_P(d) is the output value of neuron ‘P’ at the output layer, for instance, ‘d’, then the class for the given sample ‘d’ can be:

(12) $$\phi \left(d\right)\equiv \arg ma{x}_{p\in \left\{1,2,3,\dots ,C_N\right\}}{O}_P\left(d\right)\mathrm{\forall }d\in T$$

The error in the network, for instance, ‘d’ can be defined as follows:

(13) $$\epsilon \left(d\right)\equiv \{\begin{array}{c}1,if\lambda \left(d\right)\ne \phi \left(d\right)\\ 0,if\lambda \left(d\right)=\phi \left(d\right)\end{array}$$

Hence, for the training set T, the classification error i.e., incorrectly classified instances, can be calculated in percentage and represented as:

(14) $$E\left(T\right)\equiv {\displaystyle \frac{100}{\mathrm{\#}T}\sum _{d\in T}\epsilon \left(d\right)}$$where #T is the number of instances in a given set T.

Generation of population

In the proposed methodology, the generation of the population can occur in four different cases:

Case 1: This is the case when methodology starts for a particular hidden layer. Other population parameters are randomly initialized except for bat position (Bat [p]). At the same time, the bat position (represents network architecture) is initialized by choosing a number between the range of $\left[{\displaystyle \frac{\left(I+O\right)}{2},\left(I+O\right)\times {\displaystyle \frac{2}{3}}}\right]$. (Bat [x, y] means solution having two hidden layers, and x, y is the respective neurons).

Case 2: After getting the initial solution, the population of the bat is used to generate new solutions by using Eqs. (1)–(3).

Case 3: If the newly generated bat is not in the domain (the number of hidden neurons for any layer going outside the range), then use Eq. (4) to generate a new bat.

Case 4: When the Pulse of any bat is lesser, use tabu search to generate a new bat as shown in Algorithm 2.

For better exploration, tabu search divides the list of generated neighbors (Algorithm 3) into two parts. The first part increases the hidden neurons by ‘K,’ and the second part decrease the hidden neurons by ‘K,’ see Eqs. (15) and (16):

(15) $$H_N=\{\begin{array}{l}K+,r\ge p\\ nochange,r<p\end{array}$$and,

(16) $$H_N=\{\begin{array}{l}K-,r\ge p\\ nochange,r<p\end{array}$$

Here, ‘r’ is the uniformly distributed [0, 1] random number, and ‘p’ is the probability of change in neurons.

Stopping criteria

In this work, the optimizations stop if it reaches maximum iteration. Furthermore, there are some other conditions also when the algorithm will stop updating the hidden neurons and switch to successive steps: (i) when it tries to add hidden neurons using Eq. (15) while the network reaches its maximum limit, (ii) when it tries to subtract hidden neurons using Eq. (16) while the network reaches to its lower limit.

Experimental result

As discussed in previous sections, choosing the correct architecture for a neural network is crucial and time-consuming. To estimate the configuration, hit and trial experiments are in trend, but this will not guarantee ideal architecture. Therefore, this section includes the findings in random experiments and BatTS experiments.

All experiments are implemented in R using the H₂O package. The implemented methodology uses multinomial distribution, 0.2 dropout ratio, 0.7 momentum term, and 0.0001 learning rate. Each model is validated by using 20% of each dataset as a validation dataset.

Random experiments

Generally, in random trial-based experiments, the user repeatedly selects a neural network model by changing its parameters manually and running these models on a given dataset. Finally, the best one based on minimum testing/training error is chosen. Here is a random experiment for every dataset, we run 30 different topologies for a single hidden layer, then 30 different topologies for two hidden layers, and so on up to maxHid = 5. We fixed the maximum hidden layer (maxHid) = 5 which can be increased on the basis of problem complexity. Neurons for hidden layers were selected randomly by using a thumb rule [(I+O)/2, (I+O) × 2/3]. Where ‘I’ is the Input features and ‘O’ is the output classes. At the end, we choose the best architecture on the basis of minimum testing error. Every network selected for evaluation was fully connected and trained by using GDM. The results are shown in Table 2.

For the face classification, the optimal architecture returns by random experiments having two hidden layers with {499, 262} hidden neurons. The topology gives a 12.561% mean square error (MSE). The best architecture for the gas drift dataset received two hidden layers holding {83, 52} units and MSE as 6.473%. In the case of the MNIST data, the MSE of 1.839% was calculated with single hidden layer architecture and 521 hidden units. The best performance for the ISOLET dataset measured MSE as 2.266%. The topology for ISOLET using random experiments received with two hidden layers having {403, 211} neurons for respective hidden layers.

BatTS experiments

The BatTS starts with one hidden layer network and randomly chosen hidden units. After creating this as the initial topology, the fitness function is calculated with the help of the H₂O package and recorded as the best solution. Every solution here in this BatTS is iterated by Iter = 10, and then in every iteration, every solution will create a population of size, popSize = 20. If the generated solution is not in the domain (i.e., hidden units are going outside the lower and upper range), then a random walk is used to create the new solution and adjust this in the domain for any bat having a Pulse less than a certain threshold, Tabu search (Algorithm 2) help to generate a new solution and return its fitness value. A bat is only accepted if fitness is optimal and loudness is high. Once the solution is accepted, the Pulse is increased, and Loudness, Loud is decreased. The BatTS runs for maxHid = 5. The probability of changing neurons that is, one can increase or decrease (in Algorithm 3), p = 0.5. The size of the architecture is increased or decreased by K% = 3.

The experimental statistics are presented in Table 2. In the case of the face dataset, BatTS proposes architecture with H_L= 2 and the respective neurons are {423,259}. The mean square error (MSE in percentage) was 9.281%. For Gas-Drift, the optimal architecture suggested by the proposed algorithm was H_L = 2 with {83,56} hidden processing units, and the MSE was 4.208%. In the case of MNIST, MSE return by the algorithm was 1.6501, while the optimal topology has H_L= 1 with {522} hidden units. For the ISOLET, the optimal configuration required Hidden layers, H_L = 2 and H_N = {355, 198}, and the MSE noted as 1.532%.

The fitness function is evaluated based on testing error rather than training error. Table 2 shows that BatTS performed better than Random experiments and the Tabu-based approach (Gupta & Raza, 2020). Figs. 2 and 3 present the performance comparison of the proposed BatTS method with the Tabu search based method and random experiments on the four benchmark datasets which clearly shows a significant reduction in the classification error.