Multi-objective simulated annealing for hyper-parameter optimization in convolutional neural networks

CNNs—an overview

Neural Networks (NNs) receive an input and transform it through a series of hidden layers each of which is made up of a set of neurons that receives some inputs, performs a dot product followed with a non-linearity. In any layer, each neuron is fully connected to all neurons in the previous layer, that is why Regular NNs don’t scale well to inputs in the form of images. Similarly, CNNs are made up of neurons that have learnable weights and biases, and the whole network still expresses a single differentiable score function. In general, CNNs assume that the inputs are images, and they constrain the architecture in a more sensible way. There are three types of layers in CNNs: Convolution Layer, Pooling Layer and Fully Connected Layer.

A convolution layer is a fundamental component of a CNN architecture that performs feature extraction through a combination of linear and nonlinear operations, that is, convolution operation and activation function. Convolution is specialized type of linear operation used for feature extraction with the help of a small array of numbers called filter or kernel navigating over the input tensor. At each location of the input tensor, an element-wise product between the elements of the filter and the input is applied. These products are then summed to obtain the output value in the corresponding position of the output tensor which is called a feature map. This operation is repeated using multiple kernels resulting in multiple number of feature maps at a single layer. Figure 1 illustrates a CNN with two convolution layers each of which contains different number of filters with differing size. In the figure, images of size 96 × 96 × 3 are fed into the CNN where the first two dimensions denote the width and height of the image, and the third dimension denotes the number of channels which is 3 for a color image. As can be seen in Fig. 2 which illustrates the convolution process, the number of channels (depth) of the filters always equals to the number of channels of the input image, but the size and the number of those filters are among the hyper-parameters whose values should be selected carefully. The convolution process can be speed up by using dimension reduction adjusting the stride value. If the stride value takes a value other than 1, then the filter skips the input by this amount. If input size reduction is not desired, padding is used. In the pooling layer, size reduction is done in the same way as in the convolution layer with one difference is that the number of channels remains unchanged. There are two main types of pooling methods: max and average pooling. Pooling type, the size and the stride of the filter are among the hyper-parameters in a given network. In the example CNN in Fig. 1, the first convolution layer includes 32 filters each with a size of 5 × 5 × 5, and after the first convolution operation, linear structure is transformed into a non-linear structure using ReLU activation function. After the pooling layer, input weight and height is reduced to 22 × 22, but the depth is increased to 32. After the second convolution and pooling operations, input width and height is reduced even more. Strive method is proposed as an alternative to the pooling method (Springenberg et al., 2014). A strive layer is a kind of convolution layer with 3 × 3 or 2 × 2 filter sizes with a stride of 2. Filters in this layer do not have weights to learn, because, only size reduction is applied. After the convolution and pooling operations, a flattening operation is applied to transform all the feature maps in the final layer into a one-dimensional vector which is then fed to the fully connected layer where classification results are obtained. There might be multiple fully connected layers in a CNN; however, the output size of the last fully connected layer should match the number of classes in the problem. For a given input, a classification score is computed for each class and the input is assigned to the class with the highest score. Based on the prediction accuracy, an error is computed which is then used to update the weights in the network. This process is repeated until a desired error rate is achieved. For a detailed review on the topic please refer to (Goodfellow, Bengio & Courville, 2016).

Hyper-parameters like the total number of layers, the number and size of the filters at each layer along with filter related parameters like stride and padding define a CNN configuration or an architecture. On the other hand, total number of weights in all of the layers of a CNN defines the size or the number of parameters of that network. The size of a CNN is calculated differently for convolution and fully connected layers. The number of parameters in a convolution layer equals to $\left(\left(d\times m\times n\right)+1\right)\times k$ , where d is the number of input feature maps, and m and n are the filter width and height, respectively (1 is added because of the bias term for each filter), and k is the number of output feature maps. The number of parameters in a fully connected equals to $\left(c+1\right)\times p$, where c is the number of input nodes and p is the number of output nodes.

SA algorithm

Simulated Annealing (SA) uses an adaptation of the Metropolis algorithm to accept non-improving moves based on a probability (Kirkpatrick, Gelatt & Vecchi, 1983). In a single-objective SA, a new solution, X′, is selected within the neighborhood of current solution X, where the neighborhood of X is defined as all the solutions that can be reached by a single move from X. The moves that can be performed in a SA are defined based on the representation used to encode a feasible solution. Solution representation, on the other hand, is highly dependent on the problem domain. If the objective function value of X′ is smaller than X (for a minimization problem), then X′ is accepted. If X′ is worse than X, then it is accepted with a probability, $p_{acc}$,which is calculated based on the worsening amount and the current temperature of the system as, $p_{acc}=min\left\{1,exp\left(-\mathrm{\Delta }F/T_{cur}\right)\right\}$ where $\mathrm{\Delta }F$ is the worsening amount in the objective function and $T_{cur}$ is the current temperature. If the temperature is high, then the probability of accepting a worsening move would be higher than the probability with a lower temperature. In general, the system is initiated with a high temperature to allow exploration during initial steps. As shown in the SA pseudo-code below, after a certain number of solutions are visited at the current temperature level which is defined by the parameter $nbr\mathrm{\_}inner\mathrm{\_}iter$, the temperature is lowered gradually according to a predefined annealing scheme. Geometric cooling that uses a decay factor smaller than 1 is the most widely used cooling scheme in SA. At each outer iteration, $T_{cur}$ is reduced to ensure the convergence of the algorithm, because the probability of accepting worsening moves drops as $T_{cur}$ reduces. The total number of solutions visited during a run of a SA is defined as $nbr\mathrm{\_}out\mathrm{\_}iter\times nbr\mathrm{\_}inner\mathrm{\_}iter$. However, the number of outer iterations is not always fixed before the start of the algorithm. A minimum temperature level, or a minimum objective function value can be defined as the stopping criterion. Initial temperature, $T_{init}$, cooling rate (if geometric cooling is employed) and the final temperature, $T_{final}$ are important parameters that should be selected carefully.

In this study, the SA algorithm considers only one objective which is the error rate, and it accepts a new solution X′ only if it is better than current solution X with respect to this single objective value. However, if X and X′ have the same error rate, then the SA selects the one with the smaller number of FLOPs. The composite move used to generate X′ is defined in “Local Moves”. In order to define $T_{init}$, we use a real time initial temperature selection strategy (Smith, Everson & Fieldsend, 2004). In this strategy, $p_{acc}$ is not calculated using the formula $p_{acc}=min\left\{1,exp\left(-\mathrm{\Delta }F/T_{cur}\right)\right\}$. Instead, a fixed initial probability value which is recommended as 0.5 is defined for accepting the worsening moves. Then, $T_{init}$ is calculated as $-\left(\mathrm{\Delta }{F}_{ave}/\left(ln\left(p_{acc}\right)\right)\right)$, where $\mathrm{\Delta }{F}_{ave}$ is the average worsening penalty amount which is calculated executing a short “burn-in” period. A similar real time temperature adjustment approach is also used to define $T_{final}$. In this study, the total iteration budget defines the SA stopping criterion, and the number of inner and outer iterations are defined according to this iteration budget and the cooling scheme.

MOSA algorithm

There are many types of MOSA algorithms in the literature, and basically, there are two main differences between multi-objective and single-objective SA algorithms. The first difference is the design of the acceptance rule, and second one is the maintenance of an external archive of non-dominated solutions which will eventually yield an approximation of the Pareto front. Let $z_k\left(X\right)$ for $k\in \left\{1,2,..,K\right\}$ be the value of the $k^{th}$objective function of a solution $X$. If a new solution $X^{\mathrm{\prime }}$ yields in objective function values that are superior to $X$ in terms of all of the objectives; that is, $\mathrm{\Delta }{z}_k=z_k\left(X^{\mathrm{\prime }}\right)-z_k\left(X\right)$ and $\mathrm{\Delta }{z}_k\le 0,k\in \left\{1,2\right\}$ assuming the two objectives are to be minimized, then it is always accepted. Otherwise, probabilistic acceptance rules as discussed in “SA Algorithm”. are used, but before this, multiple objective values should be combined into a single objective value. There are several approaches to combine all objective function values into a single value. (Ulungu et al., 1999) use a criterion scalarizing function which takes the weighted sum of the objective functions. Czyzżak & Jaszkiewicz (1998) use a diversified set of weights to obtain a diverse set of solutions in the final front in their SA-based algorithm which they call Pareto Simulated Annealing (PSA). Suppapitnarm et al. (2000) propose another MOSA, which they call SMOSA that suggests maintaining different temperatures, one for each objective. In the study a “return-to-base” strategy which restarts the search from a random archive solution is introduced. In dominance-based acceptance rules, $X$ and $X^{\mathrm{\prime }}$ are compared with respect to the dominance relation. $X$ is said to dominate $X^{\mathrm{\prime }}$ which is denoted as $X\succ X^{\mathrm{\prime }}$,if it is better than $X^{\mathrm{\prime }}$ in at least one objective and it is not worse than $X^{\mathrm{\prime }}$ in all of the other objectives. If at any iteration, $X\succ X^{\mathrm{\prime }}$ then, $X^{\mathrm{\prime }}$ is accepted with a probability which is also computed according to the domination status of the solutions. Suman (2004) proposes a Pareto Domination-Based MOSA which uses the current domination status of a solution to compute its acceptance probability. The acceptance probability of a solution is determined by the number of solutions in the current front dominating it. Bandyopadhyay et al. (2008) proposes Archive-based Multi-objective Simulated Annealing (AMOSA) which again uses a dominance-based acceptance rule. A detailed survey on single-objective and multi-objective SA can be found in Suman & Kumar (2006).

The MOSA algorithm proposed in this study iterates in a similar fashion as the single-objective SA algorithm. However, there are now two objective values to take into consideration while moving from $X$ to $X^{\mathrm{\prime }}$, where the first objective being the number of FLOPs required by the network, and the second being the error rate achieved by the network. Criterion scalarizing rules can be used to combine the two objective values into one, but this method requires both objective values to be in the same scale and proper weights to be defined for each objective. On the other hand, probability scalarizing rules calculate the acceptance probability of $X^{\mathrm{\prime }}$ for each objective individually, then a decision rule is applied to aggregate these probabilities such as taking the minimum, maximum or the product of these probabilities. As different probabilities are evaluated for each objective, a different temperature should be maintained for each objective in this approach. Due to the difficulties mentioned above, we adopt the Smith’s Pareto dominance rule (Smith et al., 2008) in our MOSA algorithm. Starting from the first iteration, all non-dominated solutions encountered during the search are collected in an external archive, $A$, which is updated whenever a new solution is accepted. The state of $A$ is expected to improve as the algorithm iterates, and archive solutions in the final iteration form the Pareto front which is presented to the decision maker. As shown in the MOSA pseudo-code below, as the algorithm iterates from $X$ to $X^{\mathrm{\prime }}$, if $X\succ X^{\mathrm{\prime }}$ ( $X$ dominates $X^{\mathrm{\prime }}$), then $X^{\mathrm{\prime }}$ is accepted based on Smith’s Pareto dominance rule which calculates the acceptance probability of a given solution by comparing it with the current archive which contains potentially Pareto-optimal set of solutions. Smith’s rule uses a difference in the energy level, $\mathrm{\Delta }F$, to compare two solutions with respect to all objectives as follows: Let $\mathrm{A}$ denote the current potentially Pareto optimal set and $\tilde{A}$ denote the union of this set and two competing solutions, $\tilde{A}=\mathrm{A}\cup X\cup X^{\mathrm{\prime }}$, then $\mathrm{\Delta }F=\{1/\left|\tilde{A}\right|\}\cdot \left\{\left|F\left(X^{\mathrm{\prime }}\right)\right|-\left|F\left(X\right)\right|\right\}$, where $F\left(X\right)$ denotes the solutions in $\mathrm{A}$ that dominate $X$ plus 1. The probability of accepting $X^{\mathrm{\prime }}$ is then computed as $p_{acc}=exp\left(-\mathrm{\Delta }F/T_{cur}\right)$, where the only difference with a single-objective version is the calculation of $\mathrm{\Delta }F$. In this approach, only one temperature is maintained regardless of the number of objectives. In a multi-objective version of SA algorithm, archive maintenance related rules should also be defined. If $X\nsucc X^{\mathrm{\prime }}$ ( $X$ does not dominate $X^{\mathrm{\prime }}$), then $X^{\mathrm{\prime }}$ becomes a candidate for being a member of the archive. This is determined as follows: If $X^{\mathrm{\prime }}$ dominates any solution in $\mathrm{A}$, then $X^{\mathrm{\prime }}$ is accepted and the archive is updated by inserting $X^{\mathrm{\prime }}$ and removing all the solutions dominated by it. If $X\nsucc X^{\mathrm{\prime }}$ but there is a solution $a\in A$ dominating $X^{\mathrm{\prime }}$, then no archive update is performed and the current iteration is completed either by accepting $X^{\mathrm{\prime }}$, $a$ or $X$ as the current solution. If $X^{\mathrm{\prime }}\succ X$, then $a$ and $X^{\mathrm{\prime }}$ compete for being selected as the current solution according to the probability calculated with the same rule described above. Here, $a$ represents a random archive solution dominating $X^{\mathrm{\prime }}$. Continuing the search process with a previously visited archive solution is known as return-to-base strategy. If $X\nsucc X^{\mathrm{\prime }}$ and $X^{\mathrm{\prime }}\nsucc X$, then first these two solutions compete, and then the winning solution competes with $a$ according to the same probabilistic acceptance rule. If $X\nsucc X^{\mathrm{\prime }}$ and there is no solution $a\in A$ that dominates $X^{\mathrm{\prime }}$, and $X^{\mathrm{\prime }}$ does not dominate any solution $a\in A$, then $X^{\mathrm{\prime }}$ is accepted and inserted into the archive. As the archive is updated by removing all dominated solutions, the size of the archive does not grow very large. Selection of other MOSA parameters such as $T_{init}$, $T_{final}$, cooling rate and also the number of inner and outer iterations are given in detail in Section “MOSA parameter tuning”.

Performance evaluation criteria

In multi-objective optimization problems, final fronts are evaluated according to two performance criteria: (i) Closeness to the Pareto-optimal front and, (ii) Diversity of the solutions along the front (Zitzler, Deb & Thiele, 2000; Deb, 2001). Generational Distance (GD) is the most-widely used metric to examine the closeness of the final front to Pareto-optimal front (Van Veldhuizen & Lamont, 2000). In order to measure the diversity which is comprised of two components, namely distribution and spread, two metrics, spacing and maximum spread are used. Spacing evaluates the relative distance among the solutions, whereas spread evaluates the range of the objective function values.

GD metric requires a Pareto-optimal front in order to compare any two fronts. Since this optimal front is not known in advance, it is approximated by an aggregate front, ${\mathrm{A}}^{\ast }$, which is formed by combining all solutions in the two fronts. Then, the amount of improvement achieved in each front is measured with respect to this aggregate front. GD for a given front is calculated as given in Eq. (1), where $d_i$ is the Euclidean distance between the solution $i\in A$ and the closest solution $k\in {\mathrm{A}}^{\ast }$. In Eq. (2), $P^{max}$ and $E^{max}$ denote the maximum objective values, whereas $P^{\ast }$ and $E^{\ast }$ denote the minimum objective function values observed in ${\mathrm{A}}^{\ast }$. For this GD metric, smaller the better.

(1) $$GD\left(A\right)={\displaystyle \frac{\sqrt{{\sum }_{i\in A}{d}_i^2}}{\left|A\right|}}$$ (2) $$d_i=\underset{k\in A^{\ast }}{min}\sqrt{{\displaystyle \frac{1}{2}\left({\left({\displaystyle \frac{P_i-P_k}{P^{max}-P^{\ast }}}\right)}^2+{\left({\displaystyle \frac{E_i-E_k}{E^{max}-E^{\ast }}}\right)}^2\right)}}$$Zitzler’s spread metric which is computed as in Eq. (3) is used to measure the extent of the fronts. This metric simply calculates Euclidean distance between the extreme solutions in a given front. If a front $A$ includes all the extreme solutions in ${\mathrm{A}}^{\ast }$, then this front takes a value of 1 according to this metric.

(3) $$S\left(A\right)=\sqrt{{\displaystyle \frac{1}{2}\left({\left({\displaystyle \frac{\underset{i\in A}{max}{P}_i-\underset{i\in A}{min}{P}_i}{P^{max}-P^{\ast }}}\right)}^2+{\left({\displaystyle \frac{\underset{i\in A}{max}{E}_i-\underset{i\in A}{min}{E}_i}{E^{max}-E^{\ast }}}\right)}^2\right)}}$$

Spacing metric (Schott, 1995) is used to measure the diversity along a given front. For each solution, distance to its closest neighbor is calculated as shown in Eq. (4). Then, the spacing metric is calculated as the standard deviation of these distances as shown in Eq. (5), where $\overline{d}$ is the average distance value. If the distance between the closest solutions are distributed equally, then the value of this metric approaches to zero which is the desired condition.

(4) $$d_i=\underset{k\in A\wedge k\ne i}{min}\left({\displaystyle \frac{\left|P_i-P_k\right|}{\underset{j\in A}{max}{P}_j-\underset{j\in A}{min}{P}_j}+{\displaystyle \frac{\left|E_i-E_k\right|}{\underset{j\in A}{max}{E}_j-\underset{j\in A}{min}{E}_j}}}\right),\mathrm{f}\mathrm{o}\mathrm{r}i\in A$$ (5) $$Sp\left(A\right)=\sqrt{{\displaystyle \frac{1}{\left|A\right|}\sum _{i\in A}{\left(d_i-\overline{d}\right)}^2}}$$

Implementation details

Experimental setup

MOSA parameter tuning

In our study, the MOSA algorithm adopts a real time initial temperature selection strategy in which the worsening moves are accepted with an initial probability, $p_{acc}$, of 0.5. $T_{init}$ is then calculated as $T_{init}=-\left(\mathrm{\Delta }{F}_{ave}/\left(ln\left(p_{acc}\right)\right)\right)$. Average initial worsening penalty amount, $\mathrm{\Delta }{F}_{ave}$, is approximated executing a short “burn-in” period in which worsening moves as well as improving ones are all accepted. We run a burn-in period with 100 iterations and 39 of them resulted in worsening solutions. In Table 2, some information regarding only a few of those iterations are given. In the table, $F\left(X\right)$ denotes the solutions in $\mathrm{A}$ that dominate $X$ plus 1, $\left|A\right|$ denotes the size of the archive, $F\left(X^{\mathrm{\prime }}\right)-F\left(X\right)$ denotes the worsening amount in terms of domination count and $\mathrm{\Delta }F$ denotes the objective value calculated as $\mathrm{\Delta }\mathrm{F}=\{1/\left|\tilde{A}\right|\}\cdot \left\{\left|F\left(X^{\mathrm{\prime }}\right)\right|-\left|F\left(X\right)\right|\right\}$, where $\left|\tilde{A}\right|$ equals $\left|A\right|+2$. The average $\left|A\right|$ value obtained in this burn-in period is used to calculate $\mathrm{\Delta }{F}_{ave}$. This period results in a $\mathrm{\Delta }{F}_{ave}$ value of 0.40 which also gives a $T_{init}$ value of 0.577. We adopt a similar temperature adjustment approach to determine $T_{final}$ value, where in this case we assumed a $F\left(X^{\mathrm{\prime }}\right)-F\left(X\right)$ value of at most 1 to be accepted with the same probability. In order to calculate $T_{final}$, final front size needs to be approximated, and a value of 10 for the final front size seemed to be reasonable according to preliminary experiments, and $T_{final}$ is calculated as 0.12.

Table 2:

Worsening moves encountered during burn-in period.

Iteration no	F(X)	F(X′)	|A|	F(X′)–F(X)	ΔF
3	1	4	3	3	0.600
5	5	4	4	4	0.667
9	1	2	5	1	0.143
11	1	6	6	5	0.625
13	1	7	7	6	0.667
⋮	⋮	⋮	⋮	⋮	⋮
Average	2.69	5.94	6.02	3.25	0.40

DOI: 10.7717/peerj-cs.338/table-2

In this study, the MOSA algorithm is allowed to run for a number of iterations defined by the iteration budget of 500, which means that at most 500 solutions are created during a single run. The amount of this total iteration budget to be allocated in outer and inner iterations is determined by the cooling rate parameter. We tested for several cooling rate values keeping the total number of iterations at 250 due to large computational times. Table 3 shows the number of outer and inner iterations calculated for different cooling rate values under the same budget and temperature values. Based on these iteration numbers, we selected cooling rate values of 0.95, 0.90 and 0.85 for the parameter tuning process. We allowed the MOSA algorithm run for 3 times for each of those cooling rate values using different random number seeds. The fronts obtained under each cooling rate value for three runs are given in Fig. 3. As can be seen in the figure, three Pareto fronts are obtained for each cooling rate value. Although one can notice that there is no difference among the fronts formed with different cooling rate values visually, we applied Kruskal–Wallis h-test to determine whether there are significant differences. This test is applied separately for GD, Spread and the front size metrics, but none of those metrics proved a significant difference among the fronts. Therefore, we selected a cooling rate of 0.85 arbitrarily.

Table 3:

The number of outer and inner iterations calculated for different cooling rate values under the same budget and temperature values.

Iteration budget	Tinit	Tfinal	Cooling rate	#Outer iterations	#Inner iterations
250	0.577	0.12	0.99	156.2	1.6
250	0.577	0.12	0.95	30.6	8.1
250	0.577	0.12	0.9	14.9	16.7
250	0.577	0.12	0.85	9.6	25.8
250	0.577	0.12	0.8	7.0	35.5

DOI: 10.7717/peerj-cs.338/table-3

Results analysis

We first present the objective space distribution of all configurations obtained during each independent run of the MOSA. In order to show the search ability of the proposed algorithm, the MOSA search space is compared to the search space generated by the single-objective SA method also to the search space generated by Random Search (RS) method in which all solutions are accepted regardless of their objective values. Final fronts are also compared in terms of the front metrics given in “Performance Evaluation Criteria”. Then, some solutions in the final front are selected to be trained for longer epochs in order to perform post search analysis. The configurations generated by the MOSA are compared to both human designed state-of-the-art configurations and the configurations generated by other search methods like evolutionary algorithms in terms of the test error, the number of FLOPs, the number of parameters and also the search cost which is reported as GPU-days.

Fronts analysis

A MOSA search is repeated three times with different initial random seeds, and the objective space distribution of all configurations encountered during each of those search processes are illustrated in Fig. 4. In all of those runs, SA especially focuses on the areas with small classification error but with large number of FLOPs, es expected. In order to show the search ability of the MOSA over the SA, we compare the final fronts generated by each method in terms of closeness to the Pareto-optimal front and the diversity of the solutions along the fronts. The configurations generated by the SA seem to be much complex than the configurations generated by the MOSA which also takes into account the complexity as the other objective. Due to the large computational time requirement (at least 2 × longer), SA is run only once. As there is no archive mechanism in the SA, the final SA front is formed by selecting all non-dominated configurations encountered during a single run. The MOSA and the SA fronts are also shown in Fig. 4 along with the objective space distributions of those algorithms. In addition to making comparison using graphical plots, we used the metrics given in “Performance Evaluation Criteria” to compare the fronts generated by each approach in a quantitative manner. These metrics are as follows: Generational Distance (GD), Spread (S) and Spacing (Sp) metrics. The comparison results using these metrics are given in Table 4. In the table, MOSA_01 represents the MOSA front obtained at the first run, and MOSA_02 represents the MOSA front obtained at the second run and so on. GD of each front is calculated with respect to $A^{\ast }$ which is formed by combining all four fronts. S of each front is calculated with respect to the extreme points considering again all four fronts. Sp metric is calculated for each front separately, because it measures within front distribution quality of a given front.

Table 4:

Evaluation of the final fronts with respect to three metrics and the number of solutions.

	MOSA_01	MOSA_02	MOSA_03	SA	RS
GD	0.0011	0.0629	0.0387	0.0765	0.0286
S (spread)	0.7127	0.7944	0.5867	0.6681	0.4755
Sp (spacing)	0.0891	0.1483	0.1384	0.1519	0.1268
#Solutions	11	10	10	7	10

DOI: 10.7717/peerj-cs.338/table-4

The search ability of the proposed method is also validated by comparing it to the RS method. The same comparisons performed between the MOSA and the SA are applied to compare the MOSA and the RS methods. When the MOSA and the RS solutions are combined to create $A^{\ast }$, none of the RS front solutions take place in $A^{\ast }$ as in the case of SA. $A^{\ast }$ is composed of only the solutions coming from the MOSA which means that GD calculations are made against the same reference front; therefore we decided to present all these comparison results on the same table, Table 4. Figure 5 illustrates the objective space distribution and the fronts obtained by the MOSA and the RS methods.

After search analysis

In order to measure the actual classification performance of the configurations generated by the MOSA method, nine configurations are selected and subjected to longer training using the original CIFAR-10 training dataset as detailed in “Training After the Search Process”. Among these nine configurations, the networks with the lowest error rates are selected for comparison with the networks reported in the literature. As the MOSA algorithm considers two objectives, namely, error rate and the FLOPs, during the search process, a multi-objective comparison with respect to the front evaluation metrics should be performed for a fair comparison. Unfortunately, most of the studies do not report the objective values of the solutions in the final frontiers, or they consider different objective functions to estimate the complexity of the generated networks. In some studies, computational complexity is measured as computational time in GPU-days. However, a comparison with respect to computational time may not be reliable due to some external factors such as the temperature in the computing environment, even if the same GPU models are used for the experiments. Therefore, the number of FLOPs that a network carries out during a forward pass is the most reliable complexity measure. In Table 5, the final performance of three MOSA configurations in terms of both objectives are presented. In addition, a graphical comparison is given in Fig. 6 where the dominance relation between different configurations can be easily noticed. As in “Fronts Analysis”, a comparison to the configurations found with the single objective SA algorithm is also performed and the results are presented in the table in order to show the effect of using a multi-objective solution approach for this bi-objective optimization problem. From the SA trade-off front, three configurations with the lowest error rate are selected for longer training, and the performance of each configuration is reported in the table. The same approach is followed for the RS, as well. Other search generated architectures are included in the second part of the table. In the table, the accuracy, the number of FLOPs and the number of parameters columns all represent the values reported in the original papers. As stated before, we did not consider the search cost in terms of time in the comparisons; however, in order to give an idea about the search cost of the MOSA algorithm, we run one MOSA search on the same GPU as in NSGA-Net (Lu et al., 2019a). We observe that MOSA takes 50 hours on a single NVIDIA 1080Ti GPU, which equals to 2.08 GPU-days, whereas it takes 8 GPU-days for NSGA-Net.

Table 5:

Comparison of MOSA architectures to other search generated architectures.

Architecture	Search method	Test accuracy (%)	FLOPs (M)	Parameters (M)
MOSA_soln1	MOSA	91.97	16.2	3.329
MOSA_soln2	MOSA	91.52	1.7	0.854
MOSA_soln3	MOSA	91.14	1.2	0.641
SA_soln1	SA	92.65	6.8	5.9
SA_soln2	SA	92.85	11.7	3.572
SA_soln3	SA	90.04	7.1	3.45
RS_soln1	RS	88.27	4.1	2.052
RS_soln2	RS	91.25	5.1	2.603
RS_soln3	RS	86.26	1.7	0.887
NSGA-Net (Lu et al., 2019b)	EA	96.15	1290	3.3
PPP-Net (Dong et al., 2018)	EA	95.64	1364	11.39
AmoebaNet-A + cutout (Real et al., 2019)	EA	97.23	533	3.3
NASNet-A + cutout (Zoph et al., 2018)	RL	97.09	532	3.2

DOI: 10.7717/peerj-cs.338/table-5

A comparison of the MOSA configurations to the human designed state-of-the-art configurations are also performed. In order to be able to make a fair comparison, especially in terms of test accuracy, each of these state-of the-art architectures are rebuilt and trained using the same augmentation techniques as in the MOSA training process. The results are presented in Table 6.

Table 6:

Comparison of MOSA architectures to human designed state of the art architectures.

Architecture	Test accuracy (%)	FLOPs (M)	Parameters (M)
MOSA_soln1	91.97	16.2	3.329
MOSA_soln2	91.52	1.7	0.854
MOSA_soln3	91.14	1.2	0.641
LeNet-5 (LeCun et al., 1990)	71.86	0.162	0.081
VGGNet-16 (Simonyan & Zisserman, 2014)	86.38	67.251	33.638
ResNet-50 (He et al., 2016)	76.01	47.052	23.604
DenseNet-32 (Huang et al., 2017)	89.38	2.894	0.494

DOI: 10.7717/peerj-cs.338/table-6