To measure the performance of the proposed approaches, we use two metrics: hypervolume and the fraction of solutions dominated by other algorithms.
5.2.1. Hypervolume
Hypervolume is the metric that indicates the volume of the hyperspace dominated by a set of solutions. This metric is widely adopted in practice both for evaluation of genetic optimization algorithms and for guiding the evolutionary process in indicator-based methods [
38]. The value of hypervolume depends on the chosen reference point. To simplify the computational process and due to the fact that we solve a maximization problem, we adopt the origin of coordinates as a reference point for hypervolume calculation. In our case, this point also corresponds to the worst possible solution. Larger hypervolume values indicate better performance.
The values of the hypervolume indicator obtained for different numbers of knapsacks for
NSGA-II are presented in the second column of
Table 3. The first part of the table corresponds to random selection and the second to tournament selection. We can see that the hypervolume values obtained with random selection are of the same scale as those obtained with tournament selection. In addition, when
, random selection results in higher values of hypervolume. However, when the number of knapsacks reaches 10, tournament selection becomes more effective in terms of hypervolume maximization.
Columns 3–6 show the relative increase (positive number, bold font) or decrease (negative number) in the hypervolume indicator for other algorithms. For a more intuitive representation, the values of these columns are also graphically presented in
Figure 9. The first observation that we can make is that
NSGA-III, while developed for many-objective optimization, almost always results in lower hypervolume values, even for larger numbers of objectives. This supports similar observation from [
39].
NSGA-III reaches its minimum in relative increase in hypervolume of approximately
% for
for random selection and for
for tournament selection. After that, it starts increasing if random selection is used, and it does not change much for tournament selection.
The values of relative hypervolume increase for PO-count are very close to 0. This means that PO-count results in a population covering the same hypervolume as NSGA-II. The largest difference between NSGA-II and PO-count is observed for with tournament selection. In this case, the hypervolume of PO-count is 1.86% less than that of NSGA-II. Contrarily, both PO-prob and PO-prob* improve hypervolume significantly as compared to NSGA-II. This difference is visible for small and is especially prominent for large numbers of knapsacks. When , PO-prob and PO-prob* increase the hypervolume by up to 4%. After that, we observe an inverse pattern. However, the relative decrease in this case does not exceed %. Finally, starting from for random selection and for tournament selection, PO-prob again results in larger hypervolume values. For PO-prob*, this happens even faster: for random selection and for tournament selection. From this point on, we see a rapid growth with a maximum relative increase observed for : % and % for random selection, and % and % for tournament selection. The fact that PO-prob* is closer to NSGA-II than PO-prob is due to PO-prob* turning into NSGA-II during the last 150 generations. We can also notice that the effect of the PO-prob algorithm is less visible in the case of tournament selection.
5.2.2. Fraction of Dominated Solutions
We calculate the percentage of dominated solutions as follows. For a given pair of algorithms,
algorithm1 and
algorithm2, we calculate how many solutions of
algorithm2 (
dominated algorithm) are dominated by solutions of
algorithm1 (
dominating algorithm). After that, we average the obtained results among all 30 independent runs. The detailed results for
,
, and
are shown in
Table 4 and
Table 5 for random and tournament selection respectively. For example, for random selection and
, on average, the solutions of
NSGA-II dominate 47.38% of the solutions produced by
PO-prob*, see the third row and sixth column in
Table 4. At the same time, on average, only 6.24% of solutions produced by
NSGA-II are dominated by the solutions of
PO-prob*, see the seventh row and second column of the same table. Naturally, a better algorithm has a lower number of dominated solutions and a larger number of dominating solutions. This means that we want to achieve minimum per column and maximum per row.
The rows in bold show the average number of dominated solutions for all dominating algorithms, denoted by . The value of is calculated as a mean per column, and better algorithms have lower values of . For example, for the same setup, on average, 21.77% of solutions produced by NSGA-II are dominated by other algorithms. The corresponding value for PO-prob* is 36.86%. This means that for this configuration, NSGA-II performs better than PO-prob*.
Results for with random selection. Analyzing the results from
Table 4, we can see that for two knapsacks and random selection,
PO-prob is dominated the least number of times.
NSGA-II,
NSGA-III, and
PO-count, on average, dominate no more than 13.52% of the solutions of
PO-prob.
PO-prob*, however, dominates on average 23.46% of the solutions of
PO-prob. At the same time,
PO-prob almost never dominates other algorithms. The only corresponding non-zero value is 8.37%, which represents the fraction of solutions of
PO-prob* dominated by
PO-prob. This means that solution spaces of
PO-prob and other algorithms are distinct and do not intersect much. This supports our previous observation, see
Figure 4.
PO-prob* is dominated the most often by others, with more than 40% of solutions being dominated by NSGA-II, NSGA-III, and PO-count.
NSGA-II, NSGA-III, and PO-count form a group of algorithms that are not usually dominated by the probability-based algorithms, PO-prob and PO-prob*, but have high domination values among themselves. Among these three algorithms, PO-count seems to provide the best configuration: it is dominated the least number of times by NSGA-II and NSGA-III; the corresponding values are 39.21% and 34.51%, respectively. At the same time, PO-count dominates 43.34% of the solutions of NSGA-II and 45.01% of the solutions of NSGA-III. This corresponds to +5% and +10% compared to the inverse domination relationship.
Results for with tournament selection. For the same number of knapsacks,
, and tournament selection, we can see that the probability-based algorithms are dominated more often than the counting-based algorithms, see
Table 5. The average number of dominated solutions through all algorithms,
, is 44.52% for
PO-prob and 52.31% for
PO-prob*. At the same time, the values of
for
NSGA-II,
NSGA-III, and
PO-count stay relatively close to the corresponding values for random selection.
The results for this configuration, also confirm that NSGA-II performs the best: it is dominated the least often and on average dominates more solutions that other algorithms. This difference becomes especially prominent when comparing it with probability based algorithms. NSGA-II dominates 47.81% of the solutions of PO-prob and 61.20% of solutions of PO-prob*. The latter algorithms, however, dominate only 6.61% and 7.78% of the solutions of NSGA-II, respectively.
Results for . However, this pattern changes when the number of objectives increases. Already for , NSGA-II and PO-count are substantially outperformed by other algorithms, both for random and tournament selection. More than 60% of solutions of these two algorithms are dominated by the probability-based algorithms PO-prob and PO-prob*. The level of domination in the inverse direction is less than 1%.
It is interesting to note that
NSGA-III also results in a set of solutions that are rarely dominated. The only algorithm capable of dominating a significant fraction of solutions of
NSGA-III is
PO-prob. The corresponding values are 27.51% for random selection and 14.72% for tournament selection. This shows the superiority of
NSGA-III for many-objective optimization as compared to
NSGA-II. As it was shown in
Section 5.2.1, the solutions of
NSGA-III cover less hypervolume than the solutions of
NSGA-II. However, as we can see now, approximately 40% of the solutions of
NSGA-II are dominated by the solutions of
NSGA-III. Thereby, despite covering less hypervolume,
NSGA-III should be preferable in practice.
A considerable fraction of solutions of PO-prob* is dominated by NSGA-II (19.00% and 15.69%) and by PO-prob (48.65% and 39.24%). At the same time, only a tiny fraction of solutions of PO-prob is dominated by solutions produced by other algorithms. The largest fraction of solutions of PO-prob are dominated by NSGA-III: 0.08% for random selection and 0.53% for tournament selection.
Results for . When the number of knapsacks increases even further, we can notice that all algorithms tend to produce more distinct sets of solutions, as the domination fractions reduce. However, the general pattern stays the same. The solutions of NSGA-II and PO-count are more often dominated by the solutions of other algorithms. The solutions of PO-prob are almost never dominated. The next best performance is demonstrated by NSGA-III, with 9.59% and 3.60% of solutions dominated by PO-prob for random and tournament selection, respectively. PO-prob* has approximately 20% of solutions dominated by PO-prob and 10% of solutions dominated by NSGA-III.
Distribution of . To further analyze how the domination fraction changes for different numbers of knapsacks, we demonstrate the distribution of
for different values of
in
Figure 10. Recall that
shows the fraction of dominated solutions averaged over different dominating algorithms, and its values are present in bold in
Table 4 and
Table 5.
We can notice similar tendencies for both random and tournament selection. NSGA-II and PO-count behave very similarly. For , the value of for these algorithms is around 23%. After that, it starts increasing and reaches its peak of approximately 45% for . Finally, it gradually decreases to 20% for .
NSGA-III starts at a similar level. It reaches its peak of approximately 30% for for random selection and for tournament selection. After, it decreases below 10% for and stays below this value for random selection, and relatively close to zero in the case of tournament selection. These results once again demonstrate the superiority of NSGA-III over NSGA-II for large numbers of objectives.
PO-prob starts at around 15% for random selection and 45% for tournament selection. However, the value of drops to 0 very fast. This shows that the solutions produced by this algorithm are almost never dominated by any other algorithm for large numbers of knapsacks.
PO-prob* behaves similarly to PO-prob until . Next, starts going up and reaches its maximum at for random selection and at for tournament selection. For larger values of , gradually decreases, but it is always larger than the corresponding value for NSGA-III.