An active learning Gaussian modeling based multi-objective evolutionary algorithm using population guided weight vector evolution strategy

: The inverse model based multi-objective evolutionary algorithm (IM-MOEA) generates offspring by establishing probabilistic models and sampling by the model, which is a new computing schema to replace crossover in MOEAs. In this paper, an active learning Gaussian modeling based multi-objective evolutionary algorithm using population guided weight vector evolution strategy (ALGM-MOEA) is proposed. To properly cope with multi-objective problems with different shapes of Pareto front (PF), a novel population guided weight vector evolution strategy is proposed to dynamically adjust search directions according to the distribution of generated PF. Moreover, in order to enhance the search efficiency and prediction accuracy, an active learning based training sample selection method is designed to build Gaussian process based inverse models, which chooses individuals with the maximum amount of information to effectively enhance the prediction accuracy of the inverse model. The experimental results demonstrate the competitiveness of the proposed ALGM-MOEA on benchmark problems with various shapes of Pareto front.


Introduction
Many practical problems in the fields of machine learning [1][2][3][4] and engineering optimization [5,6] often involve two or more optimization objectives, which are called multi-objective optimization problems (MOPs).These objective functions always conflict with each other.Therefore, it is impossible to have a unique optimal solution to optimize all objectives at the same time.For MOPs, the ultimate goal is to find a set of compromise solutions which are non-dominated each other.One of the most widely studied MOPs is the box-constrained continuous MOP which is defined as follows. .

. ( , )
T m where x and  represent decision variable vector and search space, respectively. ( )F x denotes the objective function vector consisting of m objective functions.
As a kind of random search algorithms, evolutionary algorithm is very suitable for MOPs because of its characteristics of population-based search.Currently, multi-objective evolutionary algorithms (MOEAs) have used to solve difficult MOPs with complex shapes of Pareto front (e.g., discontinuous, degenerated or multi-modal) or with many objectives.Many branches, such as dominance modification-based approaches, indicator-based methods and decomposition-based strategies have been developed in recent years.The ultimate goal of MOEAs is to obtain a set of widely distributed Pareto optimal solutions to approach true Pareto front (PF).In particular, decomposition-based algorithm is a kind of effective algorithms due to easy execution.To design an effective decomposition-based algorithm, two factors are crucial to the quality of solutions.One is how to generate and adjust the search directions in the objective space to guide the population towards a true PF [7,8], and another is how to generate new high-quality solutions to ensure convergence and diversity [9,10].
One interest of this study focuses on dynamically generating search directions for adapting to cope with MOPs with various shapes of PF.In general, the search vectors in objective space can be represented by a number of weight vectors, which divide the MOP into several sub-problems of singleobjective optimization.The proper setting of the weight vectors helps to generate the high quality nondominated solution set [11].If the PF shape is regular, uniformly distributed weight vectors have the potential to obtain uniformly distributed solutions.However, if the PF shape is irregular, it becomes very difficult to generate appropriate weight vectors to ensure the diversity of solutions.So far, lots of weight vector generation and adjustment strategies have been developed to adjust the weight vectors for different shapes of PF, such as the schemes of random or predefined adjustment, fitting-based adjustment and adaptive adjustment.Serafini [12] used randomly modified reference vectors to guide the search.Jin et al. [13] proposed predefined weight vectors which can be adjusted periodically in a fixed way to guide the population evolution.Gu et al. [14] adopted fitting or interpolation strategy to approximate the shape of PF, and re-sampled the uniformly distributed weight vectors on the estimated Gu et al. [15] designed reference points generation approach by training self-organizing mapping (SOM) network.Wang et al. [16] co-evolved the population and the weight vectors.Furthermore, some authors propose the weight vector adjustment strategies based on the information of local population or local archive.They identify the validity of the weight vectors with the help of the relation between the current weight vector and the solution, and dynamically adjust the weight vectors by a deletingand-adding operator.Li et al. [8] used an archive and added new weight vectors in unexplored promising regions after deleting poorly invalid weight vectors.Yi et al. [17] adopted maximum-vectorangle-first principle to sequentially choose individual as weight vectors.Ge et al. [18] regenerated new reference vectors in the neighborhood of valid reference vectors combined with reference point incremental learning.These methods constantly update the weight vector with the help of local information of population, which have shown their effectiveness for many problems with certain kinds of irregular PF problems.We plan to adopt a learning strategy to dynamically adjust the distribution of weight vectors based on population information.
Another interest of this study is the reproduction strategy.Two kinds of approaches are adopted to generate offspring, namely a genetic-based operator and estimation distribution-based algorithms (EDAs).In genetic based MOEAs, simulated binary crossover (SBX), particle swarm algorithm (PSO), differential evolution (DE) and hybrid operators are often used to generate new individuals, which are commonly adopted in most MOEAs.Besides, EDAs are a specific framework, which initially collect relevant statistical information of selected solutions and establish probability models to generate new solutions by sampling.Different from genetic based operators, EDA-based methods [19][20][21] can better estimate the distribution of population via statistical information, and thus can be easy to create offspring in promising locations.Laumanns et al. [22] made use of the relationship of decision variables and developed a binary decision tree-based Bayesian optimization model.Karshenas et al. [23] established a joint probabilistic model of Bayesian network employing the information of objectives and variables.Zhang et al. [24] took advantage of the regularity property of MOPs, and constructed a probability distribution of candidate solutions by using local principal component analysis (PCA).Cheng et al. [25] utilized the statistical information of population and established Gaussian process-based inverse models (IM-MOEA) from objective space to decision space to generate offspring.In the procedure of reproduction, once the new test points in objective function space are given as input, the established Gaussian regression model will generate new points in decision space as offspring.Recently, many scholars have made several variants to IM-MOEA.Farias et al. [26] adopted decomposition framework for evolution.A two-stage environmental selection [27] was incorporated with IM-MOEA to improve the both convergence and diversity of population.Zhang et al. [28] proposed a non-random grouping strategy to enhance the accuracy of models.
Although IM-MOEA and their variants perform well on MOPs with regular PFs, there exist some aspects for improvement.First, a number of evenly distributed weight vectors are adopted to partition the population into sub-problems in IM-MOEA.However, if the shape of the true Pareto front is not regular (e.g., discontinuous, degenerate, inverted PF or PF with long tail and peak), the evenly distributed weight vectors cannot guarantee the uniform partition of the objective space, thereby affecting the uniformity and diversity of solutions.Second, in the process of establishing Gaussian process based inverse model, the training samples are selected randomly, and this will affect the accuracy of the prediction model to some extent.
To address the above issues, the proposed study aims to develop a learning-to-adjustment scheme that dynamically adjusts the reference vectors in objective space, and establish more accurate Gaussian regression models to estimate the location of true Pareto optimal set.An active learning Gaussian modeling based multi-objective evolutionary algorithm using population-guided weight vector evolution strategy (ALGM-MOEA) is proposed.The main contribution focuses on the following two aspects.
1) To properly cope with multi-objective problems with different shapes of Pareto front (PF), a new population-guided weight vector evolution strategy is developed, which dynamically adjusts the weight vector according to the distribution of generated PF.
2) To effectively enhance the prediction accuracy of the inverse model, an active learning-based training sample selection method is designed to build Gaussian process based inverse models, which selects individuals in the population with the maximum amount of information, and this helps to improve the prediction accuracy of established models and generate satisfactory offspring.
The rest of the paper is organized as follows.The basic framework of IM-MOEA is described in Section 2. In Section 3, we elaborate the proposed algorithm.Section 4 presents experimental comparisons of the proposed algorithm with the state-of-the-art algorithms.Finally, the conclusion and future work are put in Section 5.

Related works
Estimation of distribution algorithm (EDA) utilizes the statistical information from current individuals and establishes the probability distribution model reflecting the geometric shape of Pareto front or Pareto set.Subsequently, sampling is executed according to the established model to generate new offspring.16endwhile IM-MOEA can be classified as a type of estimation of distribution algorithms, which aims to build Gaussian regression inverse models (GPR) to approximate the Pareto front.In order to simplify the modeling process, the population is partitioned into K sub-populations by a number of evenly distributed weight vectors after calculating the position information of each individual and its neighboring reference weight vector.
In each sub-population, the operators including environment selection, inverse modeling and reproduction are executed independently.Given a MOP with m objective and n decision variables, it is difficult to train a m input and n output multivariate model to approximate the relation from objective vectors to decision vectors.Instead, a group of decomposed univariate probability distribution models are built for each objective.To be specific, IM-MOEA takes the j th  objective values and the i th  decision values of all solutions in one sub-population as the training data, denoted by , [ , ] ( , ) , ( , ) , t N represents the population size used to train the model), and then constructs the conditional probability distribution ( | )   i j

P x f
to reflect the mapping relationship between them, which can be estimated by training a Gaussian process regression inverse model In each model group for one objective j f , random grouping strategy is adopted to assign a part of , to establish L Gaussian process regression models decision variables will remain unchanged.For example, if the decision variables 1 x and 2 x are assigned to objective 1 f , then two models ) formulated in (2).Using this trained Gaussian regression model, we can input the j th  objective values 1 ( , ) of all solutions in one subpopulation as the training data, and a new point * j f sampling in the objective space as test data, and then predict the mean and variance of output * i x in the decision space as offspring.The detailed induction can be referred to [25].
where I represents an identity matrix, and ( ) is covariance matrix between any twotraining sample and p q j j f f .Here, ( , ) is the covariance parameters between any twotraining sample and , and ** C is a covariance parameters of test data * j f .

Motivation
In order to enhance the search efficiency of distribution estimation algorithm and improve the prediction accuracy of Gaussian model when dealing with complex multi-objective optimization problems with irregular shape of Pareto front, and an active learning Gaussian modeling based multiobjective evolutionary algorithm using population-guided weight vector evolution strategy (ALGM-MOEA) is proposed.We integrate a learning-to-adjustment scheme into the procedure of update weight vector and modification of Gaussian modeling.First, in order to cope with problems with complex shape of PF, we dynamically adjust the weight vectors by learning the distribution of current population, thereby ensuring that the search occurs in the potential region where the solutions locate.Second, an active learning-based training sample selection method is designed to build Gaussian process regression inverse models, which selects individuals in the population with the maximum amount of information to effectively enhance the prediction accuracy of the inverse model.The outline of the proposed algorithm ALGM-MOEA is shown in Figure 1.
In each generation of ALGM-MOEA, the weight vectors representing search direction first divide the population into K subpopulations.Subsequently, environment selection, the establishment of Gaussian process regression models and reproduction are carried out independently in each subproblem.The offspring produced by each subpopulation will be combined with the parents to form combined population to participate in the next generation.After several generations, the proposed ALGM-MOEA dynamically adjusts the weight vector according to the distribution of the population.It can be shown that the key components of ALGM-MOEA, including population-guided weight vector evolution strategy and active learning based Gaussian inverse model, which will be elaborated in the following.

Population-guided weight vector evolution strategy
In order to dynamically modify the weight vectors in MOPs with different shapes of PF, a population-guided weight vector evolution strategy is proposed.During the evolution process, the weight vectors co-evolve with the population, and the weight vectors can be dynamically adjusted by using the evaluation of the population, further guiding the evolution of the solution towards the optimal Pareto front.Specifically, we calculate the fitness of weight vector based on the number and density of individuals within its neighborhood.Individuals with high fitness are defined as effective weight vectors and survived.Weight vectors with little relevance to candidate solutions will be defined as invalid weight vectors and deleted.Thus, the selected weight vectors will be simultaneously improved with the evolution of the population.
The detail of the proposed population-guided weight vector evolution strategy is as follows.Initially, calculate the angles between each individual for each solution i s , we identify its neighborhood weight vectors by a predefined threshold  .Specifically, if the angle between a solution i s and the weight vector j  is less than the threshold  , then the individual is labeled as a neighbor of the weight vector; otherwise, the solution is beyond the neighborhood, and the element ij r in angle matrix R is set as Inf.Second, sort each row of angle matrix R in ascending order, and obtain another sorted matrix ' ' ( ) , and the calculation of Finally, the fitness of one weight vector j w , termed ( ) j fit w , is formulated in Eq (4), where represents the number of individuals within the neighborhood of the weight vector j w .Algorithm 2 presents the detailed implementation steps.In the operation, the weight vectors are updated every few generations.Figure 2 shows the population guided weight vector evolution strategy with a bi-objective minimization problem as an example.Initially, there are six evenly distributed weight vectors, denoted by 1 w , 2 3 4 5 6 , , , , w w w w w , in the objective space.Currently there are four individuals, denoted by 1 2 3 4 , , , s s s s .For each individual, the angles between the individual and all weight vectors are calculated first.Then we use a threshold value  to determine the neighborhood of each individual, and calculate its ranking value by fitness values of the weight vectors within its neighborhood.e.g., the different colored area is the neighborhood of each individual, 5 w is within the neighborhood of 4 s and 3 s , 4 w is within the neighborhood of 2 s and 3 w is also within the neighborhood of 1 s and 2 s .Then, calculate the fitness value of each weight vector.It is observed that 3 w and 5 w are in the neighborhood of two individuals, and gain the best two fitness function values.4 w is in the neighborhood of one individual, and the fitness ranks the third.Finally, several weight vectors with the highest fitness values are selected as effective weight vectors, and the rest of the weight vectors are discarded.

Active learning based Gaussian inverse modeling
The generation of new individuals includes two stages, including the building of Gaussian inverse model and sampling.When evaluating the quality of generated solutions, two key factors need to be considered.One is how to enhance the accuracy of the established inverse Gaussian regression model to reflect the mapping relationship between the objective function and decision variables; the other is how to accomplish highly efficient search to balance the convergence and diversity by sampling in the objective space.To enhance the accuracy of Gaussian model and the quality of the generated individuals, a new individual generation method is developed.
In the proposed algorithm, we use active learning theory to continuously update Gaussian process regression inverse model by selecting the most uncertainty training samples.In the process of reproduction, an adaptive sampling strategy is proposed to dynamically adjust the range of the values of the test sampling points by using the established model.In the following, we will introduce active learning-based model adjustment strategy, followed by reproduction strategy.

Active learning-based model adjustment strategy
The quality of the training sample set directly determines the learning effectiveness of the model, and the information contained in the training samples directly affects the accuracy of the model.In order to enhance the prediction accuracy of Gaussian process regression inverse model, a sample selection method based on active learning with uncertainty is developed to choose test samples with large amount of information.As the training samples are constantly updated, the Gaussian process regression models are also continuously modified.Figure 3 , and establish a Gaussian regression inverse model In order to enhance the prediction accuracy of established model, a new sample selection method based on active learning is developed in the proposed algorithm.We will use the active learning method to select the individuals with the most information in the population as new training samples to constantly revise the Gaussian regression model, and the modified model is denoted by The framework of sample selection method based on active learning with uncertainty is presented in Algorithm 3. The key step in Algorithm 3 is to select some new individuals from the current population as unlabeled individuals in objective space in accordance with the rules of maximum and minimum distance, and use the established Gaussian model to calculate the variance of these

Reproduction to generate new points
When the inverse Gaussian process regression models are obtained, we will sample the input test points * j f within the estimated range of the objective space, and then use the mapping relationship established by models to generate newly candidate solutions * i x as output.
In the proposed ALGM-MOEA, we adjust the range of estimating range dynamically [29] in different stages of the evolution.In the early stage of evolution, the search range is relatively narrow to speed up the convergence, while in the late stage of evolution, the search range will be extended to improve diversity and avoid to finding local optimal.The estimated range is calculated by Eq (5), where a and 2 a are range coefficients set by the decision maker.gen is the current generation, and the threshold for range adjustment 1  is set to 500.

Compared algorithms, test cases and performance metrics
In the experiments, we compare the performance of the proposed ALGM-MOEA with six algorithms, including IM-MOEA [25], RM-MEDA [24], MOEA/D [31], RVEA [30], RPD-NSGAII [32] and PREA [33].IM-MOEA and RM-MEDA are two representative types of estimation of distributionbased algorithms.MOEA/D, RVEA and RPD-NSGAII are three popular decompositions based MOEA, which use weight vectors or adaptive reference points to divide the MOP into a set of subproblems.PREA is a promising-region-based MOEA which integrates the user's preference.
In the simulation, two types widely used test cases are employed with different characteristics of Pareto front.The first type is the modified benchmarks [25] of ZDT and DTLZ cases, named IMMOEA_F1 to IMMOEA_F9.They have various complex features, such as convex or concave with linear variable linkage or non-linear variable linkage.The two control parameters 5   and 3   , and the number of decision variable is 30.The second type test cases contain various irregular shape of PFs, which have degenerated PF, disconnected PF and convex multimodal PF.

Parameter settings
Table 1.Parameter settings of the compared algorithms.

Algorithms
Parameter settings IM-MOEA the number of reference vectors 30 The parameters for the compared algorithms are configured to the recommended values in their original publications, which are shown in Table 1.In the simulation, the population size is set to 300 for all algorithms, and the termination condition is set to 300,000 fitness evaluations.For the proposed ALGM-MOEA and IM-MOEA, the number of reference vectors and the group size are set to 30 and 3.In the proposed ALGM-MOEA, the range coefficient 1 a is set to 0.2 and 2 a is set to 0.5.The threshold of angle  is set to 18  .The statistical results of IGD and HV of compared seven algorithms are displayed in Tables 2 and 3, and 20 independent runs for each algorithm on each test case are executed.In the table, the mean and the standard deviation for each algorithm are displayed in the first and second lines, respectively, and the best results are highlighted.The Wilcoxon rank sum test at the significance level of 0.05 is used to pairwise compare the proposed ALGM-MOEA and other algorithm in each test case.Symbol '+', '-' and '=' in front of the results indicate that the proposed ALGM-MOEA is superior to, inferior to or not significantly different from the compared algorithm.Some observations in Table 2 can be concluded below.

Performance comparison on IMMOEA_F1 to IMMOEA_F9
1) The proposed ALGM-MOEA performs the best on IMMOEA_F1, IMMOEA_F2, IMMOEA_F5 and IMMOEA_F6, while IM-MOEA obtains the best mean value of IGD on IMMOEA_F9.Besides, promising-region-based PREA which incorporates user preferences obtains the best IGD mean on test questions IMMOEA_F3 and IMMOEA_F7.
2) From the results, we can also observe that the performance of three distribution estimation algorithms, ALGM-MOEA, IM-MOEA and RM-MEDA are significantly outperforming that of the rest of the four algorithms on test cases IMMOEA_F1, IMMOEA_F2 and IMMOEA_F4.The reason is that for the adopted test cases IMMOEA_F1-F9 with correlation between decision variables, the probabilistic models built by estimation distribution algorithms is very effective for generating offspring in the preference area by sampling, and maintains better convergence and diversity.
3) It can also be seen that the IGD mean value of ALGM-MOEA is superior to that of IM-MOEA, especially in problems IMMOEA_F5, F6 and F8, where the relationship between decision variables is non-linear.This verifies that the proposed active learning-based training sample selection method can improve the prediction accuracy of Gaussian model and enhance the quality of solutions.
Table 3 summarizes the results of HV of seven algorithms.The best results on IMMOEA_F2, IMMOEA_F4, IMMOEA_F5 and IMMOEA_F6 are performed by ALGM-MOEA, while IM-MOEA wins on two test cases, IMMOEA_F3 and IMMOEA_F9.The decomposition-based algorithm MOEA/D obtained the best HV values on IMMOEA_F7 and IMMOEA_F8.The non-dominated solutions with the best IGD values obtained by the ALGM-MOEA and the state-of-art compared algorithms on IMMOEA_F1 and IMMOEA_F6 are displayed in Figures 5 and 6.On IMMOEA_F1 with convex shape of Pareto front, the three algorithms based on estimation distribution (ALGM-MOEA, IM-MOEA and RM-MEDA) achieve good convergence and distribution.The non-dominated solution sets obtained by the remaining four algorithms only cover part of the PF, and the distribution is not good.On IMMOEA_F6 with concave shape of Pareto front, the solutions achieved by ALGM-MOEA show the best convergence and diversity, and IM-MOEA and PREA obtain the second-best performance.The quality of the solutions obtained by the remaining algorithms cannot guarantee convergence and diversity.In test instance IMMOEA_F6, the correlation relationship between the decision variables is nonlinear.The sample selection method based on active learning in the proposed algorithm can more effectively get information between the decision variables and the objective function, and obtain offspring with high quality by Gaussian inverse model.4. 1.2805e-1(1.00e-3)-1.9243e-1(1.18e-3) MaF7 1.5238e+0(6.44e-2)+2.2723e-1 (7.43e-3) +/=/-3/0/2 The statistical results of IGD and HV of IM-MOEA and ALGM-MOEA are shown in Tables 5  and 6.The Wilcoxon rank sum test at the significance level of 0.05 is used to pairwise compare the proposed ALGM-MOEA and other algorithm in each test case.Symbol '+' '-' and '=' in front of the results indicate that the proposed ALGM-MOEA is superior to, inferior to or not significantly different from IM-MOEA.It can be shown that ALGM-MOEA outperforms IM-MOEA in problems DTLZ7 and MaF7 with disconnected PFs, MaF5 with convex multimodal PF, since the population-guided weight vector evolution strategy can dynamically adjust search direction by learning the distribution of the population, and delete invalid weight vectors in the region where no solution locates.Therefore, the convergence and diversity are enhanced compared with IM-MOEA.In addition, ALGM-MOEA behaves worse than IM-MOEA on DTLZ5 with degenerated PF.This may be due to the fact that the new sample point selection strategy proposed in ALGM-MOEA pays more attention on individuals with uncertainty, and this will lead to a certain deviation in the establishment of the inverse model, thus affecting the distribution of the offspring.

Conclusions
This work introduces an active learning Gauss modeling based multi-objective evolutionary algorithm using population-guided weight vector evolution strategy (ALGM-MOEA).A new population-guided weight vector evolution strategy is proposed to alter the weight vectors with the knowledge of the current non-dominant solution set.Besides, we develop a Gaussian regression modeling based on active learning to generate new individuals, and an adaptive searching strategy to explore the unexplored sparse area in objective space and improve the diversity.The proposed ALGM-MOEA exhibits its competitiveness on search ability for test cases with linear or non-linear correlations between the decision variables.Moreover, the performance of ALGM-MOEA and IM-MOEA for test cases with irregular PF is compared, and it can be observed that the proposed ALGM-MOEA is superior to IM-MOEA, which verifies the effectiveness of the proposed ALGM-MOEA.
In the future, we will study how to build a more efficient new model to reflect the mapping between variables and objective functions, thus further reducing computational complexity.

Use of AI tools declaration
The authors declare that they have not used Artificial Intelligence (AI) tools in the creation of this article.

Algorithm 2 .
The pseudo-code of Population-guided Weight Vector Evolution Strategy Input: Candidate solutions set S , weight vector set W ; threshold of angle  ; Output: The effective output weight vector set output W 01 Calculate the angle matrix R 02 Construct the ranking matrix ' R 03 for each weight vectors j W vectors with the highest fitness rank are selected as the effective weight vectors.

Figure 2 .
Figure 2. Illustration of population guided weight vector evolution strategy with a biobjective minimization example.
presents the schematic diagram of the idea of active learning-based model adjustment strategy.

Figure 3 .
Figure 3.The schematic diagram of the idea of active learning-based model adjustment strategy.

.
individuals.Lines 3-6 calculate the Euclidean distance between any two individuals in the alternative query test points set AQ and the current training sample set CS , and lines 7-12 fulfill the goal of selecting new individuals by maximum and minimum distance.Line 13 calculates the variance of each candidate set individuals in sel Index x The individuals with the largest variance will be selected as the new query samples and added to the training sample set.Since the larger the variance is, the bigger the possibility of the test sample which cannot be represented by the model obtained by the current training samples.Therefore, we can add above samples with the maximal variance to the training set as the most informative sample, and train the modified Gaussian regression inverse model by new training samples.

Figure 4 .
Figure 4.The example of active learning-based training sample selection process.
and lower bounds of population in the j th 

Figure 5 .
Figure 5. Non-dominated solutions with the best IGD values obtained by each algorithm in objective space on IMMOEA_F1.

Table 2 .
The average of IGD for compared algorithms.

Table 3 .
The average of HV for compared algorithms.

Table 4 .
The properties of test cases with irregular PFs.

Table 5 .
The statistical results of IGD on test cases with irregular PFs.

Table 6 .
The statistical results of HV on test cases with irregular PFs.