Numerical results for the multiobjective trust region algorithm MHT

In this data article, we report data and numerical results related to the research article entitled ”A trust region algorithm for heterogeneous multiobjective optimization” by Thomann and Eichfelder in SIAM Journal on Optimization. The method MHT which is presented there is designed for multiobjective heterogeneous optimization problems where one of the objective functions is an expensive black-box function, for example given by a time-consuming simulation. Here, we present the data of numerical tests with a set of 78 test problems mainly collected from literature and only complemented by few self-chosen test problems. The presence of expensive functions is artificially introduced in the test problems by defining one of the objective functions as expensive.


Data
We report the data related to the results of the numerical tests of the algorithm MHT presented in Ref. [13]. All computations have been run on an Intel(R) Core(TM) i3-2105 CPU 3.10GHz with Matlab 2017a.
All considered test problems are listed in Table 1 with references, number of objective functions and further information. The output data of MHT is presented structured by convex, nonconvex and scalable test problems. The output data of MHT presented includes starting points, the points that are given as output of MHT and required function evaluations. For some instances also tables with a comparison of used function evaluations are given.
The presented figures illustrate the behavior of the algorithm and show, besides the starting point, all iteration points and all further points that are required during the algorithm. Furthermore, there are also figures which show performance profiles of MHT and two other algorithms, DMS [1]and EFOS [12].
In subsection 2.1 an overview of the test problems used for the algorithm in Ref. [13] is presented followed by numerical results in subsection 2.2.

Test problems
We consider the general optimization problem min x2U f ðxÞ (MOP) with U⊆R n and f ðxÞ ¼ ðf 1 ðxÞ; …; f q ðxÞÞ u . The objective functions f i : R n /R are assumed to be twice continuously differentiable for all i ¼ 1; 2; …; q and max i¼1;…;q f i ðxÞ is assumed to be bounded from below. The constraint set U contains either box constraints or is the whole domain. In MHT, heterogeneous optimization problems with one expensive function are considered.
The following set of 78 test problems is based on test problems from the literature for general multiobjective and derivative-free algorithms [1e3,5e11, 14,15] and completed with some self-chosen problems. The latter are listed in subsection 3. All considered problems are test problems and do not involve an actual expensive function. For these problems the efficient points can be computed which is necessary to compare the results of the algorithm to the actual efficient solutions. For evaluating the results one of the functions is declared as expensive and the amount of function evaluations for this function is counted. Output data of the optimization algorithm MHT from Ref. [13] Data format Raw, filtered, summarized, analyzed Experimental factors Output data of the optimization algorithm MHT was stored, these raw data were filtered and summarized for the presentation in this paper Experimental features List of test problems Presentation and illustration of results of test runs with algorithm MHT from Ref. [13] and comparison to two other algorithms Data source location Ilmenau, Germany, Technische Universit€ at Ilmenau Data accessibility Only in this article Related research articles J. Thomann and G. Eichfelder. A trust region algorithm for heterogeneous multiobjective optimization. SIAM J. on Optim., 29(2), 1017e1047, 2019 [13] Value of the data Output data reported represents a benchmark for future comparisons, among algorithms for heterogeneous multiobjective optimization problems Output data illustrates how a trust region idea can be used in the context of heterogeneous optimization problems Output data reported illustrates the behavior of the trust region method MHT and its theoretical properties Output data illuminates similarities and differences of MHT, the direct search approach DMS and the trust region method EFOS Among the test examples are quadratic and nonquadratic functions, convex and nonconvex problems, either unconstrained or with box constraints. Table 1 shows an overview of all 78 considered test problems with information about the dimension of the domain (n), the constraints and the convexity of the problem. It also includes information about the geometry of the Pareto front, the set of all nondominated points (convex, nonconvex, disconnected). We say the Pareto front PF is convex (nonconvex) if the set PF þ R 2 þ is convex (nonconvex). Besides, the table contains in its last column which of the objective functions is declared as expensive for the test runs of MHT. If there are significant differences regarding the difficulty of the functions, the more difficult function is declared as expensive.

Numerical results
The multiobjective trust region algorithm MHT as presented in Ref. [13] has been implemented in MATLAB (version 2017a) and tested for the multiobjective problems with two or three objective functions from subsection 2.1. We used the realization of the trust region update as described in Section 5.2 in [13] with the parameters h 1 ¼ 0:001; h 2 ¼ 0:9. The stopping criterion is implemented according to Section 5.1 in [13]. It uses a maximum number of function evaluations given by the user, the size of the trust region and a necessary condition for local weak efficiency. Furthermore, model information for the as expensive declared function is reused as often as possible to save function evaluations, see Section 5.4 in [13].
We compared MHT with two other methods. On the one hand, since MHT computes only one point fulfilling a necessary optimality criterion and does not approximate the set of efficient points, we used the weighted sum approach with equal weights and apply EFOS (Expensive Function Optimization Solver) [12] to it with the predefined standard parameters. It is a solution method for expensive, simulation-based scalar optimization problems also using the trust region approach. As a stopping criterion a criticality measure using the gradients of the model functions is applied in conjunction with a validity criterion for the models. For convex multiobjective optimization problems every efficient point can be computed by a weighted sum of the objectives with suitable weights. For nonconvex problems only a subset of the efficient points can be computed. This needs to be regarded when comparing the results.
On the other hand, and to circumvent the disadvantages of the weighted sum approach, the multiobjective method DMS [1] is used as a comparative method. It is a direct search approach and therefore derivative-free and suitable for expensive functions. It approximates the whole set of efficient points, but offers also the option to compute only one efficient point. We used the latter option with the predefined standard parameters varying the maximum number of function evaluations. As a stopping criterion DMS uses a maximum number of function evaluations given by the user and the step size for the search step. If the step size is lower than the predefined value ð10 À3 Þ, DMS stops. Furthermore, DMS includes a method to compute starting points on its own which is chosen by default. To use the starting point the user passes as input to the algorithm, the parameter 'list' in the parameter file needs to be changed from the predefined value 3 to 0.
Of course also the way of the implementation of the algorithms influence the performance for the test problems. In the currently available implementation of EFOS often internal errors occur and runs are terminated without having computed a solution.
As a main comparison criterion we use the number of function evaluations until the considered method terminates and set the maximum number of allowed function evaluations for all algorithms to 2000. Firstly, we present some selected test instances to illustrate the procedure of MHT and to compare the results to the methods DMS and EFOS. In the end of this section, we present performance profiles for all considered test instances.

Convex test problems
At first we consider the quadratic, convex test problem (BK1) from Refs. [1,7] given by . The domain resp. image set is represented by scattered gray points, the starting point is marked black and the solution is marked orange. For MHT the iteration points are marked black and connected by a dotted line, the interpolation points that are evaluated to compute the model functions are marked as unfilled circles. In the domain the trust regions are depicted as gray shaded, transparent circles (the more areas overlap, the darker the gray shade). For EFOS and DMS it is not possible to distinguish between EFOS computes more points than MHT and they are more spread both over the image space and the domain. During the run even infeasible points are generated. This can be seen only in the domain (middle left) since for illustrative reasons we used the same range in the image space for all figures on the right.
DMS also computes more points than MHT, but they are not spread over the domain as for EFOS, but accumulate in a local area. Apart from this, the bottom left figure illustrates the search along the coordinate directions. In the last iterations the step size decreases. In one run of MHT one Pareto critical point is computed. In general, different starting points generate different Pareto critical points due to the search strategy. A multistart approach with randomly generated starting points is illustrated in Fig. 2. The starting points are marked as unfilled circles and the obtained points are marked black. Fig. 2 shows that MHT (top left) and DMS (bottom) compute different nondominated points, whereas EFOS (top right) generates only one nondominated point. A reason for this is the weighted sum approach. Furthermore, not all resulting points from DMS are efficient points, some have still a large distance to the Pareto front. As second test example we consider the convex, but not quadratic problem (T6) given by For all test instances of this optimization problem EFOS is prematurely canceled due to an internal error. Consequently, we only show results for DMS and MHT. The as expensive declared function f 1 is not quadratic and therefore the interpolation model used in MHT is not exact. Though, the algorithm reuses old model information as often as possible which is illustrated for one specific instance, see the top left (domain) and top right image (image space) in Fig. 3  Another convex problem is (T7), but with a three-dimensional domain, given by and with f 1 declared as expensive function. The unique efficient point for this optimization problem is x ¼ ð0; 0; 0Þ u with the function values f 1 ðxÞ ¼ f 2 ðxÞ ¼ 0. For all considered starting points all three algorithms compute this unique nondominated point respectively a point with vanishing distance to it. This is shown in Fig. 5 for one instance. The range of function evaluations for all instances is similar for MHT (29e57) and EFOS (9e75). Compared to DMS with 206e348 evaluations, both MHT and EFOS save function evaluations. The reason for this large difference is the direct search approach of DMS which produces, as depicted in Fig. 5 for one instance, dense evaluated points in the image space. Again, EFOS computes also infeasible points during the runs which is also the case for the instance depicted in Fig. 5.

Nonconvex test problems
MHT is a local method and as proved in Ref. [13], the accumulation points of the generated sequence satisfy, in case several assumptions are fulfilled, a necessary condition for local weak efficiency. For convex problems local and global optimality is identical. For nonconvex problems this is in general not the case. In the following, we report on how the algorithm performs for nonconvex problems and exemplarily show the results of three nonconvex test problems.
The first one is (Deb513) from Refs. [1,2] defined by with gðxÞ ¼ 1 þ 10x 2 , hðxÞ ¼ 1 À ðx 1 =gðxÞÞ 2 À ðx 1 =gðxÞÞsinð8px 1 Þ and a disconnected Pareto front. For this test problem f 2 is declared as expensive function. All three algorithms are capable of computing a nondominated point for all instances of this test problem as Fig. 6 shows. Furthermore, it shows that again MHT (top left) and DMS (bottom) generate several nondominated points whereas EFOS (top right) computes only one nondominated point. The different search strategies are recognizable in Fig. 7 which shows the result of one run with the same starting point for all three algorithms. EFOS (top right) computes the individual minimum of function f 1 within 7 function evaluations for all starting points. This is due to the weighted sum As second noncovex test problem we consider (Jin2) with n ¼ 4 from Refs. [1,9] defined by x i and f 2 declared as expensive. EFOS could not compute an efficient solution for this test problem, but stopped with an internal error for the considered starting points. DMS and MHT compute nondominated points for all instances of this problem. The multistart approach with randomly chosen starting points in Fig. 8 shows that both compute different nondominated points given different starting points. However, the points computed by DMS are better spread than the points computed by MHT. Regarding the required function evaluations no clear statement can be made which algorithm needs less (DMS 92e126, MHT 30e169).
In all runs MHT needs many function evaluations in the end of the procedure. Due to the nonconvexity and the local search strategy this number of function evaluations is needed to ensure the stopping criterion being fulfilled. This is exemplarily shown for one specific run in Fig. 9. This instance again illustrates the coordinate search of DMS.
As last nonconvex test problem we consider (FF) with n ¼ 3 from Ref. [5] given by with f 1 declared as expensive function. This test problem illustrates that Pareto criticality is only a necessary condition for local efficiency and that MHT does not necessarily generate an efficient point if it stops legitimately according to the stopping criterion. Fig. 10 shows the result of MHT for one test instance of (FF). The algorithm terminates with the point x ¼ ðÀ0:0604; À1:2138; À0:8433Þ u with the function values f 1 ðxÞ ¼ 0:9964 and f 2 ðxÞ ¼ 0:5234. The stopping criterion indicates that x is Pareto critical, since the step size t of the auxiliary Pascelotti-Serafini problem that is used in MHT is small enough(t ¼ 3:319810 À7 ). For further details see Section 3.2 and Section 5.1 in [13].
Additionally, another criterion using an auxiliary function denoted by u and described later in subsection 2.2.4 confirms that x is a Pareto critical point (uðxÞ ¼ 8:260410 À3 ). Though, this point is not an efficient point as the illustration in the image space in Fig. 10 shows.

Scalable test problems
We considered test problems with different dimensions for the domain and did also runs for MHT with two scalable test problems up to dimension 50. As expected the numerical effort rises when the dimension rises. The higher the dimension is, the more function evaluations are required to build up a model. If in every iteration a complete new set of interpolation points would be computed, a total number of ðn þ 1Þðn þ 2Þ =2 function evaluations would be needed for the expensive function in every iteration. However, in MHT the model is not updated in every iteration, but only if necessary and if it is  Furthermore, we suggest in Ref. [13] to use for higher dimensions a linear interpolation model also based on Lagrange polynomials. This needs only n þ 1 interpolation points and therefore the number of function evaluations also reduces. Though, a linear model needs to be updated more often since it is less accurate. Table 2 gives an overview of how many function evaluations are required to compute one model function (quadratic/linear interpolation with Lagrange polynomials) depending on the dimension of the domain.   In the numerical tests we used the linear models for all instances with dimension n ! 10. To illustrate the behavior of MHT with rising dimension we consider the scalable test problem (Jin1) from Refs. [1,9] defined by with f 1 declared as expensive. It has been tested with MHT for n2f2; 3; 4; 5; 10; 20; 30; 40; 50g. Fig. 11 shows runs with MHT for n ¼ 5 on the left and n ¼ 10 on the right. For n ¼ 10 a linear model is used which as expected worsens the predictions of the model functions. Since the trial point acceptance test in MHT does not demand a strict decrease in every component, but is instead a weaker formulation, also points that increase one of the objective functions can be accepted. Fig. 12 shows the results for DMS applied to (Jin1) with n ¼ 5 and n ¼ 10 with the same starting points as used in the runs depicted in Fig. 11. MHT needs 42 function evaluations for n ¼ 5 (21 for n ¼ 10) and therefore significantly less than DMS which needs 78 evaluations (152 for n ¼ 10). . Even though DMS computes many function values, it explores only the area close to the starting point and terminates with a point close to the starting point.
In the following we give a short overview of the number of function evaluations required by MHT and DMS for the scalable test examples (Jin1) and T4, see Test Problem 4 in section 3. Table 3 gives an overview of the range (R) and mean value (M) of used function evaluations for MHT.  As already seen in the instance of (Jin1) presented above, the ranges in Table 3 show that there are instances for which a higher dimension required less function evaluations than a lower dimension, e.g. dimension 5 and 10. This is due to the choice of starting points and due to the different kinds of model functions. From dimension 10 onwards linear model functions are used for the as expensive declared objective function. Along with the choice of a starting point and the local search strategy this can cause a lower total number of function evaluations for single instances. However, in general, the tendency of rising function evaluations with rising dimension is apparent.
The comparison method DMS is a direct search approach from Ref. [1]. For the general direct search approach 2n function evaluations are needed in every iteration. Table 4 gives an overview of this number up to dimension 50.
For DMS as it is implemented and available, Table 5 shows the range (R) and mean value (M) of function evaluations needed for the two scalable test problems. We set the maximum number of allowed function evaluations to 2000. This is not enough for some instances of T4 as Table 5 shows. For these instances DMS terminated with the maximum number of function evaluations reached without having computed an efficient point. Tables 3 and 5 give a first impression of how the dimension n influences MHT in comparison to DMS. As expected, the direct search approach needs significantly more function evaluations with rising dimension than MHT. The mean values of function evaluations per dimension for all instances of the scalable test problems FF, Jin1, Jin2, Jin3, Jin4, T4 (see Table 1 in subsection 2.1) are listed in Table 6.
This indicates that, compared to the direct search approach of DMS, the algorithm introduced in Ref. [13] can also save function evaluations in higher dimensions. However, it is important to note that DMS does not make use of any derivative information, also not of the cheap function. Therefore, it must  Table 4 Function evaluations per iteration for general direct search.   be expected that DMS needs more function evaluations than any method that does use such information.

Performance profiles
For classifying the test runs as successful or not successful the distance to the Pareto front, the set of nondominated points, is used. If it falls below a problem dependent constant, the test run for an instance is classified as solved. To compare the performance of the algorithms, the number of function evaluations for the as expensive declared function is counted until the algorithm terminates. Fig. 13 Figure 2 in [13] shows a performance profile for all 348 convex test instances in full range on the left and zoomed in on the right.
If up to 480 function evaluations are allowed for the expensive function, DMS and EFOS behave similar. With further function evaluations DMS is capable of solving further test instances whereas EFOS stagnates and cannot solve more instances.
In general, Fig. 13 shows that MHT needs less function evaluations than EFOS and DMS to solve the convex test problems. It solves all 348 convex test instances within at most 1459 expensive function evaluations. This high number is due to the high dimensional test instances included. If considering only test instances up to dimension 10 all convex instances are solved by MHT after 667 expensive function evaluations as the performance profile in Fig. 14    As already noted, MHT is a local method and generates a sequence of iteration points. The accumulation points of this sequence fulfill, in case several assumptions are fulfilled, a necessary condition for local weak efficiency (Pareto criticality). The fact that Pareto criticality is a necessary condition for local optimality must be regarded when considering nonconvex problems since local optimality is in general not synonymous to global optimality. If the algorithm stops legitimately in a Pareto critical point, the distance to the Pareto front does not need to converge to zero. Thus, for a performance profile over all considered test examples we do not only use this distance to classify test instances as solved, but complement it with a measure for Pareto criticality. The auxiliary function u : R n /R defined by uðxÞ : ¼ À min kdk 1 max i¼1;…;q V x f i ðxÞ u d for f i : R n /R continuously differentiable functions, i ¼ 1; …; q, characterizes Pareto criticality. According to Ref. [4] u is a continuous function, it holds uðxÞ ! 0 for all x2R n and a point x2 R n is Pareto critical for (MOP) if and only if it holds uðxÞ ¼ 0.
Consequently, given x the solution generated by one of the considered algorithms (MHT, DMS, EFOS), we classify a test instance as solved if either the distance of f ðxÞ to the Pareto front is small enough or if it holds uðxÞ ε. We chose ε ¼ 0:1 for the data analysis.
Using these classifications, the performance profile in Fig. 15 shows how many of all 802 considered test instances are solved depending on the required function evaluations for MHT, DMS and EFOS. The full range is shown on the left and on the right the performance profile is zoomed in to 500 function evaluations. Fig. 15 illustrates that by applying MHT 98.13% of all test instances are solved with either an efficient or a Pareto critical point. Within the same number of function evaluations (1459) EFOS solved 57.98% and DMS solved 84.04% of all considered test instances. Thus, MHT solved more test problems than both comparison methods and needs less function evaluations. Although the behavior of DMS is similar to our method, still MHT saves computation time and solves more instances in terms of distance to the Pareto front or Pareto criticality.