Visualizations for decision support in scenario-based multiobjective optimization

We address challenges of decision problems when managers need to optimize several con-ﬂicting objectives simultaneously under uncertainty. We propose visualization tools to support the solution of such scenario-based multiobjective optimization problems. Suitable graphical visualizations are necessary to support managers in understanding, evaluating, and comparing the performances of management decisions according to all objectives in all plausible scenarios. To date, no appropriate visualization has been suggested. This paper ﬁlls this gap by proposing two visualization methods: a novel extension of empirical attainment functions for scenarios and an adapted version of heatmaps. They help a decision-maker in gaining insight into realizations of trade-offs and comparisons between objective functions in different scenarios. Some fundamental questions that a decision-maker may wish to answer with the help of visualizations are also identiﬁed. Several examples are utilized to illustrate how the proposed visualizations support a decision-maker in evaluating and comparing solutions to be able to make a robust decision by answering the questions. Finally, we validate the usefulness of the proposed visualizations in a real-world problem with a real decision-maker. We conclude with guidelines regarding which of the proposed visualizations are best suited for different problem classes. (cid:1) 2021 The Authors. Published by Elsevier Inc. ThisisanopenaccessarticleundertheCCBY license (http://creativecommons.org/licenses/by/4.0/).


Introduction
In many real-world optimization problems, decision-or policy-makers are faced with several (typically conflicting) criteria or objectives that should be optimized simultaneously.When satisfying all the conflicting objectives is not possible, decision support tools may help in finding a satisfactory balance between them.Because of the conflicting nature of the objectives, we do not usually have a single optimal solution but several so-called Pareto optimal ones with different trade-offs.Over the years, many multiobjective optimization methods have been designed to help and support the decision-maker (DM) to find their most preferred Pareto optimal solution.
Besides multiple conflicting objectives, real-life problems are characterized by uncertainty.It is desirable to make robust decisions that are not too sensitive to the consequences of uncertainty, i.e., they perform relatively well in a wide range of future states or events [25], like in long-term strategic planning problems.The term scenario is used in the literature for various purposes, but usually, it refers to a framework to capture uncertainty in the absence of reliable probability distributions.
For example, a case study of the 2015 Nepal earthquake response considered seven scenarios-based on different uncertain parameters such as demands and various types of transportation [2].Scenario-based multiobjective optimization approaches can be powerful tools in supporting decision-making processes in practice.
When we have a multiobjective optimization problem to be solved in the presence of uncertainty, we seek to find solutions that are sufficiently good in all scenarios from the DM's perspective.There are various terms like satisfactory or acceptable that one may use to describe such solutions.However, for consistency henceforth, we will utilize the term satisfactory, referring to this concept.Recently, authors from different disciplines have studied multiple scenarios in multiobjective optimization (although they may have utilized the term scenario in different ways) in real applications such as design problems [11], energy [48], portfolio optimization [13], maritime disruption management [8], berth allocation scheduling [52], location-allocation models for disaster response [2], bridge management [34], and water resources planning and management [12,50].Scenarios provide a way to consider uncertainty and prepare oneself for different futures.Besides, even if scenarios represent ''different loading or operating conditions" and not uncertainty, as described in [11], solutions must have satisfactory performances in all scenarios.
In scenario-based multiobjective optimization problems, the number of scenarios is finite and, possibly, small.Scenariobased multiple criteria decision analysis problems often consider 4-6 scenarios [42] or some tens of scenarios at most [38].However, the number of scenarios in some real-world problems may increase to a few hundred (e.g., [40]).
In multiobjective optimization without scenarios, suitable graphical visualizations are often used to understand and compare different Pareto optimal solutions in the objective space, thus helping a DM to observe, learn and evaluate trade-offs between objectives and eventually choose one's most preferred solution [29,31].In scenario-based multiobjective optimization problems, the performance of a decision should be evaluated regarding each objective under conditions of different scenarios [40,42].This way of scenario consideration brings an additional dimension to the performance evaluation and complicates the task of the DM.Therefore, our goal is to help a DM understand and compare trade-offs between different objective functions and evaluate and analyze trade-offs between the performances of a solution in various scenarios (we refer to it as trade-offs between scenarios hereinafter).
As one of this paper's contributions, we first enumerate possible questions that a DM may ask when facing a scenariobased multiobjective optimization problem.Existing visualization methods for multiobjective optimization fail to capture trade-offs between scenarios.Thus, we propose specific visualization methods for different cases of scenario-based multiobjective optimization, which are the main contribution of our study.In particular, we adapt ideas from the visualization of empirical attainment functions and heatmaps and propose three visualizations based on adapted empirical attainment functions.We also report encouraging feedback from a real DM on the usefulness of the proposed visualizations in a real-world production problem.Finally, we provide recommendations for applying different visualizations depending on the number of objectives, scenarios, and solutions considered.
This paper is organized as follows.After a brief description of scenario-based multiobjective optimization problems and some visualization approaches in Section 2, we formulate in Section 3 examples of questions that a DM may wish to be answered with the help of visualizations when dealing with scenario-based multiobjective optimization problems.In Section 4, we propose appropriate visualization methods for various needs followed by different examples in Section 5.In Section 6, we collect feedback from a real DM on the usefulness of the proposed visualizations in a real-life problem.Finally, after further discussion and suggesting guidelines for applications in Section 7, we conclude and mention future research directions in Section 8.

General concepts in multiobjective optimization
Before concentrating on scenario-based problems, let us consider the following deterministic multiobjective optimization problem (following, e.g., [30]): minimize ff 1 ðxÞ; . . .; f k ðxÞg subject to x 2 S; ð1Þ where k P 2 is the number of objective functions f i : S !R (i ¼ 1; . . .; k), x ¼ ðx 1 ; . . .; x n Þ T is a vector of n decision variables in the feasible region S in the decision space R n (S # R n ) defined by constraint functions, and z ¼ fðxÞ = ðf 1 ðxÞ; . . .; f k ðxÞÞ T is called an objective vector in the objective space R k .A decision vector x Ã 2 S is called Pareto optimal if there does not exist another x 2 S such that for all i, f i ðxÞ 6 f i ðx Ã Þand f j ðxÞ < f j ðx Ã Þ for at least one index j.Furthermore, for any two objective vectors z 0 ; z00 2 R k , we say that z 0 weakly dominates z00 if and only if for all i; z 0 i 6 z i 00 and dominates if also z 0 j < z j 00 for at least one index j.If z 0 and z00 do not dominate each other, they are called mutually nondominated.The image of the set of Pareto optimal decision vectors in the objective space is sometimes called a Pareto front.In what follows, we refer to objective vectors as solutions, as our focus is on the objective space.

Scenario-based multiobjective optimization
As mentioned earlier, a scenario-based multiobjective optimization problem involves multiple conflicting objectives under uncertainty, in which scenarios (representing plausible future states) are utilized to deal with the existing uncertainties.In this paper, we assume that no reliable probabilities related to scenarios are available (called deep uncertainty) [40].
Let us assume that each scenario contains the same number of objective functions with the same meaning, and they are all to be minimized.We denote the number of plausible scenarios defining the scenario set X by s (and the number of objective functions by k).Given x 2 S # R n , its image in the objective space with a set of s scenarios is given by Y ¼ ðy 1 ; . . .; y s Þ, where y p ¼ ðf 1p ðxÞ; . . .; f kp ðxÞÞ 2 R k (p 2 f1; . . .; sg) is the image of x under the conditions of scenario p and f ip ; i ¼ 1; . . .; k, are objective functions in scenario p.
A decision vector x Ã is called Pareto optimal with respect to the scenario set X, if there does not exist any x 2 S such that for all i; p; f ip ðxÞ 6 f ip ðx Ã Þ (i.e., weakly dominated in all scenarios) and f jq ðxÞ < f jq ðx Ã Þ for at least one objective j in at least one scenario q (i.e., dominated in one of them).In other words, by solving scenario-based multiobjective optimization problems, we mean finding a decision vector that is feasible in all scenarios, and there does not exist another feasible decision vector with better values in one objective function in one scenario without losing in another objective function in any scenario.Note that, even in theory, it is hardly possible to find a decision vector that is Pareto optimal in every scenario (i.e., the image of this decision vector lies on the Pareto front in all scenarios), except in a particular class of problems [21].Therefore, here we assume it is sufficient that the decision vector is Pareto optimal for at least one scenario; i.e., the image of the decision vector lies on the Pareto front in the objective space of at least one scenario.Accordingly, in this paper, a solution to a scenario-based multiobjective optimization problem refers to a set of images of a Pareto optimal decision vector in the objective spaces of all scenarios.The task of a DM is to compare trade-offs between objectives in different scenarios and choose the most preferred one based on his/her preferences.However, as our focus in this paper is on developing visualization methods, we do not concern ourselves from where the solutions come from.Different types of preferences can be found in the multiobjective optimization literature.An example of that is a reference point consisting of aspiration levels that represent objective values that are desirable for a DM.
With a best-case scenario(s), we refer to the scenario(s) in which the best possible values for at least one objective function can be reached.Conversely, the scenario(s) in which the worst possible values, for at least one objective, can be achieved is (are) called the worst-case scenario(s).

Multiobjective visualizations
There are many ways to visualize high-dimensional data [26].In the context of multiobjective optimization without scenarios, there are many visualization methods proposed in the literature [29,31].For example, glyph plots 1 and parallel coordinate plots 2 have been used for visualizing trade-offs between objectives in problems with more than three objectives.In addition, dimensionality reduction techniques map the objectives into lower dimensions for visualization purposes (see, e.g.[6,14,35,44]).However, some relevant information, including dominance relationships between solutions, will be lost in such a mapping [10].
Other methods are used for specific purposes besides supporting a DM.For example, in multiobjective evolutionary algorithms (MOEAs), visualizations can be utilized to approximate the range, location and the shape of the set of Pareto optimal vectors in the objective space, or to monitor the progress and convergence of optimization [24].Two critical reviews of visualization in MOEAs are given in [46,47].
In principle, the above visualizations could be used for scenario-based multiobjective optimization problems by plotting each scenario separately.[50] utilized both glyph and parallel coordinate plots to show that optimizing under varying conditions of uncertainty (different scenarios) yields different trade-offs.They performed five optimizations, one for each scenario, and generated a Pareto front for each scenario.Then, they displayed all the solutions in a glyph plot to compare the trade-offs between solutions in various scenarios.These trade-offs, together with decision variables, were also shown in a parallel coordinate plot.In addition, they compared the performances of all five solution sets under conditions of two scenarios separately.However, even for few scenarios, utilizing a series of separate plots in a scenario-based multiobjective optimization problem becomes challenging for a DM since there is much information to digest and combine.
In environmental modeling (e.g., water resource planning and management), multidimensional visualizations like glyphs, parallel coordinate plots, and scatter plot matrices have been used for modeling and/or scenario analysis [12,22,23,50] in which the objective values of generated solutions, obtained from different models or scenarios, are evaluated and compared to find a suitable model and/or robust solutions.However, these visualizations do not illustrate trade-offs of a solution in different scenarios in a way that is easy to comprehend since they visualize individual solutions for different scenarios.
Further uses of visualizations in multiobjective optimization in a scenario-based context (although not particularly suited for simultaneous comparison of many solutions) include scatter plot matrices for visualizing a few solutions of biobjective 1 In general, any symbol utilized in the visualization of data to describe a characteristic or an attribute is called a glyph [49].Glyphs can simultaneously represent multiple attributes of multivariate data [31]. 2Parallel coordinate plots can show many dimensions at once [15,51].problems with a few scenarios in theoretical studies in robust multiobjective optimization [7,21].However, for problems with more objectives and/or more solutions, such visualizations get messy and hard to read, and then, extracting information (like trade-offs) and getting insights are extremely difficult, if even possible.Separate scatter plots were also used in [11] for different scenarios to compare the results in bi-scenario, biobjective design problems.In addition, a scatter plot matrix (also called a pairwise scatter plot) was utilized in some environmental applications [12].However, separate scatter plots and a scatter plot matrix are only useful for problems with few scenarios and few solutions.They are not suitable for other scenario-based multiobjective optimization problems.A real DM also brings this issue up in our real-world evaluation (discussed in Section 6).
In summary, scatter plot matrices, parallel coordinate plots, and glyph plots are simple to use and understand.They may be applied to each scenario separately if the number of scenarios is small, and it becomes the task of a DM to compare different plots to understand trade-offs between scenarios.Other visualization methods reduce any additional dimensions to a 2D space via some mapping, requiring some training to understand and interpret.They may be applied to a scenario-based context by considering scenarios as additional dimensions to be reduced; however, scenarios and objectives are two different types of dimensions, and the interpretation of such visualizations becomes even more complex.Although various dimensionality reduction techniques have been proposed in the literature for multiobjective optimization problems, we did not find any scenario-based examples.Given the above summary, we believe that, in order to support a DM in comparing solutions according to trade-offs between objectives and scenarios, the existing visualizations need to be adapted and/or utilized in a different way, which is not straightforward.For example, in [26], uncertainty in high-dimensional data visualization is mentioned as an emerging area deserving future research.
In the next sections, we describe two approaches (heatmaps and visualizations of an empirical attainment function).They will form the basis of our proposed visualizations for scenario-based multiobjective optimization.

Heatmap
Heatmaps were used in [36] for simultaneously visualizing both objective and decision spaces in population-based multiobjective optimization methods.A heatmap is a 2D matrix, where rows are related to different solutions, and columns express corresponding objective function values.The magnitudes of the values of different objectives are represented by different colors/shades of the cells.One of the advantages of heatmaps, in addition to simplicity and scalability, is that they are a direct representation of the original data [46] instead of summarising it, except for limitations of the human eye in identifying shades/colors.Furthermore, trade-offs between the objectives can also be explained through the heatmap visualization [36,45].
Although heatmaps are more suitable for clustering solutions, they have also been utilized for comparing solutions [29].Heatmaps have also been utilized to compare different algorithms in multiobjective optimization problems [1,18].Later, in this paper, we extend heatmap visualizations and describe a new way of utilizing them in scenario-based problems.

Empirical attainment function
Given several mutually nondominated sets of objective vectors in the traditional (i.e., not scenario-based) context of multiobjective optimization, the empirical attainment function (EAF) [20] calculates how many of these sets weakly dominate (attain) each point of the objective space.If these sets are produced by some stochastic process, such as independent runs of a stochastic multiobjective metaheuristic algorithm [27], the EAF, in this traditional context, represents the estimated probabilistic distribution of the output of such algorithm in the objective space [16,17].With two objectives, the visualization of the EAF is rather straightforward and helps to understand the expected output of such algorithms [28].Visualizing differences between the EAFs of two different algorithms helps to analyze which regions of the objective space are more easily attained by each algorithm [27].
With three objectives, computation of an EAF is still straightforward.However, visualization must be done in three dimensions (3D) or somehow projected into 2D [45], preferably in some interactive fashion.Visualizations of an EAF in higher dimensions are rarely discussed in the literature since they are much more challenging, both computationally and cognitively.In this paper, we adapt the concepts behind EAF visualization to the context of scenario-based multiobjective optimization.

Perspectives to decision making in scenario-based multiobjective optimization
In this section, as the first contribution of this study, we raise possible questions that a DM may ask when facing a scenario-based multiobjective optimization problem.We do not claim that these questions cover all possible cases of relevance.However, these are aspects that either are found in the relevant literature, as we will show next, or have been faced in our practical experiences.
As mentioned, in scenario-based multiobjective optimization, evaluating decisions in terms of all objectives in all scenarios means that a DM has to analyze a lot of information.In general, the objective function values corresponding to a decision must be satisfactory for a DM in all scenarios.However, in some particular cases, when no feasible solution with satisfactory performances in all scenarios exists, a DM may be satisfied with a solution that performs better in sufficiently many scenarios.In this case, achievable performances in a given number (or percentage) of scenarios or, in some particular or critical scenarios, may be in the interest of the DM.Indeed, a DM's expectations may change during the decision-making process when (s)he gains a better understanding of the problem, the feasibility of desires, and trade-offs between objectives in different scenarios.Visualizations can help a DM in various ways.We assume that we have generated Pareto optimal solutions to different scenarios.In what follows, we discuss examples of questions that a DM may wish to answer with the help of visualizations.
Question 1. ''How does a solution perform in different objectives under the conditions of all scenarios?"This question means that a DM should be able to compare trade-offs among objective function values in different scenarios, both for an individual solution and for several solutions.The overall aim of a DM is to understand what kind of trade-offs there are among objectives in different scenarios.Some authors have indirectly or explicitly mentioned a similar question in various practical applications (e.g., [34,38]).
Even in the case of two objectives (f 1 and f 2 ) and two scenarios (s 1 and s 2 ), choosing the preferred solution is not apparent.For example, assume there are only two feasible solutions, if a solution z 1 has satisfactory values for f 1 in s 1 , but the value is unsatisfactory in s 2 , and another solution z 2 has a satisfactory value for f 1 in s 2 but its objective value in s 1 is unsatisfactory (this describes trade-offs between scenarios s 1 and s 2 ), which of these solutions is more desirable?In problems with more objective functions, scenarios, and solutions to compare, the task of a DM is even more challenging.However, visualizations can support a DM in choosing the most preferred solution.
Question 2. ''How many (or what percentage of) scenarios can reach the desired objective function values?"In this case, a DM must provide objective function values of interest called aspiration levels.Here, the aspiration levels do not depend on the scenarios as we are studying achievable performances in all considered scenarios.In this question, a DM faces a situation where it is impossible to satisfy all the preferences in all scenarios.Examples of this are real-life problems facing deep uncertainty, such as environmental planning problems.Because of the higher number of scenarios (typically hundreds or thousands), it is impossible to find an optimal solution for all scenarios, even for a single objective.Therefore, in these problems, a DM is looking for solutions that perform relatively well in most scenarios [5,12,22,23,25,50].In this case, a DM must understand how achievable the desires are by analyzing the number of scenarios where the desires can be met.Depending on the findings, visualizations can support a DM in revising the aspiration levels or finding a solution that is acceptable in a sufficient number of scenarios.If it is not possible to have satisfactory values in all scenarios, one can set the percentage and also study the effects of different percentages.This concept is also called domain criterion and has been utilized as a robustness measure in the literature [3,37].
Alternatively, a DM may have some important scenarios where the objective functions should reach the desired objective function values.In such a case, paying particular attention to these scenarios should be prioritized.
Question 3. ''Which objective function values can be reached in all or selected scenarios?"A DM may want to see the best and worst objective function values that can be reached in all scenarios or scenarios that (s) he has selected (as described above).With this information about the ranges of the objectives, a DM can adjust one's expectations on a realistic level, if needed.

Proposed visualization methods
In this section, we propose different visualizations to be used in scenario-based multiobjective optimization.First, we adapt the concept of EAF to the scenario-based context and propose three new ways of using the adapted EAF in decision-making problems with two objectives.Then, we propose a novel way of utilizing heatmaps for problems with more than two (or three) objectives.

Scenario-based EAF
We can adapt the definition of the first-order EAF [19,20,Def. 5.8] to scenario-based multiobjective optimization as follows.Given a decision vector x to a problem with k objectives and s scenarios, where y p ¼ ðf 1p ðxÞ; . . .; f kp ðxÞÞ, is the objective vector corresponding to evaluating x on scenario p, we can define where IfÁg : R k !f0; 1g is an indicator function.The scenario-based EAF (SB-EAF) can be understood in the following way: -SB-EAF indicates the number of scenarios in which a given decision vector x attains (dominate or equal) each point z 2 R k in the objective space.If the number of scenarios is very large, it may be useful to describe the SB-EAF in terms of percentiles, that is, the value 100 s Á a x ðzÞ gives the percentage of scenarios that attain z.
There are some conceptual differences between the classical EAF and this scenario-based counterpart.The classical EAF is computed from several random nondominated sets of objective vectors, typically sampled from some stochastic process, where the number of sets is equivalent to the sample size and the cardinality of each set is itself a random number.By contrast, the SB-EAF for a given x is constructed from exactly s objective vectors, and each of them corresponds to a different scenario.In a sense, the SB-EAF is a special case of the EAF where the number of samples is exactly s, and each sample has cardinality 1.However, under deep uncertainty, there is no probability associated with each scenario.Thus the SB-EAF does not represent a probability distribution, unlike the classical EAF.The region of the objective space where the SB-EAF takes a particular value, that is, the region that may be attained by a given number of scenarios 0 < t 6 s, is completely characterized by a finite set of objective vectors given by: L t ¼ minfz 2 R k : a x ðzÞ P tg; ð3Þ where each L t can be understood as the best possible objective vectors that are attained by at least t scenarios.When expressed as a j% percentage of scenarios, j ¼ 100 s Á t, then L j% is called the j%-attainment surface.For example, the median attainment surface (L d s 2 e or L 50% ) describes the best possible objective vectors attained by at least 50% of the scenarios.In other words, the j%-attainment surface is the best (in terms of Pareto optimality) set of objective vectors that may be attained under the conditions of j% of the scenarios.However, each of those objective vectors may by attained by different combinations of scenarios.
From the above definition, the best attainment surface (L 1 or L100 s % ) corresponds to the best possible objective vectors attained by at least one scenario, in other words, the nondominated set that is only reachable under the conditions of the best-case scenario(s).Each point in this set may be attained by a different best-case scenario.Conversely, the worst attainment surface (L s or L 100% ) describes the best possible objective vectors attained by all scenarios, i.e., the nondominated set that is reachable in the worst-case scenario(s).The worst-case may correspond to a different scenario for each solution in this front, yet we can guarantee to attain these objective function values under all conditions, i.e., for any scenario realization if z is the solution chosen by a DM.
In the case of the SB-EAF, an attainment surface separates the objective space into two regions.On one side, we have the best possible objective vectors attainable by a number of scenarios and, on the other side, we have those objective vectors that are not attainable under the conditions of some scenarios.Therefore, a DM can simply observe the best possible objective vectors attainable under selected scenarios for a given decision vector.This would answer Question 3 identified in Section 3.Moreover, by simultaneously portraying the attainment surfaces of different solutions in one graph, a DM is able to compare different solutions and choose the most preferred one.
If the best possible objective vectors cannot satisfy a DM's preferences in all scenarios, (s) he can set some aspiration levels and observe in how many scenarios these values can be reached.A DM can also choose to track the attainments of different objective vectors in selected scenarios if desired.All of these actions can be performed in the presence of more than one solution for comparison purposes and choosing the most preferred one.Accordingly, SB-EAFs are also helping a DM in answering Question 2.
Trade-offs between objective functions in different scenarios are also tractable in this visualization, both for an individual solution and several solutions.This can help a DM to learn about different options and limitations in various scenarios and support in making the final decision.Thus, Question 1 is also answered.
When comparing two solutions with corresponding decision vectors x 1 and x 2 , it is also possible to visualize the differences between their SB-EAFs (a x 1 ðzÞ À a x 2 ðzÞ 2 ½Às; s), that is, in how many more scenarios a given point z in the objective space is attained by a solution corresponding to decision vector x 1 than by a solution corresponding to x 2 .Such a visualization (referred to as SB-EAF differences from now on) can help a DM focus on the actual differences between the two solutions.Finally, it is also possible to visualize the combination of the SB-EAFs of multiple solutions with corresponding decision vectors X ¼ fx 1 ; x 2 ; . ..g in one single plot by computing their combined SB-EAFs as follows: that is, by considering the maximum value of their individual SB-EAFs at each point.We call this visualization all-in-one SB-EAF.In this manner, a DM is able to visualize at a glance the feasibility of attaining a certain vector in the objective space within the desired number of scenarios given the solutions visualized.Moreover, objective vectors that correspond to some decision vector but do not belong to an attainment surface L t of X (having (3) with X instead of x) are dominated by some other solution in at least t scenarios.
In total, we introduced three ways of applying the proposed SB-EAFs.Although they are powerful tools, unfortunately, practical visualization is limited to biobjective optimization problems.SB-EAF visualizations of a large number (e.g., hundreds) of solutions simultaneously can be nonintuitive for a DM due to the large amount of information.However, the SB-EAF can still be computed and, smaller subsets of solutions can be visualized according to filters set interactively by the DM.Nonetheless, SB-EAFs are useful, especially when many scenarios are to be considered.In Section 5, we illustrate the use of SB-EAFs in various examples.

Heatmaps for scenarios
In this section, we propose an adapted version of the heatmap visualization for scenario-based problems.The main idea is to utilize a 2D (k Â p) matrix representing normalized values of the k objective functions in p scenarios.In other words, rows are related to objective functions, columns describe scenarios, and each matrix refers to a solution (see Fig. 1 for an example).
It is also possible to utilize some indices describing the values of each objective function under conditions of each scenario.For example, one may consider distances between attainable values of a relevant objective function and an ideal point consisting of best individual objective function values under conditions of different scenarios.Then, different levels of colors/ shades in cells can display the variation.For example, for the distance of zero (the best-case) and one (the (normalized) worst-case) use black and white, respectively, while different levels of gray can be utilized for intermediate values.Moreover, using different colors with various shades may increase the readability of the heatmaps, especially when there are many scenarios, although, in this paper, we only use different levels of gray as an example.
A DM is able to compare heatmaps for different solutions (as long as they are not too many) and evaluate them from both dimensions (i.e., objectives and scenarios): the darker the color, the better the solution.An ideal solution would be a full black matrix, while a white matrix would describe the worst possible values.Accordingly, when comparing solutions, darker solutions would be preferred.Different levels of shades/colors in each solution can help a DM to observe how this solution performs in different objectives in various scenarios and study trade-offs.Therefore, we are able to answer the DM's Question 1.
Moreover, it is also possible to display numerical objective function values in each cell of the matrices in heatmap visualizations.This allows a DM to perceive and compare the performances of different objective functions in various scenarios.Thus, one can simply identify the highest objective values achieved in all selected scenarios (Question 3) or scenarios in which the desires are achievable (Question 2).
One of the most important advantages of using heatmaps in scenario-based problems is the applicability in problems with more than two objectives and quite a large number of scenarios.However, the number of objectives and scenarios cannot be arbitrarily large.Another advantage of applying heatmaps is that they enable a DM to compare the performance of a solution for different objectives in various plausible scenarios, which is important.
Although comparing arbitrarily many solutions is not convenient with this visualization method, heatmaps (in integration with a small multiple design 3 ) can help a DM in comparing the objective values of quite many solutions in different scenarios and choose the preferred one.On the other hand, regarding the cognitive limits of human judgment [33], comparing a higher number of solutions may need a different way of utilizing the heatmaps or other visualizations.In that case, for example, solutions can first be clustered, with or without the use of heatmaps, and one can interactively ask a DM to compare some selected solutions until the DM is satisfied with the final choice.
In all heatmaps shown in this paper, the range of gray levels is defined according to the maximum and minimum values shown for each objective (hence, there is always the worst value in white and the best value in black).However, it is possible to configure the gray levels relative to the given maximum or minimum values known for the problem or given by a DM and/ or focus on specific regions of the objective space.Also, if many solutions are available and a DM wishes to compare a small subset of them, we can first use all solutions to derive the gray levels and then only display the solutions of interest.In this way, the gray levels reflect the objective values relative to the best/worst values possible, although the heatmap only shows a subset of solutions.Similarly, in the SB-EAF plots, the minimum and maximum ranges can be configured if there is a priori knowledge available.

Examples of proposed visualization methods
In this section, we demonstrate the use of the proposed visualization methods through various examples.The examples have been selected to illustrate the use of visualizations in different types of scenario-based problems.Moreover, relevant data is available in this section, so the examples are fully reproducible.We classify the examples according to the number of objective functions, scenarios, and solutions to be compared.We also discuss how the visualizations in different classes can address the questions presented in Section 3. 3 Representing multiple charts/graphs side by side or in a matrix-based form is known as small multiples in multi-dimensional data and information visualization [43].Fig. 2 visualizes this solution using the proposed SB-EAF.As shown in the figure, the performance of the solution in different scenarios is clearly observable to answer Question 1.One can also compare the achievable values in each objective function for any scenario as well as the best and the worst attainment surfaces to answer Question 3. In this figure, the area in dark purple ( ) bounded by s 3 corresponds to the worst attainment surface, and it is guaranteed to be attained by the solution in all five scenarios, whereas the area in yellow ( ) bounded by fs 1 ; s 5 g corresponds to the best attainment surface attained in at least one scenario.Finally, if a DM determines a reference point of desired aspiration levels, it is easily possible to recognize how many scenarios can reach the desired values to answer Question 2. For example, the point ð15; 11:78Þ, which corresponds to the intersection of s 4 and s 5 , is attained in 4 scenarios, which is not obvious by looking at Table 1.
Example 5.2 (Biobjective, 3 scenarios, 5 solutions).We revisit the farming example proposed in [40], considering only the first two objectives to get a biobjective problem.The objective functions f 1 (costs) and f 2 (liquidity) are to be minimized and maximized, respectively.We have three scenarios.The objective function values for five solutions obtained with the twostage scenario-based framework (see [40]) are given in Table 2.
A simple visualization of the solutions, as shown in Fig. 3, can help a DM to gain insight about the robustness of the solutions across scenarios.That is, the Pareto fronts in different scenarios are closer to each other on the left side (solutions 4 and 5) than on the right side (solutions 1-3) of the objective space.Moreover, solutions 1-3 show the same trade-off across scenarios and across objectives: (1) solutions in scenario s 3 dominate the corresponding solutions in s 2 which dominate the ones in s 1 ; and (2) moving from solution 1 to solution 3 implies, in every scenario, trading off, i.e., losses in f 2 for improvements in f 1 .On the other hand, trade-offs are not so obvious in the case of solutions 4 (4) and 5 (5).In this case, a DM may compare trade-offs between objectives in different scenarios by visualizing the SB-EAF differences between solutions in that specific region, e.g., as shown in Fig. 4 for solutions 4 and 5, and choose the preferred one to answer Question 1.The SB-EAF differences show that solution 4 would be generally preferable, except in two regions of the objective space where solution 5 offers a slight advantage.Note that, without analyzing the differences as represented via SB-EAF differences, we can only say that solution 5 provides better performances for f 1 in all scenarios.In contrast, solution 4 is better than solution 5 when the second objective is concerned in all scenarios.An alternative is to visualize the SB-EAFs of each solution separately sideby-side, as shown in Fig. 5.
Furthermore, a DM may identify the scenarios in which desired aspiration levels can be reached, the best and the worst attainment surfaces, and achievable values for each objective function in every scenario, which allow answering Question 2 and Question 3. By connecting the related performances (objective values) of each solution in different scenarios, tracking and comparing the variation of the performances of the solutions in different scenarios can be much easier in some problems (see Example 5.5).
On the other hand, the solutions in Table 2 can also be visualized by the proposed heatmap approach, as shown in Fig. 6.Since we assume here that darker colors in heatmaps represent better objective values, a full black matrix would be the ideal.A DM can study trade-offs between objectives in different scenarios by comparing the shades of different solutions in heatmaps.For example, solution 5 provides the best values (identifiable by the darkest cells) for the first objective function in each scenario, while it has the worst values (recognizable by the lightest cells) for the second objective function in each scenario.It means that solution 5 provides the extreme objective values in each scenario (the best values for the first and the worst values for the second objective functions).The opposite is true for solution 1, which provides the best (darkest) per- formances for the second objective and the worst (lightest) performances for the first objective function in all scenarios.Therefore, if a DM was only interested in the performances in one of the objectives, (s) he would select one of these two solutions.
According to Fig. 6, if a DM is looking for the most balanced solution, solution 3 (the most neutral) will be the most preferred one.Therefore, a DM can answer Question 1. Besides, the numerical values of the objective functions, shown in every cell of the matrices, allow a DM to simply recognize the highest/lowest objective values achieved in all or selected scenarios (Question 3) or find scenarios in which the desires are achievable (Question 2).
Example 5.3 (3 objectives, 3 scenarios, 5 solutions).We now consider the original version behind Example 5.2 from [40] with three objective functions: investment costs (f 1 ) to be minimized and liquidity at the end of the year (f 2 ), and environmental benefits (f 3 ) to be maximized.We have three scenarios (for different weather conditions).Table 3 shows the objective function values for five solutions in different scenarios.The solutions are visualized by the proposed heatmaps in Fig. 7.In this figure, different shades in each solution demonstrate trade-offs between the objectives in different scenarios (Question 1), and the objective function values can be helpful in identifying extreme attainable values in all or selected scenarios (Question 3) and comparing them with the desires (Question 2).For example, suppose that a DM has aspiration levels of minimum 6 (millions of South African Rand [40]) for the second objective (the liquidity at the end of the year) and maximum 4 (millions of South African Rands) for the first objective function (investment costs).Then, first, a DM can ignore solutions 3 and 5. Furthermore, among the remaining solutions (1, 2, and 4), only solution 2 satisfies the investment limitation of 4 million Rands in all scenarios.On the other hand, if a DM intends to see the most attainable values of each objective function, solutions 3, 4, and 5 provide the best values for the first, second and third objectives, respectively.However, if the most balanced solution is desired, it can be identified by comparing the first two solutions.Example 5.4 (Biobjective, 324 scenarios, 1-3 solutions).We revise a sugarcane problem [41] by considering only the first two objectives.The objective functions are total profit and CO 2 emissions to be maximized and minimized, respectively.These objective functions need to be evaluated in 324 different scenarios.By applying a revised version of the models proposed in [41] for two objectives, three different solutions are obtained.Fig. 8 visualizes these solutions as a scatter plot.Although a careful examination of the plot may reveal that solution 1 is different from solutions 2 and 3, which are more similar to each other, simply plotting such a large number of points does not help a DM to understand which objective values may be attained by each solution in all, most or some scenarios; nor the actual differences between these solutions.The need for a more informative visualization is often faced in real-world problems (like this example), highlighting the importance of using the visualization methods proposed in this study.
Plotting the SB-EAF values of each solution (Fig. 9) precisely summarizes which objective values are attained by each solution in a given number of scenarios.In cases like this, where the number of scenarios is large, it is better to plot the SB-EAF values as percentages of the total number of scenarios and consider a small number of intervals.The separate SB-EAF visualizations can help a DM to answer Question 1 and Question 3 by comparing solutions and observing trade-offs from various aspects such as the best and the worst attainable values for different objective functions, as shown by the lower bounds of the regions colored in yellow ( ) and dark purple ( ), respectively.For answering Question 2, a DM may set the desired aspiration level and observe in what percentage of scenarios the level may be reached for each solution.Neither of these answers is obtainable from Fig. 8.As an example, a DM may be interested in solutions satisfying f 1 P 0 and f 2 6 7e þ 8 in more than 50% of scenarios.Looking at the top-left regions of Fig. 9, a DM can see that only solution 1 satisfies this requirement.The DM may then relax the aspiration level to f 1 P 0 and f 2 6 9e þ 8, but require that this level is reached in all scenarios.Again, looking at the darkest regions in Fig. 9, it is trivial to see that solution 3 is the only one satisfying the DM's preferences.
Finally, comparing the differences between SB-EAFs of two solutions at a time helps a DM to assess the actual differences between solutions in terms of the number of scenarios attained at each region of the objective space.Fig. 10 shows SB-EAF differences between solutions 1 and 2 and between solutions 2 and 3.The left-side plot shows that solution 1 is better than 2 only in the region where f 1 < 0:5e þ 10, while there are some small differences in favor of solution 2 when f 2 > 9e þ 8.The difference in the number of scenarios that attain any other region of the objective space is not larger than 10% for solutions 1 and 2. This answers Question 2. The location of these differences is not evident in the simple scatter plot (Fig. 8).Solution 1 has a much better CO 2 emission level (f 2 ).However, if differences in profit (f 1 ) are worth generating extra pollution, a DM may choose solution 2. Furthermore, comparing the differences between solutions 2 and 3, as shown on the right-side plot in Fig. 10, explains that solution 3 is better in almost every region.We could not conclude any of these facts from classical scatter plots, highlighting the usefulness of the proposed visualizations.4 describes the objective values for 12 solutions in three scenarios.Fig. 11 shows a classical visualization of this example.Again, comparing (or even distinguishing) the performances of different solutions in various scenarios is not an easy task for a DM.In this case, comparing 12 separate SB-EAF plots would not be convenient.Instead, a DM may visualize the combined SB-EAFs of all solutions in a single plot (all-in-one SB-EAF), as shown in Fig. 12.An interactive interface may easily take advantage of this visualization by enabling a DM to hide/show specific solutions and their associated SB-EAF regions.With such a visualization, a DM may observe at a glance the best and worst cases possible (best and worst attainment surfaces), answering Question 3, and achievable values for each objective function in every scenario that can satisfy DM's desires, answering Question 2. Also, those solutions for which there is another solution that dominates it in a particular scenario will appear within a colored region instead of at the border of the region (e.g., solutions 12 and 9 in scenario s 3 ).Furthermore, connecting the relevant performances of a solution with lines is useful in identifying and comparing different solutions as well as tracking trade-offs between objectives in different scenarios (to answer Question 1).These connected lines are shown in Fig. 12 for all solutions, although they could also be applied in Fig. 11.The shaded regions in Fig. 12 represent not only different attainment surfaces but are also helpful in identifying solutions that are Pareto optimal in all scenarios.Moreover, some solutions may be dominated by others in some scenarios while they are nondominated in other scenarios.For example, the ninth ( ) and twelfth ( ) solutions are dominated in the second and third scenarios, while they both are nondominated in the first scenario.Also, the second ( ) and the eleventh ( ) solutions are nondominated in the first two scenarios, while they both are dominated in the last one.Table 5 shows all the dominated and nondominated solutions in different scenarios, where a check-mark (U) and cross symbol ( ) indicate a solution being nondominated and dominated, respectively, within that scenario.Finally, if a DM selects a smaller subset of solutions for further analysis, the visualizations discussed above based on SB-EAF differences and heatmaps will be useful.

Real-world evaluation and comments from the DM
In this section, we provide some empirical evidence about the usefulness of the proposed visualization techniques in a real-life scenario-based multiobjective optimization problem.We evaluated the visualizations with real data and a real DM to examine the strengths and limitations of the proposed visualizations in practice as a proof of concept.It is crucial to show how the proposed visualizations can support a DM in gaining insight into different aspects of a complex problem and making better decisions.We consider a production allocation problem of a Finnish forest-based bioindustry company.This company operates on a global platform for manufacturing sustainable and innovative products and has production plants in 12 countries.Their 2019 sales were around 10.2 billion euros.
The company gave the problem and the data.The problem has two objective functions: maximize profit (f 1 ) and minimize emissions (f 2 ).Depending on different uncertain parameters such as demand, sales price, and machine operation rates, seven scenarios are considered.The DM selected six different solutions.The solutions were calculated by relaxing 0%-10% of the optimal values for the sales margin.They are ordered as solutions 1-6 representing 0%, 0.5%, 1%, 2.5%, 5% and 10% decrease in sales margin, respectively.Therefore, the DM had prior knowledge about the problem, scenarios, and solutions.He also had used different visualizations such as bar graphs and pie charts and found them inefficient, particularly in the case of multiple scenarios, even in a single objective optimization problem.Thus, he was looking for more suitable visualization techniques to compare the results in various scenarios.
To record his first impressions about the proposed visualizations, we did not provide him any information about the visualizations before the experience.The experience was started by a brief description of the elements, colors, and symbols used for objectives, scenarios, and solutions in the visualizations.Then, different visualizations were shown to the DM in the following order: heatmaps, SB-EAF for each solution separately (similar to Figs. 2, 5, and 9), all-in-one SB-EAF (similar to Fig. 12), and SB-EAF differences for each pair of solutions (similar to Figs. 4 and 10).The DM's feedback related to each visualization can be summarized as follows.
Heatmaps (Fig. 13): In general, the DM found heatmaps informative, intuitive, and useful compared to the numbers in tables they usually use.The DM mentioned that ''very big trends are visible", meaning that the best and the worst objective values can be recognized (Question 3).He then set an aspiration level of 2 million euros for the first objective function (profit) and examined how well it can be achieved in different scenarios (Question 2).He also found it easy to compare the trade-offs between solutions across scenarios by looking at the levels of gray (Question 1).Therefore, he was able to answer all three questions raised earlier in this paper.In addition, from a different perspective, the DM was able to observe the effects of scenarios in each objective function.He mentioned that ''in the profit row there is a lot of variation among scenarios and profit sensitivity is high in scenarios but emissions do not vary that much and are rather evenly colored -scenario does not matter that much to emissions."The DM felt that more insight might be gained if he could rearrange some rows and/or columns, which suggests that an interactive variant of the heatmap visualization would be even more useful.
SB-EAF for each solution: Here, again, the DM was able to answer all three questions in a similar manner as with heatmaps.However, he found trade-off comparisons ''easier here than in heatmaps".The DM also observed that the best/worst attainment surface could be changed in different solutions, so there is no single scenario that could be called the best/ worst-case scenario.For the sake of conciseness, we do not show these plots here.
All-in-one SB-EAF (Fig. 14): The DM was impressed by this visualization.He liked the way that all the elements (i.e., objectives, scenarios, and solutions) have been ''fitted together".The DM mentioned that ''assessing solutions one at a time is a natural choice but not necessarily enough".This supports our claim about the DM's challenges in comparing a series of separate plots in scenario-based multiobjective optimization problems.In summary, the DM found separate plots useful to compare details of particular trade-offs; nevertheless, the DM found all-in-one SB-EAF more beneficial to obtain an overall view.In addition, the DM gained particular insights into the problem that, as he pointed out, were not available earlier without the help of this visualization.
SB-EAF differences: The DM reported that he found this visualization more different than what he is used to thinking and less easy to understand how it works.He got the idea but will need more time to get used to it.He felt that it could be useful for considering certain pairs that have been first selected.For the sake of conciseness, we do not show these plots here.
General: Thereafter, when asked to give his overall impression of the visualizations, the DM stated that ''I would select the all-in-one SB-EAF (Fig. 14)) to present the big picture and take the key elements and understanding of the system as a whole.Then I would use heatmaps as there are many aspects that are easiest to see in heatmaps".The DM also mentioned that ''people want to see the same thing in multiple ways" and having different visualizations can complement each other.Overall, he found various benefits in all the proposed visualizations, even in the SB-EAF differences.However, it is better first to find key details from other visualizations (all-in-one SB-EAF and heatmaps) and then use SB-EAF differences for specific pairs.
To conclude, heatmaps and all-in-one SB-EAF can complement each other in biobjective scenario-based multiobjective optimization problems from two perspectives: (1) observing the same facts from different views and (2) getting various insights (big picture thinking vs. detail-oriented).Besides, developing a flexible and interactive visualization platform including different visualizations with the ability to switch between visualizations, rearranging the objectives, scenarios, and solutions and/or choosing some of them for a closer look can be helpful in practice.

Discussion and guidelines for applications
We have proposed different visualizations to be applied in scenario-based multiobjective optimization problems.However, because of the complexity of these kinds of problems and various perspectives of a DM, one method may be more suitable than another one to be utilized in different cases or classes of problems.In this section, following the examples of the previous sections, we discuss different classes of problems in more general terms and discuss how our visualizations could have supported studies reported in the literature.To this end, we borrow some other examples from the literature where the proposed techniques could be applied.However, as not all the details are available to reproduce the visualizations, we can only discuss them on a general level.Table 6 compares the proposed visualizations with those found in the literature based on their capability to answer the three DM's questions raised in this paper (Q1-Q3).The tick sign (U) shows that the corresponding visualizations, applied in the particular example considered in that paper, can completely answer the relevant question, while the cross sign ( ) represents the opposite.The sign (%) means that the assigned visualization can answer the question, but only in some limited problems or with some difficulties in large cases.According to this table, amongst the visualizations utilized in the literature for some scenario-based context, only scatter plots could answer all three questions for biobjective problems with very few scenarios and solutions.
According to this table, amongst the visualizations utilized in the literature for some kind of scenario-based context, only scatter plots could answer all three questions for biobjective problems with very few scenarios and solutions.Therefore, as discussed earlier, its application is mostly limited to theoretical studies to describe the theory in an elementary example as a proof of concept, and not in a real-life problem.Furthermore, in biobjective problems, SB-EAFs provide extra valuable information, such as dominance relations between solutions in different scenarios and differences between solutions, in addition to answering all three questions.Let us note that the papers mentioned in Table 6 did not consider all these three questions in the same way we have formulated them.Some of them (e.g., [2]) did not look at trade-offs between scenarios, even though they did plot objective values in different scenarios in a single scatter plot.We will discuss it in more detail in the following subsections.
Further discussion and some suggestions on where the proposed visualizations are best suited to be used as a guideline for real applications are accomplished in the following subsections.As mentioned earlier, we classified problems based on the number of objective functions, scenarios, and solutions.Accordingly, we discuss different cases separately in this section.

Biobjective problems
As discussed earlier, the proposed SB-EAF is well-suited for biobjective optimization problems.It can handle many scenarios and solutions (although visualizing many solutions in a single plot may need further attention as discussed at the end of Section 4.1) while providing a clear overview of the best and worst possible values of each objective in any or all scenarios for those solutions.In many real applications, hundreds or thousands of scenarios are simulated, and many solutions are compared to find the best one.In this case, a visualization technique should handle quite many scenarios and/or solutions.
The proposed SB-EAFs can be applied to perform analysis in a wide variety of applications such as bridge management [34], location-allocation models for disaster response [2], and berth allocation scheduling [52].For example, biobjective models for optimizing bridge adaptation strategies under multiple future climate scenarios were proposed in [34].Columbia and Mississippi Rivers in the United States were studied in five scenarios, and Pareto optimal solutions were presented by separate scatter graphs.Applying SB-EAFs visualization could enable comparing trade-offs between objectives in all scenarios and make the decision-making process easier and more accurate.
Furthermore, a biobjective case study of the 2015 Nepal earthquake response was studied in [2].The minimization of the total logistics costs and response time were considered.Separate sensitivity analyses were conducted considering seven scenarios for various uncertain parameters such as demands, number of available ground transportation, air and ground transportation costs, the weight of delivered relief packages, number of available air transportation, and recurring and human resource costs.The results were visualized in several separate scatter plots (using lines to connect solutions within each scenario).Utilizing SB-EAFs in this work could help to visualize the results of such a sensitivity analysis more appropriately.SB-EAFs could also be helpful to study the large number of scenarios resulting from all the uncertain parameters.
A berth allocation problem for container terminals was studied in [52], where a biobjective optimization model was proposed for cost minimization and maximizing the robustness of schedules.Although the method was not applied in a real problem, the problem formulation is generic enough to be applied to realistic environments.Instead of presenting the performances of schedules under different scenarios in a long table or separately displaying the average costs of schedules for each scenario, the proposed SB-EAFs visualization could be applied to help a DM compare trade-offs between objectives in all scenarios.The authors had to portray only 4 or 5 out of 9 solutions (also referred to analyzed schedules in the paper) in separate scenarios to be able to visualize them with different line graphs, while our proposed SB-EAFs could visualize all the results clearly in one figure .Naturally, the applicability of the proposed methods is not limited to the above-mentioned papers.Furthermore, as discussed earlier, all the DM's questions of Section 3 can be addressed by SB-EAFs in biobjective problems.Besides, as in the previous section, if the number of scenarios and solutions is lower than 10 (or a DM wants to concentrate on some specific scenarios and/or solutions), heatmaps can also be utilized to compare details and, thus, visualizations can complement each other.

Moderate number of objectives, scenarios and solutions
As discussed in Section 4.2, the proposed heatmap visualization is suitable (i.e., it addresses the DM's questions discussed in Section 3) for problems with more than two objectives if the number of solutions is not too high.In this case, comparing the performances in different objectives and diverse scenarios is possible.Even in the case of too many scenarios and/or solutions, comparing the performances in some selected scenarios and solutions is meaningful and applicable in real-life applications.For example, a water resources problem in [50] (in the Lower Rio Grande Valley of Texas, USA) had six objective functions and five scenarios, and 3D glyph and parallel coordinate plots were used to illustrate objectives and decision values.Nevertheless, as individual solutions for each scenario were visualized, trade-offs between scenarios were not traceable.The performances of six selected solutions were also compared in five scenarios.However, the comparisons were made with separate plots for only two out of five scenarios.Thus, again, studying trade-offs between scenarios was not supported, even for only five scenarios.In this case, the proposed heatmap visualization could help and support performance evaluations and trade-off comparisons in various scenarios, as well as the analysis in scenario effects and finding more adaptable and robust solutions.
Moreover, the proposed visualizations can also be utilized to compare different solutions in scenario-based multiple criteria decision analysis problems and strategic planning.For example, the proposed heatmaps could easily be used to compare the performances of five strategic options in [38] involving food security in Trinidad and Tobago.Three criteria in 12 (reduced) plausible scenarios were considered while, originally, there were 108 possible scenario combinations.No visualizations were used to compare the performances of different alternatives (strategic options) in different scenarios, and only a plot of the regret values was used.Instead of multiple tables to present results, heatmaps proposed here would help a DM track and compare the overall performances of applying each strategic option (alternative) in different scenarios as well as the best and the worst-case scenarios.Heatmaps could also help in regret analysis and in finding the most robust option.

More than two objectives with many scenarios and/or solutions
Because of the complexity of problems with several objectives and many scenarios and/or solutions, getting an overview is very challenging.Usually, we need to either reduce the dimensions and visualize results in lower dimensions or perform separate comparisons (e.g., pairwise comparisons).Decreasing the dimensions can be done by either reducing the number of scenarios or limiting the number of solutions considered.As mentioned in [38], reducing the number of scenarios regarding the DM's preferences does not breach the philosophy of scenario planning.Similarly, the number of solutions can also be decreased to reflect the DM's preferences.Therefore, if we can reduce the number of scenarios and/or solutions in problems with more than two objective functions, the proposed heatmaps will be applicable and can address the DM's questions raised in Section 3 for the reduced problem.
Nevertheless, if the number of scenarios and/or solutions to be compared is considerably higher than, heatmap visualizations may not help much in comparisons and analysis.In this case, a 3D version of the SB-EAFs may be useful in getting an overview of the attainable values of objective functions in various scenarios, although not as intuitively as in biobjective problems.
Table 7 summarizes our discussion and recommendations as a quick guideline in applying the proposed visualizations in different cases of problems.We have separated cases with less or more than 10 scenarios and solutions.The numbers in parentheses refer to the number of objective functions.

Conclusions
In this paper, we have proposed visualizations to support managers in understanding, evaluating, and comparing the performances of management decisions in scenario-based multiobjective optimization problems.First, we identified some fundamental questions that a DM may want to answer by employing a suitable visualization.Then, we proposed the concept of scenario-based empirical attainment function (SB-EAF), which summarises the number of scenarios that may attain a par- ticular point in the objective space.Then, we proposed three visualization methods based on SB-EAFs.In addition, we proposed an adapted version of heatmaps.We illustrated via various examples how they can support a DM in evaluating and comparing solutions, making a robust decision, and answering their questions.SB-EAFs can be applied in almost all biobjective scenario-based problems without any limitation in the number of scenarios or solutions from the computational point of view.However, in the case of many solutions, interactive filtering and visualizing a smaller subset of solutions is recommended to reduce the cognitive load.It also provides a clear overview of the best and worst possible values of each objective in any or all scenario(s).The main advantage of using SB-EAFs is the ability to treat quite a large number of scenarios and/or solutions.Such a need is often faced in real applications.This capability makes SB-EAFs applicable in a wide variety of applications.However, unfortunately, their applicability is limited to biobjective problems, and a different visualization is needed in problems with more than two objectives.
Therefore, we proposed an adapted version of heatmaps, which is suitable for problems with more than two objectives if the number of objectives, scenarios, and solutions is not too high (say less than 10).Even in the case of too many scenarios and/or solutions, comparing the performances in some selected scenarios and solutions can still be meaningful and applicable in real-life applications.The proposed visualizations can help a DM both understand and compare trade-offs between different objective functions and evaluate and analyze trade-offs under conditions of various scenarios.This was also clearly confirmed in an evaluation involving a real DM.He also recommended an order of showing visualizations to support decision-making in his case.
Furthermore, we classified scenario-based multiobjective optimization problems into three different classes: (1) biobjective problems; (2) problems with more than two objectives and a moderate number of scenarios and/or solutions; and (3) problems with more than two objectives and many scenarios and/or solutions.The proposed SB-EAF and heatmap visualizations are useful in the first two cases, respectively.Our suggestions can be used as a guideline in applying the proposed visualizations in real applications.
The third case is open for future directions of research, but the proposed heatmaps can be utilized in some kinds of reduced problems, and there are proposals in the literature for visualization of multi-dimensional EAFs [45,46] that may be adapted to the SB-EAFs proposed here.Developing these and other novel visualizations for scenario-based multiobjective optimization problems, converting the ideas proposed into interactive visualizations that DMs can conveniently tailor and getting additional feedback from real DMs are our future research interests.Another important future direction is conducting a behavioral study.
Reproducibility.Source code and data to reproduce the visualizations in this research can be found at https://doi.org/10.5281/zenodo.5040421.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Fig. 1 .
Fig. 1.Visualizing a solution with k objectives and p scenarios by proposed heatmaps.

Fig. 5 .Fig. 6 .
Fig. 5. SB-EAFs corresponding to solutions 4 (left) and 5 (right) of Example 5.2.Points denote the objective vectors evaluated on different scenarios.Colored areas show regions of the objective space that can be attained by a particular number of scenarios.(see online version for color figures)

Fig. 9 .Fig. 10 .
Fig. 9. SB-EAFs corresponding to solutions 1 (top), 2 (left) and 3 (right) of Example 5.4.Points denote solutions evaluated on different scenarios.Colored areas show regions of the objective space that can be attained by the solution in a particular percentage of the 324 scenarios.(see online version for color figures)

Fig. 14 .
Fig. 14.All-in-one SB-EAF of the real-world evaluation.Points that represent the same solution evaluated in different scenarios are connected with a line.(see online version for color figures)

Table 1
Objective function values of Example 5.1 in five scenarios.

Table 2
Five solutions in three scenarios for Example 5.2.

Table 3
The objective values of five solutions in three scenarios for Example 5.3.

Table 7
Visualization appropriate for various cases (k is the number of objectives).Difficulties with very large set of solutions.c For selected solutions and not with very large set of scenarios and objectives. b