Portfolio Decision Analysis: Recent developments and future prospects

Portfolio decision analysis (PDA) refers to the body of theory, methods and practice which support decision makers in making informed multiple selections from a set of alternatives with the help of mathematical models that account for relevant constraints, preferences and uncertainties. In this review, we take stock of recent advances in PDA research, based on a representative sample of 148 PDA articles in operations research and management science journals from 2006 to 2019. In particular, we analyse relevant methodologies and discuss prominent PDA application areas. Our analysis indicates that PDA is a vibrant research field with close ties to practice, as a substantial share of articles present real applications or contain illustrative examples which are motivated by such applications. For continued knowledge accumulation, there is substantial promise in exploiting PDA concepts in deriving recommendations from decision models for problems which may not have been viewed as PDA problems; fostering the cross-fertilization of conceptual and methodological advances across application areas; and ensuring that new methodological advances are systematically evaluated through engagements with real decision makers.


Introduction
In firms and public organizations alike, activities towards the attainment of strategic and tactical objectives are often organized by first identifying a large number of alternatives and by selecting those that are expected to best contribute to the attainment of the relevant objectives, subject to constraints on the availability of resources, time and expertise. In effect, there is a broad range of problems which can be viewed in this framework. For instance, high tech companies pursue growth targets by selecting R&D projects (Toppila et al. 2011); public transportation agencies have annual maintenance budgets to ensure the quality of infrastructure assets (Mild et al. 2015;Mild & Salo 2009); military planners seek to build capabilities through cost-efficient combinations of weapon systems (Kangaspunta et al., 2012); and healthcare organiza-tions invest in services and facilities with the aim of maximizing patients' health benefits (Airoldi & Morton 2011).
While these problems are seemingly different, they have so many structural similarities so that they can be addressed by employing decision analytic approaches for portfolio selection and resource allocation. Such approaches can be collectively referred to as portfolio decision analysis (PDA).
In this review paper, we take stock of the emerging body of PDA literature, characterising PDA activities in terms of their main application domains, model features and solution techniques. By doing so, we seek to inform researchers about the current and prospective strengths of PDA and to give indications as to what kinds of future contributions would be particularly welcome; we also hope to inform practitioners about the application areas in which PDA has been or is likely to be particularly successful. Towards these aims, our review combines extensive structured queries in the OR/MS literature, guided by our definition of PDA and our subjective screening. We also give statistical summaries and reflective comments on the literature, followed by our thematic subjective discussion of the PDA field.
There are four partly overlapping types of contributions that are particularly relevant to this review. First, the PDA literature is connected to the literature on project portfolio management which, among other things, offers theoretical frameworks for the design and implementation of decision processes in which formal approaches such as PDA can be helpful. Second, there are papers which, instead of providing immediately actionable decision recommendations, stimulate management insights by capturing the salient features of generic PDA decision problems and by analysing what implications these features have for managerial decision making such as budgeting. Third, theoretical and methodological contributions underpin the formulation and solution of models which yield actionable decision recommendations to decision makers (DMs) faced with PDA decisions. Finally, reports on applications elaborate how actual decision problems have been addressed through PDA approaches. Against this backdrop, we focus on the two latter types of contributions, recognising that most OR researchers work on theory and methodology, or innovative applications that are enabled by them.
Historically, problems of project portfolio selection have spurred plenty of methodological and applied 2 research in OR. The origins of this research can be traced back to the 1950s, with roots in capital budgeting, financial portfolio optimisation and project scoring models (for an overview see Salo et al. 2011b). During the last decade, efforts have been taken to consolidate some of these expanding and interrelated strands of research in order to establish an identifiable subarea of OR/MS research focused on portfolio decisions. For instance, Kleinmuntz (2007) provides an overview of decision analysis approaches for resource and capital allocation decisions. A few years later, three of us defined the term Portfolio Decision Analysis (PDA) as "...theory, methods and practice which seeks to help decision makers make informed selections from a discrete set of alternatives through mathematical modelling that accounts for relevant constraints, preferences and uncertainties" (Salo et al. 2011a).
Portfolio decisions are often complicated by several factors. First, these decisions are expected to contribute to the attainment of multiple decision objectives. For instance, healthcare organizations must consider patient satisfaction as well as operational efficiency; and as firms may find it difficult to measure how individual projects generate shareholder value, they may instead use multiple, more readily measurable proxy attributes (e.g., profits, market-share). Second, because the 'value' of a project is rarely known when it is launched, the project selection decision has to be based on uncertain estimates of project values. For in-stance, a funding agency providing research grants has to make choices on project proposals without knowing what the eventual results of these projects will be. Furthermore, the value generated by a product portfolio can be heavily affected by exogenous uncertainties (e.g., intensity of competition, changes in the regulatory environment) whose realizations are not known when the product development decisions are made. Third, there can be interactions among the projects. For instance, if two R&D projects benefit from access to a shared research infrastructure, the cost of implementing them both may be less than what they could cost if implemented separately. Finally, the number of alternative portfolios is often large. For instance, in case of 30 simple 'go/no-go' projects the number of possible portfolios is two to the 30th power, or approximately one billion, while with 300 projects the number of possible portfolios is two to the 300th power -a number which exceeds the number of atoms in the observable universe.
In view of these subtleties, we have scoped this review by focusing on OR/MS journal articles which have been published during the past dozen years and which develop, deploy and explore mathematical models to support decisions in choosing a portfolio of decision alternatives. We realise that much of PDA research has appeared in application area-specific journals (e.g., in environmental management), but surveying this vast set of journals would have necessitated a more application focused review of its own. Furthermore, there are recent papers which list recent work on PDA in specific application areas such as environmental decision making, R&D project selection, healthcare, and military planning (Lahtinen et al. 2017;Morton et al. 2016).
Likewise, project portfolio selection, without special focus on decision analytic methods, is well-established in the project management literature (see, e.g., Martinsuo 2013). Finally, we believe that the level and nature of PDA research activity in methodologically oriented journals is a telling indicator of the 3 relevance of PDA for OR/MS researchers. Thus, although our focus on OR/MS journals risks missing some potential PDA developments in application-oriented journals, the review still serves the valuable objective of analysing the significance of PDA research within OR/MS. Nevertheless, we are aware that the resulting sample of articles does not give an unbiased representation of all PDA literature, which needs to be borne in mind when interpreting the results of this review.
Even among OR/MS journal articles, our definition of PDA does not provide a clear-cut rule on which articles to include in the review. For instance, multi-objective optimisation (MOO) techniques and algorithms are often employed to solve PDA models. Yet, including all MOO articles on problems with binary decision variables would have led to a loss of focus. Furthermore, while many financial portfolio models address uncertainties and investors' risk preferences with discrete decision variables and multi-criteria methods (see, e.g., Greco et al. 2013), we have purposely excluded financial models from our scope, partly because there are reasonably up-to-date review papers (see, e.g., Kolm et al. 2014). Finally, there is no clear boundary between PDA problems which involve decisions as to when the chosen projects should be implemented, and the kinds of problems which are studied in resource-constrained project scheduling (see, e.g., Hartmann & Briskorn 2010). In general, the project scheduling literature has a stronger focus on the development of computationally efficient algorithms to identify optimal schedules, albeit many of the articles reporting real-life applications also discuss the portfolio-aspects of the problem. This paper is structured as follows. Section 2 revisits our definition for PDA. Section 3 describes the search and screening of articles for this review. Section 4 discusses how these articles have advanced PDA theory, models and methods. The applications reported in them are analysed in Section 5. Section 6 provides a subjective overview on the current strengths and weaknesses of PDA research and identifies some future opportunities and threats. Section 7 concludes.
2. What is portfolio decision analysis? Salo et al. (2011a) define PDA as 'a body of theory, methods and practice which seeks to help decision makers make informed selections from a discrete set of alternatives through mathematical modelling that accounts for relevant constraints, preferences and uncertainties ' (p. 4).
This definition is useful in distinguishing PDA from its siblings in the field of OR. General applications of OR-the allocation of physical aircraft to an airline schedule for example -may require the modeller to incorporate a number of constraints, but present relatively little need to model preferences and uncertainties.
In these areas, optimisation methods may be an entirely appropriate modelling tool. Similarly in strategic decisions -the choice of an approach for the disposal of radioactive waste, for example -although preferences and uncertainties abound, the problem may be more naturally framed as one of choosing a single option from a shortlist. In this case, standard decision analysis methods might be adequate. In effect, PDA 4 problems tend to be characterised by their tactical focus, sitting between more operational problems (in which contested preferences and uncertainties do not have much of a role) and more strategic problems (in which the problem may not be sufficiently well-defined to develop an agreed set of decision options with a clear combinatorial structure).
Because our PDA definition is based on problem characteristics and solution needs, it is intentionally inclusive in terms of methods, spanning approaches such as stochastic programming, multiobjective metaheuristics, variants of established decision support methods, including multi-attribute value theory, outranking methods, and ad hoc scoring-and-weighting. In particular, we note that not all the articles which we consider to be 'portfolio decision analysis' employ this term; yet there are legitimate reasons for covering them in this review.
Central notions in PDA problems include the following: • The resource/budget constraint. Not all decision opportunities can be pursued, because resources are invariably scarce (most notably money, but also staff time or political attention). At the relevant organisational level of decision making, these constraints may not be easily changed.
• Benefits. Choosing an alternative into the portfolio is expected to produce some positive benefit which can be tangible (e.g., revenues) or intangible (e.g., contribution to knowledge), but which often will be multidimensional, mixing both tangible and intangible components.
• Interrelationships. The value of selecting a portfolio of alternatives relates in some systematic (but not necessarily simple) way to the benefits gained by selecting the constituent alternatives in the portfolio.
Specifically, depending on the context, doing two projects jointly may produce more or less value than the sum of the value of doing each individually; and, analogously, doing two projects together may cost more or less than the sum of these projects' individual costs.
• The incumbent portfolio. In many cases, there may be a portfolio which has a special status, for example, because it was the portfolio that was chosen in the previous planning cycle. Then, once the new portfolio is selected, part of the implementation challenge is to design a path from the incumbent portfolio to the new one. Sometimes this implementation challenge may be so substantial that it impacts the selection of the new portfolio.
• Decision areas. Often alternatives can be grouped in decision areas which may correspond to areas of managerial responsibility, for instance. Within decision areas, it is often easier to compare projects with each other because they are relatively homogenous (in the familiar metaphor, within a decision area, one is comparing apples with apples rather than apples with oranges). However, challenges may be encountered when moving from an incumbent portfolio to a rather different portfolio which calls for changes in the pattern of resource use across decision areas.
In problems which share characteristics such as these, one can often employ similar kinds of problem and solution representations (see Morton et al. 2016 for examples): • Bubble plots show projects as bubbles on a grid in which the size of the bubble is the resource footprint of the project and the dimensions of the grid are the two most important (or other selected) benefit dimensions. Such plots give an overview of the decision problem at the level of the individual alternatives.
• Triage plots which show projects ranked on a scale from "must do" (i.e. definitely include in selection) to "must die" (definitely do not include), taking into account uncertainty about benefits and about value tradeoffs. Such a triage plot can be useful in indicating those projects about which decisions can be taken based on tentative initial analyses, allowing more detailed analyses to be focused on the other remaining alternatives.
• Pareto plots show the overall value of the selected portfolio graphed as a function of the available budget. Such plots give decision makers a feel for the difference between an incumbent and selected portfolio in the cost and benefit space simultaneously, as well as a sense of what the opportunity of an increased budget or the threat of a reduction therein would imply.
We note that some of the above concepts have been proposed independently by multiple researchers in different application domains with different technical modelling choices, which reflects the fragmentation of the scientific literature. One of the aims of this review is to point out commonalities in the core ideas that underpin multiple research efforts and, by doing so, to foster systematic and cumulative knowledge building.

Survey methodology
The Web of Science (WoS) database was used to search for PDA articles published between January 2006 and March 2019 in journals included in the 'Operations Research & Management Science' -research area.
Specifically, the article's WoS topic (consisting of title, abstract and keywords) was required to include (i) the word 'portfolio*' and (ii) at least one of the words 'decision*', 'resource allocation*', 'budget allocation*', and 'project*'. We did not consider conference proceedings, books or book chapters. The 1049 articles produced by this search were manually screened based on their abstracts to include only those that develop, apply or study mathematical models for choosing a portfolio of decision alternatives. At this stage, many articles discussing financial portfolio models were excluded. We also required some use of decision analytic approaches in the modelling of preferences, multiple attributes/criteria/objectives and/or uncertainties. This screening reduced the number of articles to 212. In our final screening phase, we manually scanned the main body text of the articles. We discarded some articles that were not concerned with a portfolio of alternatives even if the term 'portfolio' appeared in the abstract or keywords. Also, some articles that developed stylised 6 macro level models describing optimal portfolio strategies of hypothetical companies were discarded. We also noticed that some PDA papers that we were aware of had not made it to the list. One reason for this was that the WoS research area 'Operations Research & Management Science' did not include all journals that are effectively OR/MS journals (e.g., Decision Analysis). Some obvious PDA articles did not have the keyword 'portfolio*' in their title, abstract or keywords. After the screening the full text of the articles and the addition of some PDA articles that had not been captured by the systematic search we arrived at the final set of 148 articles.
The journals in which these articles were published are reported in Table 1 However, the journals in Table 1 differ greatly in their publication volume. For instance, according to data in WoS, EJOR published some 9000 articles in total between 2006 and 2019, while Omega published 'only' some 1500. In order to account for the effect of the journal size, the last column of Table 1 Table 1 has barely doubled from some 1800 to 3100 articles. However, after 2016 the number of PDA articles has not grown even if we extrapolate that the year 2019 will see some 12 articles based on first quarter data from that year. We are not able to provide definitive reasons for this decline, but are inclined to speculate that in 2016 and 2017 there may have been an exceptionally high number of PDA articles due to purely random effects and that similarly 2018 may have been an outlier with low number of articles. In consequence, we are not inclined to infer far-reaching conclusions about publication trends; or to endorse the interpretation that the PDA field has become more mature, making it more difficult to produce novel methodological contributions and advances -which the journals included in this review are seeking.
The 148 articles were categorized using 42 characteristics of which most were binary (see Appendix A).
These characteristics were defined with an eye towards collecting information that would be relevant to researchers and practitioners planning their future efforts. Inevitably, such a categorization involves some 7 degree of subjectivity that cannot be completely eliminated due to shifting uses of terminology and missing details on the reported models and applications, for instance. To ensure the consistent interpretation of these characteristics across the articles, the characterization was carried out by only one of the authors.
Essentially half of these characteristics pertained to the models that were developed and/or deployed, and half to the reported application (if any). The application related characteristics included the application area (e.g., industry), the type of the portfolio elements (e.g., R&D projects) as well as the size of the portfolio problem (number of elements, constraints and objectives; monetary value). The decision model characteristics and optimisation model characteristics were treated as two separate categories. Decision model characteristics provide information about the decision setting and address questions such as whether or not there are decisions other than project selections (e.g., work allocation); which methods (if any) are used to capture preferences among multiple decision objectives (e.g., MAUT, outranking); and are uncertainties explicitly modelled and, if so, how are they quantified (e.g., probabilities, fuzzy sets). The optimisation model characteristics considered the computational properties of the PDA model, for instance, is the model linear or non-linear; does it have integer variables or multiple objective functions; and what solution approaches (e.g., exact or heuristic algorithms) are used to derive decision recommendations.
The following sections report key findings resulting from the application of the above categorizations to the selected 148 articles. Section 4 considers the types of decision analytic and optimisation methods and models, and Section 5 analyses the types of applications.

Decisions and constraints
Although PDA is, by our definition, concerned with the selection of a subset of alternatives, there can be several kinds of accompanying decisions that are contingent on which projects are selected; thus, it can be useful to include these decisions explicitly in the portfolio model. This typically requires the introduction of additional decision variables as well as the specification of adjoining constraints that ensure the values of these decision variables are consistent with project decisions. An analysis of the articles suggest that these contingent decisions are often relevant: 32% of the models involved further decisions in addition to project selections, and almost 50% included constraints that were not concerned with resources.
Perhaps the most obvious decision that is intertwined with project selection is the specification of the total resource expenditure. Several of the articles study models in which the resource limits (e.g., budget) are not fixed but are treated as (continuous) decision

Project interactions
In the PDA literature, the term project interaction usually refers to situations in which the selected projects' combined value or cost differs from the sum of their individual values or costs. While this definition is intuitive, it is technically somewhat inaccurate as it does not distinguish between violations of additivity caused by the mechanism through which the projects yield outcomes and those that pertain to preferences.
For instance, if two products cannibalize each others' demand, the value that they provide (e.g., NPV) will be non-additive; and analogously, if two R&D projects benefit from access to the same research infrastructure, the aggregate consumption of resources due to the implementation these projects may be non-additive.
However, value interactions can also be caused by non-additivities in the DM's preferences, such as decreasing marginal value. In general, interactions can be associated with any subset of the project candidates, and consequently they cannot be adequately modelled by limiting the attention to pairs of projects.
In many of the reviewed PDA articles, the lack of tools for capturing project interactions was mentioned as one of the motivations for methodological development. In effect, 23% of the articles developed or deployed PDA models with project interactions, which is a clear indication that tools for handling interactions are available. There seem to be two main approaches for modelling project interactions. The first is to introduce constraints which ensure the interaction-specific dummy project is included in the portfolio if and only if the interaction is triggered by the composition of the portfolio. Then the dummy project's value and resource parameters can be employed to indicate how the interaction affects the portfolio's value and resource consumption, respectively (see, e.g., Liesiö et al. 2008, Doerner et al. 2006, Carazo et al. 2010. The Still, identifying and quantifying the relevant interactions in the first place can be more challenging than dealing with the computational complications resulting from these interactions. The interaction parameters that affect the portfolio's value or resource consumption can rarely be estimated from hard data, even in PDA applications that build on extensive project data bases (e.g., Mild et al. 2015). Thus, quantifying interaction effects often requires judgemental estimates from multiple experts. PDA papers tend to be more focused on the technical modelling aspect of interactions and provide little detail on the process through which the expert judgements were obtained, nor do they discuss the accuracy or reliability of the obtained estimates. Still, there are some exceptions. For instance, Chowdhury & Quaddus (2015) report experiences from three applications in which portfolio models were used to select strategies for enhancing supply chain resilience. In these applications, semi-structured interviews of 2-5 executives were deployed to examine the pairwise resource synergies (cost savings) among 13-14 strategy options. These interviews helped identify 6-21 synergy effects ranging from 20,000 to 100,000 euros.

Multiple objectives/criteria/attributes
Two-thirds (67%) of the articles contained portfolio models with multiple dimensions of performance measurement referred to as objectives, criteria or attributes. Such models are needed, for instance, in order to describe how implementing projects builds the capabilities of organization (see, e.g., Doerner et al. 2006;Gutjahr 2011;Gutjahr & Froeschl 2013;Gutjahr et al. 2008Gutjahr et al. , 2010; to consider the multiple ways in which different new products contribute to the performance of the firm (Chen et al. 2007); or to account for the different beneficiaries of publicly funded projects (Almeida & Duarte 2011). does not include multi-objective optimisation approaches, which were frequently used together with one of the above methods. The types of optimisation models in the articles are discussed in more detail in Section 4.5.
Given this diversity of multi-attribute methods, one may ask if research in this area is motivated primarily by pure intellectual curiosity as to how a specific multi-attribute method might be used for portfolio selection.
However, this seems not be the case, as 70% of the articles presenting a real application or data featured multi-attribute portfolio models. Moreover, 82% of the articles involving actual decision makers also used multi-attribute portfolio models. Hence, the pervasiveness of multi-attribute models seems to be motivated

Uncertainty and risk
Portfolio decisions, like many other decisions, must often be made without knowing the decision outcomes for sure. From modelling and decision support perspective, a key question is whether these uncertainties need to be modeled explicitly, or if it is appropriate to build a model with deterministic parameter values.
The answer to this question is guided by at least three aspects. First, if the relationship between the model input and output has non-linearities, it is more important to model uncertainties. For instance, if there is a non-linear relationship between project and portfolio values (e.g., Baker & Solak 2011), then the expected portfolio value cannot (generally) be obtained by simply mapping the expected project values through the non-linear transformation. Second, when concerns of risk management and mitigation matter in the formulation of decision objectives, modelling uncertainties can be important, although less formal approaches such as risk scoring can also be used (e.g., Ferreira et al. 2015). Finally, the extent to which it pays off to model uncertainties depends on the available data and/or access to expert judgement: assessing a probability distribution, a fuzzy number or a set of scenarios requires more effort than a single crisp number.
Of the reviewed articles, 74% developed and/or applied portfolio models with a formal representation of uncertainties. Half of these articles employed probabilities (e.g., Sawik 2013a), and about half of these 14 considered probabilistic dependencies among the random variables (e.g., Bhattacharjya et al. 2013). Other approaches were also common, with 11% of the articles using fuzzy numbers/sets/distributions (e.g., Collan et al. 2013) and 16% mentioning the use of scenarios (e.g., Martinez et al. 2011). A fifth of all articles considered models with set valued parameters (cf. robust optimisation), and the use of dominance relations or worst-case analyses to evaluate portfolios' value and feasibility (see, e.g., Hassanzadeh et al. 2014).
Roughly one third of articles discussed risk-averse preferences, whereby the most popular approaches were based on constraining a selected risk-measure (e.g., Guo et al. 2012) or maximizing the weighted sum of the risk measure and expected portfolio value, with weights being used to control the level of risk-aversion.
Only a few papers introduced concave utility functions to capture risk-aversion (e.g., . Explicit models of uncertainties are relevant in practical applications: they were present in 73% of articles based on a real application and 64% of those involving actual decision makers. (2015) take a higher-level view and examine the optimal management policies of high technology projects.
In particular, they consider how resources available at each stage should be allocated between launching new projects, and evaluating and funding on-going projects.
Looking at the research on risks and uncertainties in PDA models, the topic of Value-of-Information seems to offer room for contributions that are of both theoretical and practical significance. In particular, Value-of-Information has received plenty of attention in the general DA literature as well as other fields (for a survey see Keisler et al. 2014). One can argue that because portfolio models have a much higher number of parameters than models for selecting a single alternative, the cost of obtaining estimates for PDA models will be higher. Hence, methods that help focus information acquisition efforts on the parameters whose accuracy matters most can expedite the decision process considerably. Moreover, Value-of-Information-models can provide general guidelines for information acquisition, which can be useful in portfolio decisions that are made without formal model based support.

Optimisation models and algorithms
Typical PDA analyses provide recommendations for the selection of projects or the allocation of resources.
Because these recommendations are usually generated through optimisation, it is instructive to examine what kinds of general optimisation models and solution algorithms were presented in the articles.
Mathematical optimisation models were formulated in 82% of the articles. Of these models 81% contained integer (or binary) decision variables and are thus non-convex. Perhaps more interestingly, almost half of the models were non-linear (e.g., Hosseininasab & Ahmadi 2015). This suggests that, although the roots of PDA lie in the capital budgeting literature and associated linear programming techniques, the PDA community is not limited to linearity; capabilities to deploy non-linear optimisation techniques exist and they are used when required. Moreover, the shares of linear and non-linear models do not change when considering only those articles which present a real application or involve actual decision makers.
In 56% of the articles, the optimisation models were solved using exact algorithms, i.e., algorithms that guarantee the identified solution to be (globally) optimal. These include sophisticated commercial and open source MILP solvers as well as brute force explicit enumeration of all possible portfolios (see, e.g., Eilat et al. 2006). The remaining articles deployed heuristic algorithms that identify a feasible portfolio satisfying all the constraints and having a 'reasonably good' objective function value without guarantees of optimality (see, e.g., Doerner et al. 2006). When examining only those articles which were based on real application or involved actual decision makers, the shares of articles deploying exact and heuristic algorithms remained roughly the same.
Roughly one third of the articles involved multi-objective optimisation models and deployed algorithms that generate a set of Pareto optimal solutions to this problem (see, e.g., Rauner et al. 2010). However, 38% of these models contained only two objectives.

Application areas of PDA
Practically all the reviewed articles present at least an illustrative application demonstrating how the developed approaches can be used in practice. In some cases, the applications are purely illustrative, while other articles discuss in detail how the actual decision makers where involved in the process and how the models and results were used by the client.
An important question is the degree to which the reported application is 'real'. Here we analysed with yes/no values whether (i) the application is based on data from a real application, (ii) actual decision makers were involved in, e.g., model building, parameter assessment, or utilizing the model results, and (iii) the article discusses the organizational context of the application. For instance, an article using real data from some other source would have a 'yes'-answer on the first question, but a 'no' on the remaining two. An article developing a new model and briefly reporting a real application to demonstrate that this model works would likely receive the answer 'yes' for the first two questions, but 'no' to the last one. An application or a case article would likely have the answer 'yes' to all three questions. It should be noted that articles do not always clearly state which aspects of the model were used in a real application. Furthermore, some real applications cannot be disclosed, which might lead researchers to publish a toy version of the real model.
Hence, our assessment should be interpreted with some caution, as we had to make some judgement calls in interpreting whether a particular application really involved actual decision makers.
In our review, we also seek to differentiate the applications based on their domain and model size.
In particular, these attributes differentiate applications along several dimensions capturing what is the industry or branch of government; for which organizational level the model is aimed; what are the portfolio elements (cf. projects, products, patents, systems) on which decisions are made and how many elements are considered; what is the monetary value of these decisions; and how many objectives and constraints the model includes.
Almost two thirds of the articles present results from or based on a real application or real data. Half of these involved actual decision makers (see, e.g., Vilkkumaa et al. 2018) and one third discussed the organizational context in which portfolio models were deployed (see, e.g., Montibeller et al. 2009). One might expect that PDA as a subarea of decision analysis, which, almost by definition, is an applied science, the share of real applications would be higher. However, our review focused on OR/MS journals, most of which have a strong methodological orientation. To get published in these journals, articles are usually required to make a novel methodological contribution. Articles reporting applications of existing models and methods in an innovative way and/or in a new application area, are more likely to be published in journals with a domain focus on, e.g., environmental management or the petroleum industry. Thus, in our view, the fact that two thirds of articles in the sample of the methodologically oriented PDA papers report realistic applications is a signal of a healthy balance between developing novel methodology, on one hand, and applying the results of such development, on the other hand. Figure 2 shows how the shares of used PDA methods, models and algorithms would change for those articles which report an application involving real decision makers. Overall, there is not much of a difference, which indicates that real applications do in fact utilize a variety of multi-attribute methods and optimisation models. However, although a large proportion of methodological research tackles portfolio problems with increased complexity (i.e., multiple resources, interactions, contingent decisions, uncertainties), the share of applications that actually exploit the advanced methodological capabilities called for by these complexities is somewhat lower.
Most applications had been carried out in one of a relatively small set of industries or application areas.
In 20 of the 148 articles, this area was hi-tech (e.g., electronics, computers, semiconductors and telecom; see, e.g., Gemici-Ozkan et al. 2010). Altogether 16 articles were in energy (e.g., power and electricity generation, energy policy, see, e.g., Cranmer et al. 2018), including 5 in oil and gas. There were 11 articles in health, and of these 7 focused on the pharmaceutical industry (see, e.g., Phillips & Bana E Costa 2007). Of the 8 articles of PDA tools for infrastructure asset management, half dealt with transportation infrastructures (see e.g., Mild et al. 2015). Other noteworthy application areas include e-commerce (6 articles; see, e.g., Finally, it is worth noting that 21 articles used the term 'R&D' to characterize their application area or the elements of the portfolio, but most still focused on a specific industry (e.g., pharmaceutical or ecommerce). Of the 148 articles reviewed, 23 reported applications for public or non-profit non-governmental organizations. This distribution of the application areas of PDA is not striking. Indeed, already Salo et al.
(2011a) noted the successes of PDA in oil and gas exploration, pharmaceutical drug development and R&D management, i.e., application areas which have discrete investment points and predictable processes through which uncertainties are resolved.
Figures 3 and 4 illustrate the differences in the PDA methods and models used across the 7 largest application areas. These figures suggest that the core elements of PDA models are quite similar across the different application domains. For instance, in all domains the share of portfolio models with multiple attributes/criteria is more than 43%. Multiple attributes/criteria were deployed in all articles with applications in infrastructure asset management, supply chain management and the military. This is intuitive, as, for instance, the monetary value of infrastructure maintenance projects can be hard to estimate, and hence the prioritization of such projects often relies on multiple measures on how the project improves the condition of the targeted asset (see, e.g., Gurgur & Morley 2008, Mild et al. 2015. The high number of multi-attribute/criteria supply chain applications is explained by the numerous applications which consider both expected monetary value and risk (see, e.g., Sawik 2013b). Also, the share of decision models that have other than resource constraints is 40%-70% across the application areas. The optimization models frequently involve integer variables, with the share of (mixed) integer models varying between 56% in energy and 100% in supply chain management.
However, the application domains are different in view of some characteristics. For instance, in military applications there are more project interactions than in other domains. This seems intuitive, because the effects of deploying, e.g., a new weapon system will depend on what other systems the current portfolio contains (see, e.g. Kangaspunta et al. 2012, Davendralingam & DeLaurentis 2015. Moreover, compared to the other domains, PDA applications in supply chain management use probabilities more frequently in order to capture uncertainties and to account for risk preferences (see, e.g., Sawik 2013b). In this domain portfolio models seem to be frequently formulated as mixed integer linear programming problems, which are then solved with exact algorithms.
The 'sweet spot' for PDA between 10 and 100 portfolio elements seems intuitive. If there are fewer than 10 elements, it may be less useful to consider the decision as a portfolio problem, and above 100 elements, it is more challenging to use decision analytic techniques that rely on the elicitation of expert judgements in scoring all prospective elements. With more than 100 portfolio elements, PDA tools often utilize existing project databases on top of which value/utility models are built. We conjecture that in the 10-30 range, the additional value brought by PDA stems more from characterizing projects, while in the 31-100 range, there is relatively more focus on the portfolio aspects of the problem.
The numbers of constraints and objectives are further measures of the size and complexity of the PDA models. Although not all articles report the number of constraints explicitly, rough estimates provide insights into the magnitude of this number. Most articles used a single constraint (33 cases; see, e.g., Barbati et al. 2018) or between 2 and 10 constraints (46 cases; see, e.g., Almeida & Duarte 2011), although many applications also used between 10 and 100 constraints (22 cases; see, e.g., Vetschera & de Almeida 2012).
Some articles did not explicitly model any constraints (14 cases; see, e.g., Lawryshyn et al. 2017), while some used more than 100 constraints (19 cases; see, e.g., Mild & Salo 2009). The number of constraints does not necessarily coincide with the number of resources, as there can be additional constraints for modelling multiple time periods with specified budgets or synergies between projects, for instance.
The considerable attention given to the methodological development of multi-attribute/criteria models is visible in the applications as well. Specifically, 98 of the 148 articles reported applications with multiple attributes (objectives/criteria). Bi-attribute models were used in 19 of these applications (see, e.g., Sampath et al. 2015), while 38 articles employed 3-5 attributes (see, e.g., Fliedner & Liesiö 2016), and 41 articles used more than 5 attributes (see, e.g., Doerner et al. 2006).
In order to give an indication of the importance of decisions supported with PDA tools, we set out to estimate the monetary value of these decisions; but this turned out to be difficult. First, complete data on, e.g., projects' values and costs is not always reported. Second, it is not always clear whether the data is real or if it has been generated for illustrative purposes, in which case the generated data may not retain the key characteristics of the real data for the purpose of estimating the monetary value of the decision.
To address these difficulties, we chose to report a monetary value as an estimate of the total expenditure of monetary resources (i.e., budget) at the time of making the portfolio decision. In some cases, we had to settle for the total cost of all project candidates, while in others we had to use the projected monetary value of the optimal portfolio. Although these measures are not fully correlated, they give a ballpark estimate of the significance of the decision. For instance, an application in which the aggregate cost of available 22 project candidates runs to millions of euros will hardly have a budget of 10,000 euros. Similarly, one cannot realistically expect a portfolio payoff of billions of euros with an investment budget of a hundred thousand euros. For 36 articles, we were able to produce a numerical estimate on monetary value, ranging from some hundred thousand euros (see, e.g., Tan et al. 2016) to several billion euros (see, e.g., Davis et al. 2016) with an average of a couple of hundred million euros and a median of around ten million euros. These statistics were approximately the same both for articles which reported an application based on real data and for articles which reported an application involving actual decision makers. Thus, these figures seem to give a realistic picture of the financial impact of PDA decisions in the sense that they are not overly inflated by the articles that report illustrative applications only. This wide range is interesting as it suggests that PDA methods are easily scalable.
6. Discussion: Strengths, weaknesses, threats, and opportunities of PDA We adopt here the well-known SWOT (strengths, weaknesses, opportunities, threats) framework as a convenient means of structuring our discussion of PDA and its prospects from several angles. We innovate by reversing the order of the final two items in order to close on a positive note, and by categorising opportunities separately for expanding the modelling frame and for advancing the mathematical foundations and computational capabilities. In contrast to business uses of SWOT -which aim to protect or grow a profitable enterprise -we deploy SWOT as scholars and practitioners with the motivation of giving indications for the further advancement of the PDA field. These strengths, weaknesses, threats and opportunities of PDA as a structured approach for solving portfolio problems are based not only on the authors' interpretation of the literature survey results, but also on their expertise and observations beyond the systematic survey. • Availability of models for specific classes of applications: For many frequently encountered PDA problems -such as the selection of R&D projects -the literature contains insightful papers which report useful experiences and distill 'lessons learned' which will be of value to those who address comparable problems (Section 5). This contextualisation is a clear strength for practical applicability, as it is only through the development of domain-specific case literature that evidence-based guidance for practice can be offered.
• Access to tool support: Many PDA models can be solved with the help of dedicated decision support tools or by relying on the usual functionalities of general purpose mathematical software tools. Only in situations in which the problem is either substantially larger or more complex -for instance due to the availability of a much higher number of projects or the presence of many more interdependencies between them (e.g., extensions of PDA problems to project scheduling) -the capabilities of standard mathematical software tools do not suffice for PDA problems.

Weaknesses
• Understanding of organizational success factors: While the PDA literature is rich in theory, methods and tools, it gives much less advice in guiding how the PDA apparatus can be best leveraged across different problem contexts which tend to exhibit considerable variability in terms of the roles, interests and responsibilities of decision makers; the availability and quality of relevant information; and the demands on the decision in terms of requirements such as speed and transparency. This may reflect the fact that most OR researchers seek to contribute primarily to the advancement of their own field (with ensuing emphases on theory, methodologies and applications). However, the implication is that the processes of deploying PDA involve a fair amount of tacit knowledge which often accumulates through experience. Most notably, the PDA literature is scarce in systematic empirical studies on the success factors of PDA interventions, even though such studies are common in neighbouring fields such as product portfolio management (see, e.g., Kester et al. 2014). This scarcity seems to apply not only for the selection of articles in this review but, based on the authors' experiences, also for PDA articles published in more application-oriented journals outside the core OR/MS literature.
• Availability of efficient computation techniques: In the more complex and specialized applications, the attainment of the guaranteed optimal solutions may still require a prohibitive amount of computational resources, despite the phenomenal growth in computing power over the last few decades. Fortunately, many portfolio problems can be adequately tackled through heuristics (Section 4.5). Yet, the lack of generic computational tools that would not have to be tailored for each problem separately may limit the usability of more complex models by less technically-minded analysts.

Threats
• Methodological fragmentation: As the literature on PDA problems continues to grow and becomes more voluminous within different methodological sub-fields and application areas (Figures 2, 3 and   4), there is a possibility of increasing fragmentation which slows down the prompt diffusion of useful 24 methodological advances across application areas. There is a clear role for action through the relevant professional societies, such as EURO and INFORMS, to provide fora for mutual and cross-sectoral learning that helps researchers to take full advantage of the methodological diversity.
• Ad hoc risk and multi-criteria preference models: As a rule, much of the work in PDA rests on solid theoretical and methodological foundations, anchored in decision theory and mathematical optimisation.
Yet there is a substantial share of the literature which draws upon the more contentious approaches (Sections 4.3 and 4.4). While such approaches can serve a useful role in improving organizational decision making, it is important to fully understand their theoretic properties. Our personal predilection is for methodologies with a strong decision theoretic foundation. This is the case especially in regulatory decision making in which demands on the rigor of approaches are stringent. Here, the quality of information concerning alternatives and their outcomes can be analysed to determine how such information contributes to the performance of the portfolio (Kettunen & Salo, 2017), and at what point(s) it is optimal to take decisions about different kinds of alternatives.
• Broadening the aims of decision making: While the maximization of expected benefits subject to given resource and other relevant constraints is the most common objective in PDA application, this objective can nevertheless be questioned and, at times, be even replaced by other objectives, such as improving the efficiency of the selected portfolio (see, e.g., Liesiö et al. 2020). Furthermore, in the context of safety-critical systems, one of the rationales for PDA studies is that of ascertaining how much resources are needed to achieve the required threshold level of reliability. In short, extending the range of purposes and objectives for which PDA can offer useful inputs may suggest promising avenues for future work.
• Modelling of stakeholder preferences, interests and beliefs: With some notable exceptions, PDA models have been formulated assuming that it suffices to build a decision model which portrays the values, objectives, and attributes of a single decision maker. However, in many application areas -of which environmental decision making is a good example -portfolio decisions need to be reached in a dialogue with multiple decision makers. Even in the corporate world where firms are increasingly interdependent, the explicit modelling of the interests of the different parties may be useful in generating suggestions for 'win-win' solutions. Also the implications of different beliefs for the composition of the optimal portfolio can be systematically explored to generate insights (for an example, see Baker et al. 2020).
• Exploiting synergies between PDA, project portfolio management and technology management: While there are considerable synergies between them, these areas of research and practice have had distinct evolutionary paths with rather modest interaction. Multidisciplinary research spanning these areas holds great potential, given that such research may provide empirically grounded evidence about in what situations and under what conditions the use of PDA methods can be particularly useful.
Conversely, such research may provide insights into the inherent limitations of any formal approach in situations in which the decisions may be governed more by 'ad hoc' political pressures than the rationales of rational decision making.
• Developing and deploying dynamic PDA models. For classical decision analysis problems, in which one from a set of mutually exclusive alternatives is selected, decision trees offer a general tool for supporting decision making over multiple time periods that also captures the information available when each decision is made. The literature review suggests that there is room for contributions that would introduce similar tools into the theory and practice of PDA, especially in view of the popularity of dynamic models in financial portfolio applications. Here, we believe that a key challenge is to accompany rigorous model developments with high impact applications. In particular, such applications would demonstrate that enough data or expert judgements can be realistically obtained in a real-life setting to estimate the higher number of parameters in dynamic PDA models.

Opportunities for deepening mathematical and computational understanding
• Nurturing interfaces to complementary methodologies: Because PDA models are often solved with mathematical programming, there are noteworthy synergies at the juncture of PDA methods and mathematical optimisation, given that advances at this juncture can be leveraged to solve either larger problems or to tackle previously intractable problems. Such synergies may be particularly significant in PDA applications where the required solution is not limited to the static selection but, rather, is expected to guide the tentative scheduling of projects as well, as such problems tend to be large. Furthermore, tools such as simulation or Bayesian networks can prove useful for assessing the consequences of projects.
• Development of libraries for tool support: In some application areas -such as the 'plain vanilla' selection of projects within the MAVT framework -PDA models have considerable similarities. While this has made it possible to develop specialised software tools, these dedicated tools do not usually offer sufficient functionalities for tackling more complex problems involving probabilistic chance constraints and several different kinds of interactions, for instance. Thus, the PDA field would benefit from modifiable software tools and libraries on top of widely used mathematical software environments, as these would facilitate the uptake of the more advanced PDA capabilities as well.
• Analytics, machine learning and data sciences: Advances in analytics, data science and machine learning provide opportunities in that they, among other things, help produce usable information about the alternatives from which portfolios are built. Depending on the problem context, these advances can serve to alleviate one of the hurdles which may have impeded the proliferation of PDA approaches, i.e., the need to spend a considerable amount of time and effort to assess how the individual alternatives (e.g. projects) perform with regard to the evaluation criteria.
• Introducing PDA concepts into modelling tools: At its core, PDA hinges on generic core concepts (e.g., objectives, alternatives, constraints, Pareto optimality) which are not limited to the standard project portfolio selection problem; in effect, these concepts can be readily embedded into other problem representations. This has, for instance, propelled the development of methods for improving the reliability of safety-critical systems by merging PDA concepts with the use of Bayesian networks as a representation of industrial systems (see, e.g., Mancuso et al. 2017).

Conclusions
Portfolio problems -the problems which are amenable to PDA -are pervasive and important in most, if not all organisations, even if they are not always explicitly recognised as such. Unlike much of what are commonly thought of as 'OR problems', portfolio problems have relatively modest and hard-to-specify formal structure, and precise measurement of key parameters is often not possible -indeed that is one of the motivations and challenges for an analytic approach.
Yet, as we have aimed to demonstrate in this paper, the relatively modest level of structure of these problems is not the same as no structure: there are useful concepts (budgets, benefits, interrelationships, the incumbent portfolio, decision areas) and tools (e.g bubble plots, triage plots and Pareto plots) with a wide reach. This also means that there are possibilities for cumulative learning. Indeed, one of the heartening things which we see in this review is that this learning is taking place: the level of methodological sophistication, and the complexity of problems being tackled, is greater in the current literature than in the literature of a decade or two ago; and this learning is taking place across a wide range of different application domains, and by researchers experienced in multiple methodologies.
As our discussion section of this paper demonstrates, although we, as researchers centrally engaged in this area for several years, are proud of what has been accomplished, though with some regrets and reservations, the main thing we take from this review is renewed enthusiasm about the many opportunities for further work in this area. In keeping with the general ethos and tenor of Decision Analysis, Decision Aid, and indeed OR more generally, we believe that exploiting these possibilities will add to our understanding both of the mathematical structure of human decision, and the social science of the management of human organisations.