Renewable and Sustainable Energy Reviews Risk-based methods for sustainable energy system planning: A review

The value of investments in renewable energy (RE) technologies has increased rapidly over the last decade as a result of political pressures to reduce carbon dioxide emissions and the policy incentives to increase the share of RE in the energy mix. As the number of RE investments increases, so does the need to measure the associated risks throughout planning, constructing and operating these technologies. This paper provides a state-of-the-art literature review of the quantitative and semi-quantitative methods that have been used to model risks and uncertainties in sustainable energy system planning and feasibility studies, including the derivation of optimal energy technology portfolios. The review ﬁ nds that in quantitative methods, risks are mainly measured by means of the variance or probability density distributions of technical and economical parameters; while semi-quantitative methods such as scenario analysis and multi-criteria decision analysis (MCDA) can also address non-statistical parameters such as socio-economic factors (e.g. macro-economic trends, lack of public acceptance). Finally, untapped issues recognised in recent research approaches are discussed along with suggestions for future research.


Introduction
Global investment in renewable energy (RE) in 2015 increased by 5% to $285.9 billion in relation to 2014, surpassing the last record of $278.5 billion in 2011 [1]. The annual increase in power capacity has also reached its highest level across all regions in 2015. Wind and solar photovoltaics (PV) account for an approximately 77% of new capacity, with hydropower accounting for most of the rest [2].
As the number of RE investments increases, so does the need to measure the associated risk and uncertainty from the perspective of different stakeholders throughout planning, construction and operational phases [3]. Energy developers, investors and policy makers face a future that implicitly involves technological, financial and political risks and uncertainties. Although, RE technologies potentially have a lower risk profile than conventional energy sources because they are disconnected from fossil fuel prices, they still entail considerable technological, financial and regulatory risk exposure, depending on the technology, country and regulatory regime. Fluctuation of cost components of power generation units, volatile crude oil prices, 1 electricity price and carbon costing in the context of the global climate change mitigation strategy, are examples of uncertainty components encountered by energy developers, investors and policy makers investors in the energy sector [4]. Often these risks are mitigated by governments in the form of price protection, but this can have a large budgetary burden, which often passes on to consumers through taxes and electricity bills [5].
Another stream of studies has focused on the identification and assessment of risks and uncertainty, as well as risk management solutions for sustainable energy projects [3,7,8,[17][18][19]. In general, risk in the power generation investment sector is considered to be multi-dimensional and depends on the perspective of different stakeholders [9]. An array of analytical methods has been used to analyse various aspects of risk from the perspectives of different stakeholders. This results in a bewildering mix of studies that look at different sides of the same problem. However, there has been no systematic review of which techniques are most appropriate for reviewing individual, or groups of risks and how useful the outputs are to various stakeholders.
The aim of this paper is to provide an extensive, systematic literature review (SLR) of how risk and uncertainty has been analysed with respect to sustainable energy system planning. This will focus on identifying the attributes of risks (or modelled uncertainties) that each analytical method is most suited to address, as well as a critical comparison of the main outputs of such studies. The outputs of this review will map appropriate analytical techniques to specific risks, as well as comment on their application from the perspective of different stakeholders. The outputs are intended to provide a guide to researchers as to common practice in the assessment of risk and uncertainty for sustainable energy developments as well as indicating any possible gaps or new avenues for research.
The rest of this paper is set out as follows: Section 2 presents an overview of risk/uncertainty factors affecting investment decisionmaking in sustainable power generation planning and feasibility studies, along with an overview of the different perspectives among stakeholders. The risk-based evaluation methods are introduced in Section 3, and the cross-method comparison is conducted in Section 4. Finally, Section 5 summarises the findings of this work and suggests some focal points for future research.

Overview of risks and stakeholders' perspectives in sustainable energy generation systems
Risk in the power generation investment sector is generally considered to be multi-dimensional and depends on the perspective of different stakeholders. The "Comprehensive Actuarial Risk Evaluation -CARE" paper produced by the International Actuarial Association (IAA) provides a comprehensive taxonomy of risks faced by enterprises [9]. Among other classification schemes, the paper suggests a new perspective for risk categorisation into statistical and nonstatistical risks. The former are the risks that can be measured or modelled with mathematical or statistical methods, such as stochastic modelling, while the latter are those that are difficult to model with existing knowledge. 2 Risks associated with sustainable energy projects depend largely on a number of factors that are technology-, country-and regulatoryspecific, while they also vary according to different stakeholders' perspectives. Authors working on risk identification, analysis and management in the sustainable energy investment sector have developed different risk categorisation schemes according to their intended focus. Table 1 summarises the most cited risks by employing a political, economic, social, technology, legal and environmental (PESTLE) approach.
Stakeholders involved in the field of RE investments comprise: project developers, project investors, insurers, manufacturers, consumers, affected local communities and policy makers. Each stakeholder tends to have different concerns and objectives from renewable energy investments. This means that risks will vary in importance across these different groups.
From a project developer's perspective, the objective is to make a sufficient return on investment (capital and other resources) through the sale of an RE project to an investor [12]. Investors are mostly interested in minimising risks of technical reliability, costs and risks of revenue disruption [14], while policy makers are concerned with designing efficient and effective policy schemes, which would provide the appropriate level of incentives to potential investors of RE projects that allow government targets to be met [15]. As such, risk analysis in RE projects has been performed in a generalised style covering numerous RES technologies and stakeholders' perceptions by some authors [6,[16][17][18][19], while others distinguish risks through the related stakeholders' perspective (e.g. from the investor's and developer's view) [20] or by technology-specific risk factors [3,21].

Results of the literature review
Studies in this area tend to focus on the analysis of specific risk(s) from the perspective of a stakeholder or stakeholders. Therefore, the results section will map this research area in terms of which risks have been analysed by which methods and which stakeholders have been included.

Overview of the methods
The literature review was conducted on the basis of a SLR approach, which provides the synthesis of the research in a systematic, transparent, and reproducible manner, while also restricting the researcher's bias [22]. A description of the main steps followed to conduct the SLR approach is summarised in Appendix A. Analysis of the SLR results finds several methods used in the analysis of risk involved with sustainable energy generation systems. Table 2 provides a tally of how many times a paper using a particular method was identified by the systematic review process. This paper takes these methods forward for further analysis. As indicated in Appendix A, the total number of references considered for the review was 161 out of which, 113 originated from the SLR process, while the rest 48 references were identified through additional checks (e.g. via citation tracking or journal websites searching) in order to complement information on a particular topic which was not fully covered by the systematic review.
The review focuses on critically assessing which risks have been analysed by which methods, what are the common outputs of these methods and which stakeholders have been included in a number of widely cited representative risk-based methodologies applied in sustainable power generation planning and feasibility studies. These methods have been classified, for reasons of simplicity, into quantitative and semi-quantitative methodologies (see Fig. 1).
Quantitative risk-based evaluation methods deal with (statistical) risk factors that can be described by probability distributions. Widely cited methods falling into this category are: Mean-variance portfolio (MVP) theory, Real options analysis (ROA), stochastic optimisation methods, and Monte Carlo simulation (MCS). Semi-quantitative methods have the flexibility to take into consideration statistical and nonstatistical risks. Semi-quantitative methods that were identified through the SLR are: MCDA and scenario analysis. Table 3 matches the risk-based methods with risks/uncertainties as identified by the systematic review. The table can potentially provide guidance as to what methods are most suitable to address/model the specific risk and uncertainty factors listed.

Mean-variance portfolio analysis (MVP)
MVP is an established method of economic theory, based on the pioneering work of Harry Markowitz, who focused on the diversification of securities towards the construction of efficient portfolios, which would correspond to high expected return and low variance [97,98]. Later, Awerbuch [51] applied MVP for deriving optimal (or efficient) energy generation portfolios yielding maximum expected return in combination with minimised risk.
An energy generation portfolio constitutes a mix of generating assets put together to reduce total investment risks; as such, an efficient portfolio of energy generation technologies (with higher RE shares) reduces the threat of abrupt supply disruptions, hence reinforcing energy security through the mitigation of volatile fossil fuel price dependence.
Diversifying the power generation portfolio has been highlighted by a number of authors [18,20,[99][100][101][102] as an effective strategy of risk hedging due to the creation of portfolio effects resulting in efficient power generating portfolios (i.e. optimum shares of different energy technologies in the portfolio resulting in a minimum level of risk towards attaining a given generating-cost objective). Diversification dimensions may be geographical, technological or value chain related. Numerous reports by international agencies, organisations, as well as 2  scientific papers [23,24,49,51,55,[103][104][105] have stressed the importance of de-emphasising stand-alone energy generating costs and levelized cost assessments in generation planning, since these ap-proaches do not capture the contribution of renewable and non-fossil fuel technologies to the electricity portfolio, in terms of reducing the variability of electricity costs and hence their impact on economic activity. At any point, some assets in the energy generation mix may have higher costs than others; yet, in another instance, the combination of alternatives serves to minimise overall expected generating cost relative to the expected risk. Portfolio risk is usually measured as the standard deviation of historic annual outlays for fuel, operation and maintenance (O & M) and construction period costs examined on the basis of historical data [50]. Numerous papers have attempted to generate models that consider risks as the cost variance of a technology portfolio [23,[49][50][51][52]103,[105][106][107].
Huang and Wu [52] introduced portfolio risk by means of volatile fuel prices and uncertainty of technological change and capital cost reduction, while another MVP paper deemed market electricity prices and wind    Table 3 Risk and uncertainty parameters 1 addressed by risk-based methods. Risk and uncertainty factors addressed by the outlined methods may be summarised as:

Risk-based methods
• Climate change policy risks mainly include fluctuations in CO 2 prices and reduction targets, changes in the climate change policy schemes (e.g. (retrospective) changes in RES subsidy/promoting policies).
• Variation in power generating costs may include variation in pre-development costs, fixed and variable operational and capital costs of the power generation technology.
• Financing/fiscal risks reflect the uncertainty in the financing of the power generation investment, variation in taxes and interest rates, sales and revenues as well as variation in the investment profitability (e.g. variations in IRR).
• Market risks are referred to in the literature usually by means of variability of revenue due to uncertain electricity market prices and fluctuations of electricity demand.
• Fuel risks usually capture variations in fuel prices, in fuel production, in the fuel/output ratio, disruptions in fuel delivery/supply, as well as fuel transportation risks in power-plant operation.
• Macroeconomic parameters mostly seek to reflect uncertainty in macroeconomic metrics, such as inflation rate and GDP.
• Social risks can potentially involve risks associated with the lack of public acceptance, as well as health risks (e.g. occurrence of accidents).
• Technical risks involve lack of access to the grid, construction risks, reliability of components (e.g. damage to turbines), variation in capacity factors, and unavailability of power plants and skilled labour.
• Technological/Innovation risks relate to cost uncertainties due to learning curve effects.
• Resource/power output risks can be associated with revenue loss due to intermittency, availability of natural resources, physical supply disruptions, curtailment of power generation sources and/or electric power produced (including intermittency of RES).
• Environmental risks may entail global warming (GHG emissions) effects, environmental damages (e.g. CO 2 emissions) and natural hazards.
A. Ioannou et al.

Renewable and Sustainable Energy Reviews 74 (2017) 602-615
resource availability as uncertain inputs represented by probability distributions with approximately normally distributed probability functions to compare the relative attractiveness of investing in a wind park under two RE policy support instruments, namely, feed-in tariffs (FiT) and feed-in premiums (FiP) [25]. Adopting a private investor's perspective, some authors have used cash flow models to calculate risk in terms of earnings, costs of O & M, credits, depreciation of facilities, and benefits [49,62,108]. Muñoz et al. [62] used the Internal Rate of Return (IRR) to represent the returns on investments, while the associated portfolio risk was reflected by the standard deviation of IRR. IRR proved to be a useful measure of the return from the real project, capable also of considering the uncertainty in electricity prices and future subsidies (introduced as stochastic inputs in the cash flow model). Roques et al. [109] concluded that in the absence of long-term power purchase agreements, optimal portfolios for a private investor are significantly different from socially optimal portfolios; since, from a private investor's viewpoint, there is little diversification value in a portfolio of mixed technologies, due to the high empirical correlation between electricity, gas and carbon prices. Bearing the above in mind, MVP theory is a method well suited to the problem of electricity generation portfolio planning and evaluation at a national and regional level (hence from a policy maker's viewpoint), since it can be used to derive efficient power generating portfolios, which reduce generating costs and enhance energy security, while the method has also been used to assess the maximum losses (or returns) of a private investor's (portfolio) investment within a specified confidence level.

Real-options analysis (ROA)
ROA is particularly applied to the analysis of the impact of uncertainty on investment decisions when management actions can be timed flexibly. This enables the investor to evaluate available options and take capital budgeting decisions (such as deferring, abandoning, expanding, staging, or contracting) as new information arises and uncertainty about market conditions and future cash flows is reduced [110]. ROA supplements the information provided by static discounted cash flow analysis and is based on the concept that it may be preferable to postpone irreversible decisions (e.g. in capital intensive investments) and wait to make a better informed decision at a future point in time [109]; hence, adding the ability of an investor to respond dynamically to changing market conditions. Common applications of ROA in low carbon energy projects include investigating the impact of climate policy uncertainty on private investors' decision-making in the power sector [33][34][35][36]111], such as the diffusion of various emerging RE technologies [73] or the investment timing and capacity choice for RE projects [33].
In more detail, [33] adopts ROA to analyse the flexibility of the investment timing (based on the investor's right to postpone investment once the licence is granted if the economic environment is not as favourable as desired) and capacity selection for RE projects under two different subsidy schemes (feed-in tariffs and RE certificate trading), by examining investment behaviour under these conditions. The option of investment timing and capacity choice is assessed taking into account the special characteristics of RE sources (wind power, solar power, and run-of-river hydropower), namely the intermittency of these power sources, as well as the uncertainties in capital costs, subsidy payments and electricity prices. Kumbaroğlou et al. [73] presented a policy planning model based on the ROA method featured through a dynamic programming process for recursively evaluating a set of investment alternatives on a year-by-year basis under uncertainty. They used the operational and cost data for existing power plants, electricity price data and capacity expansion structure, in order to derive annually added capacities and technologies from 2006 up to 2025 under different scenarios. The dynamic programming model allowed them to check the impact of uncertainty and technical change on the diffusion of various emerging RE technologies, concluding that market actors need, in the short-term, financial incentives to achieve a more widespread adoption of RES technologies in the longer run.
Other applications of the method focus on the impact of market uncertainty on investment electricity industry decision-making. Market uncertainty is expressed into stochastic CO 2 prices and policy uncertainty [36,55,111]. Authors in [36,111] emphasise the distinction between uncertainty coming from fluctuations in CO 2 prices around a known trend, which would arise in a market with emissions permits, and uncertainty emanating from the absence of clear policy signals. It has been shown that some market uncertainty may induce earlier investments in carbon capture and storage (CCS) equipment than in the case of perfect information. However, policy uncertainty may also lead to prolonged accumulation of CO 2 emissions in the atmosphere, since investors prefer to wait for the final decision of government before investing in climate change mitigation technologies. Hence, a clearer, long-term policy plan would leverage emission abatement actions. In both [34] and [35] the uncertainty is represented by carbon price uncertainty, which is modelled through stochastic variations in the carbon price. Results from Blyth et al.'s work [34] demonstrated that such uncertainty creates a risk premium for electricity investments which needs to be offset with extra incentives in order to overcome the effects of uncertainty on the timing of the investment decision. An important conclusion of their work suggests: the shorter the time before a future climate policy event, the higher the impact of climate change policy risks on the investment decision (a conclusion also reported in [35]). It is thus concluded that the method can derive useful outputs for both investors and policy makers. On the one hand, investors can evaluate available options and take capital budgeting decisions on the best timing; on the other hand, policy makers could be assisted to better understand the impact of market uncertainty (e.g. costs induced by an environmental policy) on the investment decisions of investors.

Stochastic optimisation techniques
Stochastic optimisation has been extensively used in a number of energy planning and feasibility problems, such as the determination of optimal energy mix planning at a national level (i.e. Indonesia [26], China [112], Korea [29], and Croatia [113]), expansion planning of sustainable energy systems [65,69,82,[114][115][116][117][118][119], design of hybrid systems [120,121], and numerous others energy systems-related problems like unit commitment, energy storage management, bidding energy resources, pricing electricity contracts [122], introducing uncertainty in one or more of the input parameters subject to stochasticity. In this review, we focused on problems that are associated principally with the deployment of stochastic optimisation methods in investment planning decisions. Usually, the constraints considered in these problems depend on the perspective of the stakeholder. As such, studies looking at the problem from a policy maker's perspective, seek to develop least-cost optimisation models to allocate energy sources for sustainable development, under constraints such as energy security (demand), renewable penetration, satisfaction of greenhouse gas (GHG) emission reduction targets, budget constraints and maximum technology capacity [26,30,112]. An investor would aim at minimising both the cost (or alternatively maximising the revenues) and investment risk (e.g. by minimising CVaR measure), while the potential constraints would further include risk-aversion constraints [70,83,123,124]. Uncertainties that are usually represented include market electricity prices, fuel prices, production costs of existing and future power plants, CO 2 emission policy, energy demand, technological efficiency, and utilisation factors [26,30,112]. Stochastic optimisation problems are characterised by an array of fragmented modelling approaches, such as fuzzy, (dynamic) stochastic and interval mathematical programming [125], often leading to inconsistent and inaccurate results [122].

Monte Carlo Simulation (MCS)
MCS involves the random sampling of probability distributions of the model input parameters with the purpose of producing numerous scenarios. The sampling from each parameter's probability distribution is realised in a way that reproduces the shape of the output distribution; hence, the distribution of the values deriving from the application of the method reflect the joint probability distribution of the outcomes [126]. MCS offers many advantages but it also requires a considerable range of data as input variables, such as the probability density functions of uncertain or fuzzy values or forecasted variables. There are numerous studies performing risk analysis of sustainable energy systems with MCS in the literature [56,57,59,63,89,92,127,128]. Existing works disclose a number of advantages of the method, such as the ability to obtain fast results when modifying the variables of the problem, the ability to calculate the risk undertaken because of uncertain or stochastic input variables, as well as the ability to model the correlations and other interdependencies of the system. Input variables need to be statistically independent; otherwise the simulations will lead to inaccuracies and shortcomings in the interpretation of the results. In studies employing MCS, the best fitting probability density function (PDF) assigned to the input variables is determined either by using historical data of the variable (statistical or experimental methods) [5], or by using subjective judgements (e.g. performing interviews with experts) on the empirical worst, base and best case estimates (confidence intervals) usually interpreted as quantiles of a probability density function [57]; most often, both methods are used in order to derive the PDF of numerous variable inputs [56,89,128].
Studies performing stochastic financial risk analyses of sustainable energy systems by means of the MCS method tend to derive joint probability distributions of annual energy production and investment profitability metrics (i.e. net present value (NPV), IRR) at a plant level [92]. For the selection of input variables, a sensitivity analysis method can initially be carried out for checking the effect of a number of potential input variables on the NPV. Risks/Uncertainty factors that have been taken into consideration include fluctuations in wind resource potential, wind curtailment, access to the grid and macroeconomic parameters [89]. MCS integrated in a typical financial model can assist investors to perform a first exploratory analysis to decide whether and where to invest and policy makers to assess policy parameters and explore possible scenarios of investing in an RE technology. For example, Pereira et al. [57] evaluated the risk in project implementation, under stochastic equipment costs, market financial conditions, O & M costs, and policy implications. They considered as independent variables the total initial costs, the interest rate and the value of energy produced and sold to the grid or utility; matching them with exponential, triangular and Bradford probability distribution functions, respectively, while NPV and the produced energy cost have been defined as the dependent variables.

Semi-quantitative methods
Along with the quantitative risk-based methods dealing with statistical risk and uncertainty in decisions associated with sustainable energy planning and feasibility problems, scenario analysis and MCDA have been identified by the SLR as methods that can consider nonstatistical risks.

Scenario analysis
The potential impact of risks on the profitability of RE investments can be evaluated by the discounted cash flows under various scenarios, reflecting different potential future developments. A scenario incorporates the dynamics and the drivers resulting in a specific conceptual future [129]. Usually, these scenarios represent either the most probable situations (situations that are most likely to occur) or extreme cases (worst-case, and best-case scenarios). Each scenario usually assumes values of elements, such as the future price of electricity, CO 2 costs, and produced electricity among others. The elements used for the construction of the scenario depend on the area on which the researcher seeks to focus [129].
Scenario analysis can potentially assist the planning of robust energy technology portfolios that will achieve set objectives under a range of future scenarios [42,76,130]. For example, [42] considered three scenarios, reflecting strong, mediocre and poor technological breakthrough and policy support for the development of the RE industry. This allowed the encompassing of uncertainties with regard to the relationships among the technology alternatives and the decision values of elements. The latter were divided into two dimensions: the importance of each technology (assessed through the market value, and the compound market growth) and the technology risk (indicators considered were the position of the technology and the manufacture capability). Conclusively, technology portfolio planning implications were derived for each of the three scenarios generated. On the other hand, Kannan [130] investigated the uncertainties in the future UK power generation mix via a range of power sector-specific parametric sensitivities under a 'what if?' scenario analysis framework, to provide a systematic exploration of least-cost energy system configurations, while [76] investigated the impact of energy price uncertainties on the supply structures of four EU countries using a stochastic risk function incorporated into a partial equilibrium energy systems model. Scenario analysis has also been used for the quantification of policy risks in the wind power industry [131].

Multi-criteria decision analysis (MCDA)
MCDA is a family of decision support methods which has been widely used in the energy sector and specifically in the evaluation of alternative energy sources as well as the consideration of risk perceptions, due to their ability to incorporate multiple actors' opinions, bringing along multiple different criteria, stemming from the political, economic, social, technological and environmental context [13,[132][133][134][135]. MCDA methods rely on relationships such as priority, outranking and distance among the alternatives and factors (i.e. criteria) that influence the decision. These methods are categorised as semi-quantitative since they can also accommodate criteria or attributes whose numerical values are hard to obtain or even cannot be quantified (intangible criteria) through the deployment of qualitative scales (i.e. a Likert scale) [136]. An example of a work using both quantitative and qualitative attributes can be found in [137]. Several authors have carried out reviews on MCDA methods with applications in the field of sustainable energy systems [132,138,139].
A few common outputs of these applications associated with sustainable energy generation technologies when risk and uncertainty is embedded in the investment decision, include: evaluation/ranking of the different RE technologies according to a number of risks/criteria [90,136,140,141], prioritisation of feasible projects through a risk analysis process [46] and risk prioritisation of RE technologies [13].
Types of uncertainty encountered in such problems stem from either the inherent valuation uncertainties (i.e. problem-specific technical parameters determined by the decision maker) or from the technical empirical uncertainties related to the data (such as the carbon emissions and technology costs) which are outside the decision maker's control [86].
Apart from the basic MCDA methods which are usually set to assess the strengths and weaknesses of the pre-determined energy options without re-defining them, another group is the continuous MCDA models seeking to identify the optimal design of the option. These methods are usually employed to deal with problems comprising multiple (usually conflicting) objectives, where decision variables are infinite variables, subject to constraints and are known as multiobjective optimisation methods. These methods have also received considerable attention in sustainable energy applications [14,47,85,86,93,142]. Goal programming is a category of multi-objective optimisation methods assimilating LP to handle problems with multiple, potentially conflicting objectives. For example, goal programming can be used to address the compromise between the cost per kWh of an electricity generation portfolio and the total risk for an investorowned utility [14]. A common application of the method in the field of sustainable energy system planning is to forecast optimum RE supply percentages under different conditions of portfolio risk and cost [14,83,143]. For example, in [14] the authors presented a multiobjective model for determining the share of different energy generation assets in an investor-owned utility portfolio that reduces risk while providing the lowest cost per kWh of electricity generation possible. The failure mode and effects analysis (FMEA) was employed to assign risk priority numbers (RPNs) to each risk. Subsequently, the share of each type of energy (i.e. solar, coal, and natural gas) in the mix was determined through a multi-objective model for the minimisation of levelized cost of electricity (LCOE) and minimisation of the aggregated RPN of each technology.
It is often encountered that the numerical values of the criteria or attributes are not easy to obtain and there is therefore a need to express them in linguistic terms. In this case, fuzzy logic is employed to address the uncertainty in human judgement by applying membership functions to vague information. There are numerous studies in the literature using fuzzy analysis in energy planning [61,[144][145][146][147][148][149].
As mentioned above, we recognise that there are also other methods dealing with risks and uncertainties in investment decision making; for example, parametric sensitivity analysis can be employed to identify sensitive input parameters (focusing on uncertainty in technical empirical parameters) by analysing their effects on the model output [86]. However, here we focus our review on methodsexported through an SLRwidely implemented to solve planning and feasibility problems seeking to investigate: the risks/uncertainties each method is best suited to cover, the stakeholder perspective each method addresses; while also critically assess their most common outputs and reveal advantages/disadvantages regarding content and methodology.

Combinations of quantitative and semi-quantitative methods
Methods described above are frequently combined with each other or with other methods in order to produce different kinds of results, e.g. in ways that the output of the one method works as the input for the other method. Subsequently, we present indicative papers combining different risk-based methods in the field of energy system planning and feasibility.
A number of studies have combined ROA with portfolio theory in order to derive optimal portfolio strategies towards meeting specific climate change stabilization targets under different socio-economic scenarios [37,38]. Fuss et al. [37] employed the real options model, in order to analyse the impact of uncertainty on investment decisions at the plant level. The Greenhouse Gas Initiative (GGI) Scenario Database was considered as a starting point for obtaining optimal technology portfolios which are robust across a number of socio-economic scenarios and across climate change targets. In [38], a multidimensional table indicating the best option (regarding the retrofit of a fossil fuel-fired plant and a biomass plant with CCS units) for each time period, possible state and possible carbon price realised during that period was produced. The implementation of the ROA resulted in the distribution of coal, gas, and biomass technology costs (for given parameters on fuel and CO 2 prices), which subsequently entered a portfolio optimisation model to provide the optimal strategy across all possible scenarios.
Methods employing portfolio theory are usually combined with optimisation methods, such as linear programming (LP) to determine optimum RE technology percentages under different conditions of portfolio risk and cost. Bhattacharya and Kojima [5] used the method of MVP risk analysis to create experimental electricity supply portfolios with high diversity (more fuel choices) and conducted a special type of optimisation method, namely simulation optimisation, in order to incorporate the various stochastic variables in their model so as to minimise the risk of the supply portfolio. The major sources of risk that were identified during the development and operation of power projects in Japan were the variation in capital costs, fuel costs, O & M costs, along with the price of CO 2 traded in the world market. Kumar et al. [105] determined optimum portfolios through the minimisation of portfolio fuel cost, portfolio fuel risk and CO 2 emission by employing a multi-objective genetic algorithm. They concluded that the limitation of the MVP theory from the perspective of a developing nation such as India lies in the fact that the method only considers risks associated with cost components while neglecting barriers associated with the implementation of projects; thus, a comprehensive risk barrier index is needed to indicate the combined impact of risks and implementation barriers associated with each portfolio.
A number of studies have combined scenario analysis with other methods as a way to incorporate uncertain situations emerging from political, economic, environmental, technological and environmental futures. Such methods include: portfolio theory [23,24,43,52,103], ROA [33,37,38,73], energy system modelling [76,130] and MCDA [148,150]. The latter study concerns the application of multiple criteria decision analysis to prioritise investment portfolios (with the overall objective of the generation mix corresponding to the anticipated electricity demand while fulfilling specified constraints), while at the same time testing the robustness of the prioritisation against several scenarios. Each portfolio reflects the distribution of the alternatives' power generation capacity denoted as X p p =[ , …, ] where p ki is the proportion of each energy asset capacity of portfolio X i to be gained by alternative a k belonging to a set A a a =[ , …, ] n 1 of n technologies. Performance criteria alternatives are assessed against economic, technical (e.g. availability and energy security risks) and environmental dimensions, with the goal to rank technologies and portfolios and then apply scenarios to validate the sensitivity of the results. 3 Emerging conditions considered for the construction of scenarios (elements) concern, among others, different projections on electricity consumption annual growth and high price volatility for natural gas and oil, as well as combinations of these. A similar approach is followed by Heinrich et al. [86] ranking power expansion alternatives for given multiple objectives and uncertainties, using a value function multicriteria approach, across different scenarios yielding information regarding the power expansion alternatives' relative performance and credibility. Energy system models are also often used in combination with scenario analysis in relevant studies [76].

Risk measures and common outputs of the methods
Having laid out widely cited and applied risk-based evaluation approaches from the literature (Section 3), this section discusses and summarises the key findings of the literature review by providing a comparative overview of the most significant outputs of each method as well as by highlighting the weaknesses and strengths of each approach as identified by authors that employed them in sustainable energy technology planning and feasibility problems. Fig. 2 illustrates the main outputs of the bulk of the studies that have employed these methods.
MVP method measures risk in several ways [151]. Usually, the standard deviation of historic periodic returns calculated through the Sharpe ratio, which is defined as the ratio of expected excess return to standard deviation of the return [152], is used; this definition assumes that financial returns follow a normal distribution, hence the prob-3 Alternatives (power generation portfolios) are assessed against the performance criteria by means of a Likert scale rating measuring the degree the alternative meets each criterion (1-High, 0.5-Low, 0-Blank).

A. Ioannou et al.
Renewable and Sustainable Energy Reviews 74 (2017) 602-615 ability dimension of the portfolio risk cannot be accurately reflected through this measure. However, Value-at-risk (VaR) is another traditional risk measure utilised by MVP theory approximating the probability that the value of an asset or portfolio will drop below a particular value over a specified confidence level and in the context of a planning horizon. The method can be applied to a power generation asset portfolio with available periodic market parameter values not necessarily following a normal distribution. Given the probability distributions of all portfolio assets, VaR values can be used to approximate the maximum loss for the whole portfolio. Being a widely used risk measure embraced not only by risk managers and actuaries but also by researchers and in investment banking, VaR (also known as percentile risk measure) is always specified with a given confidence level α (usually with values 90%, 95% or 99%) and can be used for portfolio optimisation when the cost/return distributions of the different technologies are not necessarily normal (in contrast to the Sharpe ratio metric). In the majority of MVP studies, risk is approached by the variability of the generation cost components originating from the market (deviations in demand for power, electricity price, fuel price), economic and financial (CAPEX, OPEX, project delay, capacity factor, energy generation) and political (such as retroactive/prospective regulatory changes, uncertain CO 2 prices) contexts. The method's applicability is subject to the availability of historic data of cost components and other statistical parameters of the RE project, as well as the availability of correlation values of risks among assets [109]. ROA supplements the information provided by static evaluation approaches, by recognising that in an uncertain future one needs to have the flexibility to adjust the timing of the investment decision [109,153]. Real options methods help to evaluate the value of waiting as part of the decision-making problem. The method commonly uses dynamic programming which allows the sequence of investment decisions to break down into options and systematically derive and compare the expected NPVs from immediate investment, waiting and all subsequent remaining decisions. In most studies in the domain of energy technology evaluation, uncertainty is introduced by means of forecasted input fuel prices, average wholesale price of electricity, uncertainties in policy support schemes (e.g. subsidy payments) and capital costs. The output of ROA can subsequently inform portfolio optimisation, while the importance of different energy technology options under specific political, technological and socio-economic circumstances can be captured by scenario analysis, providing valuable insight for policymakers about the incentive mechanisms needed to promote robust long-term climate risk mitigation.
Optimisation methods with stochastic inputs have been widely implemented to the problem of allocating optimal power generation assets. This may apply to long-term optimal energy mix planning in a national level, minimising total discounted (annualized) cost against a number of constraints ensuring the energy security, attainment of environmental targets, maximum capacity of different technologies, etc. This is thus a method that can be potentially derive policy recommendations for more efficient energy technology roadmaps [26]. The method can, however, be useful from an investor's (energy producer) viewpoint, e.g. for the determination of the optimal expansion planning of the power generation capacity over a long term horizon [65].
Scenario analysis recognises that altering individual variables whilst holding the remainder constant is not realistic. Depending on whether scenario analysis is embedded in a qualitative or quantitative methodological framework, risks considered may vary. Empirical scenario analysis techniques can provide a first-step in understanding inherent risks and uncertainties of future energy systems under different socio-political scenarios [154]. Outcomes of scenario analysis in empirical studies could also be the rating of electricity generation technologies and their mixes across different scenarios. Scenarios simulate the development trajectory of RES technologies between a status quo (current projection) and alternative scenarios which deviate from the status quo because of considering a different development in a number of driving forces, e.g. technology progress, climate change policy and situation of global warming. Although scenario analysis, when used on its own (potentially in an empirical framework) lacks the scientific rigour for assessing the frequency and quantified impact of risk and uncertainty on the RE technology value; when combined with other methods, such as portfolio theory and ROA, it can be a valuable tool to simulate various interconnected conditions. In this case, scenarios can derive optimal technology portfolios across different socio-economic scenarios resulting in different stabilization targets [37].
Monte Carlo is a method that allows accounting for numerous stochastic or uncertain input parameters and can be employed to produce probabilistic valuation models which incorporate risk assessment in the evaluation of RE technologies. Thus, it is a method that can capture statistical fluctuations of input variables and derive probability density distributions of cash flows.
MCDA establishes preferences between project options in accordance with a set of criteria and objectives, normally stemming from policy/project objectives as well as other financial, social, technological, and environmental factor considerations. MCDA is often applied as an alternative risk assessment technique because it is able to accommodate multiple criteria and is not constrained to use only monetary values; rather, subjective scales can be employed to rate each option (such as Likert scales). For example, when considering the problem of deciding on whether to invest in a power plant project and determine the order of priority of the projects in the company's portfolio, an investor has to consider a number of risk factors, such as the country risk (the political and economic instability as well as the level of corruption), risk of change in energy policy which may undermine the reliability of the project's economic feasibility, risk of changes in policy premiums, etc. [46], which may be hard to monetise and therefore the application of appropriate multi-criteria methods can prioritise the alternatives through pairwise comparisons in terms of each risk factor (e.g. Analytic Hierarchy Process).

Strengths and weaknesses
This section outlines briefly some of the strengths and weaknesses of the risk-based evaluation methods, which were not explicitly examined in the previous sections.
As such, the Sharpe ratio has been widely used as a metric for riskadjusted return in power generation and feasibility studies employing MVP methods [25]. However, the metric has received much criticism since it assumes that financial returns follow a normal distribution, as well as the assumption that investors only focus on the mean and variance of costs of an investment. Nevertheless, several studies have shown that financial returns of assets very often have non-normal characteristics, such as (negative) skewness. This shortcoming of the method can be potentially overcome by using alternative risk measures such as the VaR reflecting the amount that losses will not exceed a specified confidence level over a predetermined time schedule, while another measure often used is the Conditional value-at-Risk (CVaR) (also known as Tail-VaR, mean excess loss and mean shortfall) which is considered a more consistent measure of risk than VaR [155]. From an applicability perspective, the method lacks managerial flexibility since the investors are not able to assess the dynamics of the investment environment and take decisions on the portfolio rebalancingwithin the specified investment timeframeaccordingly. Additionally, conventional MVP theory disregards costs of moving from inefficient to efficient energy asset portfolios. Nevertheless, these costs are essential for electricity generation portfolios since there are usually significant salvage and decommissioning costs for existing technologies. The decommissioning cost might be included in the cost of energy, but the costs of shifting from one set of technologies to another are not explicitly addressed.
On the one hand, probabilistic approaches (such as MCS) provide the flexibility to assign probability density functions to input variables using historical data to foresee future developments of parameters; on the other hand, they cannot capture the extremities which might have a critical impact on the power generation system [108]. Each point on the output distribution represents the outcome of the joint probability function of the uncertain input variables. It should be noted that accuracy in the result depends on the appropriate statistical modelling of the stochastic input variables as well as the proper selection of the quantile value for the joint probability distribution function.
Investment planning decision making problems involving deterministic mathematical programming have been developed in standardised modelling frameworks, facilitating the validation and reproducibility of results. Nevertheless, the introduction of uncertainty in one or more of uncertain input parameters has generated a fragmented number of works following different approaches to modelling uncertainty leading to significant lack of precision and conflicting results [122].
Finally, scenario analysis does not provide the flexibility of probabilistic analyses while the uncertainties are not specifically integrated into the solutions explored [86]. Nevertheless, when combined with other risk-based methods, it can be a valuable tool to simulate various interconnected conditions. Further, the strengths and weaknesses of the methods cited above are outlined in Table 4.

Conclusions
The analysis of different risk factors (technological, political, social, environmental, etc.) assists stakeholders (developers, investors, utilities) in the RE sector to speak the same language in reference to what risks are associated with a sustainable power generation project and which of these can be transferred, mitigated, avoided or accepted.
The present paper brings together an array of methods that has been widely employed to address/model/incorporate risk and uncertainty attributes (related to energy security, generating costs, market risks, climate change risks, etc.) in sustainable power generation planning and feasibility studies. It was observed that MVP, ROA, MCS and (stochastic) optimisation methods are usually employed to address/model statistical risk factors, while semi-quantitative methods such as scenario analysis and MCDA may also be employed to address non-statistical parameters such as social factors and the emergence of competitive technologies.
Financial risks (e.g. variations in the investment return [62] or energy sale prices) have been widely accounted for in MVP and MCS methods; while the emergence of competing energy technologies (i.e. nuclear power) has been principally captured through scenario analysis [26]. Technology/innovation risk parameters are usually encountered in studies employing ROA, MCS, optimisation and scenario analysis by means of variation in future technology costs (learning curve effects). Stochastic optimisation models are frequently applied to assist policy makers in the definition of optimum energy mixes, taking into consideration uncertainties in the energy demand (i.e. macroeconomic factors), variation in electricity prices, generating costs, fuel risks, technological risks and carbon emission reduction targets. Finally, technical risks, such as reliability of components and access to the grid have been found to be frequently modelled by goal programming methods (i.e. MCDA methods) and optimisation methods.
A general conclusion of the review process is that no modelling approach can combine every element of the problem. Each approach requires different assumptions and views from different perspectives of the socio-techno-economic systems depending on what it attempts to investigate. As an example, microeconomic analysis models (such as ROA) cannot replace models with a wider view of national or regional markets (such as energy system models), rather these methods should complement each other [159]. Untapped issues recognised in the recent methodological approaches reviewed dealing with risk and uncertainty in sustainable power generation planning are summarised below: • MVP theory is one of the key methods advocated to support that diversification of energy technologies can ensure long-term electricity generation under a balanced risk-return relationship [160]. Yet, an important issue neglected to date in the technique is the consideration of the load structure of the technology combination so that technologies can cover demand during peak hours [37]; hence results derived by the method may ultimately not be insightful for policy makers and practitioners. For providing recommendations on the optimal energy mix, the load structure of the technology mix needs to be incorporated in the model, for example by introducing minimum constraints on peak-load technologies.
• Scenario analysis is particularly useful for explicitly modelling trend uncertainties and plausible future technology developments, especially when conducted according to industry's perceptions, since their actions are grounded on their perceptions, while scenarios constructed by policy makers should be used to derive the expected behaviour of the agents that participate in the market.
• Long-term uncertainties (those that cannot be hedged in forward markets) are usually represented by stochastic input parameters (such as energy demand, electricity price, CO 2 costs) and modelled through probabilistic methods (such as MCS), assuming that they follow a probability distribution. However, the development of their values critically depends on future policies and/or macroeconomic developments, so one has to be sceptical regarding the stochastic process assumption.
• Diversification of technologies has been widely cited as an effective risk mitigation technique also for investor-owned utilities which usually distribute their investments among different power genera-tion technologies. Methods employed to address risk/uncertainty in investor-owned power generation utilities mostly emphasise the statistical risks. However, it is increasingly accepted that nonstatistical risks are frequently the drivers of failures (such as policy instability, economic instability, lack of public acceptance, restrictions in terms of land availability) [105]. Translating non-statistical risks (e.g. aggregated through a risk priority number) into a cost per kWh for a number of sustainable energy technologies could contribute towards deriving more cost-effective solutions [14]. The quantification of such risks could be achieved with the support of expert opinions.
In the absence of data, risk factors identified in reference to a sustainable power generation project could be used to create specific scenarios (or else failure modes) that experts could possibly rate in terms of their probability of occurrence and impact [131]. Accordingly, quantitative risk impact evaluation methods could be employed to take advantage of the obtained values. The development of a structured riskbased evaluation framework, focusing on determining the risk-cost profile of sustainable energy generation technologies and mixes of technologies could, thus, constitute a focal point that future research in modelling risk and uncertainty in energy planning and feasibility studies should take into consideration.

Appindx A. Description of systematic review approach
The literature review was conducted on the basis of a systematic literature review (SLR) approach, which provides the synthesis of the research in a systematic, transparent, and reproducible manner, while also restricting the researcher's bias [22]. To this end, a literature review protocol was produced to frame the research methodology. The literature review protocol outlines the aim and questions underlying the review, the search strategy, the inclusion and exclusion criteria and the plan for data extraction.
Important criterion when selecting the keywords of the research was to be as inclusive as possible in order to avoid missing important studies. Key words selected, were clustered into four (4)  . Terms belonging to the same category were inserted with a Boolean operator 'OR' in the search box, while accordingly terms of Categories 1,2,3 and 4 were combined via a Boolean operator 'AND', resulting in 5*4*2*5=200 search strings.
After the search strategy was defined, a number of inclusion/exclusion criteria as regards the papers retrieved was determined to eliminate papers that fall outside the scope of the research topic. The search was limited to scientific peer-reviewed papers to ensure a collection of robust and validated works. Papers were retrieved from Scopus, while the final inclusion of papers considered for full-text analysis was determined following a quality assessment process (Fig. 3).
The initial literature was supplemented with additional works through a bespoke process, when further information to cover a particular topic was needed, or a key text in the literature had been missed by the systematic review.