The importance of stakeholders in scoping risk assessments – lessons from low-carbon transitions

Identifying the risks that could impact a low-carbon transition is a prerequisite to assessing and managing these risks. We systematically characterise risks associated with decarbonisation pathways in fifteen case studies conducted in twelve countries around the world. We find that stakeholders from business, government, NGOs, and others supplied some 40 % of these risk inputs, significantly widening the scope of risks considered by academics and experts. Overall, experts and academics consider more economic risks and assess these with quantitative methods and models, while other stakeholders consider political risks more. To avoid losing sight of risks that cannot be easily quantified and modelled, including some economic risks, impact assessment modelling should be complemented with qualitative research and active stakeholder engagement. A systematic risk elicitation facilitates communication with stakeholders, enables better risk mitigation, and increases the chance of a sustainable transition.


Introduction
Mitigating climate change requires transitions of human systems that comprise a wide range of choices to be made by different stakeholders, at different jurisdictional scales and for different time horizons.In this paper, we define transition pathways as coherent sets of policy, technological and/or behavioural choices intended to reach a desired low-carbon future and considering the trade-offs that come with these choices (compare Markard et al., 2012;Rosenbloom, 2017;Tàbara et al., 2018;Köhler et al., 2019).
The desired low carbon future may be made explicit from the top down in policy plans, such as the European Union 2030 Climate & Energy Framework (European Union, 2019), or implicit as seen in the grass root student movement, "Fridays for Future" (Fridays for Future, 2019).As with any overarching strategy, nations and communities must then determine their own specific transition pathways, that is, their unique sets of context-appropriate policies.
At the international and national scales, these low-carbon transition pathways may be about policy strategies, overarching targets, market mechanisms, and taxes; at the firm and government department scale, about technologies, budgets, and procurement; at the household scale, about behaviour and personal lifestyle.All of these choices carry risks, both from exogenous factors jeopardising the success of the transition and the implementation of the actions promoting it, and from undesirable side effects brought about by the transition itself.There are many factors that limit which pathways can reach the low carbon future (Wise et al., 2014).For example, communities may reject a technology outright or only its implementation within their borders (Siegrist and Visschers, 2013;Hanger et al., 2016).Lack of finance may also hinder the development and implementation of policy choices (Ekholm et al., 2013;Geddes et al., 2018), and incentives or policies may turn out incoherent, thereby causing negative impacts in non-target sectors (Lieu et al., 2018), such as leakage effects (e.g.van Vliet et al., 2003).Furthermore, the process through which policy choices are made can improve or reduce the legitimacy and acceptability of policy outcomes (Visschers and Siegrist, 2012;Ciupuliga and Cuppen, 2013;Lienhoop, 2018).Thus, ultimately "the selection of climate policies should be an exercise in risk management" (Kunreuther et al., 2013).Identifying the risks that could impact a low-carbon transition is a prerequisite to assessing and managing the potential impact of these risks and thereby improving the chances of a successful transition.
If examined in sufficient detail, risks may accompany every separate choice of policy, technology or behaviour of every possible transition pathway, leading to many risks layered on top of each other.Scientists, however, do not typically systematically organise, catalogue or frame most risks in transition pathways.Indeed, most efforts to assess or take stock of transition pathways, e.g.IPCC reports, use or refer to climate-economy or integrated assessment models (IAMs).These models are used to quantitatively explore the impact of different policy choices, within wider socioeconomic, temperature or GHG concentration scenarios (Nikas et al., 2019).However, using these models (or any other formalised modelling frameworks) naturally limits the types of risks that can be effectively assessed (Doukas et al., 2018).Furthermore, recent research in the wider transition community focusses on stakeholder, justice and social issues (Köhler et al., 2019;Hopkins et al., 2020), an approach that is very complementary to model-centred assessments.
To combine the research approaches from the transition research community and the assessment modelling community (similar to Turnheim and Nykvist, 2019), we conceive of the transition pathways in our research as described in two components: narratives and scenarios.We define narratives as mostly qualitative descriptions of the desired futures proposed by different stakeholders in policy, industry or society.There can be multiple transition pathways for any economic sector, that vary with the stakeholder groups that propose each narrative.This paper does not focus on the specific narratives of any transition pathways; it focuses instead on assessing risk in the policies, technologies and behaviour choices in the transition pathways (see Hanger-Kopp et al., 2019b for detailed discussion of transition pathway narratives in 11 countries).We define scenarios as quantifications of the narratives specific to climate-economy models or IAMs, combined with assumptions for the wider modelled system in which the desired futures would occur.
For this paper, we collected and analysed primary data from fifteen case studies in twelve countries that featured the design and assessment of low-carbon transition pathways.These case studies were conducted in Austria, Canada, Chile, China, Greece (3 cases), India, Indonesia, Kenya, Netherlands (2 cases), Poland, Spain, Sweden, Switzerland, and the UK, and are described in more detail in Table 1.Each case connected to ongoing low-carbon transition efforts in the respective country.Each case also included a mixed method approach with two interlocking components.First, a stakeholder engagement component, where stakeholder narratives for low-carbon transitions in a specific sector were elicited in interviews, surveys and group events like workshops -our paper focusses on the risks that were elicited as part of this process.Second, a modelling component, where the low-carbon transition pathways were assessed quantitatively within a set of wider scenario assumptions with the aid of energy system, economic, and/or IAMs, or ensembles of such models.
In reviewing the elicited risks associated with the low-carbon transition pathways from the fifteen case studies, this paper shows the value of a mixed-methods approach that combines stakeholder engagement with economic and energy modelling at an early stage of decision-making.Stakeholders can meaningfully expand the scope of risk assessment to include risks that would be left out of the assessment and any subsequent decision-making process if only macro-scale integrated assessment, energy, and/or economic models were considered.More specifically, we address three research questions: RQ 1: Are specific risks or categories of risks more or less frequently represented in risk assessment for climate mitigation?RQ 2: Do domain experts (e.g.researchers and academics) consider and value climate action-related risks similarly to other stakeholder groups?RQ 3: Do models sufficiently represent, and feature the capacity to assess, the broad spectrum of real-world risks that stakeholders talk and worry about?
To address these questions, we first discuss the background and framing of our study within the wider body of literature on these topics, then describe our methods and data collection.We then present and discuss our findings, followed by conclusions on our three research questions and implications of our study.

Background
We first frame the approach to risk we take in this paper and discuss how stakeholders were involved in risk assessments in past literature.

Risk framing
By risk, we mean "[t]he potential, when the outcome is uncertain, for adverse consequences on lives, livelihoods, health, ecosystems, economic, social and cultural assets, services (including environmental services), and infrastructure" (Agard et al., 2014).Risk in this paper therefore refers to a specific possible consequence of an uncertain state, event (e.g.climate change) or action (e.g.climate policy) that: a) is perceived to be negative; b) stems at least in part (but not necessarily exclusively) from a given uncertainty; and c) depends in large part on the transition pathway.This dependence must either be quantified as a 'probability' (drawing from existing knowledge) or be attributed a qualitative 'likelihood' (based on stakeholder knowledge, expertise, and/or perception).
Our definition is more specific than the ISO 31000 definition as "effect of uncertainty on [expected] objectives" (Lark, 2015).Conversely, it is also considerably wider than risk as monetarily quantified uncertainty that is frequently used in economics and proposed almost a century ago by Knight (1921), or as probability × damage × vulnerability used in modern quantitative risk analysis (e.g.Koornneef et al., 2010).Our definition also encompasses the risks discussed in the research around systems of innovation, like the technological innovation systems (TIS) framework (Song et al., 2020).This research focuses on the stakeholder processes that lead to technologies going from niche to mainstream and the growth of the stakeholder 'ecosystem' around emerging technologies (interconnections between actors involved in innovation processes, see Hekkert et al., 2007;Suurs, 2009;Probert et al., 2013;Bergek et al., 2015).Risk to a low-carbon transition in a TIS is framed as stakeholders deprioritising the emerging technology in question, thereby starving the TIS of attention and resources, or incumbents acting to limit innovations (Smink et al., 2015).Stakeholders can do this for their own reasons or in response to external (macro-level, global) influences (Edsand, 2019).This is potentially a large hurdle, as new technologies are an important element of any transition pathway.
For a consistent classification of risks in policies that aim to promote a low-carbon transition, we distinguish between 'barriers to' and 'negative outcomes of' these policies (Hanger-Kopp et al., 2019a;Song et al., 2020).Possible barriers include risks that may discourage or hinder the design, implementation and/or success of a technology, policy strategy or a specific policy instrument.Possible negative outcomes are risks that may manifest as a result of the uptake of a specific technology, policy instrument or strategy.As such, we refer to possible barriers as implementation risks, and to possible negative outcomes or consequences as consequential risks.This distinction is clearest when applied to specific policy instruments and technology choices.For example, legal challenges to an unpopular large hydropower dam are an implementation risk, but increased inequality in disposable income due to a tax is a consequential risk.
We recognise that risks can be connected, with multiple causes leading to the same effect and thereby reinforcing each other, for example with slow economic growth and poor land use planning both potentially leading to limited diffusion of large-scale renewable energy technologies.Conversely, one cause can lead to multiple impacts.Risk can also form chains or cascades, when one impact influences the likelihood of another impact.When policy instruments are combined into policy mixes and strategies, the number of risks increases, and new risks may emerge from the interactions between instruments and the effects they have (Lieu et al., 2018).
We only address key risks in this paper that are connected to technologies and policies in our transition pathways; climate change in general, like any worldwide environmental, economic, or social trend, carries too many risks to cover in this paper.Similarly, we focus on issues that could break pathways, and do not address possible synergies and positive spillovers from climate change, which have been widely documented (e.g.van Vliet et al., 2012;Tàbara et al., 2018).
In addition to the split between implementation risks and consequential risks, we divided risks into six categories of risk: political, social, regulatory/institutional, economic, environmental, and technological (see Appendix A for examples of implementation and consequential risks in each category).

The role of stakeholders in risk assessment
Risk assessment has a long tradition in many areas, such as insurance, finance, medicine, engineering, psychology, and anthropology (Renn, 2008).Approaches vary widely in these fields, depending on the data available.Risk assessment in insurance and medicine only works where sufficient quantitative data is available to calculate likelihoods.Particularly in engineering, experts on certain technologies and processes define likelihoods to calculate fault trees.In psychology and anthropology, risk is assessed via perceptions from affected stakeholders (Slovic, 2000) and/or collective behaviour (Douglas and Wildavsky, 1983).Emerging risk assessments are therefore done in multi-or interdisciplinary teams to fill the gaps caused by specialisation; examples of such studies can be found in this special issue (e.g.Nikas et al., 2018;Antosiewicz et al., 2019;Mayer et al., 2019;Silaen et al., 2019;Taylor et al., 2019).While there is much knowledge on risk assessment within these areas that have found its way into climate policy, this has resulted into a somewhat confused terminology on risks (Hanger-Kopp et al., 2019a).For many risks-particularly the manifold variety of risks found in a broad field such as climate change mitigation-there is insufficient quantitative data, thus we need to rely on experts and stakeholders to identify and assess risks associated with transition pathways.Ultimately, comprehensive assessments rely on methods from a variety of specialised areas.
One important option for identifying risks in decision making is therefore to ask 'lay experts'.These are individuals not employed in the field but have other reasons to have relevant knowledge, or 'interested' citizens, who are to some extent impacted by these decisions (e.g.Polk, 2015;Kochskämper et al., 2016).Including stakeholders or 'lay knowledge' is part of transdisciplinary research methods where the research problem and societal problem (e.g.climate change) is analysed and knowledge is co-produced with both researchers and stakeholders (Polk, 2015).Stakeholders inputs are therefore valued as 'legitimate' knowledge and their concerns and interests are also integrated into the transdisciplinary research process including the co-development of transition pathways to the analysis of risks in transition pathways.Thus, including stakeholders in research can be done for three reasons (Fiorino, 1990;Wesselink et al., 2011): 1 to empower stakeholders to express their concerns and priorities (normative reason); 2 because having stakeholders involved increases legitimacy and buy-in (pragmatic/instrumental reason); and/or 3 because 'lay knowledge' is at least as sound and useful as experts' knowledge (substantive reason).
Involving relevant stakeholders in defining and legitimising new technologies and practices as well as considering their motives and strategies is critical to assessing different risks associated with climate action, and reaching feasible, robust and sustainable transitions (Polk, 2015;Turnheim et al., 2015;Lieu et al., 2019).As several authors have noted (Renn and Schweizer, 2009;Polk, 2015;Moser, 2016;Doukas et al., 2018), such stakeholders should include the private sector, national governments, the research community, non-governmental organisations (NGOs), labour and trade unions and associations, representatives from other relevant institutions, and civil society.Previous work has shown that stakeholder participation, at least for environmental issues, changes decision making and can add ideas, information and resources (Beierle, 2002;Johnson et al., 2004;Reed, 2008;Nikas et al., 2017;Reed et al., 2017;Reichardt et al., 2017;Frantzeskaki et al., 2019;Köhler et al., 2019).Beierle (2002) carried out a landmark ex-post literature analysis of stakeholder participation in 239 environmental decisions.His analysis found that, in the 70 case studies that provided information on joint gains, participants found solutions that were not obvious when the process began.This process therefore increased net total benefits in 69 % of the cases, while net benefits decreased in only 6% of the cases.After investigating 59 self-described participatory R&D projects in the area of natural resource management, Johnson et al. (2004) found that some 62 % of projects adjusted their research priorities and formed feedback links with stakeholders.More recent work shows that 'citizen science' can effectively complement academic research (Bonney et al., 2014).
However, other studies point out difficulties in demonstrating that including stakeholders actually improves decisions and sometimes may artificially suggest societal buy-in if stakeholders concerns, especially those who are marginalised, are not seriously considered (Lieu et al., 2018).Inadequate stakeholder engagement processes can therefore question the instrumental reason for participatory decision-making (e.g.Cox, 2007).Several authors quantitatively examined the effect of participation on the effectiveness of decision-making and found it statistically inconclusive (Newig and Fritsch, 2009;Sterling et al., 2017).Reed et al. (2017) noted that stakeholder participation has led to success and failure in equal measure, and extensive involvement of stakeholders does not lead to better outcomes because such processes have their own pitfalls.In particular, social learning can only happen when a sufficient diversity of perspectives are involved, specifically perspectives of marginal groups whose viewpoints are poorly represented by experts (Cuppen, 2012;Polk, 2015).
All of these studies focus on stakeholders and their effect on decision-making.While the need to include stakeholders in climate change decision making is receiving increasing attention (van de Kerkhof and Wieczorek, 2005;Geels et al., 2016), few studies point to the value stakeholders can add in identifying risks in low-carbon transitions (Nikas et al., 2017).For instance, the analysis of Beierle (2002) of environmental decision making in general only suggests that stakeholders contribute to better assessment of risks, but does not address the scope of risks considered.He further mentions that constricting scope was used as a way to exclude stakeholders, in which cases stakeholders were clearly only involved after the scope was settled.Reed (2008) similarly suggests that local knowledge should be integrated but then mostly refers to using stakeholders to acquire better data rather than emphasise changes in scope.Butler et al. (2015) argue that engaging with different framings can aid in managing risks associated with public acceptability of a low-carbon energy transition.
Only de Vente et al. ( 2016) mention factors that contribute to successful participatory processes and that these lead to "better problem identification and awareness".Prell et al. (2007) show a detailed example of a successful participatory risk assessment using modelling in which stakeholder input meaningfully influenced the scope of the assessment.Few recent studies aside (e.g.Nikas et al., 2018;Antosiewicz et al., 2019;Song et al., 2020), there is still an overall lack of studies of stakeholders' contribution to meaningfully expanding the scope of assessment of low-carbon pathways to include risks that would otherwise be ignored.This is to some extent expected, given that many studies on risks associated with low-carbon transition pathways are modelling-driven, which usually sees pathways and their policy instruments translated into quantitative parameters for model scenarios.
However, many aspects of pathways do not translate (directly or indirectly) to model scenarios (Doukas and Nikas, 2020), especially decision-making processes (e.g.Gigerenzer and Selten, 1999;Wise et al., 2014;Li, 2017).Moreover, due to the limited ability of economic models and IAMs to incorporate a diverse set of risks (Doukas et al., 2018), assessments of transition pathways usually fail to include a broad, stakeholder-driven risk scoping effort.This is detrimental to the assessments, as a broader scope of risks may be helpful in the interpretation and communication of modelling results and how they relate to risks in these pathways.
Our case studies applied principles of transdisciplinary research by explicitly including stakeholders throughout the research process: from problem definition (identifying the problems related to transitions pathways that are co-developed with stakeholders), to analysis of the problems (analysing the risks in the pathways) and exploring the impact of the research (assessing the potential negative consequences of the pathways) (Pohl et al., 2017).Thus, our case studies include stakeholder knowledge not only to verify model results or to discuss their preferences for model scenarios, but we also take on board stakeholders concerns and their priorities when assessing risks.With this approach, decision makers can consider both model outputs that address national economic and social issues and qualitative outputs that address local issues identified by stakeholders.How the outputs can be applied is not explicitly part of our research and is typically an underexplored area of research (Pohl et al., 2017;Braunreiter and Blumer, 2018), but continuous monitoring of policy impacts and adapting policies where needed is advised (Frantzeskaki et al., 2019;Rosenbloom et al., 2019).
This paper seeks to add to these promising findings and contribute to closing the research gap on participatory risk scoping, by exploring how much stakeholders can contribute to scoping risks.Unlike previous studies, which rely on secondary case study data (e.g.Beierle, 2002;Sterling et al., 2017), we use both primary data from stakeholders and secondary data from literature in fifteen case studies.Based on our experience with the analysis in this paper, we also discuss the fit between modelling and assessment of pathway risks.

Data and methods
The data we examine in this study contains risks collected in fifteen case studies, in twelve countries around the globe (Table 1).Each of these cases studies explored one or more possible pathways for a transition to a more sustainable mode of operation, in a particular economic sector.More than 70 interdisciplinary researchers across fourteen institutions in total worked on various aspects of their case studies.Some case study leads were modellers while other did not have modelling experiences.This leads to difference in outcomes as discussed in Section 4.3.
The case studies had many components in common, including an initial (grey and academic) literature review of current and proposed policies, exploratory expert interviews, stakeholder workshops, and surveys.However, the cases were otherwise tailored to national, regional, and local policy contexts and the respective narratives that described the low-carbon transition pathways (Hanger-Kopp et al., 2019b).In this paper we will not discuss the detailed narratives of each case study's low-carbon transition pathways but will refer to the transition pathways more generically.
We designed the risk elicitation process, shown in Fig. 1, separately to capture the broadest set of risks possible and enable crosscase comparison across low-carbon transition pathways.O. van Vliet, et al. Environmental Innovation and Societal Transitions 35 (2020) 400-413 In standardised 'risk elicitation' tables we collected data on case-specific risks.We created different tables for implementation and consequential risks, as well as for risks from qualitative and quantitative (modelled) research processes.The risks that were modelled were in some cases a strict subset of the risks that the case study leaders had identified qualitatively, but in other cases the modellers added further risks to the tables, drawing from the epistemology of their own models.As very few models contain barriers explicitly but rather focus on exploring consequential risks, modellers were asked to also describe what barriers they took into account when preparing their model runs.
Most importantly, we collected detailed information on each risk identified during our study, grouped these into six broad risk categories, identified the source of each risk, and also identified their potential to be quantified in models.All elements of the risk elicitation tables are listed in Appendix B, while Appendix C presents the template for risk elicitation tables.
Overall, we provided initial guidelines, i.e. definitions of risk categories and examples, but let the case study leads, modellers and other stakeholders define and describe risks based on their understanding of these definitions and examples.Iterations between case study researchers and study coordinators took place in online and in-person follow-up meetings, as well as deliberations over email.Afterwards, we categorised the provided risks based on our framing as well as our understanding of their detailed input.

Results and discussion
We collected 145 implementation risks and 121 consequential risks.We present our characterisation of these risks in this section, examine the results of risk categorisation in Section 4.1 to answer RQ1, examine risk sources in Section 4.2 to answer RQ2, and then compare treatment of risks by stakeholders and modellers and stakeholders in Section 4.3 to answer RQ3.In Section 4.4, we address some consequences of our cross-case comparison and the robustness of our analysis.
It is also noteworthy that we did not assess probability and severity of risks, even though these are two key dimensions of risk assessment in this paper.This does not mean we claim that all of these risks are equally important; the aim of this paper is not to quantitatively assess risks but to assess or evaluate which risks are usually looked into, how these risks tend to be assessed, to what extent they are adequately represented, and by whom.For an example of quantitatively assessing and weighing risks, see the Chinese case study (Song et al., 2020).

Risks by category
The six risk categories we used (see Section 2) show some potential overlap, depending on the disciplinary background and perspective of the person classifying the risk.For example, unemployment, as a socioeconomic indicator, may be considered and therefore be classified both as an economic and as a social risk: it can be seen first and foremost as an economic problem of paying the bills, and at the same time as an ingredient in social unrest or political upheaval resulting from many people losing their occupations and livelihoods.The distinction between implementation and consequential risks is also imperfect.For example, knowledge of any potential negative consequence (e.g.job losses) is also a cognitive barrier and thus a political implementation risk, and any implementation risks that come to pass (e.g.rising costs of nuclear power) will change the technologies and behaviours and therefore change the consequential risks that may manifest.
We find slightly more implementation risks than consequential risks for the case study transition pathways, except in the Dutch agricultural and Canadian case studies.To put it another way: most of the time, there are more reasons to worry about a future not happening at all (barriers to implementation) rather than to worry about outcomes (consequences of actions and decisions) if and when that future comes to pass.However, we cannot proclaim any implementation risks as more important than any consequential risks without a deeper analysis of the significance and potential impact of these risks, the appraisal of which is very subjective.Furthermore, we see that consequential risks are often analysed in a different way than implementation risks, with the former more amenable to comparison between model outputs (e.g.changes in GHG emissions and GDP in integrated assessment models), and the latter more used as structural elements in the implementation of specific pathways and broader scenario choices.Though far from absolute, this general split between implementation risks that are qualitatively addressed and consequential risks that are addressed through quantitative modelling is also reflected in literature1 .
In Fig. 2, we see that economic risks are the most studied among both implementation and consequential risks, and that this is mostly done through quantitative modelling and at a national or international level.We cannot determine why this is the case, but it follows established practices.We hypothesise that reducing the risks of a low-carbon transition to investment decisions and economic impacts may be more appealing, in terms of viewing a low-carbon transition as a challenge that is both manageable and solvable; in other words, both researchers and stakeholders may tend to highlight the economic dimension of transitions, when acknowledging that these transitions must be both effective and financially viable.However, some authors have raised questions about the ability of (neo-classical) economics to contribute meaningfully to complex real-world problems (e.g.Taleb, 2007;MacKenzie et al., 2008), especially with regard to transitions research (Foxon, 2013;McDowall and Geels, 2017;Patt and Lilliestam, 2018).The risks we found in the case studies suggest that any low-carbon transition crosses multiple disciplinary boundaries.
We also find that, among consequential risks in the case studies and aside from economic risks, environmental risks are noticeably more numerous than the remaining categories, and mostly assessed through modelling.As the risks studied are associated with climate action, one would expect at first glance that the environmental consequences of this action would be negligible, if not overall positive.However, climate and environmental risks are not at all identical; for example, a pathway based on large-scale deployment of renewables may entail significant interventions to the environment, including biodiversity and local pollution; while a pathway based on structural change in a heavy industrial sector may lead to carbon leakage effects.Furthermore, these pathways represent transitions that are both complex and case-specific (see Table 1); for instance, nuclear phase-out in Switzerland may be achieved through energy imports, which in turn may be less green than nuclear power in the first place (van Vliet, 2019).In this environmental subset of consequential risks, less than a third refer to climate, and the others cover bioversity, local pollution, and water availability.
The aforementioned case studies clearly did not all cover every category; some were focused on modelling and, therefore, mostly consequential risks, while others were focused on implementation risks and political processes.We posit that this diversity reflects approaches of different research groups and needs of different transition pathways that also apply outside the scope of this study.

Risk sources
In Fig. 3, we see first of all that most risks are suggested by researchers or independent experts interviewed by the researchers.This is to be expected, as research groups will take on case studies that they have prior experience with, or consult with someone who does.A much smaller share of the risks comes from literature, both peer-reviewed and grey literature (e.g. government agency reports), and shows researchers bringing themselves up to date with the scientific state of the art as published.
Even though researchers in the project had access to all relevant scientific literature, stakeholders and media together supplied 46 % of implementation and 40 % of consequential risks.Thus, a combination of scientific and non-scientific ('lay') knowledge was necessary to assess risks relevant to all stakeholders, rather than purely focusing on risks identified by experts that are mainly interesting for existing scientific discourse and well-known concerns for policymakers.
For implementation risks in particular, media and stakeholders together suggested almost half of the risks that were considered.We draw three important lessons from this: 1) studies are likely to miss risks if they do not involve stakeholders or their opinions as expressed in the media; 2) this is especially important for implementation risks; and 3) stakeholders inform about risks that are important to them but that experts, researchers and scientific literature may disregard when they focus on existing modelling-based risk assessment capacities.
As implementation risks in a transition tend to translate to different policy pathways in model scenarios, all three lessons suggest that stakeholder input is especially relevant, when framing and elaborating plausible transition pathways that have any chance of being implemented in the real world.
A further question raised by the many risks suggested by stakeholders and media is why the case study researchers did not find these in scientific literature.We illustrate this with some examples of implementation and consequential risks that were suggested by stakeholders (Table 2).
We suggest four possible reasons why we needed stakeholder input to find these risks: 1) the risks are too context-specific to have been encountered before, which would apply to very specifically-phrased risks; 2) the risks are underrepresented in scientific literature because they were perceived as not as important as other risks that have been studied more; 3) the risks are too difficult to assess with existing methods and therefore fail to make it into publications; or 4) the literature is dominated by modelling exercises, which by nature have more capacity to explore and calculate consequential risks than to identify and assess implementation risks.

Social
Friction from the shift of jobs and workers from dirty to green sectors.

Reflection on model-focused cases vs. stakeholder-focused cases
When examining our dataset, we noticed a difference between case studies conducted exclusively by research groups that have modelling as their core activity, and case studies that were led by groups who do not.In the former, we typically received few risks (around five each for implementation and consequential), which were fairly generally formulated.These risks were suggested by the researchers themselves, and occasionally by experts or media.The consequential risks suggested by modellers were typically also the ones that were going to be analysed with their (energy, economic or integrated assessment) model.In case studies run by experts in stakeholder engagement, the stakeholder engagement was more intense, and led to the identification of more risks (up to twenty implementation and/or consequential risks), and also more diverse sets of risks, with significantly more detailed descriptions.However, there was also learning throughout the project and some case studies (e.g.Austria's steel sector) that were led by modellers who had high interactions with industry stakeholders.This interaction helped to widen the range of risks identified beyond those already built into models.
This difference reflects the different approaches to case studies -model-focused vs. stakeholder-focused -and broadly the advantages and disadvantages of each approach.Modellers generally limit their assessments to specific risks that their models represent well, which are most often consequential risks.Conversely, stakeholder-focused cases approach the subject in a broader perspective, aware that stakeholders will bring in diverse risk perceptions.This is reflected in Fig. 4, where we see that experts bring in more consequential risks than other sources (bottom figure, left bar), and stakeholders bring in the largest number of implementation risks (top figure, left bar).The risks that stakeholders bring in are also spread more evenly across categories than any other source.
Reviewing the descriptions given in the risk elicitation tables, furthermore, suggests that the stakeholders were more precise in their risk definitions, whereas the risks suggested by researchers and experts were more general and harder to operationalise or make use of in the implementation of transition pathways.For instance, in the Canadian case studies, researchers identified environmental risks but stakeholders more specifically indicate the types of environmental risk on the local biodiversity and made links to impacts on local community's health and welfare.This, in turn, suggests that stakeholders tend to stay with their concrete case situations and researchers are more inclined to approach their case studies either from a theoretical perspective or to generalise from a particular context to a broader set of circumstances.A consistent method and workflow to transpose outcomes from stakeholder engagement into models seems needed to connect the two.
Media accounts for eleven implementation risks as a source, and these were found in only five out of fifteen case studies.However, given that every case study included at least one reference to media and/or stakeholders, we conclude that almost any case study could benefit from including stakeholders.This would ensure that no key risks are neglected simply because they are not welldocumented in scientific literature, not known to the case study expert researchers, or difficult to frame within a model.

Comparison across case studies
Our study is a meta-analysis of fifteen case studies with different contexts and stakeholders.Each case therefore features different risks, different timings of data collection, and different models used.The differences between case studies raise questions about the robustness of our results that are based on aggregating data from all fifteen case studies.However, the differences among the case studies can be attributed to the scope and methods of data collection that led some case studies to gather more risks than others (e.g.some case studies were extensions of previous research projects or had strong stakeholder engagement processes).Another factor was whether the case studies dealt with a pathway of general interest or a specific sectoral pathway that was mostly interesting to experts (e.g. the Austrian case).To circumvent these differences, the risk categorisation, framing and analysis was uniform and strictly coordinated, and the results of our analysis, discussed below, can therefore lead to robust conclusions that provide broader insights to risks analysis.
The stakeholder engagement process was ongoing when we completed data collection, which may have biased the risks we found towards more abstract, 'vision'-related risks over risks encountered in operationalising low-carbon transition pathways.However, more recent, qualitative results suggest that the risks that were discussed initially remained relevant throughout the entire case study process (Hanger-Kopp et al., 2019b).

Conclusions and implications
In this study, we examined the risks found in fifteen case studies and have identified a number of gaps in expert knowledge and how risks are addressed in scenario modelling.From our analysis, we can draw the following conclusions for the three research questions driving our study: In answer to RQ1, we have noticed that economic risks are more frequently represented in risk assessment practice-or, more accurately, other categories of risks are represented noticeably less frequently.We attribute this to the fact that both domain experts and other stakeholders usually highlight the economic aspects of sustainable transitions and the need for these transitions to be financially viable.Potential negative economic outcomes are most often highlighted in climate action research.
In answer to RQ2, we also conclude that climate change and policy domain experts, including academics and researchers, consider different risks and value them differently than other stakeholders when assessing implications of climate action.Specifically, experts, academic literature and grey literature all suggest more economic risks than others, while stakeholders suggest more political implementation risks.
In answer to RQ3, our findings also suggest that models represent and can be used to assess risks that are essentially different from those that stakeholders talk about.Models also focus on more specifically defined and quantified risks.To avoid losing sight of risks that cannot be represented in models, or would have never been considered in the first place, contextual information should be preserved and used as much and as early as possible.This can be achieved by embedding use of models in mixed-methods studies, complemented with qualitative research and active stakeholder engagement.We also conclude that some real-world economic risks can push pathways in a direction that economic models cannot sensibly represent, such as clientelism stemming from political structures or low quality of subsidy applications.This is mostly a non-issue for implementation risks, as they are not usually represented in models but rather in scenario choices.For consequential risks, it is more of a practical problem than a conceptual problem: every model has a narrow epistemological range and can only answer a limited set of questions.For every case study, we could find economic risks suggested by stakeholders or suggested in literature that the models employed in that case study cannot address.This is not a failure of the models or the modellers involved, but the following three steps are necessary to adequately represent the social, economic, technical, political, regulatory, and environmental complexity of a low-carbon transition, and to assess both how feasible such a transition would be and what impacts we can expect: First, the research design for a case study should include alternative and complementary qualitative methods to deal with such risks in addition to the risks that are captured in models' structure and datasets.Second, the models used in the assessment of low-carbon pathways should, ideally, be chosen for their ability to assess the risks and concerns raised by experts and stakeholders.Third, in the process of modelling different low-carbon pathways, the scenarios can and should reflect a broader representation of implementation risks as drivers of the modelling simulations.
The most prominent and common example of modelling scenarios from the past few years is the framework of Shared Socioeconomic Pathways (SSPs, see O'Neill et al., 2014).The SSPs were designed as reference socioeconomic conditions that cover a wide spectrum of possible futures of the world, to be used as drivers in model scenarios (e.g.Van de Ven et al., 2019).The SSPs comprise broad narratives to illustrate these futures, without specifying specific policies (c.f.Frantzeskaki et al., 2019), and reference SSP datasets with quantitative model data that is compatible with these narratives.Both of these have their own expert audiences (Braunreiter and Blumer, 2018), and the approach for creating SSPs has been applied to more specific transition scenarios as well (e.g.Small et al., 2019).Our broad risk framing motivated us to try to use the SSPs as a reference framework for a wider global narrative in which our pathways might fit, and as a source for the complementary quantitative data for our scenario modelling.However, this did not work as intended.
We find that many of the risks that we had identified in the case studies, especially those supplied by stakeholders, would interfere with the developments projected in the narratives of the SSPs.This resulted in mismatches between the specific narratives of the pathways developed with our stakeholders and the generic narrative contexts of the SSPs.In the cases where we tried, it was also difficult to elicit stakeholder views on the generic narratives of the SSPs (see story factors in Appendix D, based on O' Neill et al., 2017), much less on the values of some of the quantitative parameters in the SSPs.Instead, stakeholders in our case studies focussed on country-, sector-, and technology-specific risks, narratives and scenarios.Furthermore, our case study leaders found that bottomup or agent-based models were more suitable to inform stakeholders than global IAMs (van der Gaast and Spijker, 2019).
We attribute this in part to the bottom-up development of our pathways.By contrast, the SSP reference framework was intended to facilitate down-scaling from global trends to the national context (see e.g.Absar and Preston, 2015;Nilsson et al., 2017;Tàbara et al., 2018).However, the downscaling of SSPs to country and sectoral level did not fit with our bottom-up approach to assessing pathways.Furthermore, a split between policies and context, as exemplified in the separation of SSPs and shared policy assumptions (SPAs), does not capture the complex and path-dependent interaction between individual technologies and wider social, economic, and technological developments.This was found in the stakeholder engagement in our cases and suggested by the wider literature on technological transitions (e.g.Perez, 2009;Wise et al., 2014;Turnheim and Nykvist, 2019).
This insight supports concerns about the plausibility of selected SSP scenarios (e.g.Ritchie and Dowlatabadi, 2017) and compatibility of SSP scenarios with sector-specific insights (e.g.Roson and Damania, 2017).More importantly, however, our findings are in line with those of localisation/down-scaling attempts, which show that scaling down SSPs while at the same time incorporating uniquely local pathway elements can lead to several-fold increases of plausible futures (e.g.Frame et al., 2018).
Pragmatically, it is impossible to include all locally relevant risks in a scenario modelling study.We therefore suggest that SSPs, and other generic reference scenario frameworks by extension, are best used in bottom-up pathway assessment as a source of possible exogenous risks and developments.Researchers may consider these when assessing pathways that were developed by stakeholders; a well-designed scenario framework can suitably broaden the possible future developments that are considered.Our experience also serves to remind scientists and other users of scenario studies to avoid the anchoring effect of scenarios and keep an open mind about relevant risks and other pathways elements that are not part of the scenario framework documents and graphs, and should be assessed with complementary methods.
In the process of collecting risk elicitation tables, we found that they facilitated communication between researchers involved in stakeholder engagement and modellers.This finding was a fortunate coincidence of the timing of our study, as several case studies in the larger project were at the point of starting to model case-specific pathways based on earlier scoping work.Specifically, the elicitation tables guided the researchers to be sufficiently explicit and detailed in their discussions of risks.This let the modellers clarify exactly what their scenarios could offer and the case study leaders further clarify what risks exactly they wanted to see in the scenarios to be modelled.
Eliciting risks potentially also improves communication between scientists and stakeholders.Anecdotal evidence suggests that our elicitation process increased understanding and/or trust as part of a larger transdisciplinary case study process (cf.Chess and Purcell, 1999;Konisky and Beierle, 2000;Wolsink, 2007;Arvai and Froschauer, 2010;Wallquist et al., 2012;de Vente et al., 2016).This may make an interesting topic for further study.
Once identified and assessed, risks can point out the 'adaptive space' (see Wise et al., 2014), where pathways can be enumerated within real-world constraints, and can also help identify robust pathways that come with the fewest, least likely, and/or least harmful barriers and negative outcomes (also see Tàbara et al., 2018).As with the assessment itself, making these pathways a reality is facilitated by improving the capacities of stakeholders to make the changes needed (e.g.Frantzeskaki et al., 2019) In this paper, we do not prescribe a particular format for risk elicitation, similar to how previous research found that stakeholder engagement can work (or not) in a variety of modes and formats (Chess and Purcell, 1999;Konisky and Beierle, 2000;Renn and Schweizer, 2009).Having sufficient diversity in stakeholders was noted as a key requirement in participatory governance (Cuppen, 2012), and this applies similarly to a systematic risk elicitation process.We find that, if done properly, a systematic risk elicitation allows researchers to assess a much more comprehensive set of potential barriers and negative outcomes of low-carbon transitions.This will enable better risk mitigation, and in turn increase the chance of a successful transition.lies with the authors; the paper does not necessarily reflect the opinion of the European Commission.The authors express their thanks to the case study leaders and researchers, especially Gabriel Bachner, Brigitte Wolkinger, Luis Virla Alvarado, Paula Díaz, Leonhard Späth, Christiane Plum, Song Lei, Chen Ying, Cynthia Ismail, Takeshi Takama, Xaquín García-Muros, Tim Suljada, Oliver Johnson, Rocio Alvarez, Annela Anger-Kraavi, Katerina Foruli, Sotiris Papadelis, Serafeim Michas, Duke Ghosh, Krisztina Szendrei, Björn Nykvist, and Luis Gonzalez.

Table 2
Case study countries and research subjects.
Political Perception that government support for low-carbon transitions depends on which parties are in power, and is therefore unreliable.Political Contradictory goals at different levels of government, e.g.local, state, national.Social Perception that climate mitigation and social justice are in conflict.Consequential risks: Economic Cost of compensating for loss of natural habitat in construction of renewable power plants.Environmental Missing elements and emissions in life cycle analyses.