The risk of surprise in energy technology costs

As countries begin to reassess and expand their energy research and development programs, it is appropriate to understand the likely costs of alternate approaches. Unfortunately, communities of experts have a well-known optimistic bias in their judgments. We review findings from several disciplines that underscore this tendency toward overconfidence as well as some proposed alternatives to incorporate it in public decision-making. We further argue, based on a disaggregated analysis of US nuclear power costs, that incorporating a second-order uncertainty in the shape of the distribution of cost components is essential for capturing important elements of uncertainty in moving forward with expanded energy R&D programs.


Introduction
Considerations of energy security and climate change will likely impel increased societal interest in technologies that enable a reduction in the use of fossil fuels. Although governmental R&D for energy technologies went through a period of relative decline in the 1990s [1], recent budgets have increased support for certain energy research and further resources have been called for [2,3]. Yet a dramatic increase in energy research funding, while appropriate to the policy goals, presents a major challenge to ensure that the funds go toward addressing clear policy goals and are administered in a way that both encourages incremental developments in existing technologies but also, and more importantly, drives innovation in new areas [4].
Clearly, different energy technologies present varied risks and obstacles. Large technological projects that enable continuity in the energy system-such as biofuels, carbon capture and storage, and nuclear technology-are attracting particular attention. While these advanced technologies have shown some promise, and while their low-carbon characteristics provide investors with a hedge on high carbon 5 Author to whom any correspondence should be addressed. prices [5], they also come with risks of delay and cost overruns. In this letter, we argue that overconfidence in the rates of future technological advance should be expected, and such overconfidence can hamper a broad-based research program by focusing funds on high-visibility, large technologies that seem to offer easy solutions to pressing problems. Assessing the risks of higher cost in these technologies becomes important when considering a portfolio of energy research options, some of which might be more likely to achieve their technological and financial goals [6,7]. These challenges are further complicated by decisions on how to allocate funding between, for example, the R&D, demonstration/prototyping, and commercialization phases [8]. We illustrate the potential pitfalls of such approaches with an example of the historical development of nuclear power, and then discuss how some of these risks might be anticipated in both public policy debates and in decisions for R&D funding.

Surprise and overconfidence in the context of large energy technologies
The topic of cost surprise for energy technologies presents numerous challenges for research. Foremost of course is the border between true surprises and surprises that, in retrospect, might have been foreseeable based on past experience or assessment, a much wider ontological discussion with broad applications in regulatory science. We argue in this letter that some surprises are indeed not surprising, that they can be reframed as the result of overconfidence, and that this overconfidence can be incorporated explicitly into policy discussions. Yet even in these cases, the question remains of how fully one should try to quantify the surprise stemming from overconfidence.

Observations of overconfidence
The topic of overconfidence itself is a subset of a much wider inquiry into the disparities between so-called 'rationalactor' or 'expected utility' models and the decisions that are frequently observed in actual behavior, especially in cases of uncertainty. Behavioral economics approaches [9] illuminate such long-observed patterns like overconfidence, as well as loss-aversion, endowment effects, and a bias toward the status quo [10]. In parallel, studies of peoples' perceptions of risks has similarly underscored differences between observed attitudes and 'rational' or expert judgments [11][12][13]. In one example, Bazerman [14] has applied these observations to argue that the rate and consequences of climate change are likely to be surprising to many actors, despite ample documentation of the risks and possible impacts.
Anecdotal experience with overconfidence is abundant, and numerous patterns have emerged from empirical studies [15]. Several studies have estimated the degree of overconfidence by quantifying expert errors. One approach to defining overconfidence quantitatively is to relate a prospective expert assessment of confidence intervals of a phenomenon to retrospective empirical observations of that phenomenon [16]. A panel of experts may, for example, estimate a country's future population in a certain year with 95% confidence. When the target year arrives, one can observe the 'true' value and compare it to the expert forecast. If enough before-after comparisons are available, one can assess how accurate the predictions are.
Shlyakhter and others [17,18] performed such an investigation of expert assessments from disparate fields, including measurements of physical constants, projections of future energy demand, and population. They found that, in all cases, the 'true' or observed values of each of these assessments tended to lie disproportionately outside the bounds of confidence reported by the expert groups. Importantly, such mistakes can stem not only from the experts' incorrectly narrow assessment of statistical uncertainty (e.g. what they thought was 95% confidence might really be only 66% confidence), but they can also stem from a mistaken assumption about the functional form of the distribution of the possible results. They further noted that such errors tended to be greater for systems with a social component. Both of these mistakes are arguably types of overconfidence: one dimension is an underestimate of the potential variance (spread) of the outcomes, and another is a misrepresentation of the pattern of outcomes.
Complex systems that link social attitudes and behavior with technological elements seem particularly prone to both kinds of uncertainty, and social-technical energy systems fit this characteristic. Such systems exhibit uncertainty on multiple levels in both quantitative and qualitative elements [19][20][21]. Spurred by early work on the uncertainties of complex energy systems modeling [22], several studies of forecasts of US energy consumption found evidence of systematic overconfidence in model forecasts. Craig et al reviewed the accuracy of energy forecasts from the 1960s and 1970s, and concluded that 'forecasters in the 1950-1980 period underestimated the importance of unmodeled surprises.... Not only were most forecasts of that period systematically high, but forecasters systematically underestimated uncertainties' [23], a finding echoed by O'Neill and others [24].
Overconfidence in forecasting seems to stem from multiple factors operating at both an individual and group level [25]. Forecasts that arise from group or committee reports have a tendency to overconfidence [26,27], and economic evaluations are no exception [28,29]. For example, one of the elements affecting economics of large energy technologies is the construction time for new plants. Planning timelines for the future is particularly subject to optimistic bias [30,31], especially when such planning is conducted through group discussion. This bias emerges as participants in groups tend to focus on factors promoting success [32] while systematically downplaying pessimistic scenarios [33].

Assessments of low-probability outcomes
The common assumption that results will be distributed approximately normally seems often to be unjustified. Shlyakhter and others, for example, found evidence of strong non-Gaussian errors in the three arenas they studied [17,18]. This empirical result echoes earlier findings that people often perceive low-probability events as more unlikely than they really are [34]. Some of this difference in perception can be attributed to people's source of information. Specifically, people often overestimate the likelihood of low-probability events if they are presented with explicit scenarios of different outcomes (decisions based on 'description'). In contrast, people tend to underestimate low-probability events based on personal experience-and, by extension, expertise [35][36][37]. Weber et al used this perspective to illuminate some aspects of the public debates about climate change [38], but similar arguments have not been made about energy technologies, despite the fact that past experience with both nuclear power and solar photovoltaic cells demonstrated excessive optimism on cost reductions.
Using an assessment framework based on expected cost distributions, an underestimate of low-probability events can stem from assuming a normal distribution if the underlying phenomenon is indeed non-normal. Accordingly, confidence intervals based on such an assumption will not only be too narrow but would by definition underestimate the probability of large deviations from current best estimates [39]. Thus, unless great care is taken to avoid possible biases, we might expect that groups of experts would tend to underestimate the future costs of technologies, as well as the possible timeline for the arrival of those technologies.

Expected surprises in energy technologies
The empirical evidence of overconfidence in expert judgment, models, and forecasts raises the question of how much one might expect it to affect assessments of future development of societally important technologies such as energy technology. While much attention in energy technology forecasting has focused on learning or experience curve analysis, there is evidence that such models are subject to the same pitfalls of overconfidence. Nemet [40], for example, in investigating learning in PV, concluded that experience 'only weakly explains' the most important changes in plant size, module efficiency, and the cost of silicon. Trancik [41] argues that nonexperiential factors, such as the existing scale of the current technology's infrastructure and the size of new units, can greatly influence the rate at which technologies can adapt to a changing regulatory regime. Finally, we elsewhere have presented evidence that learning in nuclear power was erratic, negative, and did not fit well with models linking experience with decreasing costs [42,43].
Several approaches have been proposed to make large technology forecasts more general and accurate.
The most basic approach is to use an engineering rule-of-thumb contingency, which assumes that costs might be greater by a rough amount, for example 20% or 30%, for a given project. While this approach does encompass the concept of unanticipated cost overruns, it does not capture possible alternative distributions of possible results. Such distributions can be incorporated in probabilistic simulation [44], although determining the expected distribution remains a challenge. Refinements of this contingency technique, when used appropriately, can provide accurate early guidance on costs for large projects [45,46]. Scenario planning exhibits additional options for overruns, but similarly does not give a sense of distributions or probabilities. A further, extensive refinement of combined qualitative and quantitative planning is provided by the 'NUSAP' approach, which aims to illuminate the many dimensions of uncertainty as well as the sources ('pedigrees') of assumptions that necessarily inform any model forecast or projection [47,48]. In a specific application to large technology projects, Dillon [49], for example, outlines a method to assess cost uncertainties in a way that accounts for the tendency to be optimistic, drawing from the literature on modeling expert behavior and surveying and combining expert judgments [15,50,51]. Others, such as Kujawski [52], argue for direct integration of biases into probabilistic forecasts of engineering cost analysis. AbouRizk approaches the problem by testing statistical distributions on observed cost and duration data, and finds similarly that lognormal and normal distributions very often do not adequately capture the observed patterns [53][54][55].

Case: nuclear power in the US 1970-2005
Several research and policy teams have assessed the potential for nuclear power [56,57] and outlined plans for both incremental and fundamental changes to nuclear reactor design that are hoped will increase safety and decrease costs [58].
The new reactors would encompass evolutionary improved designs derived from the 'Generation III+' reactors in the near term and more radical 'Generation IV' designs in the medium term. Many studies have examined historical capital costs of US nuclear reactors, focusing on the scope of, and reasons for, the unanticipated capital cost escalation that afflicted this technology. In earlier work, we have presented an integrated estimate of the busbar cost for each US reactor (see figure 1), compared historical experience to projections of future costs [43], and pointed out the possibility for cost surprise in future generations of nuclear power [42].
Using past experience to guide future technology assessments is necessarily a challenge as both the social and technological components change over time. However, the past does provide some indication on the degree of possible variation in the individual elements leading to final cost outcomes for large technology projects. Combining past experience with the earlier discussion of expectation of overconfidence and surprise can provide a transparent assessment of the probabilities of different cost outcomes. In the case of nuclear power, we have reported that the delivered costs of nuclear electricity depend most sensitively on the capital cost and the time series of the realized capacity factor, both of which depend in turn on management, regulatory regime, technological development, and cultural context.
Other elements contributing to costs include construction duration, additions to capital after the initial construction (incremental capital additions), and operations and maintenance costs: where C = cost of electricity (cents per kWh), c k = capital cost of plant, c om = operations and maintenance cost, c ica = cost of incremental capital additions, c r = reserve margin cost, c d = decomissioning cost, c f = fuel cost, c wd = waste disposal cost, c f = capacity factor, and r is the discount rate. The historic distributions of many of the most important components are often not well characterized by a normal or lognormal approximation. Figure 2 plots distributions of six major cost components that have been normalized for ease of comparison. Both Shapiro-Wilkes and Anderson-Darling goodness-of-fit tests demonstrate non-normality of all six distributions in figure 2 with p < 0.001. Importantly, historical experience does not provide much insight into several of the most uncertain components-including waste disposal and decommissioning costs.
We argue that any model of future energy technology costs should encompass not only the distribution or spread of possible outcomes but the uncertainty in the distribution. Figure 2 provides a sense of the substantial degree of this uncertainty.
One approach to incorporating such uncertainties is to utilize multiple, alternate probability density functions (PDFs) for probabilistic analysis of possible future costs. These alternatives could encompass both alternate specifications of the same distribution function and even alternate distribution functions. In selecting alternatives, we suggest that particular attention be paid to the ability of the functions to account for uncertainty in the tails of Figure 1. Total levelized cost of electricity for 99 existing US nuclear reactors [43], ranked by increasing cost and demonstrating cluster of high-cost reactors. Reproduced from [42]. Shoreham levelized cost is based on average operation. the distributions. For example, one of the most significant contributors to realized technology costs for nuclear reactors was the capital cost (including interest). These capital costs can be modeled approximately with lognormal, beta, and gamma functions. However, a cubic function actually provides the closest fit to the observed data for both the very low-cost and high-cost reactors ( figure 3). Both ends of this distribution could be important to understand the probabilities of future surprises, both positive and negative. Our data indicate that specifying the shape of PDFs in advance is highly uncertain and sensitivity analyses should incorporate this second-order uncertainty, such as non-normal distributions of the reserve Figure 3. Deviation of several alternative distributions (normal, lognormal, gamma, and cubic) fitted to one of the major cost components of nuclear power: capital costs with interest (normalized such that 0 = lowest cost and 1 = highest cost). Regions above (below) the zero line indicate that the hypothesized distributions overestimate (underestimate) the observations. Capital costs (which include 6% per year real interest during construction) are a significant contributor to overall levelized costs, and standard distributions do not represent the data well through all cost ranges. margin cost c r , and especially the discount rate r and associated risk premiums on underlying cost components [5].

Discussion: approaches to incorporating uncertainty in future cost assessments
While few observers, experts included, would be truly surprised to find evidence of overconfidence in expert conclusions, appropriate methods to include such awareness in policy debates are not obvious [59]. Indeed, the policy discussion subsumes two significant but separate inquiries: the first is the question of 'surprise for whom?', and the second is the question of 'who pays?'. When considering investments in energy technologies, utilities, legislators, and entrepreneurs clearly face fundamentally different questions deriving from their disparate perspectives on risk, reward, and uncertainty. Overall, the set of policies enacted to encourage research, development, and deployment of new energy technologies should, among other goals, strive to create effective options to enhance future climate and energy security at a reasonable cost.
Our research demonstrates the potential pitfalls of what might be seen as 'expected overconfidence' in forecasting the rates and character of future technological costs. Yet how to approach this expectation is far from clear. One option is to infuse the concept in debates qualitatively (or in alternative scenarios postulating extreme events). A salient example of this approach is provided by the extensive discussions about the precautionary principle in Europe. This approach has advantages in ensuring that the concept of uncertainty becomes part of the fabric of political discussion, which in many cases might be a preferable approach to relying on competing, but largely inscrutable, quantitative results provided by disagreeing experts.
One important element that the cost data do not illuminate is the question of option value of alternative investments. The expectation of a future price on carbon will clearly make low-carbon investments more attractive [60], and the same qualities that give incentives for integrated planning of both utility generation portfolios [61,62] also argue for broad-based research and development portfolios that include a wide variety of low-carbon technologies, perhaps with a bias toward more modular and smaller-scale options [41] in situations of higher risk [63].
In conclusion, it is evident that no single approach can adequately capture the many facets of new technologies that are important to societal choices. Our work with nuclear costs has illustrated the importance of making overt reference to future cost uncertainties. Indeed, it is possible that US nuclear power is itself an outlier, the victim of an unfortunate confluence of events of a specific time and place that will not be repeated again. Yet even this situation can serve to illustrate the risks of committing to massive, capital-intensive technological initiatives. Uncertainties will always remain, but some uncertainties can be incorporated both qualitatively, in public discussions, and also quantitatively, by utilizing multiple PDFs in probabilistic cost assessments. Particularly helpful in this effort would be empirical research to illuminate the causes of non-normal and non-lognormal outcomes, and to what extent these are linked to social processes like overconfidence. In addition, expert scenario elicitation processes focus on the possibility for high cost surprises from areas that are currently uncertain to enable more informed public discussion of technological funding choices.