Assessing confidence in management adaptation approaches for climate-sensitive ecosystems

A number of options are available for adapting ecosystem management to improve resilience in the face of climatic changes. However, uncertainty exists as to the effectiveness of these options. A report prepared for the US Climate Change Science Program reviewed adaptation options for a range of federally managed systems in the United States. The report included a qualitative uncertainty analysis of conceptual approaches to adaptation derived from the review. The approaches included reducing anthropogenic stressors, protecting key ecosystem features, maintaining representation, replicating, restoring, identifying refugia and relocating organisms. The results showed that the expert teams had the greatest scientific confidence in adaptation options that reduce anthropogenic stresses. Confidence in other approaches was lower because of gaps in understanding of ecosystem function, climate change impacts on ecosystems, and management effectiveness. This letter discusses insights gained from the confidence exercise and proposes strategies for improving future assessments of confidence for management adaptations to climate change.


Introduction
Dealing with highly uncertain decision outcomes has always been a key challenge of long-term ecosystem management. Today, global climate change is simultaneously making this challenge more difficult and more urgent. While it is widely acknowledged that improved characterization and communication of uncertainty should lead to more robust decisions, achieving this is easier said than done for actually implementing ecosystem management solutions to deal with climate change. This is because, at least in part, we so far lack systematic explorations of uncertainty (and uncertainty approaches) for this problem.
Natural resource management practices were developed during the relatively stable climatic conditions of the last century, under a theoretical notion that ecological systems tended toward a natural equilibrium state for which one could manage (Dixon 2003, US-GAO 2007, Heller and Zavaleta 2009. As a result, many ecosystem planning, management, and monitoring methods were based on the assumption that climate patterns and ecological processes would remain stable, save for the direct impacts of management actions in the context of historical short-term (e.g., year to year) variability. Indeed, many government entities traditionally have identified a 'reference condition' based on historical ranges of variability as a guide to future desired conditions (e.g., see Dixon 2003).
In recent years scientists and resource managers have recognized that climatic influences on ecosystems in the future will be increasingly complex and outside the range of historical variability (e.g., USDA 2007, IPCC 2007, USGCRP 2009, West et al 2009 and that management plans must be periodically reevaluated and revised to ensure that goals can continue to be met. This new reality necessitates transformation of management and goal-setting approaches to a more dynamic and variable framework if we are to manage ecosystems effectively under climate change.
However, even as resource managers become increasingly aware of the threats posed to their management goals due to climate change, a major limitation to addressing this threat is lack of guidance on what concrete steps to take (Hulme 2005, Wilby et al 2010. Any proposed guidance must consider the institutional cultures that have shaped past decisions, and build from the practical experiences that managers have accumulated from years of dealing with other stresses such as droughts, fires and disease outbreaks. A number of recent efforts have attempted to address this need by developing systematic frameworks for climate change vulnerability assessments that take into account specific management contexts (e.g., Johnson and Weaver 2009, Matthews and Wickel 2009, Glick et al 2011. Such frameworks incorporate particular management goals and endpoints of concern; appropriate time horizons and spatial scales for monitoring, management, and planning processes; and other factors such as risk preferences and tradeoffs of alternative management options.
In addition, any adaptation guidance should explicitly address the uncertainties surrounding the potential effectiveness of available approaches. Evaluation of uncertainty will be influenced by the management context mentioned above, including institutional cultures and the types of practical experience that managers have had with their systems. A method for estimating levels of uncertainty, along with an analysis of sources of variability among the estimates of different groups, is the subject of this letter.
The test bed for our exploration of uncertainty is a recent report prepared under the auspices of the US Climate Change Science Program (CCSP 2008). This report provides a preliminary review of adaptation options for climatesensitive ecosystems and resources in the United States 1 . It examines the founding mandates, management goals, and key ecosystem components and processes fundamental to six different management systems: National Forests, National Parks, National Wildlife Refuges, Wild and Scenic Rivers, National Estuaries and Marine Protected Areas. For each management system, an author team of 6-11 experts reviewed adaptation options for building resilience to climate change, drawing on the peer-reviewed literature, expert opinion, and the results of a series of workshops that brought together resource management scientists and representatives of the managing agencies.
The point of interest for this letter is the qualitative uncertainty analysis presented in the report in which the author teams assessed the current level of scientific confidence in the examined adaptation approaches. The design of the analysis was informed by general guidance from recent large assessment efforts such as those of the Intergovernmental Panel on Climate Change (IPCC) and the US CCSP (e.g., Moss andSchneider 2000, IPCC 2005 2 ). One fundamental principle of this guidance is the distinction between uncertainty expressed in terms of quantitative likelihood of an outcome versus the qualitative level of confidence in the science underlying the finding. Likelihood is relevant when assessing the chance of a defined future occurrence or outcome. When the maturity of the scientific knowledge base is sufficient, it is considered best practice to assign numerical probabilities to qualifiers such as 'probable', 'possible', 'likely', 'unlikely', etcetera, to avoid differing interpretations among people and contexts (CCSP 2009). This is in contrast to level of confidence, or the degree of belief within the scientific community that knowledge, models and analyses are accurate, expressed via expert judgment about the available evidence and the degree of consensus in the interpretation of this evidence. As the study of management adaptation options to support ecological resilience to climate change is still emerging as a field of research, the CCSP (2008) team deemed it most appropriate to focus on qualitative evaluations of relative levels of confidence in the efficacy of different adaptation approaches.
Here we explore the results and insights that emerged from this confidence exercise. In addition to systematically examining scientifically based confidence in the relative effectiveness of different adaptation approaches, the exercise also yielded insights into: (1) where gaps in research and understanding of ecosystem features and responses to management applications make confidence difficult to gauge; (2) how different perspectives (based on management system) may affect interpretation of adaptation theory and practice (and hence, confidence); and (3) areas for caution in using this type of confidence exercise methodology.

Methods
The confidence exercise of CCSP (2008) elicited qualitative confidence levels for the efficacy of each of seven adaptation approaches identified and reviewed in the report (table 1). The author team for each management system assessed confidence in the approaches independently from the other author teams.
The author teams were instructed to consider confidence statements (table 1) for the adaptation approaches using two attributes of the state of knowledge regarding each statement: (1) the amount of evidence available to assess the effectiveness of a given adaptation approach in supporting resilience; and (2) the level of agreement or consensus in the scientific community regarding the different lines of evidence. These attributes were further defined for the author teams in the following way: • High/low amount of evidence. Is this adaptation approach well studied and understood, or instead is it mostly experimental or theoretical and not well studied? Does your experience in the field, your analyses of data, and your Focusing management protections on structural characteristics, organisms, or areas that represent important 'underpinnings' or 'keystones' of the overall system Strategic protection of key ecosystem features is an effective way to preserve or enhance resilience to climate change Representation Protecting a portfolio of variant forms of a species or ecosystem so that, regardless of what climatic changes occur, there will be areas that survive and provide a source for recovery Representation is effective in supporting resilience through preservation of overall biodiversity Replication Maintaining more than one example of each ecosystem or population within a reserve system such that if one area is affected by a disturbance, replicates in another area provide insurance against extinction and a source for recovery Replication is effective in supporting resilience by spreading the risks posed by climate change

Restoration
Rebuilding ecosystems that have been lost or compromised.
Restoration of desired ecological states or processes is effective in supporting resilience to climate change Refugia Using areas less affected by climate change as sources of 'seed' for recovery or as destinations for climate-sensitive migrants Refugia are an effective way to preserve or enhance resilience to climate change at the scale of species, communities, or regional networks Relocation Human-facilitated transplanting of organisms from one location to another in order to bypass a barrier (e.g., urban area) Relocation is an effective way to promote system-wide resilience by moving species that would not otherwise be able to emigrate in response to climate change understanding of the literature and performance of specific adaptation options under this type of adaptation approach indicate that there is a high or low amount of information on the effectiveness of this approach?
• High/low amount of agreement. Do the studies, reports, and your experience in the field, analyzing data, or implementing the types of adaptation strategies that comprise this approach reflect a high degree of agreement on the effectiveness of this approach, or does it lead to competing interpretations?
The expert teams considered a variety of sources and types of evidence in making their judgments, including peer-reviewed literature, gray literature (e.g., government reports, management plans and workshop reports), data and observations, and model results, as well as the authors' own experience (CCSP 2008). To enable others to understand how the confidence judgments were reached, each team provided a narrative explaining the information and reasoning through which they arrived at each decision.

Results
Based on the exercise, the state of knowledge underlying any judgment can be sorted into a 2 × 2 matrix of categories (following Moss and Schneider 2000, figure 1): • Well established = high evidence/high agreement (HH).
A summary of the results of the confidence exercise is shown in figure 2, which graphically represents the confidence categories of figure 1 for the seven adaptation approaches, according to management system. Across all Figure 2. Confidence categories associated with seven different adaptation approaches, examined across six different management system types. HH: high evidence/high agreement; HL: high agreement/low evidence; LH: low evidence/high agreement; LL: low evidence/low agreement; NA: not rated. CCSP (2008). six management systems, reducing anthropogenic stresses emerged as the adaptation approach ranking highest in overall confidence, as far as taking action today using the best available information. Relocation stands out as the weakest based on current information 3 . The remaining approaches, protecting key ecosystem features, representation, replication, restoration and refugia all received more variable confidence rankings across the different systems/teams. Each of these remaining approaches received a top confidence rating of HH from at least one author team and each also received the lowest confidence rating of LL from at least one author team. A simple ranking system (presented in table 2) confirms the clear prevalence of reducing anthropogenic stresses in terms of highest confidence and the equally clear separation of relocation from the other approaches as the weakest in terms of confidence. The remaining five approaches cluster in a middle ground of confidence rankings.

Discussion
Given the qualitative nature of the exercise, each team provided a narrative explanation of the basis for their estimates under each adaptation approach (included as Table 2. Rankings based on figure 2 results. Rankings are based on tallying results according to the following system: HH = 3 points; HL and LH = 2 points; LL = 1 point; NA = 0.
Adaptation approach rankings across management systems 1 Reducing anthropogenic stresses (17 points) 2 Protecting key ecosystem features (13 points) 3 Representation and replication (tie) (12 points) 4 Restoration and refugia (tie) (11 points) 5 Relocation (5 points) annex B in CCSP 2008). Together, the narratives and the confidence ratings yield a number of insights. First, as intended, it is possible to infer some differences in the perceived effectiveness of alternative adaptation approaches based on amount of scientific evidence and agreement, both within and across management systems. Where it is not possible to distinguish relative effectiveness, gaps in understanding of ecosystem responses to climate impacts and management adaptations are revealed. The results also provide some clues about the effects of differing institutional viewpoints and management perspectives. We discuss these insights in more detail below.

Gaps in research and understanding
There was only one approach for which the amount of research and understanding was sufficient to warrant a rating of high confidence across almost all of the groups (figure 2, table 2). Reducing anthropogenic stresses ranked highest in confidence because of the wide management experience that has accumulated across time in dealing with such stresses. This broad synthesis finding is consistent with some other recent work (see, e.g., Prober et al 2012). Strategies to address anthropogenic stresses are readily accessible to the expert teams via their own experience and the literature. Managers have had many opportunities to study the effectiveness of these types of strategies at building resilience in the past, which increases confidence that the same strategies would be similarly effective under climate change. All teams except for the Marine Protected Areas group rated this approach HH. The rating of LH by the Marine Protected Areas group indicates that less scientific information on this approach has accumulated for this system compared to others; but there is still a perception of high agreement in the scientific community that the approach does increase resilience. In fact, the 'evidence' component of confidence is lower for Marine Protected Areas across the board (i.e., across all of the adaptation approaches examined) (figure 2), which could be a reflection of the shorter period of time that scientists have been studying marine systems compared to many terrestrial systems such as forests.
In contrast to reducing anthropogenic stresses, all teams reported little information (evidence) available about relocation (i.e., human-facilitated transplantation of organisms from one location to another in order to bypass a barrier) or its implications for ecosystem resilience (figure 2) and little agreement that it is a robust approach. The author teams cited a paucity of examples of relocations being attempted, with even less information on their success, and no information on implications for resilience. Two teams (National Estuary Programs and Marine Protected Areas) did not even provide confidence scores for this option due to lack of information upon which to judge confidence. This resulted in the lowest confidence ranking for this approach (table 2). At the same time, it is important to note that future research may change this ranking (as well as the rankings of other approaches) at any time.
The remaining five approaches are in the middling group, with little distinction among their rankings (table 2). Protecting key ecosystem features, replication and representation 4 , and refugia are all precautionary approaches (i.e., where one 'bet-hedges' with multiple reserves, or focuses protection on particular areas or features within reserves). They involve 'simple' protection of the ecosystem from outside forces, as compared to more 'complex' modifications of the ecosystem to strengthen resilience. While such precautionary measures are widely perceived as logical and potentially effective for many ecosystems, there was less evidence for their efficacy, such that confidence in these approaches did not rise to the level of that for reducing anthropogenic stresses. This could be due in part to (a) uncertainties surrounding geographic shifts in climate that might compromise the habitats of spatially fixed reserve systems (for replication, representation and refugia) or (b) uncertainty regarding the characteristics of a given ecosystem that contribute most to conferring resilience (for protecting key ecosystem features).
In contrast, restoration involves proactive direct modification of the ecosystem, such as construction of a habitat to include an assemblage of critical species. An additional layer of uncertainty is introduced by these types of manipulations since it is difficult to predict their degree of effectiveness as well as the long-term response of the ecosystem.
In addition to the above sources of variation in confidence, the narratives also indicated a number of other potential gaps in research and understanding. Some teams noted that for several approaches, there is more information available at the species level than at the ecosystem level. Likewise, there is a different and typically greater knowledge base available for plants versus animals with regard to their potential responses to some approaches. Finally, several teams noted that in some cases an approach may be well documented at small scales (e.g., management units), but there is little information on whether this translates to resilience at larger scales (e.g., system-wide).

Varying interpretations based on differing management perspectives
Beyond differences in scientific understanding (about species or ecosystem responses), the narratives also revealed some philosophical differences that helped drive differences in confidence across the various author teams. Perspectives could differ based on the institutional culture of the management system community, the types of ecosystems found in the management system, and the composition of the author team.
One example is the philosophical divide between confidence gained as a result of theory versus practice. This explains the difference between the confidence ratings assigned to refugia by the Wild and Scenic Rivers (H/H) and National Forests (H/L) teams. Both teams cited high amounts of evidence of past and current examples of climate refugia, but they differed in their interpretations of agreement as to whether refugia could be used to enhance resilience. The rating of high agreement by the Rivers team was based on a perceived concurrence within the scientific community that the theoretical approach would work in practice. In contrast, the Forests team was explicit in stating that lack of empirical testing of the use of refugia for resilience necessitated a rating of low agreement for the approach. Ebi (2011) has criticized the ambiguity inherent in the lumping together of theory and empirical findings in this type of IPCC-style rating system, and our results suggest that this can indeed introduce variability among the conclusions of different expert teams.
The theory versus practice dichotomy was also influential in determining how the different groups perceived the value of preserving biodiversity for guiding adaptation planning and implementation. Biodiversity played a key role in team discussions of the protecting key ecosystem features and representation approaches. These approaches were evaluated by some expert teams as having strong evidence for effectiveness in practice, based largely on their perception of the scientific community's general agreement on the theoretical evidence. Institutional culture also may have played a role in the faith different expert teams displayed in biodiversity-based approaches. For example, the presence within originating legislation of mandates to preserve diversity, and subsequent programmatic enactments of the legislation, may have positively influenced the confidence assessments of some expert teams 5 .
Another philosophical difference that emerged centered on whether information at the species level was sufficient to constitute high evidence for an adaptation approach, or whether evidence of ecosystem-level effectiveness was required. This distinction explains the diametrically opposed ratings for representation and replication given by the National Wildlife Refuges and Wild and Scenic Rivers teams (H/H) compared to the National Forests and National Parks teams (L/L). The Refuges and Rivers teams based their ratings on a large amount of information available at the species level for the utility of the approaches for supporting resilience at the species/population level. Conversely, the Forests and Parks teams concluded that-although information was available at the species level-there was insufficient information at the ecosystem level (which they regarded as essential), so their ratings were low.
Finally, a similar phenomenon of opposing interpretations occurred with regard to perspectives on geographic scale. Judgments on confidence varied based on whether the team required evidence of resilience only at the scale of small management units, or whether evidence at the larger scale of the overall system was necessary. For example, although many studies exist of restoration done at small scales, the National Wildlife Refuge team concluded that the overall amount of evidence was low for the restoration approach and rated it LL because few studies had documented effectiveness at large scales. This contrasts with the HH rating by the National Parks team, which was based on their familiarity with abundant studies at smaller scales.

Conclusions
In this qualitative assessment of confidence in adaptation approaches for a range of federally managed systems in the United States, there was near-consensus in judgments for two approaches: high confidence in adaptation options that reduce anthropogenic stresses and low confidence in adaptation options that rely on relocation of species. Meanwhile, where gaps in research and understanding made confidence difficult to gauge, the ratings of the five remaining approaches were highly variable (table 2).
Analysis of this across-group variability reveals some pitfalls inherent to this type of expert-judgment approach. Examination of the narratives reveals the importance of (1) types of evidence considered when evaluating effectiveness and (2) divergent assumptions made in the interpretation of 'amount' of evidence and 'agreement' in that evidence. The types of evidence considered by expert teams included the scientific literature, data and observations, model results, and personal experience (CCSP 2008). When there was less overall evidence available on adaptation approaches, some teams gave greater weight to empirical evidence gained through experience than evidence based on theory alone. When paleoclimatic data existed, some expert teams used this information when evaluating the amount of evidence available while other teams made no mention of it because they either gave it no weight, or very little weight.
Divergent assumptions about what constitutes 'high' versus 'low' evidence and agreement also affected the confidence results. Some teams assumed that studies conducted at relatively small geographic scales or on specific species qualified as evidence for approaches applied at scales of entire ecosystems. Other teams considered only evidence at the ecosystem level in their evaluations. Agreement sometimes became conflated with evidence when little evidence of the effectiveness of an approach was available. For example, when few papers were available on an approach, some expert teams equated that with no agreement on effectiveness among the wider scientific community, even in situations where agreement with the few papers available may have been at or near 100%.
These variations highlight the critical role of decision context in the outcome of any confidence assessment. In this exercise, outcomes were the result of the particular contexts invoked, implicitly or explicitly, by the different author teams as they carried out the exercise. Each author team had to operate within a chosen context in order to make meaningful evaluations, but these contexts also led to differences in how the evidence was evaluated.

Areas for improvement in this confidence exercise
Improvements can be made to this confidence method to improve the accuracy of the exercise and advance the design for future application to achieve higher levels of usefulness for management adaptation planning. Suggested improvements: • Provide more detail and clarity in the rules of the confidence exercise, including processes to: * more clearly define terminology to reduce inherent subjectivity of judgments about evidence and agreement; * determine weighting criteria for different sources and types of evidence (e.g., theory versus practice); * define rules for where the line will be drawn between high and low evidence and high and low agreement.
• Plan for multiple iterations of the exercise, with consultations both within and among teams in order to: * ensure consistency in the application of the approach; * resolve any unforeseen issues (e.g., controversy over the relative importance of evidence at different spatial scales); and * achieve consensus in confidence scores where appropriate.

Next steps for application beyond this study
Today there is new interest in analyzing and revisiting the treatment of uncertainty in these types of large assessment reports. Recent work has identified opportunities for improved consistency in applying the uncertainty guidance described above and has explored deficiencies in the characterization and communication of uncertainty by the IPCC (IAC 2010, Yohe andOppenheimer 2011, Spiegelhalter andRiesch 2011). This letter contributes to this emerging conversation about lessons learned and best practices for treating uncertainty in climate change vulnerability, impacts and adaptation assessments. The next level of refinement for confidence assessments for management adaptation planning may involve evaluating confidence in individual adaptation options within each approach. Methods are needed to: • Identify and conduct more detailed evaluations of confidence in individual adaptation options where levels of confidence in the approach are highly variable across management systems or across ecosystems • Aggregate individual options into summary adaptation approach scores, with the proviso that: * splits in scores may have to occur where aggregation hides large differences in effectiveness of options at the level of species versus ecosystems; * specific adaptation options may have to be noted as exceptions to the aggregated results.
• Refine groupings to ensure comparability of approaches in terms of scope (e.g., replication is one very specific action, whereas reducing anthropogenic stresses is composed of a large variety of actions).
• Ensure appropriate diversity of expert participants, i.e., sufficient to evaluate the diverse sources of information and knowledge required for different options.
Since consensus-based approaches can mask important within-group disagreements, future assessments should also consider reporting the full range of expert judgments to improve the clarity and policy relevance of the assessments (Socolow 2011, Yohe andOppenheimer 2011). Such reporting also makes results transparent and replicable.
Management planning to select and prioritize adaptation approaches will always involve some assessment of confidence, whether implicitly or explicitly. Explicit estimations of confidence, while difficult, afford managers a better understanding of the nature, implications, and risks of different adaptation approaches. However, evaluations of confidence reported here only deal with management adaptation in the near term (the next 30 yr or so). Eventually, confidence exercises will have to address adaptation approaches that meet long-as well as near-term management planning needs. This means that evaluations of confidence will have to consider the existence of thresholds, where and when they may occur, and approaches that manage for transitions into alternative ecosystem states. Evaluating confidence in management approaches under a range of ecosystem states and threshold conditions will greatly increase the complexity and difficulty of this task. Despite these difficulties, such evaluations are essential to enable managers to make difficult but informed decisions about risks, tradeoffs, triage, and planning for future inevitable state changes.