DOs and DON'Ts for using climate change information for water resource planning and management_ guidelines for study design

Water managers are actively incorporating climate change information into their longand short-term planning processes. This is generally seen as a step in the right direction because it supplements traditional methods, providing new insights that can help in planning for a non-stationary climate. However, the continuous evolution of climate change information can make it challenging to use available information appropriately. Advice on how to use the information is not always straightforward and typically requires extended dialogue between information producers and users, which is not always feasible. To help navigate better the everchanging climate science landscape, this review is organized as a set of nine guidelines for water managers and planners that highlight better practices for incorporating climate change information into water resource planning and management. Each DOs and DON'Ts recommendation is given with context on why certain strategies are preferable and addresses frequently asked questions by exploring past studies and documents that provide guidance, including real-world examples mainly, though not exclusively, from the United States. This paper is intended to provide a foundation that can expand through continued dialogue within and between the climate science and application communities worldwide, a two-way information sharing that can increase the actionable nature of the information produced and promote greater utility and appropriate use.


Introduction
The increasing availability and ever-expanding size and number of archives of climate observations and model outputs relevant to water management shifts a practitioner's dilemma. No longer are they simply trying to find climate data, instead they have to now choose the most appropriate data from various options, assess its credibility, and use it wisely (Barsugli et al., 2013). For example, at the time of this writing, there were 629 downloadable climate-related datasets on data.gov (URL accessed 7 August 2017), and 118 with the topic category of water. There is also growing recognition of the need to assess not just the magnitude of impacts, but also to quantify and grapple with the range of uncertainties that arise when estimating future change (Brekke et al., 2009;Weaver et al., 2013;Clark et al., 2016). A growing number of documents are available to help practitioners navigate this ever-expanding landscape (e.g., Table 1). While helpful, these documents can be challenging to keep relevant as climate change research and applications advance. Herein lies an opportunity to design a platform for information exchange that can build and revise guidance alongside other advances and serve to better inform both groups to the work and perspective of the other. It is this possibility that motivates this paper.
Here we introduce a use-focused, modular set of guidelines designed to be expandable and updated. This document's first version, presented here, was built by reviewing past research and synthesizing existing guidance relevant to water resource planning and management mainly, but not exclusively, in the United States (U.S.) (Table 1). It is intended to be used at multiple levels. It can be used by those seeking to understand big-picture needs, opportunities, and challenges of including climate change information in water resource planning and management, but it also provides details and direction to further guidance and resources for those engaged in the technical work. Throughout, we refer to these as DOs and DON'Ts, which encompass both guidelines and interpretations. This is guidance as advice (lower-case g), which differs from Guidance issued as a set of governing principles (by agencies or associations) that may have legal implications.
We demonstrate this framework and focus on DOs and DON'Ts for study design when including climate change information in water resources planning and management. We highlight strategies which we refer to as "low-regret" as they aim to effectively leverage existing resources, create multiple benefits, and reduce the chance of errors and information misuse. This collection of DOs and DON'Ts is intended to help facilitate a dialog among information producers, information   Vano et al. Climate Services 12 (2018) 1-13 users, and others that work at this science-application interface.
To promote common understanding, several terms which have a range of definitions depending on their context are defined, as they are used here, in Appendix A.

Guiding principles
With ever-increasing information, it is easy to get disoriented or buried in details and lose sight of the value the information is intended to bring. We therefore begin with two guiding principles that motivate this work and the research on which this work depends: 1) it is important to evaluate climate risk together with other risks in short-to long-term planning, and 2) models can be helpful tools, if used appropriately.

It is important to evaluate climate risk
Stationarity is the concept that there are no changes in statistical distributions of geophysical variables (e.g., annual streamflow or annual flood peak) as a function of time. If the climate is stationary, past records can provide an adequate estimate of future variability in waterresource engineering. This concept has, however, been undermined by climate change (Milly et al., 2008) and long-term persistence more generally (Lins and Cohn, 2011). Therefore, innovation is needed in considering future change and uncertainty in water resource planning (Milly et al., 2015;Clark et al., 2016;Lehner et al., 2017b).
Evidence is overwhelming that climate change is already affecting managed and natural water systems (IPCC, 2014b;Melillo et al., 2014), e.g., earlier snow melt in Western North America (Stewart et al., 2005), increases in annual precipitation in the U.S. Midwest and Northeast (Georgakakos et al., 2014). The future evolution of the climate system depends on changes in external forcings (e.g., future emissions) and non-linear feedbacks, as well as large-scale human alterations such as land use change and demographic changes. While there will always be uncertainty from human behavior, natural climate variations, and our ability to simulate them (IPCC, 2012), there are many things we do know -e.g., warmer temperatures will increase evaporation potential, reduce snowpack, and change the seasonality and magnitude of streamflow. As a consequence, water resources planning studies must explicitly account for climate change alongside other important but uncertain changes such as population growth and land use change. For agencies authorized to manage water now and in the future, to continue delivering on authorized purposes, it is important to understand changing conditions and prepare to operate in them. This is simply due diligence and it would be neglectful and irresponsible to do nothing.
Money, livelihoods, and lives are at stake. This urgency is evident in a policy statement from the American Society of Civil Engineers that emphasizes the need for consideration of climate change in planning, building, and maintaining U.S. infrastructure, "Engineering practices and standards associated with these facilities must be revised and enhanced to address climate change to ensure they continue to provide acceptably low risks of failures in functionality, durability and safety over their service lives" (ASCE, 2015).
Additionally, climate change may provide opportunities that appropriate planning can exploit (Overpeck et al., 2011). Examples include adjusting agricultural systems to benefit from longer growing seasons and adapting hydropower systems to adjust to changing hydrographs (NRC, 2012a). It is also advantageous for those managing the systems to direct the climate change analysis (Reclamation, 2014a). It is easy to imagine challenges (e.g., confusion over operating constraints, unworkable adaptation measures) which could arise if external parties frame the climate change analysis instead of those involved in current system management.
Various methods exist to better understand and prepare for future changes in climate. All evaluations of a water system's vulnerabilities to climate change need to start somewhere and can be refined and become more comprehensive over time. Depending on a project's goals, resources, and other tradeoffs, consideration of climate impacts in water planning can occur at multiple levels, from general regional temperature trends to detailed evaluations of changes in reservoir storage. The option exists of doing nothing, but this should be a conscious and informed decision. If not, simply ignoring climate risk may result in being unprepared or unable to adapt -a risk water agencies, cities, counties, states, and many others are increasingly less willing to take (Vogel et al., 2015;EPA and CDWR, 2011).

Models can be helpful tools, if used appropriately
Watershed-relevant climate change scenarios can provide information useful in assessing how the system is vulnerable to climate change and help identify adaptation options. To generate climate change information at the global, planetary scale and make it relevant to local watersheds, many methodological choices must be made by both information producers (on how to generate the datasets) and users (on how to apply the climate data to their decision). In the U.S., for example, the U.S. Army Corps of Engineers has 21 regional reports Models, including global and regional climate models, as well as watershed models, are used to explicitly characterize possible futures as well as historical and current conditions. These simulated futures, often referred to as projections, when used together with simulated historical conditions, can then be used to assess potential changes. More specifically, evaluating relative differences (modeled historical vs. modeled future) in system performance over time can provide improved perspectives on potential improvements as well as risks 1 . In this, it is important to recognize that model outputs are not intended to be predictions, and should be treated instead as possible future 'scenarios' which can complement existing monitoring and performance evaluation systems. They provide an opportunity to explore how natural and managed systems may respond to and influence future changes and to investigate uncertainties (Weaver et al., 2013;IPCC, 2014b;Milly et al., 2015;Reclamation, 2016). Scenarios 2 can be viewed as narratives that can be used to stress-test water systems and infrastructure (Moss et al., 2010;Weaver et al., 2013). As such, a single stress test can be misleading when viewed in isolation; multiple stress tests, especially when they span a range of possible stresses, are preferred and can be added to as resources and time permit.
In performing these stress tests, current approaches often capitalize on "ensembles of opportunity" -that is, collections of available datasets -to evaluate the range of future impacts and their uncertainties. This may be the most appropriate path forward at present; although as the field of climate change impacts advances and computing capacity improves, it will be possible to better understand and quantify underlying uncertainties (Harding et al., 2012;Gutmann et al., 2014;Clark et al., 2016), evaluate and account for model dependencies (Knutti, 2010b;Bishop and Abramowitz, 2013), and improve how models are selected for use including ensuring they capture features that make them appropriate for particular uses (Knutti et al., 2010a;Tebaldi et al., 2011;Sanderson et al., 2015). This paper reviews important considerations when designing studies so models can be useful tools in exploring future change.

Methods
This review is framed as a series of DOs and DON'Ts for those who want to design a study that uses climate change information in water management and planning. Each of these DOs and DON'Ts contains: (1) explanations for why each is important, providing insights into the decisions made by information producers, (2) real world examples and frequently asked questions (FAQ) by information users, and (3) recommended paths forward. This framework serves as a foundation for further refinement (e.g., not all questions are asked or answered here, and even the ones that are answered can be expanded on and further refined), which we designed to facilitate dialogue between information producers and users.
This guidance is drawn from the experiences of the authors and supported by existing peer-reviewed literature and reports used by practitioners with operational examples ( Table 1). The nine DOs and DON'Ts were developed by this author team, through consensus. Each had to be: (1) relevant to water managers and planners, (2) useful in the process of designing a study, and (3)  This paper is intended to help those interested in applying climate change information to support water management and planning. However, this framework also encourages the research community to consider, through describing DOs and DON'Ts, what appropriate uses of different types of climate change information should be. Many DOs and DON'Ts are most relevant to studies that use downscaled climate model outputs and hydrologic models, but it is clear that no one-size-fits-all approach exists and that other approaches may be more appropriate in certain situations, the nuances of which are described throughout this paper. Individual DOs or DON'Ts stand alone yet reference each other; they can be read in any order but are sequenced here in a logical progression of considerations when designing a study.
Published guidance that relates to water resources (e.g., Table 1) have a variety of formats, come from a diversity of organizations, and are intended for many different audiences. This diversity provides a comprehensive review of ways climate change information is used to inform water management and planning in the U.S. Notably, this review is focused on the use of climate change information, not on how the information is created. For those interested in a primer on climate change information, many resources are available (Miller and Yates, 2006;NRC, 2012b; Appendix 2 in Reclamation, 2014a; http://nas-sites. org/climate-change, among many others).

DOs and DON'Ts when setting up a study
Below are nine recommendations to consider when setting up a study that highlight low-regrets strategies. Each recommendation is supported by an explanation of why, examples and/or frequently asked questions, and a path forward.

Why
Including climate change information in water resource planning and management can reveal new vulnerabilities of a system to climate change and uncover ways to improve system resilience. This aligns with the traditional planning process, which tests system responses to a range of hydroclimate conditions (from normal to extreme) along with a broad range of other factors that determine system performance, including changes in demographics, system demands, environmental constraints, infrastructure concerns, and finance scenarios. Evaluations of a system's vulnerability to climate change must be undertaken via the addition of a broader range of possible futures than are indicated by historical records alone, although they need not be a separate project (e.g., mainstreaming climate change considerations, Vogel et al. (2016b)).
A major benefit of considering how a water system will be impacted by future change is that it raises questions of how the current system is understood and how it might be understood better (Vanrolleghem, 2010). This can lead to innovations that can help improve a water system's hydrologic modeling capabilities generally. For example, the Portland Water Bureau used a climate assessment as an opportunity to set up an in-house hydrologic model for the Bull Run watershed, Portland's primary water supply (Vogel et al., 2016a). Additionally, understanding ways to be more prepared for climate-driven impacts and addressing the inherent uncertainties in future climate in planning can improve a water system's adaptive capacity for non-climate-driven impacts as well (Olsen et al., 2015;Freitag et al., 2012) and motivate and facilitate improvements in ongoing management, including drought monitoring, streamflow forecasting for floods and droughts, and enhanced water conservation.
Climate change evaluations also build human capacity. They often require decision makers address difficult questions such as how to value dissimilar types of impacts (Willows and Connell, 2003). They can motivate people with different management goals such as irrigation districts and environmental groups to work together (Eberhart et al., 2013;Malloch and Garrity, 2015). Ultimately, the ability of a community to rapidly recover from a disaster is increased when a diverse group of stakeholders can work together to monitor potential hazards and modify plans and activities to accommodate future change (NRC, 2011).

The Observational Method
The American Society of Civil Engineers' Committee on Adapting to Climate Change recommends a modified version of the Observational Method as a useful approach for including climate change in planning that would also make the system more adaptable to other changes (e.g., population and land use change). The goal of the Observational Method, developed in geotechnical engineering over 60 years ago, is to design a system with contingencies for all foreseeable problems the system might face. Then, the system is continuously monitored (in a reliable, transparent way) for metrics which indicate when it is time to enact the contingencies. This assures safety and allows for more economic design, as long as changing conditions are monitored and design modifications are possible including funds, authority, and willingness. An example would be increasing water storage incrementally through time to store water to compensate for disappearing snow pack. If future flexibility is not guaranteed or there is no solution for all hypothetical problems, even those with low probability of occurrence, then the design should be based on least favorable conditions (Olsen et al., 2015). The method is helpful for gradual changes that can be monitored. For water resources, this could help in adapting to sea-level rise or changing permafrost, but may not be appropriate for extreme events where damages may occur before changing conditions can be observed (Olsen et al., 2015).

Yakima River Basin Integrated Plan
Consideration of climate impacts can provide motivation to overcome longstanding differences. As a newspaper article jointly written by an irrigator, an environmentalist, and a tribal member states, "After 30 years of conflict and increasingly frequent drought in the Yakima River Basin, a diverse coalition of farmers, conservationists, the Yakama Nation, and government officials have hammered out a national precedent-setting vision to improve water security for farms and communities, bring back salmon and steelhead, and protect and restore the streams and forests of the river's headwaters" (Eberhart et al., 2013). This unusual coalition was largely motivated by climate concerns, coupled with endangered fisheries and a failed attempt to build the Black Rock Reservoir (Malloch and Garrity, 2015). The climate impact studies, done at the University of Washington from 2007 to 2009, showed the basin was especially susceptible to loss of snowpack, a water supply source that allows the basin's five constructed reservoirs to hold water for irrigation from the region's wet winter to the dry summer Vano et al., 2010b).

Path forward
Highlight planning win-wins. When possible integrate climate change evaluations into ongoing monitoring, planning, and operation improvement efforts. Together they can serve to improve the ability of a water system to be resilient to floods, droughts, and other hazards, both now and in the future.

DO start by determining the level of detail that fits your need and resources 4.2.1. Why
Climate change evaluations can be done at a variety of levels ranging from qualitative regional descriptions of temperature and precipitation changes as in the IPCC reports (e.g., 2014b) and U.S. National Climate Assessments (e.g., Walsh et al., 2014) to quantitative daily climate change projections of streamflow at a specific gage location (e.g., Vano et al., 2010a,b). It is, therefore, important to first understand what information is needed to answer the climate change questions posed, what is possible, and the tradeoffs between the required effort and detail. It may also be important to balance investments across various aspects of a study, for instance, considering how a water system is vulnerable to both climate and non-climate risk factors.
Generally, it is better to start with an understanding of big-picture changes and then decide what details are needed to help inform decisions, so they can be explored most effectively (Willows and Connell, 2003;Brekke et al., 2009). Region-based inquiries and qualitative analysis are usually relatively simple and cost-effective (Reclamation, 2014a) and can be a good starting point even if more involved analysis is desired. Willows and Connell (2003) describes this as a tiered approach -by first studying the problem in a broad, holistic way, risks can be characterized qualitatively and then prioritized, which allows the most significant risks to be assessed first.

FAQ: How does one know what level of detail is needed?
The required level of detail depends on the decisions being informed (see Section 4.4 for more on decision criteria). Climate change information has a wide range of applications in water resource planning, for example the information can be used to modify system operations, to make decisions on new or improved infrastructure, to establish longterm planning objectives, and to plan river restoration (Reclamation, 2014a). In all decisions, identifying the minimum level of information required to alter a decision can help. For example, some decisions can be made by just knowing a direction of change (e.g., summer temperature increases). Other decisions require a better understanding of the magnitude of change for one or more variables. Still others require an investigation of relative differences (e.g., identifying stream reaches more vulnerable to temperature increases for endangered species protection (Mantua et al., 2010;Isaak et al., 2015)).
The level of certainty must also be considered. Reclamation guidance (2014a) recommends considering both relevance and certainty when determining the appropriate level of climate change analysis. The report suggests that climate change information should be: (1) included if changes are well supported and relevant, (2) explored through sensitivity analysis if changes are highly uncertain, but still relevant, and (3) excluded if changes are irrelevant or too uncertain. If too uncertain, one should consider more carefully the costs and risks involved and consider planning for more severe scenarios or other contingency-based planning (e.g., Observational Method example above).

FAQ: How does one know what climate change information is appropriate?
Fortunately, many resources, tools, and techniques already exist (suggestions in Section 4.3). The questions below, included in the UK Climate Impacts Programme's report Climate adaptation: Risk, uncertainty and decision-making (Willows and Connell, 2003), can be used to help evaluate appropriate tools or techniques for specific situations: • How much will it cost? (tool development, staff time, expert assistance) • How long will it take? (no matter how useful the tool, it is of little use if it cannot make the decision deadline) • To what extent will the analysis improve the decision? (what information is required to make a different decision) • Can appropriate data and information be obtained? (if not, reconsider costs and timeline) • Who will undertake the analysis? (in-house, external expert)

Path forward
Recognize that climate change evaluations require varying levels of detail. When starting, a qualitative understanding of change is useful (big-picture, regional changes), and will help bound more quantitative, detailed analysis which may be required depending on the decisions the information is informing.

Why
In the past two decades, the number of studies and people working on applying climate change information to water management and planning has increased (EPA and CDWR, 2011;Reclamation, 2013;EEA, 2017). Therefore, those just beginning do not need to start from scratch. There is a steadily growing resource of climate change information, guidance, case studies, and networks to connect to. Increasingly more organizations, both public and private sector, exist that serve to connect practitioners with researchers to generate more decision-relevant information (for example, see the diversity of organizations involved in creating guidance documents in Table 1). These efforts are underway and increasing in many arenas (e.g., public utilities, federal and state agencies, private sector, non-governmental organizations) throughout the world. Because climate science requires complex models that may not be well-understood by water managers, and climate scientists are unfamiliar with challenges of planning, designing, operating and maintaining water systems, building partnerships, trust, and shared resources between information producers and users is critical, and an increasing number of guidance documents exist on how to effectively foster these partnerships (Jacobs, 2002;Ferguson et al., 2014;Addor et al., 2015;Beier et al., 2016).

FAQ: How to discover who is doing what, where?
The Intergovernmental Panel on Climate Change (IPCC) provides a global perspective on the state of climate science (IPCC, 2014a,b). In the U.S., the U.S. Global Change Research Program has compiled U.S.based research by sector and region in the National Climate Assessment (Melillo et al., 2014;NRC, 2017). Other countries have similar regional assessments (e.g., Warren and Lemmen, 2014;EEA, 2017). For localscale studies related to water in the U.S., the Bureau of Reclamation has three editions of a literature synthesis on climate implications for water and environmental resources that covers 17 western States (Reclamation, 2009(Reclamation, , 2011b(Reclamation, , 2013. The Climate Change Handbook for Regional Water Planning has a summary of climate change information relevant to integrated regional water management planning (EPA and CDWR, 2011), reviewing 167 articles. The U.S. Army Corps of Engineers has 21 regional reports that summarize hydrological and climate changes and their subsequent impacts on USACE projects (http:// www.corpsclimate.us/rccciareport.cfm). In Europe, the Copernicus Climate Change Service (C3S) provides case studies and access to climate information for water management through two proof of concept projects: SWICCA, Service for Water Indicators in Climate Change Adaptation and EDgE, End-to-end Demonstrator for improved decision making in the water sector in Europe (https://climate.copernicus.eu/ about-c3s).
We reference these reports as starting points to navigate to the many universities, national centers, federal, state, and local agencies, and other entities working on these challenges. The organizations and authors of these reports and the literature they cite provide insights into who is doing what, where.

FAQ: Where does one get climate data?
For a better understanding of large-scale regional changes, the reports mentioned above (e.g., IPCC, 2014a,b; Melillo et al., 2014; EEA, 2017) provide regional maps. Additional maps and links to downloadable data are located in many places (e.g., http://gdo-dcp.ucllnl. org; https://www.data.gov/climate/portals; Section 4 in Reclamation, 2016). Notably, some locations distribute raw data, and others cater to specific uses, which may not be appropriate for other applications. In practice, products are often selected because they are easy to access, in a convenient format, or are otherwise familiar to the user -yet these criteria do not necessarily align with what would be the most appropriate (Barsugli et al., 2013).

Path forward
Many resources and organizations exist that can help one better understand climate change information and approaches to using it; consult them to leverage past work and connect with local experts.

Why
Clearly articulating how climate-related decisions will be assessed before starting to evaluate the data will provide a more objective evaluation, more quickly. It can also help in deciding the best approach (e.g., the level of detail needed, Section 4.2) and guide model selection. All together, this will help insure the climate change information is fit for purpose in that the information obtained is appropriate for the questions that are being asked of it.

What are climate change evaluation criteria?
Evaluations can include a variety of criteria (Table 2). Being specific about the criteria helps to define an approach and determine whether the change is significant, i.e., how big of a change matters and with what degree of confidence. Rarely, however, is there just a single concern and decision makers must prioritize (Palmer et al., 1999). This list provides examples; additional water-related climate impacts can be found in past guidance (e.g., EPA and CDWR, 2011, Box 4-1) and an overview of a range of hydroclimate metrics and their relative ability to characterize hydrologic changes can be found in Ekström et al. (2018).

How can evaluation criteria help model selection?
The general idea (expanded on in Section 4.5) is that models should be used that have appropriate capabilities and can be evaluated based on those capabilities to show they adequately represent the variable(s) of interest. The National Research Council (NRC) report on advancing climate models (2012a) shows the time scale and spatial extent for key climate phenomena and the relative climate model reliability. Most water resource impacts involve processes that occur at a local scale and thus require downscaling of the climate model outputs and the application of hydrologic models. For example, if floods are the focus, the downscaling method should adequately capture flood-generating precipitation events and the hydrology model should adequately represent peak flows in current climate, and the processes that could lead to flooding in a future climate (e.g., rain on snow).
Knowing the variables of interest can help determine the climate variables and models needed to simulate changes -for example, streamflow estimates require more climate variables (e.g., daily temperatures, precipitation, wind speed) than temperature changes alone (Reclamation, 2014a). Hydropower assessments will require consideration of reservoirs (e.g., Hamlet et al., 2010;Kao et al., 2015); sealevel rise and streamflow estimates require completely different approaches, although in some places both are required (Hamman et al., 2016).
Knowing the planning horizon can help determine the approach. For shorter periods (e.g., 20 years into the future), the various greenhouse gas emission scenarios will be more similar and a qualitative analysis and literature review might be adequate (Reclamation, 2014a).

Path forward
A first step in assessing impacts should be to define the climatedependent decisions and consider the type of changes that would cause concern. Evaluation criteria in Table 2 can help inform those discussions.

Why
Future changes and uncertainties due to climate change are most meaningful when placed in context. Therefore, once the variables of interest have been identified (Section 4.4), it is important to understand how well they are currently known and can be simulated both now and in an altered climate. This includes model simulation uncertainties (e.g., how well does the hydrology model used capture peak flows, low flows, seasonality), but also other uncertainties that could impact decisions (e.g., operation changes because of deteriorating infrastructure or increases in water demand) (NRC, 2009;Brekke et al., 2009). whether it is shifted (e.g., uniform shift to warmer climate), expanded (e.g., increased temperature variability), or otherwise altered (e.g., increased asymmetry toward the warmer part of the distribution), see SPM.3 in IPCC (2012) Time horizon of interest when in the future, e.g., by 2020s, 2050s, or 2080s Risk tolerance e.g., 80% of future simulations indicate X Model performance thresholds how well models must perform to be viable, see Section 4.5 J.A. Vano et al. Climate Services 12 (2018) 1-13 4.5.2. FAQ: How should the ability to simulate an impact be evaluated? Ideally, models should represent all relevant processes well. If certain processes are poorly captured, the model's ability to simulate the climate sensitivities of dominant processes could be in question. Yet models will always be limited by being simplifications of the real world (Clark et al., 2008;Carslaw et al., 2018). Therefore, for practical purposes, models are most often evaluated on how well they do at simulating key, measurable processes, especially those relevant to the impact of interest. For example, if the decisions relate to flooding, then hydrology model performance on short timescales matters. If, however, the decisions relate to water needs for drought, performance on shorter timescales may be less relevant. Evaluations should include how well model outputs are simulated historically (what is the current ability to simulate the variable of interest) and how sensitive they are to an altered climate. The latter can be done through evaluating whether modeled values respond accurately to a range of different climate conditions or through simple perturbations of the most relevant climate variables (e.g., Vano et al., 2012). This does not provide a comprehensive evaluation of how well future changes can be simulated, as this may not be knowable, but it can provide confidence that model sensitivities are physically reasonable and that further exploration using a model or approach is warranted. Additionally, techniques exist that can be used to evaluate how well a model performs under climatic conditions significantly different from those it was developed to simulate (Refsgaard et al., 2013).

FAQ: How should model sensitivities inform study design?
Evaluations of how sensitive models are to change can reveal the extent to which climate impacts can be adequately simulated for decision making purposes. In many cases, this provides helpful context for a more detailed analysis.
In some cases, however, this can change the trajectory of the study. There are several possible outcomes: (1) Evaluations reveal variables of interest are insensitive to climate.
This could be because they really are, e.g., rain-dominant basins do not experience a seasonal shift in their hydrograph because there is no snow to melt . Or, it might be an artifact of the hydrologic model design (e.g., temperature sensitive parameters, such as evaporative demand, have been fixed) which does not allow the model to account for climate change (Willows and Connell, 2003). In these situations, it makes little sense to do full climate simulations unless more climate-sensitive impacts are also of interest (e.g., streamflow temperature in rain-dominant basins) or the hydrologic model is reconfigured to be sensitive to changes in climate (e.g., evaporative demand, the key pathway by which temperature might influence water balance is no longer fixed).
(2) Evaluations reveal other non-climate uncertainties like population changes on exposure to extreme heat (Jones et al., 2015) or land use and resource availability (Olsen et al., 2015) have an equal or greater impact than climate. It is then up to the decision maker to decide which future impacts should be explored first. (3) Evaluations reveal large variability in results and the climate change signal is less noticeable in the midst of the noise or has not yet emerged from the range in natural variation. This could be either from natural variability of climate systems (e.g., Hamlet, 2011) or model uncertainties (Reclamation, 2014a). Notably, however, relative uncertainties change depending on the region (local, regional, global), time horizon, and amount of models included to represent the variable of interest. For example, uncertainties from emission levels dominate other types of uncertainties as planning horizons increase Sutton 2009, 2011) and other sources of uncertainties are introduced as more models are added (Eisner et al., 2017). Overall, even when relative uncertainties are large, there is still a need for understanding the range of plausible futures to use in stress tests, and practices designed for future flexibility and appropriate safety factors or freeboards should be used (Olsen et al., 2015).

Path forward
Evaluate the tools/models used in climate impact evaluations according to whether they adequately capture decision-relevant variables and respond to altered climates. When they do not, see Section 4.6. 4.6. DON'T expect every climate change question will be answerable with currently available models and datasets 4.6.1. Why Not every climate change-related question can be answered in a way that leads to a clear choice (Averyt et al., 2013). For the information producers, being clear about what is possible and what is not is an ethical responsibility (CSPWG, 2015) and can help identify knowledge gaps and direct research in application-relevant ways (NRC, 2012a). For information users, just knowing when approaches reach their limits is useful information as it allows managers and planners to develop appropriate practices for making decisions in a changing world (NRC, 2009). Most water management and planning design decisions depend on the tails of the distribution (floods and droughts) which are inherently difficult and require planning for the unknowns by designing system redundancy and adding safety factors (Stakhiv, 2011).

FAQ: Can one determine whether questions are answerable before investing time and resources?
In many cases, it is hard to know before doing some analysis. There are, however, several situations where it is best to proceed with caution.
(1) Sparse observations: many locations have a limited historical baseline from which to build an understanding of how the system will be influenced by an altered climate. While global climate model output is by definition global, many applications require finer spatial resolution. If, however, there is nothing to ground truth models to, downscaling approaches and hydrologic modeling can be misleading and give a false sense of precision. In these cases, other approaches may be more meaningful (see Section 4.9) (2) Spatial scales are too small: in some locations topography, coastal winds, fog, cool-air pooling or other local effects significantly affect local hydrology and are not sufficiently captured in existing datasets (Reclamation, 2016;Curtis et al., 2014). If this is the case, it is important to determine whether this affects the impact(s) of interest. Additionally, when information is provided for specific locations it should be considered within its larger context to check whether the precision and spatial variability are appropriate. (3) Temporal scales are too small: extreme events (on daily or subdaily timesteps) are often difficult to observe and capture because of limitations of observation networks and biases in models and downscaling methods. In these cases, it might be better to do climate-informed perturbations of the system using stochastic hydrology (described in Section 4.9).

Path forward
Recognize there are limits to what climate change scenarios can provide. Identifying these limits requires clearly communicating decisions that will be made on the basis of the analyses and asking information producers about the ability of the models, data, or methods to be used in such analyses. Being honest about these limits provides opportunities to learn. In these situations, other approaches such as climate-informed stochastic hydrology may be helpful.

Why
Climate change evaluations for water resource planning and management usually require data processing and linking one model's output to the next model's input (Reclamation, 2016;Brekke et al., 2009). Therefore, setting up and running the model chain in its entirety with a single simulation is arguably a large portion of the work, especially if processes can be automated (see Section 4.8). Each step can have unforeseen challenges, and important lessons that might reshape how project goals are achieved. For example, having preliminary numbers to work with can help refine evaluation criteria (Table 2). Early feedback on how the information is shared can prevent time being wasted creating information that is not useful. Additionally, in many cases, the newest climate change projections provide similar trends to earlier versions and can be useful for preliminary evaluations.
Earlier consideration of climate impacts has the potential to save resources, as it is easier to consider climate impacts during the design phase of a project than to restructure mature facilities (PIEVC, 2008).

Washington Climate Change Impacts Assessment (WACCIA)
The first time going through all the steps in the modeling chain can reveal the need for changes in earlier steps, as experienced in the WACCIA. The goal of this assessment was to update climate change projections and use them to assess climate impacts on nine key sectors in the state of Washington, including hydrology and water resources Vano et al., 2010a,b). The assessment used a chain of models approach that used output from 20 global climate models, two emission levels, for three future periods, downscaled, run through a hydrologic model, then run through a reservoir operations model to assess impacts. In the first year of the project, when simulations were run through reservoir operation models, unforeseen errors arose. First, the team realized their configuration of the disaggregation of monthly to daily data contained unrealistic daily precipitation estimates that were artifacts of the subsampling -in short, a few isolated storms in dry months were sampled too frequently in wet months. This discovery prompted a reconsideration of the downscaling technique and a delta method was used instead. These challenges are described in Hamlet et al. (2011) along with a new downscaling technique designed to overcome these challenges in future work. Second, new streamflow conditions required several alterations to the reservoir models that would enable them to continue to run during unprecedented extremes (e.g., extending the interpolation of anticipated flow values in September). Modifications were possible (e.g., described in Vano et al. (2010b)), but required considerations best not left until the end of the project.
This example illustrates the value of making it all the way through the modeling chain prior to completing all simulations at a single step. Additionally, more extreme projections can cause impact models to fail because they were designed only to evaluate more moderate conditions. Traversing the entire modeling chain as soon as possible in the project can be useful in uncovering necessary model modifications early on (Vano et al., 2010b).

FAQ: What are common challenges the first time a study is conducted?
• learning where and how to download the data • using unfamiliar data formats (e.g., NetCDF) • slicing data for a particular region or time period • converting from one data format to another • automating the process • running a model with new, more extreme inputs, can create un- Climate Model Intercomparison Projects (CMIP) 3 and 5 contain archives of future climate projections that differ in: number of models, the model versions, and their emission levels. Regional comparisons show some differences (e.g., Knutti et al., 2003;Sun et al., 2015;Rupp et al., 2016). However, both datasets capture the global-scale features (temperature and precipitation changes) of climate change similarly (Rogelj et al., 2012;Knutti et al., 2003;Sun et al., 2015). As such, at a IPCC Expert Meeting on Assessing and Combining Multi Model Climate Projections CMIP5 is seen as an addition to rather than a replacement of CMIP3 (Knutti et al., 2010a;https://gdo-dcp.ucllnl.org).

Path forward
Begin evaluating your impact with the information that is currently available, with the mindset that more information will be coming. Set up interim products that can test the process and, if appropriate, give information users an opportunity to provide early feedback.

Why
Datasets will continue to be updated as observations increase, models improve, errors are corrected, computational capacity increases, and hydrology and climate science advance. As such, iteration and updating should be included in a risk management framework, where new strategies are allowed to develop as new knowledge becomes available. This includes revisiting decisions when new information (e.g., new climate projections) are available (Willows and Connell, 2003).

FAQ: When do iterations stop?
Adaptive management means that decision-making is a continuous process, and needs to continually keep information used to support decisions up to date (a circular process). However, decisions can be made along the way. As an example of what this looks like, see the eight stages of the decision-making framework in Willows and Connell (2003): 1) Identify problem and objectives, 2) Establish decisionmaking criteria, 3) Assess risk,4) Identify options, 5) Appraise options, 6) Make decision, 7) Implement decision, and 8) Monitor, evaluate and review. These steps continue to provide guidance to those setting up studies, as developed by the UKCIP Adaption Wizard (UKCIP, 2013). Importantly, decision-making occurs in the midst of iterations with the best available information at the time and with knowledge that it will be readdressed as more information becomes available (NAVFAC, 2017). This needs to take into account that decisions occur at specific moments in time and therefore questions need to be constructed to be tractable within a timeline and the whole process designed to incremented and, when possible, flexible, modular, and updatable.

Path forward
Plan for iterations (e.g., automate the process, track dataset updates and model versions, document all steps in the process). This will also help recovery when unforeseen errors are uncovered. When possible sign up for notices that provide information on new releases or error updates.
4.9. DO be aware of multiple ways to evaluate future change -climate change scenarios are helpful, but there are other tools too 4.9.1. Why Climate change scenarios can provide valuable information to help better understand how the past will differ from the future. They reveal a non-stationary climate and are often the best tool available (NRC, 2012a). They do, however, have limitations. For example, the spatial or temporal scales of the data might be too coarse for certain decisions (Section 4.6), or other changes (e.g., changing demographics, socioeconomics, land use, and infrastructure demands (Brekke et al., 2009)) eclipse climate pressures (Section 4.5). Additionally, even when climate change scenarios are being used, other investigations may add important insights (Vano et al., 2014;Lehner et al., 2017a).

FAQ: What are different approaches to evaluating future change?
For perspective, below is a brief overview of four approach categories. This is not an inclusive list, as more exist and more will likely be developed.
(1) Climate change scenario studies: These approaches are often characterized as a chain-of-models approach where global climate model projections are downscaled and the downscaled climate change information (e.g., 30 years of daily precipitation, temperature) is then used as input to hydrology models, which generate streamflow and snowpack information, which can be used as input to reservoir operations models. This type of study is often the focus of existing guidelines because it most explicitly uses global climate model information and often requires decisions on model selection to translate global information to a local scale.
(2) Paleoclimate studies: Paleoclimate or paleoflood information is generated using information collected from the environment which can be proxies for past climate and flood events that date back further than the instrumental record (e.g., the width of tree rings can be correlated with streamflow) . These analogs from the past can date back thousands of years, and provide improved perspectives on natural variability, such as the length of dry periods (Woodhouse and Lukas, 2006), the characteristics of past floods (Raff, 2013) or how sensitive river basins are to temperature increases (Lehner et al., 2017a). Studies have also used a combination of scenario-based and paleoclimate studies to evaluate future change (Reclamation, 2011a;McCabe and Wolock, 2007). (3) Stochastic hydrology studies: Stochastic precipitation and hydrology timeseries can be used to stress test a system (Rodriguez-Iturbe et al., 1987;Salas, 1993;Wilks and Wilby, 1999;Yates et al., 2003;Erkyihun et al., 2016). The perturbations can be informed by historical information (e.g., paleoclimate information) or by global climate model trends. These techniques aim to avoid some of the uncertainties associated with using global climate models directly, yet address risk-based issues analytically (Olsen et al., 2015). In many cases, stationarity is assumed, although there are techniques that have included non-stationary stochastic methods (Kilsby et al., 2007;Erkyihun et al., 2016). It is, however, important to recognize that these timeseries are based on statistical models that do not capture process-based understandings, which limits how these can be used to interpret future change. (4) Climate-informed water system vulnerability analysis: These approaches are commonly referred to as decision support modeling and include techniques such as decision scaling , scenario-neutral approaches (Prudhomme et al., 2010), and robust decision making (Lempert et al., 2003). Typically, the focus is first on defining the decision context and exploring sensitivities by perturbing the climate incrementally to identify system vulnerabilities to changes in temperature, precipitation, or other climate variables before considering whether and how to apply climate change information Brown and Wilby, 2012;Weaver et al., 2013). EPA and CWDR (2011) describe strengths and limitations of using different decision support tools.

FAQ: How do approaches differ?
Approaches vary in complexity (Ludwig and van Slobbe, 2014)i.e., in which processes are represented and at what spatial and temporal scales. Simpler approaches (e.g., simple perturbations, simple water balance models) can be easier to understand, but may not include processes that provide more realistic representations of climate change (NRC, 2012a).
Approaches also differ in the order in which they evaluate aspects of the system. They are often referred to as top down or bottom up, reflecting either those that start with the climate change information first or those that start with the decision context first, respectively (Brown et al., 2011). In reality, this dichotomy is blurry. Climate change scenario studies should consider the decision context in model selection, and Climate-informed water system vulnerability analysis should consider realistic climate change perturbations when evaluating system performance in a changed climate.

FAQ: How are approaches similar?
More similarities exist than what the top down v. bottom up dichotomy suggests. Most approaches use global climate model information as part of their analysis process. All approaches have goals to better understand how the past will differ from the future and usually aim to find low-regret, robust alternatives that do well across a range of possible futures (Clark et al., 2016;Olsen et al., 2015). All approaches recognize the importance of climate variability and the importance of other changes (e.g., land cover, population changes). Often too, different approaches can complement each other (e.g., stochastic hydrology studies use perturbations based on paleoclimate information (Brekke et al., 2009). Additionally, all approaches deal with uncertainty whether it is in tree-ring reconstructions , how climate variables are correlated (Yates et al., 2003), or how well hydrology is being simulated (Mendoza et al., 2015). And, importantly, each approach has benefits and challenges that require professional judgment to navigate.

Path forward
Recognize there are multiple ways to consider future change. Approaches include climate change scenarios, paleoclimate, stochastic hydrology, and climate-informed water system vulnerability studies (described above). Together these approaches can complement each other and broaden our understandings and explore water system vulnerabilities from multiple angles.

Directions forward
This paper provides nine DOs and DON'Ts for how to design a study that effectively uses climate change information, which are grounded in past research studies and leverage existing guidance documents. We intend for this v1.0 edition to serve as a foundation for further refinement (e.g., not all frequently asked questions are asked or answered here, and even the ones answered here can be expanded on and changed). Currently examples are focused on the U.S. as this is where the project originated, but our hope is to expand and enhance learning from international experiences as well. Our goal is to facilitate dialogue between information producers and users and to codify the best of that information in easily accessible formats and examples. We also anticipate that by articulating dos, and especially don'ts, we have stated rules that may raise questions and need further refinement. This is an important conversation which we hope this structure promotes. With rules, there are exceptions, some of which are already discussed, but others that may warrant more discussion. However, for the majority of cases, the rules will be appropriate and important for helping water managers and planners effectively navigate the growing archives of climate change information and select and apply the most appropriate and useful concepts, data, and tools to their study questions.
To promote this conversation among and between information producers and users, we envision this forming a living document that can be amended with frequently asked questions, real-world examples, and additional DOs and DON'Ts (see https://ncar.github.io/dos_and_ donts). It could also be modified as discussions and climate science evolve. Community models in the sciences have benefited from opensource platforms where individuals and groups can participate in the development of specific elements that are reviewed and updated according to set protocol (e.g., github, described by Ram, 2013;Mergel, 2015); similarly, public repositories of information (e.g., Wikipedia) are made most useful when those using the information have opportunities to contribute. We intend this to be a foundation so the DOs and DON'Ts can progress in a similar fashion. More specifically, the DOs and DON'Ts will begin life on a public github site. The site will have a web interface that will allow for easy navigation and opportunities to submit additional materials, questions, and revisions that will be subject to review protocols and be version controlled. Lins and Cohn (2011) states "successful water resource management is an adaptive and multidisciplinary activity based on data, physics, statistics, economics, politics, nonquantifiable factors, and, above all, humility." This endeavor aims to create guidance that mirrors this statement.

Acknowledgements
This project is supported by the U.S. Army Corps of Engineers Climate Preparedness and Resilience program and benefits from the ongoing work of scientists and engineers in the Hydrometeorological Applications Program at the National Center for Atmospheric Research (NCAR), U.S. Army Corps of Engineers, Bureau of Reclamation, and University of Washington. NCAR is sponsored by the National Science Foundation. We are grateful for valuable feedback we received from both the research and applications communities while developing the concept, especially from the U.S. Army Corps of Engineers, Bureau of Reclamation, members of the Water Utility Climate Alliance, and two anonymous reviewers.

Declarations of interest
None.

Appendix A. Glossary
Adaptive capacity (IPCC, 2012): The combination of the strengths, attributes, and resources available to an individual, community, society, or organization that can be used to prepare for and undertake actions to reduce adverse impacts, moderate harm, or exploit beneficial opportunities.
Projection (IPCC, 2012): A projection is a potential future evolution of a quantity or set of quantities, often computed with the aid of a model. Projections are distinguished from predictions in order to emphasize that projections involve assumptions concerning, for example, future socioeconomic and technological developments that may or may not be realized, and are therefore subject to substantial uncertainty.
Resilience (IPCC, 2012): The ability of a system and its component parts to anticipate, absorb, accommodate, or recover from the effects of a hazardous event in a timely and efficient manner, including through ensuring the preservation, restoration, or improvement of its essential basic structures and functions.
Risk: In this report, unless otherwise defined, risk means the chance of loss. As described in Eco-Adapt's glossary (https://climate-adapt.eea. europa.eu/help/glossary): The word "risk" has two distinctive connotations: in popular usage the emphasis is usually placed on the concept of chance or possibility, such as in "the risk of an accident"; whereas in technical settings the emphasis is usually placed on the consequences, in terms of "potential losses" for some particular cause, place and period. It can be noted that people do not necessarily share the same perceptions of the significance and underlying causes of different risks.
Scenario (IPCC, 2012): A plausible and often simplified description of how the future may develop based on a coherent and internally consistent set of assumptions about driving forces and key relationships. Scenarios may be derived from projections, but are often based on additional information from other sources, sometimes combined with a narrative storyline.
Vulnerability (IPCC, 2012): The propensity or predisposition to be adversely affected.