Supporting local adaptation through the co-production of climate information: An evaluation of collaborative research processes and outcomes

Water for the Seasons was a five-year collaborative research project which aimed to assess the climate resiliency of the Truckee-Carson River System, a snow-fed river system in the western United States. The collaborative research design featured iterative interactions involving an interdisciplinary research team and local stakeholders that produced climate science information identified by stakeholders as necessary to support adaptation to climate-induced water supply variability. This information included plausible climate scenarios to test the resiliency of the river system and models simulating hydrologic results to examine changes in water availability. In this paper, we present formative and summative evaluation data collected over the course of the project to determine the extent to which this collaborative research project met stakeholders’ climate science information needs. Results indicate that over a five-year period, the project: 1) co-produced new climate science information to support local climate adaptation; 2) consistently engaged stakeholders in research that facilitated social learning to identify innovative strategies to adapt water management; 3) provided iterative interactions between stakeholders and researchers to ensure resulting information services were useful to stakeholders; 4) combined diverse, practical stakeholder knowledge with rigorous scientific research to co-produce legitimate climate science information; and 5) effectively utilized Extension as a boundary organization to design, implement, and evaluate collaborative research processes and outcomes. Further empirical work is necessary to continue testing and standardizing metrics that illuminate collaborative research successes and failures, and to identify best practices that guide future collaborations to co-produce climate science information. Practical implications Water for the Seasons was a five-year collaborative research project funded by the National Science Foundation’s Water Sustainability and Climate program that aimed to assess the resiliency of snowfed river systems to climate-induced water supply variability. The project utilized the Truckee-Carson River System in eastern California and northwestern Nevada as a case study to examine the climate resilience of snow-fed river systems comprised of diverse and competing water management interests where prior appropriation doctrine regulates water use under assumptions of climate stationarity. The project featured a collaborative modeling research design that served as the vehicle for systematic and iterative interactions among an interdisciplinary research team (herein, research team) and key local stakeholders (herein, stakeholders) who represented urban, agricultural, environmental and regulatory interests from river system headwaters to terminus. The research team comprised extension research faculty with expertise in hydrologic sciences, resource economics, and collaborative research methods, and modelers with expertise in surface and groundwater hydrology, climatology, and econometrics. More than 124 structured researcher-stakeholder interactions occurred throughout the life of the 5-year project that co-produced climate science information with the goal of facilitating and supporting local climate adaptation. That is, the research team and * Corresponding author at: University of Nevada, Reno, Department of Economics, 1 664 North Virginia Street MS 0204, Reno, NV 89557, USA. E-mail address: singletaryl@unr.edu (L. Singletary). https://doi.org/10.1016/j.cliser.2020.100201 Received 16 May 2019; Received in revised form 24 September 2020; Accepted 6 November 2020 Available online 19 November 202

Water for the Seasons was a five-year collaborative research project which aimed to assess the climate resiliency of the Truckee-Carson River System, a snow-fed river system in the western United States. The collaborative research design featured iterative interactions involving an interdisciplinary research team and local stakeholders that produced climate science information identified by stakeholders as necessary to support adaptation to climate-induced water supply variability. This information included plausible climate scenarios to test the resiliency of the river system and models simulating hydrologic results to examine changes in water availability. In this paper, we present formative and summative evaluation data collected over the course of the project to determine the extent to which this collaborative research project met stakeholders' climate science information needs. Results indicate that over a five-year period, the project: 1) co-produced new climate science information to support local climate adaptation; 2) consistently engaged stakeholders in research that facilitated social learning to identify innovative strategies to adapt water management; 3) provided iterative interactions between stakeholders and researchers to ensure resulting information services were useful to stakeholders; 4) combined diverse, practical stakeholder knowledge with rigorous scientific research to co-produce legitimate climate science information; and 5) effectively utilized Extension as a boundary organization to design, implement, and evaluate collaborative research processes and outcomes. Further empirical work is necessary to continue testing and standardizing metrics that illuminate collaborative research successes and failures, and to identify best practices that guide future collaborations to co-produce climate science information.

Practical implications
Water for the Seasons was a five-year collaborative research project funded by the National Science Foundation's Water Sustainability and Climate program that aimed to assess the resiliency of snowfed river systems to climate-induced water supply variability. The project utilized the Truckee-Carson River System in eastern California and northwestern Nevada as a case study to examine the climate resilience of snow-fed river systems comprised of diverse and competing water management interests where prior appropriation doctrine regulates water use under assumptions of climate stationarity. The project featured a collaborative modeling research design that served as the vehicle for systematic and iterative interactions among an interdisciplinary research team (herein, research team) and key local stakeholders (herein, stakeholders) who represented urban, agricultural, environmental and regulatory interests from river system headwaters to terminus. The research team comprised extension research faculty with expertise in hydrologic sciences, resource economics, and collaborative research methods, and modelers with expertise in surface and groundwater hydrology, climatology, and econometrics. More than 124 structured researcher-stakeholder interactions occurred throughout the life of the 5-year project that co-produced climate science information with the goal of facilitating and supporting local climate adaptation. That is, the research team and stakeholders, selected through a stakeholder analysis, interacted through project introductory sessions, face-to-face and phone interviews, focus groups, and workshops to: 1) characterize baseline water management challenges; 2) identify local adaptation strategies, implementation barriers and climate science information needs; 3) specify climate scenarios that test river system resiliency; 4) validate hydrologic and operations model simulations tailored to the river system; and 5) validate the results of an empirical analysis of the economic performance of prior appropriation doctrine in the case study area. Co-produced climate science information was then disseminated through a variety of products and services including: 1) the analysis of qualitative and quantitative data collected from five years of stakeholderresearcher interactions that identified viable adaptation strategies, implementation barriers and related climate information needs; 2) climate "stress-test" scenarios that were informed through stakeholder narratives and guided by climate scientists' expertise; 3) hydrologic and river operations model simulation results that assessed the effects of these climate scenarios on water availability to meet diverse and competing water demands across the river system; and 4) econometric models that assessed the performance over time of prior appropriation doctrine in reallocating water from lower-valued to higher-valued uses.
Here, we present formative and summative evaluation data collected to measure the extent to which project processes and outcomes met stakeholders' climate science information needs. Formative evaluation, conducted over the course of the project to guide improvements, assessed stakeholders' perceptions regarding the effectiveness of the collaborative research processes.
In year four, a summative evaluation gauged the project's overall performance in achieving its intended outcomes. Reflecting upon these combined evaluation data provided additional insights into project features that worked well and those which might have stood improvement. Evaluation results are depicted through five key outcomes. First, the project facilitated and supported local climate adaptation through the development of plausible climate scenarios to explore hydrologic and operational implications. Stakeholders cited the relevancy of research activities which included reassessing current and future water management decisions, watershed planning, agricultural water use and irrigation management, and infrastructure improvements to overcome identified water supply challenges. Second, the project identified and consistently engaged stakeholders in collaborative research and facilitated social learning. Stakeholders applauded the research team for incorporating diverse local perspectives surrounding water uses, adapting the process to meet emergent climate information needs, and inviting stakeholders' staff and constituents to participate in the research, resulting in climate information useful in supporting local adaptation. For example, in examining managed aquifer recharge as a viable strategy to adapt to earlier snowmelt and enhance groundwater sustainability, stakeholders provided key insights concerning barriers to implement this strategy. These included the risk of oversaturating fields that might prevent infiltration and existing institutional constraints that prevent earlier diversions for irrigation purposes. Modelers, in turn, used local stakeholders' knowledge to develop decisionmaking criteria, arriving at model simulations representative of the river system. Third, the project provided iterative and structured interaction between stakeholders and the interdisciplinary research team to ensure results are useful to local decision-making. Biannual workshops provided a consistent forum for these interactions with notable attributes including pre-workshop lunches, structured facilitation, relevant agenda topics, appropriate science expertise, and "take-home" information binders to disseminate materials to stakeholders' constituents. Fourth, the project combined diverse, practical stakeholder knowledge with rigorous scientific research to coproduce legitimate climate science information. Stakeholders acknowledged the research team's careful selection of the Stakeholder Affiliate Group, who interacted frequently with researchers, to ensure diverse water use interests and perspectives were embodied in the research. Stakeholders also reflected on the transparency and fairness of facilitated interactions, whereby every local representative had an equal voice in workshop discussions. Fifth, the project effectively utilized Extension as a boundary organization to conceptualize, implement, and formatively evaluate the efficacy of the collaborative research design. Having extension researchers with disciplinary expertise embedded within the research team was instrumental to guide and support the co-production of legitimate climate science information while navigating the social complexities that surround water management issues. Research team members interviewed as part of the summative evaluation highlighted the need for continued use of extension research faculty to facilitate effective collaborative research outcomes especially when simulations lead to contention among stakeholders who perceive implementation as infeasible.
Review of the evaluation data collected affirms that the project achieved its desired outcomes concerning river system-wide effects of stakeholder-informed climate scenarios, simulated adaptation strategies identified as viable by stakeholders, and ways in which improved communication and coordination among stakeholders might enhance system-wide climate resiliency. A retrospective assessment following the sixth and final workshop illustrates ways in which the collaborative research undertaking in this project could have improved so as to increase water managers' capacity to adapt to climate-induced water supply variability. With regards to procedure, stakeholders requested that in future projects researchers: 1) include multiple stakeholders for each water use sector; 2) conduct public workshops in locations across the river system; and 3) increase opportunities for engagement between researchers and stakeholders. With regards to research substance, stakeholders requested that in future projects researchers: 1) incorporate scenarios of population growth and development; 2) expand the research to include implications for water quality; and 3) provide technical support to help stakeholders integrate research results into their adaptation planning. Further empirical case study work is necessary to test and standardize metrics to assess collaborative research successes and failures, and to verify emergent practices that guide the coproduction of new climate information for the purpose of supporting local adaptation in snow-fed river basins.

Introduction
Water for the Seasons was a five-year (July 1, 2014 to June 30, 2019) collaborative research project funded by the National Science Foundation (NSF) and U.S. Department of Agriculture (USDA) National Institute for Food and Agriculture (NIFA) Water Sustainability and Climate program. Across the western United States, snow-fed arid land river systems face increasing challenges as climate change alters upstream snowpack accumulation and snowmelt that influences downstream water availability to support diverse and competing water management interests (Dettinger et al., 2015;Li et al., 2017;USGCRP, 2017). Using the Truckee-Carson River System in California and Nevada as a case study, this project developed and implemented a collaborative modeling research design to assess the climate resiliency under climate-induced water supply variability Sterle, 2017, 2018). Resiliency in this context is defined as the capacity of natural and human systems to absorb and adapt to climatic disturbances while retaining their essential purpose and function by allowing key components to cooperatively reorganize as necessary (Aldrich, 2012;Folke et al., 2010;Matarrita-Cascante et al., 2017;McGinnis and Ostrom, 2014;Walker and Salt, 2012;Wilson, 2012).
The Truckee-Carson River System comprises the Truckee (195 km) and Carson (211 km) river basins which originate as snowpack in the Sierra Nevada of California and flow northeastward into the arid Great Basin in northwestern Nevada. The river system typifies challenges observed across the western United States, where water supply is highly dependent on accumulated snow that melts through spring to meet late summer demand . The 18,197 square-kilometer area encompasses multiple and historically contentious water management challenges common to snow-fed river dependent communities in the arid western United States including allocating limited water resources to: 1) support municipal and industrial development, population growth, and recreation; 2) provide for irrigated agriculture through an interbasin transfer; 3) rehabilitate and protect ecological systems; and 4) honor federal treaties with Native American tribes to allocate water to reservation lands (Gautam et al., 2014;Horton, 1997;1996;;Wilds, 2014).
The collaborative modeling research design convened researchers with stakeholders who were selected to represent the diverse agricultural, environmental, and municipal and industrial water management interests across the river system, from headwaters to terminus, and included the system's pertinent water regulatory authority. Climate science information generated was disseminated through a variety of products including: 1) an analysis of qualitative and quantitative data reflecting stakeholders' perceptions of river system functionality under climate change (e.g., Singletary and Sterle, 2017;Sterle et al., 2019;; 2) climate "stress-test" scenarios informed from stakeholder narratives and guided by climate scientists' expertise (e.g., Albano, 2019;Dettinger et al., 2017); 3) hydrologic and river operations model simulation results quantifying the extent to which climate scenarios alter water availability and challenge water managers' ability to meet diverse and competing water demands (e.g., Morway et al., 2016;Niswonger et al., 2017;Sterle et al., 2019); and 4) econometric models that assessed the economic performance over time of prior appropriation doctrine to reallocate water from lower to highervalue uses (e.g., Lee et al., 2020). These products were intended to provide climate information services to support stakeholders' climate adaptation planning.
In this paper, we present new insights gleaned from formative and summative evaluation of the Water for the Seasons project to assess the extent to which collaborative research generated climate science information. First, we synthesize scholarly literature that demonstrates the prominent role of collaborative research in climate adaptation research. Next, we briefly describe the collaborative research design developed and implemented for this project. Using formative and summative evaluative data collected over the duration of the project, we then report stakeholders' and researchers' perceptions of project processes and outcomes. Lastly, we discuss the need for additional empirical research to assess and document the successes and failures of collaborative research processes and outcomes more generally, in order to verify emergent best practices to guide collaborative research that is intended to support climate adaptation.

Collaborative research, co-production of climate science information, and local climate resiliency
Scholarly work increasingly demonstrates that collaborative research can co-produce climate science information useful to address complex natural resource issues and enhance local decision-making (Djenontin and Meadow, 2018). In the context of climate research, this includes collaborative and participatory modeling, group modelbuilding, multi-criteria/multi-decision-making, and cyber-structure decision support tool development, to name a few (Beall King and Thornton, 2016;Biehl et al., 2017;Bourget et al., 2013;Campos et al., 2016;Langsdale et al., 2013;Prokopy et al., 2017;Voinov et al., 2016). Collaborative research is characterized by thoughtful group procedure, including facilitated discussion and deliberation (Djenontin and Meadow, 2018;Meadow et al., 2015), supported by evolving analytical and modeling tools (Biehl et al., 2017;Voinov et al., 2018), and designed to facilitate information exchange between scientists and stakeholders. These characteristics can aid in the co-production of climate science information that effectively balances and legitimizes the interests of all participants involved (Lemos et al., 2018).
Studies that explicitly examine the efficacy and outcomes indicate that collaborative research facilitates civil, constructive dialogue, which supports social learning while managing tensions to create space to identify common ground. This approach is particularly useful in highly politicized contexts that are common with multi-party resource management disputes (Lemos et al., 2012) and can result in useful sciencebased information (Dilling and Lemos, 2011;Lemos, 2015;Prokopy et al., 2017). Collaborative research requires: 1) researchers' familiarity with past and existing water conflict within the study area that can provide social context for information co-production; 2) a thorough analysis of the study area's historical and current water management challenges to objectively define and identify stakeholders with whom to partner; and 3) an effective structure for consistent and iterative communication and information exchange between researchers and stakeholders (Devente et al., 2016). Developed specifically for research focused on climate adaptation, collaborative research that features structured interaction is intended to provide stakeholders, or end-users of climate science information, with a role in the co-production of these information products tailored to support local decision-making (Lemos et al., 2018). A common attribute of successful initiation is researchers' purposeful engagement with stakeholders early and throughout the life of the project (Djenontin and Meadow, 2018) to: 1) co-define the research problem and research question(s); 2) jointly review modeling results and validate the on-the-ground potential of model simulations; and 3) through collaborative stakeholder engagement, disseminate the resulting climate information services to water management organizations (Lemos et al., 2012;Lemos and Morehouse, 2005). Hence, researchers must objectively identify stakeholders Reed et al., 2009) who assume these critical project roles. Effective collaborative research also requires that researchers and stakeholders share similar project goals, as well as clearly defined respective roles and responsibilities (Bremer and Meisch, 2017;Hegger et al., 2012;Hegger and Dieperink, 2014).
In the context of climate adaptation (Moser and Ekstrom, 2010;Moser and Boykoff, 2013), researchers leading applied climate science research and stakeholders tasked with decision-making have high stakes in project outcomes (Djenontin and Meadow, 2018;Fazey et al., 2018;Hegger and Dieperink, 2014). For example, opportunity costs associated with collaboration exceed those for traditional top-down research. These costs include necessary increases in funding and time, diverse technical expertise, and a conceptual framework that explains and justifies the additional costs, time, and labor involved in science convergence to address complex resource problems. Collaborative research also requires experts skilled in interpersonal communication and group procedure, and the sustained commitment of the active participation of researchers, stakeholders, and their respective institutions (Gober, 2018;Wall et al., 2017). Researchers and stakeholders alike may perceive these additional factors as burdensome and the continuous interaction and deliberation processes as intimidating. Stakeholders in particular may experience fatigue if they are asked repeatedly to volunteer substantial time from their daily management responsibilities to participate in collaborative research projects (Lemos et al., 2018).
Collaborative research inherently presents potential pitfalls and barriers, including stakeholders' conflicting interests and values and thus lack of consensus in research agenda setting, unrealistic expectations for research deliverables, lack of accessibility to research results in real time, and lack of confidence in the research results (Briley et al., 2015;Ferguson et al., 2017). The potential exists also for stakeholders and researchers to influence research towards special interests, which can ultimately exacerbate conflict and lead to tensions during interactive events that undermine the integrity of the research (Lemos et al., 2018).
Expectations for the research team to integrate their respective disciplinary knowledge can further challenge collaborative research, due to the lack of a common set of scientific terms, techniques, and mental models useful to arrive at and attack a shared research problem Table 1 Primary and secondary data collection methods and purpose, excluding collaborative modeling workshops (See Table 2).

Method
Purpose 2014: Archive secondary data.
2014: Initial conversations on-site at 15 various water management meetings (1 hr each; ~ 250 stakeholders total; 3 extension researchers per meeting).
Profile historical and contemporary water use in the region. Identify stakeholders to participate in baseline interviews. Introduce stakeholders to the Water for the Seasons project that will include stakeholder participation. Clarify the intent and purpose of the project, and limitations. Discuss and document stakeholders' questions, comments, concerns. Gather baseline understanding of river system function by identifying stakeholders': water supply challenges under normal, drought, and flood conditions; adaptation strategies; implementation barriers; climate information needs; and project expectations. Examine model simulations with stakeholders using hypothetical climate scenarios. Document water management challenges under climate scenarios while exploring viable adaptation strategies. Identify climate science information needs to be addressed through ongoing research. Assess ongoing water management challenges and shifts in adaptation strategies and implementation barriers. Identify additional information useful to overcoming identified adaptation barriers. Gather stakeholder input on the viability of locally identified strategies to enhance water supply. Discuss conditions mobilizing a change in water management practices. (Pennington, 2016;Pennington et al., 2016). Additionally, researchers may argue that highly localized collaborative research lacks sufficient rigor and assumptions necessary for generalizability and replication. However, such case study research is imperative to better understand coupled human-natural system dynamics and to test collaborative and interdisciplinary research methodologies necessary to assess and enhance climate resiliency (Mostert, 2018). These numerous challenges and potential pitfalls make boundary organizations very useful to collaborative research design and implementation appropriate for case-or place-based studies (Dilling and Lemos, 2011;Kirchhoff et al., 2015Kirchhoff et al., , 2013Prokopy et al., 2015). Such projects typically require considerable planning and careful execution to engage stakeholders consistently and effectively (Meadow et al., 2015;Reed et al., 2014). Boundary organizations with well-established stakeholder and science research networks, such as Extension, can broker information exchange, create new partnerships, create space for constructive dialogue that facilitates social learning, while also assessing stakeholders' climate information needs and subsequently gauge the extent to which collaborative research meets these needs Flagg and Kirchhoff, 2018;Kirchhoff et al., 2015Kirchhoff et al., , 2013Lemos, 2015;Prokopy et al., 2017).

Establishing collaborative research criteria
The collaborative research design methods developed for this project adopted a normative lens of information co-production (Bremer and Meisch, 2017) that relied upon iterative interaction between stakeholders and researchers Sterle, 2017, 2018). Researchers included faculty with Extension (i.e., the authors) with expertise in hydrologic science, resource economics, and interdisciplinary and collaborative research methods. Extension researchers worked closely with other team members with expertise in surface and groundwater hydrology, water resource engineering, climatology, and econometrics, to generate applications of their modeling methods and to gather information from stakeholders to further inform model simulations and empirical analyses (e.g., Sterle et al., 2019Sterle et al., , 2020aSterle et al., , 2020b.
Extension researchers established collaborative research criteria at the project's outset that included: 1) conduct a stakeholder analysis to identify a core group of individuals to represent the diverse and competing water management interests and organizations across the river system and whose knowledge complements researchers' technical expertise; 2) clarify the roles, responsibilities, and expectations of researchers and stakeholders; 3) provide well-designed, structured opportunities for iterative interaction to ensure stakeholder input is consistently incorporated throughout the project; and 4) co-produce climate science information services consistently responsive and accountable to support stakeholders' climate adaptation (for further description of the stakeholder analysis and research design, see Singletary and . Through ongoing interactions, researchers sought to examine interrelationships among climate-induced water supply variability, the performance of existing water management practices to absorb changing conditions, and overall river system functionality to meet diverse and competing water demands.

Identifying stakeholders and iterative interaction methods
In the context of this project, stakeholders were defined as local water managers who: 1) consume, deliver, protect, or supply water for diverse and competing agricultural, environmental, and municipal and industrial water uses; 2) regulate or have the potential to influence water management; and 3) possess technical expertise and knowledge concerning local water resource management . Extension researchers identified a suite of primary data collection methods to structure iterative interaction between stakeholders and researchers. These included face-to-face semi-structured interviews, focus groups, telephone interviews, and workshops. Table 1 describes each primary data collection method and purpose. Table 2 details content featured in six collaborative modeling workshops that convened a core group of stakeholders with researchers. The University of Nevada, Reno Office of Research Integrity reviewed and approved each primary data collection activity as described.
We began by reviewing archived documents outlining the water management issues and conflicts across the river system, including litigated outcomes and negotiated settlements which aided in identifying key water management organizations with considerable stakes in water management decisions. This review allowed us to create a list of relevant water management organizations from which to collect baseline data concerning perceptions of climate change impacts and their organization's respective information needs to cope with observed impacts. We then conducted 66 face-to-face semi-structured interviews with individuals representing the leadership of these organizations. The results of these interviews, combined with a meta-analysis of archival documentation, helped researchers to identify a subset of key stakeholders willing to participate in the collaborative research project, each representing their respective water management organizational interests. The resulting Stakeholder Affiliate Group included leadership Table 2 Summary of content, facilitated discussion topics and ongoing research featured during each of the six collaborative modeling workshops that provide a consistent forum for iterative interaction and information exchange, including research identified to meet climate adaptation information needs. 13-year drought scenario . • Introduce hydrologic and operations models .
• Discuss strategies to adapt to simulated conditions.
• Explore the performance of existing water management institutions to meet water demand. • Discuss recent hydroclimate variability, inclusive of prolonged drought and high flow, in the context of climate change (Harpold et al., 2017).
• Present model simulations of locally identified adaptation strategies. • Examine hydrologic changes according to salient river system metrics, such as Lake Tahoe.

#6 -Spring 2019
• Present econometric analysis demonstrating the effective of prior appropriation-based institutions in agricultural-rich regions of the system. • Assess hydrologic water management implications under high and low frequency scenarios according to local river system metrics.
• What climate scenarios are most concerning?
• Develop first stakeholder-informed climate scenario. What adaptation strategies are you currently • Integrate hydrologic and operations models (Morway et al., pursuing, and what barriers do you face in 2016). implementing those strategies?
• What information produced by these models • Conduct follow-up focus groups with stakeholders' organizations would be most useful for your adaptation to review modeling tools. planning?
• Simulate hydrologic change and operational implications under 13-year drought scenario.
• What water management institutions are tied • Specify model simulations to explore viable adaptation strategies, to historical climate records?
including reservoir reoperation (Sterle et al., 2019) and managed • What are viable water management strategies aquifer recharge (Niswonger et al., 2017). to adapt to earlier snowmelt?
• Continue to disseminate research through extension outreach • What other climate scenarios should we materials Singletary, 2017, 2018). explore?
• How useful is this information?
• Conduct follow-up phone interviews with key stakeholders to • What are we missing?
track shifts in adaptation strategies and implementation barriers. • How do we craft adaptation strategies to deal • Develop a climate scenario to explore high and low frequency with hydroclimate extremes? climate variability (Albano, 2019).
• What other ways might we enhance water • Conduct final round of interviews to gather input for final supply in the river system? synthesis report. • What additional modeling outputs would help • Simulate high and low frequency climate scenario through your organization overcome barriers?
hydrologic and operations models. • What information are we missing?
• What key-takeaways should we feature in our • Produce final synthesis report that features key takeaways from final synthesis report? collaborative research project. • How do we ensure research products continue • Disseminate guidance on accessing open source data archives and to provide information to support adaptation simulation modeling tools. planning? from 12 diverse organizations that represent the primary water management interests and whose responsibilities were spatially distributed across the river system from its headwaters to its natural terminus lake and manmade system terminus. These 12 stakeholders, and the organizations they represent, provided researchers with more than a century of practical management expertise and local knowledge pertaining to the diverse and competing water demands placed upon river system's water resources. These competing demands included water used for irrigated agriculture, municipal and industrial development, environmental instream flows inclusive of wildlife habitat and fisheries, and Nevada water law enforcement. Fig. 1 illustrates the river system boundaries and the approximate location of the 12 water management organizations.
Telephone interviews provided a means for extension researchers to interact annually with stakeholders. Focus groups conducted in the third year of the project provided a means for modelers, focused on particular geographic areas of the river system, to interact with stakeholders with management responsibilities focused in those areas. Collaborative modeling workshops were conducted roughly twice per year and provided a consistent forum for information exchange, convening the entire research team and Stakeholder Affiliate Group. Through these structured, iterative interactions, stakeholders provided modelers with feedback concerning hydrologic conditions and the water management challenges they faced. These narratives were used to specify climate "stress-test" scenarios to test the resiliency of the river system (Albano, 2019;Dettinger et al., 2017). Temperature and precipitation data outputs were used as inputs into a suite of theoretically-grounded hydrologic models, including the Coupled Groundwater and Surface-Water Flow Model (GSFLOW), the Precipitation Runoff Modeling System (PRMS) and river and reservoir operations models (RiverWare and MODSIM) tailored to the river system (Huntington et al., 2013;Labadie, 2006;Markstrom et al., 2008;Morway et al., 2016;Zagona et al., 2007).
Workshops were designed as opportunities for stakeholders to provide feedback concerning the project's progress and findings (see Table 2). Extension researchers established with the team and the Stakeholder Affiliate Group a set of principles and ground rules for interactive communication and group procedure to follow during workshops (Burgess and Burgess, 1997;Carpenter and Kennedy, 2001). These ground rules encouraged all workshop attendees to actively participate, seek to clarify the diverse interests surrounding water management issues rather than preconceived solutions, and avoid arguments, criticisms, and judgments of other stakeholders, researchers, and research results. Extension researchers structured and facilitated each 5-hour workshop to elicit discussion around primary data collection findings and hydrologic and operations model simulation results. Stakeholders were encouraged to validate these results and further clarify climate conditions that challenged the river system, and brainstorm adaptive water management strategies viable for the region (Sterle et al., 2019;.

Formative and summative evaluation data collection and analysis
Extension researchers conducted formative project evaluation over the duration of the project. In year one, the questionnaire administered during face-to-face semi-structured interviews with 66 stakeholders featured one open-ended question that asked stakeholders to describe their perceived project outcomes or climate science information needs. Responses informed an evaluation instrument administered in years two through five following each of the six collaborative modeling workshops to track progress as a function of these expected project outcomes. This instrument featured four Likert-type scale question items that asked stakeholders the extent to which, as a result of the collaborative modeling, they had acquired information useful to their respective organizations to: 1) improve climate adaptation planning; 2) improve daily operations; 3) identify viable adaptation strategies; and 4) improve communication and coordination among local water managers. Each question included a choice set of responses, 1 to 5, with 1 being strongly disagreed and 5 being strongly agreed.
Qualitative data resulting from open-ended questions were examined using content analysis, a method commonly used to objectively document patterns and trends to obtain a quantitative description (Rossman and Rallis, 2016), and then descriptively coded (Miles et al., 2014). Intercoder reliability assessment (Kurasaki, 2000) was undertaken to ensure minimization of coder bias or random error arising from judgments made about categories and themes emerging from the qualitative data sets. The coded data, as well as data resulting from Likert-type scale questions, were analyzed using Statistical Package for the Social Sciences (SPSS), Version 25 to calculate basic descriptive statistics.
In year four, the project funder contracted an external party to conduct a summative project evaluation (see Babin, 2018). 1 The evaluator conducted approximately one-hour telephone interviews with each of the Stakeholder Affiliate Group representatives and selected researchers, including the authors. Open-ended questions aimed to gage perceptions of the project's successes and failures relating to: 1) stakeholder involvement in the research and whether it tied to project outcomes; 2) quality of collaboration between researchers and stakeholders, and how collaboration was fostered; and 3) how the project may have contributed to capacity-building of partner (e.g., stakeholder) organizations. Researchers interviewed were asked similar questions, including to describe his/her role in the project, what about the project they perceived was successful and why, and the biggest project impact to date (Babin, 2018).
With permission from Babin (2018), extension researchers used content analysis to reanalyze Babin's qualitative data to further assess and interpret the project outcomes (Hinds et al., 1997). Secondary analysis of qualitative data is appropriate in this context, as extension 1 USDA National Institute of Food and Agriculture awarded contract for external project evaluation to the Natural Resources Social Science Lab, Department of Forestry and Natural Resources, Purdue University. *Question items responses using Likert-type response scale, with 1 = strongly disagree; 2 = disagree; 3 = neither agree nor disagree; 4 = agree; 5 = strongly agree.
researchers have in-depth knowledge of the project and thus can provide an alternative lens using the same set of data (Berg, 2005;Long-Sutehall et al., 2011). Secondary analysis of these data was advantageous also in aligning emergent themes with the results of formative evaluation data collected over the course of the project. The resulting themes are presented as five collaborative research project outcomes and quotes that illustrate the perspectives of stakeholders and researchers.

Formative evaluation
At the project outset, extension researchers asked 66 stakeholders to describe their expectations for project outcomes. Of those who responded to the question (n = 65), 90% requested as their two top outcomes information about how the river system would be impacted by hypothetical climate scenarios and adaptation strategies identified by stakeholders as plausible, respectively. Stakeholders also desired as a third outcome improved stakeholder communication and coordination, with the goal of enhancing system-wide resiliency. Nearly half (49%; n = 65) of these stakeholders requested the project develop an outreach program aimed at educating the general public about projected climate change impacts on the river system and viable adaptation strategies tailored to this system (for additional discussion of these survey results, see Sterle, 2017, 2018;Singletary et al., 2016). Fig. 2 presents formative evaluation results collected over the six collaborative modeling workshops, featuring four question items to track the project's progress in meeting the 12 organizational stakeholder representatives' expected project outcomes (for mean scores and standard deviations, see Appendix, Table A1). Positive gains can be observed over time across all question items, suggesting that at least for these expected outcomes, stakeholders' expectations generally were being met. This includes new information that could improve climate adaptation planning (mean Likert scale score (m) = 3.64 to 4.80), information to improve water management organizations' daily operations (m = 3.60 to 4.00), and information about viable adaptation strategies (m = 4.36 to 4.55). For question 4, "I believe communication and coordination are improving among local water managers," the mean scores were more variable over the six workshops but ended 0.27 points higher (m = 4.27) than the first workshop (m = 4.00). Since collaborative co-production of new information and knowledge relies heavily on social interaction, some variance is to be expected over time and may be due to multiple factors, including saturation effects from being asked to complete an evaluation following each workshop or co-production fatigue. These variable scores might also be attributed to the particular content featured in a given workshopwhereby the model simulations focused on a very particular stretch of the river system not of interest to every stakeholder or some stakeholders were not accepting of the model simulations as viable adaptation solutions.

Summative evaluation
Secondary analysis of the summative evaluation data collected by an external evaluator in year four of the project identified key themes relating to project processes and outcomes as perceived by Stakeholder Affiliate Group representatives and researchers. Evaluation results reveal that the project: 1) facilitated and supported local climate adaptation planning initiatives; 2) consistently engaged stakeholders in research that facilitated social learning; 3) provided consistent iterative and structured interaction involving stakeholders and researchers; 4) combined diverse practical stakeholder knowledge with legitimate science research; and 5) effectively utilized extension as a boundary organization to support collaborative research. The following subsections describe each of the five outcomes in more detail.

Outcome #1: Facilitated the identification and simulation of local climate scenarios and adaptation strategies
Stakeholders indicated that the collaborative research undertaken resulted in new information that could support local climate adaptation across the river system, including the following examples: • Plausible climate scenarios: "I am very happy with what we've received. The [climate scenarios] information, it was eye-opening for me. You get into these things you think you know, and then it's like, "Oh, okay. The science shows this… I'm able to bring that information to the watershed." • Future water management: "But honestly, things have to adapt, and we have to figure out a way to do it… that's probably one of the biggest benefits of the project, to help inform future policy changes in the water rights structure." • Several stakeholders expressed a desire to see the project continue to explore additional climate scenarios and simulate the effects of additional locally identified water management strategies. For example, as one stakeholder asked: "How are they taking that data now and moving forward with it? We got good data. We can now see what's going to happen with the climate. We can see how it impacts the flows of the river, now. So now what do we do?" One stakeholder even encouraged the research team to apply for more funding: "It's a study that's found itself to be true and good and, hey, next go-around of grants, next goaround of whatever, continue it, because it's really beneficial to this community, and it's working well."

Outcome #2: Consistently engaged stakeholders in research that facilitated social learning
Stakeholders acknowledged that the project provided multiple and consistent opportunities to collaborate with researchers to guide their research as well as "ground truth" their findings. Stakeholders explained, for example, that the project: • Incorporated diverse local perspectives: "The people that were doing the research weren't just saying, 'Look, I did this, and this is the way it is.' They weren't like that. They were like, 'I did this, but there could be variables on this'…And they were very good about reaching out to the stakeholders of the river and saying, 'What would you like to see? What scenario would really interest you?'… It just seemed to me like everybody was really kind of conscientious about that and not just you know, 'Let's slap it together and then here it is.'" -S • Adapted the collaborative research process to meet emergent climate information needs: "The leadership, with input from the participants [stakeholders], made some changes in the project scenarios…The group that was leading the project was very flexible and took interest in listening to what the participants had to say…if someone had a question or concern they would address it in the next meeting by adapting the project or bringing in a specialist."" • Valued stakeholder participation: "I think that we [the stakeholders] launched with this [project] at the ground floor and helped throughout our discussions at the [collaborative modeling] workshops." • Co-produced new climate science information useful to supporting local climate adaptation: "I thought, 'Why don't we work with this group, get actively involved, and use that data in our basin planning process?'… And we've been very active to make sure that what is coming out of this is usable. Because too often you get studies that are great but how do you implement it? This study was much better than that. We were able to customize some of the work that would apply more directly to what we're doing here." As a result of these consistently facilitated and iterative interactions, stakeholders indicated that they valued both the collaborative research experience and findings, which likely contributed to their ongoing highlevel of participation over the life of the project. Stakeholders noted that workshops created collaborative space for individuals who share a history of water conflict to learn from one another and work together to evaluate implications of results of climate scenarios and adaptation strategies on water availability across the river system. As one stakeholder reflected: "It's kind of like water's such a touchy issue, you know, and it gets really personal. Like, a shovel in your hand, you're ready to whack a guy that's taking your water a little too early again… I think that the study really allowed us to just look into it from different per-spectives…We're all stakeholders. And I think it really opened up that discussion." As another stakeholder reflected: "The atmosphere that was developed made it a collaborative effort to share info and perspectives, or thoughts or recommendations. It was the setup and the environment they [extension researchers] created. Collaborating and having more open communication with other entities in the area was a highlight."

Outcome #3: Provided iterative and structured interaction between stakeholders and interdisciplinary research team
Stakeholders described the workshops as a critical component of project success. As described in the Methods section, these 5-hour workshops were designed to provide a collaborative workspace to convene the research team with stakeholders in order to ground truth or validate research findings, which helped to ensure that the resulting information co-produced was useful to stakeholders (see Table 2). Stakeholders highlighted workshop attributes such as: • Pre-workshop lunches: "They made an optional lunch available [before the workshops], which was a great idea because it's that oneon-one peer interaction that happens ahead of the meeting, which was valuable." -• Structured facilitation: "I think the [workshop] facilitation led towards a greater level of participation and collegiality, quite frankly." • Relevant agenda topics: "I was amazed that people attended to the degree that they did… I did not find stakeholder fatigue. So, I would say that, to me, was a big value of this [project] because the stakeholders were engaged at a super high level, and they were interacting in a way that I think was actually solving problems on the ground." • Research expertise: "I think the quality of the science and the sci- Okay. Well, this is awesome,' because they had copies of the presentations, which has been helpful because I've been able to refer to that… It was very user-friendly." The information binders were created to support social learning during workshops and guide stakeholders' further dissemination of findings to their respective organizations. As one stakeholder described: "I go to their [workshops]. Then I go and present [at my organization's meetings], I'm actually giving a lot of that information and findings, and bringing it forward to them… a foundation that I can say, 'Based on the science, here's the direction we ought to be going…" As another stakeholder described: "This was the kind of information that was shared with our Board of Directors, who are, themselves, all [agricultural]  Stakeholders acknowledged that extension researchers had carefully recruited Stakeholder Affiliate Group participants who possessed extensive local knowledge and expertise concerning varied water use interests across the river system. As one stakeholder shared: "I really think that the collaboration that they were able to achieve as part of the process was very valuable, of just bringing the right stakeholders together." As another stakeholder explained: "They had high-level people too. They didn't have entry-level positions, which sometimes people than just don't engage. They had the right level of the organization participating…you want all levels of organizations to be engaged, but something that is this important, you really needed more of a higher management focus. And it seemed like they [the extension researchers] were getting it." Coincidently, stakeholders commented that the project included the right selection of modelers, well respected within their disciplinary fields. As one stakeholder shared: "I really had a high level of confidence in those that made presentations. It involved people with a lot of acquired skill and learning… I'm hard-pressed to tell you there wasn't anyone there that didn't have some skin in the game." As another stakeholder mentioned: "The diverse presenters and the information they brought to the workshops was very valuable, it wasn't just in one area, it was very diverse, and there was a lot of knowledge that came forward." 4 Stakeholders noted how researchers remained objective in their analyses and transparent in their interactions with diverse water use interests and information needs. As one stakeholder reflected: "What I was really impressed with by the study was it's probably the most unbiased study I've been involved in because nothing that I could tell got left out. I mean, every drop of water that entered the watershed, to the best of their ability, was recorded and they didn't leave out anything." As another stakeholder explained: "The value was really just being a part of the cutting edge of the research and the science that was helping shape water policy and environmental policy in the region."

Outcome #5: Effectively utilized extension as a boundary organization to guide and support collaborative research
Researchers who were interviewed identified the partnership with Extension as a boundary organization as advantageous to facilitating interactions between stakeholders and researchers, tailoring workshop content to reflect and ascertain stakeholders' information needs and helping to revise research objectives accordingly. Research team members interviewed contributed to particular project attributes that worked well, including: • Setting a research agenda characterized by transparency and flexibility: "I think we were very clear that we didn't have an agenda.
We're not working for any one stakeholder. They all have a voice… That first workshop, we spent, I think, a fair amount of time describing, 'This is what the project is.' But then, each workshop, we open with a couple of slides just reminding everyone, 'Yeah, you all have a voice, and we're not here to solve any long-standing issues… And so, creating that semi-safe environment -that safe space for everybody." • Building confidence in the models and model results: "You meet with [stakeholders] several times before the actual models are run and you build that confidence. So, it took a long time. It wasn't just-you don't just send them an email and say, "Well, show up at our meeting…It's meeting with them regularly, and building that trust over time." • Building relationships and trust in the research team: "Convening as often as needed and having the funding to kind of run back and forth and talk to the stakeholders as often as needed. I think it's really strengthened the relationships with these stakeholders. And as a result, they're listening to the research results that maybe they don't necessarily want to hear, or they're skeptical about. So, I think we've really built a lot of trust." Another identified strength concerning extension researchers involved their expertise in navigating sociopolitical complexities of multi-party water issues with the goal of finding common ground. Research team members described how extension researchers helped facilitate dialogue surrounding potentially contentious model simulations. For example, modelers sought to use the Carson River integrated hydrologic-operations (GSFLOW-MODSIM) model to explore water supply benefits of more far-reaching adaptation strategies, such as constructing a surface water reservoir in the headwaters even though such an undertaking is highly unlikely due to environmental concerns. Also, from the project's outset, modelers expressed interest in using this same model to simulate additional strategies, such as managed aquifer recharge, which would divert water away from the river system terminus during winter months to recharge Carson River Valley groundwater (e.g., Niswonger et al., 2017). Some stakeholders regarded this as viable from a physical perspective and were open to the model results, while others voiced disinterest since this strategy is prohibited currently under the prior appropriation doctrine and a federal decree which allocates and regulates Carson River water use. The outreach materials we created for stakeholders translated research findings to increase the usefulness of the co-produced climate science information. Extension researchers created and maintained a responsive collaborative research process that carefully facilitated workshop discussions designed to provide a safe space for stakeholders to voice and validate their information needs. For example, stakeholders voiced concern over potential adverse impacts to downstream users if upstream flows were diverted for managed aquifer recharge. Extension researchers facilitated these social learning experiences to balance to the greatest extent possible historical power disparities, including the tensions involving water allocations from irrigated agriculture to environmental instream flows and Native American tribal lands and fisheries.

Interpreting evaluative results to identify project successes
In the Water for the Seasons project a collaborative research design adopted a normative lens in the co-production of new climate science information tailored to support local climate adaptation (Bremer and Meisch, 2017;Dilling and Lemos, 2011;Lemos, 2015;Lemos et al., 2018). This research design facilitated interaction between researchers and stakeholders representing diverse and competing water management interests across the Truckee-Carson River System. Analyses of the formative and summative evaluative results suggest these ongoing interactions were effective in identifying the impacts of climate-induced water supply variability on the river system and supporting water managers with information needed to strategize climate adaptations across the system. Formative and summative evaluation helped to test the efficacy of project methods, capture the progress and fluidity of the project, and provide an overall picture of project success in achieving its targeted goals (Fazey et al., 2018).
Recall that the formative evaluative data collected through face-toface interviews in year one of the project provided baseline understanding of stakeholders' expected outcomes, including climate information needs that would support local adaptation and improve stakeholder communication and coordination. These outcomes allowed extension researchers to create and implement a stakeholder-informed evaluation instrument critical to tracking the project's developmental progress over the remainder of the project. This formative evaluation helped to determine whether the modeling and associated workshops featured in our collaborative research included diverse water use interests and perspectives which theoretically would serve to increase the usefulness of resulting climate science information in supporting local adaptation decisions (Lemos et al., 2018).
Situating the results from this evaluation instrument in the context of the summative evaluative data, collected by an external evaluator, validated the efficacy of the iterative interactions that were designed to clarify and address stakeholders' climate science information needs through simulating adaptation strategies to manage increasingly variable water supplies (Gober, 2018). As Outcomes 1-4 demonstrate, the collaborative research methods deployed co-produced legitimate climate science information concerning the effects on river system function of climate change and strategies for adapting to these changes. That is, the research team established and maintained as a common goal to identify and satisfy the diverse information needs of a carefully selected core group of diverse water management organizations key to planning and implementing adaptation to enhance climate resiliency across the river system. Additionally, stakeholders reported communication and coordination had generally improved over the course of the project -validated by Outcome 4.
As Outcome 5 illustrates, Extension performed as an effective boundary organization in designing and implementing a collaborative research undertaking. The modeling workshops, for example, attributed to project successes as illustrated by both stakeholders' and researchers' perspectives documented in year four of the project. Several workshop features that appeared to work well, including sharing the appropriate amount and type of information so that stakeholders were able to digest the information and immediately gage its usability in the field. Workshop preparation therefore included extension researchers assisting modelers with science translation, including the content and quantity of material presented. To support stakeholders' snowball dissemination of research results to the broader stakeholder community, presentation slides, recent academic and outreach publications, and related materials were assembled into the "take-home" information binder. These educational materials supported continued learning and were referenced positively by the majority of stakeholders interviewed.
In advance of the workshops, extension researchers went to great lengths to ensure effective facilitation of the meeting agenda and interactive discussion. Part of this effort involved investment in time to design workshop content and organize presentations and discussion of content so it followed a logical, and generally spatial, order that made sense to stakeholders representing water management from system headwaters to terminus. Another critical feature of workshop preparation was developing a seating chart to increase communication among stakeholders representing diverse interests, in addition to facilitating information exchange between stakeholders and modelers.
Related to these collaborative research design elements was the importance of identifying the roles and responsibilities of stakeholders and researchers early in the project. This included ascertaining from individual stakeholders their organization's commitment to partner in this research, despite a longstanding record of conflict involving all of the water management interests and organizations represented by the Stakeholder Affiliate Group. Extension researchers clarified with stakeholders early on, and reminded at the beginning of each workshop, that the project was not intended to address long-standing conflicts or revisit adjudicated and negotiated outcomes. Rather, we conveyed that the sole intent of the project was to harness stakeholders' knowledge to inform research and modeling activities and co-produce climate science information that might enhance climate resiliency of the river system.
Similarly, modelers were encouraged to share their disciplinary expertise, while incorporating stakeholder input in their modeling tools. Extension researchers worked closely with modelers to ensure workshop materials complemented identified information needs and facilitated structured discourse to provide stakeholders opportunities to offer new insights to viable strategies to potentially enhance river system resiliency to climate change. Ultimately, these interactions generated new information of river system function, allowed modelers to push the envelope of their modeling methods and empirical analyses, evaluated the viability of stakeholder identified adaptation strategies under existing water management institutions, and identified further research needed to navigate the spatial and temporal scales of climate change impacts across the river system.
Drawing from results from additional evaluation questions asked following the sixth and final collaborative modeling workshop suggests that despite documented successes, areas remained for improvement. That is, when asked what the project team could have done differently to support climate adaptation, stakeholders provided several recommendations. With regards to procedure, stakeholders requested that extension researchers: 1) incorporate multiple stakeholders from each part of the river system; 2) conduct public workshops in different locations across the river system; and 3) increase communication between stakeholders and researchers through more frequent engagements. With regards to research substance, stakeholders requested that future projects: 1) incorporate climate scenarios that include population growth and development; 2) expand the scope of analyses to examine implications for water quality; and 3) provide technical support to stakeholders in order to integrate research results into the adaptation planning efforts of the respective organizations. As one stakeholder remarked, " [Our] funding is limited and receiving support from a hydrologist isn't possible."

Evolving best practices to guide collaborative research
The experiences and outcomes reported here lend additional insight into efforts underway to identify and empirically test collaborative research practices in the context of climate resiliency (BenDor and Scheffran, 2019; Klink et al., 2017;Voinov et al., 2018;Wall et al., 2017). For example, an interdisciplinary team of scientists working alongside a core stakeholder group throughout the life of a project can synchronize research objectives to address salient water management challenges and meet climate information needs to support local climate adaptation. As others have established, collaborative research requires significant time, energy, and funding. Additionally, specialized skills are necessary to design, coordinate, and facilitate dynamic structured interactions involving multiple models, research methods, academic disciplines, and research agendas (Hamalainen, 2015), as well as stakeholders willing to commit the time and resources needed. Therefore, paramount to the success of collaborative research is careful identification early on of historical, existing, and/or potential environmental disputes and conflicts and have a conflict mitigation plan prepared (BenDor and Scheffran, 2019; Djenontin and Meadow, 2018).
While collaborative research has gained substantial credibility and funding in recent years, particularly in the context of climate adaptation, scientists continue to grapple with selecting the most effective methods of engaging stakeholders in meaningful and ethical ways to achieve mutual project goals Vincent et al., 2018;Klink et al., 2017;Singletary and Sterle, 2018;Voinov et al., 2018). Sufficient evidence confirms that such approaches tend to select methods aimed at improving communication and building relationships (Norstrom ¨et al., 2020;Djenontin and Meadow, 2018;Cundill et al., 2019). This may include experimenting with various social networking events, such as workshops, designed to convene scientists and stakeholders around problems of common interest. While frequent interactions over time can forge a mutual understanding of information needs, language, and culture, stakeholders and scientists remain members of their respective communities which include different expectations for performance . Regardless of stakeholders' needs for useful climate science information, researchers must satisfy professional expectations to produce high quality, theoretically-driven research with generalizable and publishable findings (Hegger and Dieperink, 2014).
In this regard, researchers are also stakeholders and risk-takers in the context of collaborative research (Lang et al., 2012), and candid discussions should occur early in the project to identify individual and collective expectations. Thus, to increase the likelihood of realizing substantive outcomes, projects must identify both stakeholders' and research team members' expectations and information needs (Ferguson et al., 2017;Reed et al., 2014), acknowledge when collaborative research methods are underperforming, or failing, and revise accordingly (Allen et al., 2017;White et al., 2010). For these reasons, it is necessary to evaluate, as a process element, the frequency of interaction and quality of communication that occurs among researchers, among stakeholders, and between researchers and stakeholders (Fazey et al., 2018;Ferguson et al., 2017;Meadow et al., 2015;Singletary and Sterle, 2018).
While systematic evaluations of collaborative research outcomes remain rare, reports from the field, such as this one, indicate that efforts to assess strengths and weaknesses can inform future collaborative undertakings. As demonstrated here, at the project outset, stakeholders were asked to establish criteria as goals and outcomes that were used to assist in evaluating the extent to which the project was achieving its goals over time. Because collaborative research intentionally engages stakeholders with diverse interests and information needs, it follows that stakeholders may have diverse expectations for project outcomes (Campos et al., 2016;Fazey et al., 2018;Klink et al., 2017). Therefore, a retrospective evaluation is necessary to more fully assess the extent to which the project achieved its intended goals and also to gage its longerterm effects on both stakeholders and researchers. That is, formative evaluations can identify improvements needed throughout the collaborative undertaking and summative evaluations can identify immediate outcomes, but retrospective evaluations, six to 18 months post-project, can reveal what worked well and did not -to identify longer term effects that might inform best practices to strive for in future undertakings (Wall et al., 2017;Klink et al., 2017). And, additional research is needed to compare and contrast similar case studies in order to identify benchmark collaborative research milestones and warning signs of trouble that should be addressed upon discovery (Falconi and Palmer, 2017;Voinov et al., 2016). Collaborative research could benefit from a set of empirically tested metrics that: 1) evaluate its effects on science and society as evidenced through changes in natural resource management practices and policies; 2) facilitate consistent institutional support for collaborative research; 3) acknowledge the additional time and resources necessary to undertake large interdisciplinary undertakings and the facilitation skills needed to support collaboration; and 4) funding to investigate and address identified barriers that emerge during the coproduction process (Voinov et al., 2018). Empirically testing commonly used metrics should help to verify which collaborative research methods work more effectively than others and under what conditions. Until a standardized framework is established, however, a broad range of practices, expectations, experiences, and thus processes and outcomes, are likely, which may inadvertently undermine the advancement of collaborative research (Thompson et al., 2017).

Conclusions
Formative and summative evaluations of Water for the Seasons, a fiveyear collaborative modeling research project in the Truckee-Carson River System Sterle, 2017, 2018), suggest that the collaborative research processes featured were generally successful in generating new climate science information that local water managers can use to aid in enhancing the resiliency of that river system. That is, the normative collaborative research process described here, that relied on structured, iterative interactions involving clearly identified and willing key stakeholders and researchers, generated new climate information intended as services that could support local climate adaptation. Resulting climate information services unique to this study included hydrologic and operational simulated adaptation outcomes to climate scenarios. These adaptations included reservoir reoperations on the Truckee River, managed aquifer recharge on the Carson River, and assessing the performance of existing institutions to reallocate water to higher-value land uses (see Sterle et al., 2020aSterle et al., , 2020bSterle et al., , 2019Lee et al., 2020).
The empirical evidence presented here demonstrates linkages between collaborative research processes that stakeholders perceived as meaningful and project outcomes that stakeholders perceived as useful to climate adaptation decision-making, respectively. These results suggest that structured social learning and information exchange may serve to enhance climate resiliency in the future particularly when stakeholders have consistent, practical opportunities to collaborate with researchers to generate new climate information and use that information to support climate adaptation (e.g., BenDor and Scheffran, 2019). Evidence of positive collaborative research outcomes, in addition to realistic collaborative process challenges, such as those described here, may be useful to interdisciplinary research teams embarking on collaborative research and deciding whether or to what extent boundary organizations can play a key role in their success. While careful design and implementation of collaborative research is important, it is an insufficient condition to ensure successful processes and outcomes (Cundill et al., 2019). Success is also strongly influenced by relational features such as leadership style, mutual respect, communication skills, and interpersonal trust. Systemic features of collaborative research must be considered, including past and existing institutional cultures and values, power asymmetries among stakeholders and researchers alike, and any binding contractual or legal agreements (Turnhout et al., 2020;Cundill et al., 2019). Given these multiple, coincident factors, it is likely that experienced boundary organizations, willing to provide dedicated project leadership and coordination at multiple levels, can help to improve the likelihood of successful collaborative research processes and outcomes.
Additional research is needed to compare and identify which collaborative research methods are most effective under which conditions and circumstances. Comparisons of collaborative research project outcomes are rare because each project involves different stakeholders, research teams, and group dynamics and therefore subjective values, biases, and perspectives and the selection of methods approaches and tools vary (Voinov et al., 2018). Nevertheless, formative, summative, and retrospective evaluations, conducted up to 18 months after the project, should comprise a standard component of collaborative research in order to arrive at a set of best, or at a minimum, ideal guiding practices.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.