Understanding the Impacts of Research Synthesis

Research synthesis is the integration of existing knowledge and research findings pertinent to an issue. The aim of synthesis is to increase the generality and applicability of those findings and to develop new knowledge through the process of integration. Synthesis is promoted as an approach that deals with the challenge of ‘information overload’, delivering products that further our understanding of problems and distil relevant evidence for decision-making. However, despite the increasing prominence of synthesis efforts in the science and policy landscape, we know very little about the impacts these initiatives have on research, policy and practice and the assumptions underpinning how they will lead to change. This paper presents a framework for considering the conceptual, strategic, instrumental and network-based impacts of research synthesis on policy. This framework provides insight into the range of underlying assumptions and impacts on policy and practice from 10 case studies of research synthesis related to contemporary sustainability challenges. Findings suggest that research synthesis is having diverse impacts on research, policy and practice including creating a new understanding of problems, establishing new networks, and contributing to changes in policy and practice. These impacts emerged across a range of contexts, synthesis methods, assumptions and operating models. This suggests that there is no single ‘correct way’ to design research synthesis for impact, but rather a need to tailor the approach for the context of intended use.


Introduction
Research or scientific synthesis is the integration and assessment of knowledge and research findings pertinent to a particular issue with the aim of increasing the generality and applicability of, and access to, those findings (Hampton & Parker 2011, Magliocca et al. 2015, Baron et al. 2017). Synthesis of existing research and case studies can also generate new knowledge. Synthesis efforts often bring together different academic and non-academic forms of knowledge and evidence.
Assumptions underpinning the value of syntheses are multiple. Synthesis is hailed as a means of taking science up an evidence hierarchy to have greater impact on policy processes (Dicks et al., 2014), of addressing the challenge of 'information overload', delivering products that can help improve scientific understanding in decision-making (Hampton & Parker, 2011), and providing critical knowledge to solving environmental problems (Carpenter, 2009). The production of 'summaries for policy-makers' is indicative of a drive to tailor scientific information for decision-makers' needs. Studies have shown that synthesis contributes to the scientific community through initiating new collaborations and producing new knowledge (Hampton & Parker, 2011;Barron et al., 2017).
Yet the impact of synthesis on policy and practice, and the validity of the assumptions underpinning impact have rarely been examined empirically. Concerns have been raised about the marginal influence of large, costly assessments on the complex world of global policy (Pahl-Wostl, 2015). Those searching for a simple linear translation of 'sound science' to 'evidence-based policy' in global processes like the Intergovernmental Panel on Climate Change (IPCC) assessments or the Millennium Ecosystem Assessment (MA) will be disappointed. The archetypal experience of the IPCC shows that greater rates of scientific certainty do not correlate simply with policy action. However, there is growing evidence to suggest that global assessments have had a profound, if hard-to-measure impact, on policy agendas at different scales (Beck, 2015;Risousset et al., 2017).
With the growing prominence of synthesis initiatives in the science-policy landscape (Baron et al., 2017;Specht et al., 2015), it is important to ask what assumptions are guiding these initiatives, what types of impact they have had, and under what conditions they are most likely to lead to impact, by which we mean significant changes in research, policy or practice. This paper presents findings of an exploratory review conducted to understand the rationale, approaches to, underlying assumptions and impact of synthesis initiatives.
Below we present our methods and conceptual framework used to support analysis. We present summary data from 10 case studies and arrange review findings under five headline insights. The discussion reflects on these findings and other emergent observations. As an exploratory review, the findings should be taken as hypotheses to be tested or further explored. We conclude by discussing implications and areas for future work.

Methodology and Methods
Assessing research impact is a complex but growing field, and empirical examination of the impacts of synthesis published in the academic literature is limited. Consequently, this review draws on both published and grey literature, and in particular on case studies for which the authors acted as programme designers, facilitators, or analysts. The primary focus is biodiversity conservation and natural resource management. While the review initially focused on impacts on policy, impacts on practice and research also emerged.
For the purposes of this review, research synthesis is conceptualised as a process of reviewing, assessing and synthesising existing literature or data to produce a series of outputs (products and services). Synthesis is often conducted by academic disciplinary experts, but can involve inter-or transdisciplinary working groups drawing on knowledge from across academia and beyond. Policy is defined as a formal decision or an outline of an overarching plan made by groups seeking to implement these decisions to achieve a particular goal (Richards and Smith, 2000). Such groups include local governments, non-government organisations (NGOs), corporations and community groups, as well as nation states and international bodies. Following Game et al. (2015), practice is considered as the actual application of methods that lead to the design, implementation, management and monitoring of projects or programmes. Impact is understood broadly as a range of the positive and negative, primary and secondary, direct or indirect, or intended or unintended effects of a programme or initiative (see Hearn and Buffardi, 2016).

Conceptual Framework
Synthesis initiatives and associated impacts cannot be understood in isolation of their current and historical context, or of the processes used to conduct synthesis (see Figure 1). The enabling environment in which the synthesis is situated includes the policy context; the governance of the initiative itself; and the capacities of individuals (from both science and policy) to undertake synthesis and act on findings; and the sources of funding (Clark et al., 2006;Clark et al., 2016). Other contextual factors that can influence the synthesis process itself and its impacts include the political contentiousness of an issue, legacy of past decisions or actions, and the scale of a problem or initiative. While knowledge products are often the focus of synthesis, the synthesis processes itself can be a vehicle of change, and the governance and process itself is a key determinant of policy impact (Clark et al., 2006).

Case Studies
The processes, assumptions and institutional arrangements supporting research synthesis are many and diverse. The review set out to explore this diversity, considering a number of approaches and institutional contexts, including global assessments, specialist centres of synthesis and analysis, thematic assessments and ecosystem service assessments. As a result, some of the cases considered are time-bound initiatives, others are ongoing; some cases consider the synthesis initiative itself, while others are organisations or initiatives that support or conduct synthesis.
10 case studies were identified through a selective sampling strategy to encompass a range of types and approaches to synthesis, across a diversity of scales and contexts. Data was compiled by the authors based on their involvement in the design, research or implementation of these initiatives, drawing on the authors' experiential knowledge as well as project plans, outputs and other relevant material.
Case study data was compiled using a template that considered the following: • Aims and objectives: audience; assumptions underpinning impact pathway; and definition of impact. • Context: scale (global, national, local); focus; history; size and drivers of the initiative.
• Governance: public, private, or NGO; funding sources and budget; presence or absence of a policy mandate; partners. • Methodology: nature of partnership and collaboration; type of synthesis; types of knowledge accepted as evidence.
• Outputs and impacts: types of products; identification of impacts; scale and timeframe of influence. • Reflections by the authors: strengths and weaknesses of approach; achievement of desired impact; key lessons learned. While we noted the synthesis methods used, we have not focused in detail on these, nor on the questions and data analysis techniques. Rather, our analysis considered the broader context in which synthesis takes place within the landscape of research, policy and practice, the assumptions underpinning initiatives and what impacts it has had in these domains. The review did not set out to systematically collect information on questions of independence, susceptibility to bias, transparency, rigour or reliability (primarily because of the broad range of methods considered in the case studies). However, insights on these issues emerged through the analysis and are referred to below.

Literature Review
A literature review was conducted to complement and support case study analysis. This review was used to identifying the state of current knowledge (and knowledge gaps) on the impact of research synthesis and to inform the development of the conceptual framework. Within the thematic subject areas and specific synthesis categories, the review used the following search terms: • synthesis research; research synthesis; interdisciplinary research; transdisciplinary research; • AND policy impact; research impact; impact; knowledge transfer; knowledge exchange. The search was conducted using the Web of Knowledge database and Google Scholar in January and February 2017.

Analytical Approach
The lead author conducted a comparative analysis to identify similarities and differences across cases and the published literature. This analysis drew on the conceptual framework ( Figure 1), with the objective to identify and categorise the assumptions about, and impacts research synthesis achieve, and under what conditions. Analysis was structured around the following types of impacts from research synthesis (adapted from Hezri, 2004;Waylen and Young, 2014;Owens, 2015): • Conceptual: introducing new ideas, new issue framings or perspectives, or shifting understanding of the efficacy of different response optionsknowledge is used to change understanding of a system or phenomena (for example, demonstrating the connection between human well-being and ecosystem health); • Relationships: networks emerge or are catalysed through the research process or as a product of its findingsknowledge changes understanding of the interconnections between actors, sectors or systems (for example, demonstrating the migratory pathways of species is used to justify collaborations between different public and private land managers); • Strategic: to justify a policy or activity, proposals for new work, advocacy. This use is sometimes characterised as biased or as 'policy-based evidence' where knowledge is used to support a particular view of a problem (for example, promoting climate change as an economic challenge to motivate political action).
• Instrumental: for action and problem solving through models, planning tools or organisational policies, where research is used directly or rationally as the prime source of information to inform policy change or reform (for example, the use of decision-support tools in fisheries management). • Capacities: building or enhancing the capacity to engage in sharing and use of scientific information to inform decision-making, where knowledge use is dependent on the capacity to understand or apply knowledge in a given context (for example, communication training for scientists and policymakers) These categories were also used to interrogate the assumptions underpinning the goals and approaches of research synthesis initiatives, as they represent different perspectives about the nature of the relationship between knowledge and policy change. While it is possible to analyse the processes and methods of synthesis research, we have restricted our analysis to impacts and assumptions as these areas have received less attention within the published literature.
The impact of research is notoriously difficult to track with clear causal chains of evidence, particularly given the underdeveloped monitoring this field (Alcamo, 2017;Bauer et al., 2016). The primary limitation of this review relates to the methodology used to compile case study data, which involved self-reporting by people involved with producing rather than using the synthesis in a given policy context. Impacts presented below therefore represent authors' individual perspectives, requiring further analysis to assess their consistency across a variety of perspectives. The literature review was restricted to literature and documents written in English, and this is recognised as a shortcoming.

Case Study Summary Data
Data is presented below using the conceptual framework presented above. For more details see supplementary online material.

Context
Here we provide a brief overview of the context of the initiatives, each operating within a range of geographical scales, drivers and origins.
Millennium Ecosystem Assessment (MA) (2001)(2002)(2003)(2004)(2005): The MA was a global environmental assessment that focused mostly on biodiversity and ecosystem services. It involved over 1,000 experts worldwide from across natural and social sciences who produced five technical volumes and six synthesis reports addressing both thematic and methodological issues, as well as multiscale assessments.

UK National Ecosystem Assessment (UK NEA) (2009-2011):
The UK NEA was the first analysis of the UK's natural environment in terms of the benefits provided to society. It was an inclusive process involving many government, academic, NGO and private sector institutions. A second phase, reported in 2014, provided new information and tools to help decision-makers across all sectors understand the wider value of ecosystems and the services they provide.

Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services (IPBES) (2012-present):
IPBES is an intergovernmental body which aims to increase access to knowledge that will support decisionmaking with respect to biodiversity and ecosystem services. IPBES currently has 127 governments as members, and involves over 1,000 experts worldwide. One of the four main functions of IPBES is to carry out global, sub-global, thematic and methodological assessments.
National Socio-Environmental Synthesis Center (SESYNC) (2011-present): SESYNC supports fundamental but actionable science focused on solutions to pressing socio-environmental problems. The Center cultivates and supports an interdisciplinary community of natural, social and computational scientists by providing opportunities to conduct and build capacity for research synthesis and engagement. SESYNC is part of the University of Maryland and receives support from the U.S. National Science Foundation.
Science for Nature and People Partnership (SNAPP) (2013-present): SNAPP brings together multi-disciplinary groups to tackle pressing issues facing people and nature (Lortie and Bonte 2016) that individual institutions and researchers often lack time and resources to address quickly and effectively (Kareiva et al. 2014). SNAPP is a partnership between The Nature Conservancy, the Wildlife Conservation Society and the National Center for Ecological Analysis and Synthesis at UC Santa Barbara. SNAPP working groups include government, NGO, foundations, development banks, and the private sector, as well as humanitarian, sustainable development and conservation practitioners.

EU Knowledge and Learning Mechanism on Biodiversity and Ecosystem Services (EKLIPSE) (2016-present):
EKLIPSE aims to, inter alia, provide knowledge syntheses on biodiversity and ecosystem services-related topics upon requests from policy and societal actors. The process aims for a high level of transparency and engagement, offering a set of more than 20 potential synthesis methods to be used based on the question, timeframe and knowledge needs identified. EKLIPSE is funded by the European Commission's Horizon 2020 programme.
Collaboration for Environmental Evidence (CEE) (2008-present): CEE -an independent NGO -is an open community of stakeholders who promote the synthesis of evidence on issues related to environmental policy and practice. It supports the conduct of evidence syntheses to CEE-established guidelines and standards, publishing all products in open-access format. CEE engages with conductors, commissioners of evidence syntheses, consumers of evidence and wider stakeholders to develop suitable questions for synthesis.

Lenfest Ocean Programme at The Pew Charitable Trusts (Lenfest) (2004-present):
Lenfest is a grant-making programme that funds scientific research on policy-relevant topics concerning the world's oceans. Lenfest uses syntheses as one research approach, and commits significant resources to engaging with management bodies to connect its research results to decision-making about ocean ecosystems, without advocating for specific policy positions.

WWF Global and Regional Policy Hot House (WWF Hot House) (2009-present):
The 'Hot House' is a generalist support unit designed to help WWF by undertaking literature reviews of published and grey literature and drawing on internal and external expertise. They engage directly with decision-makers to deliver political, economic and security arguments in favour of environmental conservation in language relevant to the context of the specific international event (ex: Rio+20, MDGs, SDGs, G-20, UNESCO, and other UN processes), as well as on the overall political agenda.

Commission of Inquiry on Peatlands (IUCN UK Peatland Programme) (2010-2011):
The Peatlands Inquiry involved a series of national conferences bringing together members of the research, policy, practitioner and business communities. The Inquiry comprised 12 working groups focused on a range of issues to address scientific uncertainty and develop recommendations for restoration policy and programmes. The Inquiry is being refreshed in 2018 in light of new evidence to meet new policy and practice needs.

Governance and funding
Initiatives reviewed have a variety of governance arrangements, ranging from those singly housed within academia (SESYNC, EKLIPSE) or within NGOs (Lenfest and the WWF Hot House), to stand-alone NGOs (CEE) and those that had a mixture of entities involved across academia, private sector and NGOs (MA, UK NEA, Peatlands Inquiry). SNAPP was the only explicit partnership between academia and an NGO, while IPBES was the only intergovernmental process considered. Within this diversity, governance mechanisms ranged from strategic or scientific advisory boards operating with memoranda of understanding or collaborative partnerships (SESYNC, SNAPP, EKLIPSE, MA) to formal governmental or intergovernmental processes (IPBES, UK NEA). Initiatives also benefited from a diverse range of funding sources, including philanthropic (SNAPP, Lenfest, CEE, WWF Hot House), research (CEE, EKLIPSE, SESYNC), government (UK NEA, IPBES), and international multi-lateral donors (MA, IBPES). Total funding varied from hundreds of thousands to many million dollars.

Connection to a decision-making context
The extent to which initiatives are directly connected to a decision-making context -in either policy or practice -was variable, as is the nature of the connection. Five case studies had direct mandates from a particular policy audience who could act upon the research findings (MA, IPBES, UK NEA, UK Peatlands Inquiry, EKLIPSE). In many cases, while the whole is not embedded in a policy context, individual research projects might be (SNAPP, CEE). For example, CEE as an entity is purposely free of mandate to retain independence, but authors can be mandated to conduct an Evidence Synthesis that CEE may endorse depending on standards of conduct. IPBES assessments are requested by governments, according to agreed scoping documents, and the 'summary for policy-makers' is formally approved when governments receive the final reports. In other cases, the relationship between the initiative and its intended policy audience is more informal. For example, WWF Hot House is 'associated' with many UN bodies, and CEE discusses 'networking' with policy-makers (for reviews that aren't directly commissioned).

Capacities and expertise
The initiatives drew on a diversity of capacities depending on their scope and scale. For example, a single researcher undertakes synthesis for the Hot House and is supported by a small team of WWF staff who take the work into policy arenas. By contrast, IPBES assessments, the UK NEA, MA and the Peatlands Inquiry involve large groups of authors. The range of participants in the synthesis processes also varied greatly. Some included end-users as members in working groups (SESYNC), as government representatives overseeing the process (the UK NEA,) or reviewers (the MA). CEE involves end-users and stakeholders in the planning process and production of a synthesis protocol, however, they are not involved in the conduct of the synthesis, because of concerns over potential for introducing bias.
There is also variability in attention to the composition of researchers. Some initiatives explicitly include diverse disciplinary perspectives, gender and geographical representation (SNAPP, SESYNC, EKLIPSE, IPBES). IPBES, for example, has operational guidelines and a formally agreed process to achieve this, whereas SNAPP and SESYNC's calls for proposals and review process are used to support this objective. SESYNC provides guidance on the make-up of teams, including disciplinary, institutional and demographic diversity, and requires that the majority of a team's participants have not previously collaborated. Some initiatives also included a specific focus on building the capacity to produce and/or use research synthesis (IPBES, MA, SESYNC, CEE and EKLIPSE).

Objectives
All initiatives reviewed generally aim to provide policy-relevant science; however, within this broadly defined goal, individual objectives vary. Four subcategories emerged through our analysis. These are not exclusive, as some initiatives aim to address multiple objectives or to use several strategies in addressing their objectives.
• Providing information through reviews and assessments: These initiatives tended to be oriented to presenting the state of knowledge on a given topic (CEE, IPBES, the MA, EKLIPSE and the UK NEA).
Cases here included those focused on collating and synthesising existing knowledge, as well as identifying knowledge gaps and in some cases also gaps in capacity.
• Conducting research synthesis: Here, objectives mostly focused on the research process itself, with the aim to gather different scientists together and conduct a novel kind of research. Not all these cases have an explicit focus or mandate on getting research into policy. For example, SNAPP is explicitly focused on direct connection to implementation (although producing publications is also an objective), whereas SESYNC's mission does not include direct policy advocacy. CEE has developed an evidence synthesis methodology (http://www.environmentalevidence.org/guidelines).
• Delivering support to policy or policy-makers: This includes both initiatives with a clear objective to change policy and those aiming to inform policy without being policy prescriptive. For example, the WWF Hot House is explicit about goals to affect the science-policy interface and the UK Peatlands Inquiry was created to inform policy and programmes development. IPBES and the UK NEA aim to be non-prescriptive in this respect, informing policy-making and facilitating the use of policy tools and methodologies.
• Building capacity: IPBES, SESYNC, CEE, and EKLIPSE seek to build capacity to produce or use synthesis, although this is not often the primary objective. For example, SESYNC undertakes graduate and postgraduate training in social-ecological synthesis methods, while CEE conducts training and capacity building courses to produce and use systematic reviews, including working with the Global Evidence Synthesis Initiative targeted at lower to middle income countries. Since 2008, the CEE network has grown to six centres globally. EKLIPSE and IBPES support capacity building to work at the science policy interface, and IPBES also builds capacity to engage in IPBES assessment processes.

Synthesis Methods
The suitability of synthesis methods depends on the scope and scale of a problem, available sources of data, and the intended use in decision-making (Dicks et al., 2017;Pullin et al., 2016). Consequently, initiatives reviewed used diverse methods, with four subcategories emerging through our analysis.
• Working groups: SESYNC and SNAPP conduct synthesis through inter-or transdisciplinary working groups. Generally this involves three to four, three-five day meetings, spread over two-years. The specific synthesis methods depend on the problem focus of the working group. Both require working groups to include implementers capable of influencing policy and practice.
• Systematic reviews: CEE uses a systematic review and mapping methodology that involves the following steps: question setting, creating a protocol, conducting a systematic search using a repeatable strategy, article screening, critical appraisal and data extraction, data synthesis, and finally a written report using a specific CEE template (http://www.environmentalevidence.org/guidelines/section-10).
• Expert working groups formed around assessment chapters: The working groups of IPBES, MA, UK NEA, and Peatlands Inquiry are predominantly made up of teams of researchers, with no explicit requirement to involve implementers in the drafting process. IPBES author teams, for example, undergo a nomination and selection process, and their work is guided by the agreed scoping documents and rules of procedure. EKLIPSE expert working groups are required to follow an explicit method protocol; however, the synthesis method varies depending on the topic and context.
• Individual researcher: Lenfest sometimes provides grants to researchers who conduct the synthesis, while an individual within the WWF Hot House collects external reports and research from around the world and extracts arguments to serve the needs of WWF's advocacy agenda.
Approaches to peer review ranged from mechanisms embedded within the process of producing the synthesis, to use of the formal academic peer review system to no peer review at all.

Engagement Methods
The initiatives revealed a range of methods for engaging with policy audiences. This included policy-makers participating as members of research working groups (SESYNC, SNAPP), as government representatives overseeing the process (UK NEA, IPBES), or reviewers (MA, IPBES). Some initiatives used iterative dialogue to ensure research meets policy-maker needs (WWF Hot House, Lenfest). Engagement also involved commissioning work, as is sometimes the case for CEE, or on consultation or review mechanisms used to ensure that the research conducted by experts is appropriately tailored (EKLIPSE, Lenfest, IPBES).

Outputs
Products commonly included technical reports with summaries for policy-makers, policy briefs, peerreviewed papers, and stakeholder-specific summaries. Beyond these more traditional outputs are decisionsupport tools and models (SNAPP, SESYNC), online information sharing tools like databases (SNAPP, CEE) or knowledge exchange platforms (EKLIPSE), or an online community of practice (MA). Other outputs include conceptual frameworks (MA, IPBES), manuals on how to undertake assessments (MA, IPBES) or systematic reviews (CEE), capacity-building initiatives (MA, IPBES, CEE, SESYNC), and open-access journals (CEE). Some SNAPP working groups have spawned new, self-sustaining conservation initiatives, and both the MA and IPBES have promoted national ecosystem assessments. The diversity of non-traditional products, multiple outputs and targeted communication strategies for different audiences suggests a recognised need to do more than just produce synthesis to ensure its use.

Analysis
Below we present the analysis of underlying assumptions and impacts of synthesis research using the categorisation presented above.

Assumptions
About half the case studies have explicit assumptions about how synthesis might be structured to achieve or lead to impact, or theories of change (a process that describes how a connected set of activities and outcomes will contribute to change). Of these initiatives most mention the relationship between researchers and research users. Although articulated in various ways, most assume that end-users should be involved in the process of research synthesis. Of those who do not use theory of change language or explicitly state their assumptions, it is possible to make inferences about how each initiative views its contribution to making change by reviewing mission and vision statements, goals and objectives. Further, among those that have explicit statements about assumptions we also deduced a number of unstated assumptions about their modes of operation (see Table 1).

Conceptual -
knowledge is used to change understanding of a system or phenomena

Explicit assumptions stated by initiatives • Finding solutions to complex problems in socio-ecological systems requires interdisciplinary, solutions-focused synthesis
Relationshipsknowledge changes understanding of the interconnections between sectors or systems Explicit assumptions stated by initiatives • By engaging with decision-makers, synthesis will be more relevant and have a better chance of informing policy • The impact of synthesis is improved through direct connections with the context within which it will be used • Policy and decision-making is improved by mutual interactions and learning through transparent process and methods • Networks of individuals and organisations situated across science and policy are critical to developing and integrating synthesis into policy Implicit assumptions identified through analysis • Synthesis of existing research will have more impact with policy-makers if they are aware of, or have been involved in, scoping or design of an initiative • Science can inform policy, but research scientists also need to know from policy-makers what they are likely to need in the future Strategicknowledge is used to support a particular view of a problem

Explicit assumptions stated by initiatives
• There is no best approach to synthesis for different questions or contexts, each needs to be tailored through individual scoping • The outputs of research synthesis needs to 'speak' the language of policy for it to engage with appropriate audiences Implicit assumptions identified through analysis • That policy-makers are the 'right' audience to target to improve the use of research in policy • Reframing issues so that they are more clearly understood and relevant to policy audiences will help motivate action • The production of assessment reports or synthesis and their subsequent use in decision-making is an effective approach to catalysing political action Instrumentalwhere research is used directly or rationally as the prime source of information to inform change Explicit assumptions stated by initiatives • Increasing quality and quantity of synthesis will improve use of evidence in policy Implicit assumptions identified through analysis • Increasing scope, quality and scale of available evidence and capacity to undertake synthesis will increase the use of evidence in decision-making • If decision-makers have access to synthesis they will have the capacity and willingness to act on it • High-level meta-analysis, assessment reports, and accompanying summaries for policy-makers or other stakeholders have use across a range of audiences • Research that has been commissioned to answer policy-relevant questions is likely to be used by those who commissioned it Capacityknowledge use is Explicit assumptions stated by initiatives • Building capacity to conduct and use systematic reviews will improve use dependent on the capacity to understand or apply knowledge in a given context of evidence in policy and management decisions Implicit assumptions identified through analysis • Connecting research with policy requires dedicated staffing, skills and resourcing, and/or new partnerships to support this

Impacts
At the outset, we anticipated a range of possible types of impacts that synthesis could have. We found all five expected categories of impact (see table 2).

Category of Impact Examples of Impact References/Links
Conceptualknowledge is used to change understanding of a system or phenomena The MA shaped understanding of the connections between ecosystems and human well-being, and spawned a number of spin-off initiatives seeking to bring the concept of ecosystem services into decision-making.
Spin-offs summarized in the Sub-Global Assessment Network: http://www.ecosystemassessments.net/ The IPBES conceptual framework articulates connections between scientific and indigenous knowledge, and between different 'world views', and this is being applied in IPBES assessments. Diaz  The Government of Rwanda used results of a SNAPP working group to help integrate ecosystem services and social equity into long-term plans for sustained economic growth. Lal et al., 2017 Lenfest provided scientific advice to support ongoing policy deliberations about management of fisheries catch shares. Bednarek et al. 2015 Relationshipsknowledge changes understanding of the interconnections between sectors or systems The IPBES pollination assessment led to change in recognition of the role of indigenous knowledge in assessments, created a global network within the pollination community, and led to a Coalition of the Willing on Pollinators.
Coalition of the willing: https://promotepollinators.org/author/pollination/, Tengö et al. 2017 Involvement of IPBES member governments in reviewing and adopting the 'summary for policy-makers' of each assessment builds their engagement in the understanding and communication of the assessment findings and their relevance to policy and practice (as was also true for the MA and UK NEA).

IPBES Procedures for the preparation of Platform deliverables (IPBES 2015)
Participants in SESYNC and SNAPP working groups formed new collaborations that extended beyond the initial scope of an initiative. The summary for policy makers of the IPBES Pollination Assessment was extensively used by the Convention on Biological Diversity in making recommendations for action by governments and others. CBD decision XIII/15 (https://www.cbd.int/doc/decisions/cop-13/cop-13dec-15-en.pdf) The UK Peatlands Inquiry was used as a foundation for building a policy mechanism that distributes public and private funding for peatland restoration in the UK. http://www.iucn-ukpeatlandprogramme.org/publications/commissioninquiry SESYNC teams developed new models and tools that inform decision-making for endangered species listings, and engaged with policy-makers regarding the use of ecosystem services assessments across US federal government.
Best practices for integrating ecosystem services into Federal decision-making: https://www.sesync.org/best-practices-forintegrating-ecosystem-services Endangered Species Act Decision-making: https://www.sesync.org/project/ventures/esadecision-making Decision-makers and stakeholders in Alaska, British Columbia and California used the Lenfest Ocean Programme's task force report on forage fish to inform how to more effectively manage these fisheries. Bednarek et al. 2015 EKLIPSE delivered an 'evaluation framework for nature-based solutions projects' that has been taken up by a number of EU projects. full framework: http://www.eklipsemechanism.eu/apps/Eklipse_data/website/EKLIPSE_ Report1-NBS_FINAL_Complete-08022017_LowRes_4Web.pdf; paper:  The UK NEA was used as a foundation for a Department of Environment, Food and Rural Affairs White Paper on the Natural Environment and then used to set up a Natural Capital Committee. Capacityknowledge use is dependent on the capacity to understand or apply knowledge in a given context SESYNC teams build new inter-and transdisciplinary communities to address specific socio-environmental problems and build a durable capacity for future efforts. Palmer et al (2016) The capacity-building work of IPBES, MA, SESYNC, CEE and EKLIPSE aims to broaden the capacity of experts to work in synthesis activities, and/or to use the results of research synthesis. http://www.environmentalevidence.org/trainingworkshops

Findings from case study review and literature
This exploratory review identified a number of insights about the variety of impacts from, and assumptions underpinning synthesis. Below we present our findings, drawing on case study data, the literature review and authors' personal experience.

Finding 1: The impacts of synthesis can be diffuse and difficult to trace
Attribution of policy or decision impacts to specific research synthesis projects or products is difficult, even when specific impacts are the goal. In most cases, policy change arises from a confluence of diverse research, political, legal and other factors. This is true even when research is embedded in a deliberate, specific, and long-term science-to-policy initiative.
Initiatives reviewed contributed to a range of impacts including: creating a new understanding of problems, establishing new networks, and contributing to changes in policy and practice. Other perhaps less tangible impacts focused around building individual and institutional capacity to conduct or use synthesis, raising awareness, building new knowledge and collaborations, or changing perceptions of the value of evidence in policy-making.
The UNGA resolution on illegal wildlife trafficking illustrates the challenges of impact attribution. Based on a literature review conducted by an in-house researcher, the WWF Hot House advocacy team provided government partners with draft text in policy negotiations, which substantially contributed to the resolution that was adopted by the UNGA. This impact would not have been evident without inside knowledge, as the WWF press release simply states that "WWF played a key role in the process by providing technical advice and advocacy support in the run up to its adoption, and the final version incorporates all of WWF's major asks." Such statements demonstrate the reluctance of some organisations to publicly state their role in influencing policy change as it may compromise relationships with key government partners. In such cases, the objective is to get the policy changed, not necessarily to attribute credit.
A more direct connection to a decision-making context, on the whole, enabled those initiatives to articulate tangible impacts on policy or practice. This finding is supported by the literature which suggests that identifying a specific policy audience and carefully considering the policy context is critical to impact (Oliver et al., 2014), and that research needs to be tailored to stakeholder needs (Ash et al., 2010;Cvitanovic et al., 2017;Morton & Seditas, 2016;Cash et al., 2003). This includes adapting research findings and implications to the audience in terms of both comprehensible language and physical accessibility to peer-reviewed literature or evidence (Balian et al., 2016;Cherney et al. 2015;Dicks et al., 2014).
Articulating impact is more challenging without a direct connection to a policy context. Entities focused primarily on producing knowledge do not have access to the 'stages' in which impact plays out. For example, CEE reports that their reviews are being used, and while they have not yet evaluated whether their use is changing how decisions are being made, they draw on evidence from other sectors (medicine, education, international development), suggesting that systematic reviews make a difference (see for example Pullin and Knight, 2001). SESYNC focuses on supporting research synthesis that yields high-impact publications in a non-advocacy context, but aims to increase relevance and impact through including individuals from government, non-governmental organisations and the private sector on synthesis teams.

Finding 2: There is a wide variety of assumptions underpinning the role of audience and engagement in the relationship between synthesis and impact.
The review identified a variety of assumptions about the connections between synthesis and impact, with the dominant assumptions relating to relationships and strategic engagement with the context of desired impact (see Table 1). This suggests that many initiatives are consciously thinking about the intended audience and use of synthesis, and, in varying degrees and ways, opening up the synthesis process to nonacademic actors. Generally the assumptions identified align with existing literature on the credibility, relevance, legitimacy (CRELE) criteria, which is unsurprising given their longevity as concepts within sustainability science. These criteria were embodied in the importance placed on engagement with science and policy communities in the process of designing and conducting synthesis, as well as efforts to target specific types of products to specific audiences. While the terminology of engagement varies (i.e. framed as 'relationships, ' 'involvement,' or 'partnership' with 'stakeholders' 'policy-makers' or 'decision-makers') there seems to be consensus across initiatives that end-users should be involved in the process of scoping and conducting research synthesis, not just receive information at the end.
IPBES, the MA and the UK NEA did not set out explicit assumptions about the specific nature of changes they seek beyond general statements like 'strengthening the science-policy interface', and each is framed as being 'policy relevant but not policy prescriptive'. However, they have implicit assumptions driving their existence, namely that the production of assessment reports and their subsequent use in decision-making, is an effective approach to catalysing political action on biodiversity loss, and that assessment reports are widely and consistently authoritative across the diverse audiences. In support of this, their conceptual frameworks articulate a range of assumptions about the relationships between biophysical and social systems that have had various impacts on the way these relationships are understood within science and policy communities.
With respect to involving 'end-users' in the synthesis process, there are differing assumptions about how such involvement improves the credibility of the work. The growing emphasis on 'knowledge coproduction' (iterative engagement with diverse knowledge holders to produce knowledge relevant to decision-making), was evident in many cases (eg SESYNC, SNAPP, Peatlands Inquiry). This approach rests on the assumption that more relevant, and therefore more useful knowledge will be produced if it is done so in collaboration with those who will use it. If this stakeholder engagement is not done sufficiently, or does not engage with multiple perspectives of stakeholders, there is a danger that a synthesis could be skewed to a particular perspective. To address such concerns, some cases have clear procedures to reduce potential bias that could be introduced through collaboration with end users. For example IPBES has clear operating principles, rules of procedure and open peer review, whereas CEE's method incorporates stakeholders perspectives in the design phase but the review is completed by academics.

Finding 3: Synthesis can influence policy and practice, however, in addition to compiling evidence, engagement with decision-makers and the context of use is important.
In all cases of impact, success was attributed to knowledge exchange that resulted from direct engagement with key stakeholders, decision-makers and other end-users throughout each stage of the synthesis process. Engagement consists of ensuring participation of all relevant actors (Biermann, 2002;OECD, 2015;Riousset et al., 2017;Rothman et al., 2009), whereby most successful initiatives involve an inherent partnership between researchers and stakeholders (the Peatlands Inquiry, SNAPP, SESYNC, see also Hedger et al., 2006;Lynch et al., 2015,) and/or are commissioned by at least some of the intended end-users (IPBES, UK NEA see also Ash et al., 2010;Hedger et al., 2006;Wilson et al., 2014).
Iterative and prolonged engagement builds trust, sustains dialogues, and builds shared goals (Angelstam et al., 2013;Campbell et al., 2015;Cherney et al., 2015;Elliot & Popay, 2000;Kowarsch et al., 2016). When all key stakeholders are active collaborators, their input should ensure the research is conducted at the appropriate scope and scale to address relevant questions (Campbell et al., 2015;Pullin et al., 2016). For example, the EKLIPSE synthesis process engages both the original requester (e.g., a policy body) and other relevant stakeholders in scoping, to identify the exact knowledge needs. The same is effectively true in scoping IPBES assessments. CEE actively promotes stakeholder engagement in the planning phase, although it advocates independence when conducting synthesis.
Our findings suggest that engagement processes are critical to the conduct of synthesis for impact. Assessment processes can facilitate learning by all participants, reduce gaps between science and policy, and lead to changes in behaviour, beliefs and overall discourse (Riousset et al., 2017;Kowarsch et al., 2016;Clark et al., 2006). The process produces multiple kinds of outcomes and impact realised throughout time, rather than simply when an assessment product is completed. This suggests that emphasis should be placed on the process of conducting synthesis rather than on specific outputs (Morton & Seditas, 2016;Posner et al., 2016;Reid & Mooney, 2016;Rissout et al., 2016). In this regard, sufficient resources (both time and staff) are critical to support knowledge exchange, brokering and networking to ensure that synthesis processes and products are effectively situated within a particular context (after Reed et al., 2017).

Finding 4: There are several models for impactful synthesis, with varying institutional arrangements, governance models, research and engagement processes.
There is a range of operating models and approaches for undertaking synthesis, and for connecting the synthesis to a decision-making context. These models cross a spectrum of long-term, targeted relationships among entities, to shorter-term working groups and focused commissions, to centres and platforms that focus on supporting evidence synthesis or global assessments. Different efforts are configured based on the context, funding sources and the expectations of outputs and impact from a given donor.
Below we list the range of characteristics identified across the cases. While some characteristics are presented as dichotomies, it is more appropriate to view these as a spectrum of differences: • Manner in which synthesis is called for, mandated or formally recognised; • Form of relationship between an initiative and a given use context; • Stages in which decision-makers, or end-users are engaged, and modes of engagement; • Whether the objective is to advocate for a specific outcome or to inform a decision-making process; • Role and level of engagement of non-academic experts in the production of synthesis; • Focal scale of an initiative, both in terms of the subject matter and spatial coverage; • Methods used to conduct synthesis; • Extent and transparency of peer review processes; • Staffing of the synthesis initiative, and their role in facilitating policy engagement; • Inclusion of diverse forms of knowledge and evidence in the synthesis.
The range of impacts found across diverse operating models suggests that there is no clear 'right' institutional arrangement for synthesis, or way to undertake and connect synthesis to a particular context. This finding resonates with the recent reviews conducted by Dicks et al. (2017) and Pullin et al. (2016) which suggest that most appropriate synthesis methods to apply are context dependent. Our analysis further demonstrates that variability and context dependence are also found in institutional arrangements and processes to connect synthesis to policy and practice.

Finding 5: Having a clearly defined context, audience and impact pathway can enable synthesis to target the appropriate scale and format of information.
Ensuring policy and practice impacts from synthesis requires a nuanced understanding of the policy-making process, and recognises the diffuse ways that research affects policy (Bauer et al., 2016;Elliott & Popay, 2000). In their review of the evidence of the use of research in decision-making, Langer et al. (2016) found that unstructured interventions, with a lack of conceptual and causal clarity about how evidence will be used in decision-making, are less effective at improving decision-makers' use of evidence. A focused subject matter and target audience can enable an initiative to build a coherent knowledge base and the long-term relationships or networks critical to impact. This was evident in many of the case studies. This includes not just understanding the knowledge gaps relating to a problem but how synthesis will be used in a particular policy context, and the constraints, barriers or enablers of the use of knowledge or evidence in that particular context.
A clearly defined context and audience can enable targeted impact. For example the UK Peatlands Inquiry built the impact pathway into the design of the synthesis through direct and ongoing knowledge exchange and coalition building with key stakeholders and policy actors. This included a synthesis of policy options and pathways, which was developed in tandem to draw on knowledge produced by other elements of the assessment. This enabled the policy community to understand their options for action, and their implications, rather than just being presented with a detailed description of the problem. For a body carrying out synthesis activities, this also means a substantial amount of work is required to properly identify the relevant context and audience of a new topic and consider it in the content and process of the synthesis.

Discussion
It is important to reiterate that reported impacts come from the experiential knowledge and perceptions of the authors who have been involved with the conduct of synthesis. Therefore, we are unable to make definitive claims about the nature of these impacts in policy and practice, and precisely how they were brought about. Rather these insights should be taken as provisional hypotheses that could be explored or tested in future research. The contribution therein lies in the analysis across a diverse range of cases, and the corroboration of similarities and differences within these cases and the literature. Our analysis extends existing work on the internal processes and methods of synthesis and knowledge integration (see Dicks et al. 2017;Pullin et al. 2016;Hoffmann et al. 2017), to consider how these are situated within a broader institutional context that influences the various impacts of synthesis. Moreover, much of the extant literature does not examine the underlying assumptions about the relationship between synthesis research processes and impact, and there is even less empirical analysis to test whether dominant assumptions are well founded.
This review surfaced some of the dominant assumptions about the relationship between research, policy and practice. We found a broad range of approaches producing various types of impacts, and no doubt there are other approaches we have not addressed. We are wary of producing a 'hierarchy of impacts' that identifies changes in policy as the ultimate aim. Rather, there is a need to be open to, and to value, a range of approaches and types of impact. This review indicates that the nature of impact, and where it occurs, be it on research, policy or practice, is diffuse and sometimes intangible. This reiterates findings commonly stated within the social science literature that challenge assumptions that there will be a rational and linear translation between research outputs and policy change. Indeed, the initiatives considered undertook a significant amount of 'work' to embed or shape synthesis within a particular context. However, it is not possible to demonstrate whether the findings reported here simply reinforce existing assumptions about the role of engagement in enabling impact in policy and practice.
The scale and complexity of a question and intended context of use of synthesis will influence what research methods and levels of engagement are appropriate. Also critical is the manner in which synthesis is intended to inform decision-making. In some settings the objective is to support a directional change towards a particular outcome, while in others the objective is to provide scientific evidence to inform a decision-making process without necessarily advocating for a particular outcome. There is a range of philosophical and ideological stances on what the role of science 'should be' in informing policy (see Pielke, 2007), and how independent a synthesis process should be for the results to produce robust science to inform policy (see Pullin et al., 2016). Each approach has its value, and the suitability of different approaches will be shaped by the decision-making context, as well as an individual's perspective or organisational mandate.
There is broad agreement across the literature of both barriers to and enablers of research uptake. Identified barriers include lack of relevant research, shortage of time or opportunity to use evidence, timing within the policy cycle, limited funding, and limited ability to understand research or to integrate knowledge from multiple contexts or disciplines (Balian et al., 2016;Elliott and Popay, 2000;McNie et al., 2016;Nutley, 2003;Oliver, 2014), and the professional academic rewards system (Born et al., 2009;Hampton and Parker, 2011;Lynch et al., 2015). Enablers include easy access to relevant research, a culture of research use, and sustained collaboration and relationships between policy-makers and researchers (Cherney et al., 2015;Elliott and Popay, 2000;McNie et al., 2016;Oliver et al., 2014). Other studies suggested that there is still only weak evidence of the outcomes and impacts produced by collaborative approaches to research engagement in policy (Voorberg et al., 2015;Breckon and Dodson, 2016). The diversity of barriers outlined here suggest that developing more and better evidence, and building more effective relationships with policy-makers will be insufficient in themselves (Hansson and Polk 2018). Many of these barriers stem from incentive structures and governance arrangements that undermine efforts to generate knowledge that is useful to and used in decision-making (van Kerkhoff, 2013;Clark et al., 2016). Consequently, it is naive to assume that efforts to produce synthesis will be effective, unless there is a focus on understanding the specific barriers to and enablers of the use of synthesis in a particular context.
There can be a tendency to oversimplify the policy process and its political nature, and to homogenise 'stakeholders' or 'decision-makers.' The literature on evidence-based decision-making oversimplifies the policy process and needs of policy-makers (after Head, 2016), with many studies examining barriers to and facilitators of evidence use focusing on a single element of the policy process (Oliver et al., 2014). A more general critique is the absence of a clear definition of 'stakeholders' or 'decision-makers' (e.g. Hedger et al., 2006), and neglect of the heterogeneity of policy-making contexts and associated barriers and enablers of up-take (Lynch et al., 2015). Some literature also uses language about 'policy-makers,' yet provides examples that support the use of research in relatively tractable management decisions, potentially conflating complex policy-making with less-complicated management contexts (e.g. Dicks et al., 2014b).
This suggests that to be effective it is important to ensure that synthesis processes consider all aspects of the policy or decision-making process being addressed, and the full range of relevant stakeholders.
The nature of the influence on policy will vary depending on the stages of the policy process and the scale (local, regional, national, global) at which it is intended to inform. While policy processes are generally recognised to be non-linear and complex, it can be useful to characterise different stages, from agenda setting, policy formulation, legitimation, implementation and evaluation to policy maintenance, succession or termination (Cairney, 2016). Across these different stages, the role of research (synthesis or otherwise) differs, for example, at the agenda-setting stage, research is more likely to support on discovery or awareness raising, while at evaluation stages it is more likely to support an appraisal of what has worked or has not (see Vogel et al. 2007).
The impacts reported here were largely confined to the experiential knowledge of the authors. Gaps in the literature likely relate to the challenges of impact attribution, and the fact that for the research community as a whole, measuring impact is a relatively new endeavour. One avenue for future research would be to come at this review from the other end -i.e. look for evidence of policy change, or at decisions in intergovernmental bodies, and search within these for an indication that they have been informed by research synthesis (see for example Boyd and Kramer, 2017). The impact case studies compiled for the UK Research Excellence Framework and the Excellence in Research Australia process may offer a fruitful starting point for a future comparative analysis of the mechanisms underpinning impact. Further analysis could cross-check our findings with a wider sample of people involved in these initiatives. A focus on those engaged with the use of synthesis will provide critical insight into whether or not a particular initiative had an impact in a given context, and ask the critical question as to whether the nature of the research as 'synthesis' inherently increased its utility. Understanding such perspectives is vital to improve the design of synthesis for impact, as their perspectives on the value of synthesis research and its role in decision-making are likely to be very different from those of technical experts.

Conclusion
This exploratory review found a range of impacts from research synthesis and assumptions underpinning research synthesis and impact. The synthesis initiatives examined have contributed to change in all three domains of research, policy and practice, including changes in understanding, new relationships, networks, changes in policy and practice, and improvements in capacity. We found that: • The impacts of synthesis can be diffuse and difficult to trace • There is a wide variety of assumptions underpinning the role of audience and engagement in the relationship between synthesis and impact • Synthesis can influence policy and practice, however, in addition to compiling evidence, engagement with decision-makers and the context of use is important • There are several models for impactful synthesis, with varying institutional arrangements, governance models, research and engagement processes • Having a clearly defined context, audience and impact pathway can enable synthesis to target the appropriate scale and format of information The categorisation of impacts used to support this analysis provides a framework through which to understand and interrogate the various types of impacts synthesis research is having, and also the assumptions underpinning the relationship between synthesis and impact. This will enable those involved in the design of research synthesis to be more explicit about the intended role and contribution of synthesis to change, and will further enable an examination of whether these assumptions are valid as an initiative progresses. Moreover, being explicit about these assumptions from the outset will enable an initiative to consider the range of institutional support and capacities needed to realise changes in policy and practice.
A variety of strategies emerged to improve impact in policy and practice, the most prominent being engagement of 'decision-makers' or intended 'users' throughout the design and conduct of synthesis. Potential modes of engagement varied widely ranging from participation in research synthesis to targeted consultations, as did methods for conducting synthesis, governance models and enabling environments of synthesis. While we found consistent features that enabled impact -particularly direct connection to a policy context, and iterative engagement with the intended users of the synthesis -we did not find a consistent pattern of institutional arrangements for the synthesis processes themselves that was more or less likely to lead to impact. This suggests that there is no one institutional configuration in which to undertake synthesis.
Most of the initiatives reviewed had very nuanced narratives about how the effort had been designed specifically for a policy context, yet all suggested that their impact was not assured, and required delicate navigation of processes which often unfold over long timeframes. This highlights the need to understand and engage with the processes and contexts in which the results of synthesis will be used, including a systematic assessment of when, how, and in response to what types of knowledge, decisions are made.
There is no 'silver bullet' for impact in the complex arenas of policy and practice, and it is best to acknowledge this from the outset and tailor processes accordingly.