Is themethodology used in reviews of restoration outcomes reliable? A systematic map protocol

1 Laboratory of Ecology and Forest Restoration (LERF), Department of Forest Sciences, Luiz de Queiroz’ College of Agriculture, University of São Paulo, Piracicaba, Brazil 2 Secretariat for EnvironmentalManagement and Sustainability (SGAS), Federal University of São Carlos, São Carlos, Brazil 3 Laboratory of Tropical Forestry (LASTROP), Department of Forest Sciences, Luiz de Queiroz’ College of Agriculture, University of São Paulo, Piracicaba, Brazil

practitioners to identify critical points where restoration evidence syntheses must be improved to move forward.

K E Y W O R D S
evidence-based restoration, reproducibility, restoration ecology, restoration reviews, review methodology INTRODUCTION

Background
The relentless drive for economic growth and the increase in global population has resulted in several societal challenges and environmental degradation such as climate change, food and water security, social equity, human health, and socioeconomic development, biodiversity loss and pollution (Cohen-Shacham, Walters, Janzen, & Maginnis, 2016;WWF, 2016). These unprecedented challenges have pressed the field of restoration ecology and conservation biology to re-envision the way they set goals, and how they manage ecosystems to conserve bio- Traditionally, observations and field experiments have been the main way of obtaining knowledge about empirical patterns in ecology and ecosystem sciences (Cadotte, Mehrkens, & Menge, 2012). Primary studies provide vital insights into the real-world application of a specific intervention or conservation strategy under particular conditions (O'Leary et al., 2016), but they can be limited in time and spatial scale (Wortley, Hero, & Howes, 2013), designated as 'provincial case studies' (Cadotte et al., 2012;Lawton, 1999).
The increasing number of published primary studies has resulted in ever-increasing evidence of variable quality for decision makers to draw from Li and Zhao (2015), O'Leary et al. (2016) andPautasso (2012). Specifically, the scientific field of restoration ecology is facing a considerable expanding volume of primary research (Guan, Kang, & Liu, 2018;Romanelli, Fujimoto, Ferreira, & Milanez, 2018), reaching a point where is necessary to apply effective and well-documented strategies to produce reliable synthesis (Fazey, Salisbury, Lindenmayer, Maindonald, & Douglas, 2004;Lawler et al., 2006;Brancalion &  Evidence syntheses may substantially boost the effective power of individual studies (Cooke et al., 2018;Stewart, 2010). Here we use the concept of 'evidence synthesis' to describe the whole methodology used to gather and collate evidence, for instance, systematic review or systematic maps (James, Randall, & Haddaway, 2016 Unsurprisingly, there are numerous examples of reviews reporting conflicting results (Berlin & Golub, 2014). In the broad field of conservation and environmental sciences, most reviews are non-systematic and exhibit a diversity of methods, objectives and approaches to reporting evidence (Woodcock et al., 2014).
While it is known that environmental reviews can substantially improve in quality (Diefenderfer et al., 2016;Grames & Elphick, 2020), the methodological reliability and reproducibility of restoration reviews remain poorly investigated. As a consequence, restoration practitioners must be prepared to critically evaluate each relevant literature synthesis that may be used in decision-making and policyrelevant questions (Cooke et al., 2018;Pullin & Knight, 2012;Woodcock et al., 2014).
The variation in the methodological rigour of restoration reviews may lead to inaccurate conclusions and biased results (Lajeunesse & Forbes, 2003;Reid, Fagan, & Zahawi, 2018;Stewart, 2010;Whittaker, 2010). For example, in the restoration sector, several recent metaanalyses aimed to determine whether natural regeneration is more effective at recovering tropical forests than active restoration ( Given such problems, the systematic map protocol proposed here will yield a useful overview of systematic review principles for various restoration practitioners. Moreover, the resulting paper will help researchers and decision makers to identify critical points where restoration evidence reviews must be improved to move forward. We will apply the Collaboration for Environmental Evidence Assessment Tool -CEESAT -to assess the methodological reliability (in terms of objectivity, comprehensiveness and transparency) of restoration reviews (see definitions in Table S1; Appendix A).

Primary question
• How reliable is the review methodology across restoration evidence syntheses addressing terrestrial ecosystems?

Question components
The question components were structured according to the PICOstructure (population, intervention, comparator and outcome), according to Collaboration for Environmental Evidence (Pullin et al. 2018).
• Population: Reviews should be a synthesis of primary research, being described by their authors (in title, abstract, or keywords) as a systematic review or a meta-analysis. • Intervention: Reviews should address the ecological restoration of terrestrial ecosystems.
• Comparator: Reviews published by the CEE journal Environmental Evidence.
• Outcomes: Any outcome will be eligible for inclusion.

Secondary questions
• What is the level of confidence that restoration practitioners can place in restoration reviews based on information reported within each study, through CEESAT? • What is the relationship between SCImago Journal Rank (SJR) and reliability scores?
• Are journal impact factor and more reliable reviews correlated?
• What is the relationship between reliability scores and synthesis type (narrative synthesis vs. meta-analysis)?
• How reproducible is the evidence review methodology in restoration ecology?
• How restoration reviews can improve in reliability and reproducibility?

Generic questions
• What are the main research topics addressed among restoration reviews?
• What are the main bibliographic sources reported in restoration reviews? • Are discrepant the review concepts among restoration authors?

Searching for articles
Restoration ecology is the scientific field that supports the practice of ecological restoration (Aradottir & Hagen, 2013;Romanelli et al., 2018); thus, we will use the terms ('ecological restoration' and 'restoration ecology') to retrieve titles, abstracts and keywords of related publications. We will also combine the above-mentioned terms with ("metaanalys*" OR "meta analys*" OR "metaanalys*" OR "metanalys*" OR "systematic* review*") to define our population of reviews. We have selected only two terms to retrieve publications related to ecological restoration. Therefore, the total number of all available evidence may be underestimated. Documents described with other terms (e.g. forest restoration, ecological rehabilitation, ecosystem reclamation or habitat restoration) could also be relevant for analysis. For clarity, terminology related to evidence synthesis used in this paper is defined in Table   S1 (Appendix A in the Supporting Information).
We have selected four widely used bibliographic sources in the field of conservation biology and environmental sciences to test our pre-established questions: (i) Web of Science (WoS Core Collection: SCI-E, SSCI, ESCI); (ii) Scopus, (iii) CAB Direct and (iv) SciELO; we also used Google Scholar as a search engine. We chose these bibliographic sources because they are subject of several comparative studies (e.g. Harzing & Alakangas, 2016;Meho & Yang, 2007) and are often used to evaluate researcher's productivity and whole disciplines, as well as undertake bibliometric analysis and evidence reviews (e.g. Harzing & van der Wal, 2008;Jacsó, 2011;Côté, Curtis, Rothstein, & Stewart, 2013). We will include the term 'forest restoration' to restrict the search on Google Scholar to exclude the most unwanted documents.

Search string
The final search string is shown in Table 1.

Grey literature
For our analysis only peer-reviewed articles that are relevant are selected.

Languages
Searches in bibliographic databases will be conducted only in English using the above-mentioned search string. However, we will consider TA B L E 1 Search strategy for locating reviews

Article screening and study eligibility criteria
The review team conducted a scoping exercise to assess the effectiveness of search terms (Table 1), testing them against a set of about 20 articles known to be relevant. All retrieved reviews will be first screened on their title and keywords to remove irrelevant documents.
Posteriorly, each retained paper will be screened for relevance on the basis of the abstract.
We will use the following inclusion criteria at the full-text screening stage: (i) reviews should be described by their authors (in the title, abstract or keywords) as a systematic review or a meta-analysis -although this searching strategy is partly dependent on how the authors describe the review, our purpose is to detect possible misuses of review concepts; (ii) reviews should be directly related to ecological restoration of terrestrial ecosystems; and (iii) reviews should be a synthesis of primary research.
Studies that do not match our research question or inclusion criteria will be excluded at the title, abstract or full-text levels. Reasons for exclusion will be provided for all articles excluded at the full-text assessment. A second and third assessor will perform the same screening process for a sample of ∼20% of the retrieved papers on the full-text screening process and Cohen's kappa will be used to check for inclusion consistency. Cohen's kappa coefficient is a statistic that is used to measure inter-rater and intra-rater reliability for qualitative (categorical) items (Cohen, 1960). If the kappa score will reach < 0.6, the inconsistencies among the reviewers will be discussed and the inclusion criteria possibly redefined. We will list all excluded papers to ensure transparency (Pullin et al. 2018). The list will be provided in an additional file (in the Supporting Information) along with the reasons for exclusion. If inclusion consistency is met, the main reviewer will finish the screening with the remaining articles.

Study validity assessment
We will not be undertaking a conventional study validity assessment of studies since we will perform the methodological assessment of reviews through CEESAT criteria. In this stage, all documents that fit our eligibility criteria will be considered for analysis.

Data coding and extraction strategy
Regarding the primary question (i.e. the reliability assessments) selected publications will be evaluated according to the CEESAT (O'Leary et al., 2016;Woodcock et al., 2014). CEESAT consists of a set of 13 criteria projected in alignment with environmental systematic review methodology (O'Leary et al., 2016;Pullin et al., 2018). A random selection of 20 screened articles will be double scored by two reviewers (scorers). We will analyse the scoring decisions between scorer 1 and scorers 2 and 3 by considering the magnitude of disagreement between scorers using a weighted kappa test of agreement. Potential disagreements will be discussed until a consensus is reached before continuing with the full list of screen articles.
In this stage, the key variables that will be analysed based on the content of reviews include: • Presence or absence of an a priori protocol, • Analysis of the use of a comprehensive range of bibliographic sources, • Analysis of repeatability and transparency of search strings, • Analysis of inclusion criteria to all potentially relevant studies, • Repeatability of inclusion/exclusion decisions, • Transparency of inclusion/exclusion studies decisions, • Assessment on critical appraisals of the methods of each study, • Analysis of objectively, according to the methodological quality, • Repeatability and consistency in data extraction, • Analysis of quantitative synthesis, • Analysis of the heterogeneity in the effect of the intervention/exposure and • Consideration of publication bias.
Each article will be coded with keywords and expanded comments fields describing various aspects of the reviews. These keyword and comment fields were developed among members of the Laboratory of Ecology and Forest Restoration (LERF) at the University of São Paulo and are designed to prove the summary information required to answer generic questions. We will create bibliometric network maps with the VOSviewer software (version 1.16.15), using the text mining functionality to construct and visualize co-occurrences of keywords, and then to present trends across the included literature. VOSviewer is a software tool expressly designed for the analysis of bibliometric data (van Eck & Waltman, 2010).
In this stage, key variables will include: • Study type (narrative synthesis or meta-analysis), • Research topics according to the synthesis type.
As far as possible, controlled vocabulary will be employed to code the variables using thesaurus or concepts employed in academic reporting. We will use the Guidelines and Standards for Evidence Synthesis in Environmental Management (Koricheva and Gurevitch 2014;Pullin et al., 2018) and Koricheva, Gurevitch, and Mengersen (2013) to categorize concepts and align reviews terminologies.

Data synthesis and presentation
The resulting paper will include summary figures and tables of the CEESAT results that cover specific points for review improvements.
Concerning our primary question, we will not provide a list of included reviews that will compose our database and either the individual scores obtained by the analysis of each one. We are focused on the methodological quality of restoration reviews to identify strengths and weaknesses in this literature body and offer guidelines for the improvement of future review works, not to criticize individual papers. Nonetheless, we ensure transparency and repeatability of our search strings so other researchers can access our retrieved information. Furthermore, we will provide the complete results of all repeatability tests (kappa test) for inclusion/exclusion decisions and CEESAT criteria assessment. We will discuss any point that needs to be improved in restoration syntheses methodology by providing groups of scores obtained through CEESAT.
We will also discuss the main causes of disagreements about the inclusion/exclusion of articles and the attribution of scores through CEESAT criteria.
Concerning generic issues, at the data-gathering stage, we will provide an overlap analysis from the bibliographic sources' information (percentage of duplications among these sources) to demonstrate the proportion to which each contributed to the initial retrieval of publications. We will also provide graphs of network analyses based on the keywords co-occurrence criterion to assess which are the main research topics covered by restoration reviews, considering both review types separately. From these results, restoration practitioners will be able to check the recent status of research involving restoration syntheses and seek to develop fields that are still little explored.
This joint analysis of network analyses and CEESAT scores recommendations will allow us to reveal future research topics on which the most robust studies may focus. The number of articles by publication year and important mark points in the literature body (e.g. the establishment of concepts such as 'evidence-based restoration') will be provided in graphs.
We will use descriptive statistics (median, mode and mean) to enable comparisons between synthesis types. Differences in the mean scores obtained by each review assessment and assigned to different categories (narrative syntheses vs. meta-analyses) will also be tested statistically. A Mann-Whitney U test may be applied for pairwise comparisons. Pearson's correlation coefficient may be used to analyse the relationship between journal impact factor (SJR) and CEESAT scores. Network analysis based on the co-occurrence of the author's keywords will be developed using VOSviewer software. The entire protocol complies with the ROSES reporting standards (see Appendix B in the Supporting Information).

Limitations
Systematic reviews and systematic maps must have, as one of the main tenets of their methodologies, comprehensive search strategies (Haddaway, 2017;Pullin et al. 2018), to capture as much of the relevant evidence to the synthesis as possible (Abdulla, Abdulla, Krishnamurthy, & Krishnamurthy, 2016;Bayliss & Beyer, 2015). Reviews according to the CEE standard search a mean of nine bibliographic databases (±8.2 SD) and 18 grey literature sources (±15.9 SD) (Haddaway, 2017). Here, we place our search strategy by using three databases into the Web of Science platform, besides CAB Direct, SciELO, Scopus and Google Scholar as a search engine. We will not search for grey literature because they are not relevant to answer our review question.

DISCUSSION
The main objective of this review is to gather restoration evidence syntheses published across environmental journals to assess their methodological reliability and reproducibility, and therefore the confidence that end users can place in the conclusions of these syntheses. To better serve the environmental policy and management communities, this assessment can enable researchers and decision makers to select reviews that are most likely to accurately reflect the evidence base.