Large scoping reviews: managing volume and potential chaos in a pool of evidence sources

56 Scoping reviews can identify a large number of evidence sources. This commentary 57 describes and provides guidance on planning, conducting, and reporting large scoping 58 reviews. 59 This guidance is informed by experts in scoping review methodology, including JBI Scoping 60 Review Methodology group members, who have also conducted and reported large scoping 61 reviews. 62 We propose a working definition for large scoping reviews that includes approximately 100 63 sources of evidence but must also consider the volume of data to be extracted, the 64 complexity of the analyses and purpose. We pose six core questions for scoping review 65 authors to consider when planning, developing, conducting, and reporting large scoping 66 reviews. By considering and addressing these questions, scoping review authors might 67 better streamline and manage the conduct and reporting of large scoping reviews from the 68 planning to publishing stage. 69 70

Scoping reviews often contain many varied evidence sources which can be problematic for the review team in terms of resource and data management constraints.
Planning for a scoping review is a fundamental first step for reviewers and consideration of the likelihood of identifying large numbers of evidence sources needs to be discussed by the review team and factored into the protocol.However, no specific guidance currently exists for teams planning to undertake large scoping reviews or teams who encounter an unexpectedly large volume of evidence sources once the review has commenced.As such, this article was developed by a group consisting of expert methodologists in scoping reviews, including members of the JBI Scoping Review Methodology group, who have conducted large scoping reviews [8][9][10][11][12] .The below guidance is based on our experience and is intended to aid other researchers regarding how to undertake a large scoping review.

What is a large scoping review?
There is no current definition on what constitutes a "large" scoping review.In their review of scoping review conduct and reporting, Tricco and colleagues (2016) 12 reported a mean of 118 included studies (range 1 to 2600, median 46) across 494 identified scoping reviews.
More recently, a search of two journals by this group, JBI Evidence Synthesis and Systematic Reviews, identified 53 scoping reviews that had been published between January 2021 and November 2022, with an average of 66 included studies (range 3 to 638, median 31).Only six (11%) scoping reviews included over 100 evidence sources in their final analysis.From this evidence and the experiences of the authors, a large scoping review is open to definition.For this article, we propose a working definition of a large scoping review as "a scoping review that includes approximately 100 sources of evidence.This will be dependent on not only the number of included studies but also the volume of data to be extracted and the complexity of the analyses and the purpose (e.g., only descriptive J o u r n a l P r e -p r o o f statistics or a more time-consuming qualitative content analysis or conceptual framework development)".

Benefits and Challenges of large scoping reviews.
Large scoping reviews can make a valuable contribution to understanding broad evidence in a given field by enabling mapping of a whole body of evidence, characterising its strengths and limitations, and identifying future research priorities 10 .They can also be a useful precursor to subsequent systematic reviews, ensuring the review questions are appropriate and can be addressed by the evidence-base 8 .Despite their value, authors of large scoping reviews face several challenges.Management of a large amount of evidence sources contributes to increased workload during each step of the review, and factors to consider include human and non-human (e.g., software) resource implications, ensuring sufficient consistency and reliability checks, robust data management processes, managing version control, and attempting to avoid delays in review completion and publication.
When planning a scoping review, authors may not be fully aware of the potential volume of evidence sources that may be identified.Therefore practical guidance can be helpful for authors to successfully plan and complete large scoping reviews in a methodologically rigorous and timely manner.This article will explore and discuss practical steps that we have utilised in planning, managing, and conducting large scoping reviews.Although this article focusses on scoping reviews that identify and include large numbers of information sources, the guidance is applicable to all scoping reviews regardless of size.

Preparing for a large scoping review
Preparing for a potentially large scoping review is just the same as preparing for any scoping review or evidence synthesis, regardless of potential size or scope.Some initial questions to be considered by authors are presented in Box 1 and are discussed below.These questions can apply to all evidence synthesis approaches, but the focus here relates to the concept of large scoping reviews.Time spent planning the review and creating a robust protocol 13 J o u r n a l P r e -p r o o f should identify the likelihood of a large volume of evidence being identified.Authors can then consider whether they have the appropriate resources and can develop a search strategy to answer the review question(s) that includes all eligible evidence sources (i.e.published peerreviewed literature and unpublished grey literature), within a relevant time frame.Other considerations include the use of crowd sourcing, sharing the work between teams, and the use of automation technology to increase the efficiency of the process 14,15 .
Questions for authors to consider when planning a scoping review 1.Who do I need to include as part of the review team? 2. Can I be more specific in my question(s) and search strategy to reduce the number of evidence sources that can be included while still answering the original review question?3. How can I best manage the review?4. How will I effectively manage the data extraction and analysis stage? 5. How can I best present the included data to answer the review question/s? 6.How and where will I publish this scoping review?Box 1: Questions to ask when preparing for a scoping review

Who do I need to include as part of the review team?
A scoping review team can include a range of individuals with different expertise such as methodologists, topic experts, information specialists, and knowledge users (such as patient partners, policymakers, non-government representatives etc) 16 .Besides bringing their specific expertise to the table, most of the team members will have a dual role conducting screening, extraction, analysis of the data, and contributing to the generation of the final review reporting.All scoping review authors, regardless of the review size, should consider what the best review team would look like in terms of specific expertise as well as having enough individuals to conduct and complete the scoping review.Operationally, more individuals will be required for a large or complex review to be completed in a timely manner.
Alternatively, reviewers need to acknowledge that the review will take much longer if there are only two reviewers for each step.The 'rule of thumb' that suggests most evidence J o u r n a l P r e -p r o o f 8 syntheses might be considered out of date by journal editors if the search was conducted more than 12 months before submission has been challenged dependent on topic area and evidence synthesis approach 17 .This may be applicable to large complex scoping reviews; therefore, authors may need to consider approaching journal editors before submission to clarify their position on the "rule of thumb".Authors may also need to consider if a search update is warranted and how to include it prior to submission to ensure that the scoping review is still relevant.Reviewers may also wish to consider specific individuals to lead on each step of a large review to aid data management and transparent recording.
Knowledge users should be engaged in all evidence synthesis approaches including scoping reviews 16 .Authors should consider identifying all potential knowledge users at the beginning of the scoping review to identify and recruit knowledge users to the review team, or steering committee (if needed) before developing the protocol.In a large scoping review, knowledge user inclusion can help focus the review, provide insight to guide review decisions (potentially saving time and improving comprehensiveness of the results) and support relevant research recommendation development and prioritisation.

Can I be more specific in my question(s) and search strategy to reduce the number of evidence sources that can be included while still answering the original review question?
The development of the initial scoping review question(s) can guide the breadth of the review.However, as scoping reviews are exploratory in nature 7 with iterative processes, the full breadth of relevant literature may not initially be apparent.Development of the question(s) is a key element that must be considered, discussed, and refined by the review team.From one specific overall review question, there can be specific sub-questions or objectives developed that will operationalise the scoping review to ensure the required evidence is included, analysed, and presented.This helps to provide structure and focus within a large scoping review.For example, a recent scoping review aiming to map the evidence on exercise therapy for the treatment of tendinopathy was broken up into two J o u r n a l P r e -p r o o f questions 8 : 1) Which exercise interventions had been reported for which tendinopathies? and 2) Which outcomes had been reported in studies investigating exercise interventions for tendinopathies?Another example of a large scoping review mapped the palliative care research undertaken in Australia during an 18-year period 18 .The review question was operationalised via sub-questions including types of studies conducted, topics researched, population involved, and gaps and areas of need identified for the future.
In theory, when developing a search strategy, preliminary screening of the literature should provide an indication of the available evidence and expected resources required.However, due to the exploratory nature of scoping reviews, you may find an unexpectedly large scoping review.The inclusion of grey literature (i.e., difficult to locate or unpublished material) in addition to peer-reviewed evidence, which is typical in a scoping review 12 , can also increase the number of eligible sources.Options here could be to review the protocol and identify justified deviations which you report in the final review.For example, reducing the date range to 2020 onwards or to the last five instead of 10-years for relevance.
However, reducing the date range could be considered a potential limitation, dependent on the review question.Another is to refine your scoping review question(s) and objectives/sub-questions, as there may be too many or too broad, so you then prioritise and re-focus.
Searching for and screening articles can be time consuming.Initial use of resources during scoping review protocol development to conduct preliminary searches (in collaboration with information scientists) to understand capacity and breadth of the literature is beneficial.It is helpful to discuss and identify which grey literature sources will be included and what management approach will be used to identify resource requirements.Reviewing reference lists of included studies for additional articles can be challenging with large numbers of included sources, therefore use of citation analysis tools (e.g.Web of Science, Scopus) to track citations (retrospective and prospective) can be an alternative for managing this step.
J o u r n a l P r e -p r o o f 10

How can I best manage the review?
For all scoping reviews, management of the process and resources required need to be considered including: Do I have the available resources, time, and experience to complete this scoping review?i.e., dedicated time, an involved team, methodological support, topic experts, knowledge users, financial resources, and access to software to support the review process.Do I need to use crowd sourcing, sharing work between centres or automation technology to ensure my review is completed on time and within budget?
There is no optimum sized team when dealing with a large scoping review.In our experience, a good team is one that is engaged and committed to completion of the scoping review.It is helpful to have an experienced team to assist with data extraction and analysis, typically the most complex and biggest time commitment for large scoping reviews.It is imperative teams meet regularly and during screening, and data extraction stages, all team members involved undergo extensive piloting.Training of the team (topic, methodology, and software training) and piloting of tools should be factored into the review timeline appropriately.This will allow opportunities to troubleshoot issues that occur earlier rather than having to re-do work at a later stage.Other tips we have found to assist a cohesive team environment when managing large scoping reviews are: 1. Discuss roles (are they involved in all parts of the scoping review, or are they assisting in certain parts e.g., screening or extraction dependent on skills?), expectations (how many meetings?Expected time commitment per week over how many months?and authorship) in preliminary meetings.Allocate senior reviewers to lead screening or extraction to share responsibility.

Regular meetings and core communication (online chat via MS Teams or
Slack can support regular communication) to discuss the progress of each stage (who will be responsible for sending team updates?How frequently will you update knowledge users?What information will you provide?Who is the J o u r n a l P r e -p r o o f main contact for knowledge users?) using progress update templates (such as https://sporevidencealliance.ca/resources-for-research-teams/).
3. Developing tools and resources/guidance for using the tools easily and consistently (such as a detailed extraction guidance tab on Excel/Covidence) 4. Be gracious with your team!It is not possible to suggest how long a scoping review can or should take.Our team has found that large scoping reviews can still be completed in a timely fashion if there are dedicated human resources to complete the screening and extraction stages.
Planning for tasks such as data screening and extraction is essential , especially with large numbers of sources and complex data extraction and analysis plans.Automation tools such as Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia) can greatly assist these stages provided team members are trained in using them and they are freely available through the researchers' institutions 20 .Other options include the use of crowd sourcing or sharing work between centres.Minimum software requirements include a reliable reference management system; however, several software packages (such as Excel, Google sheets and NVivo) are also available that can make a large scoping review more manageable at different stages.

How can I ensure data extraction and analysis are manageable?
Planning for what type of data you will extract and how it aligns to the review question is essential when managing large amounts of evidence sources as well as considering the complexity of the planned analyses.How you extract data can influence the analysis and ultimately how long it will take to complete a scoping review.
There are some strategies to assist the management and organisation of scoping reviews at this stage.
• Choosing an appropriate software program that allows team members to extract (even in different time zones) without issues with version control of documents.From J o u r n a l P r e -p r o o f our experience Covidence (Veritas Health Innovation, Melbourne, Australia) or Excel (using a shared drive) have worked well, however there are other programs available.
• Ensuring responses (if possible) are closed-ended on the data extraction tool.Ideally creating a list of possible responses where team members can choose rather than each team member individually writing a response is easier for the team member and aids consistency for analysis.
• Deciding the number of team members required to extract per evidence source is an important decision in the pre-planning stages.Ideally each evidence source would be independently extracted by two individual team members.However, this may not be feasible in large scoping reviews, and you may only have an evidence source extracted once with another person checking over extraction.If the latter occurs there is an increased chance of error in your extraction, and this must be noted as a limitation of your review.Piloting of data extraction among the review team is essential and will mitigate for some of this limitation.
• Deciding to abstract only high-level data without presenting any detailed results.
• If you are working with knowledge users who require the scoping review findings in a tight timeline for decision-making, you may wish to undertake a rapid review instead of a standard scoping review.However, any shortcuts need to be clearly described and the results interpreted with caution 19 .
• For some reviews, it may be appropriate to allocate data extraction to different teams related to specific review objectives.For example, in one review of health worker roles, data extraction using NVivo (NVivo V.12 Pro Software QSR) was initially undertaken by separate teams for roles in acute settings, primary care and emergency care 20 .The results were then integrated in the final analysis.Challenges to this approach are ensuring all teams are interpreting data extraction criteria in the J o u r n a l P r e -p r o o f same way and extracting the same level and type of information.Hence, regular team meetings, piloting and ongoing quality checking is essential.
The data extraction stage can also be an iterative process meaning you may have a situation where you are extracting from sources and realise that you may need to extract another variable.It is important to recognise this as early as possible to avoid having to reextract from a large number of sources.

How can I best present the included data to answer the review question?
To date, the results of scoping reviews have been typically presented in tabular form.Whilst tables are particularly useful in the organisation of data, they are not friendly for readers of large scoping reviews due to their size.Additionally, within a large scoping review, authors need to consider how best to present the results to answer the review question(s) whilst avoiding large unmanageable "death by tables" 2 .Presentation of scoping review results, regardless of size, should be considered and documented in the protocol.Innovative ways of presenting data from large scoping reviews include box plots, bar graphs, scatter plots, funnel plots, pie charts, mind maps, tree maps, word clouds, geographic maps, dot plots, histograms, schematic representations, and bubble plots 8,21 .Software such as Excel, NVivo, Eppi-viewer, and statistical packages including R and SPSS can be used to create these presentations.

How and where will I publish this scoping review?
It can be challenging to find a journal that will publish a large scoping review.Full transparent reporting in congruence with the PRISMA-ScR 5 within the standard word count for applied health journals is tricky.Thus, it is important to plan a priori how data will be presented.Scoping review teams should consider strategies to: • Manage referencesthe excluded studies list (with reasons for exclusion) as well as the included studies list can be presented as supplementary files.The included J o u r n a l P r e -p r o o f studies can lead to a very large reference list in an article.Therefore, additional thought on how to manage that is required and will depend on journal guidelines.
• Choose journals that utilise numbered referencing conventions.This will make the review easier to read and create manuscripts with fewer words.Alternatively, contacting journal editors and describing how the included studies have been numbered to aid readability can be adopted.
• Supplementary files can support a transparent reporting process with additional files accessed online.File type requirements of specific journals need to be considered.If the journal limits supplementary files, online registries, such as Open Science Framework, can be used.
• Interactive tables and online resources are another way to openly share results.However, teams need to consider the cost and maintenance of these over time as well as the initial skills and cost for creating them.
• Approaching journal editors prior to submission can help review teams to navigate these challenges and identify feasible solutions to increase the likelihood of publication.

Conclusion
Large scoping reviews are not technically more difficult than smaller/standard scoping reviews as both will be conducted according to recognised methodology.They simply present a different challenge in terms of volume, complexity, project management, and the related points raised here.As scoping review methodology continues to progress and more scoping reviews are published, it is hoped the questions discussed in this article can support future scoping review authors.Consideration of the above questions in the preparation or conduct of large scoping reviews will help scoping review authors to ensure the review can be conducted and delivered within the timeframe required and to a high standard of rigor.
J o u r n a l P r e -p r o o f