Guidance on review type selection for health technology assessments: key factors and considerations for deciding when to conduct a de novo systematic review, an update of a systematic review, or an overview of systematic reviews

Background A systematic review (SR) helps us make sense of a body of research while minimizing bias and is routinely conducted to evaluate intervention effects in a health technology assessment (HTA). In addition to the traditional de novo SR, which combines the results of multiple primary studies, there are alternative review types that use systematic methods and leverage existing SRs, namely updates of SRs and overviews of SRs. This paper shares guidance that can be used to select the most appropriate review type to conduct when evaluating intervention effects in an HTA, with a goal to leverage existing SRs and reduce research waste where possible. Process We identified key factors and considerations that can inform the process of deciding to conduct one review type over the others to answer a research question and organized them into guidance comprising a summary and a corresponding flowchart. This work consisted of three steps. First, a guidance document was drafted by methodologists from two Canadian HTA agencies based on their experience. Next, the draft guidance was supplemented with a literature review. Lastly, broader feedback from HTA researchers across Canada was sought and incorporated into the final guidance. Insights Nine key factors and six considerations were identified to help reviewers select the most appropriate review type to conduct. These fell into one of two categories: the evidentiary needs of the planned review (i.e., to understand the scope, objective, and analytic approach required for the review) and the state of the existing literature (i.e., to know the available literature in terms of its relevance, quality, comprehensiveness, currency, and findings). The accompanying flowchart, which can be used as a decision tool, demonstrates the interdependency between many of the key factors and considerations and aims to balance the potential benefits and challenges of leveraging existing SRs instead of primary study reports. Conclusions Selecting the most appropriate review type to conduct when evaluating intervention effects in an HTA requires a myriad of factors to be considered. We hope this guidance adds clarity to the many competing considerations when deciding which review type to conduct and facilitates that decision-making process.


Background
A systematic review (SR) helps us make sense of a body of research, while minimizing bias, by identifying, analysing, appraising, synthesizing, and interpreting research findings using a standardized, pre-defined method and an a priori protocol [1]. In the context of health care, the SR methodology provides an essential tool for summarizing the available published knowledge around a specific research question to support organizations and professionals who make recommendations and decisions about wide-ranging issues [2,3]. In health technology assessments (HTAs), which use explicit methods to determine the value of a health technology at different points in its lifecycle [4], SRs are routinely conducted to answer research questions around the effectiveness and safety of drugs, medical devices, and clinical interventions and provide clinical evidence to support their appropriate use. This paper focuses on such use of SRs of quantitative research in HTAs, that is, in the evaluation of interventions to prevent, diagnose, or treat diseases or conditions.
The traditional de novo SR combines the results of multiple primary studies to answer a specific research question [5,6]. There are, however, two alternative review types that use systematic methods and leverage existing SRs, namely an update of an SR and an overview of SRs. An update of an SR incorporates new evidence (by extending the source-or time-related domains of its search, including additional databases or longer periods), new methods, or new analyses into a previously completed SR [6][7][8]. An overview of SRsalso known as a review of reviews, an umbrella review, a meta-review, or other variations of these terms-compiles data from multiple SRs to provide a single summary of relevant evidence [5,9]. These three review types-de novo SRs, updates of existing SRs, and overviews of existing SRs-are most often used to evaluate intervention effects and form the basis of clinical evidence in HTAs. As such, other review types (e.g., scoping reviews, integrative reviews, and realist reviews) [10,11] were out of scope for this paper. Rapid reviews [12] and living reviews [13] were also out of scope for this paper, as these approaches can be used with any review type (e.g., an overview may be conducted rapidly with a single reviewer and updated regularly and frequently as a living document).
Definitions de novo SR: uses systematic and explicit methods* to combine the results of multiple primary studies to answer a specific research question [5,6].
Update of an SR: uses systematic and explicit methods* to incorporate new evidence, new methods, or new analyses into a previously completed SR [6][7][8].
Overview of SRs: uses systematic and explicit methods* to compile data from multiple SRs to provide a single summary of relevant evidence [5,9]. *Systematic and explicit methods usually involve: publishing a priori protocols that are detailed enough to be reproducible by others; searching multiple databases and grey literature to identify as much of the available relevant data as possible; and engaging two reviewers for each step during the review process to minimize bias.
Given the exponential growth in the number of published SRs [14,15] and high-resource requirements for completing new SRs [15,16], conducting updates or overviews of existing SRs, instead of de novo SRs, may be considered by reviewers under the right circumstances. Using existing SRs in updates or overviews can help minimize duplication of effort and redundancy in research and can contribute to a more efficient use of resources and a further synthesis of the extant literature [17]. However, there are unique challenges associated with conducting updates or overviews of existing SRs, arising from the need to rely on the work of the SR authors, which can be resource-intensive to address [15]. In other words, there are trade-offs. Therefore, potential benefits and challenges of leveraging existing SRs instead of primary study reports must both be considered. While there is some guidance on how to select a review type to conduct [2,15,[18][19][20], we are not aware of guidance that incorporates the myriad of factors that we have come to consider when making that decision in our work.
The objective of this paper is to share our guidance that is based on our work. It can be used when selecting the most appropriate review type that uses systematic methods to answer a research question about intervention effects in an HTA, while ensuring an appropriate balance between evidentiary needs and methodological rigour. The specific goals were as follows: (1) to identify factors that should be considered when deciding to conduct a de novo SR, an update of an existing SR, or an overview of existing SRs, and (2) to develop an algorithm to help with that decision-making process. The intent of this paper is not to provide a definitive answer on which review type to conduct in a specific scenario or to offer guidance on how to conduct a review using a specific review type. Rather, this paper identifies key factors that should be considered and organizes them into a summary and a flowchart that can be used as a decision tool to identify the right circumstances in which one review type over the others may be favoured or ruled out.

Process
This work consisted of three steps. First, a guidance document was drafted by methodologists from two Canadian HTA agencies based on their experience. Next, the draft guidance was supplemented with a literature review. Lastly, broader feedback from HTA researchers across Canada was sought and incorporated into the final guidance.

Step 1. Draft guidance
The Canadian Agency for Drugs and Technologies in Health (CADTH) and the Institute of Health Economics (IHE) have conducted both updates and overviews of existing SRs, instead of de novo SRs, to answer clinical research questions in several HTAs (e.g., see references [21][22][23][24][25]). Methodologists at CADTH (JK, LW) and IHE (MP), who had been involved in deciding which review types to conduct for the abovementioned projects, reflected on the key factors and considerations that informed, or could have informed, those decisions and organized them into a guidance document that comprised a summary and a corresponding flowchart outlining the decision-making process. The guidance was originally drafted by the methodologists at CADTH and then shared with the methodologist at IHE for review and input. This collaboration across the two HTA organizations helped ensure that the guidance was both methodologically robust and applicable across different decision-making contexts. Thus, the draft guidance reflected the experiences and expertise of two Canadian HTA agencies (i.e., CADTH and IHE).
Step 2. Literature review A literature review was conducted to identify any existing related guidance from academia, SR organizations, and other HTA agencies.

Search strategy
A targeted literature search was performed by an information specialist at CADTH. Published literature was identified by searching the MEDLINE database via Ovid. The search strategy was comprised of both controlled vocabulary, such as the National Library of Medicine's MeSH (Medical Subject Headings), and keywords. Retrieval was not limited by publication date but was limited to the English language. The search was completed on March 5, 2019. Regular alerts updated the search until manuscript submission. The complete search strategy is presented in Appendix 1.
Grey literature (literature that is not commercially published) was identified by searching sources listed in relevant sections of the Grey Matters checklist (https:// cadth. ca/ grey-matte rs-pract ical-tool-searc hing-healthrelat ed-grey-liter ature) [26], primarily the websites of Canadian and major international HTA agencies. Google was used to search for additional web-based materials. These searches were supplemented by reviewing bibliographies of key papers. More information on the grey literature search strategy is presented in Appendix 2.

Screening and selection
Literature identified through the database and grey literature searches were screened by a single reviewer (JK) at CADTH. Guidance for deciding when to conduct a de novo SR, an update of an existing SR, or an overview of existing SRs to evaluate intervention effects was in scope. Further, the following eligibility criteria were applied: • Documents of the following nature were in scope and included: ○ Provide explicit guidance for selecting one review type over another ○ Describe when it is or is not appropriate to use a specific review type ○ Help identify, define, or set thresholds for factors and considerations important for deciding to conduct a specific review type (e.g., how to judge relevance, quality, or currency of an existing SR) • Documents of the following nature were not in scope and excluded: ○ Papers on review types that are not intended to answer research questions about intervention effects (e.g., scoping reviews, integrative reviews, or realist reviews) ○ Papers on methodological approaches (e.g., rapid reviews or living reviews) that can be used with any review type ○ Primary studies, SRs, or clinical practice guidelines on specific topics and their protocols ○ Guidance for conducting a review using a specific review type ○ Guidance for conducting SRs in specific fields (e.g., anatomy or nursing) ○ Guidance for integrating quantitative and qualitative evidence (e.g., mixed methods or for knowledge translation) ○ Anecdotal, editorial, or opinion-based literature

Data extraction and analysis
Text that provided explicit guidance for selecting one review type over another, described when it is or is not appropriate to use a specific review type, or helped identify, define, or set thresholds for factors and considerations important for deciding to conduct a specific review type was deemed relevant. Excerpts of relevant text from the included documents were extracted and organized by a single reviewer (JK) at CADTH according to the factors and considerations identified in Step 1. All excerpts of relevant text are presented in Appendix 3.

Synthesis
The excerpts of relevant text in Appendix 3 were used to confirm, support, complement, or refine the draft guidance from Step 1. Details and citations from the literature review were incorporated into the guidance to help ensure that our guidance agreed with existing guidance from the literature. The updated guidance was then shared a second time with the methodologist at IHE (MP) for review and input.

Step 3. Broader discussion and feedback
The updated guidance from Step 2, which incorporated CADTH's and IHE's experiences as well as existing guidance from the literature, was shared more broadly with HTA researchers across Canada who have experience with deciding which review types to conduct and with conducting clinical reviews in HTAs. Input was collected from an additional nine researchers at CADTH and IHE as well as from 13 researchers at the Institut national d'excellence en santé et en services sociaux (INESSS) and the National Advisory Committee on Immunization (NACI). This input was collected through written feedback and/or small group discussions and further incorporated into the updated guidance from Step 2. Thus, the final guidance reflected the experiences and expertise of several Canadian HTA agencies (i.e., CADTH, IHE, INESSS, and NACI). All authors reviewed and approved the final version of the guidance, which appears in this paper.

Insights
This section has two parts. The first part identifies key factors and considerations that can inform the decision to select the most appropriate review type to answer a research question about intervention effects in an HTA. The second part organizes those key factors and considerations into a summary and a flowchart that can guide the decision-making process.

Part 1. Key factors and considerations
In total, nine key factors and six considerations that are important for selecting a specific review type to conduct for evaluating intervention effects were identified through reflections on experiences with clinical reviews in past HTA projects at CADTH and IHE. The literature review found these factors and considerations to be in good agreement with existing guidance from academia, SR organizations, and other HTA agencies. In total, 19 papers were identified as relevant through the literature review. They were from Joanna Briggs Institute [27], the Cochrane Collaboration [6,9], the Agency for Healthcare Research and Quality [15,18,19,28], the Belgian Health Care Knowledge Centre [29], the World Health Organization [30], eight different HTA agencies [31], the Centre for Reviews and Dissemination at the University of York [3], and other research groups [2,8,10,17,[32][33][34][35] (see Appendix 3). The feedback received from other HTA researchers across Canada also found these factors and considerations to be representative of the experiences of different organizations and applicable across different decision-making contexts. The nine key factors and six considerations each fell into one of two categories: the evidentiary needs of the planned review and the state of the existing literature. The key factors and considerations are presented in Table 1 and described below, with supporting citations. Of note, not all three review types are discussed or discussed to the same extent under each factor, as not all factors and considerations are relevant or deterministic in favouring or ruling out each review type. Also, many of the key factors and considerations are interdependent, as the evidentiary needs of a planned review must be balanced with the state of the existing literature. Therefore, many of the factors and considerations are not sufficient on their own to help select one review type over another, which motivated us to organize them into a summary and a corresponding flowchart in the following section. Further, we kept in mind the unique challenges that overviews present, stemming from having to rely on SRs, instead of primary studies, as the unit of analysis, and therefore possibly requiring more resources, compared to SRs [15,19], and tried to balance them with the potential benefits obtained (e.g., summarizing the existing literature at higher levels or avoiding or minimizing duplication or redundancy in research).
Reviewers who plan to use these key factors and considerations to help select the most appropriate review type for their HTA will need an understanding of both the evidentiary needs of the planned review and the state of the existing literature. Specifically, there must be a clear research question with a well-defined scope. Further, a scoping exercise should be undertaken to gain a general idea of the quantity, quality, and other characteristics of relevant SRs and primary studies that are available. While it may be difficult to obtain a comprehensive understanding of the literature through the scoping exercise, and reviewers will likely work with incomplete information, the guidance provided in this paper is intended to help reviewers think through, and plan for, the issues that they are likely to encounter when deciding which review type to conduct. To that end, reviewers may find it difficult to answer the questions provided below with certainty (e.g., instead of "yes" or "no, " "likely yes, " "likely no, " or even "unclear"), especially when the existing literature is vast or complex. Nevertheless, the scoping exercise could be viewed as an "upfront" investment in the conduct of an HTA that need not be wasted (e.g., information learned through the scoping exercise can be "reused" during the review process) and may end up saving resources (e.g., by identifying an existing SR and eliminating the need for a new SR) or achieving other meaningful goals (e.g., summarizing the existing literature at higher levels or avoiding or minimizing duplication or redundancy in research).

Key factor Consideration
Evidentiary needs of the planned review 1. Review scope: Is the scope of the research question broad? I. If the planned review has narrowly defined PICO elements, a de novo SR or an update of an SR may be the most appropriate review type. If the planned review has a broader scope that expands upon one of the PICO elements, an overview of SRs may be the most appropriate review type.
2. Review objective and analytic approach: Is quantitative combination of findings needed to provide a summary measure or to rank interventions?
II. If the planned review requires quantitative combination of findings through MAs, ITC, or NMAs to provide a summary measure or to rank interventions, a de novo SR or an update of an SR should be considered instead of an overview of SRs. In an overview, a narrative, not quantitative, synthesis should be conducted. I. Evidentiary needs of the planned review 1. Review scope: Is the scope of the research question broad?
A de novo SR and an update of an SR are identical or similar in scope [8], which tends to be focused and narrow [2,34]. An overview of SRs, on the other hand, tends to have a broader scope [2,3,10,32,33,35]. When expressed in terms of the Population, Intervention, Comparator, and Outcome (PICO) framework, a de novo SR or an update of an SR tends to have narrowly defined PICO elements, while an overview tends to have expanded upon one of the PICO elements. As described by the Cochrane Collaboration, overviews are appropriate for addressing research questions relating to: • The same intervention used for different conditions or populations (i.e., broader P); • Different interventions used for the same condition or population (i.e., broader I); • Different approaches to the application of the same intervention used for the same condition or population (i.e., broader I); • The same intervention used for the same condition or population but for different outcomes or time points (i.e., broader O); or • Adverse effects of an intervention for one or more conditions or populations (i.e., broader O or P) [27,32,34].
In other words, for an overview, there are two or more relevant SRs available that each covers one "part" of a broad research question, with all SRs combined resulting in perfect or almost perfect coverage [2,9,10,27,[32][33][34][35]. 2. Review objective and analytic approach: Is quantitative combination of findings needed to provide a summary measure or to rank interventions?
A de novo SR, an update of an SR, and an overview of SRs all intend to answer a specific research question [5], using explicit, systematic methods [34]. However, the unit of searching, inclusion, data extraction, analysis, and synthesis varies from the primary study-for a de novo SR and an update-to the SR-for an overview.
A de novo SR or an update of an SR-having access to primary study-level data-is best suited for quantitative combination of findings through meta-analyses (MAs), indirect treatment comparisons (ITCs), or network metaanalyses (NMAs) to provide a summary measure or answer a question about which intervention works best or is the safest.
With the SR as the unit of analysis, there are limitations in the analytic capabilities of an overview, and hence the research objectives it can fulfill. In an overview, quantitative combination of findings is more difficult, compared to narrative incorporation of findings, without going back to the primary study reports [15,27]. This is especially true if relevant SRs are of low quality in methodology or reporting (leading to errors or missing data) or if they overlap significantly in their included primary studies (giving too much statistical power to certain studies) [15,27]. Further, unless reviewers have thoroughly examined the transitivity assumption (i.e., that the studies making different direct comparisons are sufficiently similar in all respects other than the treatments being compared) and found it to be valid, which is very difficult to do using SR reports, ITCs, or NMAs should not be conducted in overviews [9,15,31,34]. Therefore, an overview of SRs-relying on review-level data-is best suited for examining a body of SR evidence to provide overall trends in research findings or answer a question about which interventions are effective or safe or for exploring if and why the evidence base on a topic or question is heterogeneous [27]. ITCs and NMAs in overviews are explicitly discouraged, and informal indirect comparisons should also be avoided [9]. To answer this question, database searches with appropriate SR filters or targeted searches for potentially relevant SRs may be needed. Potential SRs of interest could then be assessed on whether they meet eligibility criteria of the planned review, as specified by the PICO elements [28,29]. If the scope of an SR is identical to or narrower than that of the planned review, the entire SR would be relevant. If its scope is broader, the subset of its findings that meets the eligibility criteria of the planned review would be relevant, if reviewers are able to extract the relevant results separately (e.g., from a subgroup analysis).
If there is no relevant SR, conducting a de novo SR is likely necessary.
For an update, there must be one or more relevant SRs available that each perfectly or almost perfectly covers the research question. It may be possible to integrate multiple SRs in an update [15,19], but this is discouraged due to logistical and methodological complexities, as it would likely require merging existing SRs first with each other and secondly with newly identified primary studies [5,9]. Therefore, our recommendation is to handle multiple relevant SRs through either an overview of two or more SRs or an update of a single SR that is "best" in terms of relevance, quality, comprehensiveness, and currency [18].
For an overview, there should be two or more relevant SRs available that each covers at least one "part" of a research question, with all SRs combined resulting in perfect or almost perfect coverage. There is no accepted minimum or maximum number of SRs required for an overview, although recommendations range from including a minimum of "two or more" SRs [9] to "5-10 or more" SRs [29]. While it may sometimes be appropriate to include all relevant SRs in the planned review for completeness, it is often advised to assess the methodological quality of existing SRs and only use high-quality SRs in updates or overviews to build on high-quality literature [3,6,9,15,19,28,31]. In the absence of high-quality SRs, replication in the form of a de novo SR may be deemed appropriate to produce a higher-quality SR. [6,17,36] To assess methodological quality, relevant SRs should be critically appraised for their methodological rigour [17,29]. Established critical appraisal tools, such as AMSTAR 2 [37] or ROBIS [38], can be used [17,29]. There is no broadly accepted threshold for what is considered sufficiently high quality in methodology for a relevant SR to be used in a planned review, although there is guidance for what is good, fair, or poor quality [3,18]. The threshold for sufficiently high quality in methodology may vary depending on the purpose of the review, how the relevant SRs will be used, whether there are high-quality SRs available, and what resources are available for the planned review. Nevertheless, criteria for determining whether relevant SRs are of sufficiently high quality in methodology should be established a priori for each planned review to minimize bias [28].
5. Reporting quality: Are the relevant SRs of sufficiently high quality in reporting?
Poor reporting in existing SRs would make it difficult not only to extract data but also to decide whether the SRs are relevant, of high methodological quality, comprehensive, and up to date, and hence, useful for the planned review. There-fore, high quality in reporting is desirable [15,19], especially if the SRs are to be used in overviews without going back to primary study reports.
To assess quality in reporting, reporting guidelines may be used [32], supplemented with additional criteria required for the planned review (e.g., exact data needed, such as point estimates and associated variability in a specific format). Available reporting guidelines include PRISMA Statement (for all SRs) [39], SWiM (for SRs with narrative syntheses) [40], and MOOSE (for SRs with MAs of observational studies in epidemiology) [41]. There is no broadly accepted threshold for what is considered sufficiently high quality in reporting for an existing SR to be used. However, at the least, it should be well written, describe the PICO elements and methods clearly, and report all necessary data in an easily extractable form. To answer this question, relevant SRs should be examined on whether they include all relevant PICO elements of the planned review either individually (i.e., a single SR is comprehensive on its own) or together (i.e., two or more SRs are comprehensive when combined). If they do not, using them in an overview may result in research gaps in the final review, as it is not recommended that an overview of SRs be supplemented with primary studies due to logistical and methodological complexities [5,9]. Therefore, a research question that cannot be fully addressed with existing SRs due to important research gaps in those SRs may be best addressed by a de novo SR or an update of an SR (e.g., with a search for additional data from an expansion of one or more of the PICO elements or a re-analysis of the available data for a different outcome) if primary studies are available to fill those gaps. On the other hand, if two or more relevant SRs are available that when combined result in perfect or almost perfect coverage [2,9,10,27,[32][33][34][35], an overview of SRs may be appropriate.
7. Currency: Are the relevant SRs up to date?
In general, an SR may be considered to be up to date when (1) there is no new evidence or (2) there is new information, but it is unlikely to change the review conclusions [34]. However, the definition of currency varies from one field to another, as the acceptable time period from the last date of search will depend on various factors, such as the activity or advances in the field or in methods, the importance and urgency of the research question being addressed, and the level of uncertainty around the evidence base [3,8,19,42]. Criteria for determining the currency or outdatedness of relevant SRs need to be established a priori for each planned review to minimize bias [28,43].
For those overviews that have the objective to answer a research question about intervention effects (as opposed to the objective to describe a body of literature), all relevant SRs should be sufficiently up to date [9]. Whether an SR is sufficiently up to date should be determined with the help of experts in the field of research who have the knowledge to speak to the various factors that define currency mentioned above. Therefore, a research question that cannot be fully addressed with existing SRs because they are out of date may be best addressed by an update of an SRif there is a high-quality SR available-or by a de novo SR-if there is no high-quality SR available-that is, if new evidence is available to bring it up to date [3]. If there are two or more relevant SRs addressing the same research question with matching PICO elements, and if they are of sufficiently high quality in both methodology and reporting, their results could then be compared. Results are concordant if they match with respect to the direction, magnitude, and statistical significance of the estimated intervention effects for the outcome(s) of interest and in terms of the interpretations and inferences made by the SR authors [44]. Conversely, results are discordant if they differ in any of those aspects [44]. Such differences may or may not be important enough to lead to different health care decisions [44].
If the results of relevant SRs with matching PICO elements are discordant and if the differences matter for the planned review, reasons for discordance should be explored [15,18,19,30,35,44]. Potential reasons for discordance include differences in the methods or timing of different SRs that lead to different sets of studies being included, data being extracted, analyses being conducted, or syntheses or interpretations being made [44]. If reasons for discordant findings are clear, and if there is certainty around which discrepant results to trust [15,30,35]-likely those of high quality in both methodology and reporting and most up to date-reviewers may choose one or more SRs that are most appropriate for the planned review and conduct an update or an overview. If reasons for discordant findings are unclear and if there is uncertainty around which discrepant results to trust [15,30,35], a de novo SR may be indicated, instead of using one or more of the conflicting SRs [15,19].
Consideration IV: If the results of two or more relevant SRs with matching PICO elements are discordant, and reasons for discordance cannot be reliably determined, a de novo SR may be needed.

Part 2. Making the decision
A flowchart is presented in Fig. 1 and described below. This flowchart depicts the nine key factors and six considerations outlined above in the order that we see as helpful for selecting a specific review type to conduct for evaluating intervention effects in an HTA. The flowchart should be followed from top to bottom, first with Part A and then with Part B. Each question is indicated with a "Q" followed by a number that corresponds to the numbering of the key factors identified above. Both Part A and Part B begin with Q1 and Q2 (in blue boxes), which address the evidentiary needs of the planned review, and then proceed to Q3 through Q9 (in orange boxes), which address the state of the existing literature. Some of the questions from Q3 to Q9 are combined or modified for further investigation. Depending on the responses to the questions ("yes" or "no" in black circles), some review types are eliminated through this decision-making process (in yellow boxes), leading to one decision at the end (in green boxes).
If there is only a single relevant SR that is high quality in both methodology and reporting, comprehensive, and up to date, there is likely no need for a new SR of any kind for the specific research question (see D5 in Fig. 1 Fig. 1).
For most situations, however, it is the interplay of multiple considerations regarding the key factors that informs the decision on which review type to conduct. In these cases, while no review types can be completely ruled out, one review type may be favoured over the others, as follows: • If the planned review has a narrow scope (Consideration I), instead of an overview (see D1 in Part A of Fig. 1), it may be more appropriate to conduct a de novo SR (see D4 in Part A of Fig. 1) or an update of an SR (see D6 and D8 in Part A of Fig. 1 Fig. 1), or an update if there is a single "best" relevant SR that is high quality in both methodology and reporting and comprehensive on its own (see D 6 and D8 in Part B of Fig. 1).
The flowchart in Fig. 1 outlines a decision-making process for selecting a review type to conduct, capturing all of the considerations from above.

Discussion
This paper shares guidance that can be used when selecting the most appropriate review type to conduct when evaluating intervention effects in an HTA. It identified nine key factors and six considerations concerning the evidentiary needs of the planned review and the state of the existing literature. Those factors and considerations were then organized into a summary and a corresponding flowchart that can be used as a decision tool, balancing potential benefits of leveraging existing SRs against any drawbacks of not starting a new SR from scratch. With that, a reviewer may choose to conduct no new SR at all, update an existing SR, use multiple existing SRs in an overview, or conduct a de novo SR. This guidance is aligned with the literature from various organizations [3,6,9,15,18,19,[27][28][29][30][31] and other research groups [2,8,10,[32][33][34][35]45], including a recent publication that highlights eliminating research waste by avoiding unnecessary duplication of SRs [17].
To fully contemplate these key factors and considerations for a planned review, there must be a clear research question as well as a good understanding of the quantity, quality, and other characteristics of relevant SRs and primary studies that are available. Therefore, this approach may require detailed scoping and perhaps a process change as well as additional resources as an "upfront" investment in the conduct of an HTA. This challenges the notion that leveraging existing SRs necessarily saves resources, and it reinforces the idea that existing SRs should be leveraged for other reasons, such as summarizing the existing literature at higher levels and avoiding or minimizing duplication or redundancy in research, all of which had been raised by other groups as well [15,17,19,46]. Further, any additional work performed to follow the guidance presented here and apply the decision tool need not be wasted, as it could identify existing SRs that fully meet the needs of the planned review and actually free up and save resources or it could be used during the review process, regardless of the review type chosen.
The decision tool presented here depicts general guidance and should be contextualized to each case. For example, it might not always be clear if the research scope for the planned review is broad or narrow (Q1 in Fig. 1 in both Part A and Part B). Further, even for a narrow topic, if there are multiple relevant SRs, an overview may be considered as long as there is no need for quantitative combination of findings. In another example, if there is a high-quality SR that is almost but not fully comprehensive (Q3-6 in Fig. 1 in both Part A and Part B), the reviewer may decide to update that SR with revised PICO elements, instead of conducting a de novo SR, although this would generally be considered a purposeful replication and perhaps more of a de novo SR instead of an update in any case [6][7][8]17]. Therefore, the decisions outlined in the tool should be viewed as examples in ideal situations instead of a collection of definitive and exhaustive possibilities, and going against those decisions is possible and may even be appropriate in certain cases. The flowchart should be seen as simply highlighting the many interacting considerations that need be contemplated during the decision-making process. We suspect that others who conduct evidence syntheses are also working through these decisions, perhaps on an ad hoc basis, and present this work to share our current decision-making process and invite commentary and reflections.
Regardless of which review type is used, it is essential that the project protocol and report provide a list of existing relevant SRs identified in the literature through the scoping activity (e.g., as an appendix) and the rationale behind selecting a specific review type over others for transparency purposes. During scoping, effort should be made to also identify ongoing SRs (e.g., through PROSPERO). If ongoing or planned reviews are identified, opportunities for brokering or collaborating or other ways to incorporate them into the decision-making process for selecting a review type should be explored.
We considered incorporating into the flowchart rapid approaches [12] and living approaches [13], either or both of which could be overlaid on each review type (e.g., rapid SR, rapid update, or rapid overview; living SR or living overview; living rapid SR or living rapid overview). A rapid approach alters, simplifies, or omits components of a systematic process to produce information in a shorter period of time [12]. This approach may involve searching fewer databases or engaging a single reviewer instead of dual reviewers and may be used under resource constraints (e.g., short timelines or limited funding) [20], especially if there is already some high-quality evidence with one or more relevant SRs available and no unexplained discrepant findings across the SRs. A living approach continually updates an SR or an overview, incorporating relevant new evidence as it becomes available [13]. This approach may be used if a priority research question cannot be fully addressed because there is an important level of uncertainty in the existing evidence and if new evidence that will impact the current conclusions is likely to emerge soon in a research field that is moving relatively quickly [43]. As the literature on these approaches is still actively emerging, we did not incorporate these review approaches into the flowchart but identify them as potential additional considerations, as also noted by others [17]. In other words, reviewers could use the flowchart provided here to first select the most appropriate review type to conduct and then decide to apply rapid and/or living approaches to the chosen review type as needed.

Conclusions
This work provides guidance on how to balance a myriad of factors and considerations to help select the most appropriate review type to conduct when evaluating intervention effects in an HTA. The decision-making process presented is undoubtedly complicated, yet fundamental for every review, given the exponential growth in the number of published SRs [14,15] and high-resource requirements for completing new SRs [15]. While leveraging existing SRs may or may not save resources, doing so will likely contribute to a better synthesis of the existing literature and a reduction of research waste and therefore lead to a more efficient use of resources in HTAs.
We hope this guidance adds clarity to the many competing considerations when deciding which review type to conduct and facilitates that decision-making process. As we continue to test and update this decision tool, we invite feedback from others who may have their own guidance or apply ours in their work. While we focused on SRs of intervention effects, the application of this guidance to other types of reviews, such as qualitative or mixed methods reviews, should be explored in the future. ((updating or existing systematic review* or "out-of-date" or outdated or up-to-dateness or redundant or redundancy or incorporating or overlap* or choosing or selecting or deciding or integrating or discordan* or concordan* or replicat*) and (systematic review* or comparative effectiveness review* or overviews of reviews or overview of reviews or Cochrane review* or meta-analyses)).ti,kf.
3 (overview of reviews or overviews of reviews or overview of systematic reviews or overviews of systematic reviews).ti,ab,kf. 4 (overview of overviews or overviews of overviews).ti,ab,kf. 5 ((updat* adj2 review*) or (updat* adj2 systematic review*) or (updat* adj2 comparative effectiveness review*)).ti,ab,kf. 6 review design.ti,ab,kf. 7 (existing review* or existing systematic review* or existing comparative effectiveness review*).ti,ab,kf. 8 (umbrella review* or umbrella systematic review*).ti,ab,kf. 9 (review of reviews or reviews of reviews or review of systematic reviews or reviews of systematic reviews).ti,ab,kf.     Umbrella review "Conduct of an Umbrella Review offers the possibility of addressing a broad scope of issues related to a topic of interest. " "The principal reason is to summarize evidence from many research syntheses (Becker and Oxman 2011). These may include analyses of evidence of different interventions for the same problem or condition, or evidence from more than one research synthesis investigating the same intervention and condition but addressing and reporting on different outcomes. Similarly, a researcher or reviewer may wish to summarize more than one research synthesis for different conditions, problems or populations. " "The wide picture obtainable from the conduct of an Umbrella Review is also ideal in highlighting if the evidence base around a topic or question is consistent or if contradictory or discrepant findings exist, and in exploring and detailing the reasons why. Investigation of the evidence with an Umbrella Review allows assessment and consideration of whether reviewers addressing similar review questions independently observe similar results and arrive at generally similar conclusions. " "…a systematic review is the main and often sole "study type" that is considered for inclusion (Becker and Oxman 2011; Hartling et al. 2012; Smith et al, 2011). " "The principle focus of a JBI Umbrella Review is to provide a summary of existing research syntheses related to a given topic or question and not to re-synthesize, for example, the results of existing reviews or syntheses with metaanalysis or meta-synthesis. " "…if current, multiple, good quality, systematic reviews exist about a given topic or question, any reviewer should reconsider the need to conduct yet another review addressing the same issue. Rather, these may be the basis to conduct an Umbrella Review and summarize or synthesize the findings of systematic reviews already available. " None.
Cochrane Collaboration [9] Overview "Cochrane Overviews can address five different types of questions related to healthcare interventions. Specifically, they can summarize evidence from two or more systematic reviews: • of different interventions for the same condition or population; • that address different approaches to applying the same intervention for the same condition or population; • of the same intervention for different conditions or populations; • about adverse effects of an intervention for one or more conditions or populations; or • of the same intervention for the same condition or population, where different outcomes or time points are addressed in different systematic reviews. " "The primary reason for conducting Cochrane Overviews is that using systematic reviews as the unit of searching, inclusion, and data analysis allows authors to address research questions that are broader in scope than those examined in individual systematic reviews…a second reason for conducting a Cochrane Overview is that they may be associated with time and resource savings, since the component systematic reviews have already been conducted. A third reason for conducting a Cochrane Overview is in cases where it is important to understand the diversity present in the extant systematic review literature. " "…it is preferable to conduct a Cochrane Review of interventions if authors anticipate the need to conduct searches for primary studies (i.e., many relevant primary studies are not included in systematic reviews) or to extract data directly from primary studies (i.e., the anticipated analyses cannot be conducted on the basis of information provided in the systematic reviews). Using primary studies as the unit of searching, inclusion and data analysis allows authors to extract all data of interest directly from the primary studies and to report these data in a standardized way. It is also preferable to conduct a Cochrane Review of interventions if authors wish to conduct network meta-analyses, which allow authors to rank order interventions and determine which work 'best' . " "Overviews can describe the current body of systematic review evidence on a topic of interest, or they can address a new review question that wasn't a focus in the included systematic reviews. " "There are several instances where authors should not conduct a Cochrane Overview. Overviews do not aim to: • repeat or update the searches or eligibility assessment of the included systematic reviews; • conduct a study-level search for primary studies not included in any systematic review; • conduct a new systematic review within the Overview; • use systematic reviews as a starting point to locate relevant studies with the intent of then extracting and analysing data from the primary studies (this would be considered a systematic review, or an update of a systematic review, and not an Overview); • search for and include narrative reviews, textbook chapters, government reports, clinical practice guidelines, or any other summary reports that do not meet their pre-defined definition of a systematic review; • extract and present just the conclusions of the included systematic reviews (instead, actual outcome data-narratively reported study-level data and/or meta-analysed datashould be extracted and analysed, and Overview authors are encouraged to interpret these outcome data themselves, in light of the Overview's research questions and objectives); • present detailed outcome data for primary studies not included in any included systematic review; or • conduct network meta-analyses…" "…the unit of searching, inclusion and data analysis is the systematic review rather than the primary study. " "Although Overviews often present evidence from two or more systematic reviews of different interventions for the same condition or population, they should rarely be used to draw inferences about the comparative effectiveness of multiple interventions. This means that they should not directly compare interventions that have been examined in different systematic reviews with the intent of determining which intervention works 'best' or which intervention is 'safest' . " "We discourage indirect comparisons in Overviews. This is especially relevant for authors conducting Overviews that examine multiple interventions for the same condition or population; it is also relevant for authors regardless of whether the systematic reviews included in the Overview present their data using meta-analysis or simple narrative summaries of results. The reason is that the assumption underlying indirect comparison-the transitivity assumption-can rarely be assessed using only the information provided in the systematic reviews…" "Overviews that examine multiple interventions for the same condition or population will often juxtapose data from different systematic reviews. Sometimes, these data appear in the same   Integration of existing systematic reviews into new reviews "Some described that including existing reviews sometimes enables them to cover a wider range of questions and elements of questions (as denoted by PICOTS) when existing systematic reviews address important aspects of new review key questions. " Not reported.
"When incorporating existing reviews into a new or updated review, EPC members most often described qualitative or narrative incorporation of the existing reviews, noting that quantitative combination of findings (without going back to all primary studies) is more difficult and potentially introduces error, and thus, is less commonly done. " "Methods by which relevant existing systematic reviews can be evaluated for quality of methodological approach, using AMSTAR or other commonly used tools, with a focus on potentially incorporating only reviews meeting certain quality criteria into the proposed review…" "there are other instances when it would not make sense to use an existing review, such as when there are few studies on a given topic or when there are multiple conflicting systematic reviews. " "If more than one high-quality review is found with discordant findings, it may be an indication to start a de novo review on that key question. " "Overall themes were identified from the discussions with EPCs: • EPCs most commonly used existing reviews as a source of relevant literature and as context for the introduction or discussion section of reviews.
• Existing reviews were most useful when key questions and/or PICOTS-SD (population, intervention, comparator, outcome, time frame, setting, and study design) matched or when they addressed a specific subquestion of the new review.
• Using existing reviews was often more resource intensive than completing a review from scratch.
• EPCs expressed that they often did not trust aspects of reviews conducted by others.
• When relevant and rigorous, incorporating prior reviews into the review being undertaken by the EPC was clearly valuable in at least two instances: 1) allowing larger scope of the review being undertaken without additional resources, or 2) providing summarized evidence when a new in depth review of primary literature would not be feasible (for example, existing reviews provide individual patient data analysis or include hundreds of trials, supplemented by author-provided data). " Belgian Health Care Knowledge Centre [29] Meta-review/ review of reviews Not reported.
Not reported.
Not reported.
"The main factors that have to be taken into account when deciding to perform a meta-review are the amount of reviews available on the topic and the time available for doing a review.
In general a shortage in time and/or a high number of reviews available, may favour the decision to do a meta-review. " "No general guidelines exist but from a pragmatic point of view it is advisable to discuss in the research team if a meta-review is a viable option, if there are approximately 5-10 or more recent reviews fairly matching your research question. " "Other factors to consider in the decision: • Applicability of published reviews to your topic: are the reviews really about the intervention you are interested in and did they extract data on the outcome of your interest?; • Recency of the reviews, especially the search date is essential: it is good to take the search of one of the reviews and rerun it to see how many recent primary studies will be missing and to estimate if these could possibly alter the review conclusions; "There are pros and cons for this approach.  • Availability of meta-reviews: it can well be the case that on popular topics, one or more meta-reviews already exist, and if so the decision to do a new one should be weighted with the same factors as mentioned above. It can also be considered to do a review of meta-reviews, but in general this is not advised because the primary evidence is getting out of sight.
No clear-cut decision process can be presented here; it really needs discussion in the research team. There are many trade-offs in determining whether it is more efficient, and a methodologically sound process to rely on a prior review or to start from scratch. And keep in mind that a meta-review can only be as good as the reviews and primary studies on which it is based. " World Health Organization [30] Update Not reported.
Not reported.
Not reported.
"It is not always necessary to commission a new systematic review (see Fig. 8.1). If one or more relevant, current and highquality systematic reviews exist, these should be used. Updates, if needed, are usually less expensive and time-consuming than new reviews. " "…updating an existing systematic review is a reasonable option provided a high quality, fully reported review exists that uses current methods for identifying, appraising and synthesizing the evidence. " "To assess relevance, compare the key question of the existing systematic review to the key questions that were developed during the guideline scoping exercise, considering each component of PICO. Most frequently, the existing review does not entirely match the current key question. Nonetheless, the review may address one facet of WHO's key question or provide useful background information, and the list of included studies may inform WHO's systematic review.
If the existing systematic review addresses one of the key questions of the guideline, then its quality should be assessed. The following checklists may be used to assess the quality of systematic reviews: • assessment of multiple systematic reviews (AMSTAR) (11), or R-AMSTAR (12);  (13). "Once a systematic review has been found relevant and of high quality, whether it is up to date or not must be determined. There is no rule for dismissing a review on the basis of the time since publication (for example, two years); it depends on the topic and on the availability and rate of production of new information. " Agency for Healthcare Research and Quality [18] Replacing de novo processes using existing systematic reviews Not reported.
Not reported.
Not reported.
"If the selected relevant, highquality SRs have discordant findings, EPCs should explore the reasons for these disagreements.
If EPCs cannot readily give reasons for the discordant findings, then they can regard this as an indication that they need to adopt a de novo approach to answer that key question. " "If EPCs find that several recent, relevant, and high-quality SRs are appropriate for a given CER, they then need to determine how best to proceed. One approach is to incorporate the single "best" existing SR (most relevant and least biased) into their own reports [6]. However, selecting a single review may pose the risk of introducing selection bias; EPCs must ensure transparency in their criteria for eligibility. Another approach is to conduct a meta-review (also known as an "umbrella review"), whereby they select all relevant, high-quality SRs that meet an a priori publication date threshold and then assess the consistency among them [26,27]. " "In general, good-quality SRs should be defined as those that have few or no methodological shortcomings and a low risk of bias. Fair-quality SRs have some methodological flaws but the EPC conducting the CER determined that the flaws will not seriously bias or invalidate the results. Poor-quality SRs contain a serious flaw or flaws that, in the judgment of the EPC conducting the CER, are highly likely to bias or invalidate the results. " Using existing systematic reviews Not reported.
Not reported.
Not reported.
"…reviews should be excluded when the number of primary studies is small and any gains in efficiency by using the reviews would be minimal at best. " "Systematic reviews should be as free of bias as possible if they are to reliably inform guideline recommendations or other policy decisions (2). Quality assessment of existing systematic reviews is therefore a critical step and should address both the methods used by the systematic reviewers to minimize bias as well as the transparency and completeness of reporting of review methods, individual study details, and results. In fact, the priority should be to include existing systematic reviews adhering to high methodological standards, rather than to routinely include all existing systematic reviews in order to be comprehensive. " "…when higher-quality reviews disagree, a new independent review that carefully addresses potential sources of disagreement with careful sensitivity or subgroup analyses may be the most informative strategy. " Not reported.

Recommendations
Seven of the HTA agencies were open to searching for primary studies. "According to two HTA agencies, a narrative synthesis instead of a quantitative synthesis should be performed, while the third agency only notes that an evidence synthesis is likely to be problematic. " "In the context of overviews, a metaanalysis of meta-analyses should turn out to be difficult as some of the primary studies will usually be included in more than one meta-analysis. Pooling results would give too much statistical power to multiple included primary studies (Smith et al., 2011). Other ways of presenting findings will have to maintain a reasonable balance between complexity and a potential loss of important findings from the reviews. " "Five of the HTA agencies have prerequisites for the conduct of overviews ( Table 1). All of them state that relevant SRs must be of high quality, although only one HTA agency concretizes this by setting cut-off points for the review quality, based on the assessment with a critical appraisal tool. " "Overviews should be updated when the corresponding reviews have been updated. " report the results of all included studies clearly, highlighting any similarities or differences between studies, and exploring the reasons for any variations. " "If a high quality review is located, but was completed some time ago, then an update of the review may be justified. Current relevance will need to be assessed and is particularly important in fields where the research is rapidly evolving.
Where appropriate, collaboration with the original research team may assist in the update process by providing access to the data they used. However, little research has been conducted on when and how to update systematic reviews and the feasibility and efficiency of the identified approaches is uncertain [11]. If a review is of adequate quality and still relevant, there may be no need to undertake another systematic review. " None.
Review of reviews "Reviews of reviews are likely to be helpful when a review question is very broad and a number of systematic reviews have already been conducted in the topic area. " Not reported.
Not reported.
"Reviews of reviews are likely to be helpful when a review question is very broad and a number of systematic reviews have already been conducted in the topic area. " None.