Tailored PRISMA 2020 flow diagrams for living systematic reviews: a methodological survey and a proposal

Lara A. Kahale; Rayane Elkhoury; Ibrahim El Mikati; Hector Pardo-Hernandez; Assem M. Khamis; Holger J. Schünemann; Neal R. Haddaway; Elie A. Akl

doi:10.12688/f1000research.51723.3

Home Browse Tailored PRISMA 2020 flow diagrams for living systematic reviews:...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Revised

Tailored PRISMA 2020 flow diagrams for living systematic reviews: a methodological survey and a proposal

[version 3; peer review: 2 approved]

Previously titled: "PRISMA flow diagrams for living systematic reviews: a methodological survey and a proposal"

Lara A. Kahale¹, Rayane Elkhoury^2,3, Ibrahim El Mikati⁴, [...] Hector Pardo-Hernandez^5,6, Assem M. Khamis⁷, Holger J. Schünemann^8,9, Neal R. Haddaway^10-12, Elie A. Akl^4,8,13

Lara A. Kahale¹, Rayane Elkhoury^2,3, [...] Ibrahim El Mikati⁴, Hector Pardo-Hernandez^5,6, Assem M. Khamis⁷, Holger J. Schünemann^8,9, Neal R. Haddaway^10-12, Elie A. Akl^4,8,13

PUBLISHED 28 Jan 2022

Author details Author details

¹ Cochrane Central Executive, American University of Beirut, St Albans House, 57-59 Haymarket, London, SW1Y 4QX, UK
² Infectious Disease Epidemiology Group, American University of Beirut, Qatar Foundation - Education City, Doha, Qatar
³ World Health Organization Collaborating Centre for Disease Epidemiology Analytics on HIV/AIDS, Sant Pau Biomedical Research Institute, Cornell University, Qatar Foundation – Education City, Doha, Qatar
⁴ Clinical Research Institute, CIBER Epidemiología y Salud Pública, Beirut, Riad El Solh 1107 2020, Lebanon
⁵ McMaster University, Madrid, Av. de Monforte de Lemos, 5, 28029, Spain
⁶ Iberoamerican Cochrane Centre, University of Hull, Barcelona, C / Sant Quintí, 77-79 08041, Spain
⁷ Wolfson Palliative Care Research Centre, McMaster University, Hull, HU6 7RX, UK
⁸ Department of Health Research Methods, Evidence, and Impact, Stockholm Environment Institute, Hamilton, Ontario, 1280 Main Street West 2C Area, Canada
⁹ Department of Medicine, Mercator Research Institute on Global Commons and Climate Change, Hamilton, Ontario, 1280 Main Street West 2C Area, Canada
¹⁰ University of Johannesburg, Stockholm, Linnégatan, 87D, Sweden
¹¹ Africa Centre for Evidence, University of Johannesburg, Johannesburg, South Africa
¹² American University of Beirut, Eberswalder Str. 84, 15374, Müncheberg, Germany
¹³ Department of Internal Medicine, American University of Beirut, Beirut, Riad El Solh 1107 2020, Lebanon

Lara A. Kahale
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Supervision, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Rayane Elkhoury
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Supervision, Validation, Visualization, Writing – Review & Editing

Ibrahim El Mikati
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Hector Pardo-Hernandez
Roles: Data Curation, Investigation, Resources, Writing – Review & Editing

Assem M. Khamis
Roles: Formal Analysis, Methodology, Software, Validation, Writing – Review & Editing

Holger J. Schünemann
Roles: Writing – Review & Editing

Neal R. Haddaway
Roles: Software, Visualization, Writing – Review & Editing

Elie A. Akl
Roles: Conceptualization, Methodology, Supervision, Visualization, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Research on Research, Policy & Culture gateway.

This article is included in the Living Evidence collection.

Abstract

Background: While the PRISMA flow diagram is widely used for reporting standard systematic reviews (SRs), it was not designed for capturing the results of continual searches for studies in living systematic reviews (LSRs). The objectives of this study are (1) to assess how published LSRs report on the flow of studies through the different phases of the review for the different updates; (2) to propose an approach to reporting on that flow.
Methods: For objective 1, we identified all LSRs published up to April 2021. We abstracted information regarding their general characteristics and how they reported on search results. For objective 2, we based our proposal for tailored PRISMA approaches on the findings from objective 1, as well as on our experience with conducting Cochrane LSRs.
Results: We identified 279 living publications relating to 76 LSRs. Of the 279 publications, 11% were protocols, 23% were base versions (i.e., the first version), 50% were partial updates (i.e., does not include all typical sections of an SR), and 16% were full updates (i.e., includes all typical sections of an SR). We identified six ways to reporting the study flow: base separately, each update separately (38%); numbers not reported (32%); latest update separately, all previous versions combined (20%); base separately, all updates combined (7%); latest update version only (3%); all versions combined (0%). We propose recording in detail the results of the searches to keep track of all identified records. For structuring the flow diagram, we propose using one of four approaches.
Conclusion: We identified six ways for reporting the study flow through the different phases of the review for the different update versions. We propose to document in detail the study flow for the different search updates and select one of our four tailored PRISMA diagram approaches to present that study flow.

Keywords

PRISMA statement, living systematic review, update, research methodology research reporting, flow chart, systematic review reporting standards, evidence synthesis, research transparency, research replication

Corresponding author: Elie A. Akl

Competing interests: LAK, HP, AMK, HJS, EAA have been involved in methodological work on living systematic reviews and are authors of living systematic reviews.

Grant information: The author(s) declared that no grants were involved in supporting this work.

Copyright: © 2022 Kahale LA et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Kahale LA, Elkhoury R, El Mikati I et al. Tailored PRISMA 2020 flow diagrams for living systematic reviews: a methodological survey and a proposal [version 3; peer review: 2 approved]. F1000Research 2022, 10:192 (https://doi.org/10.12688/f1000research.51723.3) First published: 08 Mar 2021, 10:192 (https://doi.org/10.12688/f1000research.51723.1) Latest published: 28 Jan 2022, 10:192 (https://doi.org/10.12688/f1000research.51723.3)

Revised Amendments from Version 2

We have made the following edits from version 2 to version 3:

We updated the affiliations of co-authors LAK, REK, IM, and NH
We updated the search from July 2020 to April 2021, our sample now includes 76 LSRs (270 living publications) instead of 32 LSRs (108 living publications). Hence, we updated the methods sections of the abstract and main-text.
We updated the survey findings in table 1 (characteristics of the living publications), table 2 (reporting on study flow), and figure 1 (Summary of the four tailored PRISMA flow diagram approaches) with the analysis of the 76 LSRs (270 living publications). Hence, we updated our results sections of the abstract and main-text.
Our conclusions did not change
Our suggestions of approaches to documenting and reporting LSR study flow did not change.
We referenced under implications for practice a web-based App that can facilitate producing flow diagrams for LSRs using the four approaches suggested.
We discussed under implications for research our plans and in developing extension to the PRISMA 2020 statement for LSRs.
We did some copy-editing.

See the authors' detailed response to the review by Lex Bouter
See the authors' detailed response to the review by Sonia Hines

Introduction

During the coronavirus disease 2019 (COVID-19) pandemic, health research has proliferated exponentially¹. Systematic reviews are essential to synthesize the evidence and inform policy and practice. Given the pace of research publication, those reviews need to be kept up to date. Living systematic reviews (LSRs) are an emerging type of systematic review that involves the continual search of the literature and incorporation of relevant new evidence, soon after it becomes available². While many evidence synthesis groups are engaged in conducting LSRs or living network meta-analyses, others have developed living databases or living maps, including resources specific for COVID-19 literature^3–17.

An essential component of systematic reviews is to keep track of and report the number of records captured while searching the scientific literature and details of the selection process¹⁸. The PRISMA statement recommends the use of the PRISMA flow diagram to depict the flow of studies through the different phases of the systematic review¹⁹. While the PRISMA flow diagram is a widely used tool for reporting original systematic reviews, it was not designed to capture the results of continual searches typically used in LSRs. Hence, it’s unclear how authors of LSRs address the issue of presenting results of these continual searches.

The objectives of this study were (1) to assess how published LSRs report on the flow of studies through the different phases of the review for the different updates; and (2) to propose an approach to documenting and reporting on the flow of studies through the different phases of a LSR, for the different updates.

Methods

For objective 1, we collected relevant data as part of a larger methodological survey aiming to assess the methods of conduct and reporting of LSRs. We have described the details of that study in a previously published protocol²⁰. Briefly, we identified all living reviews published up to April 2021 available from the following electronic databases: Medline, EMBASE and the Cochrane library (see extended data²¹ of Khamis et al.²⁰ for the search strategy). An eligible living review was either (1) a protocol for an LSR, (2) a base version of an LSR, (3) a full update version of an LSR, (4) a partial update version of an LSR, or (5) a combination of any of these (e.g., one living review may constitute of a protocol, a base version, and a full update version; another living review may constitute of only a Box 1 the definition of each type of living reviews.

Box 1. Definition of the different publication types of living reviews

• LSR protocol: the protocol that describes the planned methods of the living review

• Base version: the first version of the review that follows a living approach

• Full update version: a subsequent version of the review that includes all the typical sections of a systematic review, including an introduction, methods, and results sections. Such a version could stand-alone in terms of content.

• Partial update version: a subsequent version of the review that does not include all the typical sections of a systematic review, but instead refers to a previous version for complementary information. Such a version could not stand-alone in terms of content.

For the current study, we abstracted information about the following features of LSRs:

General characteristics:
- Publication type, i.e., protocol, base version, full update version, partial update version.
- Whether published in the Cochrane library or elsewhere.
- Field (e.g., clinical, public health)
- Whether COVID-19 related or not
- Whether the base version of the living review conducted as a rapid review or not
Reporting on study flow
- Method used to report on the study flow (including the search results and the results of the selection process):
  - ▪ Narrative format and/or flow diagram.
  - ▪ Whether the results of the base and update searches are reported separately or not.
- Type of flow diagram, if applicable (e.g., PRISMA).

For objective 2, we base our proposal for tailored PRISMA 2020 flow diagram approaches on the findings from objective 1, on our experience conducting Cochrane LSRs, and our methodological work on designing and reporting living evidence. Since 2017, our group has been responsible for the first series of three Cochrane LSRs, all of which address anticoagulation in patients with cancer^22–24. We conducted the base search in February 2016. Since then, we have been updating the search on a monthly basis. Through this experience, we have been able to apply and refine the guidance for conducting LSRs endorsed by the living evidence network group²⁵. Specifically, we explored solutions for the reporting of the study flow that would address different scenarios. Our goal was not to be prescriptive and narrow, but rather to cover all possible resulting flows by reviewing the LSRs we identified based on objective 1. Two authors developed a draft of the tailored approaches to presenting the study flow, and then circulated to the author team for review and suggestions for improvement.

Data handling and analysis

We used REDCap to collect and manage the data abstraction process. All data were exported from REDCap and analyzed using Stata v. 13^26,27.

Results

Survey findings

Our search identified a total of 279 living publications relating to 76 LSRs. Table 1 shows their general characteristics. Of the 279 living publications, 11% were protocols, 23% were base versions, 50% were partial updates, and 16% were full updates. The median number of living publications per LSR was 2 (Interquartile range 1–4). Of the 76 living reviews, 22% were published in the Cochrane library, 63% were related to COVID-19, and 25% had a base version published as a rapid review. The majority were related to clinical topics (70%).

Table 1. General characteristics of the 279 included living publications related to 76 living reviews.

	N	n (%)
Publication type	279 living publications
• Protocol		31 (11.1)
• Base version		64 (22.9)
• Partial update version		138 (49.5)
• Full update version		46 (16.5)
Living publications per LSR (Median (IQR))	76 LSRs	2 (1 – 4)
Cochrane LSR	76 LSRs	17 (22.4)
Field	76 LSRs
• Clinical		53 (69.7)
• Public health		20 (26.3)
• Health system and policy		3 (4.0)
COVID-19-related	76 LSRs	48 (63.2)
Base version published as rapid review	64 base versions	16 (25.0)

Abbreviations: LSR: living systematic review; IQR: interquartile range

Table 2 shows the results for the reporting on the study flow. Most base versions and full updates used a flow diagram to report on the search results (96% and 93% respectively), whereas only one partial update presented a flow diagram. In addition, none of the 279 living publications reported in their methods section how they plan to report on the study flow.

Table 2. Reporting on study flow.

	N	n (%)
Inclusion of a flow diagram in^a
• Base version	64 base versions	62 (96.9)
• Partial update version	138 partial updates	1 (0.7)
• Full update version	46 full updates	43 (93.5)
Approach to reporting on study flow for different versions	184 update versions
Base separately; each update separately		39 (21.2)
Base separately; all updates combined		20 (10.9)
Latest update separately; all previous versions combined (including the base)		24 (13.0)
All versions combined		12 (6.5)
Latest update version only		47 (25.5)
Numbers not reported		42 (22.8)

^a When a flow diagram is not reported, the authors reported on the search results in a narrative format.

Among the 184 update versions (Figure 1):

21% reported the search results for the base version and for each update version separately.
11% reported on the search results for the base version separately and for all update versions combined
13% reported the search results for the latest update version separately and for all previous versions combined (including the base).
6% reported the search results for all the different versions combined.
26% reported the search results for the latest update version only.
23% did not report the search results at all (e.g., ‘new studies identified and integrated’ without specifying the number).

Figure 1. Summary of the four tailored PRISMA flow diagram approaches to present that study to presenting on the study flow for the different search updates.

Proposed tailored PRISMA flow diagram approaches

Using the approach described in the methods section, we developed four approaches that allow authors to document and report the study flow for the different review update versions of an LSR.

1. Documenting LSR study flow

Authors should record in detail the results of the searches to keep track of all identified records. We propose using a spreadsheet for one LSR at a time. The format we present consists of tabs for each of the respective search sources: bibliographic databases (e.g. MEDLINE, EMBASE, Cochrane databases); conference proceedings; ongoing studies as captured in clinicaltrials.gov and WHO International Clinical Trials Registry Platform (ICTRP); other tabs as needed, and a final ‘cumulative’ tab.

We show in Figure 2 a snapshot of the ‘cumulative’ tab of the spreadsheet that keeps track of all records. It shows the study flow for a hypothetical example for an LSR published first in January 2020 (i.e., base version) and updated on a monthly basis up to August 2020. Each row corresponds to a different update version. The columns present the following information for each update (columns B to E): the number of records received, deduplicated, included at title and abstract screening, and included at full-text screening (i.e., newly included reports). Additional columns (F to I) present the distribution of the newly included reports as relating to either: (1) new studies, (2) previously included studies, (3) ongoing (unpublished) studies, or (4) preprints.

Figure 2. Snapshot of the ‘cumulative’ tab of the spreadsheet that keep track of all identified records.

After manually entering the information in the first five tabs (corresponding to the different search sources) the total is automatically computed in the ‘cumulative’ tab.

2. Reporting LSR study flow

The proposed spreadsheet can act as a basis for a tailored PRISMA flow diagram for LSRs. For structuring the flow diagram for LSR, one can select one out of four tailored PRISMA 2020 flow diagram approaches:

Approach 1: presenting the search results of the different versions separately (i.e., base and each update separately) (Figure 3).
Approach 2: presenting the search results for the different versions combined (i.e., including base and all update versions) (Figure 4).
Approach 3: presenting the search results for the base version separately, and the results of all update version combined (Figure 5).
Approach 4: presenting the results of the latest update version separately, and the results of all previous versions (including the base) combined (Figure 6).

Figure 3. Approach 1: presenting the search results of the different versions separately (i.e., base and each update separately).

Figure 4. Approach 2: presenting the search results for the different versions combined (i.e., including base and all update versions).

Figure 5. Approach 3: presenting the search results for the base version separately, and the results of all update version combined.

Figure 6. Approach 4: presenting the results of the latest update version separately, and the results of all previous versions (including the base) combined.

In our Cochrane reviews, we applied the second proposal where we present the results for the different searches combined.

Discussion

Summary

This study found that authors of LSRs are not consistent in reporting on the flow of studies through the different phases of the review for the different update versions. Thus, we propose to document in detail the study flow for the different search updates and select one of four tailored PRISMA 2020 flow diagram approaches to present that study flow.

Strengths and limitations

To our knowledge, this is the first methodological survey that assesses how LSR authors report on the flow of studies through the different phases of the review for the different update versions of LSRs. In addition, the research expertise on our team covers both living approach and regular updating of traditional SR. We believe that our assessment forms a vital baseline and allows us to propose best practices for visualization options to improve consistency whilst the production of LSRs is still at a relatively early stage. Indeed, this survey is part of a larger methodological survey aiming to assess the methods of conduct and reporting of LSRs²⁰, that would allow us to update our findings in the future.

Interpretation of findings

Authors tend to produce more partial updates of LSRs rather than continually updating the full systematic review. This might seem like a pragmatic approach particularly for a rapidly growing research field and when methods do not seem to change from one update to another. The heterogeneity observed in the ways LSR authors report on the study flow is likely to be explained by the lack of clear guidance on how to do so.

Implications for practice

We built our proposal on the PRISMA 2020 flow diagram and provide four approaches to tailor the needs for continual searchers used in LSR. The fourth approach is the closest to the current PRISMA 2020 flow diagram as it presents the results of the latest update version separately and the results of all previous versions (including the base) combined.

In addition, we proposed three other different approaches to provide options to LSR authors and publishing journals. Authors should choose one or the other approach based on the number of new citations, presentation preferences, and the impression of what provides the greatest transparency in reporting. Whatever approach one decides to follow, for transparency purposes, the systematic reviewers should ideally archive previous versions of the flow diagram (e.g., in an appendix). One major challenge will be to accommodate a large number of updates in the same diagram; some approaches would work better than others in that case. Also, advanced information technology solutions may allow fitting a large number of updates. A web-based prototype is available that allows readers to explore different reporting options across these four approaches: an R package (https://github.com/nealhaddaway/livingPRISMAflow) and web-based ShinyApp (https://estech.shinyapps.io/livingprismaflow/) were developed that allow users to enter their own data (e.g., from the spreadsheet suggested above) to produce a bespoke flow diagram according to their desired approach or to create their own interactive diagram that allows readers to toggle between different versions of the same data²⁸.

Advanced information technology can also be utilized to simplify updating and tracking the change in all LSR sections including the PRISMA diagram. It would be optimal to develop the base version in a certain platform where all SR and LSR sections are reported as units (i.e., title, authors, background, objectives, inclusion criteria, effect estimate for outcome x). With each update and for every unit, the author has the luxury to keep the same text (if no change has occurred) or edit (if change has occurred). Each unit can be updated in a differential speed based on certain criteria. The edits could be highlighted to visualize the change. For a certain section, one would easily have access to the entries in the previous versions and possibly visualize a trend across the different versions (i.e., cross-sectional view for that specific item). For example, dynamic documents can be developed using ‘R markdown’, a document preparation system, where static text can be combined with in-line code and ‘code chunks’ that produce instantly updatable documents given a modified input²⁹.

Implications for future research

This study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs (please see registration form on EQUATOR network website: Equator Network. PRISMA for LSR – Extension of PRISMA 2020 for living systematic reviews. 2021; Accessed from https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/#LSR]. This project will pilot the proposed approaches for documenting the study flow and for structuring the living flow diagram. In addition, qualitative studies would be helpful to explore: (1) the feasibility and acceptability by LSR authors, publishers, and users towards the proposal; and (2) what the end-users would like to see in an LSR update.

Conclusions

LSR authors are not consistent in reporting the flow of studies through the different phases of the review for the different update versions. We propose to document in detail the study flow for the different search updates. Authors can select one of our four tailored PRISMA 2020 flow diagram approaches to present that study flow until detailed guidance will become available. Improving the reporting of study flow in LSR methodology is essential for incorporating living evidence when developing living guidance, particularly in the context of an urgent response^30,31.

Data availability

Underlying data

All data underlying the results are available as part of the article and no additional source data are required.

Acknowledgment

We would like to acknowledge Dr. Mathew Page for his revision for the manuscript.

Faculty Opinions recommended

References

1. Nasrallah AA, Farran SH, Nasrallah ZA, et al.: A large number of COVID-19 interventional clinical trials were registered soon after the pandemic onset: a descriptive analysis. J Clin Epidemiol. 2020; 125: 170–178. PubMed Abstract | Publisher Full Text | Free Full Text
2. Elliott JH, Turner T, Clavisi O, et al.: Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014; 11(2): e1001603. PubMed Abstract | Publisher Full Text | Free Full Text
3. John A, Eyles E, Webb RT, et al.: The impact of the COVID-19 pandemic on self-harm and suicidal behaviour: update of living systematic review [version 1; peer review: 1 approved, 2 approved with reservations]. F1000Res. 2020; 9: 1097. PubMed Abstract | Publisher Full Text | Free Full Text
4. Currie G, Macleod MR, Sena E, et al.: Protocol for a “living” evidence summary of primary research related to Covid-19. 2020. Publisher Full Text
5. Geisler BP, Zahabi L, Lang AE, et al.: Repurposing Existing Medications for Coronavirus Disease 2019: Protocol for a Rapid and Living Systematic Review. medRxiv. 2020. Publisher Full Text
6. Juul S, Nielsen N, Bentzer P, et al.: Interventions for treatment of COVID-19: a protocol for a living systematic review with network meta-analysis including individual patient data (The LIVING Project). Syst Rev. 2020; 9(1): 108. PubMed Abstract | Publisher Full Text | Free Full Text
7. Maguire BJ, Guérin PJ: A living systematic review protocol for COVID-19 clinical trial registrations [version 1; peer review: 2 approved]. Wellcome Open Res. 2020; 5: 60. PubMed Abstract | Publisher Full Text | Free Full Text
8. Maguire BJ, McLean ARD, Rashan S, et al.: Baseline results of a living systematic review for COVID-19 clinical trial registrations [version 1; peer review: 2 approved]. Wellcome Open Res. 2020; 5: 116. PubMed Abstract | Publisher Full Text | Free Full Text
9. Schünemann HJ, Khabsa J, Solo K, et al.: Ventilation techniques and risk for transmission of coronavirus disease, including COVID-19: a living systematic review of multiple streams of evidence. Ann Intern Med. 2020; 173(3): 204–216. PubMed Abstract | Publisher Full Text | Free Full Text
10. Ortiz-Muñoz LE, Ferrer BM, Duarte-Anselmi G, et al.: Protocol of a Living systematic review: Gloves for the prevention of COVID-19 in healthy population. 2020. Publisher Full Text
11. Siemieniuk RA, Bartoszko JJ, Ge L, et al.: Drug treatments for covid-19: living systematic review and network meta-analysis. BMJ. 2020; 370: m2980. PubMed Abstract | Publisher Full Text | Free Full Text
12. Rada G, Verdugo-Paiva F, Ávila C, et al.: Evidence synthesis relevant to COVID-19: a protocol for multiple systematic reviews and overviews of systematic reviews. Medwave. 2020; 20(3): e7868. PubMed Abstract | Publisher Full Text
13. Cochrane: Coronavirus (COVID-19) - Cochrane resources and news. The Cochrane Collaboration. 2019. Reference Source
14. World Health Organization: Global research on coronavirus disease (COVID-19). 2020. Reference Source
15. Living Overview of Evidence (LOVE): Coronavirus disease (COVID‑19). Epistemonikos foundation. 2020. Reference Source
16. EPPI center: COVID-19: a living systematic map of the evidence. 2020. Reference Source
17. Boutron I, et al.: Interventions for preventing and treating COVID-19: protocol for a living mapping of research and a living systematic review. Syst Rev. 2020; 9: 108.
18. Millard T, Synnot A, Elliott J, et al.: Feasibility and acceptability of living systematic reviews: results from a mixed-methods evaluation. Syst Rev. 2019; 8(1): 325. PubMed Abstract | Publisher Full Text | Free Full Text
19. Page MJ, McKenzie J, Bossuyt P, et al.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. 2020. Publisher Full Text
20. Khamis AM, Kahale LA, Pardo-Hernandez H, et al.: Methods of conduct and reporting of living systematic reviews: a protocol for a living methodological survey [version 2; peer review: 2 approved]. F1000Res. 2019; 8: 221. PubMed Abstract | Publisher Full Text | Free Full Text
21. Khamis AM, Kahale LA, Pardo-Hernandez HJ, et al.: Search strategies. figshare. Journal contribution. 2019. Publisher Full Text
22. Akl EA, Kahale LA, Hakoum MB, et al.: Parenteral anticoagulation in ambulatory patients with cancer. Cochrane Database Syst Rev. 2017; 9(9): CD006652. PubMed Abstract | Publisher Full Text | Free Full Text
23. Kahale LA, Hakoum MB, Tsolakian IG, et al.: Anticoagulation for the long-term treatment of venous thromboembolism in people with cancer. Cochrane Database Syst Rev. 2018; 6(6): CD006650. PubMed Abstract | Publisher Full Text | Free Full Text
24. Kahale LA, Hakoum MB, Tsolakian IG, et al.: Oral anticoagulation in people with cancer who have no therapeutic or prophylactic indication for anticoagulation. Cochrane Database Syst Rev. 2017; 12(12): CD006466. PubMed Abstract | Publisher Full Text | Free Full Text
25. Brooker J, Synnot A, McDonald S, et al.: Guidance for the production and publication of Cochrane living systematic reviews: Cochrane reviews in living mode. Cochrane Collaboration. 2019. Reference Source
26. StataCorp: Stata Statistical Software: Release 13. College Station, Editor. StataCorp LP: TX. 2013.
27. Harris PA, Taylor R, Thielke R, et al.: Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009; 42(2): 377–81. PubMed Abstract | Publisher Full Text | Free Full Text
28. Haddaway N: SRflowdiagram: flow charts for systematic reviews and maps. 2020.
29. Haddaway NR: DynamicSMapResults: template for designing updatable systematic map results text using R Markdown (Version 0.0.1). Zenodo. 2020. Publisher Full Text
30. Akl EA, Morgan RL, Rooney AA, et al.: Developing trustworthy recommendations as part of an urgent response (1-2 weeks): a GRADE concept paper. J Clin Epidemiol. 2021; 129: 1–11. PubMed Abstract | Publisher Full Text | Free Full Text
31. Schünemann HJ, Santesso N, Vist GE, et al.: Using GRADE in situations of emergencies and urgencies: certainty in evidence and recommendations matters during the COVID-19 pandemic, now more than ever and no matter what. J Clin Epidemiol. 2020; 127: 202–207. PubMed Abstract | Publisher Full Text | Free Full Text

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 08 Mar 2021

Author details Author details

Rayane Elkhoury
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Supervision, Validation, Visualization, Writing – Review & Editing

Ibrahim El Mikati
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Hector Pardo-Hernandez
Roles: Data Curation, Investigation, Resources, Writing – Review & Editing

Assem M. Khamis
Roles: Formal Analysis, Methodology, Software, Validation, Writing – Review & Editing

Holger J. Schünemann
Roles: Writing – Review & Editing

Neal R. Haddaway
Roles: Software, Visualization, Writing – Review & Editing

Elie A. Akl
Roles: Conceptualization, Methodology, Supervision, Visualization, Writing – Review & Editing

Competing interests

LAK, HP, AMK, HJS, EAA have been involved in methodological work on living systematic reviews and are authors of living systematic reviews.

Grant information

The author(s) declared that no grants were involved in supporting this work.

Article Versions (3)

version 3

Revised

Published: 28 Jan 2022, 10:192

https://doi.org/10.12688/f1000research.51723.3

version 2

Revised

Published: 18 Mar 2021, 10:192

https://doi.org/10.12688/f1000research.51723.2

version 1

Published: 08 Mar 2021, 10:192

https://doi.org/10.12688/f1000research.51723.1

© 2022 Kahale LA et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Kahale LA, Elkhoury R, El Mikati I et al. Tailored PRISMA 2020 flow diagrams for living systematic reviews: a methodological survey and a proposal [version 3; peer review: 2 approved] F1000Research 2022, 10:192 (https://doi.org/10.12688/f1000research.51723.3)

NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 3

VERSION 3

PUBLISHED 28 Jan 2022

Revised

Views

Reviewer Report 31 Jan 2022

Lex Bouter, Department of Epidemiology and Data Science, Amsterdam Universities Medical Center, Amsterdam, The Netherlands; Department of Philosophy, Faculty of Humanities, Vrije Universiteit, Amsterdam, The Netherlands

Approved

https://doi.org/10.5256/f1000research.120090.r121494

I applaud the authors for updating their survey and for more than double the sample size. They also made some other - rather marginal - changes and explain that their publication is only the first of a series based on ... Continue reading

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Methodology, epidemiology, research integrity, open science, systematic review methods.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Version 2

VERSION 2

PUBLISHED 18 Mar 2021

Revised

Views

Reviewer Report 22 Jul 2021

Sonia Hines, College of Medicine and Public Health, Flinders University, Alice Springs, Northern Territory, Australia

Approved

https://doi.org/10.5256/f1000research.55448.r87133

This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable.

I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point.

The authors have done what they set out to do, but I suggest further work is needed before a recommended method of study flow reporting is settled upon.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Systematic reviews, evidence-based practice.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Author Response 28 Jan 2022

Lara Kahale, Clinical Research Institute, American University of Beirut, Beirut, Riad El Solh 1107 2020, UK

28 Jan 2022

Author Response
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where ... Continue reading
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 2: Sonia Hines

1. This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable. I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point.

Thank you for your positive feedback and practical advice. We agree that spreadsheets are not easily publishable, that is why we recommend using it as a tool for the LSR authors to document in detail the study flow as opposed to using it as a reporting tool. We now elaborate in the discussion how this spreadsheet would feed in an app (shinyapp) that creates nicely designed flows for these four approaches well suited for insertion into a publication.

2. The authors have done what they set out to do, but I suggest further work is needed before a recommended method of study flow reporting is settled upon.

We agree with this comment. This study is a methodological survey aiming to summarize what LSR authors are currently reporting. As this is not a guidance document, we are not making any recommendations. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs. For that purpose, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in a Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders.
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 2: Sonia Hines

1. This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable. I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point.

Thank you for your positive feedback and practical advice. We agree that spreadsheets are not easily publishable, that is why we recommend using it as a tool for the LSR authors to document in detail the study flow as opposed to using it as a reporting tool. We now elaborate in the discussion how this spreadsheet would feed in an app (shinyapp) that creates nicely designed flows for these four approaches well suited for insertion into a publication.

2. The authors have done what they set out to do, but I suggest further work is needed before a recommended method of study flow reporting is settled upon.

We agree with this comment. This study is a methodological survey aiming to summarize what LSR authors are currently reporting. As this is not a guidance document, we are not making any recommendations. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs. For that purpose, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in a Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 28 Jan 2022

Lara Kahale, Clinical Research Institute, American University of Beirut, Beirut, Riad El Solh 1107 2020, UK

28 Jan 2022

Author Response
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where ... Continue reading
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 2: Sonia Hines

1. This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable. I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point.

Thank you for your positive feedback and practical advice. We agree that spreadsheets are not easily publishable, that is why we recommend using it as a tool for the LSR authors to document in detail the study flow as opposed to using it as a reporting tool. We now elaborate in the discussion how this spreadsheet would feed in an app (shinyapp) that creates nicely designed flows for these four approaches well suited for insertion into a publication.

2. The authors have done what they set out to do, but I suggest further work is needed before a recommended method of study flow reporting is settled upon.

We agree with this comment. This study is a methodological survey aiming to summarize what LSR authors are currently reporting. As this is not a guidance document, we are not making any recommendations. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs. For that purpose, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in a Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders.
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 2: Sonia Hines

1. This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable. I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point.

Thank you for your positive feedback and practical advice. We agree that spreadsheets are not easily publishable, that is why we recommend using it as a tool for the LSR authors to document in detail the study flow as opposed to using it as a reporting tool. We now elaborate in the discussion how this spreadsheet would feed in an app (shinyapp) that creates nicely designed flows for these four approaches well suited for insertion into a publication.

2. The authors have done what they set out to do, but I suggest further work is needed before a recommended method of study flow reporting is settled upon.

We agree with this comment. This study is a methodological survey aiming to summarize what LSR authors are currently reporting. As this is not a guidance document, we are not making any recommendations. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs. For that purpose, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in a Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 28 Jun 2021

Approved with Reservations

https://doi.org/10.5256/f1000research.55448.r87481

Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there’re some issues that should to be solved in the next version of the manuscript.

Major issues

It’s disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like ‘anything goes’. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.
The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest.
The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals?
The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

No
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Methodology, epidemiology, research integrity, open science, systematic review methods.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 28 Jan 2022

Lara Kahale, Clinical Research Institute, American University of Beirut, Beirut, Riad El Solh 1107 2020, UK

28 Jan 2022

Author Response
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where ... Continue reading
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 1: Lex Bouter

Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there’re some issues that should to be solved in the next version of the manuscript.

Major issues

1. It’s disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like ‘anything goes’. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.

Thank you for your comment. This study is a methodological survey aiming to summarize what current LSR authors are reporting. Given that the full methodology of living systematic reviews is still emerging, we prefer not to make firm recommendations but lay out options. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs (please see registration form on EQUATOR network website: Equator Network. PRISMA for LSR – Extension of PRISMA 2020 for living systematic reviews. 2021; Accessed from https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/#LSR]) . Consistently with the Reviewer’s suggestions, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders. We have elaborated about this research under implications for future research.

2. The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest.

Thank you for your query. This paper is the first published study within the larger project which aims to explore how LSRs authors are currently reporting, conducting, and publishing LSRs. We cite the protocol of that project (Khamis 2019). This study focuses on reporting flow diagrams, another study will focus on general characteristics and delay in updating LSRs, and a third study will focus on methodological features of LSRs. We have now updated our results to include data from up to April 2021, and our dataset now includes 279 publications of 76 LSRs.

3. The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals?

Thank you for your comment. Unfortunately, that was the state of the science. However, as noted in the previous comment, we have now updated our search to include results up to April 2021 Including data from 279 publications of 76 LSRs. We avoided including confidence intervals for the simplicity of presentation and given the small sample size. We are happy to include these if the editor prefers so.

4. The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review.

Thank you for your suggestion. We have checked and found the following (now added to the text): ‘None of the 279 living publications reported in their methods section how they plan to report on the study flow’.
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 1: Lex Bouter

Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there’re some issues that should to be solved in the next version of the manuscript.

Major issues

1. It’s disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like ‘anything goes’. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.

Thank you for your comment. This study is a methodological survey aiming to summarize what current LSR authors are reporting. Given that the full methodology of living systematic reviews is still emerging, we prefer not to make firm recommendations but lay out options. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs (please see registration form on EQUATOR network website: Equator Network. PRISMA for LSR – Extension of PRISMA 2020 for living systematic reviews. 2021; Accessed from https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/#LSR]) . Consistently with the Reviewer’s suggestions, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders. We have elaborated about this research under implications for future research.

2. The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest.

Thank you for your query. This paper is the first published study within the larger project which aims to explore how LSRs authors are currently reporting, conducting, and publishing LSRs. We cite the protocol of that project (Khamis 2019). This study focuses on reporting flow diagrams, another study will focus on general characteristics and delay in updating LSRs, and a third study will focus on methodological features of LSRs. We have now updated our results to include data from up to April 2021, and our dataset now includes 279 publications of 76 LSRs.

3. The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals?

Thank you for your comment. Unfortunately, that was the state of the science. However, as noted in the previous comment, we have now updated our search to include results up to April 2021 Including data from 279 publications of 76 LSRs. We avoided including confidence intervals for the simplicity of presentation and given the small sample size. We are happy to include these if the editor prefers so.

4. The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review.

Thank you for your suggestion. We have checked and found the following (now added to the text): ‘None of the 279 living publications reported in their methods section how they plan to report on the study flow’.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 28 Jan 2022

Lara Kahale, Clinical Research Institute, American University of Beirut, Beirut, Riad El Solh 1107 2020, UK

28 Jan 2022

Author Response
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where ... Continue reading
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 1: Lex Bouter

Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there’re some issues that should to be solved in the next version of the manuscript.

Major issues

1. It’s disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like ‘anything goes’. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.

Thank you for your comment. This study is a methodological survey aiming to summarize what current LSR authors are reporting. Given that the full methodology of living systematic reviews is still emerging, we prefer not to make firm recommendations but lay out options. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs (please see registration form on EQUATOR network website: Equator Network. PRISMA for LSR – Extension of PRISMA 2020 for living systematic reviews. 2021; Accessed from https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/#LSR]) . Consistently with the Reviewer’s suggestions, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders. We have elaborated about this research under implications for future research.

2. The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest.

Thank you for your query. This paper is the first published study within the larger project which aims to explore how LSRs authors are currently reporting, conducting, and publishing LSRs. We cite the protocol of that project (Khamis 2019). This study focuses on reporting flow diagrams, another study will focus on general characteristics and delay in updating LSRs, and a third study will focus on methodological features of LSRs. We have now updated our results to include data from up to April 2021, and our dataset now includes 279 publications of 76 LSRs.

3. The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals?

Thank you for your comment. Unfortunately, that was the state of the science. However, as noted in the previous comment, we have now updated our search to include results up to April 2021 Including data from 279 publications of 76 LSRs. We avoided including confidence intervals for the simplicity of presentation and given the small sample size. We are happy to include these if the editor prefers so.

4. The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review.

Thank you for your suggestion. We have checked and found the following (now added to the text): ‘None of the 279 living publications reported in their methods section how they plan to report on the study flow’.
The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 1: Lex Bouter

Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there’re some issues that should to be solved in the next version of the manuscript.

Major issues

1. It’s disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like ‘anything goes’. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.

Thank you for your comment. This study is a methodological survey aiming to summarize what current LSR authors are reporting. Given that the full methodology of living systematic reviews is still emerging, we prefer not to make firm recommendations but lay out options. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs (please see registration form on EQUATOR network website: Equator Network. PRISMA for LSR – Extension of PRISMA 2020 for living systematic reviews. 2021; Accessed from https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/#LSR]) . Consistently with the Reviewer’s suggestions, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders. We have elaborated about this research under implications for future research.

2. The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest.

Thank you for your query. This paper is the first published study within the larger project which aims to explore how LSRs authors are currently reporting, conducting, and publishing LSRs. We cite the protocol of that project (Khamis 2019). This study focuses on reporting flow diagrams, another study will focus on general characteristics and delay in updating LSRs, and a third study will focus on methodological features of LSRs. We have now updated our results to include data from up to April 2021, and our dataset now includes 279 publications of 76 LSRs.

3. The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals?

Thank you for your comment. Unfortunately, that was the state of the science. However, as noted in the previous comment, we have now updated our search to include results up to April 2021 Including data from 279 publications of 76 LSRs. We avoided including confidence intervals for the simplicity of presentation and given the small sample size. We are happy to include these if the editor prefers so.

4. The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review.

Thank you for your suggestion. We have checked and found the following (now added to the text): ‘None of the 279 living publications reported in their methods section how they plan to report on the study flow’.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 08 Mar 2021

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 3 (revision) 28 Jan 22	read
Version 2 (revision) 18 Mar 21	read	read
Version 1 08 Mar 21

Lex Bouter, Amsterdam Universities Medical Center, Amsterdam, The Netherlands; Vrije Universiteit, Amsterdam, The Netherlands
Sonia Hines, Flinders University, Alice Springs, Australia

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

8 Views

31 Jan 2022 | for Version 3

8 Views Cite this report Responses(0)

Approved

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Methodology, epidemiology, research integrity, open science, systematic review methods.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

11 Views

22 Jul 2021 | for Version 2

Sonia Hines, College of Medicine and Public Health, Flinders University, Alice Springs, Northern Territory, Australia

11 Views Cite this report Responses(1)

Approved

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Systematic reviews, evidence-based practice.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Author Response

28 Jan 2022

Lara Kahale, Clinical Research Institute, American University of Beirut, Beirut, Riad El Solh 1107 2020, UK

The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 2: Sonia Hines

1. This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable. I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point.

Thank you for your positive feedback and practical advice. We agree that spreadsheets are not easily publishable, that is why we recommend using it as a tool for the LSR authors to document in detail the study flow as opposed to using it as a reporting tool. We now elaborate in the discussion how this spreadsheet would feed in an app (shinyapp) that creates nicely designed flows for these four approaches well suited for insertion into a publication.

2. The authors have done what they set out to do, but I suggest further work is needed before a recommended method of study flow reporting is settled upon.

We agree with this comment. This study is a methodological survey aiming to summarize what LSR authors are currently reporting. As this is not a guidance document, we are not making any recommendations. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs. For that purpose, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in a Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

17 Views

28 Jun 2021 | for Version 2

17 Views Cite this report Responses(1)

Approved With Reservations

It’s disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like ‘anything goes’. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.
The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest.
The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals?
The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

No
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Methodology, epidemiology, research integrity, open science, systematic review methods.

Respond to this report

Responses (1)

Author Response

28 Jan 2022

Lara Kahale, Clinical Research Institute, American University of Beirut, Beirut, Riad El Solh 1107 2020, UK

The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 1: Lex Bouter

Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there’re some issues that should to be solved in the next version of the manuscript.

Major issues

1. It’s disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like ‘anything goes’. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.

Thank you for your comment. This study is a methodological survey aiming to summarize what current LSR authors are reporting. Given that the full methodology of living systematic reviews is still emerging, we prefer not to make firm recommendations but lay out options. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs (please see registration form on EQUATOR network website: Equator Network. PRISMA for LSR – Extension of PRISMA 2020 for living systematic reviews. 2021; Accessed from https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/#LSR]) . Consistently with the Reviewer’s suggestions, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders. We have elaborated about this research under implications for future research.

2. The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest.

Thank you for your query. This paper is the first published study within the larger project which aims to explore how LSRs authors are currently reporting, conducting, and publishing LSRs. We cite the protocol of that project (Khamis 2019). This study focuses on reporting flow diagrams, another study will focus on general characteristics and delay in updating LSRs, and a third study will focus on methodological features of LSRs. We have now updated our results to include data from up to April 2021, and our dataset now includes 279 publications of 76 LSRs.

3. The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals?

Thank you for your comment. Unfortunately, that was the state of the science. However, as noted in the previous comment, we have now updated our search to include results up to April 2021 Including data from 279 publications of 76 LSRs. We avoided including confidence intervals for the simplicity of presentation and given the small sample size. We are happy to include these if the editor prefers so.

4. The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review.

Thank you for your suggestion. We have checked and found the following (now added to the text): ‘None of the 279 living publications reported in their methods section how they plan to report on the study flow’.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Nasrallah AA, Farran SH, Nasrallah ZA, et al.: A large number of COVID-19 interventional clinical trials were registered soon after the pandemic onset: a descriptive analysis. J Clin Epidemiol. 2020; 125: 170–178. PubMed Abstract | Publisher Full Text | Free Full Text

[2] 2. Elliott JH, Turner T, Clavisi O, et al.: Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014; 11(2): e1001603. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. John A, Eyles E, Webb RT, et al.: The impact of the COVID-19 pandemic on self-harm and suicidal behaviour: update of living systematic review [version 1; peer review: 1 approved, 2 approved with reservations]. F1000Res. 2020; 9: 1097. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. Currie G, Macleod MR, Sena E, et al.: Protocol for a “living” evidence summary of primary research related to Covid-19. 2020. Publisher Full Text

[5] 5. Geisler BP, Zahabi L, Lang AE, et al.: Repurposing Existing Medications for Coronavirus Disease 2019: Protocol for a Rapid and Living Systematic Review. medRxiv. 2020. Publisher Full Text

[6] 6. Juul S, Nielsen N, Bentzer P, et al.: Interventions for treatment of COVID-19: a protocol for a living systematic review with network meta-analysis including individual patient data (The LIVING Project). Syst Rev. 2020; 9(1): 108. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Maguire BJ, Guérin PJ: A living systematic review protocol for COVID-19 clinical trial registrations [version 1; peer review: 2 approved]. Wellcome Open Res. 2020; 5: 60. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Maguire BJ, McLean ARD, Rashan S, et al.: Baseline results of a living systematic review for COVID-19 clinical trial registrations [version 1; peer review: 2 approved]. Wellcome Open Res. 2020; 5: 116. PubMed Abstract | Publisher Full Text | Free Full Text

[9] 9. Schünemann HJ, Khabsa J, Solo K, et al.: Ventilation techniques and risk for transmission of coronavirus disease, including COVID-19: a living systematic review of multiple streams of evidence. Ann Intern Med. 2020; 173(3): 204–216. PubMed Abstract | Publisher Full Text | Free Full Text

[10] 10. Ortiz-Muñoz LE, Ferrer BM, Duarte-Anselmi G, et al.: Protocol of a Living systematic review: Gloves for the prevention of COVID-19 in healthy population. 2020. Publisher Full Text

[11] 11. Siemieniuk RA, Bartoszko JJ, Ge L, et al.: Drug treatments for covid-19: living systematic review and network meta-analysis. BMJ. 2020; 370: m2980. PubMed Abstract | Publisher Full Text | Free Full Text

[12] 12. Rada G, Verdugo-Paiva F, Ávila C, et al.: Evidence synthesis relevant to COVID-19: a protocol for multiple systematic reviews and overviews of systematic reviews. Medwave. 2020; 20(3): e7868. PubMed Abstract | Publisher Full Text

[13] 13. Cochrane: Coronavirus (COVID-19) - Cochrane resources and news. The Cochrane Collaboration. 2019. Reference Source

[14] 14. World Health Organization: Global research on coronavirus disease (COVID-19). 2020. Reference Source

[15] 15. Living Overview of Evidence (LOVE): Coronavirus disease (COVID‑19). Epistemonikos foundation. 2020. Reference Source

[16] 16. EPPI center: COVID-19: a living systematic map of the evidence. 2020. Reference Source

[17] 17. Boutron I, et al.: Interventions for preventing and treating COVID-19: protocol for a living mapping of research and a living systematic review. Syst Rev. 2020; 9: 108.

[18] 18. Millard T, Synnot A, Elliott J, et al.: Feasibility and acceptability of living systematic reviews: results from a mixed-methods evaluation. Syst Rev. 2019; 8(1): 325. PubMed Abstract | Publisher Full Text | Free Full Text

[19] 19. Page MJ, McKenzie J, Bossuyt P, et al.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. 2020. Publisher Full Text

[20] 20. Khamis AM, Kahale LA, Pardo-Hernandez H, et al.: Methods of conduct and reporting of living systematic reviews: a protocol for a living methodological survey [version 2; peer review: 2 approved]. F1000Res. 2019; 8: 221. PubMed Abstract | Publisher Full Text | Free Full Text

[21] 21. Khamis AM, Kahale LA, Pardo-Hernandez HJ, et al.: Search strategies. figshare. Journal contribution. 2019. Publisher Full Text

[22] 22. Akl EA, Kahale LA, Hakoum MB, et al.: Parenteral anticoagulation in ambulatory patients with cancer. Cochrane Database Syst Rev. 2017; 9(9): CD006652. PubMed Abstract | Publisher Full Text | Free Full Text

[23] 23. Kahale LA, Hakoum MB, Tsolakian IG, et al.: Anticoagulation for the long-term treatment of venous thromboembolism in people with cancer. Cochrane Database Syst Rev. 2018; 6(6): CD006650. PubMed Abstract | Publisher Full Text | Free Full Text

[24] 24. Kahale LA, Hakoum MB, Tsolakian IG, et al.: Oral anticoagulation in people with cancer who have no therapeutic or prophylactic indication for anticoagulation. Cochrane Database Syst Rev. 2017; 12(12): CD006466. PubMed Abstract | Publisher Full Text | Free Full Text

[25] 25. Brooker J, Synnot A, McDonald S, et al.: Guidance for the production and publication of Cochrane living systematic reviews: Cochrane reviews in living mode. Cochrane Collaboration. 2019. Reference Source

[26] 26. StataCorp: Stata Statistical Software: Release 13. College Station, Editor. StataCorp LP: TX. 2013.

[27] 27. Harris PA, Taylor R, Thielke R, et al.: Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009; 42(2): 377–81. PubMed Abstract | Publisher Full Text | Free Full Text

[28] 28. Haddaway N: SRflowdiagram: flow charts for systematic reviews and maps. 2020.

[29] 29. Haddaway NR: DynamicSMapResults: template for designing updatable systematic map results text using R Markdown (Version 0.0.1). Zenodo. 2020. Publisher Full Text

[30] 30. Akl EA, Morgan RL, Rooney AA, et al.: Developing trustworthy recommendations as part of an urgent response (1-2 weeks): a GRADE concept paper. J Clin Epidemiol. 2021; 129: 1–11. PubMed Abstract | Publisher Full Text | Free Full Text

[31] 31. Schünemann HJ, Santesso N, Vist GE, et al.: Using GRADE in situations of emergencies and urgencies: certainty in evidence and recommendations matters during the COVID-19 pandemic, now more than ever and no matter what. J Clin Epidemiol. 2020; 127: 202–207. PubMed Abstract | Publisher Full Text | Free Full Text

Tailored PRISMA 2020 flow diagrams for living systematic reviews: a methodological survey and a proposal

Abstract

Keywords

Revised Amendments from Version 2

Introduction

Methods

Box 1. Definition of the different publication types of living reviews

Data handling and analysis

Results

Survey findings

Table 1. General characteristics of the 279 included living publications related to 76 living reviews.

Table 2. Reporting on study flow.

Figure 1. Summary of the four tailored PRISMA flow diagram approaches to present that study to presenting on the study flow for the different search updates.

Proposed tailored PRISMA flow diagram approaches

Figure 2. Snapshot of the ‘cumulative’ tab of the spreadsheet that keep track of all identified records.

Figure 3. Approach 1: presenting the search results of the different versions separately (i.e., base and each update separately).

Figure 4. Approach 2: presenting the search results for the different versions combined (i.e., including base and all update versions).

Figure 5. Approach 3: presenting the search results for the base version separately, and the results of all update version combined.

Figure 6. Approach 4: presenting the results of the latest update version separately, and the results of all previous versions (including the base) combined.

Discussion

Summary

Strengths and limitations

Interpretation of findings

Implications for practice

Implications for future research

Conclusions

Data availability

Underlying data

Acknowledgment

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated