PRISMA flow diagrams for living systematic reviews: a methodological survey and a proposal

Lara A. Kahale; Rayane Elkhoury; Ibrahim El Mikati; Hector Pardo-Hernandez; Assem M. Khamis; Holger J. Schünemann; Neal R. Haddaway; Elie A. Akl

doi:10.12688/f1000research.51723.1

Home Browse PRISMA flow diagrams for living systematic reviews: a methodological...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

PRISMA flow diagrams for living systematic reviews: a methodological survey and a proposal

[version 1; peer review: awaiting peer review]

Lara A. Kahale¹, Rayane Elkhoury¹, Ibrahim El Mikati², [...] Hector Pardo-Hernandez^3,4, Assem M. Khamis⁵, Holger J. Schünemann^6,7, Neal R. Haddaway^8-10, Elie A. Akl^1,11

Lara A. Kahale¹, Rayane Elkhoury¹, [...] Ibrahim El Mikati², Hector Pardo-Hernandez^3,4, Assem M. Khamis⁵, Holger J. Schünemann^6,7, Neal R. Haddaway^8-10, Elie A. Akl^1,11

PUBLISHED 08 Mar 2021

Author details Author details

¹ Clinical Research Institute, American University of Beirut, Beirut, Riad El Solh 1107 2020, UK
² Faculty of Medicine, American University of Beirut, Beirut, Riad El Solh 1107 2020, Qatar
³ Iberoamerican Cochrane Centre, Sant Pau Biomedical Research Institute, Barcelona, C / Sant Quintí, 77-79 08041, Qatar
⁴ CIBER Epidemiología y Salud Pública, Madrid, Av. de Monforte de Lemos, 5, 28029, Lebanon
⁵ Wolfson Palliative Care Research Centre, Hull York Medical School, University of Hull, Hull, Hull HU6 7RX, Spain
⁶ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, 1280 Main Street West 2C Area, Spain
⁷ Department of Medicine, McMaster University, Hamilton, Ontario, 1280 Main Street West 2C Area, UK
⁸ Stockholm Environment Institute, Stockholm, Linnégatan 87D, Canada
⁹ Mercator Research Institute on Global Commons and Climate Change, Berlin, Torgauer Strasse 19, Canada
¹⁰ Africa Centre for Evidence, University of Johannesburg, Johannesburg, Sweden
¹¹ Department of Internal Medicine, American University of Beirut, Beirut, Riad El Solh 1107 2020, Germany

Lara A. Kahale
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Supervision, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Rayane Elkhoury
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Supervision, Validation, Visualization, Writing – Review & Editing

Ibrahim El Mikati
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Hector Pardo-Hernandez
Roles: Data Curation, Investigation, Resources, Writing – Review & Editing

Assem M. Khamis
Roles: Formal Analysis, Methodology, Software, Validation, Writing – Review & Editing

Holger J. Schünemann
Roles: Writing – Review & Editing

Neal R. Haddaway
Roles: Software, Visualization, Writing – Review & Editing

Elie A. Akl
Roles: Conceptualization, Methodology, Supervision, Visualization, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Research on Research, Policy & Culture gateway.

This article is included in the Living Evidence collection.

Abstract

Background: While the PRISMA flow diagram is widely used for reporting standard systematic reviews (SRs), it was not designed for capturing the results of continual searches for studies in living systematic reviews (LSRs). The objectives of this study are (1) to assess how published LSRs report on the flow of studies through the different phases of the review for the different updates; (2) to propose an approach to reporting on that flow.
Methods: For objective 1, we identified all LSRs published up to July 2020. We abstracted information regarding their general characteristics and how they reported on search results. For objective 2, we based our proposal for tailored PRISMA approaches on the findings from objective 1, as well as on our experience with conducting Cochrane LSRs.
Results: We identified 108 living publications relating to 32 LSRs. Of the 108 publications, 7% were protocols, 24% were base versions (i.e., the first version), 62% were partial updates (i.e., does not include all typical sections of an SR), and 7% were full updates (i.e., includes all typical sections of an SR). We identified six ways to reporting the study flow: base separately, each update separately (38%); numbers not reported (32%); latest update separately, all previous versions combined (20%); base separately, all updates combined (7%); latest update version only (3%); all versions combined (0%). We propose recording in detail the results of the searches to keep track of all identified records. For structuring the flow diagram, we propose using one of four approaches.
Conclusion: We identified six ways for reporting the study flowthrough the different phases of the review for the different update versions. We propose to document in detail the study flow for the different search updates and select one of our four tailored PRISMA diagram approaches to present that study flow.

Keywords

PRISMA statement, living systematic review, update, research methodology research reporting, flow chart, systematic review reporting standards, evidence synthesis, research transparency, research replication

Corresponding author: Elie A. Akl

Competing interests: LAK, HP, AMK, HJS, EAA have been involved in methodological work on living systematic reviews and are authors of living systematic reviews.

Grant information: The author(s) declared that no grants were involved in supporting this work.

Copyright: © 2021 Kahale LA et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Kahale LA, Elkhoury R, El Mikati I et al. PRISMA flow diagrams for living systematic reviews: a methodological survey and a proposal [version 1; peer review: awaiting peer review]. F1000Research 2021, 10:192 (https://doi.org/10.12688/f1000research.51723.1) First published: 08 Mar 2021, 10:192 (https://doi.org/10.12688/f1000research.51723.1) Latest published: 28 Jan 2022, 10:192 (https://doi.org/10.12688/f1000research.51723.3)

Introduction

During the coronavirus disease 2019 (COVID-19) pandemic, health research has proliferated exponentially¹. Systematic reviews are essential to synthesize the evidence and inform policy and practice. Given the pace of research publication, those reviews need to be kept up to date. Living systematic reviews (LSRs) are an emerging type of systematic review that involves the continual search of the literature and incorporation of relevant new evidence, soon after it becomes available². While many evidence synthesis groups are engaged in conducting LSRs or living network meta-analyses, others have developed living databases or living maps, including resources specific for COVID-19 literature^3–17.

An essential component of systematic reviews is to keep track of and report the number of records captured while searching the scientific literature and details of the selection process¹⁸. The PRISMA statement recommends the use of the PRISMA flow diagram to depict the flow of studies through the different phases of the systematic review¹⁹. While the PRISMA flow diagram is a widely used tool for reporting original systematic reviews, it was not designed to capture the results of continual searches typically used in LSRs. Hence, it’s unclear how authors of LSRs address the issue of presenting results of these continual searches.

The objectives of this study were (1) to assess how published LSRs report on the flow of studies through the different phases of the review for the different updates; and (2) to propose an approach to documenting and reporting on the flow of studies through the different phases of a LSR, for the different updates.

Methods

For objective 1, we collected relevant data as part of a larger methodological survey aiming to assess the methods of conduct and reporting of LSRs. We have described the details of that study in a previously published protocol²⁰. Briefly, we identified all living reviews published up to July 2020 available from the following electronic databases: Medline, EMBASE and the Cochrane library (see extended data²¹ of Khamis et al.²⁰ for the search strategy). An eligible living review was either (1) a protocol for an LSR, (2) a base version of an LSR, (3) a full update version of an LSR, (4) a partial update version of an LSR, or (5) a combination of any of these (e.g., one living review may constitute of a protocol, a base version, and a full update version; another living review may constitute of only a Box 1 the definition of each type of living reviews.

Box 1. Definition of the different publication types of living reviews

• LSR protocol: the protocol that describes the planned methods of the living review

• Base version: the first version of the review that follows a living approach

• Full update version: a subsequent version of the review that includes all the typical sections of a systematic review, including an introduction, methods, and results sections. Such a version could stand-alone in terms of content.

• Partial update version: a subsequent version of the review that does not include all the typical sections of a systematic review, but instead refers to a previous version for complementary information. Such a version could not stand-alone in terms of content.

For the current study, we abstracted information about the following features of LSRs:

General characteristics:
- Publication type, i.e., protocol, base version, full update version, partial update version.
- Whether published in the Cochrane library or elsewhere.
- Field (e.g., clinical, public health)
- Whether COVID-19 related or not
- Whether the base version of the living review conducted as a rapid review or not
Reporting on study flow
- Method used to report on the study flow (including the search results and the results of the selection process):
  - ▪ Narrative format and/or flow diagram.
  - ▪ Whether the results of the base and update searches are reported separately or not.
- Type of flow diagram, if applicable (e.g., PRISMA).

For objective 2, we base our proposal for tailored PRISMA diagram approaches on the findings from objective 1, on our experience conducting Cochrane LSRs, and our methodological work on designing and reporting living evidence. Since 2017, our group has been responsible for the first series of three Cochrane LSRs, all of which address anticoagulation in patients with cancer^22–24. We conducted the base search in February 2016. Since then, we have been updating the search on a monthly basis. Through this experience, we have been able to apply and refine the guidance for conducting LSRs endorsed by the living evidence network group²⁵. Specifically, we explored solutions for the reporting of the study flow that would address different scenarios. Our goal was not to be prescriptive and narrow, but rather to cover all possible resulting flows by reviewing the LSRs we identified based on objective 1. Two authors developed a draft of the tailored approaches to presenting the study flow, and then circulated to the author team for review and suggestions for improvement.

Data handling and analysis

We used REDCap to collect and manage the data abstraction process. All data were exported from REDCap and analyzed using Stata v. 13^26,27.

Results

Survey findings

Our search identified a total of 108 living publications relating to 32 LSRs. Table 1 shows their general characteristics. Of the 108 living publications, 8% were protocols, 24% were base versions, 61% were partial updates, and 8% were full updates. The median number of living publications per LSR was 1 (Interquartile range 1–4). Of the 32 living reviews, 31% were published in the Cochrane library, 30% were related to COVID-19, and 15% had a base version published as a rapid review. The majority were related to clinical topics (78%).

Table 1. General characteristics of the 108 included living publications related to 32 living reviews.

	N	n (%)
Publication type	108 living publications
• Protocol		8 (7.4)
• Base version		26 (24.1)
• Partial update version		66 (61.1)
• Full update version		8 (7.4)
Living publications per LSR (Median (IQR)	32 LSRs	1 (1 – 4)
Cochrane LSR	32 LSRs	10 (31.3)
Field	32 LSRs
• Clinical		25 (78.1)
• Public health		5 (15.6)
• Health system and policy		2 (6.3)
COVID-19-related	32 LSRs	10 (30.3)
Base version published as rapid review	26 base versions	4 (15.4)

Abbreviations: LSR: living systematic review; IQR: interquartile range

Table 2 shows the results for the reporting on the study flow. Most base versions and full updates used a flow diagram to report on the search results (96% and 100% respectively), whereas none of the partial updates presented a flow diagram.

Table 2. Reporting on study flow.

	N	n (%)
Inclusion of a flow diagram in^a
• Base version	26 base versions	25 (96.2)
• Partial update version	67 partial updates	1 (1.5)
• Full update version	7 full updates	7 (100.0)
Approach to reporting on study flow for different versions	74 publications of LSRs that included at least one update
Base separately; each update separately		28 (37.8)
Numbers not reported		24 (32.4)
Latest update separately; all previous versions combined (including the base)		15 (20.1)
Base separately; all updates combined		5 (6.8)
Latest update version only		2 (2.7)
All versions combined		0

^a When a flow diagram is not reported, the authors reported on the search results in a narrative format.

Among the 74 publications related to LSRs that included at least one update (12 LSRs) (Figure 1):

38% reported the search results for the base version and for each update version separately.
32% did not report the search results at all (e.g., ‘new studies identified and integrated’ without specifying the number).
20% reported the search results for the latest update version separately and for all previous versions combined (including the base).
3% reported the search results for the latest update version only.
7% reported on the search results for the base version separately and for all update versions combined.
0% reported the search results for all the different versions combined.

Figure 1. Summary of the four PRISMA tailored approaches to present that study to presenting on the study flow for the different search updates.

Proposed tailored PRISMA approaches

Using the approach described in the methods section, we developed four approaches that allow authors to document and report the study flow for the different review update versions of an LSR.

1. Documenting LSR study flow

Authors should record in detail the results of the searches to keep track of all identified records. We propose using a spreadsheet for one LSR at a time. The format we present consists of tabs for each of the respective search sources: bibliographic databases (e.g. MEDLINE, EMBASE, Cochrane databases); conference proceedings; ongoing studies as captured in clinicaltrials.gov and WHO International Clinical Trials Registry Platform (ICTRP); other tabs as needed, and a final ‘cumulative’ tab.

We show in Figure 2 a snapshot of the ‘cumulative’ tab of the spreadsheet that keeps track of all records. It shows the study flow for a hypothetical example for an LSR published first in January 2020 (i.e. base version) and updated on a monthly basis up to August 2020. Each row corresponds to a different update version. The columns present the following information for each update (columns B to E): the number of records received, deduplicated, included at title and abstract screening, and included at full-text screening (i.e., newly included reports). Additional columns (F to I) present the distribution of the newly included reports as relating to either: (1) new studies, (2) previously included studies, (3) ongoing (unpublished) studies, or (4) preprints.

Figure 2. Snapshot of the ‘cumulative’ tab of the spreadsheet that keep track of all identified records.

After manually entering the information in the first five tabs (corresponding to the different search sources) the total is automatically computed in the ‘cumulative’ tab.

2. Reporting LSR study flow

The proposed spreadsheet above can act as a basis for a PRISMA flow diagram for LSRs. For structuring the flow diagram for LSR, one can select one out of four tailored PRISMA approaches:

Approach 1: presenting the search results of the different versions separately (i.e., base and each update separately) (Figure 3).
Approach 2: presenting the search results for the different versions combined (i.e., including base and all update versions) (Figure 4).
Approach 3: presenting the search results for the base version separately, and the results of all update version combined (Figure 5).
Approach 4: presenting the results of the latest update version separately, and the results of all previous versions (including the base) combined (Figure 6).

Figure 3. Approach 1: presenting the search results of the different versions separately (i.e., base and each update separately).

Figure 4. Approach 2: presenting the search results for the different versions combined (i.e., including base and all update versions).

Figure 5. Approach 3: presenting the search results for the base version separately, and the results of all update version combined.

Figure 6. Approach 4: presenting the results of the latest update version separately, and the results of all previous versions (including the base) combined.

In our Cochrane reviews, we applied the second proposal where we present the results for the different searches combined.

Discussion

Summary

This study found that authors of LSRs are not consistent in reporting on the flow of studies through the different phases of the review for the different update versions. Thus, we propose to document in detail the study flow for the different search updates and select one of four tailored PRISMA diagram approaches to present that study flow.

Strengths and limitations

To our knowledge, this is the first methodological survey that assesses how LSR authors report on the flow of studies through the different phases of the review for the different update versions of LSRs. In addition, the research expertise on our team covers both living approach and regular updating of traditional SR. We believe that our assessment forms a vital baseline and allows us to propose best practices for visualization options to improve consistency whilst the production of LSRs is still at a relatively early stage. Indeed, this survey is part of a larger methodological survey aiming to assess the methods of conduct and reporting of LSRs²⁰, that would allow us to update our findings in the future.

Interpretation of findings

Authors tend more towards producing partial updates of LSRs rather than continually updating the full manuscript in a form of a full update. This might seem like a pragmatic approach particularly for a rapidly growing research field and when methods do not seem to change from one update to another. The heterogeneity observed in the ways LSR authors report on the study flow is likely to be explained by the lack of clear guidance on how to do so.

Implications for practice

We built our proposal on the PRISMA 2020 flow diagram and provide four approaches to tailor the needs for continual searchers used in LSR. The fourth approach is the closest to the current PRISMA 2020 flow diagram as it presents the results of the latest update version separately and the results of all previous versions (including the base) combined.

In addition, we proposed three other different approaches to provide options to LSR authors and publishing journals. Whatever approach one decides to follow, for transparency purposes, the systematic reviewers should ideally archive previous versions of the flow diagram (e.g., in an appendix). One major challenge will be to accommodate a large number of updates in the same diagram; some approaches would work better than others in that case. Also, advanced information technology solutions may allow fitting a large number of updates, for example as shown in our web-based prototypes that allow ‘toggling’ between approaches. Once an approach is selected, one may develop an interactive flow diagram that have been designed to facilitate developing and communicating flow diagrams concisely²⁸.

Advanced information technology can also be utilized to simplify updating and tracking the change in all LSR sections including the PRISMA diagram. It would be optimal to develop the base version in a certain platform where all SR and LSR sections are reported as units (i.e., title, authors, background, objectives, inclusion criteria, effect estimate for outcome x). With each update and for every unit, the author has the luxury to keep the same text (if no change has occurred) or edit (if change has occurred). Each unit can be updated in a differential speed based on certain criteria. The edits could be highlighted to visualize the change. For a certain section, one would easily have access to the entries in the previous versions and possibly visualize a trend across the different versions (i.e., cross-sectional view for that specific item). For example, dynamic documents can be developed using ‘R markdown’, a document preparation system, where static text can be combined with in-line code and ‘code chunks’ that produce instantly updatable documents given a modified input²⁹.

Implications for future research

Future research should pilot the proposed approaches for documenting the study flow and for structuring the living flow diagram. In addition, qualitative studies would be helpful to explore: (1) the feasibility and acceptability by LSR authors, publishers, and users towards the proposal; and, (2) what the end-users would like to see in an LSR update.

Conclusions

LSR authors are not consistent in reporting the flow of studies through the different phases of the review for the different update versions. We propose to document in detail the study flow for the different search updates and select one of our four PRISMA diagram approaches to present that study flow. Finally, improving the reporting of study flow in LSR methodology is essential for incorporating living evidence when developing living guidance, particularly in the context of an urgent response^30,31.

Data availability

Underlying data

All data underlying the results are available as part of the article and no additional source data are required.

Acknowledgment

We would like to acknowledge Dr. Mathew Page for his revision for the manuscript.

Faculty Opinions recommended

References

1. Nasrallah AA, Farran SH, Nasrallah ZA, et al.: A large number of COVID-19 interventional clinical trials were registered soon after the pandemic onset: a descriptive analysis. J Clin Epidemiol. 2020; 125: 170–178. PubMed Abstract | Publisher Full Text | Free Full Text
2. Elliott JH, Turner T, Clavisi O, et al.: Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014; 11(2): e1001603. PubMed Abstract | Publisher Full Text | Free Full Text
3. John A, Okolie C, Eyles E, et al.: The impact of the COVID-19 pandemic on self-harm and suicidal behaviour: protocol for a living systematic review [version 1; peer review: 1 approved, 1 approved with reservations]. F1000Res. 2020; 9: 1097. Publisher Full Text
4. Currie G, Macleod M, Sena E, et al.: Protocol for a “living” evidence summary of primary research related to Covid-19. 2020. Publisher Full Text
5. Geisler BP, Zahabi L, Lang AE, et al.: Repurposing Existing Medications for Coronavirus Disease 2019: Protocol for a Rapid and Living Systematic Review. medRxiv. 2020. Publisher Full Text
6. Juul S, Nielsen N, Bentzer P, et al.: Interventions for treatment of COVID-19: a protocol for a living systematic review with network meta-analysis including individual patient data (The LIVING Project). Syst Rev. 2020; 9(1): 108. PubMed Abstract | Publisher Full Text | Free Full Text
7. Maguire BJ, Guérin PJ: A living systematic review protocol for COVID-19 clinical trial registrations [version 1; peer review: 2 approved]. Wellcome Open Res. 2020; 5; 60. PubMed Abstract | Publisher Full Text | Free Full Text
8. Maguire BJ, McLean ARD, Rashan S, et al.: Baseline results of a living systematic review for COVID-19 clinical trial registrations [version 1; peer review: 2 approved]. Wellcome Open Res. 2020; 5: 116. PubMed Abstract | Publisher Full Text | Free Full Text
9. Schünemann HJ, Khabsa J, Solo K, et al.: Ventilation techniques and risk for transmission of coronavirus disease, including COVID-19: a living systematic review of multiple streams of evidence. Ann Intern Med. 2020; 173(3): 204–216. PubMed Abstract | Publisher Full Text | Free Full Text
10. Ortiz-Muñoz LE, Ferrer BM, Duarte-Anselmi G, et al.: Protocol of a Living systematic review: Gloves for the prevention of COVID-19 in healthy population. 2020. Publisher Full Text
11. Siemieniuk RA, Bartoszko JJ, Ge L, et al.: Drug treatments for covid-19: living systematic review and network meta-analysis. BMJ. 2020; 370: m2980. PubMed Abstract | Publisher Full Text | Free Full Text
12. Rada G, Verdugo-Paiva F, Ávila C, et al.: Evidence synthesis relevant to COVID-19: a protocol for multiple systematic reviews and overviews of systematic reviews. Medwave. 2020; 20(3): e7868. PubMed Abstract | Publisher Full Text
13. Cochrane: Coronavirus (COVID-19) - Cochrane resources and news. The Cochrane Collaboration. 2019. Reference Source
14. World Health Organization: Global research on coronavirus disease (COVID-19). 2020. Reference Source
15. Living Overview of Evidence (LOVE): Coronavirus disease (COVID‑19). Epistemonikos foundation. 2020. Reference Source
16. EPPI center: COVID-19: a living systematic map of the evidence. 2020. Reference Source
17. Boutron I, et al.: Interventions for preventing and treating COVID-19: protocol for a living mapping of research and a living systematic review. Syst Rev. 2020; 9: 108.
18. Millard T, Synnot A, Elliott J, et al.: Feasibility and acceptability of living systematic reviews: results from a mixed-methods evaluation. Syst Rev. 2019; 8(1): 325. PubMed Abstract | Publisher Full Text | Free Full Text
19. Page MJ, McKenzie M, Bossuyt P, et al.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. 2020. Publisher Full Text
20. Khamis AM, Kahale LA, Pardo-Hernandez H, et al.: Methods of conduct and reporting of living systematic reviews: a protocol for a living methodological survey [version 2; peer review: 2 approved]. F1000Res. 2019; 8: 221. PubMed Abstract | Publisher Full Text | Free Full Text
21. Khamis AM, Kahale LA, Pardo-Hernandez HJ, et al.: Search strategies. figshare. Journal contribution. 2019. Publisher Full Text
22. Akl EA, Kahale LA, Hakoum MB, et al.: Parenteral anticoagulation in ambulatory patients with cancer. Cochrane Database Syst Rev. 2017; 9(9): CD006652. PubMed Abstract | Publisher Full Text | Free Full Text
23. Kahale LA, Hakoum MB, Tsolakian IG, et al.: Anticoagulation for the long-term treatment of venous thromboembolism in people with cancer. Cochrane Database Syst Rev. 2018; 6(6): CD006650. PubMed Abstract | Publisher Full Text | Free Full Text
24. Kahale LA, Hakoum MB, Tsolakian IG, et al.: Oral anticoagulation in people with cancer who have no therapeutic or prophylactic indication for anticoagulation. Cochrane Database Syst Rev. 2017; 12(12): CD006466. PubMed Abstract | Publisher Full Text | Free Full Text
25. Brooker J, Synnot A, McDonald S, et al.: Guidance for the production and publication of Cochrane living systematic reviews: Cochrane reviews in living mode. Cochrane Collaboration. 2019. Reference Source
26. StataCorp: Stata Statistical Software: Release 13. College Station, Editor. StataCorp LP: TX. 2013.
27. Harris PA, Taylor R, Thielke R, et al.: Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009; 42(2): 377–81. PubMed Abstract | Publisher Full Text | Free Full Text
28. Haddaway N: SRflowdiagram: flow charts for systematic reviews and maps. 2020.
29. Haddaway NR: DynamicSMapResults: template for designing updatable systematic map results text using R Markdown (Version 0.0.1). Zenodo. 2020. Publisher Full Text
30. Akl EA, Morgan RL, Rooney AA, et al.: Developing trustworthy recommendations as part of an urgent response (1-2 weeks): a GRADE concept paper. J Clin Epidemiol. 2020; 129: 1–11. PubMed Abstract | Publisher Full Text | Free Full Text
31. Schünemann HJ, Santesso N, Vist GE, et al.: Using GRADE in situations of emergencies and urgencies: certainty in evidence and recommendations matters during the COVID-19 pandemic, now more than ever and no matter what. J Clin Epidemiol. 2020; 127: 202–207. PubMed Abstract | Publisher Full Text | Free Full Text

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 08 Mar 2021

Author details Author details

Rayane Elkhoury
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Supervision, Validation, Visualization, Writing – Review & Editing

Ibrahim El Mikati
Roles: Data Curation, Investigation, Resources, Validation, Writing – Review & Editing

Hector Pardo-Hernandez
Roles: Data Curation, Investigation, Resources, Writing – Review & Editing

Assem M. Khamis
Roles: Formal Analysis, Methodology, Software, Validation, Writing – Review & Editing

Holger J. Schünemann
Roles: Writing – Review & Editing

Neal R. Haddaway
Roles: Software, Visualization, Writing – Review & Editing

Elie A. Akl
Roles: Conceptualization, Methodology, Supervision, Visualization, Writing – Review & Editing

Competing interests

LAK, HP, AMK, HJS, EAA have been involved in methodological work on living systematic reviews and are authors of living systematic reviews.

Grant information

The author(s) declared that no grants were involved in supporting this work.

Article Versions (3)

version 3

Revised

Published: 28 Jan 2022, 10:192

https://doi.org/10.12688/f1000research.51723.3

version 2

Revised

Published: 18 Mar 2021, 10:192

https://doi.org/10.12688/f1000research.51723.2

version 1

Published: 08 Mar 2021, 10:192

https://doi.org/10.12688/f1000research.51723.1

© 2021 Kahale LA et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Kahale LA, Elkhoury R, El Mikati I et al. PRISMA flow diagrams for living systematic reviews: a methodological survey and a proposal [version 1; peer review: awaiting peer review] F1000Research 2021, 10:192 (https://doi.org/10.12688/f1000research.51723.1)

NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 08 Mar 2021

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 3 (revision) 28 Jan 22	read
Version 2 (revision) 18 Mar 21	read	read
Version 1 08 Mar 21

Lex Bouter, Amsterdam Universities Medical Center, Amsterdam, The Netherlands; Vrije Universiteit, Amsterdam, The Netherlands
Sonia Hines, Flinders University, Alice Springs, Australia

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

8 Views

31 Jan 2022 | for Version 3

Lex Bouter, Department of Epidemiology and Data Science, Amsterdam Universities Medical Center, Amsterdam, The Netherlands; Department of Philosophy, Faculty of Humanities, Vrije Universiteit, Amsterdam, The Netherlands

8 Views Cite this report Responses(0)

Approved

I applaud the authors for updating their survey and for more than double the sample size. They also made some other - rather marginal - changes and explain that their publication is only the first of a series based on an ongoing project.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Methodology, epidemiology, research integrity, open science, systematic review methods.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

11 Views

22 Jul 2021 | for Version 2

Sonia Hines, College of Medicine and Public Health, Flinders University, Alice Springs, Northern Territory, Australia

11 Views Cite this report Responses(1)

Approved

This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable.

I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point.

The authors have done what they set out to do, but I suggest further work is needed before a recommended method of study flow reporting is settled upon.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Systematic reviews, evidence-based practice.

Respond to this report

Responses (1)

Author Response

28 Jan 2022

Lara Kahale, Clinical Research Institute, American University of Beirut, Beirut, Riad El Solh 1107 2020, UK

The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 2: Sonia Hines

1. This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable. I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point.

Thank you for your positive feedback and practical advice. We agree that spreadsheets are not easily publishable, that is why we recommend using it as a tool for the LSR authors to document in detail the study flow as opposed to using it as a reporting tool. We now elaborate in the discussion how this spreadsheet would feed in an app (shinyapp) that creates nicely designed flows for these four approaches well suited for insertion into a publication.

2. The authors have done what they set out to do, but I suggest further work is needed before a recommended method of study flow reporting is settled upon.

We agree with this comment. This study is a methodological survey aiming to summarize what LSR authors are currently reporting. As this is not a guidance document, we are not making any recommendations. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs. For that purpose, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in a Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

17 Views

28 Jun 2021 | for Version 2

17 Views Cite this report Responses(1)

Approved With Reservations

Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there’re some issues that should to be solved in the next version of the manuscript.

Major issues

It’s disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like ‘anything goes’. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.
The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest.
The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals?
The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

No
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Methodology, epidemiology, research integrity, open science, systematic review methods.

Respond to this report

Responses (1)

Author Response

28 Jan 2022

Lara Kahale, Clinical Research Institute, American University of Beirut, Beirut, Riad El Solh 1107 2020, UK

The Reviewers’ comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript.

Reviewer 1: Lex Bouter

Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there’re some issues that should to be solved in the next version of the manuscript.

Major issues

1. It’s disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like ‘anything goes’. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.

Thank you for your comment. This study is a methodological survey aiming to summarize what current LSR authors are reporting. Given that the full methodology of living systematic reviews is still emerging, we prefer not to make firm recommendations but lay out options. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs (please see registration form on EQUATOR network website: Equator Network. PRISMA for LSR – Extension of PRISMA 2020 for living systematic reviews. 2021; Accessed from https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/#LSR]) . Consistently with the Reviewer’s suggestions, we will be following the EQUATOR Network’s guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders. We have elaborated about this research under implications for future research.

2. The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest.

Thank you for your query. This paper is the first published study within the larger project which aims to explore how LSRs authors are currently reporting, conducting, and publishing LSRs. We cite the protocol of that project (Khamis 2019). This study focuses on reporting flow diagrams, another study will focus on general characteristics and delay in updating LSRs, and a third study will focus on methodological features of LSRs. We have now updated our results to include data from up to April 2021, and our dataset now includes 279 publications of 76 LSRs.

3. The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals?

Thank you for your comment. Unfortunately, that was the state of the science. However, as noted in the previous comment, we have now updated our search to include results up to April 2021 Including data from 279 publications of 76 LSRs. We avoided including confidence intervals for the simplicity of presentation and given the small sample size. We are happy to include these if the editor prefers so.

4. The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review.

Thank you for your suggestion. We have checked and found the following (now added to the text): ‘None of the 279 living publications reported in their methods section how they plan to report on the study flow’.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Nasrallah AA, Farran SH, Nasrallah ZA, et al.: A large number of COVID-19 interventional clinical trials were registered soon after the pandemic onset: a descriptive analysis. J Clin Epidemiol. 2020; 125: 170–178. PubMed Abstract | Publisher Full Text | Free Full Text

[2] 2. Elliott JH, Turner T, Clavisi O, et al.: Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014; 11(2): e1001603. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. John A, Okolie C, Eyles E, et al.: The impact of the COVID-19 pandemic on self-harm and suicidal behaviour: protocol for a living systematic review [version 1; peer review: 1 approved, 1 approved with reservations]. F1000Res. 2020; 9: 1097. Publisher Full Text

[4] 4. Currie G, Macleod M, Sena E, et al.: Protocol for a “living” evidence summary of primary research related to Covid-19. 2020. Publisher Full Text

[5] 5. Geisler BP, Zahabi L, Lang AE, et al.: Repurposing Existing Medications for Coronavirus Disease 2019: Protocol for a Rapid and Living Systematic Review. medRxiv. 2020. Publisher Full Text

[6] 6. Juul S, Nielsen N, Bentzer P, et al.: Interventions for treatment of COVID-19: a protocol for a living systematic review with network meta-analysis including individual patient data (The LIVING Project). Syst Rev. 2020; 9(1): 108. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Maguire BJ, Guérin PJ: A living systematic review protocol for COVID-19 clinical trial registrations [version 1; peer review: 2 approved]. Wellcome Open Res. 2020; 5; 60. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Maguire BJ, McLean ARD, Rashan S, et al.: Baseline results of a living systematic review for COVID-19 clinical trial registrations [version 1; peer review: 2 approved]. Wellcome Open Res. 2020; 5: 116. PubMed Abstract | Publisher Full Text | Free Full Text

[9] 9. Schünemann HJ, Khabsa J, Solo K, et al.: Ventilation techniques and risk for transmission of coronavirus disease, including COVID-19: a living systematic review of multiple streams of evidence. Ann Intern Med. 2020; 173(3): 204–216. PubMed Abstract | Publisher Full Text | Free Full Text

[10] 10. Ortiz-Muñoz LE, Ferrer BM, Duarte-Anselmi G, et al.: Protocol of a Living systematic review: Gloves for the prevention of COVID-19 in healthy population. 2020. Publisher Full Text

[11] 11. Siemieniuk RA, Bartoszko JJ, Ge L, et al.: Drug treatments for covid-19: living systematic review and network meta-analysis. BMJ. 2020; 370: m2980. PubMed Abstract | Publisher Full Text | Free Full Text

[12] 12. Rada G, Verdugo-Paiva F, Ávila C, et al.: Evidence synthesis relevant to COVID-19: a protocol for multiple systematic reviews and overviews of systematic reviews. Medwave. 2020; 20(3): e7868. PubMed Abstract | Publisher Full Text

[13] 13. Cochrane: Coronavirus (COVID-19) - Cochrane resources and news. The Cochrane Collaboration. 2019. Reference Source

[14] 14. World Health Organization: Global research on coronavirus disease (COVID-19). 2020. Reference Source

[15] 15. Living Overview of Evidence (LOVE): Coronavirus disease (COVID‑19). Epistemonikos foundation. 2020. Reference Source

[16] 16. EPPI center: COVID-19: a living systematic map of the evidence. 2020. Reference Source

[17] 17. Boutron I, et al.: Interventions for preventing and treating COVID-19: protocol for a living mapping of research and a living systematic review. Syst Rev. 2020; 9: 108.

[18] 18. Millard T, Synnot A, Elliott J, et al.: Feasibility and acceptability of living systematic reviews: results from a mixed-methods evaluation. Syst Rev. 2019; 8(1): 325. PubMed Abstract | Publisher Full Text | Free Full Text

[19] 19. Page MJ, McKenzie M, Bossuyt P, et al.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. 2020. Publisher Full Text

[20] 20. Khamis AM, Kahale LA, Pardo-Hernandez H, et al.: Methods of conduct and reporting of living systematic reviews: a protocol for a living methodological survey [version 2; peer review: 2 approved]. F1000Res. 2019; 8: 221. PubMed Abstract | Publisher Full Text | Free Full Text

[21] 21. Khamis AM, Kahale LA, Pardo-Hernandez HJ, et al.: Search strategies. figshare. Journal contribution. 2019. Publisher Full Text

[22] 22. Akl EA, Kahale LA, Hakoum MB, et al.: Parenteral anticoagulation in ambulatory patients with cancer. Cochrane Database Syst Rev. 2017; 9(9): CD006652. PubMed Abstract | Publisher Full Text | Free Full Text

[23] 23. Kahale LA, Hakoum MB, Tsolakian IG, et al.: Anticoagulation for the long-term treatment of venous thromboembolism in people with cancer. Cochrane Database Syst Rev. 2018; 6(6): CD006650. PubMed Abstract | Publisher Full Text | Free Full Text

[24] 24. Kahale LA, Hakoum MB, Tsolakian IG, et al.: Oral anticoagulation in people with cancer who have no therapeutic or prophylactic indication for anticoagulation. Cochrane Database Syst Rev. 2017; 12(12): CD006466. PubMed Abstract | Publisher Full Text | Free Full Text

[25] 25. Brooker J, Synnot A, McDonald S, et al.: Guidance for the production and publication of Cochrane living systematic reviews: Cochrane reviews in living mode. Cochrane Collaboration. 2019. Reference Source

[26] 26. StataCorp: Stata Statistical Software: Release 13. College Station, Editor. StataCorp LP: TX. 2013.

[27] 27. Harris PA, Taylor R, Thielke R, et al.: Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009; 42(2): 377–81. PubMed Abstract | Publisher Full Text | Free Full Text

[28] 28. Haddaway N: SRflowdiagram: flow charts for systematic reviews and maps. 2020.

[29] 29. Haddaway NR: DynamicSMapResults: template for designing updatable systematic map results text using R Markdown (Version 0.0.1). Zenodo. 2020. Publisher Full Text

[30] 30. Akl EA, Morgan RL, Rooney AA, et al.: Developing trustworthy recommendations as part of an urgent response (1-2 weeks): a GRADE concept paper. J Clin Epidemiol. 2020; 129: 1–11. PubMed Abstract | Publisher Full Text | Free Full Text

[31] 31. Schünemann HJ, Santesso N, Vist GE, et al.: Using GRADE in situations of emergencies and urgencies: certainty in evidence and recommendations matters during the COVID-19 pandemic, now more than ever and no matter what. J Clin Epidemiol. 2020; 127: 202–207. PubMed Abstract | Publisher Full Text | Free Full Text

PRISMA flow diagrams for living systematic reviews: a methodological survey and a proposal

Abstract

Keywords

Introduction

Methods

Box 1. Definition of the different publication types of living reviews

Data handling and analysis

Results

Survey findings

Table 1. General characteristics of the 108 included living publications related to 32 living reviews.

Table 2. Reporting on study flow.

Figure 1. Summary of the four PRISMA tailored approaches to present that study to presenting on the study flow for the different search updates.

Proposed tailored PRISMA approaches

Figure 2. Snapshot of the ‘cumulative’ tab of the spreadsheet that keep track of all identified records.

Figure 3. Approach 1: presenting the search results of the different versions separately (i.e., base and each update separately).

Figure 4. Approach 2: presenting the search results for the different versions combined (i.e., including base and all update versions).

Figure 5. Approach 3: presenting the search results for the base version separately, and the results of all update version combined.

Figure 6. Approach 4: presenting the results of the latest update version separately, and the results of all previous versions (including the base) combined.

Discussion

Summary

Strengths and limitations

Interpretation of findings

Implications for practice

Implications for future research

Conclusions

Data availability

Underlying data

Acknowledgment

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated