Review
A retrospective comparison of systematic reviews with same-topic rapid reviews

https://doi.org/10.1016/j.jclinepi.2017.12.001Get rights and content

Abstract

Objective

To compare rapid reviews (RRs) to same-topic systematic reviews (SRs) for methods, studies included, and conclusions.

Study Design and Setting

A retrospective comparison of studies comparing RRs and SRs by searching four scoping reviews published between 2007 and 2016. Reports were included if literature searches were conducted within 24 months of each other and had common research questions. Reviews were compared for duration, studies included, population, intervention, comparisons, outcomes, study designs, quality, methods, and conclusions.

Results

Six studies containing 16 review pairs were included, covering nine topics. Overall, RRs used abbreviated methods more often: no search of grey literature, employing one reviewer to screen studies, engaging fewer experts, including fewer studies, and providing shorter reports, with poorer reporting quality and faster completion. Reviews reported similar conclusions, with two exceptions: one SR did not include a key study; separately, two RRs failed to highlight an association with early mortality identified by the SR. RRs tended to provide less detail and fewer considerations.

Conclusion

RRs used several methodological shortcuts compared with SRs on the same topic. It was challenging to discern methodological differences because of retrospective assessment and substantial nonreporting, particularly for RRs.

Introduction

Although systematic reviews (SRs) are a comprehensive form of data synthesis, they represent a significant investment in time [1] and resources [2]. SRs are defined by a structured rigorous approach to obtain, extract, appraise, and synthesize individual studies [2]. A review may take up to 2 years and cost upwards of $100,000 [3]. Rapid reviews (RRs) have arisen as a timelier, cheaper alternative, and are often requested by health care, public health, and governmental organizations [4] to address the needs of clinical, health promotion, preventive medicine, and policy decision-makers [2], [5]. RRs have been found to range from 5 days to 8 months [6]. Tricco et al. describe RRs as “a form of knowledge synthesis in which components of the SR process are simplified or omitted to produce information in a timely manner” [2]. However, a commonly accepted definition does not yet exist in the literature, because, unlike SRs, RRs lack a standard methodological approach [2], leaving uncertainty regarding the validity of their use.

In various scoping reviews examining the application of RRs, researchers have identified studies comparing RRs to SRs [2], [6], [7]. The comparisons have been examined with varying levels of detail and emphasis, and for different sets of studies. This presents an opportunity to consolidate and comprehensively compare RRs with corresponding SRs of the same topic.

Section snippets

Objectives

The objective of our review was to conduct a retrospective comparison of pairs of SRs and RRs on the same topic by examining their characteristics, methods, conclusions, and quality, as well as studies included. An a priori study design and protocol were used to guide our research (Appendices A and B).

Eligibility criteria

The unit of analysis was a SR-RR “pair”. Eligible pairs required:

  • the review question(s) or objective(s) of both RRs and SRs to be clearly stated;

  • the review questions/objectives were similarly

Included studies

We identified 12 studies [7], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20] comparing SRs and RRs, comprising 101 SR-RR pairs across a number of different topics. Six studies [10], [11], [12], [13], [14], [15] representing 85 SR-RR pairs were excluded mainly because of insufficient available information and unsuccessful attempts at contacting the authors (Fig. 1 and Appendix C). Six studies had accessible data, and were included in our present study, providing 16 SR-RR pairs

Discussion

As expected, we found that RRs were generally conducted much faster than their SR pairs. Overall, RRs had shorter reports, fewer authors, and were less likely to use an expert panel, which to our knowledge are aspects that have not been directly compared before. In most cases, SRs had more complete reporting compared with RRs. This included the research question, search strategy, whether a protocol was published, as well as details of methods for selection, abstraction, and study appraisal. We

Conclusion

Overall, RRs were generally found to use abbreviated methods compared with their SR counterparts more often, and commonly did not search the grey literature, or use two reviewers to screen and appraise studies or abstract data. However, it was challenging to discern methodological differences because of substantial nonreporting, particularly for RRs, and the retrospective nature of our study. Nonetheless, similar overall conclusions were reported for RRs and SRs with two exceptions. One review

Acknowledgments

In addition to the authors, Alissa Epworth (an experienced librarian) conducted our updated literature search. Aline Chhun, Theshani De Silva, Susan Le, and Krystle Amog formatted the manuscript and tables.

References (49)

  • A. Cameron et al.

    Rapid versus full systematic reviews: an inventory of current methods and practice in Health Technology Assessment

    (2007)
  • A.M. Abou-Setta et al.

    Methods for Developing Evidence Reviews in Short Periods of Time: A Scoping Review

    PLoS One

    (2016)
  • B.J. Shea et al.

    Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews

    BMC Med Res Methodol

    (2007)
  • P. Corabian et al.

    Rapid assessments provide acceptable quality advice

    Annu Meet Int Soc Technol Assess Health Care Int Soc Technol Assess Health Care Meet

    (2002)
  • L. Best et al.

    Rapid and responsive health technology assessment: the development and evaluation process in the South and West region of England

    J Clin Effectiveness

    (1997)
  • M. Rizzo et al.

    A rapid systematic review versus a Cochrane systematic review: an empirical comparison. 19th Cochrane Colloquium, Supplement. (Suppl(CD000003))

    (2011)
  • V. Warren

    Health technology appraisal of interventional procedures: comparison of rapid and slow methods

    J Health Serv Res Policy

    (2007)
  • Z. Saz Parkinson et al.

    Preliminary study of rapid vs. exhaustive HTA reports: comparison of their usefulness in the decision-making process [abstract]

    (2010)
  • S. Van de Velde et al.

    Medicinal use of potato-derived products: conclusions of a rapid versus full systematic review

    Phytother Res

    (2011)
  • F. Peinemann et al.

    Disagreement in primary study selection between systematic reviews on negative pressure wound therapy

    BMC Med Res Methodol

    (2008)
  • Health Technology Inquiry Service (HTIS)

    Transcatheter Aortic Valve Implantation: A Critical Appraisal of a Health Technology Assessment and Comparison with a Rapid Review

    (2010)
  • T. Lopez

    On Archimedes

    Arch Dis Child

    (2003)
  • J.E. Vlachojannis et al.

    Medicinal use of potato-derived products: a systematic review

    Phytother Res

    (2010)
  • E. De Buck et al.

    Towards evidence based emergency medicine: best BETs from the Manchester Royal Infirmary. BET 2: potato peel dressings for burn wounds

    Emerg Med J

    (2010)
  • Cited by (38)

    • Model-based security testing in IoT systems: A Rapid Review

      2023, Information and Software Technology
    • Curbing plastic consumption: A review of single-use plastic behaviour change interventions

      2022, Journal of Cleaner Production
      Citation Excerpt :

      However, in the absence of sufficient reviews, rapid reviews can also be supplemented by primary studies with high-quality designs, such as randomised controlled trials2 (RCTs). Like systematic reviews, rapid reviews involve systematic, transparent, and reproducible methods and, importantly, have been found to show similar conclusions to traditional systematic reviews on the same topic (Reynen et al., 2018; Tricco et al., 2017). The review protocol was registered with Open Science Framework3 which provides a detail description of the review methods.

    • Rapid reviews: A critical perspective

      2020, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen
    • Choose your shortcuts wisely: COVID-19 rapid reviews of traditional, complementary and integrative medicine

      2020, Integrative Medicine Research
      Citation Excerpt :

      In RRs any methodological shortcomings will be amplified.35 Trade-offs in comprehensiveness and rigour increase the risk of bias with resultant type 1 (false positive) and type 2 (false negative) errors, and reduce the certainty/quality of the review findings.35 This concept is critical to consider in RRs and particularly for TCIM, as the conclusions of the review are rarely based on the totality of the literature,36 but instead a limited subset, which is therefore more prone to bias.

    • Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study

      2020, Journal of Clinical Epidemiology
      Citation Excerpt :

      Because rapid reviews often can be completed within a few weeks or months, they have emerged as an alternative evidence synthesis tool to systematic reviews, which tend to consume more time, and thus, resources. A potential risk of speeding up the review process by streamlining the methodological approach is that the generated findings could be less reliable than those of the systematic reviews [3,4]. One option to save time and resources is to limit the reviews to English-language publications.

    View all citing articles on Scopus

    Conflict of interest: The authors have declared that no conflict of interest exists.

    Competing interests: A.C.T. is an associate editor, and S.E.S. is on the policy advisory board for the journal; all other authors do not have any conflicts of interest to declare.

    Funding: This work was supported by an Ontario Ministry of Research, Innovation, and Science Early Researcher Award that was awarded to A.C.T. S.E.S. is funded by a Tier 1 Canada Research Chair in Knowledge Translation, and A.C.T. is funded by a Tier 2 Canada Research Chair in Knowledge Synthesis. The funders had no role in the design and conduct of the study; data collection, analysis, and interpretation of data; writing of the report; and in the decision to submit the article for publication.

    Authors' contributions: E.R., J.I., A.C.T., and B.P. made substantial contributions to the conception and design, acquisition of data, and analysis and interpretation of data. S.E.S. made substantial contributions to conception and design, and analysis and interpretation of data. R.R. made substantial contributions to analysis and interpretation of data, and led the writing of the manuscript. All authors were involved in the drafting and revision of the manuscript and gave final approval of the version submitted for publication.

    Authors' information: Dr. E.R. is a third-year internal medicine resident at Queen's University in Kingston, Ontario. Dr. E.R. contributed to this project as part of a research component of her core internal medicine training. She is a graduate of McGill University's medical school. Dr. E.R. also completed an undergraduate and doctorate degree in Pharmacy at the University of Toronto. She practiced as a primary care pharmacist at Two Rivers Family Health Team in Cambridge, Ontario. Dr. E.R. has experience in systematic review, rapid review, and network meta-analysis methodology. She worked as a clinical research officer for the Canadian Agency for Drugs and Technologies in Health. Emily has worked as a research coordinator at the Li Ka Shing Knowledge Institute of St. Michael's Hospital since March of 2013. Mr. R.R. has a Masters of Mathematics from the University of Waterloo and has substantial experience in the design and analysis of clinical trials and health economic data as well as medical writing. Mr. R.R is a biostatistician and part-time research associate at Li Ka Shing Knowledge Institute. Mr. J.I. graduated from the National University of Ireland (Galway) with a BSc in Biochemistry and a MSc in Neuropharmacology. He worked for three years as a clinical research coordinator in the Wound Care Clinic of Dr. R. Gary Sibbald in Mississauga, Ontario and has been working with the Li Ka Shing Institute since January 2013, where he has gained experience in systematic review, rapid review, and network meta-analysis methodologies. Mr. J.H. is a PhD student in the Epidemiology Division at the Dalla Lana School of Public Health, University of Toronto, Toronto, Canada. He is funded by the Frederick Banting and Charles Best Canada Graduate Scholarship Doctoral Award from the Canadian Institutes for Health Research. Dr. B.P. is a research associate, health economist, decision modeler, systematic review methodologist, and biostatistician. Dr. S.E.S. is a geriatrician, clinical epidemiologist and professor, and director of the Knowledge Translation Program of St. Michael's Hospital and of Geriatric Medicine at the University of Toronto. She holds a Tier 1 Canada Research Chair in Knowledge Translation, has been awarded more than $30 million in peer-reviewed grants in the past 10 years and has published >350 peer-reviewed publications. Dr. A.C.T. is a Scientist at the Li Ka Shing Knowledge Institute of St. Michael's Hospital, Associate Professor in the Epidemiology Division of the Dalla Lana School of Public Health, University of Toronto and holds a Tier 2 Canada Research Chair in Knowledge Synthesis and an Ontario Ministry of Research, Innovation, and Science Early Researcher Award.

    View full text