ReviewA retrospective comparison of systematic reviews with same-topic rapid reviews
Introduction
Although systematic reviews (SRs) are a comprehensive form of data synthesis, they represent a significant investment in time [1] and resources [2]. SRs are defined by a structured rigorous approach to obtain, extract, appraise, and synthesize individual studies [2]. A review may take up to 2 years and cost upwards of $100,000 [3]. Rapid reviews (RRs) have arisen as a timelier, cheaper alternative, and are often requested by health care, public health, and governmental organizations [4] to address the needs of clinical, health promotion, preventive medicine, and policy decision-makers [2], [5]. RRs have been found to range from 5 days to 8 months [6]. Tricco et al. describe RRs as “a form of knowledge synthesis in which components of the SR process are simplified or omitted to produce information in a timely manner” [2]. However, a commonly accepted definition does not yet exist in the literature, because, unlike SRs, RRs lack a standard methodological approach [2], leaving uncertainty regarding the validity of their use.
In various scoping reviews examining the application of RRs, researchers have identified studies comparing RRs to SRs [2], [6], [7]. The comparisons have been examined with varying levels of detail and emphasis, and for different sets of studies. This presents an opportunity to consolidate and comprehensively compare RRs with corresponding SRs of the same topic.
Section snippets
Objectives
The objective of our review was to conduct a retrospective comparison of pairs of SRs and RRs on the same topic by examining their characteristics, methods, conclusions, and quality, as well as studies included. An a priori study design and protocol were used to guide our research (Appendices A and B).
Eligibility criteria
The unit of analysis was a SR-RR “pair”. Eligible pairs required:
- •
the review question(s) or objective(s) of both RRs and SRs to be clearly stated;
- •
the review questions/objectives were similarly
Included studies
We identified 12 studies [7], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20] comparing SRs and RRs, comprising 101 SR-RR pairs across a number of different topics. Six studies [10], [11], [12], [13], [14], [15] representing 85 SR-RR pairs were excluded mainly because of insufficient available information and unsuccessful attempts at contacting the authors (Fig. 1 and Appendix C). Six studies had accessible data, and were included in our present study, providing 16 SR-RR pairs
Discussion
As expected, we found that RRs were generally conducted much faster than their SR pairs. Overall, RRs had shorter reports, fewer authors, and were less likely to use an expert panel, which to our knowledge are aspects that have not been directly compared before. In most cases, SRs had more complete reporting compared with RRs. This included the research question, search strategy, whether a protocol was published, as well as details of methods for selection, abstraction, and study appraisal. We
Conclusion
Overall, RRs were generally found to use abbreviated methods compared with their SR counterparts more often, and commonly did not search the grey literature, or use two reviewers to screen and appraise studies or abstract data. However, it was challenging to discern methodological differences because of substantial nonreporting, particularly for RRs, and the retrospective nature of our study. Nonetheless, similar overall conclusions were reported for RRs and SRs with two exceptions. One review
Acknowledgments
In addition to the authors, Alissa Epworth (an experienced librarian) conducted our updated literature search. Aline Chhun, Theshani De Silva, Susan Le, and Krystle Amog formatted the manuscript and tables.
References (49)
Providing guidance to the NHS: The Scottish Medicines Consortium and the National Institute for Clinical Excellence compared
Health Policy
(2006)- et al.
Comparing methods for full versus single technology appraisal: a case study of docetaxel and paclitaxel for early breast cancer
Health Policy
(2008) - et al.
Infrared ear thermometry compared with rectal thermometry in children: a systematic review
Lancet Infect Dis
(2002) - et al.
ROBIS: a new tool to assess risk of bias in systematic reviews was developed
J Clin Epidemiol
(2016) - et al.
Barriers and facilitators to uptake of systematic reviews by policy makers and health care managers: a scoping review
Implement Sci
(2016) - et al.
A scoping review of rapid review methods
BMC Med
(2015) Systematic reviews in the social sciences: a practical guide
(2006)- et al.
Barriers, facilitators, strategies and outcomes to engaging policymakers, healthcare managers and policy analysts in knowledge synthesis: a scoping review protocol
BMJ Open
(2016) - et al.
Evidence summaries: the evolution of a rapid review approach
Syst Rev
(2012) - et al.
EPC Methods: an exploration of methods and context for the production of rapid reviews
(2015)
Rapid versus full systematic reviews: an inventory of current methods and practice in Health Technology Assessment
Methods for Developing Evidence Reviews in Short Periods of Time: A Scoping Review
PLoS One
Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews
BMC Med Res Methodol
Rapid assessments provide acceptable quality advice
Annu Meet Int Soc Technol Assess Health Care Int Soc Technol Assess Health Care Meet
Rapid and responsive health technology assessment: the development and evaluation process in the South and West region of England
J Clin Effectiveness
A rapid systematic review versus a Cochrane systematic review: an empirical comparison. 19th Cochrane Colloquium, Supplement. (Suppl(CD000003))
Health technology appraisal of interventional procedures: comparison of rapid and slow methods
J Health Serv Res Policy
Preliminary study of rapid vs. exhaustive HTA reports: comparison of their usefulness in the decision-making process [abstract]
Medicinal use of potato-derived products: conclusions of a rapid versus full systematic review
Phytother Res
Disagreement in primary study selection between systematic reviews on negative pressure wound therapy
BMC Med Res Methodol
Transcatheter Aortic Valve Implantation: A Critical Appraisal of a Health Technology Assessment and Comparison with a Rapid Review
On Archimedes
Arch Dis Child
Medicinal use of potato-derived products: a systematic review
Phytother Res
Towards evidence based emergency medicine: best BETs from the Manchester Royal Infirmary. BET 2: potato peel dressings for burn wounds
Emerg Med J
Cited by (38)
Model-based security testing in IoT systems: A Rapid Review
2023, Information and Software TechnologyRapid systematic reviews for synthesizing research on built environment
2022, Environmental DevelopmentCurbing plastic consumption: A review of single-use plastic behaviour change interventions
2022, Journal of Cleaner ProductionCitation Excerpt :However, in the absence of sufficient reviews, rapid reviews can also be supplemented by primary studies with high-quality designs, such as randomised controlled trials2 (RCTs). Like systematic reviews, rapid reviews involve systematic, transparent, and reproducible methods and, importantly, have been found to show similar conclusions to traditional systematic reviews on the same topic (Reynen et al., 2018; Tricco et al., 2017). The review protocol was registered with Open Science Framework3 which provides a detail description of the review methods.
Rapid reviews: A critical perspective
2020, Zeitschrift fur Evidenz, Fortbildung und Qualitat im GesundheitswesenChoose your shortcuts wisely: COVID-19 rapid reviews of traditional, complementary and integrative medicine
2020, Integrative Medicine ResearchCitation Excerpt :In RRs any methodological shortcomings will be amplified.35 Trade-offs in comprehensiveness and rigour increase the risk of bias with resultant type 1 (false positive) and type 2 (false negative) errors, and reduce the certainty/quality of the review findings.35 This concept is critical to consider in RRs and particularly for TCIM, as the conclusions of the review are rarely based on the totality of the literature,36 but instead a limited subset, which is therefore more prone to bias.
Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study
2020, Journal of Clinical EpidemiologyCitation Excerpt :Because rapid reviews often can be completed within a few weeks or months, they have emerged as an alternative evidence synthesis tool to systematic reviews, which tend to consume more time, and thus, resources. A potential risk of speeding up the review process by streamlining the methodological approach is that the generated findings could be less reliable than those of the systematic reviews [3,4]. One option to save time and resources is to limit the reviews to English-language publications.
Conflict of interest: The authors have declared that no conflict of interest exists.
Competing interests: A.C.T. is an associate editor, and S.E.S. is on the policy advisory board for the journal; all other authors do not have any conflicts of interest to declare.
Funding: This work was supported by an Ontario Ministry of Research, Innovation, and Science Early Researcher Award that was awarded to A.C.T. S.E.S. is funded by a Tier 1 Canada Research Chair in Knowledge Translation, and A.C.T. is funded by a Tier 2 Canada Research Chair in Knowledge Synthesis. The funders had no role in the design and conduct of the study; data collection, analysis, and interpretation of data; writing of the report; and in the decision to submit the article for publication.
Authors' contributions: E.R., J.I., A.C.T., and B.P. made substantial contributions to the conception and design, acquisition of data, and analysis and interpretation of data. S.E.S. made substantial contributions to conception and design, and analysis and interpretation of data. R.R. made substantial contributions to analysis and interpretation of data, and led the writing of the manuscript. All authors were involved in the drafting and revision of the manuscript and gave final approval of the version submitted for publication.
Authors' information: Dr. E.R. is a third-year internal medicine resident at Queen's University in Kingston, Ontario. Dr. E.R. contributed to this project as part of a research component of her core internal medicine training. She is a graduate of McGill University's medical school. Dr. E.R. also completed an undergraduate and doctorate degree in Pharmacy at the University of Toronto. She practiced as a primary care pharmacist at Two Rivers Family Health Team in Cambridge, Ontario. Dr. E.R. has experience in systematic review, rapid review, and network meta-analysis methodology. She worked as a clinical research officer for the Canadian Agency for Drugs and Technologies in Health. Emily has worked as a research coordinator at the Li Ka Shing Knowledge Institute of St. Michael's Hospital since March of 2013. Mr. R.R. has a Masters of Mathematics from the University of Waterloo and has substantial experience in the design and analysis of clinical trials and health economic data as well as medical writing. Mr. R.R is a biostatistician and part-time research associate at Li Ka Shing Knowledge Institute. Mr. J.I. graduated from the National University of Ireland (Galway) with a BSc in Biochemistry and a MSc in Neuropharmacology. He worked for three years as a clinical research coordinator in the Wound Care Clinic of Dr. R. Gary Sibbald in Mississauga, Ontario and has been working with the Li Ka Shing Institute since January 2013, where he has gained experience in systematic review, rapid review, and network meta-analysis methodologies. Mr. J.H. is a PhD student in the Epidemiology Division at the Dalla Lana School of Public Health, University of Toronto, Toronto, Canada. He is funded by the Frederick Banting and Charles Best Canada Graduate Scholarship Doctoral Award from the Canadian Institutes for Health Research. Dr. B.P. is a research associate, health economist, decision modeler, systematic review methodologist, and biostatistician. Dr. S.E.S. is a geriatrician, clinical epidemiologist and professor, and director of the Knowledge Translation Program of St. Michael's Hospital and of Geriatric Medicine at the University of Toronto. She holds a Tier 1 Canada Research Chair in Knowledge Translation, has been awarded more than $30 million in peer-reviewed grants in the past 10 years and has published >350 peer-reviewed publications. Dr. A.C.T. is a Scientist at the Li Ka Shing Knowledge Institute of St. Michael's Hospital, Associate Professor in the Epidemiology Division of the Dalla Lana School of Public Health, University of Toronto and holds a Tier 2 Canada Research Chair in Knowledge Synthesis and an Ontario Ministry of Research, Innovation, and Science Early Researcher Award.