ReviewHigh quality of the evidence for medical and other health-related interventions was uncommon in Cochrane systematic reviews
Introduction
The advent of evidence-based medicine has sensitized us to the need for high quality of evidence to inform clinical and public health practice, but also to the fact that such evidence is often missing in many fields [1]. With the accumulation of vast numbers of systematic reviews (SRs), one can assess systematically the current state of the evidence across very diverse specialties and medical or other health-related interventions.
Since its inception in 1993, the Cochrane Database of Systematic Reviews (CDSRs) has become established as a comprehensive resource for evidence. The database has expanded to include over 50 review groups [2] encompassing diverse areas. Cochrane reviews typically have higher methodological and reporting quality than SRs published elsewhere [3], [4], [5]. Moreover, they are often instrumental in developing policy documents and practice guidelines [6].
Various tools have been used to appraise the quality of a body of evidence within an SR [7], [8], [9]. The Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) tool can allow such a systematic appraisal, and it has been adopted by Cochrane [10]. The GRADE approach involves assimilation of the quality of the available evidence from SRs in addition to the values and preferences of the patients, safety, and costs. The quality of evidence for individual outcomes is scored for specific domains and an overall score assigned (high, moderate, low, and very low) based on five factors: risk of bias (study limitations), inconsistency, indirectness, imprecision, and publication bias and presented in summary of findings (SoF) tables. This approach has been validated for the assessment of the strength of a body of evidence [11].
The quality of the evidence has been addressed in small samples of SRs in focused areas [12], [13], but to our knowledge, an appraisal of the quality of evidence has not yet been undertaken on reviews within the CDSR across diverse topics. Moreover, it would be interesting to explore how this strength of evidence relates to the probability that authors would interpret that an intervention may be of value. Therefore, to answer these questions, we evaluated a large sample of all recent SRs in the CDSR that had used the GRADE tool to generate SoF tables.
Section snippets
Data sources and searches
Both new and updated versions of all SRs published over an 18-month period from January 1, 2013, to June 30, 2014, were identified on the CDSRs.
Study selection and data extraction
Two authors (P.S.F. and N.P.) retrieved SRs for inclusion in the review by accessing the CDSR (www.thecochranelibrary.com). Full text reports were uploaded onto a shared electronic folder. Data from the identified reviews were extracted independently and entered on prepiloted standardized forms by two authors (P.S.F. and D.K.) after the initial
Database search results
One thousand three hundred ninety-four SRs were identified in the 18-month period across 50 Cochrane research groups. Of these, 608 (43.6%) incorporated GRADE assessments with at least one SoF table. Two hundred four reviews (33.6%) had two or more SoF tables. In this study, the focus was only on the first outcome of the first SoF table (Fig. 1). Of these reviews with GRADE assessments, 316 (52%) were new and 292 (48%) were updates.
Characteristics of included interventions and outcomes
Of the 608 eligible SoF tables, over half (56.5%) related to
Main findings
Our survey of the quality of evidence underpinning recent SRs published in the CDSR found that almost half of them had GRADE evaluations. Among them, we noted a large percentage of conclusions are based on low or very low quality of evidence. Less than 1 in 7 SRs had evidence of high quality for the first listed primary outcome and less than 1 in 5 had evidence of high quality for any outcome. Reviews reporting a high quality of evidence appear considerably more likely to lead reviewers to
Conclusions
Our survey has found that high quality of evidence is uncommon in the CDSR and that favorable interpretation on the value of an intervention based on both high quality of evidence and statistically significant results is very sparse in the body of medical and health-related evidence reviewed within the CDSR. Based on this extensive cross-sectional analysis of the CDSR, for the vast majority of the recently reviewed medical and health topics, we need better interventions, better evidence, or
References (33)
- et al.
GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables
J Clin Epidemiol
(2011) - et al.
The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses
J Clin Epidemiol
(2013) - et al.
Using GRADE for evaluating the quality of evidence in hyperbaric oxygen therapy clarifies evidence limitations
J Clin Epidemiol
(2014) - et al.
Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs
J Clin Epidemiol
(2013) - et al.
Reducing waste from incomplete or unusable reports of biomedical research
Lancet
(2014) - et al.
Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists
Control Clin Trials
(1995) - et al.
Active implementation strategy of CONSORT adherence by a dental specialty journal improved randomized clinical trial reporting
J Clin Epidemiol
(2014) Underpowering in randomized trials reporting a sample size calculation
J Clin Epidemiol
(2003)- et al.
GRADE guidelines: 3. Rating the quality of evidence
J Clin Epidemiol
(2011) Evidence-based healthcare: how to make health policy and management decisions
(1997)
Epidemiology and reporting characteristics of systematic reviews
PloS Med
Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: a quality paradigm?
Eur J Orthod
Does updating improve the methodological and reporting quality of systematic reviews?
BMC Med Res Methodol
Citation of Cochrane reviews in national and international guidelines and policies. Reports of the NHS Health Technology Assessment Programme, Effective Health Care, Finnish Evidence-Based Medicine Guidelines, and Clinical Evidence
Strength of recommendation taxonomy (SORT): a patient-centered approach to grading evidence in the medical literature
Am Fam Physician
Cited by (49)
Modifiable risk factors for diphtheria: A systematic review and meta-analysis
2023, Global EpidemiologyMost healthcare interventions tested in Cochrane Reviews are not effective according to high quality evidence: a systematic review and meta-analysis
2022, Journal of Clinical EpidemiologyCitation Excerpt :Following a method used in a related review [14], we extracted the description of the outcome, outcome category (subjective/objective), number of trials and number of participants, effect type, effect size, significance level, whether there was a statistically significant positive result, and whether the original review authors deemed the intervention to be effective. The information about the review authors views was obtained from the conclusions section in the review abstract and the body of the review (subsections “implications for practice” and/or “implications for research”) [13]. An example of a positive interpretation was: “Buprenorphine should be supported as a medication to use” [18].
The certainty of the evidence in oral health has not improved according to GRADE: a meta-epidemiological study
2022, Journal of Clinical Epidemiology