Review
High quality of the evidence for medical and other health-related interventions was uncommon in Cochrane systematic reviews

https://doi.org/10.1016/j.jclinepi.2016.03.012Get rights and content

Abstract

Objectives

To appraise the quality of evidence in systematic reviews (SRs) within the Cochrane Database of Systematic Reviews (CDSRs) across diverse topics and to explore the relationship between the strength of evidence using Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) and the probability that authors would interpret that an intervention may be of value.

Study Design and Setting

We evaluated the SRs published on the CDSR from January 1, 2013, to June 30, 2014. Two authors identified relevant SRs by independent searching of the Cochrane register. We further focused on SRs that incorporated tables with GRADE [summary of findings (SoF)]. Data were extracted independently by two authors. The quality of the evidence for the first listed primary outcome in SoF tables in each review and reasons for upgrade or downgrade were recorded.

Results

Overall, 1,394 SRs were identified. Of these, 608 (43.6%) incorporated GRADE. Within these reviews, only 13.5% (n = 82) reported a high quality and 30.8% (n = 187) a moderate quality of evidence for the first listed primary outcome, whereas 31.7% (n = 193) had low level and 24% (n = 146) had very low level of evidence. High quality of evidence was more common in updated compared to new reviews and in pharmacologic than other types of interventions. Even when all outcomes listed in the SoFs were considered, only 116/608 (19.1%) of SRs had at least one outcome with high quality of evidence. Overall, only 4.1% (25/608) of SRs incorporating GRADE in SoF tables had high quality of evidence, allied both to significant results and a favorable interpretation of the intervention by the reviewers.

Conclusion

Evidence of high quality is uncommon for medical and health-related interventions assessed with GRADE within the CDSR, and favorable evidence of high quality is even more uncommon.

Introduction

The advent of evidence-based medicine has sensitized us to the need for high quality of evidence to inform clinical and public health practice, but also to the fact that such evidence is often missing in many fields [1]. With the accumulation of vast numbers of systematic reviews (SRs), one can assess systematically the current state of the evidence across very diverse specialties and medical or other health-related interventions.

Since its inception in 1993, the Cochrane Database of Systematic Reviews (CDSRs) has become established as a comprehensive resource for evidence. The database has expanded to include over 50 review groups [2] encompassing diverse areas. Cochrane reviews typically have higher methodological and reporting quality than SRs published elsewhere [3], [4], [5]. Moreover, they are often instrumental in developing policy documents and practice guidelines [6].

Various tools have been used to appraise the quality of a body of evidence within an SR [7], [8], [9]. The Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) tool can allow such a systematic appraisal, and it has been adopted by Cochrane [10]. The GRADE approach involves assimilation of the quality of the available evidence from SRs in addition to the values and preferences of the patients, safety, and costs. The quality of evidence for individual outcomes is scored for specific domains and an overall score assigned (high, moderate, low, and very low) based on five factors: risk of bias (study limitations), inconsistency, indirectness, imprecision, and publication bias and presented in summary of findings (SoF) tables. This approach has been validated for the assessment of the strength of a body of evidence [11].

The quality of the evidence has been addressed in small samples of SRs in focused areas [12], [13], but to our knowledge, an appraisal of the quality of evidence has not yet been undertaken on reviews within the CDSR across diverse topics. Moreover, it would be interesting to explore how this strength of evidence relates to the probability that authors would interpret that an intervention may be of value. Therefore, to answer these questions, we evaluated a large sample of all recent SRs in the CDSR that had used the GRADE tool to generate SoF tables.

Section snippets

Data sources and searches

Both new and updated versions of all SRs published over an 18-month period from January 1, 2013, to June 30, 2014, were identified on the CDSRs.

Study selection and data extraction

Two authors (P.S.F. and N.P.) retrieved SRs for inclusion in the review by accessing the CDSR (www.thecochranelibrary.com). Full text reports were uploaded onto a shared electronic folder. Data from the identified reviews were extracted independently and entered on prepiloted standardized forms by two authors (P.S.F. and D.K.) after the initial

Database search results

One thousand three hundred ninety-four SRs were identified in the 18-month period across 50 Cochrane research groups. Of these, 608 (43.6%) incorporated GRADE assessments with at least one SoF table. Two hundred four reviews (33.6%) had two or more SoF tables. In this study, the focus was only on the first outcome of the first SoF table (Fig. 1). Of these reviews with GRADE assessments, 316 (52%) were new and 292 (48%) were updates.

Characteristics of included interventions and outcomes

Of the 608 eligible SoF tables, over half (56.5%) related to

Main findings

Our survey of the quality of evidence underpinning recent SRs published in the CDSR found that almost half of them had GRADE evaluations. Among them, we noted a large percentage of conclusions are based on low or very low quality of evidence. Less than 1 in 7 SRs had evidence of high quality for the first listed primary outcome and less than 1 in 5 had evidence of high quality for any outcome. Reviews reporting a high quality of evidence appear considerably more likely to lead reviewers to

Conclusions

Our survey has found that high quality of evidence is uncommon in the CDSR and that favorable interpretation on the value of an intervention based on both high quality of evidence and statistically significant results is very sparse in the body of medical and health-related evidence reviewed within the CDSR. Based on this extensive cross-sectional analysis of the CDSR, for the vast majority of the recently reviewed medical and health topics, we need better interventions, better evidence, or

References (33)

  • Cochrane Review Groups. Available at http://www.cochrane.org/contact/review-groups. Accessed July...
  • D. Moher et al.

    Epidemiology and reporting characteristics of systematic reviews

    PloS Med

    (2007)
  • P.S. Fleming et al.

    Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: a quality paradigm?

    Eur J Orthod

    (2013)
  • B. Shea et al.

    Does updating improve the methodological and reporting quality of systematic reviews?

    BMC Med Res Methodol

    (2006)
  • L. Holmes et al.

    Citation of Cochrane reviews in national and international guidelines and policies. Reports of the NHS Health Technology Assessment Programme, Effective Health Care, Finnish Evidence-Based Medicine Guidelines, and Clinical Evidence

    (2001)
  • M.H. Ebell et al.

    Strength of recommendation taxonomy (SORT): a patient-centered approach to grading evidence in the medical literature

    Am Fam Physician

    (2004)
  • Cited by (49)

    • Most healthcare interventions tested in Cochrane Reviews are not effective according to high quality evidence: a systematic review and meta-analysis

      2022, Journal of Clinical Epidemiology
      Citation Excerpt :

      Following a method used in a related review [14], we extracted the description of the outcome, outcome category (subjective/objective), number of trials and number of participants, effect type, effect size, significance level, whether there was a statistically significant positive result, and whether the original review authors deemed the intervention to be effective. The information about the review authors views was obtained from the conclusions section in the review abstract and the body of the review (subsections “implications for practice” and/or “implications for research”) [13]. An example of a positive interpretation was: “Buprenorphine should be supported as a medication to use” [18].

    View all citing articles on Scopus
    View full text