BACKGROUND

Systematic reviews are an increasingly important part of the clinical evidence base. Well-conducted reviews provide solid evidence for clinical decision-making, though there are a number of important biases that can impact on the validity of study conclusions. One of the quality markers for systematic reviews is the thoroughness of the search. Our study’s purpose was to assess the frequency that systematic reviews are limited to studies published in English.

METHODS

We searched in PUBMED using the search term “systematic review [pt]” through December 15, 2018. We reviewed the first 250 systematic reviews for the presence of any language restriction and the number of non-English articles included or excluded. This review was done in duplicate with disagreement reached by consensus.

RESULTS

Among 250 systematic reviews, we found that 84 (34%) explicitly excluded non-English articles, 80 (32%) had no statement about excluding non-English trials, though none included non-English articles, and 86 (34%) indicated that they did their search without language restriction. Of the 86 reviews without language restrictions, only 19 (22%) included non-English trials, representing 2% of the total articles included in these studies.

DISCUSSION

A study that examined systematic reviews through 1993 found that most had language restrictions with 93% excluding at least one non-English randomized controlled trial.1 This exclusion of non-English studies is labeled “English-language bias” or “Tower of Babel bias.” We found that in the subsequent two decades, limiting searches to English-only is still common, with only 34% reporting having no language restrictions and few reviews including non-English trials. While 86% of journals are published in English, and the likelihood of citation is higher for English-language articles,2 this does not imply that non-English trials are lower quality. One 1995 study found no difference in quality design elements between English and other language trials,3 though this study was limited to European language articles. Studies examining the impact of including non-English trials on review estimates of effect have been mixed.4,5 Articles published in German are less likely to be significant than those published in English.6 If this is true for other languages, there is a potential for bias in outcomes among reviews that are limited to English-only. Including studies published in non-English languages may have increased resource challenges with respect to costs, time, and expertise in non-English languages; however, inclusion will help ensure generalizability and reduce the risk of bias.

CONCLUSIONS

Limiting systematic reviews to English-only is common in systematic reviews and could result in biased estimates of effect and reduce generalizability. Important research questions for those conducting systematic reviews are whether online translation programs, such as Google Translate, are sufficiently accurate to allow the translation of non-English trials and whether such trials are of comparable quality to those published in English.