Methods to select results to include in meta-analyses deserve more consideration in systematic reviews

Abstract

Objectives

To investigate how often systematic reviewers encounter multiple trial effect estimates that are available for inclusion in a particular meta-analysis (multiplicity of results) and the methods they use to select effect estimates.

Study Design and Setting

We randomly sampled Cochrane and MEDLINE-indexed non-Cochrane reviews published between January 2010 and January 2012. The first presented meta-analysis of an effect measure for a continuous outcome in each review was identified, and methods to select results to include in this meta-analysis were extracted from review protocols and reviews. All effect estimates that were available for inclusion in the meta-analyses were extracted from trial reports.

Results

We examined 44 reviews. Multiplicity of results was common, occurring in 49% of trial reports (n = 210). Prespecification of decision rules to select results from multiple measurement scales and intervention/control groups (in multi-arm trials) was uncommon (19% and 14% of 21 review protocols, respectively). Overall, 70% of reviews included at least one randomized controlled trial with multiplicity of results, but this occurred less frequently in reviews with a protocol (risk difference, −25%; 95% confidence interval: −52%, 1%).

Conclusion

Systematic reviewers are likely to encounter multiplicity of results in the included trials. We recommend that systematic reviewers always consider predefining methods to select results to include in meta-analyses. Methods focusing on selection of measurement scales and how to deal with multi-arm trials would be most valuable.

Introduction

Systematic reviews of randomized controlled trials (RCTs) of health care interventions have the potential to have a major impact on patient health, research agendas, and policy making. However, the validity of systematic review findings can be compromised by challenges in undertaking meta-analysis. One challenge is that multiple effect estimates in a trial report may be available for inclusion in a particular meta-analysis [1], [2]. For example, a trial report may present effect estimates for two depression scales, at week three, six, and nine, each analyzed as unadjusted and adjusted for covariates. Multiplicity of effect estimates may lead to “selective inclusion of results,” whereby the process for selecting the trial effect estimates for inclusion in a meta-analysis is based on the estimates themselves, which may, in turn, result in biased meta-analytic effects [3].

Several organizations that produce systematic reviews (e.g., [4], [5], [6]) have recommended methods that aim to reduce selective inclusion of results. The methods (specified a priori) aim to uniquely identify results that will be included in a meta-analysis and can be placed in two broad categories, which we label “eligibility criteria to select results” and “decision rules to select results.” Eligibility criteria to select results include specifying lists of measurement scales, intervention/control groups, time points, and analyses that systematic reviewers consider eligible to include in the review (ideally based on some clinical or methodological rationale). Providing specific criteria discourages the use of broad outcomes such as “pain,” and instead encourages specification of details such as the eligible pain measurement scales and time points of interest to the review [1], [2].

Predefining eligibility criteria to select results is an effective method to minimize the number of effect estimates available for inclusion in a particular meta-analysis. However, this method may not always identify a single eligible effect estimate per trial, and in such cases, the addition of decision rules is useful. Decision rules are strategies to either select one effect estimate, or combine effect estimates, when multiple are available. An example of a decision rule to select one effect estimate is when commonly encountered measurement scales for a particular outcome domain (e.g., depression) are ranked based on their psychometric properties, and for trials that report the results of more than one scale, the results for the tool with the best measurement properties are selected. Such a strategy has previously been referred to as an “outcome data hierarchy” [2], [7], [8]. An example of a decision rule to combine effect estimates is when a trial includes more than one active treatment arm (e.g., placebo vs. high-dose drug vs. low-dose drug), and rather than selecting data from only one of the active arms (e.g., only one dosage group), data from all active treatment arms are combined (e.g., any dosage vs. placebo) [9], [10].

To our knowledge, only two previous studies have investigated multiplicity of results in trial reports or the methods systematic reviewers use to select results to include in meta-analyses [2], [11]. In the first study, that examined interobserver variation in results extracted from trials for use in meta-analyses, decision rules to select final vs. change from baseline values were reported in 4 of 10 review protocols [11]. In the second study [2], that examined the impact of multiplicity of trial results on meta-analysis results, multiplicity was found to be common, but methods to select results to include in meta-analyses were rarely predefined. In 83 RCTs included in 19 Cochrane reviews published from 2006 to 2007, 35% of the RCTs had multiple measurement scales, 29% had multiple intervention/control groups (i.e., in multi-arm RCTs), and 36% had multiple time points that were available for inclusion in a particular meta-analysis. In all review protocols, eligibility criteria for measurement scales and intervention/control groups were always defined, and eligibility criteria for time points were defined in eight (42%). In contrast, decision rules to select measurement scales or intervention/control groups were not reported in any of the review protocols, whereas a decision rule to select time points was reported in one review protocol (5%) [2].

To inform methods guidance regarding inclusion of results when there is multiplicity, several issues still require exploration. First, the protocols in Tendal et al. studies were published before 2006, and it is unclear whether reporting of eligibility criteria and decision rules to select results has changed over time. Second, most systematic reviewers do not report working from a review protocol [12], [13], and the methods used to select results to include in such reviews have not been examined. Third, there has been no investigation of the frequency of other types of multiplicity which may arise in RCTs [e.g., reporting of results from intention-to-treat (ITT) and per-protocol or unadjusted and covariate-adjusted analyses]. Fourth, no one has examined whether multiplicity of results and reporting of methods to select results to include in meta-analyses differs between clinical conditions. It may be hypothesized that there may be less multiplicity of results for clinical conditions that have “core outcome measurement sets” available [14], [15], [16]. Core outcome measurement sets are measurement scales recommended for use in RCTs and systematic reviews of a particular health condition and are designed to increase consistency in scale selection.

Our aim was to investigate multiplicity of results in trial reports and methods systematic reviewers use to select results to include in meta-analyses. The primary objectives were to investigate the frequency and types of: (1) multiplicity of results that arise in RCTs and (2) eligibility criteria and decision rules to select results, which are reported in review protocols and reviews. Secondary objectives were to examine how the extent of multiplicity of results was modified by the existence of a review protocol and the clinical condition of the review and how the reporting of eligibility criteria and decision rules to select results was modified by the clinical condition of the review. We also plan to investigate whether there is evidence of selective inclusion of results in the sample of reviews and what impact this may have on meta-analytic effect estimates [17]; the results of this research will form a subsequent article.