Should a Trial with Lost Allocation Key be Incorporated in Meta - analysis?

Aims: On occasion randomized controlled studies may remain unpublished because the allocation key identifying which group received active treatment and which received placebo (or alternative treatment) has been lost. The purpose of this paper is to explore whether and under what circumstances an unpublished study like NCT00250237 whose allocation key has been irretrievably lost might be incorporated into a meta-analysis, allowing the information contained to contribute, however tentatively, to the evidence base around a clinical topic. Study Design: Review / meta-analysis Place and Duration of Study: Vancouver Coastal Health Research Institute and School of Population and Public Health, University of British Columbia and Department of Surgery, Rijnland Short Research Article Pijls et al.; BJMMR, 9(3): 1-7, 2015; Article no.BJMMR.17815 2 Hospital, between 2009 and 2014. Methodology: For two published trials, we calculated separate effect measures for a simulated lost allocation key, under the alternative assumptions that Groups A or B received active treatment. These two effects were compared visually to the pooled effect from a meta-analysis of all other comparable studies. We present as well several quantitative methods for reconstituting a lost allocation code, involving frequentist meta-analysis, meta-regression, and Bayesian techniques. We apply these to an analysis of NCT00250237. Results: The simulated lost allocation keys of both trials were successfully identified through visual methods using meta-analysis. Reconstitution in this manner requires that the trial’s effect depart significantly from the null and that there exist a sufficient number of other high quality studies addressing the same clinical topic to allow unbiased calculation of a pooled summary effect. While direct reconstitution may sometimes be possible, our primary approaches are implicitly Bayesian and, as with NCT00250327, allow probabilities to be estimated for each allocation assignment. Conclusion: Under some circumstances studies with a lost allocation key may be incorporated cautiously into meta-analysis.


INTRODUCTION
Systematic reviews and the techniques of metaanalysis comprise the basis of evidence-based practice, but valid meta-analysis requires that all relevant studies -published or not -be assessed for inclusion. In 1979 Rosenthal called attention to the "File Drawer Problem," observing that studies consigned to file drawers, and thus unpublished, are often disconfirmatory [1]. Such "negative" studies may reflect correct inference or Type II error, but in either case omitting them from meta-analysis risks serious bias.
Unpublished positive studies have attracted little attention. It is probable that some of such studies remain unpublished due to sloth, to a change in research direction, to retirement, or even to investigator death. Randomized trials face yet another risk precluding publication: the irrevocable loss of information recording which participants received active treatment. The current paper's authors successfully completed a trial, only to discover that its allocation key (AK) had disappeared. The study (trial registration NCT00250237) investigated haloperidol versus placebo for preventing post-operative delirium in hip fracture patients, was completed in 2009, was well-designed and executed, and achieved significant results. With AK lost, researchers could not ascertain whether participants had been benefitted or harmed; and so the study was consigned to the file drawer. At least three subsequent meta-analyses have been completed without considering the results of this stillunpublished trial. The purpose of this paper is to explore under what circumstances a study like NCT00250237 with a lost AK might be incorporated into a meta-analysis, allowing the information contained to contribute, however tentatively, to the evidence base around a clinical topic.

MATERIALS AND METHODS
For the purposes of this paper we define the AK of a randomized controlled trial (RCT) as information that allows the researcher to determine which of two or more study groups received active treatment(s) and which received placebo (or comparison treatment.) We consider the situation of a lost AK for two-arm trials comparing active treatment (T) with placebo (P). All patients in group A have received comparable -but unknown -treatments, either T or P, and all patients in group B have all received the alternative. For the indirect method we calculate two effect measures for the lost key study, under the alternative assumptions that Groups A or B received active treatment. In an implicitly Bayesian analysis these two effects were compared visually (or, with results not reported here, using a Student t test) to the pooled effect from a meta-analysis of all other comparable studies.
Since we are dealing with two possible study results in the context of other studies we are interested in the prediction interval rather than the confidence interval of the mean. The prediction interval (PI), an estimate of an interval in which effect sizes of future studies will fall with a certain probability (e.g. 95%) is given by a formula included in the appendix to this paper [2]. Data were combined for meta-analysis with the random effects model (RE) using a restricted maximum-likelihood estimator (REML) for risk differences (RD). All analyses were performed with the metafor package for R version 2.13 orfor Bayesian meta analysis -WinBugs 1.4.3 [3].

Direct Approach to Reconstituting the Randomisation Key
A direct approach uses material, examinations or information available from the study itself. If marked A or B in a drug trial, left-over medication can be chemically analyzed to determine the AK. Stored blood, urine, or tissue samples, if available, can be chemically analyzed for active treatment, or in some cases such samples can be obtained and assessed even after the study has concluded [4]. None of these or similar direct approaches was feasible for study NCT00250237.
A variant on the direct method is sometimes possible when it is known from previous studies, or can be subsequently be determined, that there is effect modification by some reliably identifiable factor. The observed risk difference in NCT00250237 had a different sign in the presence and absence of dementia, which was therefore a modifier of the observed effect. NCT00250237 authors had previously conducted an observational study of haloperidol and delirium [5]. This study found that delirium was reduced among subjects with cognitive impairment who received pre-operative haloperidol, but not among cognitively-normal subjects. That observation pointed to an allocation assignment consistent with that derived from indirect methods described below.

Indirect Approach to Reconstituting a
Randomisation Key

Worked example in the absence of heterogeneity
The study by Cannon et al. is an example of a meta-analysis with significant treatment effect and no heterogeneity [6]. Subjects given high dose statin therapy had a significant reduction in the risk of coronary death or myocardial infarction compared to standard dose statins. The SEARCH study, published 4 years after the Cannon meta-analysis, met the latter's inclusion criteria [7]. We counterfactually proposed that the SEARCH study's AK had been lost, and we assigned to it two possible AK reconstitutions, SEARCH-A (Group A=high dose) and SEARCH-B (Group B = high dose.) In the absence of heterogeneity, we adopted an empirical Bayesian hypothesis that the correctly-assigned group would be more likely to have an effect falling within the confidence interval of the Cannon meta-analysis. The corresponding Forest plot in Fig. 1 confirms that SEARCH-A has a correctly-assigned AK [7,6].

Fig. 1. Forest plot of cannon meta-analysis with SEARCH-A and SEARCH-B depicted in the plot
Note that the data from SEARCH-A and SEARCH-B was not included in the meta-analysis, but is only plotted

Worked example when heterogeneity is present
The meta-analysis by Colditz et al. [8] found that BCG vaccine (intervention group) significantly reduced risk of tuberculosis compared to placebo (control group). Ten years after this metaanalysis Aronson published a study which met the Colditz inclusion criteria [9]. As in the example above, we assumed counterfactually that Aronson had lost the AK, and we calculated separate effect sizes for Aronson-A (group A received BCG) and Aronson-B (Group B received BCG). Since there is meta-analytic heterogeneity, a prediction interval was used instead of the confidence interval. An allocation assignment whose effect fell within the prediction interval, taking into account latitude as effect modifier, was hypothesized the more likely of the two alternatives.
In a forest plot of the Colditz meta-analysis (Fig. 2), the point estimator of Aronson-A falls within the prediction interval. Although visual inspection of interval boundaries may mislead, here we (correctly) hypothesize that Group A received BCG [10].
Geographic latitude proved a significant moderator in the Colditz meta regression: subjects in northern latitudes had a stronger protective effect from BCG immunization than did those in southern latitudes [8]. In the absence of a factor like latitude it is also possible to metaregress outcome on baseline risk, usually defined as the risk in controls [11,12]. A comparison can then be made to the observed control group risk in alternative allocation assignments.

Assessment of NCT 00250237
The odds ratios for the two alternative allocations were 0.5 (A) and 2.0 (B) with non-overlapping 95% confidence intervals. Ordinary random effects meta-analysis of five published studies gave a summary OR of 0.42 (95% CI 0.24 to 0.74). Meta-regression with risk as a moderator similarly favored allocation A, a finding confirmed by Bayesian methods utilizing uninformative priors [12]. Bayesian meta-analysis can be used to predict the interval into which the nextperformed clinical trial is likely to fall, in this case a point OR estimate of 0.41 and 0.44 [13]. Six priors were used, with those that favoured between-study variance similar to that derived from already-published studies suggesting a 30 to 100-fold greater likelihood of allocation A. Even under the very wide prediction intervals generated by uninformative uniform priors on tau-squared and tau, the MCMC-simulated probability (y new < y obs ) for allocation B (y new ) was only 11% (5.6% for a prior uniform on tau).

Fig. 2. Forest plot of colditz meta-analysis with Aronson-A and Aronson-B depicted in the plot
Note that the data from Aronson-A and Aronson-B were not included in the meta-analysis, but were only plotted. the aronson study was performed at 44 degrees latitude. (latitude was a potential moderating variable recorded for all studies in the meta-analysis.) the 44 degrees prediction interval therefore gives the predicted risk difference for 44 degrees latitude with corresponding prediction interval derived from meta-regression

DISCUSSION
RCT are powerful and costly tools that often expose subjects to potentially harmful interventions in the hopes of improving future patients' clinical outcomes. It is ethically desirable that RCT results be disseminated, as emphasized by the WHO statement of April 2015 [14]. We are not aware of a previous lost AK being reported, but given the ubiquity of error in all complex human activities, we would be very surprised to learn that allocation loss had never previously occurred. The described direct and indirect methods may allow a researcher facing the misfortune of a lost AK, to hypothesize which group is most likely to have received the intervention. It should be intuitively obvious (which we have confirmed by simulations) that indirect methods best distinguish the two options for a lost AK when risk differences depart significantly from the null. The probability of correct AK assignment increases if the assignment inferred from one indirect method is confirmed by another method such as metaregression. While generally feasible, meta regression on baseline risk should be employed with caution, ideally using Bayesian methods [15].
So, should an otherwise methodologically-sound study with lost AK be incorporated within a meta analysis? There should be no debate if the study is negative: i.e. the effect size is not clinically or statistically significant. Such studies are unlikely to have a recoverable AK because the effects of alternative allocations are statistically indistinguishable, but as with any negative study they add important meta analytic evidence. The debate centers around the approach to a study, like NCT00250237 whose outcome differed significantly from the null and conferred a number-needed-to -treat (or harm) of eight. This means that in the light of a lost allocation key only one of two possible conclusions is correct for NCT00250237. One: haldol reduces the incidence of post-operative delirium after acute hip surgery compared to placebo. Two: haldol increases the incidence of post-operative delirium after acute hip surgery compared to placebo. Given the results from a recent systematic review and meta-analysis, only option one appears valid: haldol reduces incidence of delerium [16]. The implicitly Bayesian arguments in this paper would support including a lost AK study if one or more methods established that one AK assignment was substantially more probable than the other. In an unselected review of a large number of recent meta-analyses Tam et al found that the direction of effect of the first trial on a topic agreed with subsequent metaanalysis in 99% of cases (95% CI 97-100%) [17]. While limited to a first trial, these findings support our initial indirect approach relying on inspection of the Forest plot.
Removing a lost AK study from the file drawer might seem to introduce a form of circularity into meta-analytic reasoning, using a prior likelihood established thorough meta-analysis to reinforce a "Lysenkoist" bias toward what we already believe true. Adding this study to the pre-existing evidence base does risk reinforcing previouslyintroduced bias, even more there is also clinical and ethical risk from ignoring knowledge when reasonable inference about its veracity is possible. However, since one of the two allocations is "correct," a lost AK study with significant results provides additional evidence that should be taken into consideration when interpreting meta-analytic results.
If a study with lost AK is to be incorporated into a meta-analysis, a sensitivity analysis must be done with both AK alternatives separately assessed, and in the absence of previouslypublished meta analysis, results should be presented both including and excluding the lost AK study. Effect sizes of each allocation should also be reported and their relative plausibilities reflected upon. We think it is possible that both allocations should be included together in the meta-analysis, with weightings of each adjusted to reflect the inferred relative probability that each is the correct AK reconstitution, and with the total weight for both equivalent to that of only one study. This approach would be methodologically difficult to implement and was not utilized in the only published meta-analysis of which we are aware in which a lost AK study (NCT00250237) has been included [16].

CONCLUSION
Under some circumstances studies with a lost allocation key may be incorporated cautiously into meta-analysis.

CONSENT
As this is a methodological study (no patients included) dealing with the issue of a lost randomization key, informed consent was not applicable.

ETHICAL APPROVAL
As this is a methodological study (no patients included) dealing with the issue of a lost randomization key, ethical approval was not applicable.