Inconsistent results for Peto odds ratios in multi‐arm studies, network meta‐analysis and indirect comparisons

The Peto odds ratio is a well‐known effect measure in meta‐analysis of binary outcomes. For pairwise comparisons, the Peto odds ratio estimator can be severely biased in the situation of unbalanced sample sizes in the two treatment groups or large treatment effects. In this publication, we evaluate Peto odds ratio estimators in the setting of multi‐arm studies and in network meta‐analysis using illustrative examples. We observe that Peto odds ratio estimators in a multi‐arm study are inconsistent if the observed event probabilities are different or the sample sizes of treatment groups are unbalanced. The same problem emerges in network meta‐analysis including only two‐arm studies and translates to indirect comparisons of pairwise meta‐analyses. We conclude that the Peto odds ratio should not be used as effect measure in network meta‐analysis or indirect comparisons of pairwise meta‐analyses.


What is already known
The Peto odds ratio estimand in a single study is different from the odds ratio if the true treatment effect is large or group sample sizes are unbalanced. This problem transfers to the Peto odds ratio estimator and can thus impact metaanalysis results.
What is new Peto odds ratio estimators from a multi-arm study are inconsistent if the observed event probabilities are different or sample sizes are unbalanced across treatment groups. This inconsistency also impacts network metaanalyses and indirect comparisons of pairwise meta-analyses.

Potential impact for RSM readers outside the authors' field
The Peto odds ratio should not be considered as an effect measure in network meta-analysis or indirect comparisons of pairwise meta-analyses.

| INTRODUCTION
The Peto method 1 is a traditional method for metaanalysis of binary outcomes. It is a variant of the generic inverse variance method 2 using the logarithm of the Peto odds ratio and its standard error from each study. The Peto method is implemented in common software for meta-analysis: RevMan Web from Cochrane, Comprehensive Meta-Analysis, Stata, and several R packages including meta and metafor.
Yusuf et al 1 used likelihood theory to define the Peto odds ratio in a study, based on the difference between the observed and expected number of events in the experimental treatment group under the assumption of no treatment difference. The logarithm of the Peto odds ratio estimator corresponds to the first step of the Newton-Raphson algorithm away from the null effect to the maximum likelihood of the log odds ratio. Accordingly, the Peto method is sometimes called the one-step method. 3 The popularity of the Peto method received a major setback with the publication by Greenland and Salvan 3 demonstrating-through some hypothetical examplesthat the Peto odds ratio estimator in a study can be severely biased in the situation of unbalanced sample sizes in the two treatment groups or large treatment effects (i.e., away from the null effect) impacting the result of the meta-analysis. The method regained some of its reputation after a large simulation study 4 in metaanalysis with rare binary events. The Peto method was the least biased and most powerful meta-analysis method for event rates below 1 percent, however, this was only in situations with no substantial imbalance in group sample sizes and no excessive treatment effects.
It has been argued that the Peto odds ratio in a study should be considered an alternative effect measure. 5 Using the delta method, the authors derived the limit of the expected Peto odds ratio. This estimand contains the ratio of the group sample sizes as a component. This peculiar property of the Peto odds ratio is not shared by other estimands for binary data. In addition, extensive simulations of single studies with rare binary events by the same group 6 showed that the Peto odds ratio estimator does not outperform the usual odds ratio estimator in all performance measures, even in the optimal setting of equal group sample sizes and modest treatment effects.
Accordingly, the Peto method is nowadays not recommended as the default approach for meta-analysis. [6][7][8] All publications summarized so far evaluated the Peto odds ratio either in the setting of a single study with two treatment groups, that is, a single two-by-two table, or in a pairwise meta-analysis, that is, one two-by-two table per study. In this publication, we will look at the Peto odds ratio in the setting of multi-arm studies and in network meta-analysis. Like Greenland and Salvan, 3 we will use illustrative examples to show that another problem exists for Peto odds ratio estimators in multi-arm studies as well as network meta-analysis and indirect comparisons of pairwise meta-analyses.

| ODDS RATIO AND PETO ODDS RATIO IN MULTI-ARM STUDIES
Let us assume we have data from a single study comparing three treatment groups with respect to a binary outcome, that is, the number of events in each group follows a binomial distribution, 9,p. 39 . The true (but unknown) probabilities of the event of interest are denoted by p 1 , p 2 , and p 3 , respectively. Estimates of p 1 , p 2 , and p 3 are given bŷ p i ¼ x i =n i , for i = 1, 2, 3, with number of events x 1 , x 2 , x 3 and group sample sizes n 1 , n 2 , n 3 .

| Odds ratio
The odds ratio OR 12 comparing group 1 and 2 is defined as Odds ratios OR 13 , OR 23 , OR 21 , OR 31 and OR 32 are defined accordingly.
By construction, the odds ratios in a three-arm study are consistent which means, for example, that the product of OR 12 ÁOR 23 ÁOR 31 ¼ 1. Or, alternatively, that OR 12 ¼ OR 13 =OR 23 which means that the odds ratio of the direct comparison of treatment 1 versus 2 is equal to the odds ratio of the indirect comparison, that is, OR 13 =OR 23 .

| Peto odds ratio
The estimated Peto odds ratio d POR 12 comparing treatment 1 and 2 is given by Þand Var a k j…;OR 12 ¼ 1 ð Þare the mean and variance of x 1 under the hypergeometric distribution with "…" denoting the four (observed) cell margins. Under this distribution, we have Estimated Peto odds ratios d POR 13 , d POR 23 , d POR 21 , d POR 31 , and d POR 32 are defined accordingly. It is not straightforward to determine if and when the set of estimated Peto odds ratios is consistent in the way the odds ratios are, as shown in section 2.1. Therefore, we decided to evaluate this using real and artificial examples.

| Illustrative examples
For illustration, we will use data from the two three-arm studies included in a network meta-analysis to prevent bleeding in cirrhosis. 10 The actual data is listed in Table 1. We use R function pairwise from R package netmeta 11 to conduct all analyses for a single three-arm study. The R code is available as a supplement.

| Original study data
The estimated odds ratios for study 1 in Table 1  In the following subsections, to further elucidate the properties of the Peto odds ratio, we explore a set of artificially created data taking study 1 as our starting point.

| PETO ODDS RATIO IN NETWORK META-ANALYSIS AND INDIRECT COMPARISONS
The three estimated treatment effects in a three-arm study are not independent as the results of the first and second comparisons determine the third result. Two approaches exist to account for the dependency in order to include multi-arm studies in network meta-analysis. 12 We can either only consider comparisons with a reference treatment (called basic parameters) or we can consider all pairwise comparisons but increase the standard error of each comparison. Higgins et al 10 used the first approach while R package netmeta implements the second approach.
Based on our results described in the previous section, we would argue against using the Peto odds ratio in a T A B L E 2 Variants of study 1 from network meta-analysis to prevent bleeding in cirrhosis 10 network meta-analysis including multi-arm studies. For the cirrhosis dataset, a network meta-analysis including multi-arm studies by only considering comparisons with a reference treatment would not notice that the Peto odds ratios of study 1 are inconsistent. 10 A network metaanalysis of this dataset with R package netmeta results in an error stating that treatment estimates of study 1 are inconsistent (see supplementary R code S1).
Another issue with the Peto odds ratio is that the inconsistency problem also affects network meta-analyses including only studies with two treatments. In Table 3 we present the data of study 1 from the network meta-analysis to prevent bleeding in cirrhosis 10 as if they come from three independent studies. A fixed-effect network meta-analysis of these three studies using the odds ratio as effect measure results in the same odds ratios as for the data in Table 1 and the Q statistic for inconsistency is equal to 0. A fixed effect network meta-analysis using the Peto odds ratio as effect measure results in slightly different network estimates (0.2467, 0.1528, 0.6196) and the Q statistic for inconsistency is equal to 0.0297. The network estimates must be different from the original Peto odds ratio estimators as the underlying network model assumes consistency of treatment effects.
Finally, the inconsistency problem also transfers to indirect comparisons of pairwise meta-analyses. To illustrate this, we only consider data from study 1 and 2 in Table 3. In this situation, the indirect treatment estimate of beta-blockers versus sclerotherapy is calculated in the same way as described in subsection 2.3.1 resulting in an inconsistent indirect Peto odds ratio estimator.

| DISCUSSION
In this article we show that the use of Peto odds ratio estimators can lead to inconsistent treatment estimates in settings comparing more than two treatments. While we first noticed this problem for a single multi-arm study and a network meta-analysis including multi-arm studies, we later recognized that the same problem exists in network meta-analysis of two-arm studies. One reviewer pointed out that this problem also translates to indirect comparisons of pairwise meta-analyses. To our knowledge, this is the first time that this issue of the Peto odds ratio estimator is described in the literature. We assume that it has not been recognized so far as the usual approach to include multi-arm studies in network metaanalysis is to only consider comparisons with a common comparator. Accordingly, the inconsistency is not noticed but ascribed to differences in the individual studies.
We did not comprehensively evaluate the inconsistency of Peto odds ratio estimators, but considered some real and fictitious examples. The aim of this work is to inform about this problem instead of thoroughly investigating it. The Peto odds ratio estimators in a multi-arm study are only very rarely consistent in a strict mathematical sense, for example, if the event numbers and group sample sizes in two of three studies are the same; see variant 1 where odds ratio and Peto odds ratio estimators were rather different, however, both consistent. We would speculate that Peto odds ratio estimands will in general be inconsistent if event probabilities differ across treatments in a study or group sample sizes are unequal.
One may argue-like in pairwise meta-analysis-that the Peto method could be an option if events are rare, treatment effects are small or modest, and group sample sizes are (almost) balanced. However, the problem is whether these conditions are met in a network meta-analysis. As other statistical methods for network meta-analysis of rare binary outcomes are available, 13,14 we would not recommend the use of the Peto method in network meta-analysis.
Admittedly, our results do not have a real-world impact on applications of network meta-analyses as the Peto odds ratio is rarely used. However, (informal) indirect comparisons of multiple pairwise meta-analyses are quite common. For example, Cochrane reviews often report results of several pairwise meta-analyses comparing active T A B L E 3 Depicting data from first three-arm study 10 as three independent two-arm studies treatments with placebo as head-to-head comparisons of active treatments are scarce. Furthermore, the Peto odds ratio is a popular effect measure in Cochrane reviews. An empirical investigation of the impact of using Peto odds ratio estimators in such indirect comparisons could provide additional insights on the inconsistency problem.