Left-truncated effects and overestimated meta-analytic means

The meta-analytic by Mertens et al. (1) interprets nudges as a generally effective technique for increasing desirable decision-making, with an overall pooled effect size of d = 0.43. This research also reports large systematic variations (meta-analytic heterogeneity) in effects, primarily attributed to moderators such as the domain, as well as asymmetrically distributed effects, interpreted as moderate publication bias. Apart from publication bias, non-normality and high heterogeneity may be problematic for the representativeness of meta-analytic means (2). Here, we reanalyze the corrected data made available by Mertens et al. (1), finding evidence that nudges have more limited than general effectiveness. We show that effects are clearly left-truncated, likely due to substantial publication bias, consistent with another reanalysis (3). We also find that most of the pooled effects as reported in Mertens et al. (1) are overestimated and hence unrepresentative. First, we visualize the distributions of effects, by domain, using raincloud plots (4); see Fig. 1. Four domains (finance, food, other, and prosocial) show a concerning pattern of sharply left-truncated tails at or slightly below zero. The two remaining domains only have a handful of effects slightly below zero. A plausible mechanism for this left “cliff” is suppression of unfavorable results (5). Most domains also exhibit long right tails—a limited number of effects

The meta-analytic by Mertens et al. (1) interprets nudges as a generally effective technique for increasing desirable decision-making, with an overall pooled effect size of d = 0.43. This research also reports large systematic variations (meta-analytic heterogeneity) in effects, primarily attributed to moderators such as the domain, as well as asymmetrically distributed effects, interpreted as moderate publication bias.
Apart from publication bias, non-normality and high heterogeneity may be problematic for the representativeness of meta-analytic means (2). Here, we reanalyze the corrected data made available by Mertens et al. (1), finding evidence that nudges have more limited than general effectiveness. We show that effects are clearly left-truncated, likely due to substantial publication bias, consistent with another reanalysis (3). We also find that most of the pooled effects as reported in Mertens et al. (1) are overestimated and hence unrepresentative.
First, we visualize the distributions of effects, by domain, using raincloud plots (4); see Fig. 1. Four domains (finance, food, other, and prosocial) show a concerning pattern of sharply left-truncated tails at or slightly below zero. The two remaining domains only have a handful of effects slightly below zero. A plausible mechanism for this left "cliff" is suppression of unfavorable results (5). Most domains also exhibit long right tails-a limited number of effects  with large and very large magnitudes. This pattern of left truncation and long right tails strongly indicates that publication bias is greater than moderate. Second, we evaluate non-normality and the representativeness of pooled effects by domain (Table 1). Normality was assessed using Egger's regression test for asymmetry (6). Representativeness was tested by quantifying the estimated proportion of effects below meaningful thresholds (7), here, the meta-analytic means. A perfectly representative (meta-analytic) mean would have 50% of values below it.
All domains exhibited asymmetry, and all but one (other) had some overestimation in pooled effects, that is, a greater than expected proportion of effects below their metaanalytic mean. Despite left truncation of effects, nearly two-thirds of all effects were still below the overall metaanalytic mean.
Funnel plots can often be difficult to interpret (8), and, typically, all effects are plotted together; thus, the severity and nature of the non-normality in effects, especially by domain, may not be apparent in Mertens et al. (1). Here, we evaluate effects by domain; therefore, our results cannot be solely attributed to the heterogeneity and non-normality potentially caused by combining domains.
The end goal of nudges and related behavioral interventions is increasing desirable decision-making. Achieving this requires identifying factors associated with positive impacts, but also factors that have minimal and even negative effects on decisions (9,10). Publication bias impedes understanding for variations in nudge effectiveness. Data Availability. Data and code are available at https://osf.io/jydb7/ (11) and https://codeocean.com/capsule/3133766/tree/v1 (12). ACKNOWLEDGMENTS. We thank Jessica Schultheis for editing a previous version of this work.The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the US Army Combat Capabilities Development Command Army Research Laboratory or the US government. The US government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation.