Science of price experimentation at Amazon

Abstract

In order to improve prices at Amazon, we created Pricing Labs, a price experimentation platform. Since we do not price discriminate, we must run product-randomized experiments. We discuss how we randomize to prevent spillovers, run different experimental designs (i.e., crossovers) to improve precision, and control for demand trends and differences in treatment groups to get more precise treatment effect estimates.

Appendix: Power analysis with cluster randomization

There are two different approaches in estimating the power that we study here: (1) an analytical closed-form expression considering a normality assumption, or (2) simulation-based power analysis. The closed-form expression is more computationally efficient while its assumptions are more difficult to justify. In particular, the closed-form expression is not valid for our constrained randomization and causal forest treatment effect estimation.

We next perform power calculations mentioned above using the experiment data. In randomizing the clusters, we balanced across the treatment and control groups on different financial metrics and the cluster size following the process described in Sect. 4.1.3. We consider a 6%, 8%, and 10% effect sizes (since this experiment was not well-powered, we selected larger effect sizes). Table 4 summarizes the power results.

Table 4 Closed-form and simulation-based power calculations