Estimating dynamic treatment effects in event studies with heterogeneous treatment effects

Abstract

To estimate the dynamic effects of an absorbing treatment, researchers often use two-way fixed effects regressions that include leads and lags of the treatment. We show that in settings with variation in treatment timing across units, the coefficient on a given lead or lag can be contaminated by effects from other periods, and apparent pretrends can arise solely from treatment effects heterogeneity. We propose an alternative estimator that is free of contamination, and illustrate the relative shortcomings of two-way fixed effects regressions with leads and lags through an empirical application.

Introduction

Rich panel data has fueled a growing literature estimating treatment effects with two-way fixed effects regressions. This body of applied work has prompted a corresponding econometrics literature investigating the assumptions required for these regressions to yield causally interpretable estimates. For example, Athey and Imbens (2018), Borusyak and Jaravel, 2017, Callaway and Sant’Anna, 2020a, de Chaisemartin and D’Haultfœuille, 2020 and Goodman-Bacon (2018) interpret the coefficient on the treatment status when there are treatment effects heterogeneity and variation in treatment timing. Researchers are often also interested in dynamic treatment effects, which they estimate by the coefficients ${\mu }_{\ell }$ associated with indicators for being $\ell$ periods relative to the treatment, in a specification that resembles the following:

$$Y_{i,t}={\alpha }_i+{\lambda }_t+\sum _{\ell }{\mu }_{\ell }\mathbf{1}\{t-E_i=\ell \}+{\upsilon }_{i,t}.$$

Here $Y_{i,t}$ is the outcome of interest for unit $i$ at time $t$, $E_i$ is the time when unit $i$ initially receives the binary absorbing treatment, and ${\alpha }_i$ and ${\lambda }_t$ are the unit and time fixed effects. Units are categorized into different cohorts based on their initial treatment timing. The relative times $\ell =t-E_i$ included in (1) cover most of the possible relative periods, but may still exclude some periods.

The first goal of this paper is to uncover potential pitfalls associated with using the estimates of the relative period coefficients ${\mu }_{\ell }$ as “reasonable” measures of dynamic treatment effects. We decompose ${\mu }_{\ell }$ to show it can be expressed as a linear combination of cohort-specific effects from both its own relative period $\ell$ and other relative periods; unless strong assumptions regarding treatment effects homogeneity hold, the terms that include treatment effects from other relative periods will not cancel out and will contaminate the estimate of ${\mu }_{\ell }$. Importantly, this demonstrates that the widespread practice of using estimates of treatment leads in (1) as a way of testing for parallel pretrends is problematic. Roth (2019), in his survey of the applied literature, notes that checking whether ${\mu }_{\ell }=0$ for $\ell$ leads of treatment is a common test for pretrends. Our decomposition result implies that such a test would be invalid because the estimate of ${\mu }_{\ell }$ is affected by both pretrends and treatment effects heterogeneity, thus any test of ${\mu }_{\ell }=0$ cannot accept or reject the existence of pretrends without further assumptions on treatment effects.

We show how to calculate the weights underlying the linear combination of treatment effects in ${\mu }_{\ell }$ using an auxiliary regression. This auxiliary regression depends only on the distribution of cohorts and the relative time indicators included in (1). Examining the weights allows researchers to gauge how large the amount of treatment effects heterogeneity needs to be for ${\mu }_{\ell }$ to be contaminated by treatment effects from other relative periods. Our publicly-available Stata package eventstudyweights automates the estimation of these weights using the panel dataset underlying any given specification of (1).

The second goal of this paper is to propose an alternative regression-based method that is more robust to treatment effects heterogeneity than regression (1). For dynamic treatment effects, researchers are usually interested in estimating some average of treatment effects from $\ell$ periods relative to the treatment. Our alternative method estimates the shares of cohort as weights. These weights are more interpretable than the weights underlying regression (1) in the presence of treatment effects heterogeneity, and the resulting weighted average of treatment effects extends beyond a convex combination of treatment effects (Słoczyński, 2020). As discussed in Section 4.2, using the procedures of Callaway and Sant’Anna (2020a), our alternative method can also accommodate covariates.

We illustrate both our decomposition results and our alternative method via an empirical application, estimating the dynamic effects of a hospitalization. We follow Dobkin et al. (2018) in using the publicly-available dataset, Health and Retirement Study (HRS), to first estimate two-way fixed effects regressions. We then illustrate our alternative estimation method with this example. Among the outcomes studied by Dobkin et al. (2018), we focus on out-of-pocket medical spending and labor earnings. Our alternative method yields similar big-picture findings as the original paper that uses two-way fixed effects regressions: the earnings decline due to hospitalization is substantial compared to the transitory out-of-pocket spending increase. However, the two-way fixed effects estimates sometimes fall outside the convex hull of the underlying effects. In contrast, estimates using our alternative method, by construction, are guaranteed to be easy-to-interpret because they are weighted averages of the underlying effects, with weights corresponding to cohort shares.

The rest of the paper is organized as follows: In the next subsection, we review the theoretical literature. Section 2 formally introduces the event study design and discusses our definition in relation to the applied literature. Section 3 derives the estimands of two-way fixed effects regression, and introduces sufficient assumptions for them to be causally interpretable. Section 4 develops our alternative estimator. Section 5 illustrates our results using an empirical example and Section 6 concludes. All proofs are contained in the Appendix B Proofs of decomposition results, Appendix C Supplementary results related to IW estimator.

This paper makes two main contributions within an active literature on the causal interpretations of two-way fixed effects models in settings with staggered treatment adoption (Athey and Imbens, 2018, Borusyak and Jaravel, 2017, Callaway and Sant’Anna, 2020a, de Chaisemartin and D’Haultfœuille, 2020, Goodman-Bacon, 2018). Our paper is also related to the traditional literature analyzing non-separable panel and treatment effects models e.g. Heckman et al., 1998, Heckman et al., 1997, Blundell et al., 2004, Abadie, 2005, Chernozhukov et al., 2013.

The first main contribution of our paper is to interpret estimates from two-way fixed effects specifications when researchers include “dynamic” indicators for time relative to treatment and when treatment effects are heterogeneous across adoption cohorts. We derive our results for a general class of two-way fixed effects specifications where “dynamic” indicators can be flexibly specified as single relative periods $\ell$ or sets of relative periods $g$ (thus also capturing any “static” specification where all post-treatment indicators are collected in a single set). This class of specifications encompasses all specifications addressed by Athey and Imbens, 2018, Borusyak and Jaravel, 2017, Callaway and Sant’Anna, 2020a, de Chaisemartin and D’Haultfœuille, 2020 and Goodman-Bacon (2018).

As a building block for the causal interpretation of estimates, we define $CAT{T}_{e,\ell }$, the cohort average treatment effects on the treated as the cohort-specific average difference in outcomes relative to never being treated. Our choice of a “building block” is governed by the counterfactual and the type of heterogeneity of interest. This object coincides with the “group-time average treatment effect” studied by Callaway and Sant’Anna (2020a) and is more granular than the building block used by Goodman-Bacon (2018) that is an average of $CAT{T}_{e,\ell }$ over some relative period range. Athey and Imbens (2018) consider an alternate counterfactual to never being treated: being treated at a different time. Borusyak and Jaravel (2017) implicitly assume away heterogeneity across cohorts within a relative period, so their building block reduces to $AT{T}_{\ell }$. de Chaisemartin and D’Haultfœuille (2020) allow for heterogeneous treatment paths within a cohort over time across “groups”, thus their building block is at the group level. We defer a discussion of assumptions underlying the causal interpretation of these building blocks to Section 2.

The second main contribution of our paper is to propose a simple regression-based alternative estimation strategy that produces a more sensible estimand than conventional two-way fixed effects models under heterogeneous treatment effects. Our procedure is most similar to Callaway and Sant’Anna (2020a), but has the following differences. First, in the setting where there is no never-treated group, our method uses the last cohort to be treated as a control group, whereas Callaway and Sant’Anna (2020a) use the set of not-yet-treated cohorts. Our method and theirs thus rely on different, but non-nested parallel trends assumptions. Second, our estimation method can be cast as a regression specification and thus may be more familiar to applied researchers. However, a third difference is that the procedure of Callaway and Sant’Anna (2020a) allows for conditioning on time-varying covariates. de Chaisemartin and D’Haultfœuille (2020) and Goodman-Bacon (2018) respectively propose alternative estimators and diagnostic tools for estimation of causal effects in staggered settings, but do not consider the estimation of the dynamic path of treatment effects as we do.