The Effect of COVID-19 Vaccinations on Self-Reported Depression and Anxiety During February 2021

Abstract Using the COVID-19 Trends and Impact Survey, we estimate the average effect of COVID-19 vaccinations on self-reported feelings of depression and anxiety, isolation, and worries about health among vaccine-accepting respondents in February 2021, and find 3.7, 3.3, and 4.3 percentage point reductions in the probability of each outcome, respectively, with particularly large reductions among respondents aged 18 and 24 years old. We show that interventions targeting social isolation account for 39.1% of the total effect of COVID-19 vaccinations on depression, while interventions targeting worries about health account for 8.3%. This suggests that social isolation is a stronger mediator of the effect of COVID-19 vaccinations on depression than worries about health. We caution that these causal interpretations rely on strong assumptions. Supplementary materials for this article are available online.


Introduction
The COVID-19 pandemic and the associated social isolation, economic hardship associated with job loss, and social uncertainty led to an increase psychological distress ( [24]). For example, [9] showed that the same number of people experienced serious psychological distress in May 2021 as the entire year prior to the pandemic.
[27] estimated that the pandemic caused a 27.6% increase in cases of major depressive disorder and a 25.6% increase in cases of anxiety disorders globally. Younger adults have consistently been found to be among the most adversely subgroups ([27], [28]). Secondary consequences of these mental health impacts, such as suicide and drug overdoses, are also of concern (see, e.g., [16], [13]).
If COVID-19 pandemic worsened mental health, then perhaps receiving a COVID-19 vaccination improved it. This might happen, for example, by decreasing worries about health or social isolation. We investigate this hypothesis by estimating the effects of COVID-19 vaccinations on the rates of depression and anxiety, feelings of social isolation, and worries about health in February 2021. Our analysis is based on the COVID-19 Trends and Impacts Survey (CTIS) ([26]), a cross-sectional survey designed by the Delphi group at Carnegie Mellon University and administered in collaboration with Facebook. We quantify the average effect of vaccinations on these outcomes among a subset of vaccine-accepting Facebook users during February 2021. By vaccine-accepting we mean CTIS respondents who indicated that they either would receive a COVID-19 vaccine if offered one that day, or who indicated that they had already received a COVID-19 vaccine. By depression and anxiety we mean self-reported feelings, not a clinical diagnosis. A key assumption for the causal interpretation of this analysis is that conditional on vaccine-acceptance and observed covariates, vaccinated and unvaccinated respondents differed on average only in vaccination status.
We also examine effect heterogeneity and mediation. For our heterogeneity analysis, we consider both pre-specified and data-driven subgroups, using an approach similar to [3].
For our mediation analysis, we consider a model where COVID-19 vaccinations affect depression and anxiety through feelings of social isolation, worries about health, and a direct effect capturing all other channels. We allow the causal ordering of the mediators to be unknown; however, we assume that they are determined prior to the outcome, and that no other effects of vaccinations affect these mediators. Using the interventional effect decomposition proposed by [30], we estimate the proportion of the total effect on depression via each mediator. While interventional indirect effects do not necessarily reflect mechanisms without stronger assumptions ( [21]), this is the first study we are aware of that attempts to estimate the effects of COVID-19 vaccinations on depression via intermediate pathways.
Section 2 presents a brief review of the related literature and our contributions. Section 3 describes our data and provides summary statistics. Section 4 outlines our identification, estimation, and inferential strategy. Section 5 presents our results. Section 6 considers the sensitivity of our results to our causal assumptions and their robustness to analytic choices, and Section 7 provides discussion and conclusion. Additional materials are available in the Appendices.

Related Literature
Two concurrent studies examined the effect of COVID-19 vaccinations on mental health: [23] and [1]. [23] use longitudinal data from the Understanding America Survey and fixedeffects models to estimate that vaccines caused a 4% reduction of the probability of being at least mildly depressed and a 15% reduction in the probability of being severely depressed.
When testing for effect heterogeneity, they were unable to rule out that the effects were the same across sex, race, and education level. [1] use repeated cross-sectional survey data from Census Pulse. Using heterogeneity in state-level eligibility requirements as an instrumental variable, they use two-stage least squares to estimate that COVID-19 vaccination reduces anxiety and depression symptoms by nearly 30%. They find larger effect sizes among individuals with lower education levels, who rent their housing, who are unable to telework, and who live with children.
Our identification strategy instead hinges on an assumption that sample selection and COVID-19 vaccination are independent of potential outcomes given vaccine-acceptance and observed covariates. This assumption precludes confounding at the population-level and biases induced by conditioning on our sample. We also exploit auxiliary data to identify potential high impact subgroups hence mitigating the risk of false discoveries. Finally, we estimate interventional indirect effects to better understand the mechanisms through which COVID-19 vaccinations affect depression and anxiety. By contrast, the previous studies only identify average effects or local average effects, do not use sample-splitting for their heterogeneity analyses, and do not address mediation beyond speculation. A key contribution of our paper is therefore our mediation analysis. This analysis hinges on the idea that worries about health and social isolation are the two primary pathways through which vaccinations would change depression and anxiety. These are commonly cited as two major ways that the pandemic has adversely impacted mental health ( [25], [14], [11]).

Data
Our primary dataset is the COVID-19 Trends and Impact Survey (CTIS), created by the We analyze the period from February 1-28, 2021 1 We choose this timeframe to increase the plausibility of our identifying assumptions, which we discuss further in Section 4.4.
Our primary outcome (Y ) is whether a respondent reported feeling depressed or anxious "most" or "all" of the time in the previous five days. We hereafter refer to this variable as "depressed." Our treatment variable (A), is an indicator of whether the respondent ever received a COVID-19 vaccine. 2 Importantly, the CTIS does not tell us when a respondent was vaccinated. We also consider responses to how often a respondent felt isolated from others in the previous five days (M 1 ), and how worried respondents reported feeling about themselves or an immediate family member becoming seriously ill from COVID-19 (M 2 ), each containing four levels. 3 We hereafter refer to these variables as "isolated" and "worries about health," respectively. We also create the variable M as the sixteen level joint variable M 1 × M 2 . Finally, we dichotomize M 1 and M 2 when analyzing them as outcomes.
Specifically, we create indicators for the responses "most" or "all" of the time for isolation, and "very worried" for worries about health. Whenever referring to our outcomes analyses, for simplicity we refer to any outcomes as Y .
We consider several covariates recorded in the CTIS pertaining to demographic, household, employment, and health status. For demographics we consider each respondent's age category, race, ethnicity, gender, and highest education attained. For household variables, we construct indicators of whether a respondent lives alone; lives with a child attending school in-person full-time, part-time, or not in school; or lives with an elderly (over 65) individual. For employment, we record whether each respondent worked for pay in the previous 30 days, and whether their work was at home or outside. Among respondents who work, we record the respondent's occupation type, including healthcare, education, service-industry workers, protective services, other employment sector. Finally, we record information regarding each respondent's health and health behaviors, including whether they have previously been tested for and/or diagnosed with COVID-19, how many "highrisk" health conditions they have ever been diagnosed with, (0, 1, 2, or three or more), and whether they received a flu shot since April 2020 (included as a proxy for health behaviors). We also record whether the respondent took the survey in English or Spanish. For all of the CTIS survey-measured covariates, we recode missingness as a separate category to account for item non-response. Lastly, we augment the dataset with indicators of each 2 We recode the response "I don't know" to be grouped with people who indicated they did not already receive a COVID-19 vaccine. 3 The CTIS questions specifically are "In the past 5 days, how often have you felt isolated from others?" and "How worried are you that you or someone in your immediate family might become seriously ill from COVID-19 (coronavirus disease)?" respondent's state and week of response. The state indicators can control for fixed statelevel policies during the study period that may be associated with both vaccine access and depression, including eligibility requirements, while the week indicators control for time trends associated with both treatment and outcomes.
We also join several county-level variables using each respondents' FIPS code. Using data from the 2019 American Community Survey, we add the county-level population, population density, percent living in poverty, percent uninsured, and Gini index. Using the National Center for Health Statistics Urban-Rural classification scheme, we classify the type of county each respondent lives in (Large Central Metro, Large Fringe Metro, Medium Metro, Small Metro, Micropolitan, and Non-core regions). These variables help control for differential COVID-19 vaccine access. We also include the county the total number of deaths per capita from COVID-19 in each county the final two weeks in January, and divide this variable into sextiles (data obtained from Johns Hopkins University). Finally, we include Biden's county vote share at the county-level (Biden minus Trump vote totals divided by total votes). We denote all demographic and county-level covariates as X.
Our initial dataset from February includes 1,232,398 responses. We exclude 2,530 respondents whose Facebook user language was not English or Spanish, 3,690 responses from Alaska -Alaska did not report 2020 voting results at the county-level -and 39,526 responses that did not provide FIPS code information -preventing us from merging countylevel covariates. We then subset our data to only include respondents who say that they "probably" or "definitely" would get the COVID-19 vaccine if offered to them today or who  The prevalence of depression varies substantially by age group. The left-hand panel of Figure 2 shows the weekly prevalence from October 2020 through February 2021, with rates increasing monotonically across seven age groups. This may depict a real negative correlation between depression and age; however, an alternative interpretation could be that younger individuals have a different vocabulary around mental health than older individuals and respond to these questions differently. While possible, Figure 2    Our study examines whether COVID-19 vaccinations in part account for these improving outcomes during February 2021. We begin by examining the differences in the prevalence of each outcome in vaccinated versus unvaccinated respondents that month, taking these averages within separate subgroups. Figure 3 displays these results. 5 See, e.g., https://www.nytimes.com/interactive/2021/us/covid-cases.html 6 As has been observed elsewhere, the CTIS over-represents vaccinated individuals compared to the U.S.
population ([26]). This does not necessarily threaten the internal validity of our analysis (see Section 4.3), though may threaten our ability to generalize our results.  Given the high base-rates of depression and isolation among the youngest respondents, perhaps not surprisingly we observe the largest declines in the prevalence of each outcome among younger respondents relative to older ones. We also see larger declines in rates of depression and isolation among White relative to Black respondents, and non-Hispanic relative to Hispanic respondents. When examining worries about health by each demographic category, the declines appear lower among the the oldest age groups, higher among people with three or more high-risk conditions, and higher among Hispanic versus non-Hispanic respondents. Broadly speaking, patterns in depression and isolation appear more similar to each other than to patterns in worries about health.
Of course, COVID-19 vaccinations were not randomly assigned, and we cannot make causal claims from these associations. For example, individuals who were most at-risk and who had the means were most likely to receive the vaccination. If these same individuals were also more likely to be depressed or anxious without the vaccine, then these associations would overstate the absolute magnitude of any treatment effects. We therefore adjust these estimates to control for the demographic, household, employment, health, and county-characteristics described previously, as well as state and week indicators, and outline the assumptions necessary for these adjusted associations to reflect causal quantities.
Additional summary statistics are available in Appendix 8.

Preliminaries
Recall that A indicates prior receipt of COVID-19 vaccination, M 1 isolation, M 2 worries about health, and Y depression. X indicates the covariate vector described in Section 3, and V indicates vaccine-acceptance. We define S as an indicator of responding to the survey, , respectively. We observe n independent samples from a pool of active Facebook users residing in the United States We define two important functions of this distribution. First, we let This is an indicator of whether we observe all outcomes, mediators, and treatment assignment information (hereafter "complete-cases") for a respondent. Second, define Z = RSV .
In other words, Z is an indicator of inclusion in our analytic sample.

Estimands
We next define causal estimands with respect to our outcomes and mediation analyses using potential outcomes notation. Before defining these quantities, we make the following assumptions. First, we invoke the SUTVA. This implies that there is only one version of treatment and rules out interference between individuals: specifically, each individual's potential outcomes only depend on their own vaccination status. Second, we assume no carryover effects. This implies that an individual's potential outcome having been vaccinated one week prior to responding to the CTIS is identical to their potential outcome were they vaccinated one month prior. Finally, we assume no anticipatory effects, implying that future vaccination status does not affect one's potential outcomes at the time an individual responded to the CTIS.
Under these assumptions, we define the potential outcomes for each individual as Y a i i : that is, an individual's outcome when setting their vaccination status A i to level a i . This quantity does not depend on anyone else's vaccination status (SUTVA), their past vaccination status (no carryover effects), nor their future vaccination status (no anticipatory effects). Finally, we define the function µ a ( , representing the expected potential outcome under treatment A i = a for an individual given their covariates, vaccine-acceptance, and being in our analytic sample.

Outcomes
We use Y to denote any outcome considered in this section to ease notation, recalling that M 1 and M 2 are dichotomized for this analysis. Assume that there are N adults living in the United States. A natural estimand is the average treatment effect: This reflects the mean difference in the prevalence of each outcome if everyone were vaccinated contrasted against the case where no one was vaccinated, averaging over the empirical covariate distribution of adults residing in the United States.
This causal quantity is difficult to target for at least four reasons. First, the U.S. adult population consists of two types of individuals: the vaccine-hesitant and the vaccineaccepting. The vaccine-hesitant constitute a group of "never-takers," a group would never choose be vaccinated (at least during this timeframe). We cannot estimate the treatment effect among this subgroup without strong modeling assumptions. Second, the sample is taken over the population of Facebook users who reside in the United States. This population differs from the U.S. adult population, the arguably more interesting population. Third, the average over the population of Facebook users is also difficult to credibly estimate, as survey response rates in February 2021 were approximately one percent. Unless we assume that survey non-response was completely random, we cannot estimate this quantity without further assuming, for example, that the selection process was based on pre-treatment characteristics and then modeling this process. Finally, among the survey respondents, we only observe the vaccination status and outcomes among those who respond to these items. Similar to survey non-response, generalizing the causal effects to include these individuals requires additional assumptions and modeling.
We make several analytic choices to address these challenges. First, we condition our analysis among the vaccine-accepting population. Second, our primary estimands are conditional on our analytic sample. Specifically, we define our target estimand as: 7 This estimand is the average expected difference in the prevalence of the outcomes given the empirical covariate distribution of the vaccine-accepting complete-cases in our analytic sample. We abbreviate this empirical average using P n moving forward. We also examine all effects on both the risk-ratio scale (P n [µ 1 (X)]/P n [µ 0 (X)]) and averaged within distinct subgroups defined by the covariates. In Appendix 12.4 we consider estimands that do not deterministically set vaccination status, but rather shift each individual's probability of vaccination ( [17]).

Interventional effects
We next decompose the effect of vaccinations on depression into four separate channels: a direct effect, an indirect effect through social isolation, an indirect effect through worries about health, and an indirect effect reflecting the dependence of social isolation and worries about health on each other, following [30]:

Identification
We assume the following causal structure holds among vaccine-accepting individuals.  Figure 4 also encodes the assumed relationships between the variables necessary to identify our causal estimands in terms of our observed data. We instead motivate these conditions using potential outcomes, and assume that for all values of (a, m 1 , m 2 ): 9 Assumption 1 (No unmeasured confounding).
Specifically, no unmeasured confounding states that among the vaccine-accepting population, vaccinations are independent of the potential mediators and outcomes conditional on covariates. Y-M ignorability states that the mediators are jointly independent of potential outcomes conditional on covariates and vaccination status. Finally, we we assume random sample selection: this states that survey response and complete-case indicators are random with respect to the potential outcomes given the covariates and vaccination status. We additionally assume that the survey responses fully capture all population relationships ("no measurement error"), that all vaccine-accepting individuals in our sample have a probability of vaccination bounded away from zero and one, and that the joint mediator probabilities are all bounded away from zero ("positivity"). We formalize all identifying assumptions in Appendix 10 and show that they imply that our causal parameters are identified in terms of the observed data. For example, We largely defer discussing possible violations of these assumptions to Section 6.

Analytic timeframe
We choose to limit our causal analysis to February 2021 to strengthen the credibility of our identifying assumptions. The challenge is that we only observe cross-sectional data, and the causal assumptions required for our analysis simplify a more complicated longitudinal process. Our identifying assumptions arguably become less tenable over time. As one example, consider a key unobserved variable: time since vaccination. Assuming no carryover effects implies that this variable is not required either to define or identify our causal estimands. Absent this assumption, time since vaccination would act as an unmeasured confounder inducing a bias for our outcomes analysis that plausibly increases with time from the first administered COVID-19 vaccines (we discuss this point further in Appendix 10). Our mediation analysis similarly becomes less tenable over time: for example, vaccinations may improve economic conditions over time, which in turn may act as a posttreatment confounder with respect to feelings of social isolation or worries about health.
We therefore limit our analysis to February, relatively soon after the start of COVID-19 vaccination administrations.

Estimation and inference
We use influence-function based estimators throughout ( [4], [17]), requiring estimaion of several nuisance functions. For example, for the interventional effects, we estimate the outcome model, the propensity score model, and the joint mediator probabilities. For our primary results we use logistic regression and multinomial logistic regression. We allow for separate models for each treatment level for our outcomes analysis, and each treatment-mediator level for our mediation analysis. Similarly, to estimate mediator probabilities we allow separate models within each treatment level. Our propensity score model includes only main effects for each covariate. These estimators are multiply robust in the sense that the consistency of the estimates do not require that all models are correctly specified, but rather that some subset are ( [4]). We generate 95% confidence intervals and conduct hypothesis tests using the empirical variance of our estimated influence function values and standard normal quantiles. These intervals and tests are valid if all of our models are correctly specified. To alleviate concerns about model misspecification, we also estimate all nuisance functions using XGBoost and sample-splitting. Assuming the consistency of the influence-function estimates and that that the nuisance functions are estimated at n −1/4 rates in the L 2 norm, these estimates are asymptotically normal and root-n consistent with a simple expression for the variance ( [4]). Appendix 11 provides further details.
We also use a data-driven approach to identify subgroups with large or small treatment effects. Specifically, we estimate the influence function values using auxiliary data from March 1-14, 2021 using XGBoost. We regress these pseudo-outcomes on a subset of the demographic covariates using depth-four regression trees. After generating the trees, we use a "human-in-the-loop" approach to determine whether the candidate subgroups are interpretable and substantively interesting. We then conduct estimation and inference using on these candidate hypotheses using the February data. 10 All t-tests referenced below set α = 0.05 and only control the Type I error rate for each test marginally.  Table 3 displays our results when using XGBoost to estimate the nuisance functions. These estimates are quite similar to the GLM estimates.   respondents aged 65 and older. We also find larger magnitude point estimates for females relative to males and non-binary respondents for depression. Specifically, we estimate the 11 As noted in Appendix 10, we may interpret these results as bounding the effect of the pandemic on depression. For example, under some assumptions we can say that the pandemic increased depression by at least 3.7 percentage points on average, and that the percentage of depressed respondents was at least 24 percent higher on average (100*( 1 1−.19 − 1)) than it would have been absent the pandemic among respondents in our analytic sample. effect of vaccinations on depression as being -4.0 (-4.3, -3.7) percentage points among female identifying respondents and -3.1 (-3.5, -2.7) among male and non-binary respondents.

Outcome analysis
Lastly, we see larger effect estimates among respondents for worries about health among respondents with two or more high-risk health conditions (-4.8 (-5.3, -4.3)) relative to those with none (-3.7 (-4.2, -3.2)). 12 health. Because non-response is highly correlated across questions, we observe similar magnitude estimates for other non-response categories, though the point estimates for isolation tend to be comparable to other groups.

Data-driven subgroup heterogeneity
We identify three possibly heterogeneous subgroups using the March data with respect to depression: respondents aged 18-24, non-Hispanic respondents aged 25 and older, and Hispanic respondents aged 25 and older. These categories exclude non-respondents who to age, or to ethnicity among those 25 or older. Figure 6 displays the effect sizes and confidence intervals estimated on the February data. 13 Consistent with our hypotheses, we estimate effects of -12.2 (-13.7, -10.7) among respondents aged 18-24, -3.6 (-3.  We examine additional subgroups within these three primary groups. Among respondents aged 18-24, we estimate larger absolute magnitude effect sizes among those who live with an elderly individual (-21.6 (-26.5, -16.8); N = 2,542) versus those who do not (-10.9 (-12.7, -9.2); N = 20,347). We estimate effects for depression and isolation that are statistically indistinguishable from zero among Hispanic respondents whose primary Facebook user language is Spanish (N = 23,018). By contrast, all estimated effect sizes are statis- 13 We present estimates using our XGBoost models: our GLM estimates do not include interactions between the covariates, and are therefore unlikely to correctly capture this heterogeneity.
14 We display significance levels associated with the pairwise t-tests in Appendix 12.2. tically significant and negative for those whose Facebook user language is English (N = 57,236). 15 We test for differences with respect to gender and race among respondents aged 18-24, and with respect to age among Hispanic respondents aged 25 or older. We find statistically significant differences with respect to race among the youngest respondents for isolation and worries about health: the effect sizes appear smaller in absolute magnitude for non-White versus White respondents. However, our remaining hypotheses do not yield any statistically significant and substantively meaningful differences on the risk-differences scale.
Finally, we identify candidate heterogeneous subgroups using isolation and worries about health as the outcomes. Beyond the previously noted heterogeneity with respect to age, we are largely unable to confirm additional sources of heterogeneity. One exception is with respect to gender among non-White respondents aged 25 or older: we estimate that female identifying respondents had larger magnitude effects with respect to social isolation (-3.2 (-4.0, -2.5)) than non-female identifying respondents (-1.5 (-2.4, -0.7)). Additional results may be found in Appendix 12.2.
Our results reveal that being aged 18-24 is a key source of effect heterogeneity, especially with respect to depression and social isolation. Moreover, living with an elderly individual appears to have a strong interaction effect among this same group. These patterns also hold on the risk-ratio scale (see Appendix 12.2). Additional research would be valuable to further investigate hypotheses with respect to other demographic characteristics, including interactions between age, race, gender, and ethnicity, though our results suggest that age would likely remain the factor most strongly associated with effect heterogeneity.  We also find evidence of a direct effect of vaccination on depression that accounts for over half of the total effect. We interpret this as reflecting a general sense of relief that vaccination provides, perhaps related to a belief in a return to normalcy. The patterns are consistent, with social isolation appearing to account for a larger proportion of the mediated effect relative to worries about health across all groups. Perhaps not surprisingly, worries about health appear to account for a slightly greater percentage of the total effect among people with two or more high-risk health conditions compared to those with none (though we do not formally test the statistical significance of these differences).

Mediation analysis
The XGBoost results are similar. Additional results are available in Appendix 12.3. 16 We excluded Asian and Pacific Islander and Other racial categories as the estimates are very imprecise. We consider the sensitivity of our analyses to our causal assumptions, their robustness to alternative analytic choices, and our ability to generalize our effect estimates. We highlight the limitations of our analysis throughout, and also provide heuristic arguments about how violations of our causal assumptions may bias our estimates. We conclude that our outcomes analysis is fairly robust, and that most possible violations likely bias our estimates towards rather away from zero. We also argue that generalizing the effect magnitudes of our estimates beyond our sample is not unreasonable. Finally, we discuss the limitations of our mediation analysis; however, we do not conduct formal sensitivity analyses.

Unobserved confounding and selection bias 6.1.1 Sensitivity analysis
Our outcomes analysis assumes no unmeasured confounding and random sample selection.
Together these assumptions imply that the potential outcomes are independent of treatment assignment given covariates and vaccine-acceptance on our sample. This assumption may be violated either assumption fails. Unmeasured confounders may threaten this assumption at the population level, or selection into our analytic sample may induce bias. We can relax these assumptions and instead estimate obtain bounds on our target sample estimand via a variation of a sensitivity analysis proposed by [20].
We then assume there exists a τ satisfying equation (7).
For example, consider the left-hand side of equation (7) and let a = 1 and a = 0.
The bound specifies that within each covariate stratum, the counterfactual probability of depression among unvaccinated survey respondents when vaccinated is not greater than than 1/(1 − τ ) times the observed probability of depression among the vaccinated survey respondents. In Appendix 13, we show that equation (7) implies a bound on the treatment effect estimates using only functions of the observed data and τ . We then use influencefunction based estimators to estimate bounds across a range of values of τ until we find values that could explain away our estimated effects.
We estimate that values of τ greater than or equal to 0.18, 0.15, and 0.14 would explain away our effect estimates for depression, isolation, and worries about health, respectively.
In other words, our effect estimates would be statistically indistinguishable from zero if the ratios in equation (7)  This analysis is quite conservative: for example, the bounds consider cases where the biases work in opposite directions for µ 1 and µ 0 . However, the biases may work in the x, then then our estimates of the causal risk-differences would remain unbiased, despite biased estimates of the prevalence of each potential outcome in our sample.

Heuristic reasoning: no unmeasured confounding
We can also reason about the sign of the bias induced by violations of no unmeasured confounding at the population level. Two specific confounders we may worry about are vaccine-access and motivation to be vaccinated. We attempt to control for vaccine-access using a variety of county-level characteristics, the respondents' geographical location, employment status, and financial stress. However, these covariates may not sufficiently capture differential access across respondents. Similarly, vaccine-acceptance acts as a rough proxy for motivation. Yet even among vaccine-accepting respondents, some likely have higher motivation to be vaccinated than others. We reason that failing to control for these variables likely induces biases that work in opposite directions, resulting in an overall bias with no clear sign.
For example, if within every covariate stratum, the probability of being vaccinated is lower for respondents with worse access, and the probability of being depressed, isolated, or worried about health is higher for these same respondents, our estimates will be negatively biased. On the other hand, if motivated respondents are more likely to be vaccinated, and more likely to be depressed, isolated, or worried about their health generally, our estimates will be positively biased. Our data is consistent with these hypotheses: when running our analyses without controlling for financial stress, employment status, or occupation, 18 These values reflect cases where the observed covariates are assumed to deconfound the relationship on the observed sample. This does not account for scenarios where bias is induced by selection on the observed or potential outcomes.
our point estimates move further away from zero (see also Section 6.4). On the other hand, when including the vaccine-hesitant in our analytic sample, our estimates move closer to zero (see Table 36 in Appendix 14). The total bias induced by these unmeasured confounders therefore has no clear sign.

Generalizability
Assuming that our estimates among this sample of CTIS respondents are valid, or more weakly that the magnitude of the effect estimates is correct, we may wish to infer similar magnitude effects among all U.S. adults. Skepticism is warranted for several reasons. First, CTIS respondents differ from the general US population across several characteristics.
For example, [12] found that the CTIS survey over represents White individuals, those with higher education, and those who have received COVID-19 vaccinations (see also [8]).
Second, our estimates may be subject to selection-bias: even if we knew the true sample estimand, if sample selection were based on the potential or realized outcomes, these effects may not generalize to a population of non-CTIS respondents with identical covariates.
Finally, our estimates make no claims about the vaccine-hesitant, which is a substantial portion of the U.S. population ( [19]).
While caution is certainly warranted, we nevertheless argue that these concerns should not entirely prohibit generalizations from our study. First, assume that our conditional effect estimates are valid. Despite our sample's non-representative covariate distribution, we do not find evidence of substantial sources of effect heterogeneity beyond age. Averages over other mixes of vaccine-accepting populations are therefore plausibly similar. Second, sample selection may in truth be a function of the observed or potential outcomes. However, this does not imply our estimates are invalid: our sample may not be representative of the prevalence of the potential outcomes, but may still be representative of the risk-differences.
Third, if we wished to generalize across a population that includes a vaccine-hesitant subset, we can simply assume that there is no treatment effect on average in this subgroup. As long as the vaccine-hesitant comprise a minority of the target population, similar magnitude effect sizes may therefore hold. 19 19 As a more formal analysis, in Appendix 13 we extend the previous sensitivity analysis to evaluate how Such generalization may still strike some as implausible. We may instead wish to consider the more limited goal of generalizing our estimates beyond the complete-cases to the entire vaccine-accepting CTIS sample. We also run this analysis and find that our estimated effect sizes are similar to our primary analysis (see Appendix 14).

Interference, carryover, and anticipatory effects
Our analysis makes three restrictions on the dependence of the potential outcomes across Consider the following estimand: This formalizes the difference in rates of depression in a world where everyone is vaccinated versus a world where no one is vaccinated, allowing each individual's potential outcomes to depend on everyone else's vaccination status within their community. We can then consider how well have we estimated ϑ. Consider the model: our effects might generalize across a population with the same covariate distribution as our sample but include entirely cases where RS = 0. This analysis suggests that our ability to generalize our estimates may be weak; however, it is also very conservative.
Equation (9) implies that ϑ = P n (β 1X +β 2X ). This estimand captures both a direct effect of vaccinations, captured by P n (β 1X ), and an indirect effect captured by P n (β 2X ). Assume that for all x, β 1x , β 2x < 0 -that is, that both the direct and indirect effects of vaccination are negative within all covariate strata. To make these assumptions concrete, consider depression as the outcome. These assumption would be reasonable if (1) vaccinations induce people to socialize, and socializing decreases depression, and (2) socializing increases with the number of vaccinated individuals within all covariate strata.
However, our estimates are not based on equation (9), but instead on the model: Our estimators therefore target the biased quantityθ = P n [β 1X ]. The omitted variable bias formula implies that this estimand is positively biased with respect to ϑ. 20 Whether ϑ is the correct target of inference, or whether these assumptions are realistic is debatable.
Nevertheless, this logic provide justification for believing that SUTVA violations would bias our effect estimates toward, rather than away, from zero. 21 20 To see this, consider the model: Within each covariate stratum the bias of our target quantity relative to our target is equal to β 2x (δ 1x −1).
The sample average of this bias across all covariate strata is greater than zero: β 2x < 0 for all x by assumption, and δ 1x ≤ 1 by the definition of S(A c−i ). As long as vaccinations aren't perfectly correlated within every community, there exists some x in our sample where δ 1x < 1, implying positive bias.
If vaccinations were independently assigned, δ 1x = 0,β 1x = β 1x , andθ would correctly capture the direct effects of vaccination on the outcomes (but not the total direct and indirect effects). The sign of the bias of the direct effect is unclear and depends on δ 1x . 21 [1] attempt to estimate spillover effects by examining whether increases in community vaccination rates improve depression and anxiety symptoms among the unvaccinated population. They are unable to find evidence that this is the case, providing some empirical evidence that we may need not worry about SUTVA violations affecting our estimates.
We also assume no carryover effects, precluding the potential outcomes from depending on the time since receipt of vaccination. Were this violated, time since vaccination would essentially be a confounder that is positively associated with the observed treatment assignment indicator. Assuming that the effects of COVID-19 vaccinations on mental health monotonically diminish over time on average within each covariate stratum, violations of no carryover effects would be bias our estimates towards zero. Finally, we assume no effects in anticipation of receiving the vaccine. If anticipatory effects were to improve mental health on average in our sample, violations would again bias our effect estimates towards zero. In short, violations of SUTVA, no carryover effects, and no anticipatory effects most plausibly bias our effect estimates towards zero.

Bad controls
So-called "bad controls" may also bias our effect estimates ( [2]). We argue that these biases again plausibly move our estimates toward zero.
Specifically, we control for occupation, employment status, and worries about finances.
These variables are associated with vaccine-access, and -as noted above -failing to control for such variable likely induces a negative bias in our effect estimates. However, COVID-19 vaccinations may also affect these variables, meaning that they act as a mediator on the causal pathway from A → Y , and are therefore "bad controls." We argue that such effects were weak in February 2021: acquiring or switching job can take considerable time, and everyone in our sample was vaccinated for at most three months. Additionally, any bias induced by over-controlling is likely positive. For example, COVID-19 vaccinations likely boosted economic activity in aggregate and increased employment. This would then likely reduce financial stress and the associated mental health burdens. Controlling for variables along this causal pathway therefore likely results in positive bias.
We examine these hypotheses in our data by rerunning our analyses without controlling for worries about finances, occupation, or employment status. Our point estimates move further away from zero (see Table 37 in Appendix 14). This is consistent with either overcontrolling moving our point estimates closer to zero, or a negative bias induced by failing to control for these variables. Because we prefer our estimates to be biased towards zero we therefore control for these variables.

Measurement error
Measurement error may also bias our estimates. For example, our data is self-reported, and may be subject to reporting bias. Relatedly, we assume no unmeasured confounding assumption conditional on X, which includes both observed covariate values and categories for missing data. The unobserved values of these covariates may act as unmeasured confounders. We are unable to fully address these issues. However, as one robustness check we run our analysis while excluding respondents who provide implausible or unreasonable answers. For example, several respondents indicate that they live with a negative number of other individuals. Following [6], we consider a variety of implausible answers to our questions and omit these responses when running our analysis, removing 15,762 complete-cases.
All results are consistent with our primary analyses and are available in Appendix 14.

Mediation analysis
The validity of our mediation analysis requires strong assumptions. For example, our model precludes a prominent possible pathway through which COVID-19 vaccinations might affect depression: through household finances ([32], [5]). Our analysis instead assumes that vaccinations do not affect depression by changing finances. This assumption may not strictly hold: at an individual level, receipt of a COVID-19 vaccination might induce the previously unemployed to seek employment, which might then change a person's personal finances.
At an aggregate level, mass distribution of COVID-19 vaccinations may have boosted economic activity, which may affect any individual's employment status and therefore finances.
As noted in Section 4.4, this is one reason we limit our analysis to February, a timeframe when such effects are plausibly negligible.
Our analysis also requires that there exist no other post-treatment effects of vaccination that confound the mediator-outcome relationship. For example, we interpret the direct effect as reflecting the general feelings associated with a return to normalcy. Our analysis requires that this feeling does not cause social isolation or worries about health. However, causal arrows may exist from social isolation or worries about health to this variable. In this case both the direct and indirect pathways capture the effect of this variable. 22 These conditions and interpretations hold for any other hypothesized pathway between COVID-19 vaccines and depression.
Finally, the interventional effects are not robust to reverse causation with respect to Y .
We must therefore interpret these results cautiously. While we do not conduct a sensitivity analysis with respect any of these assumptions, we compute all of the robustness checks noted previously and find comparable results.

Discussion
We provide evidence that COVID-19 vaccinations reduced feelings of depression and anxi- and 15 percent reductions in the prevalence of each outcome. We also examine effects by both pre-specified and data-driven subgroups. We observe substantial heterogeneity with respect to age, with younger age groups having larger magnitude effect sizes. Our estimates on the risk-ratio scale suggest that some of this heterogeneity may be driven by differences in base rates of each outcome by age group. However, on both the risk-differences and riskratio scales, we find that the total effects on each outcome are strongest among respondents aged 18-24 relative to older respondents, with particularly large effects on depression and isolation among those living with an elderly individual. We also observe that those with multiple high-risk health conditions appear to have slightly larger point estimates with respect to worries about health. Our average effect estimates are moderately robust to violations to our causal assumptions and analytic choices, and we argue that many vio-22 [23] speculate that in addition to worries about health and social isolation, "different work opportunities" is a possible pathway through which COVID-19 vaccines may affect depression. While this pathway is related to household finances, worries about household finances could remain the same while one's employment status may plausibly differ. More broadly our analysis rules also out any pathways via employment changes. Limiting our analysis to February again mitigates potential bias if we think people are unlikely to change their employment within a short-time frame after being vaccinated.
lations would tend to bias our effect estimates towards zero on the risk-differences scale.
Finally, while our estimates pertain only to our sample of CTIS respondents, we argue that inferring similar magnitude effect sizes beyond our sample may not be unreasonable.
We conclude by positing a model where vaccinations affect depression through social isolation and worries about health. We then decompose the total effect into a direct effect and indirect effects via interventions on the distribution of social isolation and worries about health while holding vaccination status fixed. We find that the interventional indirect effect via social isolation is larger in absolute magnitude than the interventional indirect effect via worries about health. We find evidence of a direct effect that accounts for approximately half of the total effect on depression, which we interpret as reflecting a pathway via a perceived return to normalcy (see also [1]). These patterns are broadly consistent across our pre-specified subgroups.  8 Sample characteristics Table 4 displays the participant flow into the analytic sample. Tables 5-7 display the   demographic characteristics of both the analytic sample and the entire February sample.   Table 5 focuses on the treatment, outcomes, and mediators. Table 6 focuses on respondentlevel demographic characteristics.

Other estimands
As supplemental analyses we consider three additional estimands beyond those included in the main paper.

Incremental effects
We can generalize the above question to further ask: how would rates of depression, isolation, and worries about health have changed were we able to change everyone's probability of being vaccinated in February 2021? To answer this, we use the incremental intervention effects proposed by [17], formalized in equation (13): where π 1 (X) = P (A = 1 | X, V = 1) and δ represents a parameter that multiplies ev- Notice that equation (12) is an example of such a contrast between δ = 1 and δ = 0. This highlights an important distinction between equation (12) and equation (13): the former involves a causal contrast while the latter reflects a counterfactual rate. We refer to [17] and [7] for more details.

Pandemic effects
One reason we are interested in the effect of COVID-19 vaccinations on mental health is because we believe that it may "undo" some of the effects of the pandemic on mental health. We may therefore also wish to understand the effects of the pandemic on mental health in a world without COVID-19 vaccinations. Let C be an indicator of the COVID-19 pandemic. We can then consider the analogue to our primary causal estimand (ψ): We can also consider effects the risk-ratio scale. Under some assumptions, we can use our estimates of ψ to obtain lower bounds on both quantities (see Appendix 10).

Identification
This section details our causal models. The first subsection outlines all identifying assumptions. The second subsection provides expressions of our causal estimands in terms of the observed data distribution, and provides briefs proofs of these identification results. These results cover both the estimands in the main paper and the additional estimands outlined in Appendix 9.

Causal assumptions
As noted previously, the CTIS is a repeated cross-sectional sample of Facebook Users.
Additionally, the CTIS only indicates whether a respondent was vaccinated, but not when, and does not contain information on anyone's feelings of depression, isolation, or worries about health over time. While in truth the causal process may be complex (see Remark 3), we make several simplifying assumptions to identify the causal effects given this data. Assumption 5 (Consistency).
To ease notation, we drop the indices it moving forward. We again assume no unmeasured confounding and M-Y ignorability, detailed in equations (4)-(5), and reproduce these assumptions below for completeness.
No unmeasured confounding states that treatment assignment is independent of the potential outcomes and potential mediators conditional on covariates and vaccine-acceptance, while Y-M ignorability states that the potential outcomes are independent of the mediators conditional on treatment assignment, covariates, and vaccine-acceptance. This precludes the existence of other causal descendants of A that confound the mediator-outcome relationship.
Remark 1. For our outcomes analysis we only require Assumption 6 for identification. By contrast, our mediation analysis requires Assumption 7. We also implicitly assume no reverse causation between the mediators and outcome: that is, that depression does not cause feelings of isolation and worries about health.
Remark 2. These assumptions reduce a potentially more complex longitudinal process into a simpler structure that permits causal identification given our data. To illustrate, we consider our outcomes analysis, and let W t = [Y t , M 1t , M 2t ]. For simplicity we assume Consistency and No Anticipatory Effects. We also illustrate only four time points, though the insights hold across a longer time-frame: Because we do not observe each person's outcomes or vaccination status over time -only at the time they respond to the survey -we assume no carryover effects and no unmeasured confounding. Figure 10 illustrates these exclusions contrasted against Figure 9. First, we assume that vaccination status at any time t is not a function of any contemporaneous or prior outcomes. Second, we assume that that the outcomes at time t are only a function of that period's treatment status. We do allow that prior-period outcomes can affect current outcomes; however, they cannot affect treatment assignment at any point; otherwise, these variables would be a confounder. Remark 3. Allowing for carryover effects would imply that only causal estimands at time t = 0 would be identified (again assuming W −1 is not a confounder). This motivates our decision to analyze data only from February 2021: this time-period is "closer" to t = 0.
Heuristically, we may expect that the bias induced from violations of this assumption are smaller when closer to this "first" time-period and that this bias grows in magnitude over time. This may occur, for example, when treatment effects change monotonically with time since vaccination. As a related point, if our observed covariates are highly correlated with time since vaccination, controlling for these variables may reduce this bias. Unfortunately, even if we observed the differences in a pair of potential outcomes for each individual sampled at time t, we could not necessarily identify how effects change as a function of time since treatment ("timevacc"). To illustrate this point, consider the model: Consider the case where for all x, β 0x > 0, β 1x > 0, β 2x < 0, and |β 1x | > |β 2x |. That is, there is some positive initial effect of treatment which increases with the time index, but that decreases with time since treatment (i.e. the effects are not "durable"). In this case we would observe that the average effects increase with time; however, we would be wrong to infer that effects are increasing with the length of treatment. This is because in this scenario we cannot separately identify heterogeneity across time from carryover effects.
This illustrates the difficulty of modeling such effects generally. And in fact, as noted in Remark 3, we must actually preclude the existence of carryover effects identify our causal estimands given our data structure. Therefore, our particular data makes it difficult, if not impossible, to learn about effect durability, or carryover effects generally.
We next assume random sample selection to rule out collider-bias by conditioning on our sample. This assumption is detailed in equation (6), but reproduced below for completeness.
Assumption 8 (Random sample selection). where Finally, we assume positivity of the probability of treatment assignment, the joint mediator probabilities, and being a complete-case for all values of (a, m 1 , m 2 ) in the support of the data and > 0: 23 Assumption 9 (Positivity). These assumptions are sufficient to identify our causal parameters.

Identification
We provide all identification results conditional on X, noting that the target estimands are simply the empirical average of these expressions across the analytic sample.
where the first equality holds by the Law of Iterated Expectations, the second by random sample selection, the third by no unmeasured confounding, the fourth follows directly by the third, the fifth by no sample selection, and the sixth by consistency.
Proposition 2 (Joint counterfactual outcomes). The following identification result holds: Proof of Proposition 2. Let M = M 1 × M 2 . Then: where the first equality holds by LIE, no sample selection, and no unmeasured confounding Proposition 3 (Interventional effects). We provide the identifying expression for ψ M 1 , noting that the other interventional effects can be derived similarly.
Proof of Proposition 3. The proof follows directly from Propositions ?? and ??. Moreover, where the first equality follows by equation (27), the second by no unmeasured confounding, the third by equation (27), and the final line by consistency. The result follows by combining these results.
Remark 7. Unlike our other analyses, we additionally assume equation (27). This precludes the missingness process or sample selection process from being a function of treatment assignment given the covariates and vaccine-acceptance. This assumption may not hold: for example, [8] shows that the CTIS overrepresents vaccine-uptake compared to the U.S.
population, suggesting that perhaps there is selection into the sample based on this variable. On the other hand, [26] also shows that the CTIS overrepresents individuals with higher education, a variable also associated with higher vaccination rates. As long as higher vaccine-uptake among CTIS respondents can by fully explained by other observed covariates, then equation (27) could hold.
Alternatively, we could imagine these shifts as the result of an intervention that only targeted respondents to the CTIS, so that the intervention is on P (A = 1 | X, Z = 1) rather than P (A = 1 | X, V = 1). However, no feasible real-world intervention on the treatment assignment process would likely correspond to such an intervention, rendering such an estimand arguably less useful to consider.
Proposition 5 (Pandemic bounds). Assume the following: It follows that: Proof of Proposition 5. We use the identity that for all x: This holds by definition: our estimates all take place in a world in which there is a COVID-19 pandemic. The remainder of the proof follows directly from equation (28).
Remark 8. Equation 28 is perhaps better motivated by the following assumed inequalities: The first equality holds by definition.

Estimation
This section provides additional details on our estimation approach. As noted previously, we use influence-function based estimators throughout. 24 These estimators provide consistent estimates of the sample average treatment effects and possibly conservative variance estimates ( [15]). Specifically, all of our estimators take the form: whereφ(O) represents a scaled estimate of the uncentered influence function for any of the functionals considered in Section 10. To be precise, we use estimates of ϕ for the average potential outcome under treatment A = a, the incremental effects, and interventional effects via M 1 , respectively: where 24 Technically, these estimators are derived treating the covariates as if representative from some superpopulation with covariate distribution dP (X | Z = 1). π a (X) = P (A = a | X, Z = 1) with other terms defined analogously. The result for ϕ 1 (O) is well-known (see, e.g. [17]).
The derivation of ϕ 2 (O) is contained in [17] and for ϕ 3 (O) in [4]. We refer to [4] for the expressions for the one-step estimators for the remaining interventional effects.
To obtain estimates of the effects across all vaccine-accepting respondents (e.g. averaged over the entire stratum where SV = 1), we multiply the bias-correction terms (specifically, the terms involving inverse-probability weights) by We use logistic regression and XGBoost to estimate all nuisance functions. In the former case we estimate each function on the full-sample, and in the latter we use cross-fitting.
Additionally, for the XGBoost estimates, we estimate twenty specifications across a grid of hyper-parameters using the "xgboost", "tidymodels", and "tune" packages, and stack the resulting models using the "stacks" package in R.

Additional results
This section contains additional results not displayed in the main paper. We first present model diagnostics, and then present additional tables and figures with respect to our outcomes analysis, mediation analysis, and incremental effects analysis. Results from the incremental effects analysis are only contained in this Appendix.

Model diagnostics
We next examine the distribution of propensity score ("pi") and complete-case weights ("eta"), noting again that the complete-case weights are only used for our analyses in Appendix 14. Table 8 displays the quantiles of the distribution for each of these models.

Outcomes analysis
Figure 12 displays our heterogeneity analysis for pre-specified subgroups for our outcomes analysis using our XGBoost estimates, analogous to Figure 5 in Section 5.
The following tables display the subgroup estimates for the pre-specified subgroups with respect to our outcomes analysis. The first table presents results on the risk-differences scale and the second on the risk-ratio scale. The second two tables are identical but using XGBoost. In general we see that the results are quite similar except for the non-response category, which is much smaller for the XGBoost than the logistic regression models.         This next set of tables display the results of the subgroup estimates for the data-driven groups for our outcomes analysis. The first set of tables displays the results for the groups that we expected to be heterogeneous with respect to depression, which we discuss in the primary paper. In addition to this analysis, we also present the results for the subgroups that we examined for isolation and worries about health, though these results were largely inconclusive. All estimates are using XGBoost. We also estimates these numbers using logistic regression models. Many of these patterns in heterogeneity no longer appear as prominent, likely in part because we do not include any interactions between covariates in any of these models. These results are available on request.         Table 17 contains all point estimates on average across our analytic sample using our GLM estimates, while Table 18 presents our XGBoost estimates. The next sets of tables below display results contain results our subgroup effect estimates with respect to our pre-specified subgroups. The first two tables are on the risk-differences scale and set the indirect effect reference category at 0 and 1, respectively. The second two tables express the quantities as a proportion of the total effect. The final four tables repeat these analyses using our XGBoost estimates.

Other estimands
We consider the effects of the observed distribution of vaccinations and incremental effects on the propensity scores. Table 27 shows the results from the XGBoost models, displaying the estimated rate of each outcome and uniform confidence bands setting δ = 0 to δ = 5.
Again δ is a parameter that multiplies each individual's odds of being vaccinated, and reflects the expected prevalence of each outcome under the corresponding intervention We also consider the effect of the observed distribution of vaccinations, which is We also consider these estimands by age group and ethnicity. Figure  These results are consistent with our previous findings that the youngest respondents were most likely to benefit from COVID-19 vaccinations, while they were also least likely to be vaccinated: 18.0 percent of the youngest group were vaccinated, while 25.9 percent and 38.9 percent of the older Hispanic and non-Hispanic groups were vaccinated. Our incremental effects analysis is consistent with this finding, and further illustrates that the youngest respondents would obtain the largest relative returns from an intervention increasing everyone's odds of vaccination. The table below displays the incremental effects estimates using the GLM estimates.
The following tables display the incremental effect estimates using XGBoost models, stratified by subgroup. The GLM estimates are available on request.

Sensitivity analysis
This section contains two propositions: first, one containing the derivation of our sensitivity analysis used in Section 6. This result follows almost immediately from results previously shown in [20]. Our second proposition provides a simple extension to generate bounds on an estimand that does not assume random sample selection.
Our goal is to bound P n [ψ(X)] using only functions of the observed data.

Sensitivity analysis 1
In this subsection we propose a sensitivity analysis for the primary estimand P n [µ 1 (X) − µ 0 (X)]. Recall the assumption in equation (7) that there is some known τ satisfying: Proposition 6. For a fixed τ satisfying equation (7), we obtain the bounds: Proof of Proposition 6.
where the first equality holds by the law of iterated expectations, and the second by rearranging terms. For a fixed τ , equation (7) implies that: Plugging this into equation (31), we obtain the following inequalities: The result follows by taking the upper bound of µ 1 (x) minus the lower bound for µ 0 (x) to an upper bound on ψ(x), and vice versa for the lower bound.
Remark 11. To obtain bounds on the average effects, we can simply choose a τ valid for all values of (x, a) and average the expression over the relevant covariate distribution.
Remark 12. These bounds are expressed in terms of the observed data distribution. In order to estimate them we need to account for the uncertainty in our estimates of the nuisance functions. We use influence-function based estimators of these quantities (averaged over the empirical covariate distribution). We refer to [20] for more details.

Sensitivity analysis 2
Our target estimand is defined with respect to the observed sample and the above sensitivity analysis does not account for the fact that the prevalence of potential outcomes in our sample may not be representative. For instance, consider the following two estimands: Random sample selection implies that equations are equal (see Appendix 10). This equality implies that while the CTIS may have a non-representative covariate distribution, our statistical target of inference generalizes across covariate distributions of CTIS nonrespondents, and that we could reweight our estimates to generalize to populations with different covariate distributions.
We now instead consider the case where we these quantities are not equal due to nonrandom sample selection. For example, if the selection process were a function of observed or potential outcomes our sample estimand be large in magnitude while the effects among non-respondents were zero. To be clear, this would imply that our effect estimates cannot generalize to a comparable covariate distribution of CTIS non-respondents.
We therefore consider targeting equation (32) instead of equation (33). To obtain a bound on this estimand, we only assume the following: where Q a = 1(A = a)RS. We next derive bounds for equation (32) while relaxing the assumption of Random Sample Selection (equation (6)). µ 1 (x) −μ 0 (x) ≤ψ(x) +μ 1 (x, 1) 1 1 − τ + τμ 0 (x, 0) µ 1 (x) −μ 0 (x) ≥ψ(x) − τμ 1 (x, 1) −μ 0 (x, 0) τ 1 − τ Proof of Proposition 13.1. The result follows by following the same steps in the proof of Proposition 6, but iterating expectation on the variable Q a . The challenge is that we neither know nor can estimate P (Q a = 1 | X = x, V = 1), since we do not observe any characteristics about survey non-respondents. On the other hand, this value must always be less than 1. This then yields the following inequality: ≥ −τμ a (x, a)P (Q a = a | X = x, V = 1) ≥ −τμ a (x, a) The result follows by taking the upper bound forμ 1 (x) and subtracting the lower bound forμ 0 (x) for the upper bound ofψ(x), and vice versa for the lower bound.
Remark 13. We can again average this expression over the empirical covariate distribution to obtain bounds onψ, and can estimate these bounds using influence-function based estimators to account for the uncertainty in the estimates of the nuisance functions.

Robustness checks
This section presents several robustness checks. These primarily involve running our analyses on different subsets of the data, and / or including / excluding different covariates.
We describe each check in detail below. We rerun our mediation analyses across all of the specified checks. The results, when not provided, were similar to the primary analyses and are available on request.

Vaccine-accepting sample
Consider the target estimand: In contrast to our primary estimand, this estimand includes the incomplete-cases. As a first approach we directly derive point estimates for this quantity. These estimates require no additional assumptions beyond those previously invoked, but do require using inverseweighting with respect to the probability of being a complete-case (see Appendix 11). As a second approach we instead derive bounds on this quantity.

Point estimates
Our estimates are qualitatively similar to our primary analysis; however, the variance estimates increase substantially. Table 34 displays the outcome analysis using logistic regression while Table 35 displays the results using XGBoost. For our mediation analysis,

Bounds
We can obtain upper bounds on ψ va by assuming that effects on the R = 0 stratum are not greater than zero on average. This removes the positivity requirement required to estimate ψ va -equation (26) in Appendix 10.
Letting p r indicate the proportion of complete-cases over the total number of vaccineaccepting respondents, we can scale our estimates of ψ by p r (approximately 86 percent) to obtain a valid upper bound on ψ va . 25 We can generalize this approach to cover any other portion of the data that we do not include in our primary sample (e.g. those with missing ZIP code information, who live in Alaska, or who are vaccine-hesitant). Notably, we can never fully explain away our effect estimates under this assumption: we would scale them by at most 64%, the complete-case percent of the initial sample.
We may also wish to make inferences while allowing our estimates to be biased, either due to violations of no unmeasured confounding or random sample-selection. Fortunately, the sensitivity parameters we estimated previously in Section 6 remain valid for ψ va . This follows since, by assumption, the effect on the unobserved portion of the data is not greater than zero. This logic generalizes to any portion of the data excluded from our primary analysis. Table 36 presents our primary estimates when including vaccine-hesitant respondents. Consistent with our expectations detailed in Section 6, the effect estimates for the outcomes anlaysis move closer to zero.

Including hesitancy non-response
Our primary analysis excludes all individuals who report vaccine hesitancy, or who do not respond to the question. Implicitly, this analysis assumes that all non-responses were vaccine-hesitant. We run our analysis when including the 75,568 respondents who did not respond to the vaccine hesitancy question (and who either did not respond to the vaccination question or indicated that they had not previously received the COVID-19 vaccine). However, this only adds 1,053 respondents to our analytic sample. Table 38 compares the primary outcome analysis to this analysis, and shows that the estimates are consistent.

Omitting suspicious responses
We also run our analysis excluding individuals who give suspicious responses. To identify these observations, we roughly follow a script used in [6], and exclude people who report any values that fall outside the range of the responses: • Household size between 0 and 30