Upgrading Efficiency and Behavior: Electricity Savings from Residential Weatherization Programs

Residential weatherization programs have become a major component of U.S. energy policy. Through these programs, households receive heavily subsidized energy efficiency upgrades as well as informational and behavioral treatments designed to encourage conservation. While previous work demonstrates that weatherization programs provide sizable energy savings, all have measured the composite effect of efficiency upgrades and behavioral treatments. In this paper, we present the first estimates which disentangle the energy savings provided by each of the individual interventions. Our results reveal that the actual energy savings achieved by the efficiency upgrades are substantially smaller than ex-ante, engineering predictions. Moreover, we present evidence that the energy savings provided by the simple behavioral interventions can exceed the savings resulting from the much more costly efficiency upgrades.


INTRODUCTION
Making homes more energy efficient is widely viewed as a low cost option for reducing the negative externalities arising from energy use. 1 As a result, residential energy efficiency programs have become a major component of U.S. energy policy.Chief among these policies is the Department of Energy's Weatherization Assistance Program (WAP). 2 The WAP provides heavily subsidized efficiency upgrades (e.g., improved insulation, duct-sealing, etc.) to low-income households.To date, over 7 million homes have been upgraded through the program.It is important to note, however, that participating households receive more than just the physical upgrades.Recipient households generally receive instructions on how to interpret their energy bills as well as coaching on how they can alter their behavior to conserve energy. 3While an existing literature reveals that the WAP provides sizable energy savings (Hirst and Goeltz (1984, 1985), Sebold and Fox (1985), Hirst (1986, 1987), Schweitzer (2005)), the previous studies estimate the program's composite effect-that is, the combined impacts of the efficiency improvements and the behavioral interventions. 4To provide a more nuanced understanding of the impacts of the weatherization program, this paper presents the first estimates which disentangle the energy savings provided by the engineering and behavioral treatments.
Isolating the effect of the physical upgrades allows us to make two key contributions.First, we are able to directly compare the actual electricity savings achieved by the efficiency upgrades to ex-ante, engineering predictions.While several older studies attempt a similar exercise (Sebold  and Fox (1985), Hirst (1986), Dubin, Miedema and Chandran (1986)), none have isolated the savings provided solely by the efficiency upgrades-which is the value engineering models attempt to predict.In a recent study, Fowlie, Greenstone and Wolfram (2015) provide evidence that engineering predictions overstate the natural gas savings provided by residential weatherizations.In line with these results, we find that the electricity savings are similarly overstated.Second, we are able to examine whether combining behavioral treatments with the efficiency upgrades can achieve additional savings.It is important to note that the impact of the behavioral treatments is theoretically ambiguous.If providing information on conservation strategies induces behavioral changes, then additional energy savings may be achieved.Alternatively, informing households how to interpret their energy bills may make the reduction in the costs of energy services (e.g., space cooling) more salient, potentially exaggerating the rebound effect caused by the efficiency upgrades.Given that the WAP specifically targets low-income households-which are potentially the most responsive to energy price changes-the resulting rebound effect may be quite pronounced. 5o study the effects of a weatherization program, we focus on 275 low-income households in San Diego, California that received free energy efficiency retrofits.In addition to the retrofits, a random subset of the households were offered a behavioral treatment that consisted of three components: 1) households were educated on how to interpret their energy bills, 2) households were provided with information on specific energy conservation practices, and 3) households were asked to make non-binding "commitments" to specific energy-saving behavioral changes.
To determine how the retrofits and behavioral treatments affect electricity consumption, we identify the change in monthly electricity consumption that occurs following each treatment.Our results suggest that the retrofits and behavioral treatments have heterogeneous effects across households.Among the homes with air conditioning units, the retrofits reduce electricity use by an average of 7% and the behavioral interventions reduce electricity use by an additional 24%.We find that these reductions occur almost exclusively during the summer months-suggesting that the retrofitted households use less energy to cool their homes.In contrast, among households without air conditioning units, neither treatment is found to significantly affect consumption.
6.In 2009, for example, the incentive payments to the state's utilities was roughly $1.4 billion.For an overview of California's energy efficiency incentive mechanisms, see the CPUC's white paper, Proposed Energy Efficiency Risk-Reward Incentive Mechanism and EM&V Activities.
7. It is worth noting, however, that Lang and Siler (2013) explore the impact of efficiency upgrades on non-residential buildings and find that engineering models perform well at predicting the energy savings.
8. In the SDHEU program, an average of $1,726 was spent on each retrofit.In contrast, the marginal cost of conducting an in-home informational/behavioral treatment was $350.9.For eligibility requirements, see the ESAP website: http://www.sdge.com/energy-savings-assistance-program.
To examine how accurately an engineering model predicts the savings provided by the upgrades, we compare our ex-post estimates to ex-ante predictions from the Database for Energy Efficient Resources (DEER).The DEER model is used by the California Public Utility Commission (CPUC) and California Energy Commission to predict the energy savings provided by building upgrades.Moreover, the CPUC uses the DEER predictions when determining how large of an incentive payment each utility should receive for their energy efficiency programs. 6Our results reveal that the DEER predictions overstate the energy savings.For example, among the homes with air conditioning units, only 79% of the predicted savings are realized.Echoing the conclusions from previous studies (Blumstein (2010), Kaufman and Palmer (2012), Fowlie, Greenstone and  Wolfram (2015)), our findings suggest that if ex-ante estimates continue to play a role in subsidizing energy efficiency programs, more accurate predictions are needed. 7n addition to exploring the effect of the retrofits, our results provide evidence that the behavioral treatments can cause additional energy savings.While several related studies reveal that behavioral treatments used in isolation can provide modest reductions in consumption (Allcott  (2011), Allcott and Rogers (2012), Harding and Hsiaw (2014)), none have explored how the treatments interact with energy efficiency improvements.This interaction is particularly important because federal and state efficiency programs increasingly bundle the interventions together and, as outlined earlier, the impacts of information in this bundled context may well undermine the goals of the efficiency upgrades.Despite this theoretical possibility, our results suggest that not only can the simple behavioral interventions reduce energy use, the reductions can exceed the savings resulting from the much more costly efficiency upgrades. 8he remainder of this paper proceeds as follows.Section 2 describes the retrofit program.Section 3 discusses our empirical strategy.The estimates of the average energy savings are presented in Section 4. Section 5 presents a comparison of our ex-post estimates of the energy savings and ex-ante, engineering predictions.Section 6 concludes.

Retrofit Program
The Energy Savings Assistance Program (ESAP) provides free energy efficiency upgrades to low income households in San Diego Gas and Electric's (SDG&E) service territory.Program eligibility is determined by household size and income.For example, a four person household must have a combined annual income below $47,700. 9The program is advertised through monthly bill inserts as well as on SDG&E's and the CPUC's websites.Eligible households must self-select into the program.After enrolling online or by phone, the households receive an energy audit which identifies the set of upgrades that will be performed at each household.The specific improvements vary by household, but in general, the upgrades include installing energy efficient lighting and windows, as well as improving existing insulation, ventilation, and duct-sealing.
Our analysis examines the 275 low income households that received free upgrades between November 2011 and September 2012. 10In addition to the home upgrades, 144 of the retrofitted households were randomly assigned to a behavioral treatment group.After receiving the energy efficiency upgrades, the households assigned to the behavioral treatment group were contacted by telephone and offered a free, in-home visit to learn about energy saving actions.Of the 144 treatment group households, 38 selected to receive the treatment. 11These households were visited by a trained educator who explained how to interpret their SDG&E bill (e.g., clarifying the tiered pricing system) and walked through the home demonstrating ways to save energy. 12At the end of the informational session, the educator asked each participant to make a non-binding " commitment" to three energysaving actions of their choice.Each treated household chose to make the non-binding commitments.The most common energy-saving actions pledged by the households included unplugging unused appliances, turning off lights, and drying clothes on lines. 13Only one household pledged to reduce their use of air conditioning (AC).Despite this fact, our subsequent results suggest that energy savings as a result of the in-home treatments were largely confined to homes with AC unitssuggesting that the "commitments" had limited effectiveness.

Data
To estimate the impact of the retrofits and the behavioral treatments on electricity consumption, we use billing data from SDG&E.The billing data provides the monthly electricity consumption and the monthly expenditure on electricity for each household.It is important to note that the billing cycle start and end dates vary across households.For each household, we observe the monthly bills with start-dates later than July 31, 2011 and end-dates earlier than October 1, 2012.Therefore, we typically observe 12 bills per household. 14Given that the retrofits occur between November 2011 and September 2012, the number of pre and post-retrofit observations varies by household.
Table 1 reveals that 237 households receive only a retrofit and 38 households receive both a retrofit and the behavioral treatment.Table 1 also summarizes the demographic and building characteristics of the households in our sample.It is important to note that only 27% of the households have AC units. 15Given that many of the upgrades are designed to insulate the homes, the retrofits likely reduce the energy required to cool a home.As a result, the retrofits may have very different effects on the energy consumed in households with AC units versus those without.Table 2 demonstrates that there are no significant differences between the treatment and control group households for any of the household characteristics.
In addition to the actual consumption, we observe ex-ante engineering predictions of the electricity savings the retrofits will provide.The engineering estimates are from the Database for Energy Efficient Resources (DEER).DEER is sponsored by the California Energy Commission and the CPUC and is intended to provide well-documented estimates of the energy savings provided by a wide range of residential energy efficiency measures.The database uses a combination of engineering calculations, building simulations, and measurement studies to predict the energy savings that each individual energy efficiency measure will provide.For a given upgrade, the DEER estimates of the energy savings vary across premises based on the home size, vintage, and climate zone.The DEER estimates do not reflect the condition of the specific home, the existing appliances (e.g., type of AC unit), or any demographic information.The DEER estimates we observe aggregate the predicted energy savings across each upgrade performed at a given household.On average, the DEER model predicts the retrofits will reduce 2.1 kWh per day.If these predicted savings are realized, this would represent a 14% decrease in daily electricity consumption.

ESTIMATION STRATEGY
To provide ex-post estimates of the average effect of the upgrades on the retrofitted households, we estimate the change in energy consumption that occurs following the retrofits.Given that all of the households being studied have selected to receive the retrofits, we do not need to control for selection into the program.However, the timing of the retrofits may not be entirely random.The date a household is retrofitted is determined jointly by the availability of the household and the contractors performing the retrofits.Given that the timing of the retrofits is not randomized, households that receive the retrofits earlier in the sample may differ systematically from households that are retrofitted later in the sample.Table 2 compares households that receive their retrofits early in the sample-with six or fewer pre-retrofit bills-and households that are retrofitted later.While the two groups are quite similar, there are significant differences in the average number of occupants and the shares with AC units.To ensure that we control for any time invariant household characteristics that may be correlated with both household energy consumption and the number of months a household is treated, we use household fixed effects.
In addition to controlling for the heterogeneity in the average levels of consumption, we must also control for the fact that a household's energy consumption varies over time.The temporal variation in consumption is driven in part by deterministic patterns.For example, there are more 16.The weather data is provided by the National Climatic Data Center.17.On days when the average temperature is less than 65ЊF, HD equals the difference between 65ЊF and the average temperature while CD equals zero.On days when the average temperature exceeds 65ЊF, CD equals the difference between the average temperature and 65ЊF while HD equals zero.daylight hours, and therefore lower lighting demand, during the summer.Variation in consumption is also driven by the temperature.In San Diego, where winters are temperate, electricity consumption tends to peak during the summer when air conditioning use is the greatest.Left uncontrolled, these seasonal trends present a potential problem for estimating the effect of the retrofits.For example, roughly half of the retrofits took place during the spring (March through May).On average, electricity consumption will be lower during the months prior to these spring retrofits and higher in the summer months following the upgrades.Therefore, naively estimating the effect of the retrofit, without controlling for seasonal demand patterns, may understate the savings.
We use two strategies to control for seasonal consumption patterns.First, we include month-of-sample fixed effects.While these fixed effects will control for much of the seasonal variation in demand, they do not control for the fact that households experience different temperatures during their billing cycles.These differences arise for two reasons.First, the start and end dates of each billing cycle vary across households.Second, the households are in different regions of San Diego.Households closer to the coast experience milder temperatures while households farther inland face more extreme temperatures.To directly control for the impact of temperature on demand, we collect daily temperature readings from two weather stations in San Diego. 16The stations are located at San Diego International Airport (coastal zone) and Montgomery Field (inland zone).We assign households to a zone following SDG&E's designation of coastal and inland regions.Following the standard approach taken by electric utilities for forecasting demand, we calculate the daily heating degrees (HD) and the daily cooling degrees (CD). 17We do so separately for each zone.To control for the impact of temperature on monthly electricity consumption, we calculate the average of the daily HD's and CD's over each day in a household's billing cycle.
Of course, some remaining temporal variation in monthly energy consumption will not be explained by the monthly fixed effects and the average HD and CD measures.If any time-varying, household specific characteristics which affect energy consumption are also correlated with the timing of the retrofits, then the estimates of the treatment effects could be biased.While we view such concerns as minimal, absent data on time-varying household characteristics or a randomized roll-out of treatments, we cannot rule out this possibility.
Nonetheless, we can compare the trends in energy use between households retrofitted early in the sample and households retrofitted later in the sample.If time varying determinants of demand are correlated with the timing of the retrofits, then we might expect households retrofitted earlier to display different patterns in their pre-retrofit energy use compared to households retrofitted later.To explore whether this is the case, we calculate the average daily energy consumption across each pre-retrofit month for three groups: households with 1-3 pre-retrofit bills, 4-6 pre-retrofit bills, or more than 6 months of pre-retrofit observations.The top panel of Figure 1 displays the trends in pre-retrofit electricity consumption.All three groups display similar patterns.In addition, there are no significant pairwise differences between the average consumption for any of the months.In addition to displaying the levels, we also demean the pre-retrofit observations by removing the household fixed effects and the month of sample fixed effects.The bottom panel of Figure 1 reveals that there are no trends in the demeaned pre-retrofit consumption.In fact, none of the demeaned We also present estimates of the change in electricity consumption caused by the behavioral treatments.Recall, during the treatments, households were instructed on energy conservation strategies, were informed how to interpret their energy bills, and asked to make non-binding commitments to a set of energy saving behavioral changes.Our estimates uncover the combined effect of each of these interventions.It is also important to note that the behavioral treatments followed the retrofits.Therefore, our estimates capture the incremental impact of providing the behavioral treatment to newly retrofitted homes-not the impact the treatment will have when provided on its own.Despite the randomized assignment to the treatment group, three quarters of the eligible households did not receive the intervention.While household fixed effects will control for time-invariant determinants of demand that may be correlated with the decision to opt-out, we cannot directly control for household specific, time-varying determinants of demand that may be correlated with the decision to opt-out.To produce consistent estimates of the average effect of the behavioral treatments on the treated households, we use the random assignment to the treatment group as an instrument for receiving the behavioral intervention.Due to the fact that informational programs are often offered on a voluntary basis, our estimates of the treatment effect on the treated households may be particularly relevant for policy makers.
18. Estimates were also made using a third potential "treatment"-the date the households enrolled in the free retrofit program.The enrollment treatment has no significant impact on any subset of the homes' consumption.
19.The month-of-sample fixed effect represents the month at the mid-point of each billing cycle.20.To estimate the variance of the estimators defined by Eq. ( 2) and Eq. ( 3), we use the approximate unbiased variance estimator presented by Van Garderen and Shah (2002), .

Econometric Specification
To determine the average effects of the retrofits and the behavioral interventions on the treated households, we estimate the model 18 : where represents a household fixed effect, is a month-of-sample fixed effect, 19  The Retrofit and Treat variables are equal to zero prior to receiving a retrofit or behavioral intervention and one in each month after a household receives a retrofit or behavioral intervention.During the month a household receives a retrofit or a behavioral treatment, the Retrofit or Treat variables are continuous and bounded between 0 and 1.To account for serial correlation, the errors are clustered at the household level.Given the semi-log specification, and must be transformed β h to determine the percentage change in electricity consumption caused by the interventions.Following Kennedy (1981), consistent estimates of the average percentage change in electricity consumption caused by discrete changes (from 0 to 1) in Retrofit or Treat can be calculated by solving for the following values: where and represent estimates of the variance of the coefficient estimates and . 20ˆV (β) V(h) β h In addition to estimating Eq. ( 1) using OLS, we also present 2SLS estimates which take advantage of the random assignment to the behavioral treatment group in order to control for potential selection bias caused by households opting out of the behavioral treatment.Specifically, we instrument for the non-random Treat variable using the following first-stage: where includes average daily HD and CD as well as household and month-of-sample fixed X i,m effects.In addition, the first-stage includes an indicator (Intention) which reflects whether the household was intended to have received the in-home intervention.For households assigned to the control group, Intention always equals 0. For households assigned to the treatment group, Intention ideally would equal 0 each month prior to the date the household was intended to receive the intervention (the intention-to-treat date) and 1 each month after the intention-to-treat date.Unfortunately, we only observe an intention-to-treat date for households that receive the treatment.Specifically, their intention-to-treat date is the observed treatment date.For these households, Intention equals 1 each month after receiving the behavioral intervention and 0 each month prior to the treatment.During the month of treatment, Intention is equal to the share of the billing cycle the household is treated.
For the treatment group households that opt-out of the behavioral treatment, the intentionto-treat date is unobserved since a household visit is never scheduled.To estimate the model, we must designate an intention-to-treat date for these households.Recall, the behavioral interventions always occur later than the retrofits.On average, households that opt in receive the treatment 131 days after being retrofitted.The standard deviation of the lag between the treatment and retrofit dates is 51 days.In order to assign an intention-to-treat date, we use two approaches.For the first approach, the intention-to-treat date for households that opt-out of treatment is fixed at 131 days after the household's retrofit date.Therefore, Intention equals 1 for all months over 131 days after the retrofit date.As a robustness check, we also present estimates which use a second approach to define the intention-to-treat date.Rather than assuming a constant lag between retrofit and treatment dates for households that opt-out of treatment, we draw a random value from a normal distribution with a mean of 131 and a standard deviation of 51.Using this number as the lag between the household's retrofit and intention-to-treat dates, we create an Intention indicator for each household that opts-out of the behavioral treatment.

Treatment-on-Treated Effects
Table 3 presents OLS and 2SLS estimates of Eq. ( 1).The estimates in the first two columns are made using all 275 households in the sample.To produce the 2SLS estimates, we initially use the first approach described in the preceding section.That is, the intention-to-treat date is assumed to be 131 days after the retrofit date for households that opt-out of treatment.The OLS and the 2SLS estimates suggest that, on average, the retrofits and behavioral interventions do not result in significant changes in electricity consumption.These findings are not entirely surprising.The majority of the energy efficiency improvements are intended to make the inside temperature of the homes less responsive to the outside temperature.However, Table 1 highlights that less than 30% of the households participating in the study have an electric AC unit-and only 3% of the homes have electric heat.Since the largest source of residential electricity use is heating and cooling, there is little scope for energy savings among those that do not use electricity for temperature control. 21o examine how homes with and without AC are differentially affected by the retrofits and behavioral interventions, we estimate Eq. (1) separately using households without AC units and households with AC units.Columns 3 and 4 of Table 3 reveal that, in homes without AC units, the retrofits and behavioral interventions have no significant impacts on electricity usage.In contrast, columns 5 and 6 of Table 3 provide evidence that, within the subset of homes with AC units, the Note: Models are estimated using household level fixed effects.Clustered standard errors are presented in the parentheses.Explained variation of within household energy usage given by R 2 .*, significant at the 10% level; **, significant at the 5% level; ***, significant at the 1% level.
22. Consistent with these homes being more responsive to high temperatures, the coefficient on the average daily CD is also large and significant.
23.To give a sense of how plausible a 24% reduction in energy use during the summer is, we have examined the energy consumption patterns of homes with AC units.During the two warmest months, August and September, the average daily consumption is 22 kWh.During the mild month of April, the average electricity use is 15 kWh.Therefore, roughly 7 kWh of electricity is used for cooling.Given that the annual average daily consumption is 18 kWh, to achieve a 24% reduction, around 4 kWh must be reduced-or just over half of the electricity used for cooling.retrofits cause significant reductions in electricity usage. 22Moreover, in homes with AC units, the behavioral treatment is estimated to cause large, additional reductions in consumption.It is important to note, however, that the estimate of the negative behavioral treatment effect is statistically significant only in the OLS model.
The bottom rows of Table 3 present the point estimates of Eq. ( 2) and Eq.(3)-the average percentage changes in consumption caused by the retrofits and the behavioral treatments.In homes with AC units, the retrofits cause an average reduction in electricity consumption of 7%.Moreover, the point estimate of the behavioral treatment effect suggests that the intervention reduces consumption by an average of 24%.It is important to note that the post-behavioral treatment observations fall exclusively during the summer months.If the behavioral intervention has a smaller impact on consumption during the cooler months, when air conditioning is rarely used, then the 24% reduction overstates the average annual impact of the behavioral intervention. 23In contrast, the estimates of Eq. ( 2), the average retrofit effect, have a mean of -0.07%with a standard deviation of only 0.0003%.
The similarity between the OLS and 2SLS estimates in Table 3 suggests that households opting-out of the behavioral treatment does not introduce selection bias in the OLS estimates.Recall, however, the 2SLS estimates in Table 3 assumed that the unobserved intention-to-treat dates are 131 days after the observed retrofits.To examine how sensitive the estimates are to this assumption, we produce 2SLS estimate of Eq. (1) using the second approach described in the preceding Section.That is, we draw random intention-to-treat dates for the households that opt-out of the behavioral intervention.Repeating this process 1,000 times, we re-estimate the first stage Eq. ( 4) and produce 1,000 corresponding estimates of Eq. ( 2) and Eq.(3)-the average percentage changes in consumption caused by the two treatments.
Figure 2 presents the distribution of the 1,000 point estimates of Eq. ( 3) among homes with and without AC units.Focusing first on homes without AC units, the estimates of the average behavioral treatment effect are tightly centered around 0.01%.This is consistent with the results presented in Table 3 that suggest the behavioral treatment does not dramatically affect energy consumption in households without AC units.In contrast, among households with AC units, the estimates of the average behavioral treatment effect are centered around -25%.However, as Figure 2 demonstrates, there is a decent amount of variation among these point estimates-the standard deviation is 0.054%. 24Despite this variation, each of the 1,000 point estimates of Eq. ( 3) is less than -0.08 for the homes with AC units.Therefore, while the magnitude of the estimated behavioral treatment effect is somewhat sensitive to the simulated intention-to-treat dates, the sign of the estimated impact is consistently negative.
25. Estimating Eq. ( 5) using 2SLS-with the intention-to-treat date fixed at 131 days after the retrofit for households that opt-out of treatment-we find a significant retrofit effect of -0.06% (0.022) and an insignificant, but now very imprecisely estimated, treatment effect of -0.017% (0.23%).
26.Among the homes without AC units, the estimate of the average retrofit effect is 0.000% (0.027%) and the average behavioral treatment effect is 0.056% (0.102%)-which are both statistically insignificant.
The preceding results provide evidence that the retrofits and behavioral interventions can cause sizable reductions in energy use among households with AC units.Recall that the behavioral treatment consisted of an informational component (i.e.conservation strategies and interpreting bills) as well as a behavioral component (i.e.household members made soft commitments to energy savings strategies).While every household treated with the behavioral intervention elected to make non-binding commitments to three energy saving strategies, recall that only one household committed to increasing the temperature on their thermostat.If the non-binding commitments are responsible for the savings from the behavioral treatment, then we would expect the behavioral intervention to provide energy savings in households with and without AC units.However, the fact that the behavioral treatment only reduces electricity consumption in households with AC units suggests that the information component of the behavioral intervention is driving the results, not the soft commitments.

Robustness Checks
To examine the robustness of the preceding results, we estimate a number of alternative specifications.First, we examine whether our estimates are sensitive to the way we allow demand to respond to temperature.Using OLS, we estimate the following model: The specification is identical to Eq. ( 1) except the households are now allowed to have heterogeneous responses to HD and CD.Using the estimates of and , we re-estimate Eq. ( 2) and Eq.β h (3).Focusing on the homes with AC units, we estimate an average retrofit effect of -0.053% (0.023%) and an average treatment effect of -0.173% (0.059%). 25While these estimates are both slightly closer to zero compared to the corresponding estimates presented in the bottom row of Table 3, they still suggest that both the retrofit and the behavioral treatment cause significant reductions in electricity consumption among the homes with AC units. 26o examine how sensitive the results are to outliers, we drop a single household at a time and re-estimate Eq. (1) using 2SLS.If the results are driven by anomalous consumption patterns within a small number of households, then the estimates may be quite sensitive to the sub-sample chosen.Focusing on homes with AC units, we find estimates of the retrofit effect (Eq. ( 2)) that range between -0.083% and -0.059% and estimates of the average behavioral treatment effect (Eq. ( 3)) that vary between -0.294% and -0.169%.Given the small sample size, it is certainly expected that the results will vary with the set of homes included in the sample.However, the fact that none of the estimates deviate substantially from the initial point estimates suggests that the results are not being driven by a small number of outliers.
Finally, if the estimated changes in electricity consumption are being driven by the retrofits and the behavioral treatments, there should be no discernible retrofit and treatment impacts prior to the interventions, only after the interventions.To examine whether this is the case, we estimate 27.The downward trend in the average retrofit effect following the home upgrades does not imply that the monthly energy savings grow with time.The results presented in the following section will demonstrate that the retrofits save larger quantities of energy during the summer months.The finding that the retrofit effect is the largest five or more months after the intervention is driven by the fact that most of the households were retrofitted May, 2012-several months before the hottest summer months (July through September).
Eq. ( 1) using OLS and allow the estimates of and to flexibly vary across the months leading β h up to and following each intervention.Specifically, we interact the Retrofit indicator with a set of dummy variables representing billing cycles occurring 1, 2, 3, 4, and 5 or more months before a household receives a retrofit as well as dummies for billing cycles 0, 1, 2, 3, 4, and 5 or more months after a household receives a retrofit.Similarly, the Treat indicator is interacted with dummies for 1 through 5 or more months before the behavioral intervention and 0 through 3 or more months following a behavioral treatment.
Figure 3 presents the estimates of the average retrofit and treatment effects-Eq.( 2) and Eq. ( 3)-during the pre and post-treatment months.Among the homes without AC units, there are no significant impacts on electricity consumption in any of the months leading up to or after the retrofits and behavioral interventions.Among the homes with AC units, the estimates do not reveal any significant changes in electricity consumption during the months leading up to the retrofits or behavioral interventions.However, during many of the post retrofit and treatment months, significant decreases in electricity consumption do occur.These results provide strong evidence that the estimated changes in electricity consumption are in fact driven by the retrofits and the behavioral interventions. 27opyright ᭧ 2016 by the IAEE.All rights reserved.
28.The simple average of the average CD for each month, across the inland and coastal zones, is plotted.29.Jacobsen and Kotchen (2013) find a very similar seasonal pattern in the electricity savings caused by imposing more stringent building codes in Florida.

EX-POST ESTIMATES AND ENGINEERING PREDICTIONS
The results in Table 3 suggest that, among households with AC units, consumption falls by 7% after the retrofits occur and by an additional 23% after the behavioral treatments.These estimates are initially surprising for two reasons.First, the energy savings from the retrofits appear small-they are only half of the annual 14% savings predicted by the DEER model.Second, the behavioral treatment effect is over three times larger than the retrofit effect.
However, unlike the predictions from DEER, the ex-post estimates of the retrofit and behavioral treatment effects in Table 3 do not necessarily represent the average impacts the treatments will have on energy use over a full year.This is due to the fact that the post-retrofit months, and the post-treatment months, are not uniformly distributed over a year.If the retrofits or the behavioral treatments provide different energy savings during different months, then the estimates in Table 3 will not capture the average annual impacts on the treated households.In the following section, we provide evidence that the retrofits affect energy use differentially across months based on differences in the temperature.Using this fact, we provide estimates of the average annual savings from the retrofits which can be directly compared to the engineering predictions.

Average Retrofit Effect by Month
To examine how the retrofit effect varies, we estimate the following model using 2SLS: where is the original vector of controls and is now allowed to vary across months.In the X β i,m m first stage, fitted values of Treat are predicted by estimating Eq. ( 2)-again assuming the intentionto-treat date is 131 days after the retrofit data for households that opt-out of treatment.To determine the average percentage change in consumption during a specific calendar month, we solve for the corresponding values of Eq. ( 2).The first two columns of Table 4 provide the share of households retrofitted and treated by month.Given that there are pre and post-retrofit observations during each month other than October, the retrofit effect can be estimated for all months other than October.In contrast, the behavioral treatment effect only represents the impact of the behavioral intervention during the summer months.Estimates of Eq. ( 6) are made for two different subsets of households-those with AC units and those without AC units.The point estimates of the monthly retrofit effects are presented in Table 4. Focusing first on the homes with AC units, a clear seasonal pattern emerges.To highlight this pattern, Figure 4 plots the point estimates of the monthly retrofit effects as well as the average daily CD for each month. 28In retrofitted homes with AC units, large and significant reductions in electricity consumption occur during August and September. 29These are also the months with the highest average daily CD's, consistent with these households requiring less energy to cool their homes.In retrofitted homes without AC units, none of the monthly point estimates are significantly different from zero.

Annual Average Energy Savings
The estimates of the monthly retrofit effects demonstrate that the energy efficiency improvements affect energy use differentially across months.Therefore, to estimate the average annual impact of a retrofit on energy use, we must account for the fact that the post-retrofit observations are not uniformly distributed over time.We use two different strategies to estimate the average annual effects of the retrofits.
The first strategy is motivated by the results from Figure 4 which reveal that the average effects of the retrofits are highly correlated with the average CD.Therefore, we can estimate the average retrofit effect as a function of the average CD.2SLS estimates of the following model are made for households with AC units and for households without AC units: The dependent variable is now the average daily electricity consumption (kWh) during the billing cycle.Again, the first stage model (Eq.( 4)), with the intention-to-treat date set at 131 days postretrofit, is used to estimate fitted values for Treat.To estimate the average annual impact of a retrofit on the daily level of consumption, we solve for the following expression: Note: Models are estimated using household level fixed effects.Clustered standard errors are presented in the parentheses.Explained variation of within household energy usage given by R 2 .*, significant at the 10% level; **, significant at the 5% level; ***, significant at the 1% level.
30.The average daily CD during October is 0.99.This is similar to the annual average daily CD of 1.29.As a result, the unobserved retrofit effect during October is likely to be close to the average retrofit effect across the other eleven months.
the retrofits reduce daily consumption by an average of 2.05 kWh.In homes without AC units, the retrofits do not have a significant impact on energy use.
The second strategy we use to estimate the annual average electricity savings caused by the retrofits directly accounts for the fact that the post-retrofit observations are not evenly distributed across months.We re-estimate the model specified in Eq. ( 7) while restricting .To account β = 0 2 for the uneven distribution of post-retrofit observations, we weight the observations within each calendar month by the inverse of the share of post-retrofit observations occurring in the given month.This strategy places more weight on observations from months with fewer post-retrofit bills .However, given that there are no post-retrofit observations from October, this strategy will produce an estimate of the average impact of the retrofits over eleven months of the year. 30Using the second strategy, we find very similar results.In homes with AC units, the retrofits reduce consumption by an average of 1.95 kWh/day over the eleven months.This point estimate is again significant at the 1% level.Among households without AC units, the retrofits do not significantly affect consumption.
31.During the period from November, 2011 through October, 2012, the average daily temperatures in San Diego, as well as the average high temperatures, exceeded the long-run averages over the preceding 10 years.This is important because the expected savings predicted by the DEER model depend on assumptions about the future weather.During warmer than average years-when the demand for cooling is higher-the predicted savings will likely understate the actual energy savings from the retrofits.Therefore, the estimate of will likely serve as an upper bound on the share of the ( -ββ ⋅ CD) predicted savings actually achieved.32.In the subset of households that do not have AC units, the DEER model predicts that reductions in electricity consumption will be provided by the retrofits.However, no significant retrofit effects are found in this subset of homes.As a result, essentially none of the expected energy savings are realized.
33.For more information on the California Energy Upgrade program, see https://energyupgradeca.org/overview.34.Moreover, if the inflation in the predicted savings differs across types of efficiency upgrades, then the programs will bias the decision of which efficiency upgrades to receive.
35.For an overview of the literature examining the existence of an energy efficiency investment gap, see Allcott and  Greenstone (2012).

Comparing Energy Savings to Ex-Ante Predictions
To directly compare the average annual energy savings caused by the retrofits to the exante, engineering predictions, we estimate the following model using 2SLS: where fitted values for Treat are estimated from the first stage Eq. ( 4), Predicted = i,m , and is the DEER prediction of the daily kWh savings pro- vided by the retrofit in household .Solving for yields the average annual share of i ( - the expected energy savings that is actually achieved by the retrofits. 31stimates of the annual share of the expected savings achieved are presented in the bottom row of Table 5.In homes with AC units, on average, 79% of the predicted savings from the retrofits are realized. 32Re-estimating Eq. ( 9) using the alternative strategy to estimate the annual average retrofit effect-i.e.restricting and weighting the observations by the inverse of the share of β = 0 2 post-retrofit observations in the corresponding month-we find that, on average, 78% of the predicted savings are achieved during the eleven month period.

Discussion
Our finding that the DEER model overstates the energy savings provided by the efficiency upgrades has important implications.Investments in energy efficiency upgrades are often subsidized based on the predicted energy savings achieved by improvements.For example, the California Energy Upgrade program provides rebates to residential consumers for a wide variety of energy efficiency improvements. 33Homeowners first receive a home assessment from a professional energy auditor and then they can select from an array of recommended upgrades provided by a participating contractor.A rebate worth between $1,500 and $4,000 is then awarded to the homeowner based on the predicted energy savings.If the predicted energy savings are consistently higher than the actual energy savings, then the resulting home improvements may be heavily over-subsidized. 34oreover, as Metcalf and Hassett (1999) highlight, overstating the energy savings that will be achieved by energy efficiency improvements can lead to incorrect conclusions surrounding the existence of an energy efficiency gap. 35In an appendix, we estimate the reductions in the electricity expenditures caused by the retrofits.Our results reveal that, among homes with AC units, 36.The predicted expenditure savings are not directly observed.Therefore, the 13 year payback period is calculated under the assumption that, as was the case with the actual energy savings, the actual annual expenditure savings are 79% of the predicted expenditure savings.Given that SDG&E imposes increasing block pricing, this is a conservative estimate.The actual reduction in expenditure will be less than 79% of the "true" predicted expenditure savings.37.At the same time, it is important to note that while the retrofits do not significantly reduce electricity consumption in the homes without AC units, the efficiency upgrades still likely provide large private benefits to these households in the form of increased comfort levels on days with extreme temperatures.Therefore, if a weatherization program has a distributional goal, then it may not be optimal to reduce the participation among lower-income, lower-energy consuming households-despite the smaller energy savings.
each $1 spent on the retrofits reduces the households' annual electricity expenditures by $0.08.Imposing a discount rate of 5%, and assuming the retrofit effects and electricity prices are constant over time, it would take 20 years for the electricity savings alone to justify the expenditures on the retrofits.However, if we naively assume the ex-ante DEER predictions are correct, the predicted payback period would be less than 13 years. 36This analysis highlights that the upward bias that we find in the predicted energy savings-which is consistent with the results from several earlier studies (Sebold and Fox (1985), Hirst (1986), Dubin, Miedema and Chandran (1986), Metcalf and Hassett  (1999), Fowlie, Greenstone andWolfram (2015))-leads to a substantial downward bias in the payback period for energy efficiency investments.
Our estimates presented in Table 5 highlight another important result.During the spring and summer-the only months with post-behavioral treatment observations-the behavioral interventions are consistently found to reduce consumption by an average of 2.8 to 3.2 kWh/day in homes with AC units.While the savings generated by the behavioral treatments are likely smaller during the cooler months, the impact of the behavioral intervention is nonetheless sizable compared to the savings generated by the retrofits.These findings demonstrate the importance of isolating the impacts of the various treatments.If the total savings generated by the weatherization program are attributed to the efficiency upgrades, they will obfuscate the potential importance of the behavioral elements of the program and further overstate the impacts of the physical retrofits on energy use.
Finally, our results consistently reveal that there is substantial heterogeneity in the energy savings caused by the efficiency upgrades.Retrofitting households with AC units provides much larger reductions in electricity consumption compared to retrofitting homes without AC units.In many cases, rebate programs abstract from the heterogeneity in the resulting energy savings.However, if the goal of the programs is to internalize the external benefits provided by reducing energy demand, then the subsidies for the energy efficiency upgrades should reflect the heterogeneity in the expected savings-as is the case with the California Energy Upgrade program described above. 37

CONCLUSIONS
Residential weatherization programs are receiving widespread support from policymakers.These programs are thought to provide sizable reductions in energy consumption by subsidizing efficiency upgrades for low-income, credit constrained households.However, the expected savings generated by the efficiency improvements are typically based on ex-ante predictions from engineering models.Given the strong assumptions embedded in these models, it is unknown whether the weatherization programs achieve the predicted energy savings.
In this paper, we quantify the electricity savings provided by a residential energy efficiency program targeted at low-income households in California.Similar to other utility-provided weatherization programs, households in our study are treated with multiple interventions.The homeown-ers first receive a variety of efficiency upgrades to their homes.In addition, a subset of the retrofitted households receive a behavioral treatment in which they are provided with information on energy conservation strategies.By taking advantage of the fact that only a subset of the retrofitted homes receive the behavioral treatment, we are able to separately estimate the savings generated by the physical home improvements and the savings generated by the behavioral interventions.
Our results reveal that among households with air conditioning units-the set of households which experience the largest savings-the retrofits reduce electricity consumption by an average of 7%.Comparing our ex-post estimates to ex-ante predictions made using an engineering model, we find that only 79% of the predicted energy savings are actually realized.This result, combined with similar findings from previous studies, leads us to conclude that the potential energy savings from residential energy efficiency upgrades are regularly overstated.Given that the ex-ante, engineering predictions play an important role in evaluating and subsidizing energy efficiency investments, it is important to determine what is driving the overestimation and how the predictions can be improved.Is it the case that engineering estimates are based on a "best-case installation" of efficiency upgrades-which is not achieved in reality?Alternatively, is the overestimation due to a sizable rebound in energy demand among the low-income households?Answers to these questions can help serve to improve the ex-ante predictions of the benefits provided by energy efficiency.
In addition to quantifying the energy savings caused by the retrofits, our results provide evidence that the behavioral treatments can induce sizable reductions in electricity consumption.This suggests that simple informational campaigns are a potentially important component of weatherization programs.Further work isolating the mechanisms through which these savings may be achieved can assist in the improved design of future efficiency programs.Our results also highlight that in order to evaluate the benefits provided by energy efficiency upgrades, it is important to identify the impacts of each treatment included in the weatherization programs.Attributing the program-wide energy savings solely to the retrofits has the potential to even further overstate the benefits of energy efficiency improvements and lead to substantially suboptimal program design.

APPENDIX
To estimate the average reduction in monthly electricity expenditures caused by the retrofits, we estimate the following model: where represents the monthly expenditure ($'s) on electricity.Solving for now Bill β + β ⋅ CD i,m 1 2 yields the annual average reduction in the monthly electricity bills caused by the retrofits.Columns (1) and (3) of Table 6 present the estimates of Eq. ( 10).In homes without AC units, there is no significant impacts on monthly expenditures.In homes with AC units, the retrofits reduce electricity bills by an average of $13.66/month.While the average retrofit cost $1,726, the retrofits ranged between $300 and $6,000.To determine how much of the upfront costs are avoided by reduced electricity expenditures, we estimate the following specification:  Table 6 presents the estimates of of Eq. ( 12).On average, in households with AC units, each dollar spent on retrofits reduces the total expenditure on electricity by $0.08 during the first year.Under the strong assumptions that the retrofit effects and electricity prices are constant over time, we can consider the payback period for the retrofits.With a discount rate of zero, the expenditures on the retrofits would be recouped through reductions in electricity bills in just over 14 years.If the discount rate were 5%, it would take 20 years for the electricity savings alone to justify the expenditures on the retrofits.

Figure 1 :
Figure 1: Consumption Trends daily kWh's consumed during billing cycle, i,m Retrofit = Share of billing cycle post-retrofit, i,m Treat = Share of billing cycle post-behavioral treatment, i,m HD = Average daily heating degrees during billing cycle, i,m CD = Average daily cooling degrees during billing cycle.i,m

Figure 2 :
Figure 2: Distribution of IV Treatment Effect Estimates

Figure 3 :
Figure 3: Average Consumption Changes Pre and Post-Retrofit

Table 2 : Treatment vs. Control / Early vs. Late Adopters
Early Retrofit households are households that have 6 or fewer pre-retrofit observations.The Late Retrofit group have 7 or more pre-retrofit observations.For the treatment and control groups, as well as for the early and late retrofit groups, the values reported are the simple mean across the households.The values in the parentheses are the sample standard deviations.The differences between the sample means are also reported with the standard error of the difference reported in parentheses.* represents a difference that is significant at the 10% level; ** represents a difference that is significant at the 5% level.

Table 4 : Share of Households Treated and Retrofit Effects by Month
Note: Monthly levels of consumption are observed from August, 2011 through September, 2012.Retrofits took place betweenNovember, 2011 and September, 2012.

Table 6 : Effect on Monthly Electricity Expenditures
Models are estimated using household level fixed effects.Clustered standard errors are presented in the parentheses.Explained variation of within household energy usage given by R 2 .*, significant at the 10% level; **, significant at the 5% level; ***, significant at the 1% level.