Does Retrofitted Insulation Reduce Household Energy Use? Theory and Practice

We analyze the household energy use impacts of a large-scale, universally available, subsidized retrofit insulation and clean heat scheme. Theory shows that the energy-saving effects of such schemes are ambiguous. Our difference-in-differ-ence model of energy impacts resulting from each of insulation and clean heat treatment uses a sample of more than 12,000 treated houses. Retrofitted insulation treatment under the Warm Up New Zealand: Heat Smart program resulted in a statistically significant reduction in metered household energy consumption of almost 2%. Clean heat (heat pump) treatment resulted in increased electricity use but little change in total metered energy use other than at warmer temperatures, when heat pumps may have been used as air conditioners. Actual energy savings from insulation are approximately one-third of the modeled energy savings predicted by an engineering model.


INTRODUCTION
In 2009, the New Zealand government established the Warm Up New Zealand: Heat Smart (WUNZ:HS) scheme 1 to subsidize the costs to homeowners of retrofitting insulation and installing clean heat devices.The subsidies were designed to encourage homeowners to raise the heat levels, lower the humidity and increase the energy efficiency of their homes, with the aim of reducing household energy demand and improving health outcomes.These aims were in the context that, by Organisation for Economic Co-operation and Development (OECD) standards, New Zealand homes are poorly insulated and heated (Howden- Chapman et al., 2009;Phillips and Scarpa, 2010).WUNZ:HS provided homeowners up to NZ$1,300 2 towards the cost of retrofitting insulation and NZ$500 towards the cost of an efficient clean heating source. 3It was available for all houses built prior to 2000 regardless of household income (EECA, 2011a).The initial four-year program had the intention of retrofitting 188,500 houses that were insufficiently insulated (EECA, 2011b).
The large-scale, universally available, subsidized retrofit insulation scheme was introduced at a similar time to large-scale, subsidized retrofit insulation energy-efficiency schemes in Australia, USA, Canada, and the UK. 4 Previous analysis demonstrates that price mechanisms outperform command-and-control mechanisms in delivering house insulation efficiently in newly built houses (Jaffe and Stavins, 1995).In the case under study, however, the houses are already built, but without adequate insulation.Our question is whether the addition of universally available, subsidized insulation (and clean heating) to those houses resulted in energy savings in the treated houses.The study is the first analysis to evaluate the energy impacts of this large-scale subsidy-based scheme.It complements prior New Zealand-based studies of smaller-scale schemes (e.g.Howden- Chapman et al., 2005;Chapman et al., 2009;Preval et al., 2010) and international evaluations of similar schemes such as the US Weatherization Assistance Program (WAP) (Brown et al., 1993;Schweitzer, 2005) and Mexico's "Cash for Coolers" program (Davis et al., 2014).
We analyze the impact on monthly metered household energy use (electricity and reticulated natural gas) of those houses that had retrofitted insulation and efficient clean heating installed under the program. 5We find that there are statistically significant reductions in metered household energy consumption as a result of insulation treatment.Installation of efficient clean heating (heat pumps) increased electricity use but left total metered energy use broadly unchanged except at higher temperatures, when energy use increased, indicating that heat pumps were being used as air conditioners.Insulation treatment is most effective in saving metered energy at colder temperatures, with no estimated savings once mean monthly temperatures reach 20ЊC.A comparison of our findings with ex ante estimates finds that actual energy savings from insulation are approximately one-third of modeled energy savings predicted by an engineering model.This result implies that households used energy efficiency savings to boost internal temperatures, which is consistent with findings from related studies that found health benefits arising from the WUNZ:HS scheme.
Section 2 of the paper briefly reviews prior studies and provides a theoretical framework for the analysis.Sections 3 and 4 outline our methodology and data.Results are presented in section 5, with discussion in section 6. Conclusions are presented in section 7.

Prior Studies
Previous smaller-scale (often targeted) New Zealand studies have found that houses with retrofitted insulation save energy (Orion Ltd, 2004;Howden-Chapman et al., 2005, 2009;Lloyd et al., 2008;Chapman et al., 2009;Phillips and Scarpa, 2010).However, prior New Zealand estimates of energy savings from installing new heating sources show only small and/or insignificant impacts (Orion Ltd, 2009;Preval et al., 2010). 6 Impacts of major insulation and heating retrofit programs on energy use have been described in the international literature.In particular, extensive research has been carried out on the impact of the US Department of Energy's WAP, which includes audits and a range of energy efficiency-related upgrades, including insulation retrofits, heater replacement, and draft-proofing (weatherstripping).Evaluations of the WAP include a nationwide quasi-experimental analysis of the impacts of the program on household energy use (Brown et al., 1993), and later meta-analyses of state-level evaluations of the impact of WAP on energy use (Schweitzer and Berry, 1999;Schweitzer, 2005).Schweitzer (2005) finds that participation in the WAP reduces total natural gas use in a typical household by 22.9%. 7 Other recent quasi-experimental international research on the impact of insulation and heating retrofits on energy use includes Hong et al. (2006), who evaluated England's Warm Front, a targeted scheme aimed at reducing fuel poverty by providing insulation and heating retrofits to vulnerable householders.They found that insulation resulted in a 10% reduction in electricity use in central-heated properties and a 17% reduction in noncentral-heated properties, but that a gas central-heating upgrade did not reduce fuel consumption.
In addition to the evaluations of large-scale retrofit programs described above, the international literature on the impact of household heating and insulation retrofits includes a number of quasi-experimental studies.However, these studies are often limited by the absence of a suitable control group or other design issues (Sorrell et al., 2009).There are also assessments of the energy impact of insulation and heating upgrades based on engineering models (for example, Clinch and Healy, 2000;Tommerup and Svendsen, 2006).A limitation of studies based on engineering models is that predicted energy savings may be limited by the phenomenon known as the "take-back" or "rebound" effect (Berkhout et al., 2000;Howden-Chapman et al., 2007, 2009;Phillips and Scarpa, 2010;Gillingham et al., 2013;Gillingham et al., 2014;Levinson, 2014a,b). 8The direct take-back effect represents the proportion of potential energy savings resulting from an energy efficiency upgrade that is instead "spent" via additional consumption of that energy service.
In the context of insulation and heating retrofits, this direct take-back is sometimes known as temperature take-back, as occupants may respond to a reduction in the effective cost of heating their homes by changing their heating behavior in order to live in warmer homes.A recent analysis, based on a meta-analysis of 12 quasi-experimental and nine econometric studies, found an average temperature take-back effect of 20% (Sorrell et al., 2009).In some cases, the direct take-back effect may even result in increased net consumption of energy; this effect is known as "backfire."Backfire was demonstrated in a recent evaluation of the energy savings resulting from the replacement of air conditioners with more efficient models in Mexico as part of the "Cash for Coolers" program (Davis et al., 2014).
Factors that may influence the degree of temperature take-back following an insulation or heating retrofit include internal and external temperatures prior to the retrofit, socioeconomic characteristics of households, and maximum heating capacity (discussed in Section 2.2).For instance, Milne and Boardman (2000) find that initial low indoor house temperatures induce households to increase indoor temperatures as a result of energy-efficiency improvements, but as temperatures rise, energy savings are increasingly taken as cash savings.In the New Zealand context, the WUNZ:HS program was predicted to produce both energy savings and health benefits from in-creased temperature and reduced damp, which implies that some take-back effect was anticipated.In this respect, our discussion of the take-back effect should be considered in the context of a "Policy-Induced Improvement" in energy efficiency that bundles together energy efficiency improvements with other co-benefits (Gillingham et al., 2014).Previous New Zealand randomized control trials of insulation (Howden-Chapman et al., 2007) and heating upgrades (Howden-Chapman et al., 2008) demonstrated both temperature increases and health co-benefits, and thus evidence of the direct take-back effect in action.
Our study differs from the New Zealand and most international studies cited in that it pertains to a subsidy-based scheme that is universally available, restricted only by the requirement that participating homes had to be built before 2000.Unlike some previous small-scale New Zealand programs, there was no randomization of treatment within this government-sponsored scheme, so our methodology uses quasi-experimental methods to assess the scheme's energy impacts.

Theory: The Household Insulation Problem
Given the generally temperate climate in New Zealand, 9 very few houses are fitted with central heating, most households instead relying on stand-alone heating appliances.To understand the potential impacts of retrofitted insulation on energy use in this context, consider the following household problem.The household's utility (U) is defined over both internal house warmth (w) and other consumption (c), with u w Ͼ0, u c Ͼ0, u ww Ͻ0, u cc Ͻ0, where a single (double) subscript indicates a first (second) partial derivative.Prior to insulation being retrofitted, the household owns a certain number ( ≥ 0) of heating appliances.We assume that the number of appliances is held constant following insulation.Given the number of heating appliances, we hypothesize that internal house warmth (w) is a positive function of external temperature (temp), energy use for heating purposes (e), and whether the house is insulated (insul = 1 if insulated; = 0 otherwise).Energy used for heating, e, is constrained to be non-negative and is limited by an upper threshold (h), determined by the capacity of existing heating appliances within the household.Thus we have the following household problem: subject to: where Y is household income; p c and p e are the price of consumption goods (c) and energy (e), respectively; (2) represents the household's budget constraint; (3) represents the technology relating internal house warmth to energy and insulation, given outside temperatures; and (4) embodies the two inequality constraints on energy use.
Given the inequality constraints on energy use, and assuming that all income is spent, this results in the nonlinear programming problem: Maximize c,e : U = u (c, w(e, temp, insul)) + k(Yp c cp e e) + l(e) + v(eh) (5) with complementary slackness conditions: The first-order conditions yield: When energy choice is not constrained (0ϽeϽh), then l = m = 0, and hence the standard optimization condition holds, in which the household balances the marginal gains to utility from extra energy use (via increased warmth) relative to extra consumption against the relative price of energy to consumption: We assume that insulation makes a house warmer for any given energy input; thus, ceteris paribus, u w⎪insul = 1 Ͻu w⎪insul = 0 .In these circumstances, to restore optimality, the household can reduce energy use and raise consumption so as to raise u w ; thus energy savings will be observed.However it is possible, depending on the shape of the w( ⋅ ) function, that w e⎪insul = 1 Ͼw e⎪insul = 0 , i.e. a marginal increment of energy has greater effect on warmth with insulation than without insulation (e.g. because of fewer heat leaks).If this were the case, the impact of installing insulation could be an increase in energy use due to the technological superiority of using energy for heating once a house is insulated relative to the prior situation-an example of "backfire."Thus the effect on energy use of installing insulation is ambiguous and will depend on the shape of the w( ⋅ ) function as well as on the parameters of the utility function.
Prior to insulation, if mϾ0, the household would ideally like to use extra energy for heating purposes at very cold temperatures but cannot do so owing to the upper limit on energy use that the available heating appliances can utilize.In this case, installation of insulation may have either of two effects.First, it may leave e = h but result in a warmer house (since w e⎪insul = 1 Ͼw e⎪insul = 0 ).The constraint still binds after insulation in this case and so is most likely to be observed at the very coldest temperatures when all available heating appliances are being used.Second, insulation could relieve the binding nature of the constraint, resulting in eϽh.This outcome is more likely to occur at cool (but not extremely cold) temperatures when households were previously using all available heating capacity but no longer have to use maximum heating capacity once insulation has been installed.Energy savings as a result of installed insulation may therefore reach a peak at cool, but not extremely cold, temperatures.
When lϾ0, energy is not used for heating prior to insulation being installed.However, as in the nonbinding case, energy use could potentially increase after installing insulation if w e⎪insul = 1 Ͼw e⎪insul = 0 .Thus, at higher temperatures it is possible to observe an increase in energy use after retrofitting owing to the technological superiority of heating after insulation is installed. 10 One further effect may be observed.The utility function depicted in ( 1) is assumed to be invariant to the treatment.If, instead, the utility function incorporates habit-persistence, the experience of living in a warmer house post-insulation could lead to a permanent increase in the desired warmth of a house.In this case, energy consumption would be higher, ceteris paribus, for any given vector of exogenous variables (temp, p e , p c ) after insulation than before, as households become accustomed to greater warmth-another instance of the take-back effect.

METHODOLOGY
Since the WUNZ:HS program did not incorporate randomization of treatment, we adopt a quasi-experimental design to test the impacts of retrofitted insulation and heat pumps on the use of each of electricity and total metered energy (electricity plus reticulated gas).Specifically, we adopt a difference-in-difference (DID) estimation approach that leverages off the large sample of treated houses matched to multiple control houses.We show results from regressions based on an unmatched DID approach and from regressions using an explicit matching approach (see Imbens and Wooldridge (2009) for a discussion of both types of approach).For the former approach, we first include all control houses separately and then group together all control houses matched to a particular treated house as a single mean control house for that treated house.The mean control house approach is adopted in our explicit matching approach.For conceptual and operational reasons, our preferred methodology is to use the explicit matching approach (treated house relative to a mean control house), but we show that the results are similar across alternative methods.In each case, we incorporate house-by-month fixed effects (as in Davis et al., 2014) to control for idiosyncratic energy use by a household that may vary by the time of year (i.e.season).These house-bymonth fixed effects address the potential for selection bias based on time-invariant factors.In order to address one potential avenue for selection bias based on a time-varying factor, our dataset excludes any house that switches energy companies over the sample period, since such a switch may indicate that a new household has moved into the house or that the household has consciously reviewed its energy requirements and changed energy companies as a result.In our unmatched regressions we include time fixed effects to reflect all national variables (including prices) that affect energy use.
Our simplest specification is a standard unmatched DID regression as shown in ( 12): where Energy it is energy use of house i in month t (with energy use being defined respectively as electricity use and total metered energy use), insulation it is a dummy variable ( = 1 if house i has been insulated under WUNZ:HS in month t or in a previous month, and = 0 otherwise), heatpump it is a dummy variable ( = 1 if house i has received a heat pump under WUNZ:HS in month t or in a previous month, and = 0 otherwise), γ i,m is a set of house-by-month fixed effects (thus there are 12 monthly fixed effects for each house, reflecting typical energy use of house i in calendar month m), l t is a time fixed effect (to account for all national level variables), and e it is the residual term.We do not include any fixed house characteristics in ( 12) since these characteristics are accounted for through the house-by-month fixed effects, and as all our information on houses (other than energy use, insulation, and heat-pump installation status) is fixed we cannot add any variable house characteristics to (12).Equation ( 12) is estimated in two forms, which we label (12a) and (12b).In (12a), all treated and control houses are included separately.In (12b), we reflect the way in which our matched control houses are chosen.The control houses are chosen to be as similar as possible to a particular treated house, with most treated houses having multiple matched control houses.In this second approach, for each month we take the mean energy use of the multiple control houses that match a particular treated house and treat the resulting mean control house as a single control house.This approach reduces noise associated with any idiosyncratic control house energy use in a particular month.Thus, in (12a), the number of control houses exceeds the number of treated houses, whereas in (12b), the number of (mean) control houses equals the number of treated houses.
While specification ( 12) is commonly used in DID applications, it does not fully utilize the information that can be gained from having explicitly matched houses as in our dataset.Reflecting recent DID matching approaches, we can rewrite (12) separately for a treated house (superscript T) as in (12Ј) and for its mean control house (superscript C) as in (12Љ) where, in the latter, we omit insulation it and heatpump it since each of these variables is a vector of zeros for a control house: , and eЈ i,t = e T i,t -e C i,t , gives equation ( 13): Thus, in equation ( 13), the dependent variable EnergyDiff it relates to the energy use of treated house i less the energy use of its mean matched control house, each in period t.Equation (13) accounts for temperature differences across the year by subtracting the energy use of a treated house's mean control house in each month, and through the inclusion of house-by-month fixed effects, which account for idiosyncratic differences in energy use between a treated house and its control houses each month.However, both ( 12) and ( 13) imply that the energy use impacts of fitting insulation and a heat pump are the same across all calendar months.With respect to insulation, our theoretical model predicts that energy savings will be greater with cool external temperatures than 11.Quadratic temperature interaction terms are included to allow for nonlinear temperature effects; higher-order terms were not significant when added in preliminary empirical work and so are excluded.The standard deviation of monthly temperatures, when included as interaction variables, were also not significant and so are excluded.
12. See Table 2 for a list of house characteristics.
in warm temperatures.Similarly, we expect that the energy impacts of heat-pump installation will differ according to external temperatures (including the possibility that energy use may rise with warm temperatures if the heat pump is used as an air conditioner when it is warm).
To relax this restriction, we extend (13) to include interaction effects for each of insulation and heatpump.We do so initially with respect to temperature, Temp it , (where Temp it is the measured month-average temperature corresponding to the region of house i in month t).With quadratic temperature interaction effects, 11 (13) becomes: Equation ( 14) can be further extended by adding interaction terms for every available house characteristic (other than suburb location) 12 with each of insulation and heatpump (in addition to the quadratic temperature interaction terms); we label this extension as equation ( 15).
Each of (12a), (12b), ( 13), ( 14), and ( 15) is estimated for each of electricity use and total metered energy use utilizing pooled ordinary least squares (OLS) with errors clustered by house.Before presenting these results in section 5, we discuss our data and test whether pre-program trends differed between treated and control houses.

EECA Data
Data were obtained from the Energy Efficiency and Conservation Authority (EECA) detailing which houses received treatment and the type of treatment received over the period July 2009 to May 2010.A total of 46,655 houses received at least one form of treatment under WUNZ:HS during this period.Treatment is classified into two categories: retrofitted insulation and heater installation.Table 1 details the uptake of each treatment category, showing that the majority of treated houses received only insulation treatment.

QVNZ Data
Addresses of the treated houses were supplied to Quotable Value New Zealand (QVNZ), a state-owned enterprise, to be matched to their database.The matching of addresses for treated houses returned a 79.7% successful match ratio; unmatched houses were removed from the sample.Once addresses were matched, characteristics of each house were extracted to identify suitable properties to be used as controls for each treated property. 13The control houses did not receive any form of treatment under WUNZ:HS over the entire study period. 14 House characteristics used to determine suitable control houses were as follows: location (census area unit, similar to a suburb), dwelling and house type, number of levels, age (decade of build), floor area and number of bedrooms, whether there is a garage under the main roof and its size (number of vehicles), house construction material (walls and roof), whether the house was modernized, and dwelling quality (building and roof condition).Location, dwelling and house type, and number of levels are mandatory matching criteria, while the remaining characteristics form nonmandatory matching criteria.Controls are chosen first according to the mandatory matching criteria, and, second, the nonmandatory matching criteria, for which a matching score was calculated and on which potential suitable controls were prioritized.A total of 269,110 suitable control houses were found.Of the 37,163 matched treated houses, 31,423 houses possess at least one suitable control house; unmatched houses were deleted from the sample.There were 269,110 suitable control houses, with 96.6% of the matched treated houses having at least one suitable control house and 71.7% having the maximum ten control houses.

Metered Energy Data
New Zealand's five major suppliers of metered energy collectively have more than 90% of the electricity retail market share: Contact Energy (24.7% market share), Genesis Energy (23.9%), Mercury Energy (20.2%), Meridian Energy (12.5%), and Trustpower (11.5%). 15Of these companies, Contact Energy, Genesis Energy, and Mercury Energy also supply reticulated natural gas, but only to some regions.Metered energy use data are recorded at the installation control point (ICP) level and each energy supplier submits monthly ICP volumes of electricity and gas use to the respective centralized authority.Submission volumes are expressed in kilowatt hours (kWh) for both electricity and gas, and include modeled and estimated levels of use.
Data on ICP level submission volumes for the period January 2008 to November 2010 were received from four companies (Genesis Energy, Mercury Energy, Meridian Energy, and Trustpower).Using the QVNZ address-matching file, we were able to match data on metered energy use to 152,190 houses within our sample.Of the matched houses, 98.6% have observed electricity use, 13.6% have observed gas use, and 12.2% have both electricity and gas use.
Being administrative data, some data cleaning was required.Data-cleaning steps comprised: (i) removing negative submissions (indicating revisions in estimates of volumes); 16 (ii)  removing houses that switched energy retailer (indicating a possible change in the residents of the house); (iii) removing outliers (the top and bottom 1% of electricity observations and the top 1% of gas observations); 17 and (iv) removing houses with incomplete gas or electricity observations (indicating a possible switch to an energy company not included in our database). 18We note that step (ii) helps to account for a possible change in the residents of a house, and so reduces potential selection issues that might be induced by more or less energy-conscious residents self-selecting into or out of a treated house.

Climate Data
Regional climatic data were sourced from the National Institute of Water and Atmospheric Research (NIWA).Within each regional council area, we chose the weather station located within the most densely populated census area unit of the region.We use data for mean monthly air temperatures (ЊC) for each of the 16 regions in New Zealand.

Working Datasets
We combine the EECA, QVNZ, NIWA, and energy data into a panel dataset that details which houses are treated, the month of treatment, characteristics of treated and control houses, monthly climatic conditions, and electricity and total metered energy use.We distinguish between two categories of treatment: insulation and heating.Given that more than 80% of heating treatments are heat-pump installations, our heating focus is on the effects of installing a heat pump; any house that had heating treatment other than a heat pump is removed from the sample.
Our full dataset used to estimate (12a) contains 12,082 treated houses and 40,216 control houses, with 1,723,919 house-by-month observations.The matched dataset used to estimate equations ( 13)-(15) includes 12,082 treated houses, with 325,439 house-by-month observations.Table 2 provides summary statistics for the energy variables and for the key house characteristic variables (including temperature) in our working datasets for the matched sample.The average monthly mean temperature across the study period for both treated and control houses 19 is 13.64ЊC, with more than 95% of monthly mean temperatures lying between 7ЊC and 20ЊC.For all variables, the means for treated and control houses are virtually identical and the standard deviations are also very similar, reflecting the exact matching on some characteristics and preferential matching on the remaining characteristics.
For the 12 months prior to WUNZ:HS treatment, treated houses used on average 187kWh (2.6%) less electricity and 270kWh (3.4%) less total metered energy than their control houses.Figure 1 shows the difference in electricity use between houses treated only for insulation and their mean control house for the 12 months pre-and 12 months post-insulation treatment (with month 0 being the treatment month).The solid line in the figure represents the mean difference in electricity use between treated and control houses, while the two dashed lines show two standard error bands either side of the mean.The figure shows a broadly stable electricity use difference between treated  Figure 2 shows the corresponding information for the total energy use difference (electricity plus reticulated gas) for houses that received only insulation treatment.In this case, the difference in energy use in the month prior to insulation treatment was -26.1kWh, compared with the mean monthly difference in energy use for the year prior to treatment of -24.4kWh, with this change representing 0.25% of mean monthly total energy use of a treated house.The graphical evidence is consistent with a broadly stable energy use difference prior to insulation treatment and with a decline in energy use post-treatment.
Figures 3 and 4 show the corresponding graphs for houses that received only heat-pump treatment.These figures show a less obvious trend after treatment, and have greater variability and wider confidence intervals prior to treatment, reflecting the smaller sample of houses that received just heat-pump treatment compared with the large sample that received just installation treatment. 20The reduced pre-treatment stability of the energy use difference of houses treated only for heat pumps compared with those treated solely for insulation suggests that we should be more cautious about our heat pump results than for our insulation results.Figures 1-4 present the raw data on energy use differences; they do not account for factors such as seasonality, which may influence mean energy differences depending on date of treatment.These other factors are, however, accounted for in our estimates of equations ( 12)-( 15) and in our tests below (equation ( 16)) for energy use differences between treated and control houses.We test formally whether treated and control houses differed prior to the start of the WUNZ:HS program in terms of the level and trends in their energy use.For each of electricity and total metered energy, we estimate the following equation for the 18 months prior to the start of the program: where everI i = 1 if house i was subsequently insulated under WUNZ:HS ( = 0 otherwise), everH i = 1 if house i was subsequently fitted with a heat pump under WUNZ:HS ( = 0 otherwise), trend is a time-trend covering the full pre-program period, characteristics is the vector of house characteristics (as shown in Table 2), 21 l t are time fixed effects and t it is the residual.
Our null hypotheses, corresponding to a hypothesis that treated and control houses are perfect matches, are that α 1 = α 2 = 0, implying that the average level of energy use was identical for similar treated and control houses, and that α 3 = α 4 = 0, implying that the pre-program trends in energy use were identical for similar treated and control houses.Table 3 provides the results of estimating ( 16) by pooled OLS (with errors clustered by house) for each of electricity and total metered energy.13), the R-squared in the former is considerably higher as it pertains to the proportion of variance explained in a level variable as opposed to a difference variable.One cannot therefore meaningfully compare the explanatory power of the two using the R-squared statistic.
The results in Table 3 indicate that, on average, the houses that subsequently received insulation used less energy than did their control houses prior to the program (possibly reflecting that the residents were already more energy conscious).Relative to average control house energy use, the estimated coefficients on everI indicate that (after controlling for house characteristics) houses subsequently treated with insulation used 1.4% less electricity and 2.4% less total metered energy than did the control houses.There is no evidence at the 5% significance level that houses subsequently fitted with a heat pump used more or less electricity or total metered energy than did the control houses.
More importantly for the DID equations that follow, there is no evidence at even the 10% significance level that houses subsequently treated with insulation had a different trend in either definition of energy use than did control houses.There is also no evidence that houses subsequently treated with a heat pump had a different trend in total metered energy use than did the control houses.However, houses fitted with a heat pump did have an increasing trend in electricity use relative to control houses.These results again suggest that we have to treat the estimated electricity impacts of heat-pump installation with caution, but that the remainder of our results should not be affected by any systematic pre-program trends in energy use behavior that differed between treated and control houses.

Insulation
Tables 4 and 5 report the results from estimating equations ( 12a)-( 15) as specified in section 3. The standard DID specifications in (12a) and (12b) indicate that a house treated with insulation subsequently reduces its electricity use by approximately 12kWh per month and reduces its total metered energy use by approximately 14kWh per month.These each represent an almost 2% saving in energy use.When houses are matched with their control houses, as in equation ( 13), the estimated savings are estimated to be slightly higher but nevertheless still at around the 2% level. 22  When insulation treatment is interacted with the quadratic temperature terms, a more complex impact of insulation treatment on energy use is indicated.With respect to the impact of insulation on electricity use, all three terms are significant both without and with house characteristic interactions added (equations ( 14) and (15), respectively).Similarly, all terms other than the squared term in equation ( 14) are significant for total metered energy.A Wald test of the null hypothesis that all three terms are equal to zero rejects that hypothesis for each energy source, as does a test that the temperature interaction terms by themselves are zero.Thus the impact of insulation on energy use depends on the external temperatures being experienced by the household.
We find little difference in the magnitude or significance of the temperature interaction terms between equations ( 14) and ( 15), while the difference in the estimated coefficients on the level insulation term just reflects the average characteristics of houses that are interacted with that term.For ease of interpretation, we henceforth concentrate on the results of equation ( 14).
Figure 5 plots the estimated electricity and total metered energy impacts of insulation treatment (thick lines) for the temperature range that covers 95% of mean monthly temperatures across the country.At low temperatures (7ЊC) insulation is estimated to save 31kWh and 35kWh for electricity and total metered energy, respectively.Savings decline (nonlinearly) to level off at zero savings for both energy sources at a temperature of 20ЊC.This behavior is in keeping with expectations, since few houses use any heating at temperatures of 20ЊC, so energy savings from insulation are nonexistent at this temperature level.

Heat Pumps
The nonmatched DID specifications (equations (12a) and (12b) in Tables 4 and 5) indicate that a house fitted with a heat pump subsequently increases its electricity use by 11-12kWh per 23.The standard deviation of monthly temperatures in the sample is 2.1ЊC.24.Prior estimates (Grimes et al., 2011) show that our estimates are robust to the inclusion of a weather variability variable, splitting the sample by income bracket, and relaxing some of our data-cleaning procedures, albeit with a slight increase in estimated energy savings when outliers are reincluded in the sample.month and increases its total metered energy use by approximately 4kWh per month.When houses are matched (equation ( 13)), the estimated increase in electricity use falls back slightly to approximately 10kWh per month and there is estimated to be no change in total metered energy use.The lesser rate of increase in energy use for total metered energy relative to electricity is consistent with households reducing the use of reticulated gas for heating.
When the heat pump variable is interacted with temperature, virtually none of the individual heat pump terms is significant at even the 10% level.However, a Wald test indicates that for each energy source, the three combined terms are significantly different from zero (at the 1% significance level in each case).Figure 5 includes plots of the estimated impacts of heat-pump installation on electricity and total metered energy use (thin lines).Installation of a heat pump is estimated to increase the electricity use of a treated house across all temperatures.The greatest increase is at the lowest temperatures, with electricity use increasing by approximately 18kWh at 7ЊC.As temperatures increase, electricity use declines until the temperature reaches around 15ЊC, at which point there is a slight increase in electricity use.This increase likely reflects the use of heat pumps as air-conditioning units (recalling that our temperature variable is a monthly mean, so a mean temperature of 15ЊC may include some days within the month in which households use air conditioning). 23Once the mean monthly temperature reaches 20ЊC, increased electricity use is around 14kWh per month.
The effect of heat-pump installation on total metered energy use differs substantially from that for electricity.For temperatures between 7ЊC and 15ЊC, there is virtually no change in energy use (the estimated monthly impact stays within ‫3.2ע‬ kWh throughout this temperature range).This finding indicates that households with a heat pump do not materially change their energy use for heating purposes following its installation.Given that heat pumps are more energy efficient than other forms of heating commonly used in New Zealand houses, this result implies that houses are warmer as a result of the heat-pump installation.Once temperatures rise above a monthly mean of 15ЊC, we again see a rise in energy use, implying that households use the heat pump as an airconditioning unit.This behavior mirrors that found for electricity use.While these heat pump results are in accordance with expectations, we note again the caveats of section 4 that the results may not be as reliable as those for insulation treatment given the better matching of prior energy use between treated and control houses for those houses treated with insulation relative to those treated for a heat pump.

Overall Impacts
The importance of external temperatures for individual house energy use in response to both insulation and heat-pump treatment means that we must take regional temperature differences into account in assessing the overall energy impacts of the WUNZ:HS scheme.We do so by assessing the location of each treated house (for each of insulation and heat pump), then applying the relevant regional temperatures to the energy difference experienced by each house as a result of treatment.
We find that insulation treatment under WUNZ:HS reduced electricity use by 1.86% and reduced total metered energy use by 1.94% across all treated houses in the sample.Installation of heat pumps increased electricity use by 1.59% but left estimated total metered energy use almost unchanged, with an estimated increase of just 0.26%. 24

DISCUSSION
We explore the significance of these results from a policy perspective.Some evaluations of similar international programs have demonstrated greater average annual energy savings than found here.For example, the 2005 meta-analysis of WAP evaluations suggests an average saving of 22.9% (Schweitzer 2005), while the evaluation of the Warm Front program suggests savings of 10% as a result of receiving insulation, but no impact as a result of receiving a boiler retrofit (Hong et al., 2006).By contrast, a number of more recent evaluations have found much lower (or zero) energy savings.Fowlie et al. (2014), using a randomization strategy, find that homeowners who signed up to Michigan's 2009 WAP had energy savings of approximately one-third of those predicted for their houses.Jacobsen and Kotchen (2013) find that a tightening in building energy codes in Gainesville, Florida, in 2002 resulted in the new houses saving 4% for electricity use and 6% for natural gas use.Chong (2012) and Levinson (2014b) find no significant energy saving effects of California's change in building codes in 1978.
New Zealand evidence has been somewhat limited by sample size and data-quality issues, but Chapman et al. (2009) report a 13.2% reduction in metered energy use following an insulation retrofit, and Lloyd et al. (2008) report a 10% reduction in household electricity use.In this context, the electricity and total metered energy savings resulting from insulation retrofits under the WUNZ:HS program appear relatively small, albeit not inconsistent with more recent international studies.The universality of the WUNZ:HS scheme (as opposed to targeting, as in some other programs) may be one reason that the energy savings from this scheme, while statistically significant, are not as large as in studies of some prior schemes.
Previous New Zealand studies have demonstrated increases in living-room and bedroom temperatures following insulation and heating upgrades.Furthermore, a contemporaneous study (Telfar-Barnard et al., 2011) using the same treated houses as in this study shows health improvements resulting from the WUNZ:HS insulation program (though not from heat-pump installation).Accordingly, we can infer that the temperature take-back effect partially explains the small size of the savings observed.
In their overview of the temperature take-back effect, Sorrell et al. (2009) note that in evaluating data from quasi-experimental retrofit studies one can use an engineering model to predict savings in the absence of the take-back effect.The comparison of observed savings with potential savings produces an estimate known as "shortfall," which can be considered an upper bound of the temperature take-back effect.In our case, the best proxy for an engineering model is the Net Benefit Model that informed key strategic policy documents, including the 2007 New Zealand Energy Efficiency and Conservation Strategy (EECA, 2007), in the years directly prior to the adoption of the WUNZ:HS program.EECA officials made the model available to us, enabling us to explore the magnitude of any shortfall.
The Net Benefit Model is an interactive spreadsheet based on engineering assumptions sourced from various technical documents held by EECA.The Net Benefit Model contains assumptions about the annual kWh savings that can be attributed to various energy efficiency upgrades, per square meter of household size, and it factors in variation in climate via four location categories.By populating the spreadsheet it is possible to predict annual or lifetime savings for a set number of retrofits, provided that information such as retrofit type, household size, and location is available.We enter household size, location, and insulation status data for each household that received insulation in our study and, assuming a take-back of 0%, calculate predicted average annual electricity saving per household in kWh as a result of receiving insulation.(It is not possible to 25.Alternatively, it may reflect inaccuracies in the engineering model or imprecision in our estimates.26.The impossibility of obtaining data for nonmetered energy use means that we are unable to extrapolate our results to total (metered plus nonmetered) energy use impacts.carry out similar calculations for total energy use, as the Net Benefit Model does not distinguish between metered and unmetered gas consumption.)When we contrast this figure with the electricity savings predicted by our favored model, we calculate a shortfall of approximately 63%.This upper bound on the temperature take-back effect is larger than that typically observed, although it is not inconsistent with the more recent international estimates.Our estimate may reflect New Zealand's poor-quality housing stock (Howden- Chapman et al., 2009), which leaves considerable opportunities for households to increase their warmth once houses become better insulated. 25 In interpreting our take-back estimate, we highlight that this estimate is in the context of a Policy-Induced Improvement, as discussed by Gillingham et al. (2014).As a result of WUNZ:HS, households were expected to increase the warmth of their houses with resulting health benefits; Telfar- Barnard et al. (2011) and Preval (2015) confirm the presence of these health co-benefits of the scheme.Thus, as Gillingham et al. (2014) point out, this take-back effect can be considered as a benefit rather than as a cost of the scheme.

CONCLUSIONS
The WUNZ:HS program involved a major part-publicly-financed effort to improve the insulation and heating of New Zealand houses.It is similar, in many respects, to large-scale retrofit insulation schemes implemented at a similar time in Australia, Canada, USA, and the UK.Using a sample of more than 12,000 treated houses (and more than 325,000 house-by-month observations) covering the first 17 months of WUNZ:HS, we find that insulation treatment does, on average, reduce metered energy use for treated houses.The estimates indicate that retrofitted insulation treatment leads to an annual reduction in electricity use for typical energy users in the order of 2% and an annual reduction in total metered energy use (electricity plus reticulated gas) by a similar amount.Our estimated treatment effects vary according to outdoor temperature.Energy savings due to insulation decline nonlinearly from cold monthly mean temperatures for New Zealand (7ЊC) to reach zero at a mean monthly temperature of 20ЊC.
Estimates for the impacts of heat-pump treatment show increased annual electricity use for houses that had a heat pump installed.This increase occurs across the whole range of external temperatures, with the greatest increase in electricity use occurring for houses in cold months.A U-shaped pattern of electricity use increase following heat-pump installation is found, indicating that heat pumps are used for air-conditioning purposes as mean monthly temperatures increase from about 15ЊC.Total metered energy use is left almost unchanged by installation of a heat pump, apart from at higher temperatures, where there is again an indication that heat pumps are used for airconditioning purposes. 26Thus the installation of a heat pump results in a backfire result at all temperatures for electricity use and a backfire result at higher temperatures for total metered energy use.At higher temperatures, however, when the heat pump is being used as an air conditioner, the backfire effect (for both energy definitions) is as a result of the addition of a new service to the household.Households are no longer consuming more heat, but rather consuming a reduction in heat.Whether the addition of a completely new service can be termed backfire (or even take-back) is an issue of definition, but the fact of the increase in energy use is nonetheless real and should be understood by policymakers and households.
The backfire result for electricity at cooler temperatures should not necessarily be interpreted as an adverse effect of the heat-pump policy since policymakers also intended insulation and clean heat treatment to have beneficial health outcomes.Typically, health benefits can be considered a private good, implying no role for government intervention.However, two aspects argue for a public role.First, the existence of a state-funded national health-care system in New Zealand means that taxpayers pay for the costs of ill-health due to poor-quality housing, so not all costs of poor housing are internalized within the household.Second, more philosophically, children in households do not have a choice about insulation or heating devices, so government may have a role in protecting children unless caregivers are considered to be perfect guardians of their children in this respect.
Estimates of health benefits using the same sample of houses included here indicate considerable health benefits from the program and show that these benefits materially outweigh the benefits from energy savings (Telfar-Barnard et al., 2011;Preval, 2015).These results are consistent with the insulation take-back effect (and heat-pump backfire effect) found in the current study and emphasize that our estimates arise from a Policy-Induced Improvement that has both energy efficiency and other co-benefits.Our estimated energy savings due to insulation treatment are only 37% of those estimated using an engineering model applied to our sample of treated houses.A broad-based retrofitted insulation scheme such as WUNZ:HS that increases energy efficiency may therefore have only limited effectiveness in reducing household energy consumption, but may nevertheless result in considerable health (and potentially other) benefits.

Figure 4 :
Figure 4: Mean Total Energy Use Difference (ؓ2 Standard Error Bands); Treated-Control, Pre-and Post-Heat-pump Treatment

Figure 5 :
Figure 5: Energy Impacts of Insulation and Heat-pump Treatment by External Temperature

Table 2 : Summary Statistics for Working Datasets (Equations (13)-15))
Note: There are 12,082 treated houses (with an identical number of mean control houses) with 325,439 house-by-month observations.

Table 3 : Estimation Results for Equation (16)
22. Note that despite the similar results in (12b) and (