Introduction

Many governments subsidise residential energy efficiency upgrades for vulnerable households. The objectives motivating these policies include helping reduce carbon emissions from domestic heating, improving public health and assisting segments of the population that suffer from poverty and deprivation.

Although some efficiency upgrade schemes are focused mainly on a single objective, often to do with supporting climate policy or reducing energy use, domestic energy efficiency has multiple benefits that should be considered together (Ryan and Campbell 2012). Some benefits accrue directly to supported households, such as lower bills, added thermal comfort and improved health and well-being. Other benefits have a broader economic and social footprint, such as decarbonisation and long-term savings in public expenditures. Understanding how an upgrade scheme performs across multiple benefit categories can help authorities design schemes that maximise societal welfare and better anticipate how scheme benefits will vary across different groups within the population.

For example, households that receive energy efficiency upgrades make choices that affect the distribution of benefits between climate policy and poverty alleviation goals (i.e. reduction in energy use and bills) and improving health and well-being (i.e. increasing thermal comfort). Put simply, a household that receives an efficiency upgrade may save money and reduce carbon emissions, or it may take advantage of the lower marginal cost of energy services by consuming more energy and becoming more comfortable and perhaps healthier. The latter response is one element of the “rebound effect” or “shortfall” described in the energy policy literature, which we discuss further in the next section.

Both energy-saving and comfort-increasing responses could be welfare-improving if market failures had previously led to sub-optimal levels of investment in efficiency, so mixtures of both responses (probably typical in practice) may also yield net welfare gains. However, policymakers also care about how the benefits are shared among policy objectives. This is most obvious in the case of climate policy, where interim goals often take the form of target levels of abatement for a set of jurisdictions, sectors or activities. If a measure like upgrades for social housing delivers smaller reductions than anticipated, additional measures will need to be taken to compensate if overall targets are to be met.

In this paper, we examine some of the multiple benefits from funding energy efficiency upgrades to social housing. We contribute to both the academic and policy literature by focusing on a wider range of outcomes than is traditionally examined by such evaluations. Much previous research in this area focuses on fuel consumption; we examine this and a range of other self-reported measures relating to fuel poverty and dwelling condition. We also provide important evidence on the persistence of household consumption of inefficient solid fuel, despite receiving heating system upgrades. Further, we add to the growing body of evidence on the shortfall between engineering model predictions and the observed reality. Our results are very much in line with other research in this area (Fowlie et al. 2015; Aydin et al. 2017). This is important as many policy evaluations still use ex ante predictions rather than ex post observations when evaluating energy efficiency programmes.

The effects of efficiency upgrades on social housing tenants are important to understand both because of the socioeconomic status of this group and because their economic behaviour is likely to differ from households in general. The primary function of social housing is to provide accommodation that is affordable to people on low incomes. This is provided either by the state or by charities and non-profit organisations. The socio-economic characteristics of social housing residents, who rent rather than owning dwellings and receive relatively low incomes, mean they are less likely to invest in energy-saving measures than the general population. Improving the thermal efficiency and reducing carbon emissions of such residences are likely to require some intervention by the state. Such interventions may be more economically efficient than those targeting the general population, as social housing residents do not generally invest in dwelling upgrades in the absence of intervention.

Access to social housing is usually means tested, so these households are likely to suffer from high rates of poverty and deprivation, as well as family structures associated with socioeconomic vulnerability such as single parenthood or job tenures such as unemployment. To the extent that they lead to lower energy bills, upgrades may help advance anti-poverty and related distributional objectives.Footnote 1

Groups that might be particularly vulnerable to temperature-related health problems, including the elderly, the very young and people with disabilities (World Health Organization 1987; Liddell and Morris 2010), also tend to be concentrated among social housing tenants. In some jurisdictions, public health objectives are identified as an important reason for supporting dwelling upgrades.

To carry out the study, we have collected microdata from a sample of social housing tenants in Ireland. In addition to the households who received upgrades, information was collected on a control sample of similar households whose dwellings were not upgraded during the period. We focus on three main questions. First, taking the sub-sample of gas-using households, we estimate regression models to measure how gas demand responded to varying levels of thermal efficiency upgrades. This yields an estimate of the shortfall associated with upgrades for social welfare tenants. While the data do not allow us to directly explore how residents used the savings from upgrades (less spending vs. more comfort), the second part of our analysis uses some survey evidence from upgraded households to consider the possible channels of response. Third, we construct a simple difference in differences model to find out whether the upgrades led to significant improvements in subjective fuel poverty indicators or aspects of dwelling quality.

Importantly, dwellings were selected for the upgrades on the basis of need. Households did not self-select into the trial. The occupants of upgraded dwellings were very similar to the occupants of dwellings not upgraded. The differences between these groups are observable and related to the characteristics of their dwellings, primarily thermal efficiency. This allows us to control for these differences in our econometric models. We also use fixed effects panel estimations giving us further robustness against any unobserved heterogeneity among households.

Overall, we find that use of natural gas fell much less than 1:1 for each increment to thermal efficiency of the home. For the average household in this study, about one third of the marginal increase in thermal efficiency was reflected in reduced gas demand. This result points to a behavioural response from households, potentially taking back some of the savings as increased internal temperatures. Households that received upgrades reported significantly bigger declines in a measure of fuel poverty and a question on problems with draughts than the control group did. However, although there were improvements over time in other measures of fuel poverty and housing quality, the difference between groups on these dimensions was not statistically significant. Among households that were upgraded, respondents were broadly satisfied with the upgrades, agreed that their homes felt warmer, agreed that their homes were now more pleasant places to spend time in, and did not find the upgrade overly disruptive or more difficult to operate.

The rest of the paper is organised as follows: the “Related research” section details other research which relates to home energy efficiency improvements and the rebound effect, with a particular focus on socially vulnerable groups. The “Methodology and data” section contains a discussion of the data available and the econometric modelling approach used in this paper. This is followed by the discussion of the results in the “Results” section, while the “Conclusions” section draws some final remarks and policy implications from this research.

Related research

Researchers have devoted considerable attention to defining and measuring rebound effects, which Sorrell et al. (2009) define as “any increase in energy service consumption [that] will reduce the ‘energy savings’ achieved by the energy efficiency upgrade”. Three overlapping concepts of rebound effects are highlighted, including shortfall (the difference between predicted energy savings from engineering models and actual savings), temperature take-back (the reduction in energy savings associated with change in mean internal temperature after energy retrofit) and behavioural change (reduction in estimated energy savings associated with the change in heating controls or other user-related behaviour).

Empirical estimates of rebound vary widely, partly because authors may be focusing on different mechanisms or ways of measuring the effects, but also because the strength of the effect may depend upon the socioeconomic and policy context. For example, Sorrell et al. (2009) cite nine econometric studies and 12 quasi-experimental studies of rebound in household heating and found that while average long run direct rebound effects are probably lower than 30%, individual studies report effects ranging from 0 to 100%.

Similar results are set out in Sanders and Phillipson (2006), who also survey a set of energy efficiency studies and highlight the confusion surrounding the definition of “rebound effect” and the common discrepancy between predicted energy savings (from engineering-based models) and actual energy savings. They find a typical shortfall (which they term the “reduction factor”) of about 50% between the predicted savings and actual savings, with the comfort factor (the portion of the reduction factor associated with temperature take-back) roughly 15% of the entire reduction factor. In an evaluation of the UK Warm Front scheme, Hong et al. (2006) find similar results, as do Dowson et al. (2012) who suggest the reasons for the energy efficiency upgrades being only half as effective as anticipated “due to a lack of monitoring, poor quality installation and the increased use of heating following refurbishment”.

Some more recent randomised controlled trials (RCTs) in the USA and Netherlands also highlight the discrepancy between predicted and actual energy savings. In a large-scale evaluation of a weatherisation programme for low-income households in the USA, Fowlie et al. (2015) find that predicted savings are 2.5 times greater than actual savings, and fail to find evidence that this is due to increased internal temperatures. Other research examines the elasticity of energy consumption relative to predictions and finds an average rebound effect of 26.7% with substantial heterogeneity (Aydin et al. 2017). The effect is as much as 49% in the lower income groups, and considerably lower in the upper quartiles.

This result is consistent with that of previous research. Milne and Boardman (2000) review 13 studies which examined fuel consumption before and after energy efficiency upgrades for homes designated as being in fuel poverty. They find a comfort factor of 30-50%. They also show that the comfort factor is a function of mean internal temperature and that houses with lower initial mean internal temperature (often those with lower incomes) are more likely to have higher comfort factors. This finding is consistent with the idea that households seek to achieve a target profile of internal temperatures and that as temperatures move towards this profile, the comfort factor for additional upgrades decreases. If this is so, households who have difficulty paying their heating bills may behave differently from those without income constraints that bind as tightly and may have very different rebound effects from the average household. Milne and Boardman (2000) find that for low-income households, the lower average internal temperatures result in up to half of the predicted energy saving being achieved, with the other half devoted to increased comfort in the house. Other research shows that rebound is inversely related to household income in Australia (Murray 2013) and the USA (Thomas and Azevedo 2013).

Some studies have even suggested that rebound can be larger than 100%, i.e. some types of households use more energy after an efficiency upgrade than before. In an RCT, Heyman et al. (2011) find that treated homes (who receive a retrofit 1 year before the control group) tend to increase their energy consumption. However, the authors acknowledge that the results may have been subject to bias due to sample attrition over the 4-year period of surveying. The research period involved four surveys over 4 years, with upgrades in either the third (treatment) or the fourth (control) year. Ultimately, Heyman et al. (2011) favour retrofit programmes as they “generate modest but long-lasting fuel efficiency gains which translate into increased room temperatures rather than financial savings, a sign of the importance which people with limited resources place on staying warm”. Hong et al. (2006) also find that households can increase their fuel consumption after an upgrade.

Looking specifically at Ireland, Scheer et al. (2013) study effects of energy efficiency upgrades on household gas consumption. Using billing data from 210 households that self-selected into a co-funded energy efficiency subsidy scheme, they find a shortfall of approximately 36%, estimated as the difference between the predicted engineering model-based consumption change and the actual change. Respondents also reported other benefits of the energy efficiency upgrade, ranging from improved well-being and home comfort to an increase in the perceived value of their home. The authors acknowledge that selection bias is a potential problem when using data from an upgrade programme in which beneficiaries have to opt in and were required to contribute to the cost of the energy efficiency upgrade, so the results are not necessarily representative of the national population.

As noted earlier, upgrading domestic energy efficiency offers many household-level benefits including health promotion and poverty alleviation, as well as broader macroeconomic benefits. It is beyond the scope of this research to provide a detailed literature review on these related topics but for the interested reader, useful starting points include Wilkinson et al. (2009) and Hamilton et al. (2015) on health, Hills (2012) and Watson and Maitre (2015) on fuel poverty and Ryan and Campbell (2012) for a survey of multiple benefits more broadly.

Methodology and data

Research design

Background

In 2014 Respond! Housing Association received approval from Sustainable Energy Authority of Ireland (SEAI) to undertake home energy improvements in a number of its housing estates throughout Ireland. These estates are owned and managed by Respond! who provide rental accommodation for their residents. SEAI allocated support on the basis of an application made by Respond! Dwelling upgrades were then co-funded by SEAI and Respond! residents were not required to provide any additional funding towards the cost of dwelling upgrades. These dwellings form the population from which the sample for analysis is drawn.

The SEAI Better Energy Communities Scheme provides funding for community-based home energy efficiency improvements. Community groups submit a proposal to SEAI to upgrade their housing stock. SEAI evaluate the bid and decide which dwellings to upgrade. SEAI then co-fund the upgrade with the relevant community group. This scheme was launched in 2013, and by 2016 had supported over 12,000 homes, community, private and public buildings in receiving energy efficiency upgrades.

Selection of upgrade and control group

Households did not self-select into the trial but were chosen by Respond! and SEAI based on the characteristics of the dwellings. Dwellings were identified by Respond! Housing Association as being in need of energy efficiency improvements. As this is a community-based programme, selection is not based on an individual basis. Respond! decide to upgrade dwellings on an estate-by-estate basis primarily based on the age profile of the dwellings in a housing estate. This happens on an annual basis as part of ongoing management and refurbishment of their dwelling stock. In the years prior to our analysis, Respond! typically achieved a success rate of 50–60% of dwellings in their funding application to SEAI. In any previous year, this would have resulted in 40–50% of dwellings being ear-marked for funding but not receiving it due to budget constraints. These dwellings could potentially have formed a control group. In the year of our trial, 100% of dwellings ear-marked for upgrades received funding. This completely absorbed the proposed control group and required the research team to select a control group from other dwellings within Respond’s stock. This group was chosen to be as similar as possible to the upgrade group in all observable characteristics, but were not due to receive an upgrade that year. Detailed descriptive statistics comparing the upgrade and control groups is provided in the supplementary material.

Timeline and focus of study

The research team undertook a pre-upgrade survey on households in June–July 2014. Households were asked a range of questions related to factors such as family composition, income, self-reported fuel poverty and heating-related problems. Houses in the control group also completed surveys to allow before and after comparisons between groups. As part of the survey, the household manager/head of household was asked to sign a data access agreement, allowing the researchers access to their electricity and gas consumption over a 3-year period. At the time of completing the survey, households in the upgrade group would have known about the proposed upgrades in the upcoming months.

The upgrades were completed over the following autumn, and a post-upgrade survey was completed in October–November of the following year.Footnote 2 The timeline for this project is displayed in Fig. 1.

Fig. 1
figure 1

Timeline of project

Full size image

As displayed in Table 1, the final sample contains 260 households who completed both waves of the survey, 164 of which received home energy efficiency upgrades, 96 of which did not. This total sample contains two subgroups: a group which consists of 210 households with signed electricity billing access agreements; another group of 100 households who provided signed gas billing access agreements.

Table 1 Sample size and attrition
Full size table

The objective of this paper is to examine how the upgrades affected home heating and other related factors. Given this, the samples of interest are the 260 homes who completed both surveys, and the sub-sample of 100 homes with gas central heating. The focus is on these households because the upgrades are primarily related to heating, not electricity consumption and metered gas provides an objective outcome variable that can be measured pre- and post-intervention. Information is also collected on secondary fuel usage, but given that this data is self-reported, we are much less confident about accuracy. The sample of dwellings for which we have details of electricity use will be the focus of future research.

Attrition does not vary systematically between the treatment and control groups. However, given the need to include as many dwellings as possible with objectively measured energy consumption, the proportion of treatment and control households in the final gas sample is statistically different from the initial proposed dwellings. Also, households who left the study by definition did not sign agreements allowing us access to their data; we cannot compare their energy consumption with those remaining in the study.

Data

As discussed in the previous section, respondents in both the upgrade and control groups completed pre- and post-upgrade surveys. Respondents also signed a waiver allowing the authors access to their gas and electricity consumption. ESB Networks provided metered electricity consumption data and Gas Networks Ireland provided metered gas consumption data. Data on the dwellings, including dwelling characteristics, location and information on the type of upgrade and when they were completed, were obtained from Respond! Housing Association. Weather data was downloaded from The Irish Meteorological Service, Met Eireann’s website.

Dependent variable: gas use

Our dependent variable is the metered gas demand in each billing period, expressed as a daily average given the number of days in the period. In most cases, the period covered was from January 2013–December 2015, a 3-year period (n = 65). In some cases, we could only get access to a 2-year billing period, from January 2014–December 2015 (n = 35). Also, some houses had new gas boilers installed as a replacement for their previous heating system (n = 8). For these dwellings, we do not have consumption data prior to their upgrade; however, we include them in the analysis (yielding an unbalanced panel). The initial cleaning and smoothing of the raw gas data are described in the supplementary material to this paper, along with details on the 6 households removed from the final analysis. The composition of the final sample used in the econometric analysis is described in Table 2.

Table 2 Final sample used in econometric analysis
Full size table

Thermal efficiency of dwellings: BERpred

Our proxy for the thermal efficiency of each dwelling in the sample is based on its building energy rating (BER). Established by SEAI, this engineering-based metric is based on a bottom-up model of factors affecting energy efficiency. A BER must be carried out by anyone wishing to sell or rent a property in Ireland, and BER inspectors are certified by SEAI. The model predicts the average energy requirements of each dwelling given its physical characteristics, and these values are used to rate properties on a categorical scale. The BER values relating to the residences in this study were provided to us by Respond! Housing Association.

We take the raw BER score, which is in units of kilowatt-hour/square meter/year, and scale it to match the units in our dependent variable by multiplying it by the area of each dwelling, dividing by 365 (to convert it to a daily basis) and multiplying it by ratio of heating degree days to the total number of days in each billing period. This figure, which we refer to as BERpred, is a proxy for the gas demand per day expected in a given billing period for a particular residence assuming the residence uses only gas for heating. If a residence received an efficiency upgrade during the sample period, we reflected this by reducing BERpred accordingly. There was no other change to BERs during the study period. One possible limitation of the BER as a proxy for expected gas demand is that efficiency upgrades may improve the efficiency of electricity-using applications as well as gas-using applications. If this is so, not all the effect of the upgrade will translate to a savings in gas demand.

Figure 2 compares the BERpred values for our sample to actual average consumption. Consumption values are lower than predicted levels in almost all cases. Moreover, the gap tends to widen at higher level of predicted energy consumption (i.e. for lower efficiency houses). Particularly given that these households are social housing tenants, it is likely that limited income constrains energy consumption. Some households may not be able to afford to maintain the level of thermal comfort assumed by engineering-based models, particularly those living in dwellings that are inefficient and thus relatively expensive to heat. Use of secondary fuels may also help explain the divergence for some households.

Fig. 2
figure 2

Comparison of predicted heating demand based on each dwelling’s BER (BERpred) with average actual gas use per day; results are included both before and after upgrades for households that received them. BERpred includes the electricity used for lighting and ventilation in addition to the gas used for heating

Full size image

Weather conditions

Daily weather data is taken from Met Eireann website.Footnote 3 Daily data was downloaded for a number of weather stations located around the country. These were then assigned to the nearest housing estate in our data, using GIS software.

The variables we include are sunlight hours, rainfall (mm), windspeed (knots) and heating degree days. Sunlight hours is a measure of the duration of sunshine in a day. Daily rainfall is measured in millimetres; the daily mean windspeed is measured in knots (equivalent to 1.852 km/h). Heating degree day is calculated from the average daily outside air temperature. It is defined relative to a base temperature of \(15.5~^{\circ }\)C, above which it is assumed a building needs no heating. If the average daily temperature is \(1^{\circ }\) below this, it is referred to as one heating degree day. Sunlight hours and rainfall are expressed as average values for each billing period and heating degree days are expressed as a ratio to the total number of days in the period.

Socioeconomic characteristics

Earlier in the paper, we noted that socioeconomic characteristics can have a significant effect upon a household’s energy use and response to energy efficiency upgrades. To control for such effects, we include a range of socioeconomic variables in our models. In this sub-section, we list them and briefly explain their expected effects on residential heating demand.

Income should have a positive effect on energy usage, as shown in previous research (Brounen et al. 2012). However, given the limited degree of cross-sectional variation in income in our data, we may not observe a statistically significant effect.

Older households have been found to spend a significant proportion of their income on space heating, and demand is also found to increase with age (Liao and Chang 2002). In a large randomised controlled gas smart-metering trial, Harold et al. (2015) find that households with older chief economic supporters (CES) consume more gas and younger households consume less relative to a reference category of 36 to 45 years. However, the magnitude and significance of these effects are reduced once dwelling characteristics relating to energy efficiency are included. This reflects the propensity of older people to live in lower quality dwellings, on average, than their younger counterparts.

The number of occupants is positively correlated with gas consumption (Harold et al. 2015); however, scale economies have also been observed, and each additional person decreases the per-capita consumption by 26% (Brounen et al. 2012).

Only 15% of our total sample is in full-time employment (18% for the gas sample). A large proportion (61% of total, 57% of gas sample) describes their employment status as unemployed, retired, suffering from illness or disability or home duties. One might expect these groups to use more energy, on average, than others spending more time outside the home. This also highlights the importance of being able to heat the house properly for these vulnerable groups.

Fuel Allowance is a cash payment under the Irish National Fuel Scheme to help with the cost of home heating during the winter months. It is paid to people who are dependent on long-term social welfare payments and who are unable to provide for their own heating needs. Sixty percent of households in our gas sample receive this benefit. It is unclear whether receipt of this benefit will increase gas consumption. On the one hand, it might enable income-constrained households to more adequately heat their homes, increasing consumption. On the other hand, these households are likely to be more vulnerable to poverty and deprivation generally (Watson and Maitre 2015), and additional cash may be used to meet other needs.

Table 10 in the Appendix presents descriptive statistics on selected socioeconomic characteristics of the chief economic supporter (CES) for each household. In most cases, it was necessary to aggregate characteristics into larger cells when analysing the data, because there were too few households in the sample with particular individual characteristics. For example, income and age categories are each aggregated into two broader categories when we apply regression analysis.

We compare our sample across gender, education and income with the population of social housing inhabitants from the Central Statistics Office (CSO) Household Budget Survey in Table 11 in the Appendix. We observe a higher proportion of female respondents in our sample; both groups have very similar levels of education; levels of income are lower on average in our sample, with a much higher proportion earning €20,000 or less per annum. It must be noted that the most recent HBS for which we could obtain data was conducted in 2009/2010, our survey was conducted in 2014.

Table 12 in the Appendix displays the results of a binary regression model examining the probability of being in the upgrade group for the gas dwellings. These results indicate that male CES, meter-type and the energy efficiency of the dwelling are significant predictors of being in the upgrade group.

Our average family features between two and three people with a chief economic supporter who is over 55 years of age, with a leaving certificate (upper secondary) level of education, and who is unemployed.

Dwelling characteristics

Structural dwelling characteristics have been found to influence space heating demands more so than factors related to occupancy and the socioeconomic characteristics of inhabitants (Brounen et al. 2012).

Semi-detached homes account for 72% of dwellings in our gas sample. The remainder is bungalows, apartments and terraced homes. We aggregate all other groups in the analysis and use semi-detached as the reference category. Harold et al. (2015) found that relative to households living in semi-detached dwellings, those living in apartments used less gas, while those in detached homes and bungalows used more. We should expect consumption to increase with the size of the dwelling. This is measured as the number of rooms in each dwelling.

Many income-constrained households opt for a pre-paid meter to help with managing energy bills. It is possible that such households might ration their use of home heating services differently from those on ex post billing arrangements. In our gas sample, 37% of households have pre-paid meters installed. We control for meter type in our estimations in case the meter type has an impact on gas consumption.

The typical dwelling type in our sample is a three-bedroom semi-detached house with PVC windows and three occupants (including the respondent). Details on the dwellings can be found in Table 13 in the Appendix.

The energy efficiency upgrades

Table 3 highlights the type of upgrades administered and the percentage of the 164 treatment households who received each upgrade. Cavity wall insulation was the most common upgrade (92% of treatment households), with a vast majority of households also receiving a combination of heating controls, attic insulation and CFL lightbulbs (80, 80 and 72% respectively). Most dwellings had a new boiler installed and over 40% of houses had their windows and doors replaced.

Table 3 Upgrade received - Full sample
Full size table

Collins and Curtis (2016) found that most residents who applied for a grant scheme in Ireland selected either one (33% of sample) or two (63%) upgrade measures.Footnote 4 The upgrades undertaken in our study are, on average, much deeper than this. Over half of treatment households receive five upgrade measures (55% of treatment group). Most households receive multiple energy efficiency upgrades, with the most common combination being attic insulation, heat boiler, heating controls, CFL light bulbs and cavity wall insulation (23%).

Figure 3 displays the distribution of building energy ratings (BERs) for the control and upgrade group with metered gas data, both prior to and after the upgrade. The control group was on average more energy efficient prior to upgrade. The upgrades significantly improved the energy efficiency of the dwellings, shifting the distribution of BERs to the left of the control group. A similar pattern is observed for the non-gas dwellings. Details of which are in Appendix A.

Fig. 3
figure 3

BER rating for gas sample, control group and upgrade group (pre- and post-upgrade)

Full size image

Consumption of other fuels

This section focuses on the subset of gas-connected households, who had a metered gas connection both pre- and post-upgrade. In both surveys, respondents were asked a number of questions relating to their purchasing of solid and other liquid fuels, excluding metered gas. These questions enquired about the quantity and costs of their most recent purchase, the frequency of purchasing and the approximate amount purchased in the past 12 calendar months. Table 4 illustrates the proportion of gas-connected households consuming any non-zero amount of a range of other fuels. It shows that households with metered gas are consuming a range of other fuels, particularly coal. This reduces somewhat after the upgrade but a certain proportion continues to use other fuels along with gas.

From this table, it is difficult to determine the extent to which households are using these other fuels as primary or secondary heating sources. Figure 4 displays the self-reported annual spending on all other fuels before and after the upgrade for dwellings with gas boilers.

Table 4 Self-reported usage of other fuels
Full size table

These data are likely subject to measurement error as they are self-reported by households, but they give a sense of the potential magnitude of expenditure on other fuels. Many households reported spending a significant proportion of their total annual fuel expenditure on other fuels.

Fig. 4
figure 4

Total non-metered fuel spend in € (pre, post, difference)

Full size image

A small number persist with heavy usage of other fuels. This has implications for climate policy as it suggests the effect of upgrades to more efficient gas and oil boilers as a means of reducing emissions may be less than expected if some households continue to use coal and other solid fuels.

We create a dummy variable for those households that report solid fuel expenditures pre and post to control for the fact that their gas consumption will not reflect their total fuel consumption. However, this will still lead to a certain degree of error in our models as it is difficult to determine the extent to which households are substituting gas for other fuels.

Analytical methods

To explore how energy efficiency upgrades affected the households in our sample, we model the gas use of the subset of households in our sample that use natural gas using panel regression estimations. Our main interest is in isolating the effect of our energy efficiency proxy, BERpred, from confounding factors. By using this measure, we can examine treatment intensity, rather than a treatment binary variable. This allows us to measure the elasticity of actual energy consumption with respect to predicted. Because the data are longitudinal, we can also include random or fixed effects to control for household-specific effects that do not vary over time. Seasonal factors not captured in our weather variables are addressed using dummy variables for each billing period.

We estimate the average daily gas consumption of each household in each period (\(\frac {Y_{it}}{N_{t}}\)) as a function of the predicted energy use of the dwelling given its efficiency (BERpredit), other dwelling characteristics (Di), socioeconomic characteristics of household members (Xi), weather (Wit), and time dummy variables for each period (ρt).

$$ \frac{Y_{it}}{N_{t}} = f(\text{BERpred}_{it}; D_{i}; X_{i}; W_{it}; \rho_{t}) $$
(1)

An outcome of particular interest is how actual gas consumption varies with respect to the predicted consumption of the dwelling. Because our demand model is linear, the elasticity of actual consumption with respect to predicted consumption is calculated using this formula:

$$ E = \frac{\partial(Y)}{\partial(\text{BERpred})} \frac{\overline{\text{BERpred}}}{\overline{Y}} $$
(2)

The \(\frac {\partial (Y)}{\partial (\text {BERpred})}\) term is the marginal effect of BERpred on demand, equal to the regression coefficient on BERpred in our models. This is multiplied by the ratio of mean BERpred to mean demand.

We include six two-monthly time dummy variables to control for any unobserved seasonal trends not picked up by the weather variables. These seasonal dummies are also interacted with the BER variable, allowing for possible seasonal variation in the elasticity of actual consumption with respect to its predicted value.

Further analysis is conducted on a range of self-assessed outcomes, including the presence of housing quality problems and a subjective indicator of fuel poverty (going without heating or difficulty paying utility bills).

Results

Econometric analysis of gas consumption

This section will provide insights from the reduced sample of our data featuring 94 gas-connected households who completed surveys before and after the upgrade period. Occupants consented to the researchers gaining access to their gas meter readings. The billing data (mostly) covers a 3-year period (January 2012–December 2015) before and after the period of the home retrofit upgrade scheme.Footnote 5

Descriptive statistics for all continuous variables used in the following regressions are presented in Table 5.

Table 5 Descriptive statistics for continuous variables (n = 1139)
Full size table

The upper panel of Fig. 5 displays the average consumption for the upgrade and control group, over a 3-year period during which the upgrades were undertaken. It would appear that the upgrade dwellings consumed more gas on average than the control dwellings prior to the upgrade, and that this difference was reduced post-upgrade.

Fig. 5
figure 5

Bi-monthly gas consumption split by group

Full size image

The lower panel of Fig. 5 displays the average consumption for those on pre-paid and those on post-paid meters, for a 3-year period during which the upgrades were undertaken. From examining the graph, there does not appear to be much difference between the groups.

Results from panel regressions

The results from a number of estimations are presented in Table 6. Model 1 is an OLS random effects model that includes a set of socio-demographic characteristics that did not change over the sample period, as well as the energy efficiency proxy variable BERpred, weather and time dummy variables. This model is tested down to exclude collectively insignificant variables (P = 0.43), and the resulting parsimonious random effects model is shown as model 2. Model 3 includes only the time-varying controls and is estimated using fixed effects. All the models are shown with robust standard errors, because a likelihood ratio test indicated the presence of heteroscedasticity (P = 0.00).

Table 6 Regression explaining household gas demand in each billing period; random effects estimator
Full size table

As expected, there is a statistically significant positive association between BERpred, the predicted heating requirement based on the building energy rating, and households’ actual gas use in each billing period. This implies that households who received efficiency upgrades tended to have lower gas use afterwards, all other things equal. The interaction terms between bimonthly time dummies and BERpred indicate that the efficiency effect was much stronger than average in billing period 1 (January–February) and much lower in period 5 (October–November). These variables may be picking up a lagged effect of upgrades on behaviour, e.g. as upgrades tended to be done in the autumn, it may have taken time for households to readjust their use of heating services fully.

The numbers of heating degree days and other weather variables are not significant after we include time dummies and time interactions with BERpred. Since BERpred takes into account the number of heating degree days in the billing period, the absence of an independent effect of heating degree days on gas demand is not surprising. Among the time dummies, the ones denoting the third and fourth billing periods of the year (July–October) indicate significantly lower average demand compared to the sixth (November/December). Households may be less sensitive to weather parameters during the summer and early autumn when adjusting their use of heating systems .

Few socioeconomic controls are statistically significant. The sample used in this study has less variation across these dimensions than the national population, because social housing tenants are selected at least in part on observable characteristics. Some variables have the expected signs, e.g. number of rooms or low-income status, so lack of statistical significance may partly be due to the limited sample size.

The positive marginal effect of thermal efficiency is consistent across the three models. As tests of robustness, we estimated a log-log version (logging the dependent variable and BERpred), a version with a three-period moving average of gas demand in place of the smoothed gas demand series used in the models above, and a variant of model 2 omitting the interactions between BERpred and time dummies. These checks yield similar estimates of the BERpred relationship to the main models. The Sargan-Hansen test (P = 0.003) suggests that the fixed effects model (model 3) is preferred to random effects.

Table 7 below shows the elasticity of demand with respect to BERpred as estimated in the models we estimated. The elasticities resulting from these models imply that improving the thermal efficiency of an average residence by 1 kWh reduced its gas use by about 0.33 kWh, which implies a shortfall of about 0.67.

These elasticities are broadly in line with the international research discussed earlier; households on low incomes should be expected to take some of the benefits of improved energy efficiency in the form of increased thermal comfort, with the remainder feeding through into lower heating bills and carbon emissions.

Energy affordability and self-reported heating problems

In addition to improving the energy efficiency of dwellings, one of the objectives of providing home energy retrofits is to alleviate fuel poverty. We assess this by comparing how responses by treatment and control groups to certain questions change after their dwellings have been upgraded. Results from a difference-in-difference analysis are reported in Table 8. We find a statistically significant reduction in the mean number of treated households reporting that they went without heating through lack of money. Interestingly, both the treatment and control groups reported an improvement in their ability to pay utility bills. This may reflect more generally improved economic circumstances over the course of the trial.

Table 7 Elasticity of gas demand with respect to BER-based predictions of heating requirements (BERpred) for a range of specifications, evaluated with all other variables at means
Full size table
Table 8 Full sample—fuel poverty and heating problems
Full size table

Another aim of the upgrades was to improve the quality of accommodation for tenants. In both survey periods, respondents were asked to report the presence of issues which might indicate an inadequately heated home. Table 8 also displays the change in a range of heating and dwelling fabric related issues and how they vary across groups.

Over the course of the study, the group that received upgrades was significantly more likely to report a reduction in problems with draughts than the control group. For all issues, both groups reported significant improvements over time, but the differences between groups were not statistically significant for problems other than draughts. The pre-upgrade survey was conducted in June–July 2014 and the post-upgrade survey in October–November 2015. It is possible that the timing difference exerted a common unobserved effect on both groups, as external weather conditions would have been different between periods. It is also possible that improved economic conditions more generally allowed both groups to heat their home more adequately in the post-upgrade period. Another explanation is that we are witnessing a form of “Hawthorne effect”, in which the responses of both groups are altered because they are being studied (Landsberger 1958). Another possibility is that there was some “experimenter demand effect”, whereby responses were altered to reflect the behaviour that respondents thought was expected of them (Orne 1962). This issue was raised with the housing association and they were not aware of any other external factors which might have contributed to it.

This observation highlights the importance of having a control group in a study such as this, as there may be unobserved general trends affecting both groups that would otherwise be missed by the researchers.

Occupant’s satisfaction with upgrade

The final section in the results focuses on household satisfaction and awareness of energy-related issues after efficiency upgrades among the households whose dwellings were upgraded. The issues explored relate to overall satisfaction, perception of warmth, improved awareness of energy usage and behavioural change as a result of the upgrade.

From Fig. 6, it is clear that households were broadly satisfied with the upgrades, agreed that their homes felt warmer, agreed that their homes were now more pleasant places to spend time in, and did not find the upgrade overly disruptive. Most respondents did not find the new system more difficult to operate. The level of disruption expected is widely cited as a factor which makes households less likely to engage in retrofits (when they have a choice). Given that most of these households received deep retrofits and many were likely to be at home a lot, it is encouraging that they did not generally find the upgrades to be disruptive.

Fig. 6
figure 6

Household’s satisfaction with upgrade

Full size image

Awareness of energy use seems to have increased following the upgrades, and most households agreed that post-upgrade they heated more rooms when their heating is on, but that they also used less heat due to improved control. The answers to the other questions on behavioural change were less consistent across the households surveyed. Households varied in their responses to questions about whether they had changed how often the heating is on, the time spent in the home and the likelihood of going to bed due to cold. Time spent in the home is likely to be significantly affected by socioeconomic factors other than thermal comfort. Frequency of heating system use may interact with other aspects of use, e.g. whether one is heating a single room or using central heating.

Conclusions

The impact of energy efficiency upgrades on social housing tenants remains an important topic for research because this group includes many vulnerable people whose housing quality is directly amenable to policy intervention. Studies of social housing tenants also offer methodological advantages compared with field experiments involving other groups, not least because they give rise to less risk of self-selection bias. Although there may be sample selection involved, it is more likely to be on the basis of observable characteristics than would normally be the case for programmes where participants opt in.

In evaluating such programmes, it is important to understand the full range of effects, not just on energy use and carbon emissions but also on deprivation and other outcomes. While our main empirical results focus on estimating shortfall in the effects of efficiency measures on gas demand, we have also been able to cast some light on how upgrades affected fuel poverty, thermal comfort and general satisfaction.

Focusing on the sub-sample of gas-using households, our econometric results support the findings in international research that lower income households exhibit relatively high levels of shortfall (a measure of rebound) when their energy efficiency is upgraded. This is likely accompanied by relatively high temperature take-back, though we could not test this directly. Our estimates are consistent with other international work examining the rebound effect for lower income groups, while higher than that found previously for Ireland in Scheer et al. (2013), who examined a programme giving grants to households that can afford to make part of the investment themselves. There are some caveats to our analysis. One is that we were not able to measure use of secondary fuels as accurately as natural gas consumption. Though many households reduced their use of secondary fuels, self-reported purchases of coal and other fuels were surprisingly high in the sample. Another limitation is that we could not directly assess the quality of thermal upgrades. However, since the co-funded upgrade scheme and Building Energy Rating system are subject to regulation and an inspection regime run by the funding authority, we have no reason to think that these upgrades exhibited any significant level of quality problems. A third caveat is that the proxy we use for thermal efficiency is actually designed to measure energy efficiency more generally. It is possible that some of the shortfall we measure is due to improvements in efficiency of electricity-using applications. Given that this proxy is designed to also measure the electricity used in lighting and ventilation, it is possible that by using this measure, we systematically overstate gas consumption. However, the bulk of the upgrade measures taken in this case was aimed at improving thermal efficiency.

An inverse relationship between rebound and income might be taken to imply that environmental policy will be more effective when it is focused on better-off households (e.g. Thomas and Azevedo 2013). However, this is true only in the narrow sense that upgrades to such households will be more effective at reducing energy use and carbon emissions. Total welfare gains from upgrades may well be as high or higher for upgrades to low-income households, depending upon one’s distributional preferences and on the value of the benefits associated with higher dwelling temperatures.

We observed a statistically significant improvement in the self-reported proportion of households who went without heating through lack of money—a subjective indicator of fuel poverty. This improvement stands in contrast to the experience of our control group, who reported a lower level of difficulties but did not see an improvement during the study period. The broader ability to pay utility bills improved for both the upgrade and control groups, possibly reflecting more generally improved economic circumstances. A significant difference between upgrade and control households was not noticed in this case.

Conditions seem to have been improving generally for the social housing tenants surveyed during this period and several other indicators of deprivation or housing quality showed improvement for both upgrade and control households, although the changes were larger and more statistically significant for upgraded households. These measures included the incidence of mould and draughts. Along with increasing indoor temperatures, reducing the severity and prevalence of such issues has been linked to a variety of improved health outcomes (Hamilton et al. 2015).

Of particular interest is the heavy usage of solid fuels in addition to gas central heating in our sub-sample. Our households did not self-select into this trial, and a certain reluctance to switch heating source was noted by the housing association. This has clear implications for carbon reduction in the domestic sector. Certainly, publicly funded home energy upgrade programmes must take account of behavioural factors when upgrading heating systems, particularly if there is persistence in fuel choice which can only be partially alleviated by changing the central heating system.

Unfortunately, we could not measure the health effect of upgrades in this study, nor could we monitor internal temperatures. Given our limited sample size, we could not calculate the energy savings associated with certain measures, as others such as Adan and Fuerst (2015) for example have done. These are limitations of this work. Health effects tend to take more time to emerge in a measurable way than the other benefits of upgrades, which suggests that data collection may have to take place over a longer period to measure them reliably. Another possibility is to track in-home temperatures before and after upgrades, as has been done in some studies internationally, and then to infer likely health benefits from improved temperature profiles. The falling cost of sensor technology may make this a more practical proposition in future studies.