Assessing the effectiveness of gamification in reducing domestic energy consumption: Lessons learned from the EnerGAware project

The application of gamiﬁcation to encourage energy conservation behaviour in house occupants is an emerging ﬁeld of research. However, empirical evidence of its effectiveness is lacking. This paper presents lessons learnt from the EU-funded EnerGAware research project, in which an innovative serious game (a game designed for purposes other than purely entertainment) was developed to promote reduced energy consumption and carbon emissions by changing social housing tenants’ energy eﬃciency behaviour. The game was validated in a sample of European social housing using a longitudinal, two-stage experimental design, employing both pre-post and control group approaches. While some aspects of the game did not work as intended, there were nevertheless some positive impacts. The intervention increased social housing tenants’ awareness and engagement in certain energy saving behaviour and provided an average electricity saving of 3.46% and an average gas saving of 7.48%. Although savings were found not to be statistically signiﬁcant, an effect size was detected (0.2). Therefore, future steps should exploit all available opportunities to replicate the pilot and increase the sample size so as to gain stronger evidence of the game’s impact. Preliminary results support the utility of gaming investment in the household energy ef-ﬁciency ﬁeld, and provide useful insights and pathways that could be incorporated into the development of future serious game interventions to foster their effectiveness. ©


Introduction
As stated in the recently reviewed Energy Performance Directive [24] , the European Union is committed to developing a sustainable, competitive, secure, decarbonized energy system by 2050. To meet this goal, special attention must be paid to the building sector as it accounts for 38.9% of all the energy consumed in the EU-28 [27] and is among the largest end-use consumer sectors [13] .
Various approaches have been proposed to reduce the energy consumed by buildings, including the adoption of building energy efficiency standards, promoting building renovation and implementing applied ICT solutions for building automation, among others. However, findings reported by Zhao et al. [42] indicate that technological advances in building systems directly contribute to just 42% of energy efficiency, which suggests that an impact on energy savings is highly dependent on behavioral plasticity. Several * Corresponding author.
E-mail address: marta.gangolells@upc.edu (M. Gangolells). strategies for encouraging occupants to conserve energy have been proposed in recent decades, including economic stimuli, feedback mechanisms and social norms [22] . However, these effort s have had varying levels of success [ 31 ].
Most of the existing campaigns for fostering energy conservation behaviours are typically designed as information-intensive and they seem not to be enough motivating [41] . Concurrently, a growing body of literature supports the use of gamification to enhance learning and engagement in education, from kindergarten through to postsecondary levels [28] . Outside formal education, gamification has also recently gained significant traction as a method of producing attitude and behaviour change [39] . Within this context, serious games are defined as virtual simulations of real-world activities that can educate users and prompt behavioral change [36] . Despite growing interest and some initial attempts, serious games' potential to engage consumers in energy efficiency behaviours has not been researched extensively [33] .
The main objective of this paper is to advance in this direction by exploring the effectiveness of gamification in reducing domes- tic energy consumption through lessons learnt in the EU-funded Horizon 2020 project EnerGAware -Energy Game for Awareness of energy efficiency in social housing communities [20] . In the En-erGAware project, an innovative serious game was developed and implemented to promote a reduction in energy consumption and carbon emissions in social housing.
Following this introduction, Section 2 reports previous initiatives related to use of ITs to motivate pro-environmental behaviour change. Section 3 contains the research method. First, the process and main outcomes related to identification of user, building and game requirements are described. Then, the serious game, including the game mechanics, storyline and simulation engine, is detailed. Section 3 also reports the game validation, with a description of the pilot implementation process and a discussion of the results. Section 4 examines the cost-benefit opportunity of the solution. Finally, Section 5 provides conclusions and future research.

ICT solutions to motivate pro-environmental behaviour change for energy efficiency
Smart meters collect the electricity consumption in a high frequency (i.e. quarterly or hourly values) and transmit these figure to a data hub [4] . Beyond the development of applications for visualizing energy consumption data [34] , several ICT solutions have been diffused for involving users, ensuring awareness and promoting behavioral changes [37] .
Within this context, displayed feedbacks providing users with tailored suggestions or advices to save energy have been largely investigated [ 5 , 6 , 17-19 , 23 , 29 , 30 ]. In some cases, information is immediately provided to the user (direct feedback) whereas in some other cases, information is processed before reaching the final user (indirect feedback). Feedbacks have been used within the context of antecedent strategies (information is linked to a goal setting) or consequent strategies (information rewards -or notusers' behaviour). Information has been provided through different devices or interfaces such as in-home displays, web-based interfaces, apps for smartphones, online social networking tool, interfaces for TV, emails or vocal messages [37] .
Approaches based on gamification have also been an emerging area of focus to motivate, engage and educate users regarding energy consumption and related concerns. Relevant initiatives within this area are those extensively reviewed by Johnson et al. [31] , Pasini et al. [37] , AlSkaif et al. [1] , Boomsma et al. [7] and Csoknyai et al. [12] . Gamification based energy applications are mostly related to energy efficiency, self-consumption or demand response [1] and the most recent ones such as those developed under the Energy Efficiency programme of the European Commission [ 29 , 35 , 38 ] rely on the advantages of the smart metering systems massive roll-out. Existing games are either built on cooperation or competition [37] and they are based on a number of storylines including dialogues, avatars, etc. Games have been developed in domestic contexts, public buildings and workplaces, targeting several users including young people, university students, family members and workers [37] .
A thorough literature review seeking effectiveness of gamification in motivating energy conservation behaviour allows concluding that games appear to be of value, with varying degrees of evidence of positive influence. However, much of the existing research is exploratory by nature. When impact effectiveness of behaviour changes interventions is measured, results raise questions about its reliability and validity due to weaknesses in program design and evaluation [ 43 ]. Therefore and as documented by Johnson et al. [31] and Morganti et al. [33] , there is a need for more quantitative and qualitative empirical research to ascertain the effectiveness of applied games to motivate individuals to become more energy aware and to translate this knowledge into action.

Research method
The game was designed to achieve significant energy consumption and emissions reduction in a social housing pilot project by increasing social tenants' understanding and engagement in energy efficiency. By playing the game, users should learn about potential energy savings that can be made by installing energy-efficiency measures and changing their behaviour, whilst maintaining comfort at home. The game should function either without an internet connection or with a link to the actual energy consumption (smart meter data) in the game user's home using a specific energy metering system or within the context of smart metering roll-out. To maximize user acceptation and thus the impact on energy reduction, Living Lab methodology was used, and social tenants were engaged in the design of the serious game from the outset.
The research method used in the project included the following steps: − Definition of user, building and game requirements − Game development − Game validation Fig. 1 summarizes the implementation of the research method in a 3-year period. Requirements were elicited between month 1 and month 6 and led to the initial design of the serious game. A game prototype was released in month 12. The core gameplay was then refined and validated through an iterative testing process involving social tenants. A beta version of the game was released in month 24 and deployed in a UK social housing pilot project. Apart from fixing some bugs reported during the experiment, work during the third year of the project focused on making the game available to the general public (simplification of some game features that were only needed within the context of the experiment, translation of the game so that it was available in English, French, Spanish and Portuguese by the end of the project and preparing the game so that it could be downloaded from iOS and Android app stores).
To validate the game, an energy metering solution needed to be implemented in the pilot houses. When the system requirements had been established, the design and deployment of the energy metering system could begin. To assess the effects of the energy saving intervention, variables were measured and compared before the intervention (baseline evaluation) and at the end of it (final evaluation). The baseline period started in month 10, when the energy metering sensors and communication devices had been deployed and energy consumption data were available for the first time. It finished in month 24, when users started playing with the beta version of the game. The reporting period started just after the end of the baseline period (month 24) and finished when energy monitoring kits were uninstalled or tenants answered the final survey (month 35). Thus, both the baseline and the reporting period covered one year (including one heating period) to account for a minimum of one normal operating cycle and to fully assess the impact of the energy saving intervention [14] . To gain insight into the mid-term effect of the game, variables were also examined three months after game implementation (month 27).
In order to enhance the robustness of the experimental design, a quasi-randomized controlled trial where tenants are either assigned to the experimental or the control group was also used (see Section 3.3 ). Tenants in the experimental group played with the game whereas tenants in the control group did not.
The EnerGAware project received full ethical clearance from EU Commission Services and the Ethics Committee of the University of Plymouth and therefore, all research activities, especially those related to research involving human participants and personal data collection and/or processing, were conducted in compliance with fundamental ethical principles. Informed written consent was obtained from all the tenants participating in the pilot.

Definition of user, building and game requirements
The design of the serious game and the metering system solution required comprehensive identification and analysis of the specific user, building and game requirements. As reported in Casals et al. [9] , requirements were defined using a range of datasets and methods: − Literature review: a detailed review of previous projects, publications and reports related to the design and use of IT in social housing was undertaken and used as starting point for the definition of requirements. − Socio-economic characteristics, energy consumption motivations, behaviour and perceptions, game experiences and IT literacy: collected during a large-scale, city-wide survey administered in Plymouth (United Kingdom) to 2772 social houses (social housing survey). − Game experiences and game feature preferences: collected during three focus groups with social housing tenants in Plymouth (gameplay scenario focus groups). − Building characteristics of social housing stock in Plymouth: information contained in the social housing provider database (building stock condition database) was used.

User requirements
Social tenants involved in the Living Lab methodology suggested that the serious game virtual world should be based on a domestic environment, so that players can relate to it.
In addition, visual aspects of the game should consider requirements associated with the human aging process and novice users. In relation to the game's didactic approach, it should be adapted to various learning levels and provide clear and easy to understand goals. The educational content should provide knowledge on how much energy is used by typical end-uses in a domestic environment, poor use practices that might increase energy consumption and the most efficient ways to save energy. The game should help players to assess potential energy savings resulting from various behavioral actions and energy-efficient changes at home. A link to social media platforms to enhance communication and information sharing among players was found to be a relevant functionality.

Building requirements
The most common building characteristics, building envelope, building services and controls and renewable energy generation were analyzed and transformed into a typical dwelling which was used to design the virtual home in the serious game and its neighbouring houses ( Fig. 2 ). This typical house was also used to define the simulation engine underpinning the serious game. Data about internet penetration, as well as energy meter types and possible locations were used to devise the pilot implementation plan.

Game requirements
Most of the social tenants were found to have good IT-literacy, internet and social network habits, and experience in playing video games. Therefore, an online serious game approach should not be a barrier to the targeted audience. Both the focus group and the social housing survey results suggested that the serious game should be a management game with virtual house customisation mechanics. Focus groups concluded that a pseudo-realistic game setting would be better than a fantasy world (or science fiction or cartoon) or a fully realistic simulation. Finally, a touch-screen tablet was identified as the most suitable IT device (in technical terms and cost efficiency) for deployment of the serious game.

Game development
According to the Living Lab methodology [10] , requirements elicited during the first stage of the project were used as the main input for the design of the serious energy game. A prototype of the serious game was released in month 13 and several play-test prototype feedback focus groups were undertaken to present the entire concept of the game and its main features to the social tenants (months 14 and 22). Focus groups were held in a 'soft-lab' replicating a home living room environment at the School of Psychology, University of Plymouth (UK). Participants played with the early prototype of the game and provided feedback on its playability and display features. The beta version of the game was released and deployed in the pilot homes in month 24. After the evaluation phase, the final version of the serious game, called Energy Cat: The House of Tomorrow [21] , was available for the general public.

Storyline
The main character of the game is the Energy Cat and it is the one controlled by the player. The Energy Cat lives in a house with humans, that are controlled by artificial intelligence. The Energy Cat is eager to live in a comfortable, energy-efficient house and therefore corrects the human characters' actions and advises them on energy saving. Within the house customisation mode ( Fig. 3 ), players can create their dream house without any restrictions in a realistic environment using the editor function ( Fig. 4 ). The mission mode provides stories instilling knowledge about energy efficiency and educating the player about proper energy management behaviour. Some missions are in the main house, but others take place in neighbours' houses. Neighbours' houses provide tailored scenarios that cannot be shown in the player's house (e.g. a house with children), but they also provide ideas about how the player's house can evolve.
Social media features of the game give users a platform to share data on their achievements, compete, provide energy advice and form virtual energy communities. For this feature, users must be connected to the internet.

Game mechanics
According to the storyline, the player has a house with humans living in it. The main gameplay loop ( Fig. 5 ) starts with a daily pool of energy points that diminish gradually according to the energy consumed by the house. The players' main objective is to reduce household energy consumption so as to save energy points at the end of the day. Energy points allow progress in the game as they are used to unlock new items and upgrades in the catalogue, some of them are just ornamental (house furniture, wall and floor claddings and other decorative items) but others are related to energy saving measures (insulation, new windows, lighting devices, electric appliances, etc.). Implementing energy saving measures allow loosing energy points at a slower pace so at the end of the day more energy points can be saved. New items and upgrades increase humans' happiness, which in turn increases daily income. Money that is earnt can be used to buy upgrades and new items. Mission rewards add extra income. However, extra energy consumption decreases daily income.
Energy points are the game's most valuable resource and the player has a direct influence on the number of energy points he/she will save. Energy points can be saved by upgrading electric appliances (i.e. changing energy-guzzling boilers for more energyefficient models), improving the building's thermal performance by modifying external walls, roof and windows and changing the behaviour of the humans living in the house with energy efficient actions such as closing the windows whilst the house is being heated, using the shower for a shorter time and turning the light off when a room is unoccupied. Saving energy points unlocks new game content, such as house furniture, decorative items and new appliances.
Money is not saved directly by the player but by the humans living in the player's house. The income, provided on a daily basis, depends on the player's last actions, but extra money rewards are hidden in the house to encourage players to play every day and to look for bad behaviour in all rooms of the house. Money can be used to buy items that have been previously unlocked with energy points, upgrade current appliances or invest in energy efficient upgrades for the house.
Energy points and money interact with a third parameter called happiness. The happiness of the humans in the house depends on their comfort and the appliances and furniture they have. Happy humans are more productive and earn more money. Therefore,  there is a need to invest in better equipment, smart, connected devices and insulation to reduce the expenditure of energy points without decreasing the happiness of the humans in the house.
If the game is connected to smart meter data, energy savings achieved in the players' actual homes also enable progression in the serious game. The energy consumed by a house in one week is compared to the energy consumed the same week the year before. If a significant reduction is observed, the player gets an extra money bonus. To motivate the player, the bonus is proportional and cumulative (if the player does not log in every week) to the measured reduction.

Simulation engine
Energy savings achieved virtually in the game are calculated by building performance simulation using the white box modelling approach and dynamic thermal simulation software (DesignBuilder Version 4, powered by EnergyPlus). The simulation engine calculates the annual energy consumption of the player's virtual house and determines the energy savings associated with the energy efficient actions undertaken by the player, including: − Space heating energy consumption, considering the increase in demand associated with the extension of the house floor area and the savings associated with implementation of energy efficiency house upgrades (such as adding insulation to the building envelope). − The use of domestic hot water and a boiler upgrade. − The use of electrical appliances and lighting, with a focus on different use modes (active and standby), energy efficiency levels and appliance sizes and power. − Implementation of renewable energy technologies, such as solar photovoltaics, micro wind turbines or solar hot water.
Missions at neighbours' houses were simulated using a house model of known geometry and volume. In this case, the impact of (i) turning down the thermostat, (ii) shortening the heating season, (iii) zonal heating and (iv) leaving windows open during the heating season was assessed. Considering that the game requires Simulation results were transformed into an in-game score with enough sensitivity to highlight meaningful changes in energy consumption indicators and keep the game both realistic and engaging.

Game validation
Within the pre-post comparison approach, energy savings achieved by introducing any energy saving intervention cannot be measured directly, as they represent the difference between the energy that is actually consumed after the intervention and that which would have been consumed had the intervention not been undertaken. Therefore, according to the International Performance Measurement and Verification Protocol (IPMVP) [14] , the methodology used to test the effects of the serious game intervention on the social housing pilot project adopted the entire facility pre-post comparison approach (Option C) ( Fig. 6 ). In addition, following the recommendation of European ICT PSP methodology for calculating energy savings in buildings [3] , a control group approach was also implemented ( Fig. 6 ).
The experimental design assessed the impact of the serious game over time and compared it to a control group using dependent variables ( Table 1 ). Other potentially influential factors were considered using independent variables ( Table 1 ). Independent variables were used to assess the effect of some unavoidable variations between houses in the experimental and control groups. Table 1 summarizes the variables collected for each group.
Most of the data needed for the energy behaviour change evaluation were collected from the baseline and final social housing surveys for both the experimental and control groups. In this case, the sample size is related to the number of returned surveys and it changes across conditions and time ( Fig. 6 ). Data needed to evaluate energy-related dependent variables were obtained through a real-time energy monitoring system deployed in the homes of social tenants in experimental and control groups. In this case, the sample size ( N = 44) corresponds to the number of houses where the energy monitoring system could finally be deployed ( Fig. 6 ). As explained in Section 3.3.2 (Pilot results and discussion), an experimental subgroup ( N = 18) was also examined including those houses that actually played with the game.
The baseline evaluation was conducted in month 24 by sending a baseline survey to all pilot homes asking about energy consumption behaviour, energy awareness, IT literacy, and selfreported manual meter readings to cross-check automatic readings. Three months after implementation of the game, the same survey was sent again to all pilot homes, with questions to collect feedback on the game from houses in the experimental group.
Data for the final evaluation were collected through a final evaluation survey in month 35 sent to all pilot homes. After the final evaluation tasks, the real-time energy monitoring system infrastructure was removed from the houses in the pilot project and meter readings were collected to double-check the data collected by the energy monitoring systems. Face-to-face interviews were held with the tenants to gather detailed feedback on the game. In month 36, a focus group with tenants in the experimental group was held to collect more detailed information.

Pilot implementation
The following subsections describe the pilot implementation process, including tenants' recruitment strategy, the design and deployment of energy data collection and communication infrastructure and middleware and game implementation.
Recruitment strategy : The recruitment strategy was carefully designed to maximize the number of tenants involved in the project ( Fig. 7 ). The social housing survey was sent by post to 2772 social houses in Plymouth in month 4. To encourage households to complete and return the survey, a reminder was sent out on month 5 and a prize draw was used as an incentive. In total, 537 of the households had completed the survey by the end of month 5 (504 paper-based and 33 internet-based surveys), giving an overall response rate of 20%. In the survey, households were asked whether they would be interested in participating in further follow-up studies. Out of the 537 households that completed the survey, 237 stated that they would be interested. These households were asked again if they would like to take part in the monitoring stage and 137 answered affirmatively. However, 23 households had to be discarded as they did not match the monitoring solution that had been developed (i.e. they had pre-paid gas meters, dual [Economy 7] electricity meters, non-pulse electricity meters, digital meters or an old analogue dial). Tenants who did not have an internet connection at home were not discarded, as the game was designed to be played on-or offline. In any case, tenants could use internetrelated features of the game through public Wi-Fi networks. At the end of the recruitment process, 114 socials homes in Plymouth were found to be suitable for participation in the pilot project. Out of the 114 shortlisted homes, the energy monitoring system was finally deployed in 88 houses. Equipment could not be fitted in the remaining homes because tenants were not reachable after several attempts to contact them, tenants were no longer interested or the information collected during the social housing survey about existing meters was not accurate enough.
Selection of subjects participating in the project was made based on volunteers, introducing the so-called selection bias. In or- der to minimize it, all participants that wished to take part in the intervention were assigned to either the experimental or the control group. Tenants failing to consent/respond to the offer were not assigned to the control group. Social houses should be assigned by a purely random process to either the experimental and control groups so that in an ideal situation, the social houses assigned to both the experimental and control groups would be identical in terms of socio-demographic and dwelling characteristics, and the only difference between the groups would be the presence or absence of the energy saving intervention. Identical experimental and control groups are achieved in other fields, such as medical control trials. However, for applied built environment and energy-related projects, practical considerations usually prevent the rigorous application of these standards. In this case, social houses were assigned to either the experimental or control group using a pairing approach in which two identical/near-identical houses were identified and one was randomly assigned to each group. A subsequent analysis of the main socio-economical characteristics (and others) of the houses in both the control and experimental groups (44 homes each) was undertaken to identify any significant differences between them that would need to be considered during the evaluation. The results of the study did not highlight any concerns.
Design and deployment of energy data collection and communication infrastructure and middleware : The energy data collection and communication infrastructure were developed ad-hoc for game validation as a generic solution for all the houses in the pilot to minimise the demand and time commitments on the participating households and to ensure maximum compatibility with the electricity and gas meter types identified during the requirement analysis stage.
In order to read electricity consumption data, an optical pulse reader and a standard wireless M-Bus pulse counter were attached to the existing electricity meters. For the gas readings, the solution included an energy cam that visually recognises the data displayed in the existing meters and transforms it into an M-Bus parameter that can be read wirelessly. A data concentrator collected monitoring data and periodically sent it to a remote data server. As an internet connection was not available in all the pilot homes, a dedicated virtual private network (VPN) was adopted, implemented on a GPRS communication solution. The energy monitoring kit installed at each home sent all the stored information every 15 min, upon request from the remote server. The game validation strategy described above required middleware to perform data management. The middleware was built on the FIWARE platform [ 15 , 16 ] and maintained a repository with three types of data retrieved from different sources: − Pilot households' gaming experience data, available from the game server. − Energy consumption data collected by the energy monitoring system installed in the pilot homes. − Local weather data, available from an automatic web weather service, needed to analyse the weather impact on the energy consumption profile.
The middleware also provided a set of services to aggregate and export the relevant data to compute game rewards linked to realworld energy savings (used by the player to advance in the game) and ease the game validation. In accordance with the data management plan, both pilot energy consumption data and households' game experience data were anonymised so that individual homes could not be identified.
Game deployment : The beta version of the game was released and deployed in the pilot homes in month 24. First, a letter was sent to all the households in the pilot with information about tablet delivery and including the baseline survey. A few days later, psychologists visited all the experimental houses in the pilot and delivered the tablet with the game installed on it. During the same visit, the completed baseline survey was collected, and energy meter readings were taken. If the resident was not at home, a 'we missed you' flyer was put in the letterbox with contact details to arrange a revisit on a specific date and time.
Several actions were undertaken to keep the participants in the experiment involved during this 3-year project and motivated enough to attend focus group sessions, provide feedback when needed or even to play the game. To encourage responses to the surveys, several reminder letters were sent and prize draws and shopping vouchers were used as an incentive. Although pre-paid postal return envelopes were always left, tenants could also respond to the surveys online. In addition, a team of psychologists tried to maintain an optimal relationship with the tenants during the entire project and support them when needed. Lack of engagement was addressed with other actions such as sending Christmas cards, leaflets, letters with information on game updates and invitations to focus groups.

Pilot results, discussion and limitations
The energy saving impact of the serious game intervention was evaluated by estimating the average energy consumption reduction of houses in the experimental group during the reporting period in relation to the baseline. The energy consumption reduction of a given social house (i) in the experimental group in relation to its baseline (ECR exp baseline i ) was calculated using the following expression: (1) Where C exp reporting i stands for the total energy consumed in household i in the experimental group during the reporting period, expressed in kWh, and C exp baseline i, represents the total energy consumed in household i in the experimental group during the baseline period, expressed in kWh. According to the IPMVP [14] , the adjustments term in Eq. (1) was used to express both pieces of measured energy under the same set of conditions and may include energy-governing factors expected to routinely change during the reporting period such as the weather (routine adjustments) or energy-governing factors that are not expected to change such as the size, design and operation of installed equipment or type of occupants (non-routine adjustments).
Regarding routine adjustments and according to the European ICT PSP Methodology for calculating energy savings in buildings [3] , weather changes are the main reason for variability in residential consumption profiles. Considering that energy consumption is predominantly heating-related in the UK, weather-correcting energy consumption figures were based on Heating Degree Days (HDD). Heating Degree Days are a measure of the amount of time when the outside temperature falls below the base temperature (when the building needs heating). Heating Degree Days can be calculated for a certain period according to Eq. (2) as the addition of the difference between a base temperature and the outdoor temperature, when the outdoor temperature is lower than the base temperature: (2) where T base is the outdoor temperature above which the building needs heating and T outdoor is the outdoor temperature.
According to the Carbon Trust [8] , Heating Degree Days (HDD) were calculated based at 15.5 ºC at a daily basis for both the baseline and the reporting periods. The average daily temperature was calculated using outdoor temperatures retrieved from the middleware platform.
Energy consumption is closely related to heating needs. Most of the houses in the pilot (85.2%) were gas heated. In these cases, only gas consumption figures had to be weather-corrected; electricity consumption was not considered to be weather dependent. Other pilot houses were heated with electricity (14.78%). In these cases, Heating Degree Days were used in the analyses of electricity consumption figures. However, adopting such an approach may lead to some inaccuracies, because disaggregated energy consumption data were not available within the social pilot and all the electricity (not just that used for heating) was divided by Heating Degree Days. In any case and so as to minimize this potential bias, electricity heated houses were equally distributed among the experimental and control groups.
Non-routine adjustments were not considered, because no changes in energy governing factors could be identified during the reporting period in the mid-term and final surveys. Table 2 shows the average baseline electricity and gas consumption of houses in the pilot during the baseline period.
Results gathered in the mid-term survey revealed that not all houses in the experimental group really played the game. Therefore, besides assessing all the houses in the experimental group (44 homes) as a group, an experimental subgroup was created of houses in the experimental group that stated in the mid-term survey that they had played the game (offline) and houses whose ingame data records was analysed separately (18 houses).
According to the results obtained in the pilot project after recategorisation and weather normalisation, playing the Energy Cat game was associated with an electricity saving of 3.46% ( Table 3 ). These results are especially relevant when compared to those of the control group, who were found to increase their electricity consumption by an average of 1.68%. Similarly, analysed data revealed that all the houses in the experimental group used less gas during the reporting period in relation to the baseline period (2.73%). As expected, this saving was even greater in the experimental subgroup (7.48%). In contrast, houses in the control group were found to use slightly more gas during the reporting period than in the baseline period (1.15%).
The unpaired two-samples t -test was used to verify whether these differences were statistically significant. Taking into account that t -test requires independent and normally distributed samples to be applied, the Shapiro-Wilk test was used as it was considered to be the most appropriate for a small sample size. Obtained results (p-value > 0.05) showed that data was normally distributed for all the analysed groups. The F-test, used to test the homogeneity of variances, indicated that population variances for each group were different, except when analysing gas savings in the experimental subgroup versus the control group ( Table 4 ).
According to the obtained results, the classical t -test or the Welch t -test were used to evaluate whether the means of experimental groups and control group are different for both electricity and gas ( Table 4 ). In this case, the p-value was found to be higher than the alpha significance level (p-value > 0.05) for both electricity and gas ( Table 4 ). Therefore, it cannot be concluded that the average electricity and gas savings for the experimental and control groups are statistically significant.
However and as stated by the American Statistical Association [40] , smaller p-values do not necessarily imply the presence of larger or more important effects, and larger p-values do not imply a lack of importance or even lack of effect. Additional tests addressing the effect size must be considered [2] . Therefore, the Cohen's effect size was also calculated to determine the size of the difference between groups ( Table 4 ). When comparing the experimental and the control group, the effect size was found to be 0.1 for both electricity and gas, and therefore and according to Cohen's classification, it must be concluded that in this case the game had no effect. However and when comparing the experimental subgroup (made of those tenants that effectively played with the game) with the control group, the effect size was found to be low (0.2) according to Cohen's classification. Cohen [11] stated that studies should be designed in such a way that they have an 80% probability of detecting an effect when there is an effect to be detected. In this case and considering an effect size of 0.2 and a p-value lower than 0.05, the sample size of future studies should be increased to 393 for each group to avoid the type II error or what is the same, a false negative. This means that the pilot sample used in the EnerGAware project (highly restricted by availability and technical constraints) was finally underpowered to detect statistically significant results.
In addition and when the final reporting results ( Table 3 ) were compared with those obtained in the mid-term analysis, savings were found to decrease in all the analysed groups, which indicated a greater short-term impact of the EnerGAware intervention. The results showed that the intervention did not reduce average home electricity peak demand and average power demand at the net- Table 2 Baseline energy consumption of pilot houses.

Electricity Gas
Energy consumption during the baseline period [kWh s /house •day] 9.08 14.10 Table 3 Mean electricity and gas savings and corresponding standard deviation of the pilot houses at the end of the reporting period.  In terms of self-reported energy-related behaviours, engagement in the project was found to have a significant, positive impact on the perceived affordability of energy bills. Specifically, while no differences were found between houses in the experimental group and those in the control group, all subjects were more likely to state they found it difficult to pay their energy bills at baseline than in the final stage. This suggests that simply taking part in the project had a positive impact on improving perceived affordability for social housing tenants across conditions.
In general, no differences were found in engagement in twentythree energy saving behaviours between houses in the experimental group and houses in the control group or over time. The results suggest that behaviour change may have been limited due to a desire to maintain comfort levels, for health reasons, and because subjects perceived that they already used very little energy. Nevertheless, there was a significant difference in energy-related behaviours over time, with subjects in the experimental (versus control) condition being more likely to state they had set their bedroom radiator to a lower temperature than normal.
There were no differences in energy awareness between tenants in the experimental group and those in the control group in terms of understanding, perceptions and motivations, perceived control or social norms. However, there was a significant difference in understanding over time, with subjects being more likely to state they understood how their home used energy in the final stage than in baseline stages. This indicates that participation in the project helped to improve understanding across conditions.
The intervention did not affect the experience of fuel poverty. However, in the final stage, a lower proportion of subjects in both conditions stated they 'did not know what fuel poverty was'. As such, it appears that engagement in the research project helped to educate subjects about fuel poverty across conditions. No changes were found in terms of IT literacy across condition or over time.
Before ending this section, it is worth spending few words on the limitations of the game but especially, pilot implementation. A first cautionary remark is related to the high attrition rates in the experimental group, as demonstrated by the low numbers of survey respondents in the final stage who stated they had played the Energy Cat game. Although responses to the interviews and the fi-nal stage survey highlighted that there may have been reduced interest in playing digital games, feedback from tenants who played the game showed that the game's complexity and a lack of support were critical issues that appeared to have prevented subjects from effectively interacting with the game. This was reflected in the below average usability scores for all subjects. Therefore, future versions of the game should aim to improve support and instructions and reduce game complexity, in order to increase usability.
Results should be interpreted with caution because of unequal group sizes. To compensate for high attrition rates, future studies should consider recruiting higher numbers initially. When defining the recruitment strategy, authors suggest targeting a representative sample bearing in mind that only approximately 1/3 of the experimental group will probably actively engage with the programme.
In a classic treatment-control group approach, it is important to ensure that tenants in the control group did not change their behaviour because they knew they are part of the project. In this project and although effort s were made to avoid exposing tenants in the control group to the research project aims, ethical and practical constraints did not allow monitoring households' energy consumption without the tenants' informed consent.
Subjects did not perceive that the game was linked to their behaviour in real life, which provides a key explanation for the lack of real-life energy savings as a result of playing the game. In future interventions, it is important to consider how to overcome these limitations and to enhance the educational functionality of the game, with a focus on teaching users how to reduce consumption whilst maintaining thermal comfort. Incorporating this feedback into new serious game designs may prove a highly effective means of overcoming remaining psychological barriers to behaviour change identified by tenants.
Evidence was found for a change in engagement in energysaving behaviour over time, with higher reported engagement at mid-term than at baseline or final stages. This suggests that the intervention was effective in motivating positive behaviour change, but the effects were higher in the relatively short-term and diminished to the final stage. This indicates that the intervention did have a positive impact on engagement in energy-saving behaviour, and that the optimal time course for such interventions may be around three months (if we consider that the baseline survey was received in month 24, and the mid-term survey in month 27). In future interventions, it must be considered how to prolong these effects on behaviour in the longer term. One potential strategy may be to increase the salience of potential rewards associated with active engagement (i.e. longer-term reductions to energy bills). This may explain why the effects in the current intervention did not prevail over time with the same intensity, as most householders did not recognize the link between game play and real-life energy-saving behaviour. Continued interest in the game may have been reduced by this disparity, and by a perceived lack of longer-term rewards. By addressing this issue and making information about longer-term rewards more prominent and easier to access and visualize, future interventions may be able to motivate behaviour change beyond the three-month period identified in the current research.
EnerGAware is a cross-cutting project exploring the application of behavioral sciences to new energy efficiency strategies. The project was conceived as a concept testing to evaluate the use of a serious game to engage social housing tenants in proenvironmental behaviour for energy efficiency, before investing to make it available to the public. EnerGAware was the first project targeting an entire population of social housing tenants in a particular geographical location. Unfortunately and despite the effort s made by the project consortium, the final sample representativeness was not large enough (due to ethical concerns and lack of engagement, among others) to definitely ascertain the effects of the serious game. Therefore and although it may require a large amount of resources, future steps should exploit all available opportunities to replicate the pilot so as to gain stronger evidence of the game's impact regardless the geographic location and the social-cultural environment. Results obtained in the EnerGAware project show that the game may be worth under a multifactorial approach, where the game is used along with other more traditional behavioral change programs, to make the effect of the game prevail over time.

Cost-benefit opportunity
Despite the limited savings resulting from the game in one house, the solution has high scalability potential. This section examines the cost-benefit opportunity associated with future exploitation of this kind of game from a general perspective. The economic feasibility of a serious game reporting energy savings is depicted in Eq. (3) .
Where Ec i years are the net economic benefits provided by the game after i years, measured in €, e represents the increase in energy price, expressed as a percentage, and r is the discount rate or the expected rate of return, expressed as a percentage. Sav elec and Sav gas denote annual electricity and gas savings provided by the game, expressed in kWh/year and P elec and P gas represent electricity and gas prices for household consumers, expressed in €/kWh. N is the number of game downloads and C is the initial investment.
For the case of the Energy Cat serious game, the initial investment is considered to amount to €687,882, including costs incurred within the EnerGAware project that are directly related to game production (WP2 and WP3). Considering that the game can be played on a pre-existing tablet using a pre-existing router, allocated costs are negligible. The baseline energy consumption and energy savings are assumed to be the same as those found in the Plymouth pilot project ( Tables 1 and 2 ) and average European electricity and natural gas prices for household consumers for the second semester of 2017 [ 25 , 26 ] are considered. Calculations assume an annual increase in the energy prices of 2.1% and a discount rate of 3.0%. The economic benefit that the Energy Cat serious game can provide in 1 and 3 years are shown in Eqs. (4) and ( 5 ) The break-even point (number of downloads that make the game viable from an economic perspective assuming that all users become regular players) is estimated to be 14,502 downloads with a lifespan of 1 year. The break-even point is much lower when a 3-year lifespan is considered, at 4877 downloads.

Conclusions
This paper investigated the extent to which gamification can encourage energy efficient behaviour change and bring about energy savings in the residential sector through lessons learnt in the EU-funded Horizon 2020 project EnerGAware -Energy Game for Awareness of energy efficiency in social housing communities. For this purpose, an innovative serious game was developed to promote reductions in energy consumption and carbon emissions in social housing using Living Lab methodology. The game was implemented in a social housing pilot project and its effect was tested with a longitudinal, two-stage experimental design, employing both pre-post and control group approaches. Data for evaluating the energy-related impact of the serious game were collected using an energy monitoring system installed at the pilot homes for over two years. Further detailed information was gathered by sending a baseline and evaluation survey to all pilot homes, holding faceto-face interviews and focus groups with the tenants. At the end of the serious game intervention trial period and within the prepost comparison approach, the Energy Cat serious game was found to provide an average electricity saving of 3.46% and an average gas saving of 7.48%. This preliminary results were found not to be statistically significant but taking into account that an effect size was detected (0.2), future research should put the focus on trying to repeat the trial recruiting a bigger number of household energy consumers so as to assess the generalizability of the findings.
The mid-term analysis showed higher engagement in energysaving behaviours and higher energy savings than the long-term analysis. This suggests that the Energy Cat serious game was effective but that the impact was greater in the relative short-term and did not persist with the same intensity in the final stage. In order to make the effect of the game prevail over time, results obtained in the EnerGAware project show that the game should be implemented along with other energy efficient behavioral change programs.
Although it increased engagement in certain specific energy saving behaviours, the game was found to have limited impact on behaviour change, with small differences found across the experimental versus control conditions after the reporting period or over time. Nevertheless, several significant effects were found across conditions. Specifically, tenants had increased understanding of how their homes used energy over time and reported improved perceived affordability of energy bills over time. Engagement in the project was also found to be useful in educating subjects about fuel poverty. The results suggest that behaviour change may have been restricted due to a desire to maintain comfort levels and for health reasons. This may also be attributed to the fact that social housing tenants already use less energy than average [32] .
Game-based approaches promoting energy efficient behaviour hold a high replication potential, especially if the game is distributed freely by energy providers to energy customers as part of the European smart meter roll-out. Results obtained in this project point to numerous pathways for future research that will help to improve the persuasive potential of future serious game interventions, and thus to reduce energy demand in the social housing sector.

Declaration of Competing Interest
None.