Designing an integrated socio-technical behaviour change system for energy saving

Stimulating households to save energy with behaviour change support systems is a challenge and an opportunity to support eﬀorts towards more sustainable energy consumption. The approaches developed so far, often do not consider the underlying behaviour change process in a systematic way, or do not provide a systematic linking of design elements to ﬁndings from behaviour change literature and the design of persuasive systems. This paper discusses the design and evaluation of a holistic socio-technical behaviour change system for energy saving that combines insights from behavioural theories and the persuasive system design in a systematic way. The ﬁndings from these two streams of research are combined into an integrated socio-technical model for informing the design of a behaviour change system for energy saving, which is then implemented in a concrete system design. The developed system combines smart meter data with interactive visualisations of energy consumption and energy saving impact, gamiﬁed incentive mechanisms, energy saving recommendations and attention triggers. The system design distinguishes between a version with non-personalized energy saving tips and a version with personalized recommendations that are deployed and evaluated separately. In this paper, we present the design and evaluation results of the non-personalized system in a real-world pilot. Obtained results indicate reduced energy consumption compared to a control group and a positive change in energy knowledge in the treatment group using the system, as well as positive user feedback about the suitability of the designed system to encourage energy saving.


Introduction
Meeting the European targets for a reduction of CO2 emissions by 2030 (40% compared to 1990) and energy savings (27% compared to "business-as-usual") [1] requires extensive changes in consumption behaviour of European citizens. Previous studies on the effect of behaviour change interventions on energy consumption report energy savings of 4%-12% on average [2], but also point out a number of limitations and issues related to the persistence of behaviour change over time. Overall, the current body of knowledge on determinants and processes of behaviour change for environmentally conscious behaviour [3] and research from the design of persuasive systems [4], provide good groundwork for designing systems to support behaviour change. However, the combined consideration of these findings in the development of technological solutions for stimulating energy saving has been rather limited and few approaches have systematically based their design on a theoretically-grounded model. In particular, none of the existing approaches and studies have been able to validate in a real world pilot a theoretically grounded design of a holistic behaviour change system for energy saving or provide specific recommendations for the design of such systems for various types of users.
While many approaches explore the use of consumption feedback or the use of game-like motivational elements, recent research both in energy and related fields (e.g. water saving) suggest that such individual elements alone are not suitable for inducing a durable change in behaviour (see [2,5] for an overview). Recently, several European projects have started systematically investigating models where different types of elements are combined in an integrated behaviour change system [6], although few results regarding their theoretical approach and empirically validated findings are currently published. In our view, energy consumption should be considered beyond an individual decision framework, as a complex socio-technical process that takes into account social norms, technologies and infrastructures. The demand for energy is therefore indirect, created by services such as comfort, which are in turn provided by devices and infrastructures [7,8], and is "systematically configured" over the long term [9]. When it comes to energy savings we propose that effective and sustainable behaviour change cannot be achieved by a single intervention impacting a specific attitudinal or behavioural variable, but requires a holistic socio-technical approach that uses a combination of individual enablers, mechanisms and techniques, and aligns technological enablers with suitable models of behaviour change. While such integrated socio-technical systems for behaviour change in energy saving are available in theory, they have not been validated in real-world pilots.
In this paper, we propose a theoretically-grounded design of a holistic sociotechnical system supporting behaviour change for energy saving, which combines smart meter data with interactive consumption visualizations, gamified incentives, energy saving recommendations and attention triggers. We present the theoretical background, derive the socio-technical integrated behaviour change model for energy saving and then describe the design and technical implementation of the model in a concrete system that provides different types of incentives (virtual, physical) and is adaptive to the various types of user motivations. Finally, we discuss the results of its evaluation in a real-world pilot including impact on energy consumption, analysis of user activity and evaluation with the end users.

Behavioural change systems for household energy saving
One stream of research on behaviour change with respect to energy savings considers behavioural theories that aim to identify antecedents of pro-environmental attitudes and behaviour and/or describe an associated behaviour change process [3]. Frequently cited behavioural theories exploring the adoption of new behaviours by users and applying them to energy saving include the Ajzen's [10] theory of planned behaviour and Schwartz's [11] norm activation model. Studies also recognize a wide range of factors determining individual household energy consumption behaviour, most frequently subdivided into socio-economic variables, psychological factors and external contextual and situational factors (see Frederiks [12] for a review). Although some factors have been found to be better predictors of energy savings than others, these findings are not consistent across time, context, and sample type of participants and studies [12]. Additionally, very few studies have found the interventions to be effective or achieve substantial behavioural changes [12], mainly due to the fact that they have not appropriately tested causal relationships [12].
Other research stream deals with the actual design of behaviour change support systems that can aid in inducing users to change behaviour, drawing on generic design frameworks ( [13], [4]. These systems have emerged in different domains (for a review, see [14]), including various kinds of pro-environmental behaviour, such as energy saving [15]. Most authors agree that effective designs of behaviour change systems in the energy domain should incorporate different types of feedback and analysis options (e.g. allowing users to access historical series of consumption data or provide them with social comparisons). Feedback as a consequential intervention tool helps individuals associate their actions with outcomes [16] and shapes behaviour by breaking consumption habits [17]. Such feedback can be data-oriented (e.g. bar or pie charts [18]), connected to the real consumption context (e.g. floor plans [18]), metaphorical [18], playful and ambient [19,20], or connected to nature or animal habitats (eco-visualization [21,22]). Energy consumption feedback holds the promise of reducing energy consumption by 4%-12% on average, with peak savings going beyond 20% [2,23]. However, feedback is regarded as a double-edged sword: some authors find that behaviour change is likely to be reversed once feedback is no longer provided [24]. Others found that feedback should be coupled with motivational techniques and energy saving advice [2].
To be effective, incentives must consider different consumer types and needs, be presented at the right moment and provide actionable suggestions tailored to a given user and context [5]. A new direction is the use of personalized recommendations employing machine learning techniques to identify different consumer classes, create user models and map appropriate actionable suggestions in a personalized manner [5]. While the combination of consumption visualization and feedback with recommendations is promising, little evidence is available regarding the most effective combinations of feedback, or how these behavioural changes can be sustained over time.
Gamification approaches and social strategies are often employed in an attempt to induce energy-efficient behaviour with various degrees of success [25,17]. By adding game-like elements to an otherwise pragmatic application, users may become more engaged with the application and thus more likely to change their behaviour [26]. Social rewards can be induced by allowing users to compare game achievements or energy saving achievements against other users (e.g. leaderboards reflecting users ranking [25]). As an alternative to competition, cooperative approaches supported through e.g. virtual social rewards [25] have also been explored. However, in most of these approaches, actual impact on energy use was not measured or unreported.
The majority of the described research has focused on influencing behavioural determinants (e.g. attitudes, behavioural intentions) to stimulate energy saving without considering the stages of behaviour change and how intervention strategies must be tailored to accommodate individuals as they fluctuate between these stages [27]. But there is growing awareness that interventions and incentives need to be provided based on an individual's stage in the behaviour change process [28]. While early attempts have been made to adopt this approach to feedback systems for natural resource consumption [28,26,5], its systematic implementation into a real-world system and related empirical evaluation has yet to be done. The approach and model that we propose in the next section is an attempt to design a holistic behaviour change model for energy saving that integrates the insights from the trans-theoretical model of behaviour change [29], its adaptation to pro-environmental behaviour by [30] and water saving [5], and the mapping of various incentive mechanisms (such as feedback and recommendations) to appropriate stages of the behaviour change process.

Socio-technical behaviour change model for energy saving
The underlying assumption of behaviour change systems [4] is that behaviour change can be initiated by stimulating psychological factors that influence specific behaviours [14]. Accordingly, a change in energy consumption behaviour can be stimulated through a combination of specific incentive and persuasion strategies that address underlying psychological determinants. However, impacting the attitudes and the intention to change behaviour may not lead to actual change in behaviour (intention-behaviour gap). Individuals also have to develop skills and strategies needed to implement new behaviors [30]. Thus, behavioural change can be seen as a transition through a time-ordered sequence of several stages, such as in the trans-theoretical model of behaviour change [29] or the model of action phases [31]. The trans-theoretical model of behaviour change, originally from the health domain, explains behaviour change through five consecutive phases spanning from raising awareness ('pre-contemplation') to creating new habits. Each of the stages follows distinct goals, and as a result, requires different motivational drivers. The trans-theoretical model has been applied to water saving [5] and a similar model of action phases, the MAP model, has been applied to studying environmentally harmful behaviours [30].
We model behaviour change as a multistage process using a slightly adapted transtheoretical model of behaviour change. Following [5], the first two phases, the preaction and action phases are merged into one renamed phase, "pre-contemplation," to reflect the limited cognitive effort usually invested in energy consumption behaviour. The process of behavior change is not always linear, as users can relapse to earlier stages. The resulting socio-technical behaviour change model is conceived to support all phases of the change process with an integrated incentive model for energy saving that combines different incentive mechanisms matching each phase. Although the original trans-theoretical model of behaviour change gives suggestions as to the motivational goals that need to be achieved in each stage, it does not specify in detail the social, cognitive, and affective factors and processes that impact the formation of attitudes and intentions in each of the stages [30]. But the knowledge about these factors is critical for developing the incentive mechanisms to impact the formation of attitudes and intentions in each stage. Therefore, building on [5] we first specify what motivates an individual in each stage to invest cognitive effort into thinking about the behaviour and potentially changing it by using theories from environmental and cognitive psychology. We then use the input from the design of persuasive systems research to identify which incentives need to be provided to the users to increase their motivation to save energy in each stage. The resulting behaviour change model for energy saving is given in Table 1. Norm activation theory [11]: personal norm activated when people become aware of harmful consequences their behaviour has for other people and the environment (awareness of consequences) and accept personal responsibility for that (ascription of responsibility). Social norm [32]: can influence attitudes, e.g. through activation of social disapproval • Feedback mechanisms (comparison of energy consumption to historical consumption, social comparison to other households) • Impact visualizations showing possible negative impact of own energy consumption.

Contemplation
-people aware of the need for change and ready to act, but situational factors can postpone energy saving behaviour -factual knowledge about energy saving and other information that can help "tip the balance in favor of change" is needed Theory of planned behaviour: activating feeling of behavioural control [33] increases likeliness for acting in new ways • Tips on how to save energy can activate feeling of behavioural control Goal setting theory [34]: setting specific goals increases persistence and belief in ability to complete a task • Goal setting to reduce consumption by a specific degree supports saving Goal framing theory [35]: user propensity to act can be stimulated by showing how hedonic values (e.g. comfort, enjoyment) need not be impacted by energy saving, or can be achieved with small comfort reduction (e.g. one minute shorter shower) • Comfort feedback visualization relating energy savings to the maintained comfort level can be used to demonstrate that energy saving can work while preserving comfort.
Action -take first actions and require continuous reinforcements against slipping back.
Self-efficacy theory [36]: a person's belief in their ability to perform and maintain a difficult behaviour supports action • Personalized recommendations to reduce energy consumption Coping planning [37]: ability to foresee ways to cope with scenarios that can hinder the realization of the action • Virtual and physical rewards to stimulate energy saving behaviour Reinforcement theory [38]: positive reinforcements (e.g. rewards) can stimulate people to keep acting in new ways • Motivational messages to indicate if users are on the right track to achieve their energy savings goal Monitoring -need to ensure that behaviour persists over time and people don't slip to old behaviour.
Recovery self-efficacy [37]: individual's confidence in being capable of resuming a difficult behaviour after a relapse has a positive influence on sustained change. Fogg's behaviour model [?]: to sustain change, sufficient motivation and ability need to exist after the intervention, and need to be triggered appropriately.
• Reminders and notifications can reinforce positive behaviours long after the intervention is over (e.g. sending reminders at regular intervals or specific occasions) • Card or board games as objects integrated with existing habits can serve as reminders in daily life

Integrated incentive model for household energy saving
Based on the presented socio-technical behaviour change model, we develop more detailed incentive elements for designing a concrete behaviour change system for stimulating household energy saving that addresses all stages of the behaviour change process. The designed elements include interactive energy consumption visualizations, energy saving tips, goal setting and gamified incentives (virtual, social, tangible) and are implemented in a concrete system.

Interactive feedback visualizations with goal-setting for energy saving
Multiple studies show that feedback has the potential of influencing underlying beliefs regarding energy consumption and attitudes towards energy saving [39,40,5] at different stages of the behaviour change process. But the design of effective feedback visualizations for energy saving is not trivial. Energy consumption behaviour is abstract, non-sensory and of low personal relevance to most individuals [41]. The abstract units of W and kWH are difficult for household consumers to understand [42]. Consumers have different environmental goals and values [35], as well as different needs regarding energy consumption feedback [43], at different stages of the behaviour change process [30]. To address these challenges, an adaptive metaphorbased visualization approach can be used, helping users interpret complex numerical, and abstract information (e.g. [18,44,45]).
To set and monitor the achievement of the energy saving goal for a current month, a battery metaphor (Figure 1a) is used. It communicates the notion of energy as a limited resource that should not be wasted. Users can set their monthly saving goal relative to their baseline consumption in the same month of the previous year (20% by default). The user can then monitor the achievement of the goal in real time: as energy is consumed, the battery depletes. Consumption alerts are provided as colorcoded normative messages, which display the user's progress in relation to the set goal (green motivational message when the user can still achieve their goal, orange warning message when the user is close to not meeting their goal, red message stating that the user cannot achieve their goal anymore). As consumption feedback should be actionable [46], tapping on the 'Learn how' button leads directly to the tips page. This visualization is mainly intended for users in the pre-contemplation phase, to raise awareness of negative behaviour, and users in the contemplation phase, to stimulate action through goal-setting and monitoring.  To show the effect of energy savings in a way related to everyday life, the impact visualizations display achieved energy savings using three metaphors drawn from goal framing theory [35]: monetary impact, environmental impact (CO2 emissions) and hedonic value. The monetary impact metaphor (Figure 1b) is depicted as a piggy bank, the environmental impact metaphor (Figure 1c) is represented by trees corresponding to saved CO2 emissions, and the hedonic value metaphor displays jars filled with balls and badges which are earned after a user saves energy (user's gamified achievements). As users have different goals for using energy feedback systems [43], they may be more motivated by a specific impact view. Therefore, the impact view is personalized based on people's main motivation for saving energy (solicited from a questionnaire). However, according to Goal Framing theory, individuals may have several competing motivations, therefore users are able switch between these three different visualizations. These visual metaphors are intended to target users in the pre-contemplation stage to illustrate the impact of energy saving, as well as users in the action stage to create a sense of achievement that motivates users to continue saving energy.
Raising awareness about the trade-off between energy saving and comfort and suggesting recommendations for energy saving actions that do not jeopardize the user's comfort level, is a key challenge to improve user attitudes and acceptance of behaviour change systems. For this, we use a simple visualization displaying the user's achieved savings alongside their measured comfort level (determined using temperature, humidity and luminance values) that was maintained during that month (see Figure 2a). If savings were achieved and the comfort level was maintained, values are presented in a green box (alternatively, a red box provides a warning). An initial assessment of the comfort level is inferred automatically by the system (using sensors installed in the users' homes) and is refined using explicit comfort feedback from the user. While some efforts have previously been made at visualizing comfort level for e.g. building managers in an office setting [47], the consideration and visualization of comfort for households is under-investigated and is a novel element of our approach. By showing that energy savings can be achieved with only a small reduction of comfort, the display of the comfort level alongside energy savings can induce the users in the contemplation phase towards energy savings (cf . Table 1). Finally, the detailed consumption view provides data-affine users [46] with a bar chart consumption visualization, allowing users to check their energy consumption at different time scales (e.g. weekly, monthly) and compare it against their historical average.

Energy saving tips and recommendations
Educating users on how to perform the desired behaviour is a well-known behavioural change strategy [4,48]. Providing concrete actionable suggestions strengthens the sense of responsibility [11], promotes self-efficacy and the perception of behavioural control [10], and can also directly strengthen the intention to implement the desired behaviour [30]. Offering recommendations for energy saving actions allows users to learn how to save energy [43]. Our system design foresees two types of recommendations (see Figure 2b): a list of generic energy savings tips and personalized recommendations (derived from energy behaviour data, user-generated data, and sensors). In each case, users are asked to commit to execute the recommended action ('Ok, will do'), state that they are already doing it, or indicate that it is irrelevant to them. Please note that the system design presented and evaluated in this paper considers only the non-personalized version.  Gamified incentives and rewards A review in [25] found that gamified applications can positively influence energy behaviour, stimulate learning, and improve the user experience. Game-based strategies often motivate users through extrinsic rewards, but the removal of these rewards can result in the dissipation of the desired energy saving behaviour [49]. However, motivation to allocate cognitive resources is not only driven by rational thoughts, but also by hedonic values [50]. Appealing to the hedonic values with playful interaction can motivate user engagement beyond external rewards. In our approach, users are motivated to perform recommended actions via two types of gamified rewards: symbolic (points, badges) and tangible (external prizes). These are reinforced by game-mechanics such as goal setting (meeting personal savings goals are rewarded with bonus points), competition (vouchers are awarded monthly to the best performing households) and social comparison (e.g. collecting points for performing energy saving actions improves a user's leaderboard rank). An example of the leaderboard is displayed in Figure 2c. These features are intended to support user engagement in the action phase and reduce the risk of relapse by stimulating the need for competition and/or the need to demonstrate one's achievements to others [51].

Notifications and attention triggering
Given the generally low involvement of users with energy consumption [41], and the competition with other applications, there is a risk that users lose interest over time [52]. Therefore, push notifications are sent to users' smartphones comprising tips for energy saving, warnings when they are close to missing their energy savings goal, announcing gamified achievements (e.g. earning a new badge), and reactivation messages when activity becomes low. Such notifications are especially helpful in the monitoring stage to re-activate users' attention (e.g. after relapse to old behaviour).
Repeatedly triggering user's attention to an energy saving application is a difficult challenge, since a balance must be struck between keeping users engaged without irritating them. Consequently, users can limit the number of notifications per week by choosing a low (maximum of 2 per week) or high frequency setting (maximum of 5 per week), or completely disable them.The timing and type of notifications to be sent to the user in this non-personalized system is determined in a fixed manner based on input obtained from user requirements workshops (without considering different user types, actual user presence at home, or their current activity).

System implementation
The technical implementation of the system is based on a layered architecture, in which each layer is designed to address a specific task. The system architecture comprises three layers: the data layer, the business process layer, and the consumer layer. The data layer is responsible for the data acquisition, pre-process and storage of the smart meters and indoor climate sensor readings; it also exposes services for the upper-level layer to access the consolidated data. The business process layer is responsible for the execution of the business logic; it exposes business services for the Consumer layer. In the non-personalized system version it comprises: • The Gamification Engine: a configurable gamification rule engine, designed to provide services to the Consumer layer, specifically to the end-user mobile application through a service API. The exposed services allow the user application to record user activity on the platform and assign points for completed tasks, assess energy saving goals, track user achievements, manage rewards, and notify users about relevant platform events. • Inference Engine: a machine learning component that processes sensor and consumption disaggregated data to infer indicators like user comfort level and household occupancy. • Notification Engine: a component that provides notification services to other Platform components. It schedules notifications pushed by other components and delivers them based on user-defined preferences; it exploits the Google Firebase Cloud Messaging service, for the delivery of mobile alerts. • Service Integration and Orchestration Component: this component synchronizes the execution of the processes that the different components require, and provides services for the communication between components. Finally, the consumer layer comprises the client-side application for publishing the services exposed by the Business Process Layer, and the mobile App for displaying the consumption and sensor data and for integrating the gamification services.
The end-user application is divided into two components: a web application with responsive web design, which handles the communication with the business layer and the rendering of the visualizations, as well as client applications for mobile devices (Android and iOS), which wraps the web application and enables access to the native features of the devices. The responsive web application was developed following a Model-Driven Engineering approach: models representing the organization of the interface, its content and the user interaction events were specified using the OMG Interaction Flow Modeling language (IFML) and WebRatio [53], a model-driven development environment based on IFML, which automatically transforms these models into executable code using industry standard frameworks, such as Struts, JSP, CSS and jQuery. The client wrapper applications are developed using the native tools for each mobile platform, Java for Android and Swift for iOS; they consist mainly of a WebView, i.e., a simple web browser whose behaviour can be customized, that enables access to the end-user web application, enriched with native functions such as notifications, safe storage of user data and passwords etc.

Research Methodology
The system was designed with a user-centered design process, starting from eliciting the requirements of the users in a series of workshops, developing mock-ups, testing them in online crowd tests, as well as prototyping before final system development.
To assess the proposed model and system design we evaluated behaviour change for a set of residential households in a real-world pilot in Switzerland 1 . To this end, we applied an experimental design with a treatment and a control group. The treatment group used the developed system, whereas the control group wasn't exposed to this intervention. People signed up on a self-interested basis to the control and the treatment groups. We then assessed changes in energy consumption between the treatment and the control group and the interaction of the treatment group with the system. Additionally, a survey-based approach has been applied to capture behaviour and awareness change due to the intervention. Specifically, we measure changes in attitudinal and behavioural measurements before and after exposure to the system. This includes changes in behaviour, knowledge and attitudes, as well as the user evaluation of the usefulness and motivational effect of the system.

Energy savings as a result of system usage Sample characteristics
The experiment in this pilot has been organized by selecting a treatment group and a control group. The treatment group has been composed of 66 self-selected households from the district of Contone in the municipality of Gambarogno located in the Ticino Canton (southern Switzerland). A focused control group has been composed of 34 households in the districts of Gerra, Magadino, Piazzogna, Quartino and Vira, all of them in the same municipality of Gambarogno. The members of this focused control group agreed to take part in a series of questionnaires during the whole project. For the purpose of the consumption evaluation, we also have access to consumption data of a much larger control group of users (558 households) located in the same district, who are neither taking part in the project nor answering the questionnaires. This larger group is used only for a statistical assessment of the energy saving results.
The structure of the treatment group comprises 50% families with kids, 31% couples, and 9% reporting themselves as single. The control group comprises 28% families, 47% couples, and 25% reporting as single. The treatment group lives in apartments (21%), terraced houses (18%) and single homes (62%), while the control group distribution is as follows: 6% live in apartments, 35% live in terraced houses, and 59% live in single homes. The average monthly consumption for the treatment group over the baseline period spanning May 2017 until April 2018 was 574.5 kWh per month. The average monthly consumption for the focused control group in the same period was 625.6 kWh per month, and 450 kWh per month for the larger control group.

Results on energy saving
The results in this section cover the treatment period for the non-personalized system spanning June 2018 until February 2019. Therefore, we compare the consumption of the treatment period (June 18 -Feb 19) against the same sub-period during the baseline period (June 17 -Feb 18). The following formula is used to compute the consumption percentage change: Where N is the total number of users, x i is the consumption of the i th user during the treatment period so far, while x i base is the consumption of the same user during the baseline period. We find that for the treatment group, the average consumption has decreased on average by 5.81% (standard deviation 18.81%). The same analysis for the focused control group shows that the average consumption has increased by 1.33% (standard deviation 15.22%) after removing an outlier that increased the consumption by 168%. The percentage variation in consumption of the treatment group users can be seen in Figure 3. Apart from two users who, respectively greatly increased and greatly decreased their consumption (users 86 and 87), the other users in general managed to reduce their consumption (positive increments mean savings).

Figure 3: Percentage variation in consumption of treatment group users
An F-test shows that the variance of the data in the focused control group and in the treatment group can be assumed to be similar (P (F ≤ f ) = 0.09) so we performed a two sample t-Test which returned non-significance in the difference of the two means, as reported in Table 2. We thus accept the hypothesis that the average savings in the focused control group and in the treatment group are similar. Given the limited size of the focused control group, we also compared the average percentage savings of the treatment group against the average percentage savings of the larger control group from the Contone population. In this case, the variances of the two samples do not pass the F-test, therefore we employ a Welch t-Test to compare the average savings in the two samples. The average percentage savings in the larger control group are equal to 0.2% with a variance equal to 6.4%, which confirms the stability of the consumption of this group of households as expected, given that no major exogenous factors have affected the energy consumption in Contone over the study period. The results presented in Table 3 show that we can reject the hypothesis that average savings between the larger control group in Contone and the treatment group are similar. We have then analyzed possible relationships between the usage of the system (user activity in the app) and the obtained energy savings. For this, we have gathered the average number of actions performed in the app per week. Actions can be diverse types of interactions within the app, such as checking current energy consumption, reviewing a tip, updating the monthly goal and so on. Figure 4 shows the average number of weekly actions on the y-axis with consumption variation on the x-axis. We have then performed a K-means clustering on the data, setting the number of clusters to 4. The result is depicted in Figure 4, where a group of users (coloured green) who display a rather intense weekly activity on the app can be noticed. These users do not necessarily show higher savings compared to other clusters, but their savings are consistently positive, which is not the case for the other clusters (e.g. the larger purple cluster has a positive energy saving on average, but contains a subset of members who actually increased their energy consumption).

Evaluation of system usage
In the treatment group, 66 users downloaded and registered to the mobile app during the first two months of the pilot. Among these users, 52 logged in more than once and were therefore considered to be active users. This subgroup's application usage data was the subject of further analysis. Figure 5 shows the frequency of new sign-ups per month, the number of active users each month (those who logged in more than once over the course of the pilot) and the number of users who logged in each month. From Figure 5 we see that a majority of active users (in a range of 54%-100% depending on the month) are logging in to the application at least once every month. This is a healthy level of user engagement.
To estimate how frequently the users interacted with the main visualizations and pages of the application, we calculated the monthly average frequency of interaction per user per month of pilot membership 2 . The results are shown in Figure 6. The results show that the users interacted the most with the Savings and Goal page which they accessed on average over 14 times a month. The second most frequently used page was the Comfort page, with an average of 10.7 accesses per month. Similarly, the Tips page (10 accesses per month) 3 and Achievements page (9.6 monthly accesses), were also frequently used. The Consumption page with the detailed time-based overview of the consumption and the Leaderboard were used less frequently, though still more than once a week on average. In contrast, the Impact and Comfort Feedback pages were accessed less, about once a week on average (3.9 monthly accesses).
To shed more light on the interaction with the three types of impact visualizations, we analyze how many users accessed the default impact visualizations in the application (determined from their stated motivation to save energy) and how many  of them changed the default impact visualization to another metaphor. Among the users who indicated their preference to save energy, 88% were motivated by environmental considerations and 6% each by monetary or hedonic goals. Results show that most users accessed the impact visualization reflecting their stated preference at least once; for example, 95% of users who cited the environment as their preferred motivation for saving energy accessed the environmental visualization. Among users who had the environmental visualization set as the default, 50% changed to the monetary and 55% changed to the hedonic visualization at least once.
Evaluation of the change in antecedents of behaviour change and user acceptance Measurement instruments and sample In order to assess differences in the antecedents of behaviour change, we measured three constructs with validated measurement instruments, adapted to the context of energy saving. This was done using 3 items measuring intention to save energy (7pt Likert) in line with Ajzen (2002), 4 items measuring perceived behavioural control (7pt Likert) in line with Thøgersen & Grønhøj (2010), and 5 items measuring environmental knowledge (5pt Likert), with an adapted scale from Kaiser and Frick (2002) 4 . These items were measured both for the treatment and the control group, with the main hypothesis that due to the intervention with the developed system, these behavioural determinants would improve in the treatment group more than in the control group.
To evaluate user acceptance of the system, we used the items from the UTAUT model [54]. Specifically, we used 2 items measuring performance expectancy, 4 items measuring effort expectancy, 3 items measuring hedonic motivation, and 3 items measuring behavioural intention, all on a 5pt Likert scale (see Table 5). We also measured the motivation of users by the system in general and by specific elements (such as the battery, and the environmental and monetary impact views) in particular (these items were self-developed).
For the analysis of the changes in behavioural variables, we can only include users who filled out both the baseline questionnaire (given at the beginning of the pilot), and the assessment questionnaire after the pilot had been active for 4 months. Therefore, the results presented below come from 40 users in the treatment group, of whom 66% are male and 34% are female; 25% are between 31-40 years of age; 28% between 41-50, 25% between 51-60, 15% between 61-70, and 5% are over 70. In the control group, there are 26 users, of whom 72% are male and 28% are female, 18% are between 31-40 years old, 23% between 41-50%, 23% between 51-60, 14% between 61-70 and 23% are over 70.

Results of impact on antecedents of behaviour change
We measured three antecedents of behaviour change: energy knowledge, perceived behavioural control and behavioural intention. For behavioural control and behavioural intention, we computed averages for the items mentioned in the previous section, whereas for the energy knowledge we computed the sum, as the construct is formative. These constructs were measured at the beginning of the trial and after four months 5 in order to assess any change resulting from the intervention. In Table 4 we show the means and standard deviations of the change in the behavioural determinants computed by subtracting the score after four months of trial from the score measured prior to the beginning of the trial. In this table the Shapiro-Wilk test, used for small sample sizes, [55] reveals that all of the scores are normally distributed, except for the scores of energy knowledge in the intervention group.
Scores measuring energy knowledge, show a change from 14.2 to 16.8 in the treatment, and from 18.30 to 16.81 in the control groups. The scores provided by participants in the treatment group have increased, whereas the scores of the control group have decreased. As the scores of the treatment group are non-normally distributed, a Mann-Whitney test shows that people in the treatment group (Mdn=1) managed to increase their knowledge compared to the control group (Mdn=-0.5); U=735, p<0.01, r=0.35, which is considered a medium size effect [56].
As evident from Table 4, other behavioural determinants measured were not substantial; the independent samples t-test revealed that there are no significant differences in the mean changes in perceived behavioural intention between the treatment and the control group (t (64)=1.18, n.s.) as well as no differences in behavioural intention (t (64)= -1.33, n.s).

Results regarding user acceptance and motivation
The results in Table 5 reveal that the application shows above average user acceptance for all measured constructs of UTAUT model: the application was considered moderately useful in daily life (average performance expectancy score of 3.6/5) and easy to use (average effort expectancy score of 3.75/5). Users also reported that the application is fairly fun to use (average score of 3.4/5 for hedonic quality) and that they intend to use it in the future (average score of 3.7/5). The average scores of user responses for the evaluation of the developed visualizations are presented in Figure 7. They show that users evaluated positively all aspects of the developed visualizations.

Discussion and conclusions
We have presented a socio-technical model and system for behaviour change for energy saving and evaluated it in a real-world pilot. The results suggest that the designed behavior change system is suitable to stimulate energy savings among consumers. The observed positive effects on energy consumption of the treatment compared to the larger control group and a change in user's energy related knowledge suggest that such systems can be successfully applied to motivate users to save Our results show that users who used the application frequently, achieved positive energy savings, compared to users who used the application less regularly. Moreover, the positive user evaluation of the system, the high motivation of users by the developed visualizations to save energy, and the actual active usage of the application suggest that the integrated incentive model motivated users to adopt the proposed system to start changing their behaviour. These results provide both theoretical and practical contributions.
On the theoretical side, we find that the underlying socio-technical model can be applied to study behavior change in the energy saving domain. Research suggests energy knowledge is one of the first antecedents of behaviour change towards energy saving and our results show energy knowledge improving in the treatment group after users participated for four months in the pilot. Proceeding behavioural determinants such as perceived behavioural control or behavioural intention, have yet to show significant change. This suggests that four months is not enough time for the behavioural determinants to change and a post-hoc analysis must be performed to assess whether these determinants show any change in the long-term.
Additionally, our results suggest that the developed interactive visualizations are well suited to motivate users to save energy. Specifically, the battery visualization allowing the user to set and monitor the achievement of the energy saving goal was reported by the users as easiest to understand and most useful. The environmental impact visualization had the most visual appeal and was perceived as the most motivating metaphorical visualization for saving energy. While most users were interested in the visualizations matching their primary motivation, some were also interested in the visualizations matching the other motivational drivers. This confirms that users might have different motivations for energy saving, as suggested by the goal framing theory [35].
Additionally, during the four month trial users were increasingly using the application, motivated by the incentive mechanisms proposed in our model. Specifically, users have been most frequently accessing such pages of the application as: the savings and goal, the comfort, the tips, and achievements (ordered by frequency). The interest in the savings and goal page is justified as this page allows users to monitor their consumption with respect to the energy saving goal -the main aim of the application. The lower frequency of access for Impact visualization pages is surprising, although one needs to consider that accessing these pages also requires more clicks since they follow after the higher-level visualization pages.
On the practical side, the developed incentive model can guide the designers of behaviour change systems in energy use and other sustainable domains to design their systems in such a way that they are easy to use, helpful and engaging for the users. For example, we show that the energy consumption and savings visualizations help users save energy, by e.g. allowing them to set and monitor their energy saving goals. Additionally, providing users with energy saving tips and competition-based achievements can help keep users engaged in the behaviour change process. An overall recommendation for designers is to use different incentive mechanisms (those for different behavior change stages or various motivation types) that can potentially motivate various types of users.

1.
Here we report findings from evaluation of the non-personalized system in the Swiss pilot. The enCOMPASS project includes another version of the system with personalized recommendations and notifications that has been released following the evaluation of the non-personalized version, and is currently being tested. A comparison between the two versions (non-personalized vs. personalized) will be subject of a separate analysis and publication. In addition, two more pilots to evaluate external validity through application in different climatic and socio-cultural contexts are being performed in Greece and Germany. This will also be subject of a separate analysis.
2. The frequency was computed by dividing for each user the total frequency by the number of days between sign-up and the 28.2.19, multiplied by the average number of days in the months in the pilot.
3. Calculated by adding the number of times a user read the tips and the number of tips read.

4.
The limit in the number of constructs measured is due to the need to cover different evaluation aspects and a limit in the number of questions that users would be willing to answer 5. The evaluation of the presented non-personalized system was performed as planned after four months of usage. Due to the delay of the subsequent trial of the personalized system, we were able to collect two more months of data on the usage of the non-personalized system and associated consumption.

Limitations
Limited duration of the trial of 4 months bears the risk that after the intervention the users may relapse. Verifying if such short period is sufficient to change behaviour in the long-term is the subject of a follow-up study. Overall, quite small sample sizes in the intervention and control group impact the significance of behaviour change determinants. Self-selection bias of participants to participate in the trial might have impacted the results. To show causality, a longitudinal study is necessary.