IDEA: A software toolkit for energy awareness

ABSTRACT High electricity and water expenditure constitute a significant issue, as recent statistics demonstrate the continuous increase in energy consumption in the European Union (EU). This results in environmental, financial, and social issues. One such major issue is energy poverty. Software tools can influence users’ behavior, enabling them to improve certain situations and enhance their awareness and education around important topics. Such tools must also be usable and offer a positive user experience. In this context, a set of 15 innovative ICT tools were created under the Erasmus+ EU-funded project IDEA: “Innovative Direction in Energy Advising,” aiming, on the one hand, to assist individuals in decreasing their households’ electricity and water consumption through the development of energy awareness, and on the other hand, to help energy experts in their fight against energy poverty. Our previous work presented an overall brief view of the 15 ICT tools. In this work, we focus on the two core sets of tools. We present (i) the software engineering process we followed toward developing the tools, including a detailed needs analysis and important design and development aspects of the tools, as well as the final product; and (ii) a user evaluation from the perspectives of usability and user experience, during which quantitative and qualitative data were collected and analyzed, followed by a presentation of the analysis conducted and discussion of the results. We concluded that energy experts felt that the IDEA toolkit appears very useful in comparison to what they are currently using. In regards to the wider audience, participants’ experience satisfaction (Water Tool (WT) (93%), Lighting Tool (LT) (86%), and Heating Tool (HT) (81%)), and usefulness of the tools (WT (86%), LT (83%), HT (76%)) were highly rated.


Introduction
High electricity and water usage constitute a significant issue in the European Union (EU) for many years now, as there is a continuous increase in energy consumption. Electricity consumption has been increasing in the household sector since 1990, with an overall increase of 22.6% until 2004 1 and 16.5% from 2000 to 2018. 2 In 2019, the final energy consumption in the European Union, i.e., consumption by end-users, increased again at a new highest level. 3 At the same time, the EU struggles to reduce its energy consumption and reach its target goals. Final and primary energy consumption for the 27 member states of the EU from 2005 to 2019 and the indicative energy consumption targets for 2020 and 2030 are presented by the International Energy Agency. 4 A continuing growth of 2.1% per year to 2040 is demonstrated 4 . As consumption increases, electricity demand increases, leading CO2 emissions to reach a record high in 2018 4 . Adding on the above, household water usage grew by 600% since 1960. 5 Increased energy consumption results in environmental, financial, and social issues. Combining high electricity and water expenditure with low financial income and inefficient building material and individual non-energy expert users to decrease their household electricity and water consumption; and (ii) the "Energy-Advisor" toolkit, including 5 tools aiming at assisting energy experts and energy advisors in conducting household visits. In this paper, we present (i) a detailed needs analysis and results toward developing the tools; (ii) important design and development aspects of these tools; and (iii) a user evaluation from the perspectives of usability and user experience, during which quantitative and qualitative data were collected and analyzed, accompanied with a presentation of the analysis conducted and discussion of the results.
Structure Chapter 2 presents an overview of the related work on ICT tools for energy consumption and energy poverty. Then, Chapter 3, presents the software engineering steps followed for designing and developing the tools, including the needs analysis, the system architecture and the presentation of the final toolkit. Chapter 4, presents in detail the evaluation of the specific tools, via mixed methods: (i) a questionnaire survey, and (ii) a focus group session. The results are presented and discussed. The paper concludes with a discussion of the conclusions and envisioned future work in Chapter 5.

Related work
The topics of energy awareness and energy advising through educational material and digital tools, were also the subject of the IEE 7 projects ACHIEVE (ACHIEVE 2014) and REACH (REACH 2017). The material created and the tool developed in ACHIEVE, were focused on energy poverty and were aiming to help the energy advisors on their audits during household visits for calculating the energy savings. The tool was implemented in Microsoft Excel Spreadsheets, using simple functionality and formulas. Subsequently, in the project REACH the material and the tool were updated to better accommodate the advisors' needs, still remaining in Microsoft Excel. Nevertheless, both projects excluded non-energy expert individuals from the tool's direct target groups. Awareness raising of individuals in order to better comprehend their own consumption, as well as offering support to learn and implement measures on their own was neither included nor studied. Other shortcomings of these tools were: (i) absence of user-friendliness, (ii) they introduce complexity when using them for calculation of energy savings, and (iii) the inappropriate visual appearance. Compared with the current ICT solutions, the tools were outdated. Focused on the above, the IDEA project aimed to upgrade and update the tools, recreating them by using modern technologies, by receiving requirements from both energy experts and other individuals, as well as aiming to offer a better user experience. Nevertheless, as the underlying functionality of the ACHIEVE and REACH tools was deemed as useful, it was decided that the core functionality would be sustained with the necessary adjustments. Moreover, during the IDEA project lifetime it was shown that energy advising information and calculations would be valuable for individuals, i.e., non-energy experts, as well. Thus, IDEA aimed at designing, implementing and providing a comprehensive toolbox covering energy education, awareness and advising, for both energy experts and non-energy experts.

Existing ICT tools
During the initial activities of the project, a database of existing methods and practices for education and awareness about energy poverty and energy advising tools was created. We present the most relevant ICT tools below.
The "Social Electricity" project tool (Kamilaris, Pitsillides, and Fidas 2016), allows individuals to compare their current electricity consumption with the one they had in past years or months, or with their friends' and neighbors' consumption; to set up targets for energy saving and compare them; to analyze their consumption based on the electrical appliances of their houses; and, to gain a better understanding of their consumption through examples. The platform also provides educational games and material. "Local Energy Balances" tool, created by the Cyprus Energy Agency (CEA), demonstrates through graphs the energy balances for several types of energy including electricity, the type of usage including residential, and their correlation. The "ENERgy Retrofit FUNDing" (ENERFUND) Horizon2020 project (Geissler et al. 2019), developed a tool able to rate and score deep renovation opportunities based on a set of parameters such as Energy Performance Certificate (EPC) data, number of certified installers, governmental schemes running, etc. Through the visual interface, users can select a country in order to view a map on which the houses are shown in scaling colors depending on the above parameters. Users can also filter the presented houses via some parameters: energy rating, potential energy rating and construction area. This tool specifically targets energy poverty. The Agency for Sustainable Energy Development in Bulgaria (ASED BG) released the "Energy performance of Buildings" software based on the Energy Performance of Buildings directive, and the "Energy efficiency in households" ICT tool, for energy consumption assessment. The Energy Agency of Plovdiv (EAP) created the "CO2 emissions calculator" tool. The "EnerGbg" web platform shares information about: how can energy be used more efficiently by households; how can energy bills be reduced; how it is possible to reduce the release of harmful emissions into the atmosphere, while allowing users to share their own knowledge and opinions. As discussed above, the project REACH (REACH 2017), dealing with energy poverty issues and solutions produced the "REACH energy assessment tool," a comprehensive tool built in Microsoft Excel, for the assessment of energy and water consumption in households. The tool is used by energy advisors when implementing an energy audit in energy poor households. It was considered as a good starting point for development of a new, more user friendly tool. Another related tool, is the "FIESTA tool for auditors," incorporating the main energy features of a household, and with embedded formulas calculations it allows the rapid assessment of the energy performance of a building as well as of the heating and cooling appliances. The "GOLEA" e-educational materials include ICT based tools for education about renewable and energy efficiency. It has a special section of ICT games for various target groups, from small children to students and adults. It also includes tests and exercises. The "National Energy Path" Slovenia (NEP), created a web-based platform with information and measures for energy efficiency in homes. It includes a database of good (and bad) practice examples of practical implementation energy efficiency measures in Slovenian households. "Climate Literacy" provides tools focusing on the topic of climate change and related aspects. It consists of various educational modules on the topic, suggestions on how to include the modules in curricula for various target groups, and a mobile application. In addition the "Use Less" and "Let's make it happen, let's save energy" web-based applications calculates energy use in a household and potential savings in case of implementation of energy efficiency (EE) measures, and the "iEnergy" web-based interactive platform explains how the electricity system is working and how it is changing from traditional to a new, more decentralized system. Table 1 summarizes all the works recognized and discussed as relevant, along with their main target group (TG) out of: a. Non-Energy Experts (NEE); b. Energy Experts (including energy advisors) (EA); and c. All stakeholders (All), the URL on which the tool or the project is accessible, and the providing organization. In 7 cases, noted as 'Project' the tool is provided by the respective EU Project. All presented tools are provided in a free access manner. Even though there are many projects studying energy poverty from different perspectives, such as legal, political, and social, this study only presents works that delivered ICT tools. However, beyond ICT tools, there exist many educational materials in regards to energy poverty and consumption, such as in the EPOV platform in which more than 50 different educational material sets are hosted (EU Energy Poverty Observatory 2019). As mentioned, the shortcomings of the existing tools, the need for upgrading and for targeting also non-energy experts, as well as the need to focus on the actual requirements of all the users and combine them with the studies in the area to create suitable and useful tools, lead to the envisioning of the IDEA project ICT tools. Figure 1 visualizes the process followed in this work for the specification, design, development and evaluation of the IDEA ICT tools. For the design and development of the tools, the following software engineering process was applied: first a questionnaire survey was distributed to all relevant stakeholders to gather their opinion, needs and requirements, and the collected responses were analyzed. The analysis produced results, that were used as input to the next step of software engineering process, in order to draft the tools' specifications. Based on the specifications, the tools were designed and the designs were evaluated through a focus group with experts by utilizing their qualitative feedback. Then the tools were developed through an iterative process including different types of software testing, and continuous collaboration with the stakeholders, receiving feedback and re-iterating the development where needed. After completion of the tools' development, a case study was conducted from consortium members in order to test their functionality through a preliminary evaluation. We then publicly released the products. Finally, an evaluation process took place, including two methods: a questionnaire survey to collect quantitative feedback from non-energy experts and a focus group to collect qualitative feedback from energy experts and energy advisors.

IDEA tools
In this chapter, we unfold the software engineering steps toward the design and development of the tools including the needs analysis, the tools architecture, and the final product. Subsequently, in the next chapter we present the tools evaluation, and our results. Parts of the process explained above but not described in this paper, were presented in our previous work (Vanezi et al. 2020).

Needs analysis
The IDEA project performed initially a needs analysis and assessment. Via an evaluation questionnaire, the project executed the needs assessment for determining and addressing the requirements of users in terms of energy awareness and energy poverty. Gathering appropriate and sufficient data informs the process of developing an effective product that will address the group's needs and wants (Coastal Services Center 2012). Needs assessments are only effective when they are ends-focused and provide concrete evidence that can be used to determine which of the possible means-to-the-ends are most effective and efficient for achieving the desired results (Kaufman, Rojas, and Mayer 1993). In specific the target of the project was to identify the topics, approaches and tools and educational training sets that are important to key energy stakeholders. The data collection and data analysis provide the foundation and delivers concrete evidence for the actual needs of the key stakeholders (e.g., energy advisors, energy auditors, citizens).

Survey sample
The needs assessment research study presented in this paper was performed at four of the IDEA project partners countries (Cyprus, Bulgaria, Slovenia, Croatia) and involved 145 participants that responded to the developed questionnaire. The population targeted ( Figure 2) were energy experts, energy advisors, energy poor citizens, etc. The simple random sampling technique was used where from the 145 participants in the sample who had performed the survey, only 98 users have provided answers to all questions. The remainder (47) of the participants had missing responses (i.e., questions that have not been answered).
Therefore, from the total of stakeholders, N = 98 of them are valid for needs analysis and assessment. Moreover, the instrument was completed by different types of stakeholders as can be seen in the following graph, which provides a very good sample that takes into consideration any differences in terms of the views of various stakeholders. Due to the nature of the project, awareness on energy poverty, the focus of the survey was on energy advisers, energy agencies and citizens, i.e., nonenergy experts, that are the main beneficiaries of the topics, approaches, tools and educational training methods that were to be developed in the IDEA project. Nevertheless, additional relevant stakeholders were also reached for the survey.
Finally, the sampling error was calculated using the widely known sampling error formula. The result for confidence level of 95% (Z = 1.96) is as follows: = 1.96x0.682 = 9.9 = 0.135

Needs analysis process
The custom instrument 8 was defined in this work in five different languages: English, Bulgarian, Slovenian, Croatian and Greek in order for the questions to be clear to the participants and to receive optimal results. The custom instrument's items were separated into groups based on the high-level questions they were addressing: (i) energy awareness and knowledge about energy poverty, (ii) topics to learn about energy poverty and what measures to apply to alleviate energy poverty, (iii) tools considered to be of high value to the participants in terms of their involvement in energy poverty and tackling the problem through energy education and awareness, (iv) approaches and tools for energy education and for alleviating energy poverty and (v) beneficial practices for alleviating energy poverty and tackling the problem through energy education and awareness. The needs assessment methodology is based on two axes: (1) to identify the current level of knowledge and awareness on energy poverty and (2) to recognize and prioritize the topics, approaches, tools and educational training methods necessary to increase awareness. The responses were then transformed to a clean format appropriate for data analysis and statistical analysis and the complete set of responses, analysis and results can be found (anonymously) electronically. 9 The focus of this work though is on applications and tools, which were identified as two key components of the needs assessment, that are defined in two sets: (i) the set of 3 tools aimed at assisting individual households to decrease their electricity and water consumption; (ii) the set of 5 tools aimed at assisting energy experts in conducting household visits to assist individuals in this process of evaluating their consumption, gaining awareness, and taking measures to reduce it and thus counteract energy poverty and increasing energy awareness.

Reliability
Prior to presenting the results of the survey, a reliability analysis was executed on the data set using the Cronbach's Alpha measure. The alpha coefficient is a measure used to assess the reliability of any given measurement and refers to the extent to which it is a consistent measure of a concept. It is a measure of internal consistency, that is, how closely related a set of items are as a group. The coefficient of reliability ranges from 0 to 1 and coefficients that are less than 0.5 are usually unacceptable. In overall, the alpha coefficient is a measure to assess if the goal, designing a reliable instrument where scores on similar items are related (internally consistent), while for each to contribute some unique information as well. The reliability results for the custom instrument (k = 39 items), indicate a high reliability in terms of the design of the questionnaire (α = 0.916933304), but at the same time the reliability is not greater than 0.95 as this is commonly an indicator that the custom instrument's questions may be overlapping and thus be entirely redundant.

Needs analysis results
As presented in the above graph ( Figure 3), half of the participants are familiar with the term energy poverty, one-third has a basic idea about the problem, while 10% of the participants have heard the term and finally 4% have not heard about the problem at all. This provides an excellent basis for the needs assessment and analysis, since 86% of the survey participants are familiar with the problem and thus their opinion is valuable to the requirements gathering. The first set of items in the custom instrument were defined in order to identify the topics that the stakeholders believe that are more important for them in their effort to learn about energy poverty and what measures to apply to alleviate energy poverty. Figure 4 illustrates the opinions of participants on educational topics and measures. The graph clearly showcases that the participants primarily value (mean score = 4.49) having the capability to learn about the energy use of domestic devices and appliances, as well as practical training (mean score = 4.38) and measures and devices for reducing energy (mean score = 4.33) and detecting energy poverty (mean score = 4.26). These results indicate that participants are highly interested about practical measures, training and tools that can help them to get a greater understanding and receive immediate benefits in terms of energy awareness and reducing energy consumption. The above conclusion is supported by the results of the second set of items in the custom instrument ( Figure 5), which indicate that the participants believe that the online platform (mean score = 4.20) is a useful resource, as well as the database on energy poverty in households at the national and regional level (mean score = 3.93) and the statistics on relevant indicators of energy poverty (mean score = 3.91) provides valuable data to engage and tackle energy poverty. In particular, the results indicate once again that tools and data are considered to be of high value to the participants in terms of their involvement in energy poverty and tackling the problem through energy education and awareness. The next set of questions in the survey aims to further distill the opinions of the participants on the approaches and tools for energy education and for alleviating energy poverty. The results further support the fact that more practical approaches and tools are considered to be more valuable to the participants. As shown in Figure 6 practical activities are ranked higher (mean score = 4.47), as well as web-based tools (mean score = 3.77) and applications (mean score = 3.77) are next in the order of preference of the participants. In fact, web-based tools and applications can support practical activities, e.g., implement practical training in households.  Finally, the last set of items touches directly and aims to identify which practices are more beneficial for energy poor households. The results are also consistent with the fact that web tools and applications can be useful, since they can assist in educational purposes and measures for alleviating energy poverty. In specific, the three highest activities are: (i) enforcing practical measures for reducing energy and water use (mean score = 4.56), (ii) energy advising and auditing (mean score = 4.35) and (iii) training about measures they can implement themselves (mean score = 4.40).

Observations between groups
The needs assessment and analysis is concluded with the execution of a statistical test, the Welsh test. The t-test is a type of inferential statistic used to determine if there is a significant difference between the means of two groups. In specific, the statistical t-test considers two samples (N1 = 35 observations and N2 = 63 observations -as depicted also in Table 2) of unequal variances. This test is used to validate if the two groups, group 1 is using the tools and approaches (point iv) and group 2 may use the tools and approaches, share the same opinions. In the statistical analysis two groups have been taken into consideration: i) the first group is using directly the approaches and tools -e.g, energy advisers, energy agencies, social care institutions and ii) the second group can also use the approaches and tools -citizens, public authorities. The null hypothesis to be tested is that the difference of the means of the two groups are equal to zero (μ1 À μ2 ¼ 0), while the alternative hypothesis is that the means of the two groups are different (μ1 À μ2�0). Based on the results shown in Table 2, we cannot reject the null hypothesis because the p-value (0.39832111) is higher than the level of significance (0.05). This indicates that the two groups share the same opinions in terms of which tools and approaches are considered most appropriate for education about energy poverty and measures for alleviating energy poverty. This confirms the fact that both groups consider practical activities, web-based tools and applications to be the most appropriate ones for education about energy poverty and measures for alleviating energy poverty.  For the independent samples T-test, Hedges' g is determined by calculating the mean difference between your two groups, and then dividing the result by the pooled standard deviation, as follows: Hedges' g = (3.589569-3.457142) ⁄ 0.683751 = 0.193677. The literature suggested rule of thumb for interpreting results of the effect size, is Small effect = 0.2, Medium Effect = 0.5 and Large Effect = 0.8. In this study, the effect size can be categorized as small (Hedges' g = 0.193677), which indicates that the difference between the two groups is unimportant.

System specification
Based on the needs analysis results, the specifications of the envisioned ICT tools were drafted. Subsequently, a set of specific tools were designed. The tools' functionality was defined in order to cover several different parameters judged as important from the results of the needs analysis. Based on the results shown in Figure 6 that depicts the most appropriate approaches and tools, and Figure 7 that depicts beneficial practices, the following parameters (P) were chosen as most important to be covered by the ICT tools based on their ranking and their relevance to digital techologies: P1. Practical activities, approaches and tools; P2. Web-based tools; P3. Practical measures for reducing energy and water use; P4. Energy advising and auditing; P5. Training about measures they can implement themselves; and P6. Awareness raising. These parameters were set as top priority to be supported by the tools. Please note that, the IDEA project implemented other activities beyond the tools, e.g., webinars, to cover other important parameters. Additionally, two main categories of target groups (TG) are defined based on the needs analysis: TG1. Energy Advisors, and TG2. Citizens, i.e., defined as the non-energy expert individuals. Based on P2, online digital tools were planned. Based on the rest of the parameters, the IDEA Toolkit was designed, incorporating a basic core set of 8 tools: the "Open-Access" toolkit including the Water, Lighting, and Heating tools, and the "Energy-Advisor" toolkit including the Household Info, Water, Lighting, Heating, and Reporting tools. The two toolkits are similarly designed, and aim to support and accommodate the needs for P1, P3, and P6. The toolkits allow the users, either being citizens using the "Open-Access" toolkit or energy advisors using the respective toolkit, to record their potential device replacement, and obtain the calculated values for savings and costs occurring if these replacements take place. Replacements refer to lamps, taps and showerheads, and installation of insulation in doors and windows. The "Open-Access" toolkit accommodates additionally P5, and the "Energy-Advisor" toolkit P4. Furthermore, 7 additional supporting tools were developed, covering as a total the above discussed parameters and approaches: (1) the tips tool; (2), (3), (4): training tools; (5) energy wasting game; (6) advisors quiz; (7) national contacts tool. While our previous paper (Vanezi et al. 2020) provided an overall presentation of all the IDEA tools, in this paper we focus on the 8 core tools. Figure 8, presents the architecture of the "Energy-Advisor" and "Open-Access" toolkits.

System architecture
Citizens interact with the "Open-Access" toolkit, while Advisors interact with the "Energy-Advisor" toolkit. The toolkits offer interaction with their respective users via two distinct Graphical User Interfaces (GUIs). The GUIs are designed in a consistent way regarding the different tools within each toolkit. The "Open-Access" toolkit, includes the following three tools: (i) lighting Tool, (ii) heating Tool, and (iii) water Tool, while the "Energy-Advisor" toolkit includes: (i) the household info tool, (ii) the lighting tool, (iii) the heating tool, (iv), the water tool, and (v) the reporting tool. The two toolkits, receive as input a set of constant values, being either country-specific or non-country-specific, and the predefined calculation formulas. The "Energy-Advisor" toolkit, interacts with the database (DB) to receive and send household information. Finally, the calculations results are returned to the users, via the GUI.
The two toolkits were developed using web technologies: HTML and CSS for the structure of the GUI; PHP and JavaScript for the interactions and calculations; MySQL for the DB creation and queries. Figure 9 depicts the GUI of the water tool of the "Open-Access" toolkit. Through this, end-users create a "new shower" record, i.e., a row of text fields for entering values and reading results in regards to a potential replacement in the specific shower. End-users include information in the leftmost part of the row, such as defining the room that the shower is located in, providing information on the previous metered flow of that shower and stating the average time of showers in minutes. Then, by clicking the "Calculate Savings" button, the tool proceeds with computations and responds with values in the rightmost part of the row. Users can add as many rows for replacements needed, and then below, the total savings and costs are shown summing up all the replacement rows. The GUI is extendable in the sense that users may click the "Add another shower" link so that an additional row appears below the shown row. In addition, the water tool also includes a similar interface for the tap.

Open-Access toolkit
In a similar manner, the lighting tool allows the addition and savings/cost calculation of lamp replacements, and the heating tool allows the calculation of savings and cost for installing insulation in doors and windows. In the specific example of the water tool, examining the "Savings at the Shower" the room is pre-selected to "bathroom" as this would be the only option, and the same applied for the "Type of device" field. However, in the rest of the tools a list is provided in each of these two field for the user to select the room and the device that would be used for the replacement. A set of devices, e.g., lamps for the lighting tool, were selected by the energy experts and coded into the tool.

Energy-Advisor toolkit
In terms of the "Energy-Advisor" toolkit, the corresponding GUI for the water, lighting and heating tool is similar to the GUI presented in Figure 9, with the difference that, instead of adding replacement devices to compute the savings and display them to the users, the tool saves the inputs, i.e., typed values in the DB. The stored values are available for the experts to view and use on the spot while conducting a household visit, or in subsequent visits. Advisors can also access the data at any time, in order to study them. Once stored, the data from the water, lighting and heating tools are available to the reporting tool to retrieve them, conduct the computations and prepare reports, like the one shown in Figure 10.

Evaluation
The evaluation phase concentrated on the quantitative and qualitative data analysis with mixed methods.

Methodology
Data collection procedure was carried out by implementing an online questionnaire survey using multiple-choice questions (non-energy experts) and a focus group (experts) session, covering all the aspects of our research besides validating the main purpose of the paper. The experimental study aimed to evaluate the IDEA "Open-Access" Toolkit usability principles and to obtain an in-depth feedback from users' experience. The target group for this part of the evaluation were non-energy expert users. The survey was distributed by e-mail following the snowball sampling, recruiting 42 participants. It is a nonprobability method, which involves a random selection of the population. The data collection procedure continued in a semi-automatic and chain-like manner until data saturation, and guaranteed the participants' privacy concerns, anonymity, and confidentiality of the data. The questionnaires were constructed as 7-point Likert rating scales.
Users were asked to rate their interaction with the IDEA "Open-Access" toolkit from strongly disagree to strongly agree. There was a group of qualitative questions, where the participants could unwrap and express their opinion about the platform. The questionnaire was divided into four sections; the first was about the demographical data and the participants' awareness of electricity, heating, and water consumption reduction, which was completed before the interaction with the toolkit. The following sections were completed separately, with the accomplishment of each tool, electricity, heating, and water tool, respectively. Each section concerned a different tool, providing an in-detail scenario with the necessary information and guidance for users. Scenarios can be found in the link in footnote. 10 After the task's achievement, participants responded to the related questions, which remained the same for all the tools.
Quantitative data (multiple-choice questions) were analyzed via Excel data analysis software, providing descriptive statistics report and respectively. The analysis of the qualitative data was carried out through focus group discussions with energy experts and energy advisors. More specifically, considering the adoption of a user-centered design method to collect qualitative data, a focus group session was conducted with eight energy experts and energy advisors in total. An interview protocol was designed, consisting of five main discussion questions to assess their overall experience with the IDEA "Energy-Advisor" toolkit. The recorded qualitative data (focus groups) were transcribed and analyzed via content analysis, which followed the steps of analytic procedure (Polit and Beck 2018). Data were assigned to the categories and the findings derived and compiled. Direct quotations preserved the participants' voice.

Quantitative analysis
The quantitative data analysis began from the examined outcome derived before the experimental study started gathering some demographics and understanding whether there was any previous experience and knowledge on electricity, heating, and water consumption reduction.
The overall participation in the experimental study was 59,5% male and 40,5% female. According to Table 3, the majority of the users (n = 38) had the knowledge and were conscious of the energy consumption reduction. This can be seen from the high percentages collected (90.5%), which emerge their awareness and concerns about the energy consumption issues. In addition, most of the respondents (90.5%) answered positively when asked if they have considered saving energy in their household.
The second phase of the experimental study focused on the users' experience with the IDEA "Open-Access" toolkit and the three tools, water, lighting, and heating. Briefly, the participants were given instructions and guidance through a detailed description of the scenario and upon completion, they answered related questions. The evaluation was held based on the usability principles (Lund 2001) in a 7-point Likert scale, from strongly disagree to strongly agree. In our analysis we are using percentages. The users' overall experience, through the outcome, was positive. Table 3. Descriptive statistics of the pre-questionnaires.

Pre-Questionnaire
Have you ever been concerned about energy-saving issues?
Yes (n = 37-88.1%) No (n = 5-11.9%) Have you ever considered saving energyin your home/flat? Yes (n = 38-90.5%) No (n = 4-9.5%) Figure 11. Means and standard deviations (shown above bars) of participants' responses on usefulness, ease of use and ease of learning for the three tools. Figure 11 depicts the means (bars) and standard deviations (numbers above bars) of participants' responses on usefulness, ease of use, and ease of learning for the three tools. Ease of learning received the highest score and has the lowest standard deviations, while ease of use received the lowest score. However, the difference between the two is relatively small: from 6% (lighting tool) to 13% (heating tool).
As seen from Table 4, in the first group of the usability evaluation, "usefulness," the principle "is it useful" ranged in high percentages, with WT (86%), LT (83%), and HT (76%), with the water and lighting tools providing a slightly higher percentage. This concerns the "functionality" of the tools, how easy and pleasant the interaction was, with the users emphasized on the "utility," something they needed in the first place. Two more principles from the first group that showed high ratings were "productivity" and "efficiency." The "productivity" principle is based on the context in which it is used, and in combination with the "efficiency" that concerns the duration of performing a task, participants acquired positive experience through using the IDEA "Open-Access" toolkit.
The second group of usability principles, "ease of use," described how easily the users could navigate to find the targeted information in a contextual designed product, mapping the use of the most appropriate functionality to achieving the targeted goals. Following the outcome, users stood out that the water and lighting tools were simple to be used during the completion of the necessary values, with WT (79%), and LT (81%), showing, also, that the design of the tools allow users to recover quickly and easily from any errors that will arise throughout their navigation, with WT (81%), LT (79%), and HT (81%).
The last set of usability principles related to the "ease of learning" section, where users concluded on how quickly they learned, remembered, and became familiar with using all the tools. Furthermore, the principle of "ease of learning" is enhanced when an interface provides interaction cues, which imitate those the sample is familiar with. Consequently, through the high ratings results, with WT (74%), LT (81%), and HT (83%), shown that previous experiences and knowledge of the participants helped them to comprehend quickly how the functions were working, creating a very productive interaction.
Concluding the quantitative analysis, participants' satisfaction via their experience was highly rated, as shown from the data (Table 5), WT (93%), LT (86%), HT (81%). The water tool provided the highest percentage, which means that the tool's interface was "useful," "easy to use," and "easy to Table 4. Descriptive statistics of the users' experience with the water, lighting and heating tool based on usability principles.
Describe your experience with the IDEA "Open-Access" toolkit according to three tasks Water Tool (WT)

Satisfaction
Trust in "Open-Access" Toolkit Water Tool n = 39-93% n = 39-93% Lighting Tool n = 36-86% n = 37-88% Heating Tool n = 34-81% n = 34-81% learn." By setting the question if the whole procedure through the IDEA "Open-Access" Toolkit helps improve the individuals' awareness and trust in the tool for helping with energy savings and consuming reduction, it seems, from the statistics, with WT (93%), LT (88%), and HT (81%), that it has a positive impact and their intention of future use.

Qualitative analysis
As mentioned before, the experimental study was enriched with the implementation of focus group discussions for the qualitative data analysis, offering the opportunity to the participants to express their opinion and thoughts, gaining an overview of their experience. Participants were invited to a focus group interview that took place via Zoom conference call. An interview protocol was designed to obtain a full understanding of participants' experiences with, and perception of, the IDEA "Energy-Advisor" toolkit from the viewpoint of the energy advisor. Interviews took approximately an hour, were audio recorded and transcribed verbatim. At the end of the focus group, participants were also asked to provide a rating on two questions, one of which was constructed as on a 5-point Likert rating scale with multiple items. The participants assessed the items based on a Likert-type scale with 5 options for each item, from strongly disagree (1) to strongly agree (5).
Eight interviewees (5 females and 3 males) consented to participate in the focus group that took place in February 2021. Countries represented in the focus group included Bulgaria, Slovenia and Croatia. With regard to participants' educational level, six held a Master's degree, one held a PhD and one was a high-school graduate.
All participants were from the not-for-profit sector, with seven of them being employed on a full-time basis and one on a part-time basis. Three have been employed as an energy expert for a duration of less than 6 months, one for a duration of 2 years, three for longer than 5 years, while one participant did not answer. Other than two participants, who were within the age group of 36-45, six were within the age group of 25-35. Five participants had used the IDEA "Energy-Advisor" toolkit before, while the other three were aware of the toolkit and had even watched tutorial videos on it, despite not having used it.
The interview data were analyzed via thematic content analysis. A systematic and flexible method suitable for exploring energy experts and energy advisors' feedback about the platform were in line with the needs of stakeholders' and energy advisors', thus assessing qualities and characteristics such as usefulness, visual appearance, experience with using the tools and functionalities. An 'open coding' technique was employed primarily through Excel in order to code and track participants responses in relation to the previously mentioned themes within the qualitative data. Thus, the themes were identified through repeated readings and were ultimately classified according to the theoretical components established from the research (Zhang and Flynn 2020). During this process, distinct lines of argumentation were identified, revealing a number of themes significant for the discussion.

Energy advisor tasks during household visits and difficulties experienced
The first question of the focus group aimed to acquire a better understanding of what energy advisors actually do during a household visit and to what extent do they experience any technical difficulties with the current way of work. Moreover, the collection and handling of the data was also of particular interest.
Participants concurred that they conduct visits in a similar manner. An advisor is provided with a set of questions on paper that need to be answered during the visit. These relate to the household, building, and costs and bills for electric water and heating. To answer the questions, the energy advisor will use measuring devices to check devices, taps, windows, etc. In the case that it will be only one visit to the household they proceed to offer advice and to hand out devices e.g. led light bulbs, efficient showerheads, devices that generate savings. In the case that there is to be a follow-up visit they will collect the data with the paper and pencil form and insert it into an Excel tool once they return to the office. The tool will then calculate the savings and, in the follow-up visit, they will show the results to the household and hand out devices.
One participant mentioned that the paper and pencil approach is easy to learn; " . . . we trained students how to implement energy consulting to energy poverty households and it wasn't something difficult for them to monitor or to gain the information to present the objectives of the idea in the household, I don't think that there were difficulties for typing or writing the answers in the questionnaire." The overall views however pointed to two situations where the IDEA "Energy-Advisor" toolkit can make a difference and be an upgrade on the paper and pencil approach and excel tool; " . . . the excel tool is quite complex and not user-friendly, so a more user-friendly tool is very beneficial and useful. On a second note it is also possible to use the toolkit directly during the visits, which means that the data are gathered directly in the central database but on the other hand it might take longer to complete a visit since you have to provide input to many fields of the toolkit. Using paper will be quicker. With bad connection you might also lose data, but in a correct structure we might change from the paper form." This indicates that participants see the value in the IDEA toolkit and would be willing to adopt it under the right circumstances.

Experience with using the IDEA "Energy-Advisor" toolkit
The second question of the focus group aimed to acquire a better understanding of energy advisors current experience with the IDEA toolkit. Participants had taken this opportunity to point to areas for further improvement of the IDEA toolkit. One participant mentioned; " . . . it might be useful to have a list of all the households . . . ." This relates to the fact that they would like for an admin to be able to view data of all households. During the implementation stage of the IDEA "Energy-Advisor" toolkit however, for security purposes, it was agreed to enable energy advisors to only have access to view data of the households that they themselves visit. In a future upgrade the topic of admin access would thus need to be revisited.
Regarding the heating tool of the toolkit, one participant mentioned that an energy advisor should receive more explanation from the tool about how savings are actually achieved; " . . . there are no visible information on how the savings are achieved, it does not define what exactly is the measure." This could possibly be connected to educating energy advisors as well, so that they can understand also how the savings are achieved.
The impact that the IDEA "Energy-Advisor" toolkit can have has been noted by another participant, particularly on another project that they are working on. The participant felt that the IDEA toolkit can be used as one of the tools that they will be using in their local energy advising offices; " . . . this kind of tool will be good to gather information here and I want to present this in the project as one of the tools that we can really use in local offices where we will have internet connection. It will be good for our volunteers to have a really user friendly tool, to not have it all on paper . . . ." The potential of the IDEA "Energy-Advisor" toolkit surfaces in this discussion.

Functionalities
The third question of the focus group aimed to explore whether there are functionalities that could be added to expand the IDEA "Energy-Advisor" toolkit, or whether there are improvements that could be made to the existing toolkit.
One participant pointed to energy advisors being able to add new questions in the tools; " . . . possibility to add questions or groups of questions because energy advising is changing from one project to another, adding a lot of different questions to the questionnaire . . . ." Need for guidance on how to use the IDEA "Energy-Advisor" toolkit also emerged in the discussion; " . . . it would be good to have some written text on how to use it and what input to add to the fields for people using the tools for the first time . . . ." It should be stated that video tutorials are available, explaining how each individual tool within the IDEA "Energy-Advisor" toolkit is used. Participants agreed that even though these are indeed useful, sometimes an advisor doesn't want to watch videos but would prefer some quick guidance on the tool, especially novice users.
It was also understood that energy advisors feel it is important to be able to decide which sections of tools are relevant on a per project basis, since projects differ in terms of requirements; " . . . to be able to choose only the section of the tool that you will need e.g. for led lamps and water-saving taps and not use shower head and do not deal with other parts if not needed and change the preselected options in the drop-down menus . . . ." The participant further explained that it is normal practice to agree before a project starts what data will be collected. Hence, in terms of the IDEA "Energy-Advisor" toolkit, this would relate to deciding beforehand which sections of the tools will be used in a specific project, a result of the energy advising field changing. Moreover, to further support this change, there should be an option to add more questions if needed within the tools for a specific project. Another participant also supported this view; "advisors should have rights to access one project that has one set of questions and another project that has another set of questions as predefined." Another participant touched on the aspect of note taking. Energy advisors should be able to make notes or other observations. This could be supported by adding textboxes within the tools. Henceforth, notes could be taken on something that is unrelated or not included in any menu of the tool.
Considering further improvement, another participant suggested that in addition to the predefined values for light bulbs within the tool, it would be likewise useful for an energy advisor to be able to enter the related information for new light bulbs. This again has to do with the changes in the field and new light bulbs surfacing.

User-friendliness
The fourth question of the focus group aimed to examine the user-friendliness of the IDEA toolkit. It thus explored whether the IDEA "Energy-Advisor" toolkit improves on user-friendliness in comparison to some previous and existing tools (e.g., REACH), which had been deemed as not too user-friendly.
Participants agreed that the IDEA "Energy-Advisor" toolkit is indeed more user-friendly; " . . . yes it is better and more user-friendly . . . ." Yet, scope for further improvement was likewise pointed out by participants. At the same time, it is also understood amongst energy advisors that the complex nature of the topic, makes it difficult to improve upon the IDEA "Energy-Advisor" toolkit. For example, the formulas built into the tools, to calculate savings, should be able to be updated but it is complicated to update these formulas in general. Hence, more time would be needed to further improve on this specific aspect. The participant mentions: " . . . going from the excel tool to the IDEA toolkit looks far better . . ., " and continues; " . . . the other thing is the formulas, this thing stays the same, it would be beneficial to have an update but its hard as it is a complex tool . . . ." The COVID-19 pandemic also affected the testing of the IDEA "Energy-Advisor" toolkit in the field, which was acknowledged by the energy advisors. One participant mentions: "When we have a real situation we will have better testing of the tool."

Visual appearance
The fifth question of the focus group focused on the visual appearance of the IDEA "Energy-Advisor" toolkit and whether it improves their experience.
Participants expressed positive experiences regarding the design. One participant mention " . . . nice colors, they are comfortable and soothing . . ., " and suggested that it may be more engaging to energy advisors to be made aware of their progress; " . . . when you have different categories filling out (e.g. household info and lighting) have a percentage bar on how much information has been filled in, or feel like collecting points and the icons change color, from reddish to greenish . . . ." Another participant was curious about how the IDEA "Energy-Advisor" toolkit would appear in mobile and tablet displays; "it would be very practical if a person could login to the phone and have everything there on the phone it would be more readable." Participants were duly informed that the IDEA "Energy-Advisor" toolkit has a responsive design, hence can be used on different size displays.
The IDEA "Energy-Advisor" toolkit displays savings in Euros and one participant from Bulgaria noted that it would be more sensible to display the savings in local currency; " . . . prices in Euros but it would be better to change to the national currency, would be more user friendly for advisors and everyone who use it . . . " and continues; "currency and savings are in Euros, it would be better to have national currency." At the end, participants liked the design and felt that the IDEA "Energy-Advisor" toolkit is user-friendly.
In conclusion, participants indicated certain aspects for improvement. Beyond that, they also felt that the IDEA "Energy-Advisor" toolkit appears very useful in comparison to what they are using now. They especially see value in being able to easily share household information and results to different organizations, and believe that the IDEA "Energy-Advisor" toolkit has potential to deliver on this. In closing, participants were interested to know whether there will be further progress with the IDEA "Energy-Advisor" toolkit. Testing of the IDEA toolkit in the field, post COVID-19, and more dissemination of the toolkit was also expressed.
The data from the two questions that required participants to provide ratings for, which they had completed after the focus group discussion, reinforced the data from the focus group interviews concerning the usefulness of the IDEA "Energy-Advisor" toolkit and its potential to be a useful tool in the household energy advising space. It should be noted that seven out of the eight participants had submitted their ratings. Regarding the results, participants rated the following subscales for the IDEA "Energy-Advisor" Toolkit: i) making it easier to do their work at 4.28 out of 5; ii) enabling them to do their work more quickly at 4.28 out of 5; iii) improving their performance in doing their work at 4.28 out of 5; iv) increasing their productivity in doing their work at 4.28 out of 5; and v) enhancing their effectiveness in doing their work, also at 4.28 out of 5. With a rating of 4.28 for all of the items, there is evidence of correlation between their focus group answers and their ratings. Lastly, for the question on whether they believe that they would revisit the IDEA Toolkit regularly if it was available for use, five participants had replied that they would while two of them responded maybe.

Conclusions and future work
This paper has presented the work done under the Erasmus+ IDEA project. The project evolved around the notions of energy awareness and energy poverty, aiming to help in resolving the latter with the creation of a set of ICT tools. The purpose of the current work was to focus on 8 specific tools, present their architecture and conduct an extensive evaluation to examine their usability and user experience. Based on the evaluation results we concluded that energy experts felt that the IDEA toolkit appears very useful in comparison to tools they are currently using, and at the same time indicated certain aspects for improvement. The participants were interested to know whether there will be further progress with the IDEA toolkit. At the same time, in regards to the wider-audience participants' satisfaction via their experience was highly rated, as shown from the data (WT (93%), LT (86%), HT (81%)), while usefulness of the tools ranged in high percentages (WT (86%), LT (n = 83%), HT (n = 76%)).
As a future work we envision extending the tools with additional functionality based on the evaluation results. An admin role can be created for the "Energy-Advisor" toolkit, that will be in charge for a number of "advisors," to imitate the way energy advising agencies work. The admin of an agency will be able to view and edit all records created by the respective advisors. Furthermore, energy experts think it would be beneficial to be able to add new questions, and to select subsets of questions to be answered in specific households, based on the specific needs of a certain energy advising project. Moreover, energy advisors would be assigned to specific projects within the toolkit, and they would be able to access only those. The above are not supported at the moment, and in order to incorporate them, both the tools design as well as the developed modules, interface and DB would need to be adjusted accordingly. The current permissions scheme, limiting each user to access only their own records, would need to be extended. On the other hand, the infrastructure of the IDEA toolkit is built incorporating the potential to allow energy advisors to edit the set of constant values, e.g., kWh price in each country, however there is still no interface allowing them to do that directly. At the moment, the advisors can only edit such values by submitting a request to the technical team, which will edit the values within the DB. Future work includes the creation of such interfaces. The formulas would need to be updated, but first this needs to be done from the energy-experts perspective, and then to be incorporated into the tool. At the moment, the tool incorporates the latest version of calculation formulas. New devices should be able to be added to the existing lists, e.g., light bulbs, as this information is rapidly changing while new devices are appearing in the market. Such changes would be propagated to both toolkits and benefit all users, experts or not.
Additionally, other important functionality would be: for countries with a national currency, the calculations should be shown in accordance and not in Euro; a progress bar and different colors could accommodate the advisors in understanding their progress in completing the needed data; more information can be provided in each tool interface to aid the users. Testing of the IDEA toolkit in the field, post COVID-19, and more dissemination of the toolkit are also envisioned.
We should also mention, as a limitation, that we did not at this point estimate nor assess the actual effect on the energy consumption of households after using the IDEA tools. A future work could include studying the energy consumption and energy awareness of households before and after using them.

Notes
interests and expertise include Web-based Software Engineering, User Management Systems Development, Privacy in Software Systems, Data Protection in Software Systems, Privacy verification in Software Systems, Formal Methods in Software Engineering Methodologies and Privacy by Design in Software Development. Her skills include programming in several languages, and developing of web and mobile applications.
Thomas Photiadis (PhD) completed his PhD at the Cyprus University of Technology, at the Department of Multimedia and Graphic Arts, which was on Virtual Well-Being: Emotional and Behavioral Appraisal Patterns an Immersive User Experience". His Bachelor and MSc degrees were at the University of the Aegean at the Department of Product and Systems Design. His bachelor degree was in Product and Industrial Design Engineering, and the MSc degree focused on the building and promotion of Online Brand Identity using Human-Computer Interaction research methodologies. He is currently a Post-Doctoral Researcher at the University of Cyprus and a member of the Software Engineering and Internet Technologies (SEIT) Lab of the Computer Science department. During his PhD studies, he participated as a research collaborator or an operator in diverse research projects (a) research algorithmic media-critical data studies (University of Cyprus), (b) CYberSafety II (Pedagogical Institute of Cyprus), (c) European MediaCoach Initiative (Erasmus +) (European University Cyprus) (d) Cyprus Ports Authority (EU-funded MONALISA 2.0 project), (e) Eye tracking evaluation (Cyprus University of Technology) (f) Online gaming experience (Folkwang University of Arts (Essen-Germany)). His research interests focus on Digital Media, Media Literacy, Human-computer Interaction, User Experience, and Online Branding.