Keywords

1 Introduction

User Experience (UX) is a term that is appearing more and more frequently in the IT industry [1, 21]. Today, the preceding concept of usability, which emphasizes mainly pragmatic aspects of IT products, is to a large extent being taken for granted by contemporary users. To offer more, IT companies have started to focus on products that are both practical tools and deliver interesting experiences for the users [21]. Introducing UX work in the software industry turns the attention towards the user’s thoughts and feelings before, during and after using a product. Thus design and evaluation of UX is about capturing and predicting these thoughts and feelings, including the study of e.g. emotions, preferences, brand image, functionality, attitudes, and context of use [10].

An increasing number of researchers create new UX design and evaluation methods and evaluate qualities of existing methods. Some of these researchers depart from the UX challenges that the IT industry is facing. Unfortunately, only few of them also conduct research on adoption and use of UX methods in that industry [1, 5, 15, 18]. Accordingly, it is exceedingly difficult for IT companies to move from being inspired by UX methods to adoption of these methods in their development practices.

Although the IT industry is turning its focus towards UX, it is far from all IT development companies that have already adopted UX work, and some of those that have, have not embraced it as intended by UX research [1]. Many IT companies are motivated to adopt UX work in their development processes, in order to enhance the user experience of their products and make them more appealing, but they are missing guidance on strategies for carrying UX adoption through.

This paper describes an action research study focusing on adoption of UX and UX evaluation methods in an IT company. The aim of adopting UX in the company was to exceed a traditional approach to design of banking applications. The action research study deals with a specific project in the company where UX was introduced to stimulate design of more interesting services in a banking system to manage loans with a smartphone. The study involved collaboration over several months between the researchers and a group of developers with no prior knowledge of UX. The study demonstrates how UX methods can be introduced in an IT organisation and how UX decisions can be prioritised in the software industry.

The following section presents related work, including existing qualitative research about UX and usability in the IT industry. Then the framework of our empirical study is presented, followed by the description of method of the study. Next, the findings from the study are presented, followed by a discussion that compares our results to existing studies of UX in industry. Finally, we provide our conclusion on the study.

2 Related Work

Research investigations of UX work, including UX evaluation in companies are limited. Only few articles have made qualitative research about the adoption of UX in companies [1, 5, 18]. This section outlines some of the inquiries, and because of their limited number, also a study of usability work in industry. In the description, we emphasize the setup and methods of the investigations.

2.1 Study of Usability Evaluation in Industry

Since UX can be seen as related to usability, studies about usability in industry can be used as inspiration to conducting UX studies in industry. A good example of a qualitative process-oriented study about usability in industry is Nørgaard and Hornbæk [16]. They conducted an explorative study of 14 think-aloud sessions in order to investigate what usability evaluators do in practice. The researchers used observations to investigate seven companies and their use of test sessions in a usability evaluation. The study resulted in an overall picture of how companies are working with usability test sessions as they did not investigate one company in depth.

2.2 Definitions of UX

There is still no agreed upon definition of UX [11], but various academic papers have been given their take on how to define UX. Hassenzahl [6] describe that people perceive interactive products along two different dimensions: Pragmatic quality, referring to the functional usefulness of the product, and hedonic quality, referring to the feelings associated with the product. This definition has been developed to: “A consequence of a user’s internal state, the characteristics of the designed system and the context (or the environment) within which the interaction occurs” [7].

Law et al. [12] took a different approach and conducted a survey with 275 researchers and practitioners in search of a common UX definition. The general understanding was that UX is “dynamic, context-dependent, and subjective, stemming from a broad range of potential benefits users may derive from a product.” Later, it was added that UX is dynamic over time, and that feelings towards a product may be different during and after the interaction with it [20].

ISO 9240-2010 [10] aims to provide a common definition with a main definition and three notes with descriptions of the different dimensions of UX: “Person’s perceptions and responses resulting from the use and/or anticipated use of a product, system or service”. The notes add that UX is about emotions, psychological responses, behaviors, brand image, the user’s prior experiences, context of use, and that UX occurs before, during, and after use.

2.3 Study of Adoption of UX in Industry

In order to investigate the existing qualitative research about UX in industry, an experimental study was conducted in 2014 with a collaboration between researchers and software companies [1]. The purpose was to investigate how to adopt UX in practice and several research methods were used, including an explorative study with one company. In the explorative study, several activities were carried out, e.g. team meetings, observations, interviews, and heuristic inspections. The study suggests using empirical research to reduce the gap between the research and industry use of UX. They found collaboration studies with researchers and practitioners instrumental in showing practitioners why to improve their development processes and how to do so.

An exploratory case study from 2015 had the goal of investigating how UX knowledge can be obtained and shared within a company [5]. Three case studies were conducted with three companies, who already had integrated UX in their development process. These studies included six interviews with the designers from the companies. Different methods to spread UX knowledge within a company were analysed and these methods were centred on UX competence flow between individual UX designers and the company in which they worked.

Another case study from 2012 investigated Microsoft and their practice behind UX management [18]. The study was conducted in situ at Microsoft where several interviews were conducted with seven managers from research and development departments. Since Microsoft has already integrated UX in their development process, this study describes a successful UX integration in a company.

The above mentioned papers about UX describe qualitative studies of integration of UX in industry, but they do not work specifically with UX evaluation, are based on interviews with other actors, provide only general and short descriptions of the different cases, and two of them investigate companies where UX is already integrated and do not address how UX can be adopted.

3 Framework

The theoretical framework of our action research study is adopted from Mathiassen [13] as our purpose was to “understand, support, and improve practice as part of the ongoing professional development”. The concept of UX used in this study is ISO’s definition (see Sect. 2.2).

We selected methods for our study based on the following criteria: they should be easy to use for practitioners with little to no prior UX experience, the materials and equipment required should be minimal, they should represent different approaches to evaluation of UX, they should include both expert and user based evaluation, and they should facilitate collection of both quantitative and qualitative data. A previous study identified 96 UX evaluation methods [21]. Based on this survey and our criteria, we chose the following UX evaluation methods: SUXES [19], Product Reaction Cards [4], AttrakDiff [22] and UX Heuristic Inspection [17].

4 Method

The purpose of our empirical study was to study support of adoption of UX work in an IT organisation. We decided to base it on an action research (AR) approach. Action research was pioneered by Lewin [18] who used it in his study of field theory in the social sciences. Action research has often been used in information system research to study changes in an organisation. Thus it may be used to study adoption of new concepts in a social setting. However, in HCI this research approach is rarely used and only few guidelines and cases for inspiration are available [10]. In the following, we present the method of our AR study.

4.1 Research Approach

AR studies are a duality of action and research; practice and theory. They are working towards producing new knowledge through the creation of solutions and/or improvements to “real-life” practical problems [14]. AR has been used previously in HCI [8] but is more prevalent in information systems research [9].

We have used [14], where AR is described as “two, interlinked cycles”, one of practical problem solving and the other with production of scientific knowledge (see Fig. 1). The Problem Solving cycle aims to help our collaborating company in regards to using UX and UX evaluation in their IT-system development, and the Research cycle aims to investigate how UX evaluation can be adopted in the IT-industry.

Fig. 1.
figure 1

The AR of this paper viewed as a dual cycle process. Consist of both a problem solving interest cycle and a research interest cycle. Adapted from McKay and Marshall [14].

Full size image

In AR the cycle continues with no definite ending point, but the goal is to enable the company to maintain the positive changes that have been made, once the researchers leave the company [5]. In our case, the exit points from both cycles in Fig. 1 were based on a predefined deadline.

The limited time influenced our criterion of success. Ultimately, the adoption process is successful if the methods adopted are used competently and effectively. However, we did not expect that to be realistic because of the limited amount of time allocated to the study. Instead, we saw the adoption as successful if the participants could use the techniques we introduced and intended to keep using them in the future.

4.2 Research Practice

The AR study was carried out in cooperation with IT developers from the IT department of a Danish bank. The IT department had recently shifted their development method to a user focused approach.

The company requested us to support them in adopting methods and techniques that were relevant to their new focus. We offered to conduct an AR study where we supported adoption of UX methods that could enhance their user focus and help them develop more interesting services. The purpose was to help them solve the challenges they were experiencing with selection and adoption of relevant UX methods. This was accepted by the company.

In cooperation with the company, we chose a new project where the purpose was to develop a banking system to manage loans with a smartphone.

The development team of 10, which we have been working with, consisted of UX designers, a method expert, a product manager, bank domain experts, business architects and software developers. In the collaboration, we were mostly in contact with the following key development team members: P1; our contact person in the company who also had the role of method consultant and UX specialist, P2; Project manager, P3; the chief specialist who worked with the design of the new system including UX, P4; senior specialist who worked with both the design and technical aspects of the system.

The AR study lasted 6 months, using 98 company employee hours. During this period, idea creation and the construction and evaluation of prototypes were done. Table 1 provides a timetable of relevant AR activities conducted in regards to the Problem Solving Interests Cycle. Not included in Table 1 are the weekly design workshops the development team held. We did not attend these meetings but the findings were reported back to us.

Table 1. Action Research in regards to the key steps in Problem Solving Interests Cycle. Showing the activities conducted in a project in the IT-department of a Danish bank company.
Full size table

Initiating meeting (activity 1).

The first meeting with the company was held at the end of 2014. The meeting was conducted with P1. The purpose of the meeting was to gain information about the new HCI software development method, created by P1, as well as getting information about the new project in the company. Furthermore, we identified the main problems in regards to the adoption and work with UX evaluations, and discussed how we could contribute to the company’s new project. We also collected information about the company’s expectations from our collaboration. Team meetings (activities 2, 3, 4, 6, 7, 8, 10, 12).

Throughout the collaboration with the company, we participated in team video meetings consisting primarily of the key members (P1, P2, P3, P4). The meetings lasted between 20 min to 1 h and took place three to four times per month. In each meeting, the team members presented the status of the project. We then presented our suggestions for current and future UX integration and used these meetings to educate team members on UX topics. These topics consisted of presentation of the UX term in regards to the ISO definition as well as when, how, and why the company could benefit from UX evaluation methods. In the later stages of the collaboration, the meetings functioned as a means to evaluate the effect of our UX adoption.

Workshop - early UX evaluation (activity 5).

In this workshop we were invited as UX consultants and our role was to educate the whole team about UX as well as how they could evaluate UX in the concept design phase. The presented evaluation methods were modified so they could function as developer tests, meaning that the developers could evaluate their own system. The workshop lasted one day and was divided into several activities, designed to expose the team to UX methods.

The agenda for the workshop was: The workshop started with a 10 min presentation of UX. We then asked the team members to brainstorm potential UX strengths and weaknesses of the new system. The same activity was conducted at the end of the workshop and functioned as an evaluation of our influence on their UX knowledge.

After the brainstorm, the team members were divided into two groups (5 each). Each group was presented with a UX evaluation method (SUXES [19] or Product Reaction Cards [4]). None of the team members were familiar with the presented evaluation methods prior to the workshop. Both methods were modified to be a ‘design brainstorming tool’ and an evaluation method that they could try on their own product. They were modified on the different UX parameters being measured (such as fun, effective, complex, and sophisticated). We wanted to make sure that the methods presented relevant parameters to measure for the new banking system. SUXES provides little information on which UX parameters should be measured, while Product Reaction Card provides 118 different UX related words, which we reduced.

The SUXES method and the parameters were also first used as a ‘design brainstorming tool’ by the SUXES team, where they first chose which parameters (out of a selection of 20 words, e.g. fun, sophisticated) they wanted their new banking system to represent. Afterwards in a scale from 1–7 (as the SUXES method also suggest), they decided what they expected the different parameters to score. After a skectching session with the parameters in mind, the Product Reaction Card team used the SUXES methods to evaluate the design from the SUXES group.

After the method presentations, each group used their method as a tool for choosing which UX dimensions will be focused on in the new project. Afterwards they brainstormed ideas based on the chosen UX dimensions and lastly one group member from each team was used as a test person and conducted a UX evaluation based on the other group’s method.

The workshop ended with the development team evaluating the workshop and the methods they had been presented.

UX evaluations (activities 9, 10, 11).

As a way to show the team which kind of results you can get from an UX evaluation, we conducted both expert evaluations (UX Heuristic Inspection [17]) with HCI experts, and user evaluations (AttrakDiff [22]) with potential end-users. The evaluations were conducted in a lab and were videotaped for the development team to see how the evaluations were conducted, how the test moderator controlled the evaluations, and to hear participants’ comments. To show the diversity of UX evaluation methods, we evaluated both a paper prototype and a functional prototype on a smartphone. The prototypes were a result of the previous workshops conducted in the development team (e.g. activity 5) and were modeled to fit the size of a smartphone. This was to explore how and what one can gain from evaluating UX of different stage of development. Since the company was making a prototype which should function on a smartphone, we designed a wooden smartphone-mockup to make the evaluation of the paper prototype more realistic (Fig. 2). This made it possible to evaluate the actual screen size elements, to use the swipe function, and to change between different screenshots.

Fig. 2.
figure 2

The wooden smartphone prototype designed to test paper prototypes.

Full size image

The results from the evaluations were analyzed and presented to three of the team members (P1, P3, P4) in a video meeting where results were discussed as well as how possible design changes could address the negative comments from the experts and users in the UX evaluations. Lastly, we gave comments about the potential future uses of the UX evaluations in the company.

Last team meeting (activity 12).

The last team meeting was conducted as a final evaluation of our collaboration. Here, two of the key members of the development team (P3, P4) evaluated our adoption of UX and UX evaluation methods as well as giving a report on which promoted material has been adopted in the company so far. Since not all key members were able to participate in the meeting, we also conducted a survey, which all employees in the development team answered.

4.3 Data Collection and Analysis

Throughout the AR study, we collected data from the different meetings with the company through both audio records (in total = 5 h 33 min.) and notes taken during the meetings (36 p.). Furthermore, we used three surveys throughout the collaboration:

  • Survey 1: Conducted before activity 5 to collect information about the development team’s knowledge about UX.

  • Survey 2: Conducted the day after activity 5 to collect information about the changes in the development team’s knowledge about UX and opinion about the workshop.

  • Survey 3: Conducted before activity 12 in order to gather information from the development team about the overall opinion on the collaboration, the adopted UX material, as well as the status on the adoption of the material.

Since the development team was anonymous in the surveys, we have used the term “anonymous” after the citations from the surveys.

To analyze the statements of the development team, we used a modified version of the Conventional Analysis Method [19], where we generated names for categories from the collected data. Using this method, we first read all the collected data to get an overview of it and were then analyzed word by word in order to derive codes. Afterwards, notes about our impression and initial analysis of the data were made, and codes were sorted into categories where some were combined in order to create new ones.

5 Findings

The analysis of the statements from the development team resulted in 8 topics that are presented below. The results consist of the following categories: Process (P), obstacles (O), and definition (D).

5.1 (P1) From No UX Consensus to a Comprehensive Understanding

In the beginning of the collaboration with the IT-development team, all the members had different understandings and definitions of UX:

“There are not many (of the team members) that have knowledge about UX. We are not experts in the field of UX and when we are discussing it, it is clear that we have different views on it.” – Anonymous

These multiple views may have been due to the company not establishing a companywide definition. This concern was expressed by P1 during one of the first meetings.

To investigate the development team’s understanding of UX further, survey 1 was conducted (see Table 2). Results showed that there was disagreement about the definition of UX and the dimensions of UX. Furthermore, the results indicated that most team members saw UX as consisting mostly of dimensions related to usability such as functionality, effectiveness, cognitive load, ease of learning. Dimensions as brand image and the social aspects were not seen as part of UX. Even though 7 out of 8 said they knew about UX, the results indicated that it was limited.

Table 2. Results from survey 1 (n = 7) and 2 (n = 8) showing a selection of the UX dimensions and how many agree that these are a part of UX.
Full size table

The aim of activity 5 was to make the development team understand that UX is more than usability. The UX term was presented based on the ISO definition, and all team members worked with UX and UX evaluation. An analysis of the results of a follow-up survey showed that the team developed a uniform understanding of UX dimensions, and they saw UX as broad and consisting of different dimensions. Furthermore they saw UX as more than usability (see Table 2).

5.2 (P2) UX Adoption or No Changes in UX Focus

During activity 12, when the collaboration with the development team and our promotion of UX were evaluated, the designers seemed happy to be introduced to UX and new UX evaluation tools. They also expressed that they had learned something new about working with UX that has already had an impact on how they work in their system development process:

“If you haven’t been here so we could get the knowledge from you this early (in the process), we would not have been where we are now. I have no doubt about that… It is also good to shake the heavy company culture up a bit and it has succeeded for you. Changes have been made.” - P4

In regards to the changes and definite use of the adopted UX and UX evaluation methods, the key members of the development team, expressed a concern. They were positive towards UX and UX evaluation methods and wanted to use the methods we presented, but were concerned about how other employees in the company would accept them. When presenting the UX topics to members of other development teams, they were more interested in the outcome than the process:

“It didn’t seem that they were very interested in the process. They all just wanted to see the prototype and to be able to interact with it… it’s a little sad.” - P4

That it is difficult to promote UX to employees in a company were a problem which both P1, P3 and P4 were struggling with. However, they saw the UX workshop as a very positive way to promote UX, since people are then able to work with the presented UX tools and through this a greater possibility of seeing the value behind the methods. That other than designers and UX experts were part of the workshop, was expressed as positive, since the designers felt that the team now had a shared understanding of UX. It was also seen as being positive since the “non-designers” were able to get an understanding of the UX-workers’ tasks and decisions.

“Using workshops is the best thing you can do in a company (when adopting new methods and techniques). You can send us lots of material, but half of the people will not read it, and the other half will read it and not use it. Participating in a workshop is the way to get things changed in a company.” - P4

“When you have a workshop, people are quickly committed. As soon as you feed them with what they should do, they quickly start working with the methods and tools.” - P3

Whether the company is going to use the UX methods and tools provided is uncertain. However in the development team, small changes have already been adopted and the team is motivated towards continuing working with UX and UX evaluation, both in regards to this project and others.

5.3 (P3) Opinion and Adoption of UX Evaluation Methods

From the beginning of the collaboration with the company, P1 expressed that they had already decided to conduct user evaluations with the thinking aloud technique, but had not found a method for evaluating UX yet. When presenting the possibility of conducting UX on paper prototypes, it was clear that they did not have a method for that and P2 seemed surprised that you are able to conduct these kinds of evaluations. P2 was therefore very interested in a presentation of these kinds of evaluation methods. P1 also expressed a wish to work with both low-fi and hi-fi prototypes with the possibility of conducting user evaluations on these.

As a result of this, the two UX evaluation methods ‘Attrakdiff’ and ‘UX Heuristic Inspection’ were presented. Since the key members of the development team expressed a wish to learn about the methods, but did not have the time to conduct the evaluations, we decided to conduct them through activity 9.

After presenting the methods and results of the evaluations, the key members of the development team expressed enthusiasm in regards to the Attrakdiff user test. They found it a good evaluation tool which provided a clear overview of UX dimensions measured in the evaluation and were motivated towards conducting more, using the Attrakdiff method. They saw the method as being simple and straightforward, and were interested in the tools used to conduct the evaluation.

The two designers (P3, P4) were on the other hand a bit skeptical in regards to letting experts evaluate their system in the later steps of the process, since they saw them as being “fake users”. However, they saw a potential use of expert evaluations in the early stages of their future system’s development process as a way to find early problems and a way to get inspiration for new solutions. It was therefore seen as being an evaluation tool to test prototypes. From one of the later team meetings, it was however clear that the designers began to see the benefits of conducting evaluations, especially after reading the results from our conducted expert evaluations.

In regards to conducting evaluations on paper prototypes versus functional prototypes, they expressed that they were able to use the results from both evaluations, and were especially motivated in conducting tests with paper prototypes to catch early problems in the design phase.

When evaluating the strengths of the presented UX evaluation methods, most of the members of the development team were positive about these and wanted to continue using them:

“We have moved from having a gut feeling about the quality (of the system) to have actual knowledge.” – Anonymous

“By working with and evaluating UX with real users and experts which doesn’t have the background knowledge that we have, we are able to get closer to the best result.” – Anonymous

This indicates that the development team is open towards adopting UX evaluation methods in their development process.

5.4 (D1) UX Dimensions of Fun and Enjoyment

Since the goal for the development team was to create a bank application, one would expect that popular UX dimensions such as enjoyment and fun would not be taken into consideration. The bank industry may be seen as conservative and boring, but being part of the development team for four months demonstrates that they were open towards all sorts of UX dimensions. They e.g. included words such as interesting, fun and different when describing which product they wanted to create in activity 5:

“We have an opportunity to make it a little more fun today… I think it is possible to make it interesting.” - P4

Even though they did not end up working with these kinds of UX dimensions later on, they were open towards using them in the initial idea generation phase. This shows that they are interested in creating a different experience for future users of their bank system and express that they wanted to move away from their boring image.

In addition, after presenting the UX term for the development team, their view of UX widened, making them understand that UX is more than making systems fun to use, but is about giving the user an overall good experience in regards to the system (see Table 2). However it is important to clarify that even though the development team see UX as being broad it does not indicate if they are going to work with UX broadly in practice.

5.5 (D2) More Than One Meaning Per UX Dimension

Especially during activity 5, it became clear that different UX dimensions were interpreted differently among the team members, which resulted in many discussions:

“We spent a long time on creating a shared understanding of these (UX dimensions). When we evaluated the system with one from the other team (in the workshop), the person did not have the same understanding of the words.” - P4

However, during activity 5 the team created a shared understanding of the different meanings of the UX dimensions (see Table 2). During defining the UX dimensions, new ideas also arose in regards to the new system. Most of the development team therefore saw the time used to make a shared understanding of the dimensions as being useful:

“Now we use these words (UX dimensions) and we have this understanding of them. We have then established the conceptual universe of this project and can use the same words.” - Anonymous

Especially the presented UX evaluation method ‘Product Reaction Cards’ were seen as being a good discussion tool, since it presents various UX dimensions.

5.6 (O1) A Skepticism Towards UX Work

Even though the development team seemed excited about working with UX in their new development process, it was clear that they had some concerns in regards to the project. In one of the first meetings with the key members of the development team, P3 expressed a concern in regards to the new UX development method:

“We are all novices. None of us have tried to work like this before… so it will probably take a little longer (to work with UX). When we know the process, things will go faster. In the next project we will know the method and therefore be more confident with it.” - P3

“What worries me is that I think there is an expectation from the board of managers that this is something we just do… and I do not think that. You have to practice and we are going to have a hard time prioritizing what should be made and what should not be made.” - P3

Not only did P3 have skepticism towards working with a more human-centered and UX focused development method, the people working with the design of the system also expressed concerns towards this new shift in focus:

“It has previously been very uphill for this company to use the time and energy on these things (design and UX) and I am concerned that it will also be uphill in this project. I do not think so, but it is not a Google Labs business we’re in. It is about finding a balance, cause we could easily use 100 million (DKK) on this.” - P4

In regards to working with UX early in the development phase, concerns were expressed by the two designers, and it appeared to be a consensus that UX is something one work with in later phases of a design process:

“UX work first appears later on in the process when we test the sketches… then we begin to take UX into consideration.” - P4

“I am a little nervous about becoming lost in the detail elements, when we first start to work with UX.” - P3

The skepticism towards working with UX early and fear of not challenging the team in regards to UX was a motivation for us to make the team work with early UX evaluation methods in activity 5.

5.7 (O2) Working with Creatures of Habit in a UX Workshop

In a meeting with key members of the development team, before the UX workshop, the designers (P3, P4) were skeptical in regards to adopting new UX tools and UX thinking to the development team:

“We are creatures of habits. Many quickly fall back into the old way of doing things… the old way of thinking. One of the major challenges is to challenge the status quo.” - P4

However, they expressed that they were interested in learning new methods and techniques to bring them out of their comfort zone. Especially methods for bringing up UX early in the design process, which also functioned as a motivation for the UX workshop, facilitated by us:

“It is always great to work with UX, unfortunately I see a lack of understanding of what UX is among my colleagues and often we discuss details that are unnecessary at this stage of the process (the concept phase). I wish we could take it (UX) in earlier in the process.” - Anonymous

In the workshop, this concern became reality. Even though they were given the task of thinking about which UX dimensions they wanted their system to represent, they began talking about technical aspects and future goals of the project, neglecting their future users and UX aspect.

However, after we presented the task again for the teams, emphasizing the different aspects of UX, they began working towards UX goals, using more UX dimensions in their conversation. By doing this, a more creative idea generating process began to emerge and it was clear, that they were thinking more about what the future users might want in regards to the system.

Comparing the activity in the beginning and the activity in the end of the workshop, some changes in the team members’ way of thinking were observed. From being much focused on technical aspects, and the current way of doing things in their concept brainstorming phase, they began being more open towards thinking out-of-the-box. The workshop also resulted in the team members being more focused on specific goals for the project, including UX dimensions they wanted to focus on. Further, several team members expressed that the workshop exercises about making the developer evaluate the design ideas early in the concept creating phase were useful. However, they mostly saw the concept UX evaluation methods as a design tool for specifying key UX dimensions in regards to the new system, rather than an evaluation tool:

“I think the methods (UX evaluation methods) are good to bring up discussions in the group, but I do not like to use the method to evaluate with.” - Anonymous

“I think the many concepts (UX dimensions) are a good way to get the ideas flowing and simultaneously narrow the focus on the essentials.” - Anonymous

The key team members of the development team were expressing that they were happy to introduce new ways of thinking to the other members of the development team and liked that the workshop resulted in a shared understanding of UX. However, P4 expressed a concern in regards to a permanent change in the team:

“(The method) introduced the colleagues to other ways of doing things. The question is whether it can be kept alive or whether we fall back into old habits.” - P4

To investigate this concern, activity 6 was held three weeks after the workshop. It was here expressed that they had integrated aspects of the tools from the UX workshop, and that the UX dimension cards from SUXES and Product Reactions Cards were already integrated in their weekly workshops:

“It (the UX dimensions cards) is one of our bibles… we have decided to hang them on our whiteboards.” - P2

Further, P1 also expressed that he had already planned to use the SUXES method in an evaluation of another system in the company, and that the two designers (P3, P4) wanted to integrate the method in their project: “It shows that you have had an effect on us. SUXES has already been integrated” - P1

5.8 (O3) A Need for UX Evaluation Method Modification

When presenting the UX evaluation methods from the research literature to the development team, these methods were often modified to fit the development team’s wishes and available resources. Sometimes these modifications were made by us, but the development team also made their own modifications to the methods.

SUXES and Product Reaction Cards both use cards with prespecified UX dimensions. In activity 5, however, both teams found some of the dimensions irrelevant in regards to the goal of their new system, which resulted in both removals of cards as well as creation of new cards with UX dimensions.

From activity 5, it was also clear that the development team did not see the methods as fixed. Many of the team members talked about different techniques and parts of the methods which could be combined into one that would suit their development process better. They e.g. saw the Product Reaction Cards as being more useful as a brainstorming tool among the developers instead of being a user evaluation tool.

When presenting the Attrakdiff method to the designers of the development team (P3, P4), they also discussed possible modifications to the tool, e.g. combining the notation of ‘acceptable’ and ‘desirable’ levels of the UX dimensions from SUXES into the Attrakdiff method. The reason for this was that the company was focused on prioritizing the different goals for the new system to match team resources.

P4 further expressed that the words used in the Attrakdiff method were too academic and talked about substituting these with ones that are more interesting for the company to measure on:

“Attrakdiff uses a lot of words that are not used in the danish language anymore. Its academia and we cannot use it, but we can drag the things out which make sense to use and throw out the rest.” - P4

The development team was therefore positive towards the academic methods, but did not see them as methods that can be used without modifications in the company, since they saw parts of the methods as being too academic for practical use.

“Everything has to be modified to fit our culture.” - P3

6 Discussion

In this section, we discuss the lessons learned in this AR study.

6.1 Do Not Just Talk About the Methods, Show the Procedures and Results

Since the development team expressed that they had not worked with UX evaluation methods before and were skeptical in regards to the outcome of these, we decided to conduct the evaluations ourselves and video record the whole procedure as well as making an analysis of the results in a document form. The feedback from the key members of the development team was very positive, since they were able to see both the outcome of the evaluations methods as well as the procedure of the different methods. The videos gave a more realistic view on the methods as well as easy guidelines for conducting the evaluations.

An adoption technique of UX evaluation methods similar to ours has been used in the experimental study by Ardito et al. [1]. Here the researchers conducted heuristic evaluations by themselves to show the company they collaborated with, that these kinds of methods require limited resources and little training of company employees.

To adopt UX evaluation methods, it can therefore be beneficial to not only present the methods, but show the company the results you can get from them when evaluating the company’s own products, as well as showing them the procedure behind the methods. This is seen as being very important in companies, since companies tend to be very results-focused and want to know the procedure behind the methods in order to calculate the resources needed to use these.

6.2 Joint UX Workshops Results in Shared Knowledge

As an adoption strategy in our AR study, we facilitated a UX workshop with the development team, to teach all the members UX and UX evaluation methods. Conducting workshops in a development team is already used in Microsoft, and is described by Szóstek [18] as a way to promote UX within the company. She highlights the effects of joint UX workshops: “It is the most effective way to show others the value of UX”. In our case, we also learned that employees are very keen on using workshops to learn about new things. The designers of the development team expressed that they liked that the workshop created a shared knowledge about UX, and that non-designers now have both an understanding of UX work and why the designers want to work with it. As in the Microsoft case study, it was also expressed by the employees, that workshops are an effective way of adopting UX, since the employees are able to work with the term in practice instead of just listening to what others claim UX is.

When adopting UX in a company, it is therefore important to educate more than just the designers of a development team. Since UX and UX evaluation are often seen by non-designers as being messy and difficult to understand, they have difficulties understanding why designers are working with UX and what they can benefit from this. By educating the entire development team and making them used it in practice; it is our experience that the designers work with UX is more appreciated.

6.3 Towards UX Adoption

Even though the company we collaborated with was positive towards both our adoption strategies and the evaluation methods presented, we do not know if it is going to incorporate and use them in the long run, since our study lasted only four months.

Looking at other experimental studies of UX however shows that these are also conducted in a limited time period, making it difficult to talk about an actual UX adoption. The study by Ardito et al. [1] lasted only three months and speculations towards the effects of their adoption of UX in a company were therefore dimmed. However, their study mentioned that the company they collaborated with was positive towards the presented UX material and in the end of their study; some small elements of the adopted UX were already incorporated in the company’s development process.

In our case, we are also not capable of talking about permanent UX adoption of our UX evaluation methods. However, just as Ardito et al. [1] were able to see some indications of UX adoption, we also witnessed parts of our UX evaluation methods being integrated in the development process by the end of the study.

Since we were not able to find investigations of long term adoption of UX techniques and methods we cannot say whether the adoption strategies actually work in the long run. In addition, the concept of UX that we employed [10] is very vague and difficult to measure. It would be interesting for a software organisation to employ a concept of UX that is simpler or even possible to measure.

6.4 Academic Evaluation Methods Need Changes

In our AR study we promoted UX and UX evaluation methods from research literature. With all the presented UX evaluation methods, both we and the development team saw a need to modify these in order to make them fit the development method and the goals of the company in regards to the new system. The development team expressed that some of the methods were using words that were too academic and wanted to make their own modifications on them in order to make them useful for the purpose of their projects.

Looking at the Ardito et al. study [1], this also resulted in modifications of the material from scientific articles in order to use it in a more practical context. They concluded that researchers have to be “more careful in transferring academic work into practical value for industry” [1] and describe that “it is the responsibility of academics to translate scientific articles, which formally describe evaluation methods, into something that makes sense for companies and is ready to be applied.” [1].

From our experience in regards to this AR study, we also believe that modifications have to be made in order to make the scientific methods useful in a more practical oriented situation. The reason for this is that scientific papers often have another purpose and goal for methods, e.g. finding if certain UX dimensions are measureable and testing certain conditions, whereas the practical use of the methods is not studied often. It would also be useful if the prerequisites for applying specific UX methods were available. In our case, the company had a very low level or no UX maturity, and we did not know what level of maturity was required for the methods we selected.

6.5 The Usefulness of AR in Studies of UX Adoption

In this study, we have worked with AR in regards to UX research. In HCI, some AR studies have already been made and researchers are trying to promote the usefulness of this research approach in order to increase their use. Hayes [8] investigated the relationship between AR in HCI. She explained that some researchers do not see AR as providing scientific rigor, since it only investigates few cases that cannot be generalized. However, she states that: “AR provides a rigorous framework for generating and sharing sufficient knowledge about a solution that it may potentially be transferred to other contexts.” [8].

The purpose behind AR is therefore not to be able to construct one solution that can be used by others, but to focus on local solutions to local problems, which may be reused in other cases. Further, since studies about adoption of UX evaluation methods in companies are sparse in the literature, investigations are needed to explore this. Such an investigation requires testing different techniques. Baskerville [3] describes AR as an opportunity to study social processes by “introducing changes into these processes and observing the effects of these changes.” [3].

7 Conclusion

We have presented an action research study with an IT development team of a Danish bank, investigating how UX and UX evaluation can be adopted in industry. We used different adoption strategies with the development team. Our results indicate that changes have already been made and part of the material has been adopted by the company.

In the process of adopting UX and UX evaluation, demonstrations and workshops are effective strategies. The employees have different meanings of the different UX dimensions, so it is important for employees to create a shared understanding and definition of UX dimensions in order to work towards the same goal. It is difficult to use UX evaluation methods from research literature without modifying these to fit the individual company.

Our study is limited in regards to the period being studied. In addition, we have only conducted one action research study. Since the purpose of action research is to construct local solutions to local problems which may potentially be used in other cases, it is possible to build upon our study and results in order to investigate how UX evaluation can be adopted in practice.