A Revised Set of Usability Heuristics for the Evaluation of Interactive Systems

Usability is a critical quality attribute in the information society . Inspection based evaluation relies on various forms of design knowledge that are used to assess the usability, such as ergonomic criteria, heuristics, and guidelines. Usability heuristics are widely recognized principles against which the usability of an interactive system could be evaluated. This work presents a revised set of usability heuristics that are based on the experience acquired in previous studies targeting local e-government websites. The underlying goal is to include the most important ergonomic criteria and usability heuristics into a clear hierarchical organization, which helps evaluators to better explain and developers to better understand the usability problems. The new set has only fourteen heuristics that are structured into four groups: user guidance, user effort, user control and freedom, and user support. The main strength of this approach is the good coverage achieved with a reduced number of heuristics.


Introduction
Usability is a critical quality attribute in the information society. Providing products with a rich functionality that satisfy various users' needs is no longer enough. Apart from usefulness, a computer system should be usable, which means to enable a user to accomplish the goals with effectiveness, efficiency, and satisfaction [5]. Designing for usability requires a user-centered attitude: knowing the users, the tasks they perform, and developing a product iteratively in a design -evaluation loop. In order to incorporate usability in a product, various forms of design knowledge could be used that are structured as design principles, ergonomic criteria, design patterns, design rules, heuristics, and guidelines [1,2,13,16,23,24,25]. Usability evaluation aims at finding and explaining the difficulties a user has (or might have) in performing a task with a computer system. There are two broad categories of usability evaluation methods: usability inspection and user testing. Usability inspection, also termed as expert evaluation, is carried on by one or several evaluators that are testing the system with the purpose to anticipate usability problems. Both evaluation methods rely on some form of design knowledge that is used in the evaluation. In a previous work, a set of 24 heuristics has been proposed that have been used in the evaluation of municipal websites. Based on the evaluation results, the heuristics have been further refined [18]. A task-based inspection method has been used: all evaluators are testing the interface with the same tasks and the severity of the evaluation problems is related to the task goal. The main role of the usability heuristics is to explain and document the usability problems. Two recent studies revealed some shortcomings of this set, among which the most important was the difficulty of relating a usability problem with a usability heuristic. The goal of this work is to revise the usability heuristics in order to make them more usable for evaluators [10]. The revised set has only 14 heuristics that are structured into four groups: user guidance, user effort, user control and freedom, and user support. The rest of this paper is organized as follows. In the next section, we present related work in the usability evaluation and the development of usability heuristics with a focus on usability inspection and usability heuristics. In section 1 3 we present the revised set of usability heuristics. The paper ends with a discussion, conclusion, and future work directions in sections 4 and 5.

Related work
Usability evaluation Usability evaluation methods are classified according to various criteria. In this work, we focus on the formative evaluation methods, which aim to uncover and fix usability problems as early as possible during the development process [22]. A usability problem has been defined by Jacob Nielsen [12] as any aspect of the user interface which might create difficulties for the user with respect to an important usability indicator (such as: ease of understanding and learning how to operate, ease of use, time to complete the task, or subjective user satisfaction). According to the potential effect on the user's task, the usability problems are ranked as severe (major), moderate and minor. A usability problem is severe if the user is not able to accomplish the task goal or the task ends with a significant loss of data or time. The problem is moderate if it has an important impact on task execution but the user is able to find a solution. A minor usability problem is irritating the user but it doesn't have an important impact on accomplishing the task's goal. Prior to the first release of an application, all important usability problems (severe and moderate) should be fixed. The evaluation method could be done with users (user testing, more expensive) or without users (expert evaluation, cheaper). Regardless the method used, the evaluation report should describe and explain in detail each usability problem and suggest ways how to fix it. An evaluation report should be reliable, usable, and useful for developers [10]. Usability inspection methods (also termed as expert evaluations) are carried on by experts that are testing the user interface to find potential usability problems. In order to anticipate as many of the possible problems and to reduce the evaluator effect, several experts are needed (usually [3][4][5]. In this case, the evalua-tion is done in two phases: individual evaluation and consolidation. In the consolidation phase, the usability problems identified by each evaluator are analyzed and integrated into a list of unique usability problems. In order to do this, evaluators have to agree on the problem description, causes, and suggestions how to fix it, as well as on the severity rating. A well-known inspection method is the heuristic evaluation that has been proposed by Nielsen and Molich as a discount evaluation method that assesses the usability against a set of usability heuristics [11]. According to the authors¸ the heuristic evaluation is cheap, intuitive, does not require advanced planning, and can be used early in the development process. Several authors questioned the effectiveness of heuristics in finding usability problems. The main criticism is related to the heuristic evaluation method which is more oriented towards fault finding instead of user's difficulties in performing the task [7,8]. An approach that overcomes this shortcoming is the heuristic walkthrough, which combines the taskbased evaluation with the heuristics-based review [7,21]. In our opinion, the heuristics are a valuable aid for guiding developers and helping evaluators to explain the usability problems. In order to effectively identify usability issues, usability inspection should be done by testing the user interface in a task-based approach and by analyzing the usability problems against some evaluation criteria [17]. The evaluation criteria refer to various design knowledge structured in form of usability prescriptions, such as principles, ergonomic criteria, heuristics, usability checklists, patterns, and guidelines.

Usability heuristics
Several approaches exist to a hierarchical organization of the usability-related principles as well as to the development and validation of usability heuristics [6,16,19,23,24,25]. The widest used usability heuristics has been defined by the heuristic evaluation method [14]: visibility of system status, match between system and the real world, user control and freedom, consistency and standards, error prevention, recognition rather than recall, flexibility and efficiency of use, aesthetics and minimalist design, help users recognize, diagnose and recover from errors, help, and documentation. A checklist for these heuristics has been specified by Pierotti [15] as an aid for the evaluators. Another approach has been proposed by Bastien & Scapin [1] that is based on evaluating the user interface against ergonomic criteria consisting of 18 elementary criteria grouped into 8 categories (general principles). For each criterion, several usability guidelines are given. Ji et al. [6] proposed a set of usability principles structured into five groups: cognition support, information support, interaction support, performance support, and user support. Based on these principles and the user interface elements, they developed a usability checklist for mobile phones. A shortcoming of this approach is the lack of an explicit mapping between the usability principles and the checklist. Kamper [8] mentioned that a limitation of the existing practices is the plethora of guidelines with no comprehensive underlying basis. He argued for a simple and unified set of heuristics that could be applied across different technologies, contexts of use, and domains of knowledge. The heuristics were grouped under the Lead, Follow and Get Away (LF&G) paradigm inspired by the facilitative learning. Quinones & Rusu [19] surveyed the existing literature on developing usability heuristics. The results of the survey show a plethora of existing heuristics organized around various principles as well as many approaches to validation. Their study reveals a tendency to create and validate in-house heuristics serving specific needs. Several studies proposed various extensions of Nielsen's usability heuristics in order to address specific usability issues [4,9,16]. A different approach has been taken in [14] by integrating to well-known sets of usability principles: usability heuristics and ergonomic criteria. This approach will be detailed in the next section.

The revised usability heuristics Reasons for change
Ergonomic criteria and usability heuristics are two forms of usability design knowledge that have many things in common. For example, in both, we can find the same usability principles, related to feedback, flexibility, or error prevention, which are expressed in a less or more similar way. The table below shows the correspondence between the usability heuristics and the ergonomic criteria (criteria marked with an asterisk are elementary criteria [20]). Scapin & Bastien [20] with the heuristics of Nielsen [14]. The objective of our approach was to integrate the ergonomic criteria and usability heuristics into a coherent structure rather than creating and validating a new set of heuristics or extending an existing one. The usability evaluation has been carried out in a scenario-based approach and revealed some difficulties when using the usability heuristics to explain and document specific problems for municipal websites. Therefore, the set has been refined by merging two existing heuristics and adding two new heuristics, which resulted in a total of 25 heuristics [18]. However, our further experience with other kinds of web applications suggested that several changes are needed. First, for some evaluators having the expertise in the application domain, it has been difficult to learn and distinguish between 25 usability heuristics. Second, the evaluators faced several difficulties when trying to explain and document the usability problems related to user guidance and user workload. Third, it was relatively difficult to achieve a consensus in the collaborative consolidation phase.

Overview and organization
The revised set of heuristics is organized into four groups, addressing general ergonomic criteria: user guidance, user effort, user control and freedom, and user support. The number of heuristics has been reduced from 25 to 14 and the number of groups from six to four. Consistency has been included in the second group since it helps the understanding and learning how to use the system [14,20]. The error management (three heuristics in the previous version) has been included in the last group since is related to the assistance provided by the system. The aesthetic design has no longer included since is related to hedonic rather than pragmatic aspects. In this respect, this revised set does not address criteria that are related to the user experience (UX). The information architecture is the only heuristic mainly related to web applications. The rest of heuristics are general and could be used for any type of application.
For each heuristic, the following information is given:  Title: the usability criterion.  Statement: the usability heuristic.  Usability guidelines: a short list of related usability guidelines. The rationale of this approach is to include the most important ergonomic criteria, usability principles, and heuristics into a hierarchical organization, which makes it easier to be understood, learned, and used by both designers and evaluators. The revised heuristics are summarized in Table 2. User guidance is a general ergonomic criterion that concerns the means to inform, orient, and guide the users throughout the interaction with the computer [20]. Suitable user guidance has positive effects on the ease of use. The user guidance includes the following four heuristics: prompting, feedback, information architecture, and grouping/distinction. The second group refers to the physical, perceptual, and cognitive effort needed to learn how to use the system and the effort needed to use it effectively. Reducing the user's effort has positive effects on the learnability and efficiency of use. This group includes three heuristics: consistency, cognitive workload, and minimal actions. The third group concerns the means available to adapt and control the system as well as to use it in a flexible way. The control and freedom have positive effects on the efficiency of use. This group includes three heuristics: explicit user actions, user control, and flexibility. The last group concerns the support for using the system, including specific accessibility requirements for the users with special needs. Four heuristics have been included in this group: compatibility with the user, task guidance and support, error management, help, and documentation.

Detailed description
In this subsection, a description of each heuristic is given that is mainly based on the definitions of the ergonomic criteria [20] and usability heuristics [14].  Provide a general presentation of the system. The usability guidelines provided in this section are only a short checklist illustrating the coverage of the revised set of heuristics with respect to the previously mentioned sources: ergonomic criteria and usability heuristics.

Discussion
From a methodological point of view, there are some issues that are important to discuss. First, the task-based usability inspection is more effective than the heuristic evaluation. Not only makes it possible to uncover most important problems, but it also makes possible a comparative usability evaluation. Since the same tasks are evaluated, is possible to compare the results obtained by different methods or by different evaluators. Second, during the evaluation process, the heuristics are used to explain and document the usability problems, regardless the method used (inspection or user testing). Third, in order to assess the reliability of the evaluation results, a one-to-one mapping between usability issues and heuristics is desirable. This is difficult to achieve since the explanatory power of heuristics is various [13] and a usability problem may be caused by the violation of several principles. This situation occurs frequently in the consolidation phase when the same problem is identified by several evaluators but is documented with a different heuristic. In this case, the individual problems could be merged but all relevant heuristics should be mentioned. The revised set of usability heuristics has several strengths and inherent limitations. The hierarchical structure is simple and clear, grouping a small number of heuristics under four general principles: user guidance, user effort, user control, and user support. Second, each heuristic is defined in a general way, in order to ensure a good coverage with respect to various application and technologies. The third level in the hierarchy is used for the inclusion of related usability guidelines and enables the extension in two ways. One way is to include other relevant heuristics or guidelines from other sources. Another way is to include more specific guidelines that are addressing application or technology specific issues. Third, the usability heuristics are taken from two well-known sources of design knowledge that have been already validated and widely used for more than two decades. There are inherent limitations of any approach striving to focus on a relatively small number of heuristics. The main limitation is related to completeness but this could be addressed by extending the design knowledge in a hierarchical way, as mentioned above.

Conclusion and future work
In this paper, a revised set of usability heuristics has been proposed that integrates into a hierarchical structure two widely used design and evaluation principles. The main strength of this approach is a good coverage achieved with a reduced number of heuristics. Usability heuristics represent valuable design knowledge that could be used to create a usercentered attitude, to incorporate usability into a product, to train novice evaluators, to structure usability guidelines, to explain and document usability problems, and to analyze the ergonomic quality of an application. Although initially targeting the evaluation of web applications, the revised set of usability heuristics has a general coverage as regards the application types and technologies. In the near future, we intend to develop usability guidelines that are specific to target applications and technologies in order to better support the evaluation process.