Abstract
This paper addresses the possibility of measuring perceived usability in an absolute way. It studies the impact of the nature of the tasks performed in perceived software usability evaluation, using for this purpose the subjective evaluation of an application’s performance via the Software Usability Measurement Inventory (SUMI). The paper reports on the post-hoc analysis of data from a productivity study for testing the effect of changes in the graphical user interface (GUI) of a market leading drafting application. Even though one would expect similar evaluations of an application’s usability for same releases, the analysis reveals that the output of this subjective appreciation is context sensitive and therefore mediated by the research design. Our study unmasked a significant interaction between the nature of the tasks used for the usability evaluation and how users evaluate the performance of this application. This interaction challenges the concept of absolute benchmarking in subjective usability evaluation, as some software evaluation methods aspire to provide, since subjective measurement of software quality will be affected most likely by the nature of the testing materials used for the evaluation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Metrics for Usability Standards in Computing (CEC ESPRIT project number 5429).
The SUMI’s Background notes read as follows: “the minimum user sample size needed for an analysis with tolerable precision using SUMI is on the order of 10–12 users, although evaluations have been carried out successfully with smaller sample sizes. However, the generalisability of the SUMI results depends not so much on the sample size itself, but the care with which the context of use of the software has been studied and the design plan has been made. As summarised by Macleod (see elsewhere in this volume) this involves identifying the typical users of the software, the goals which they typically wish to achieve, and the technical, physical and organisational environments in which the work is carried out (for prototype systems, this involves determining the future context of use). The design plan requires an adequate sampling of the context of use.”
References
Background notes on the SUMI questionnaire. Retrieved. from http://www.ucc.ie/hfrg/questionnaires/sumi/sumipapp.html
Brooke J (2004, Sep. 30, 2004) SUS—A quick and dirty usability scale. Retrieved October 3rd, 2004, from http://www.usability.serco.com/trump/documents/Suschapt.doc
Chin JP, Diehl VA, Norman KL (1988) Development of an instrument measuring user satisfaction of the human-computer interface. Paper presented at the SIGCHI
Dumas JS (2003) User-based evaluations. In: Jacko J, Sears A (eds) The human computer interaction handbook. Lawrence Earlbaum Associates, New Jersey
Dybkjaer L, Bernsen NO (2000) Usability issues in spoken dialogue systems. Nat Lang Eng 6(3/4):243–271
Kirakiwski J, Corbett M (1988) Measuring user satisfaction. In: Jones DM Winder R (eds) People and computers IV. Cambridge University, Cambridge
Newell A, Simon H (1972) Human problem solving. Prentice Hall, New Jersey
Sears A (2003) Testing and Evaluation. In: Jacko JA, Sears A (Eds) The human–computer interaction handbook: fundamentals, evolving technologies and emerging applications. LEA, New Jersey
Acknowledgments
This research project was sponsored by a grant provided by Autodesk Inc.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Cavallin, H., Martin, W.M. & Heylighen, A. How relative absolute can be: SUMI and the impact of the nature of the task in measuring perceived software usability. AI & Soc 22, 227–235 (2007). https://doi.org/10.1007/s00146-007-0127-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00146-007-0127-0