Juergen Baumgartner
ABSTRACT
We have developed a pictorial multi-item scale, called P-SUS (Pictorial System Usability Scale), which aims to measure the perceived usability of mobile devices. The scale is based on the established verbal usability questionnaire SUS (System Usability Scale). A user-centred design process was employed to develop and refine its 10 pictorial items. The scale was tested in a first validation study (N=60) using student participants. Psychometric properties (convergent validity, criterion-related validity, sensitivity, and reliability), as well as the motivation to fill in the scale were assessed. The results indicated satisfactory convergent validity for about two-thirds of the items. Furthermore, strong correlations were obtained for the sum scores between verbal and pictorial SUS, and the pictorial scale was perceived as more motivating than the verbal questionnaire. The P-SUS represents a first attempt to provide a pictorial usability scale for the evaluation of (mobile) devices.
Supplemental Material
- Leonardo Angelini, Francesco Carrino, Stefano Carrino, Maurizio Caon, Omar Abou Khaled, Jurgen Baumgartner, Andreas Sonderegger, Denis Lalanne, and Elena Mugellini. 2014. Gesturing on the Steering Wheel: a User-elicited taxonomy. In Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 1--8. Google Scholar
Digital Library
- Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies 4, 3: 114--123. Google ScholarDigital Library
- Aaron Bangor, Philip T. Kortum, and James T. Miller. 2008. An empirical evaluation of the system usability scale. Intl. Journal of Human--Computer Interaction 24, 6: 574--594.Google ScholarCross Ref
- Juergen Baumgartner, Andreas Sonderegger, and Juergen Sauer. 2019. No need to read: Developing a pictorial single-item scale for measuring perceived usability. International Journal of Human-Computer Studies 122: 78--89.Google ScholarCross Ref
- Alberto Betella and Paul F. M. J. Verschure. 2016. The Affective Slider: A Digital Self-Assessment Scale for the Measurement of Human Emotions. PLOS ONE 11, 2: e0148037.Google ScholarCross Ref
- Eli Blevis, Youn-kyung Lim, Erik Stolterman, Tracee Vetting Wolf, and Keichi Sato. 2007. Supporting design studio culture in HCI. In CHI '07 extended abstracts on Human factors in computing systems - CHI '07, 2821. Google ScholarDigital Library
- Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1: 49--59.Google ScholarCross Ref
- Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1: 49--59.Google ScholarCross Ref
- Joost Broekens and Willem-Paul Brinkman. 2013. AffectButton: A method for reliable and valid affective self-report. International Journal of Human-Computer Studies 71, 6: 641--667. Google ScholarDigital Library
- John Brooke. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194: 4--7.Google Scholar
- John Brooke. 2013. SUS: A Retrospective. J. Usability Studies 8, 2: 29--40. Google ScholarDigital Library
- Heather Buchanan and N. Niven. 2002. Validation of a Facial Image Scale to assess child dental anxiety. International journal of paediatric dentistry 12, 1: 47--52.Google Scholar
- Kevin Capota, Marco van Hout, and Thea van der Geest. 2007. Measuring the emotional impact of websites: a study on combining a dimensional and discrete emotion approach in measuring visual appeal of university websites. In Proceedings of the 2007 conference on Designing pleasurable products and interfaces, 135--147. Retrieved April 27, 2017 from http://dl.acm.org/citation.cfm?id=1314173 Google ScholarDigital Library
- Timo Christophersen and Udo Konradt. 2011. Reliability, validity, and sensitivity of a single-item measure of online store usability. International Journal of Human-Computer Studies 69, 4: 269--280. Google ScholarDigital Library
- J. Cohen. 1988. The effect size. Statistical power analysis for the behavioral sciences: 77--83.Google Scholar
- Pieter Desmet. 2003. Measuring Emotion: Development and Application of an Instrument to Measure Emotional Responses to Products. In Funology, Mark A. Blythe, Kees Overbeeke, Andrew F. Monk and Peter C. Wright (eds.). Springer Netherlands, 111-- 123.Google Scholar
- Pieter Desmet, Kees Overbeeke, and Stefan Tax. 2001. Designing products with added emotional value: Development and appllcation of an approach for research through design. The design journal 4, 1: 32--47.Google Scholar
- Pieter Desmet, Martijn H. Vastenburg, and Natalia Romero. 2016. Mood measurement with Pick-A-Mood: review of current methods and design of a pictorial self-report scale. Journal of Design Research 14, 3: 241--279.Google ScholarCross Ref
- Will Eisner. 1985. Theory of Comics and Sequential Art. F.: Poorhouse press.Google Scholar
- Kraig Finstad. 2010. Response interpolation and scale sensitivity: Evidence against 5-point scales. Journal of Usability Studies 5, 3: 104--110. Google ScholarDigital Library
- Dawn P. Flanagan, Samuel O. Ortiz, and Vincent C. Alfonso. 2013. Essentials of cross-battery assessment. John Wiley & Sons.Google Scholar
- Mirta Galesic and Michael Bosnjak. 2009. Effects of questionnaire length on participation and indicators of response quality in a web survey. Public opinion quarterly 73, 2: 349--360.Google Scholar
- Gunther Gediga, Kai-Christoph Hamborg, and Ivo Duntsch. 1999. The IsoMetrics usability inventory: An operationalization of ISO 9241--10 supporting summative and formative evaluation of software systems. Behaviour & Information Technology 18, 3: 151--164.Google ScholarCross Ref
- Ramesh Ghiassi, Kevin Murphy, Andrew R. Cummin, and Martyn R. Partridge. 2011. Developing a pictorial Epworth Sleepiness Scale. Thorax 66, 2: 97--100.Google ScholarCross Ref
- Shamila Haddad, Steve King, Paul Osmond, and Shahin Heidari. 2012. Questionnaire Design to Determine Children's Thermal Sensation, Preference and Acceptability in the Classroom. 7.Google Scholar
- Kai-Christoph Hamborg, Julia Hulsmann, and Kai Kaspar. 2014. The Interplay between Usability and Aesthetics: More Evidence for the "What Is Usable Is Beautiful" Notion. Advances in HumanComputer Interaction 2014: 1--13. Google ScholarDigital Library
- Rex Hartson and Pardha S. Pyla. 2012. The UX Book: Process and guidelines for ensuring a quality user experience. Elsevier. Google ScholarDigital Library
- Timothy R. Hinkin. 1995. A review of scale development practices in the study of organizations. Journal of Management 21, 5: 967--988.Google ScholarCross Ref
- Kasper Hornbæk and Effie Lai-Chong Law. 2007. Meta-analysis of Correlations Among Usability Measures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07), 617--626. Google ScholarDigital Library
- International Organization for Standardization. 2010. Ergonomics of human-system interaction - Part 210: Humancentred design for interactive systems (Standard No. 9241--210). Retrieved from https://www.iso.org/standard/52075.htmlGoogle Scholar
- International Organization for Standardization. 2016. Ergonomics of human-system interaction - Part 11: Usability: Definitions and concepts (Standard No. 9241--11.2). Retrieved from https://www.iso.org/standard/63500.htmlGoogle Scholar
- Alice M. Isen and Johnmarshall Reeve. 2005. The influence of positive affect on intrinsic and extrinsic motivation: Facilitating enjoyment of play, responsible work behavior, and self-control. Motivation and emotion 29, 4: 295--323.Google Scholar
- Heike Jungst. 2010. Information Comics--Knowledge Transfer in a Popular Format.Google Scholar
- Philip Kortum and S. Camille Peres. 2014. The relationship between system effectiveness and subjective usability scores using the System Usability Scale. International Journal of HumanComputer Interaction 30, 7: 575--584.Google ScholarCross Ref
- Philip T. Kortum and Aaron Bangor. 2013. Usability ratings for everyday products measured with the System Usability Scale. International Journal of Human-Computer Interaction 29, 2: 67-- 76.Google ScholarCross Ref
- Theodore Kunin. 1955. The Construction of a New Type of Attitude Measure. Personnel Psychology 8, 1: 65--77.Google ScholarCross Ref
- Clayton Lewis and Robert Mack. 1982. Learning to Use a Text Processing System: Evidence from "Thinking Aloud" Protocols. In Proceedings of the 1982 Conference on Human Factors in Computing Systems (CHI '82), 387--392. Google ScholarDigital Library
- James (Jim) R. Lewis and Jeff Sauro. 2017. Revisiting the Factor Structure of the System Usability Scale. J. Usability Studies 12, 4: 183--192. Google ScholarDigital Library
- James R. Lewis. 2002. Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies. International Journal of Human-Computer Interaction 14, 3--4: 463--488.Google ScholarCross Ref
- James R. Lewis. 2018. The System Usability Scale: Past, Present, and Future. International Journal of Human--Computer Interaction 34, 7: 577--590.Google ScholarCross Ref
- Bernd Marcus, Michael Bosnjak, Steffen Lindner, Stanislav Pilischenko, and Astrid Schutz. 2007. Compensating for Low Topic Interest and Long Surveys: A Field Experiment on Nonresponse in Web Surveys. Social Science Computer Review 25, 3: 372--383. Google ScholarDigital Library
- E. McAuley, T. Duncan, and V. V. Tammen. 1989. Psychometric properties of the Intrinsic Motivation Inventory in a competitive sport setting: a confirmatory factor analysis. Research Quarterly for Exercise and Sport 60, 1: 48--58.Google ScholarCross Ref
- Samuel Messick. 1979. Test Validity and the Ethics of Assessment. ETS Research Report Series 1979, 1: i--43.Google Scholar
- Michael Minge and Laura Riedel. 2013. meCUE-Ein modularer Fragebogen zur Erfassung des Nutzungserlebens. In Mensch & Computer, 89--98.Google Scholar
- Morten Moshagen and Meinald T. Thielsch. 2010. Facets of visual aesthetics. International Journal of Human-Computer Studies 68, 10: 689--709. Google ScholarDigital Library
- Clive Nancarrow and Trixie Cartwright. 2007. Online access panels and tracking research: the conditioning issue. International Journal of Market Research 49, 5: 573--594.Google ScholarCross Ref
- Jakob Nielsen and Jonathan Levy. 1994. Measuring Usability: Preference vs. Performance. Commun. ACM 37, 4: 66--75. Google ScholarDigital Library
- J. C. Nunnally. 1978. Psychometric Theory, Second. New York: McGrawHill.Google Scholar
- Jum C. Nunnally and I. H. Bernstein. 1994. Psychometric Theory (McGraw-Hill Series in Psychology). McGraw-Hill New York.Google Scholar
- Sampo V. Paunonen, Michael C. Ashton, and Douglas N. Jackson. 2001. Nonverbal assessment of the Big Five personality factors. European Journal of Personality 15, 1: 3--18.Google ScholarCross Ref
- Laura Reynolds-Keefer, Robert Johnson, and S. Carolina. 2011. Is a picture is worth a thousand words? Creating effective questionnaires with pictures. Practical Assessment, Research & Evaluation 16, 8: 1--7.Google Scholar
- Richard M. Ryan. 1982. Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory. Journal of personality and social psychology 43, 3: 450.Google ScholarCross Ref
- Young Sam Ryu and Tonya L. Smith-Jackson. 2006. Reliability and Validity of the Mobile Phone Usability Questionnaire (MPUQ). J. Usability Studies 2, 1: 39--53. Google ScholarDigital Library
- Jeff Sauro. 2011. A practical guide to the system usability scale: Background, benchmarks & best practices. Measuring Usability LLC Denver, CO.Google Scholar
- Jeff Sauro and James R. Lewis. 2009. Correlations among prototypical usability metrics: evidence for the construct of usability. In Proceedings of the SIGCHI conference on human factors in computing systems, 1609--1618. Google ScholarDigital Library
- Jeff Sauro and James R. Lewis. 2016. Quantifying the user experience: Practical statistics for user research. Morgan Kaufmann. Google ScholarDigital Library
- Andreas Sonderegger, Klaus Heyden, Alain Chavaillaz, and Juergen Sauer. 2016. AniSAM & AniAvatar: Animated Visualizations of Affective States. 4828--4837. Google ScholarDigital Library
- R. A. Stern, J. E. Arruda, C. R. Hooper, G. D. Wolfner, and C. E. Morey. 1997. Visual analogue mood scales to measure internal mood state in neurologically impaired patients: Description and initial validity evidence. Aphasiology 11, 1: 59--71.Google ScholarCross Ref
- Noam Tractinsky. 2018. The Usability Construct: A Dead End? Human--Computer Interaction 33, 2: 131--177.Google ScholarCross Ref
- Bert Weijters, Elke Cabooter, and Niels Schillewaert. 2010. The effect of rating scale format on response styles: The number of response categories and response category labels. International Journal of Research in Marketing 27, 3: 236--247.Google ScholarCross Ref
- Matthias Wilde, Katrin Batz, Anastassiya Kovaleva, and Detlef Urhahne. 2009. Uberprufung einer Kurzskala intrinsischer Motivation (KIM). Zeitschrift für Didaktik der Naturwissenschaften 15.Google Scholar
- Rudiger Zarnekow and Walter Brenner. 2004. Integriertes Informationsmanagement: Vom Plan, Build, Run zum Source, Make, Deliver. Informationsmanagement: Konzepte und Strategien für die Praxis. dpunkt, Heidelberg: 3--24.Google Scholar
Index Terms
- Pictorial System Usability Scale (P-SUS): Developing an Instrument for Measuring Perceived Usability
Recommendations
Questionnaire experience of the pictorial usability inventory (PUI) – a comparison of pictorial and hybrid usability scales
Highlights- This article compares pictorial, hybrid and verbal usability questionnaires using classical psychometric parameters and respondent-centred measures.
- A recently developed questionnaire measuring ‘questionnaire experience’ was used to ...
AbstractIn recent years, alternative types of usability questionnaires using graphical elements (pictorial scales) or a combination of graphical and verbal elements (hybrid scales) have been introduced. Previous research indicates that these ...
Pictorial usability inventory (PUI): a pilot study
MuC '20: Proceedings of Mensch und Computer 2020The Pictorial Usability Inventory (PUI) is an image-based instrument that was designed for the assessment of perceived usability of smartphone apps. The aim was to provide users with an easy and intuitive access to the evaluation of perceived usability. ...
Revisiting the factor structure of the system usability scale
In 2009, we published a paper in which we showed how three independent sources of data indicated that, rather than being a unidimensional measure of perceived usability, the System Usability Scale apparently had two factors: Usability (all items except ...
Comments