Skip to main content
SpringerLink
Log in

Investigating the psychometric properties of the Speech User Interface Service Quality questionnaire

  • Published:
International Journal of Speech Technology Aims and scope Submit manuscript

Abstract

The Speech User Interface Service Quality (SUISQ) questionnaire is a standardized instrument for the assessment of the usability of interactive voice response (IVR) applications, developed by Polkosky (Toward a social-cognitive psychology of speech technology: affective responses to speech-based e-service, 2005; Mediated interpersonal communication, 2008). During its development, participants rated the quality of recorded interactions rather than interactions in which they participated, leaving open the question of the extent to which the findings would generalize to personal as opposed to observed interactions. The results of a large-scale unmoderated usability study of a natural-language speech recognition IVR demonstrated the utility of the SUISQ for the purpose of assessing personal experiences with service-providing speech user interfaces. The psychometric properties of construct validity and reliability were very similar to those reported by Polkosky. Additional item analyses led to the definition of two subsets of the full set of 25 SUISQ items—a reduced version (SUISQ-R, 14 items) and a maximally-reduced version (SUISQ-MR, 9 items). The SUISQ-R had similar psychometric properties to the full SUISQ, but analysis the SUISQ-MR revealed some weaknesses in its reliability and construct validity. This replication of the original SUISQ findings in a markedly different context of measurement and the availability of a shorter, psychometrically qualified, version of the questionnaire (SUISQ-R) should enhance its utility for usability practitioners who work on the development and assessment of speech-recognition IVRs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Albert, T., Albert, B., & Tedesco, D. (2010). Beyond the usability lab: Conducting large-scale online user experience studies. Burlington: Morgan Kaufmann.

    Google Scholar 

  • Cargile, A., Giles, H., Ryan, E., & Bradac, J. (1994). Language attitudes as a social process: A conceptual model and new directions. Language & Communication, 14, 211–236.

    Article  Google Scholar 

  • Cliff, N. (1987). Analyzing multivariate data. San Diego: Harcourt Brace Jovanovich.

    Google Scholar 

  • Coovert, M. D., & McNelis, K. (1988). Determining the number of common factors in factor analysis: A review and program. Educational and Psychological Measurement, 48, 687–693.

    Article  Google Scholar 

  • Dabholkar, P., & Bagozzi, R. (2002). An attitudinal model of technology-based self-service: Moderating effects of consumer traits and situational factors. Journal of the Academy of Marketing Science, 30(3), 184–201.

    Article  Google Scholar 

  • Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York: John Wiley.

    Google Scholar 

  • Hone, K. S., & Graham, R. (2000). Towards a tool for the subjective assessment of speech system interfaces (SASSI). Natural Language Engineering, 6(3–4), 287–303.

    Article  Google Scholar 

  • Hornbæk, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79–102.

    Article  Google Scholar 

  • Hornbæk, K., & Law, E.L. (2007). Meta-analysis of correlations among usability measures. In Proceedings of CHI 2007 (pp. 617–626). San Jose: ACM.

  • International Standards Organization. (1998). Ergonomic requirements for office work with visual display terminals (VDTs)—Part 11: Guidance on usability (ISO 9241-11:1998(E)). Geneva: ISO.

    Google Scholar 

  • International Telecommunication Union. (1994). A method for subjective performance assessment of the quality of speech voice output devices (ITU-T recommendation (p. 85). Geneva: ITU.

    Google Scholar 

  • Kraft, V., & Portele, T. (1995). Quality evaluation of five German speech synthesis systems. Acta Acustica, 3, 351–365.

    Google Scholar 

  • Kuo, H. J., Siohan, O., & Olive, J. P. (2003). Advances in natural language call routing. Bell Labs Technical Journal, 7(4), 155–170.

    Article  Google Scholar 

  • Landauer, T. K. (1988). Research methods in human-computer interaction. In M. Helander (Ed.), Handbook of human–computer interaction (pp. 905–928). New York: Elsevier.

    Google Scholar 

  • Lee, C.-H., Carpenter, B., Chou, W., Chu-Carroll, J., Reichl, W., Saad, A., & Zhou, Q. (2000). On natural language call routing. Speech Communication, 31, 309–320.

    Article  Google Scholar 

  • Lewis, J. R. (1993). Multipoint scales: Mean and median differences and observed significance levels. International Journal of Human-Computer Interaction, 5, 383–392.

    Article  Google Scholar 

  • Lewis, J.R. (2001). Psychometric properties of the mean opinion scale. In Proceedings of HCI International 2001: Usability Evaluation and Interface Design (pp. 149–153). Mahwah: Lawrence Erlbaum.

  • Lewis, J. R. (2011). Practical speech user interface design. Boca Raton: Taylor & Francis Group.

    Google Scholar 

  • Lewis, J. R. (2012). Usability testing. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (pp. 1267–1312). New York: John Wiley.

    Chapter  Google Scholar 

  • Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill.

    Google Scholar 

  • Patterson, M. L. (1996). Social behavior and social cognition: A parallel process approach. In J. L. Nye & A. M. Brower (Eds.), What’s social about social cognition? Research on socially shared cognition in small groups (pp. 87–105). Thousand Oaks: Sage.

    Chapter  Google Scholar 

  • Polkosky, M. D. (2005). Toward a social-cognitive psychology of speech technology: Affective responses to speech-based e-service. Unpublished doctoral dissertation. University of South Florida.

  • Polkosky, M. D. (2008). Machines as mediators: The challenge of technology for interpersonal communication theory and research. In E. Konjin (Ed.), Mediated interpersonal communication (pp. 34–57). New York: Routledge.

    Google Scholar 

  • Polkosky, M. D., & Lewis, J. R. (2003). Expanding the MOS: Development and psychometric evaluation of the MOS-R and MOS-X. International Journal of Speech Technology, 6, 161–182.

    Article  Google Scholar 

  • Sauro, J., & Lewis, J. R. (2009). Correlations among prototypical usability metrics: Evidence for the construct of usability. In Proceedings of CHI 2009 (pp. 1609–1618). Boston: ACM.

  • Sauro, J., & Lewis, J. R. (2012). Quantifying the user experience: Practical statistics for user research. Waltham: Morgan Kaufmann.

    Google Scholar 

  • Schmidt-Nielsen, A. (1995). Intelligibility and acceptability testing for speech technology. In A. Syrdal, R. Bennett, & S. Greenspan (Eds.), Applied speech technology (pp. 195–232). Boca Raton: CRC Press.

    Google Scholar 

  • Sudman, S., Bradburn, N. M., & Schwartz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass Publishers.

    Google Scholar 

  • Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • van Bezooijen, R., & van Heuven, V. (1997). Assessment of synthesis systems. In D. Gibbon, R. Moore, & R. Winski (Eds.), Handbook of standards and resources for spoken language systems (pp. 481–563). New York: Mouton de Gruyter.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IBM Corporation, Boca Raton, FL, USA

    James R. Lewis

  2. State Farm Mutual Automobile Insurance Company, Bloomington, IL, USA

    Mary L. Hardzinski

Authors
  1. James R. Lewis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mary L. Hardzinski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James R. Lewis.

Appendices

Appendix 1

The Standard SUI service quality (SUISQ) questionnaire

  1. 1.

    The system made me feel like I was in control.

  2. 2.

    The messages were repetitive.

  3. 3.

    The system gave me a good feeling about being a customer of this business.

  4. 4.

    The system used terms I am familiar with.

  5. 5.

    I could find what I needed without any difficulty.

  6. 6.

    The system used everyday words.

  7. 7.

    The system was organized and logical.

  8. 8.

    The system gave me more details than I needed.

  9. 9.

    The system spoke at a pace that was easy to follow.

  10. 10.

    The system would help me be productive.

  11. 11.

    The system seemed polite.

  12. 12.

    I could trust this system to work correctly.

  13. 13.

    I would be likely to use this system again.

  14. 14.

    The system’s voice was pleasant.

  15. 15.

    The system was too talkative.

  16. 16.

    The system’s voice sounded like people I hear on the radio or television.

  17. 17.

    I felt confident using this system.

  18. 18.

    The system’s voice sounded like a regular person.

  19. 19.

    The quality of this system made me want to remain a customer of this business.

  20. 20.

    The system’s voice sounded natural.

  21. 21.

    The system seemed courteous.

  22. 22.

    I felt like I had to wait too long for the system to stop talking so I could respond.

  23. 23.

    The system seemed friendly.

  24. 24.

    The system’s voice sounded enthusiastic or full of energy.

  25. 25.

    The system seemed professional in its speaking style.

1.1 SUISQ scales (based on specification in Polkosky 2005)

  • User goal orientation (UGO) average items 1, 3, 5, 10, 12, 13, 17, and 19.

  • Customer service behavior (CSB) average items 4, 6, 7, 9, 11, 21, 23, and 25.

  • Speech characteristics (SC) average items 14, 16, 18, 20, and 24.

  • Verbosity (V) average items 2, 8, 15, and 22 (to reverse score: Vr = 8 − V).

  • Overall average of UGO, CSB, SC, and Vr.

Appendix 2

See Table 4.

Table 4 The reduced SUI service quality (SUISQ-R) questionnaire
Full size table

Appendix 3

See Table 5.

Table 5 The maximally-reduced SUI service quality (SUISQ-MR) questionnaire
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lewis, J.R., Hardzinski, M.L. Investigating the psychometric properties of the Speech User Interface Service Quality questionnaire. Int J Speech Technol 18, 479–487 (2015). https://doi.org/10.1007/s10772-015-9289-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10772-015-9289-1

Keywords

Search

Navigation