Skip to main content
SpringerLink
Log in

Design and usability evaluation of multimodal interaction with finite state machines: a conceptual framework

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Designing multimodal systems that take the best advantage of multiple error prone recognition-based technologies, such as speech and gesture recognition, is difficult. To guarantee a robust and usable interaction, careful consideration must be given to the choice of modalities of interaction made available, their allocation to tasks, and the range of modality combinations allowed. In this paper, we present a conceptual framework for evaluating the usability and robustness of different interaction modality combinations early in the process of designing a multimodal system. First, models of multimodal elementary commands are built using Finite State Machine (FSM) modelling. Second, the most usable representations of multimodal commands are flagged up in the model by activating the FSM-based model with real or simulated user inputs. The output of this step is a collection of FSMs that more closely represent user preferences and natural behaviour. Third, the disambiguating potential of sets of multimodal commands is evaluated by observing the models response to simulated recognition errors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Bouchet J, Nigay L, Ganille T (2004) Icare software components for rapidly developing multimodal interfaces. In: ICMI, October 2004, pp 251–258

  2. Bourguet M-L (2002) A toolkit for creating and testing multimodal interface designs. In: Posters and demos from the 15th annual ACM symposium on user interface software and technology, October 2002, pp 29–30

  3. Bourguet M-L (2003) Designing and prototyping multimodal commands. In: INTERACT, September 2003, pp 717–720

  4. Bourguet M-L (2003) How finite state machines can be used to build error free multimodal interaction systems. In: The 17th British HCI group annual conference, vol 2, September 2003, pp 81–84

  5. Bourguet M-L (2006) A study into multimodal behaviour in error correction. In: Workshop on multimodal corpora: from multimodal behaviour theories to usable models. In association with the 5th international conference on language resources and evaluation, May 2006

  6. Bourguet M-L, Ando A (1998) Synchronisation of speech and hand gestures during multimodal human computer interaction. In: ACM CHI’98, April 1998, pp 241–242

  7. Bernsen NO, (2002) In: Multimodality in language and speech systems. From theory to design support tool. Kluwer Academic, Dordrecht, pp 93–148

    Google Scholar 

  8. Clow J, Oviatt SL (1998) Stamp: an automated tool for analysis of multimodal system performance. In: International conference on spoken language processing, December 1998

  9. Coutaz GFC, Paterno F, Nigay L (1993) A comparison of approaches for specifying multimodal interactive systems. In: ERCIM workshop on multimodal human-computer interaction, November 1993

  10. Coutaz C, Nigay L, Salber D, Blandford A, May J, Young R (1995) Four easy pieces for assessing the usability of multimodal interaction: the care properties. In: INTERACT 95, June 1995, pp 115–120

  11. Duarte C, Carrico L (2006) A conceptual framework for developing adaptive multimodal applications. In: The 11th international conference on intelligent user interfaces, January 2006

  12. Flippo AKF, Marsic I (2003) A framework for rapid development of multimodal interfaces. In: The 5th international conference on multimodal interfaces, November 2003

  13. Johnston M, Bangalore S (2005) Finite-state multimodal integration and understanding. Nat Lang Eng 11:159–187

    Article  Google Scholar 

  14. Karat CM, Halverson C, Horn D, Karat J (1999) Patterns of entry and correction in large vocabulary contentious speech recognition systems. In: ACM CHI’99, May 1999

  15. Mankoff J, Hudson S, Abowd G (2000) Interaction techniques for ambiguity resolution in recognition-based interfaces. In: ACM UIST00, November 2000, pp 11–20

  16. Navarre D, Palanque P, Dragicevic P, Bastide R (2006) An approach integrating two complementary model-based environments for the construction of multimodal interactive applications. In: Interacting with computers, vol 18, September 2006

  17. Nigay L, Coutaz J (1997) Multifeature systems: The care properties and their impact on software design. In: Intelligence and multimodality in multimedia interfaces, July 1997

  18. Oviatt S (1999) Mutual disambiguation of recognition errors in a multimodal architecture. In: ACM CHI’99, 1999, pp 576–583

  19. Oviatt S (1999) Ten myths of multimodal interaction. Commun ACM 42(11):74–81

    Article  Google Scholar 

  20. Oviatt S (2000) Taming recognition errors with a multimodal interface. Commun ACM 43(9):45–51

    Article  Google Scholar 

  21. Oviatt S (2002) Breaking the robustness barrier: recent progress on the design of robust multimodal systems. Adv Comput 56:306–343

    Google Scholar 

  22. Oviatt S, Coulston R, Lunsford R (2004) When do we interact multimodally? cognitive load and multimodal communication patterns, In: ICMI 2004, October 2004

  23. Palanque P, Schyn A (2003) A model-based approach for engineering multimodal interactive systems. In: INTERACT’03, September 2003

  24. Sinha AK, Landay JA (2003) Capturing user tests in a multimodal, multidevice informal prototyping tool. In: 5th international conference on multimodal interfaces, November 2003

  25. Suhm B, Myers B, Waibel A (1999) Model-based and empirical evaluation of multimodal interactive error correction. In: ACM CHI’99, May 1999, pp 584–591

  26. VanGurp J, Bosch J (1999) On the implementation of finite state machines. In: IASTED 3rd international conference on software engineering and applications, October 1999, pp 172–178

  27. Vuurpijl L, ten Bosch L, Rossignol S, Neumann A, Pflegfer N, Engel R (2004) Evaluation of multimodal dialogue systems. In: The 4th international conference on language resources and evaluation, May 2004

  28. Wasinger R (2007) In: Multi modal interaction with mobile devices: fusing a broad spectrum of modality combinations. IOS, Amsterdam

    Google Scholar 

  29. Wasserman A (1985) Extending state transition diagrams for the specification of human-computer interaction. IEEE Trans Softw Eng 11(8):699–713

    Article  Google Scholar 

  30. W3C (2007) Emma: extensible multimodal annotation markup language. December 2007

Download references

Author information

Authors and Affiliations

  1. Computer Science Department, Queen Mary University of London, London, UK

    Marie-Luce Bourguet & Jaeseung Chang

Authors
  1. Marie-Luce Bourguet
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jaeseung Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marie-Luce Bourguet.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bourguet, ML., Chang, J. Design and usability evaluation of multimodal interaction with finite state machines: a conceptual framework. J Multimodal User Interfaces 2, 53–60 (2008). https://doi.org/10.1007/s12193-008-0004-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-008-0004-2

Keywords

Search

Navigation