David Traum
ABSTRACT
Immersive virtual worlds are increasingly being used for education, training, and entertainment, and virtual humans that can interact with human users in these worlds play many important roles. However, current computational models of dialogue do not address the issues that arise with face-to-face communication situated in three-dimensional worlds, such as the proximity and attentional focus of others, the ability to maintain multi-party conversations, and the interplay between speech and nonverbal signals. This paper presents a new model that integrates and extends prior work on spoken dialogue and embodied conversational agents, and describes an initial implementation that has been applied to training in virtual reality.
- J. Allwood. Obligations and options in dialogue. Think Quarterly, 3:9--18, 1994Google Scholar
- J. Allwood, J. Nivre, and E. Ahlsen. On the semantics and pragmatics of linguistic feedback. Journal of Semantics, 9, 1992Google ScholarCross Ref
- M. Argyle and M. Cook. Gaze and Mutual Gaze. Cambridge University Press, Cambridge, 1976Google Scholar
- M. Baker. A model for negotiation in teaching-learning dialogues. Journal of Artificial Intelligence in Education, 5(2):199--254, 1994 Google ScholarDigital Library
- G. Ball, D. Ling, D. Kurlander, J. Miller, D. Pugh, T. Skelly, A. Stankosky, D. Thiel, M. van Dantzich, and T. Wax. Lifelike computer characters: the persona project at microsoft. In J. Bradshaw, editor, Software Agents. AAAI/MIT Press, Menlo Park, CA, 1997 Google ScholarDigital Library
- Boston Dynamics. PeopleShop 1.4 User Manual, 2000Google Scholar
- H. Bunt. Interaction management functions and context representation requirements. In Proceedings of the Twente Workshop on Language Technology: Dialogue Management in Natural Language Systems (TWLT 11), pages 187--198, 1996Google Scholar
- J. Cassell, T. Bickmore, L. Campbell, H. Vilhjálmsson, and H. Yan. Conversation as a system framework: Designing embodied conversational agents. In J. Cassell, J. Sullivan, S. Prevost, and E. Churchill, editors, Embodied Conversational Agents. MIT Press, Cambridge, MA, 2000 Google ScholarDigital Library
- J. Cassell, C. Pelachaud, N. Badler, M. Steedman, B. Achorn, T. Becket, B. Douville, S. Prevost, and M. Stone. Animated conversation: Rule-based generation of facial expression, gesture and spoken intonation for multiple conversational agents. In Proceedings of ACM SIGGRAPH '94, pages 413--420, Reading, MA, 1994. Addison-Wesley Google ScholarDigital Library
- J. Cassell, J. Sullivan, S. Prevost, and E. Churchill, editors. Embodied Conversational Agents. MIT Press, Cambridge, MA, 2000Google Scholar
- J. Cassell and K. R. Thórisson. The power of a nod and a glance: Envelope vs. emotional feedback in animated conversational agents. Applied Artificial Intelligence, 13:519--538, 1999Google ScholarCross Ref
- J. Cassell and H. Vilhjálmsson. Fully embodied conversational avatars: Making communicative behaviors autonomous. Autonomous Agents and Multi-Agent Systems, 2:45--64, 1999 Google ScholarDigital Library
- J. Cassell, H. Vilhjálmsson, and T. Bickmore. Beat: the behavior expression animation toolkit. In Proceedings of ACM SIGGRAPH, pages 477--486, New York, 2001. ACM Press Google ScholarDigital Library
- H. H. Clark. Managing problems in speaking. Speech Communication, 15:243--250, 1994 Google ScholarDigital Library
- H. H. Clark. Using Language. Cambridge University Press, Cambridge, England, 1996Google Scholar
- H. H. Clark and E. F. Schaefer. Contributing to discourse. Cognitive Science, 13:259--294, 1989Google ScholarCross Ref
- P. Dillenbourg, D. Traum, and D. Schneider. Grounding in multi-modal task-oriented collaboration. In Proceedings of the European Conference on AI in Education, 1996Google Scholar
- Discourse Resource Initiative. Standards for dialogue coding in natural language processing. Report no. 167, Dagstuhl-Seminar, 1997Google Scholar
- S. Duncan, Jr. Some signals and rules for taking speaking turns in conversations. In S. Weitz, editor, Nonverbal Communication, pages 298--311. Oxford University Press, 1974Google Scholar
- M. Fleischman and E. Hovy. Emotional variation in speech-based natural language generation. In Proceedings of The Second International Natural Language Generation Conference (INLG'02), June 2002Google Scholar
- W. L. Johnson, J. W. Rickel, and J. C. Lester. Animated pedagogical agents: Face-to-face interaction in interactive learning environments. International Journal of Artificial Intelligence in Education, 11:47--78, 2000Google Scholar
- Y. Katagiri and A. Shimojima. Display acts in grounding negotiations. In Proceedings of Gotalog 2000, the 4th Workshop on the Semantics and Pragmatics of Dialogue, pages 195--198, 2000Google Scholar
- A. Kendon. Some relationships between body motion and speech. In A. Siegman and B. Pope, editors, Studies in Dyadic Communication, pages 177--210. Pergamon Press, New York, 1972Google ScholarCross Ref
- A. Kendon. A description of some human greetings. In R. Michael and J. Crook, editors, Comparative Ecology and Behavior of Primates, pages 591--668. Academic Press, New York, 1973Google Scholar
- J. E. Laird, A. Newell, and P. S. Rosenbloom. Soar: An architecture for general intelligence. Artificial Intelligence, 33(1):1--64, 1987 Google ScholarDigital Library
- S. Larsson and D. Traum. Information state and dialogue management in the TRINDI dialogue move engine toolkit. Natural Language Engineering, 6:323--340, September 2000 Google ScholarDigital Library
- J. C. Lester, J. L. Voerman, S. G. Towns, and C. B. Callaway. Deictic believability: Coordinating gesture, locomotion, and speech in lifelike pedagogical agents. Applied Artificial Intelligence, 13:383--414, 1999Google ScholarCross Ref
- W. C. Mann and S. A. Thompson. Rhetorical structure theory: A theory of text organization. Technical Report ISI/RS-87 190, USC, Information Sciences Institute, June 1987Google Scholar
- S. Marsella and J. Gratch. A step towards irrationality: using emotion to change belief. In Proceedings of the First International Joint Conference on Autonomous Agents and Multi-Agent Systems, June 2002 Google ScholarDigital Library
- C. Matheson, M. Poesio, and D. Traum. Modelling grounding and discourse obligations using update rules. In Proceedings of the First Conference of the North American Chapter of the Association for Computational Linguistics, 2000 Google ScholarDigital Library
- C. H. Nakatani and D. R. Traum. Coding discourse structure in dialogue (version 1.0). Technical Report UMIACS-TR-99-03, University of Maryland, 1999Google Scholar
- A. Newell. Unified Theories of Cognition. Harvard University Press, Cambridge, MA, 1990 Google ScholarDigital Library
- D. Novick. Control of Mixed-Initiative Discourse Through Meta-Locutionary Acts: A Computational Model. PhD thesis, University of Oregon, 1988. Also available as U. Oregon Computer and Information Science Tech Report CIS-TR-88-18 Google ScholarDigital Library
- C. Pelachaud, N. I. Badler, and M. Steedman. Generating facial expressions for speech. Cognitive Science, 20(1), 1996Google Scholar
- M. Poesio and D. R. Traum. Towards an axiomatization of dialogue acts. In Proceedings of Twendial'98, 13th Twente Workshop on Language Technology: Formal Semantics and Pragmatics of Dialogue, 1998Google Scholar
- J. Rickel, J. Gratch, R. Hill, S. Marsella, and W. Swartout. Steve goes to Bosnia: Towards a new generation of virtual humans for interactive experiences. In AAAI Spring Symposium on Artificial Intelligence and Interactive Entertainment, 2001Google Scholar
- J. Rickel and W. L. Johnson. Animated agents for procedural training in virtual reality: Perception, cognition, and motor control. Applied Artificial Intelligence, 13:343--382, 1999Google ScholarCross Ref
- J. Rickel and W. L. Johnson. Virtual humans for team training in virtual reality. In Proceedings of the Ninth International Conference on Artificial Intelligence in Education, pages 578--585. IOS Press, 1999Google Scholar
- J. Rickel and W. L. Johnson. Task-oriented collaboration with embodied agents in virtual worlds. In J. Cassell, J. Sullivan, S. Prevost, and E. Churchill, editors, Embodied Conversational Agents. MIT Press, Cambridge, MA, 2000 Google ScholarDigital Library
- E. A. Schegloff and H. Sacks. Opening up closings. Semiotica, 7:289--327, 1973Google Scholar
- C. L. Sidner. An artificial discourse language for collaborative negotiation. In Proceedings of the Fourteenth National Conference of the American Association for Artificial Intelligence (AAAI-94), pages 814--819, 1994 Google ScholarDigital Library
- W. Swartout, R. Hill, J. Gratch, W. Johnson, C. Kyriakakis, K. Labore, R. Lindheim, S. Marsella, D. Miraglia, B. Moore, J. Morie, J. Rickel, M. Thiebaux, L. Tuch, R. Whitney, and J. Douglas. Toward the holodeck: Integrating graphics, sound, character and story. In Proceedings of 5th International Conference on Autonomous Agents, 2001 Google ScholarDigital Library
- D. R. Traum. A Computational Theory of Grounding in Natural Language Conversation. PhD thesis, Department of Computer Science, University of Rochester, 1994. Also available as TR 545, Department of Computer Science, University of Rochester Google ScholarDigital Library
- D. R. Traum and J. F. Allen. Discourse obligations in dialogue processing. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics, pages 1--8, 1994 Google ScholarDigital Library
- D. R. Traum and E. A. Hinkelman. Conversation acts in task-oriented spoken dialogue. Computational Intelligence, 8(3):575--599, 1992Google ScholarDigital Library
- M. A. Walker and S. Whittaker. Mixed initiative in dialogue: An investigation into discourse segmentation. In Proceedings ACL-90, pages 70--78, 1990 Google ScholarDigital Library
- S. Whittaker and P. Stenton. Cues and control in expert-client dialogues. In Proceedings ACL-88, pages 123--130, 1988 Google ScholarDigital Library
Index Terms
- Embodied agents for multi-party dialogue in immersive virtual worlds
Recommendations
Communication Behavior in Embodied Virtual Reality
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsEmbodied virtual reality faithfully renders users' movements onto an avatar in a virtual 3D environment, supporting nuanced nonverbal behavior alongside verbal communication. To investigate communication behavior within this medium, we had 30 dyads ...
Human-robot collaborative tutoring using multiparty multimodal spoken dialogue
HRI '14: Proceedings of the 2014 ACM/IEEE international conference on Human-robot interactionIn this paper, we describe a project that explores a novel experimental setup towards building a spoken, multi-modally rich, and human-like multiparty tutoring robot. A human-robot interaction setup is designed, and a human-human dialogue corpus is ...
Talking to virtual humans: dialogue models and methodologies for embodied conversational agents
ZiF'06: Proceedings of the Embodied communication in humans and machines, 2nd ZiF research group international conference on Modeling communication with robots and virtual humansVirtual Humans are artificial characters who look and act like humans, but inhabit a simulated environment. One important aspect of many virtual humans is their communicative dialogue ability. In this paper we outline a methodology for study of dialogue ...
Comments