Edward Tse
Saul Greenberg
Chia Shen
ABSTRACT
Most commercial software applications are designed for a single user using a keyboard/mouse over an upright monitor. Our interest is exploiting these systems so they work over a digital table. Mirroring what people do when working over traditional tables, we want to allow multiple people to interact naturally with the tabletop application and with each other via rich speech and hand gestures. In previous papers, we illustrated multi-user gesture and speech interaction on a digital table for geospatial applications -- Google Earth, Warcraft III and The Sims. In this paper, we describe our underlying architecture: GSI Demo. First, GSI Demo creates a run-time wrapper around existing single user applications: it accepts and translates speech and gestures from multiple people into a single stream of keyboard and mouse inputs recognized by the application. Second, it lets people use multimodal demonstration -- instead of programming -- to quickly map their own speech and gestures to these keyboard/mouse inputs. For example, continuous gestures are trained by saying "Computer, when I do [one finger gesture], you do [mouse drag]". Similarly, discrete speech commands can be trained by saying "Computer, when I say [layer bars], you do [keyboard and mouse macro]". The end result is that end users can rapidly transform single user commercial applications into a multi-user, multimodal digital tabletop system.
- Boyle, M. and Greenberg, S. Rapidly Prototyping Multimedia Groupware. Proc Distributed Multimedia Systems (DMS'05), Knowledge Systems Institute, 2005.Google Scholar
- Cao, X. and Balakrishnan, R. Evaluation of an online adaptive gesture interface with command prediction. Proc Graphics Interface, 2005. 187--194. Google ScholarDigital Library
- Cohen, P. R., Coulston, R. and Krout, K., Multimodal interaction during multiparty dialogues: Initial results. Proc IEEE Int'l Conf. Multimodal Interfaces, 2002, 448--452. Google ScholarDigital Library
- Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L. and Clow, J., QuickSet: Multimodal interaction for distributed applications. Proc. ACM Multimedia, 1997, 31--40. Google ScholarDigital Library
- Cypher, A. Watch What I Do: Programming by Demonstration. MIT Press, 1993. Google ScholarDigital Library
- Dietz, P. H., Leigh, D. L., DiamondTouch: A Multi-User Touch Technology, Proc ACM UIST, 2001. 219--226. Google ScholarDigital Library
- Greenberg, S. and Boyle, M. Customizable physical interfaces for interacting with conventional applications. Proc. ACM UIST Conference, 2002, 31--40. Google ScholarDigital Library
- Gutwin, C. and Greenberg, S. Design for individuals, design for groups: Tradeoffs between power and workspace awareness. Proc ACM CSCW, 1998, 207--216. Google ScholarDigital Library
- Lunsford, R., Oviatt, S., and Coulston, R., Audio-visual cues distinguishing self- from system-directed speech in younger and older adults. Proc. ICMI, 2005, 167--174. Google ScholarDigital Library
- McGee, D. R. and Cohen, P. R., Creating tangible interfaces by augmenting physical objects with multimodal language. Proc ACM Conf. Intelligent User Interfaces, 2001, 113--119. Google ScholarDigital Library
- Oviatt, S. L. Ten myths of multimodal interaction, Comm. ACM, 42(11), 1999, 74--81. Google ScholarDigital Library
- Tse, E., Shen, C., Greenberg, S. and Forlines, C. Enabling Interaction with Single User Applications through Speech and Gestures on a Multi-User Tabletop. Proc. AVI 2006. Google ScholarDigital Library
- Tse, E., Greenberg, S., Shen, C. and Forlines, C. (2006) Multimodal Multiplayer Tabletop Gaming. Proc. Workshop on Pervasive Games 2006.Google Scholar
- Wu, M., Shen, C., Ryall, K., Forlines, C., Balakrishnan, R., Gesture Registration, Relaxation, and Reuse for Multi-Point Direct-Touch Surfaces, Proc. TableTop2006. 183--190. Google ScholarDigital Library
Index Terms
- GSI demo: multiuser gesture/speech interaction over digital tables by wrapping single user applications
Recommendations
Sensor synaesthesia: touch in motion, and motion in touch
CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsWe explore techniques for hand-held devices that leverage the multimodal combination of touch and motion. Hybrid touch + motion gestures exhibit interaction properties that combine the strengths of multi-touch with those of motion-sensing. This affords ...
VoicePen: augmenting pen input with simultaneous non-linguisitic vocalization
ICMI '07: Proceedings of the 9th international conference on Multimodal interfacesThis paper explores using non-linguistic vocalization as an additional modality to augment digital pen input on a tablet computer. We investigated this through a set of novel interaction techniques and a feasibility study. Typically, digital pen users ...
PERCs Demo: Persistently Trackable Tangibles on Capacitive Multi-Touch Displays
ITS '15: Proceedings of the 2015 International Conference on Interactive Tabletops & SurfacesTangible objects on capacitive multi-touch surfaces are usually only detected while the user is touching them. When the user lets go of such a tangible, the system cannot distinguish whether the user just released the tangible, or picked it up and ...
Comments