Euan Freeman
Stephen Brewster
ABSTRACT
When users want to interact with an in-air gesture system, they must first address it. This involves finding where to gesture so that their actions can be sensed, and how to direct their input towards that system so that they do not also affect others or cause unwanted effects. This is an important problem which lacks a practical solution. We present an interaction technique which uses multimodal feedback to help users address in-air gesture systems. The feedback tells them how ("do that") and where ("there") to gesture, using light, audio and tactile displays. By doing that there, users can direct their input to the system they wish to interact with, in a place where their gestures can be sensed. We discuss the design of our technique and three experiments investigating its use, finding that users can "do that" well (93.2%-99.9%) while accurately (51mm-80mm) and quickly (3.7s) finding "there".
Supplemental Material
- Leap Motion Controller. https://www.leapmotion.com/Google Scholar
- Precision Microdrives. http://www.precisionmicrodrives.com/Google Scholar
- Adafruit. Adafruit NeoPixel LEDs. http://adafruit.com/products/1506Google Scholar
- Alan Armstrong and Johann Issartel. 2014. Sensorimotor synchronization with audio-visual stimuli: limited multisensory integration. Experimental brain research 232, 11 (2014), 3453--63. DOI: http://dx.doi.org/10.1007/s00221-014--4031--9Google Scholar
- Md Tanvir Islam Aumi, Sidhant Gupta, Mayank Goel, Eric Larson, and Shwetak Patel. 2013. DopLink: Using the Doppler Effect for Multi-Device Interaction. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing UbiComp '13. ACM Press, 583--586. DOI: http://dx.doi.org/10.1145/2493432.2493515 Google ScholarDigital Library
- Victoria Bellotti, Maribeth Back, W. Keith Edwards, Rebecca E. Grinter, Austin Henderson, and Cristina Lopes. 2002. Making Sense of Sensing Systems: Five Questions for Designers and Researchers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '02. ACM Press, 415--422. DOI: http://dx.doi.org/10.1145/503376.503450 Google ScholarDigital Library
- Peter Bennett, Kirsten Cater, and Mike Fraser. 2015. Resonant Bits: Harmonic Interaction with Virtual Pendulums. In Proceedings of the 9th International Conference on Tangible, Embedded, and Embodied Interaction - TEI '15. ACM Press, 49--52. DOI: http://dx.doi.org/10.1145/2677199.2680569 Google ScholarDigital Library
- Matthias Budde, Matthias Berning, Christopher Baumgärtner, Florian Kinn, Timo Kopf, Sven Ochs, Frederik Reiche, Till Riedel, and Michael Beigl. 2013. Point & Control Interaction in Smart Environments: You Only Click Twice. In Adjunct Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing - UbiComp '13 Adjunct. ACM Press, 303--306. DOI: http://dx.doi.org/10.1145/2494091.2494184 Google ScholarDigital Library
- Alex Butler, Shahram Izadi, and Steve Hodges. 2008. SideSight: Multi-Interaction Around Small-touch Devices. In Proceedings of the 21st Symposium on User Interface Software and Technology - UIST '08. ACM Press, 201--204. DOI: http://dx.doi.org/10.1145/1449715.1449746 Google ScholarDigital Library
- Thomas Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, and Sriram Subramanian. 2013. UltraHaptics: Multi-Point Mid-Air Haptic Feedback for Touch Surfaces. In Proceedings of the 26th Symposium on User Interface Software and Technology - UIST '13. ACM Press, 505--514. DOI: http://dx.doi.org/10.1145/2501988.2502018 Google ScholarDigital Library
- Emiko Charbonneau, Charles E. Hughes, and Joseph J. Laviola Jr. 2010. Vibraudio Pose: An Investigation of Non-Visual Feedback Roles for Body Controlled Video Games. In Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games Sandbox 2010. ACM Press, 79--84. DOI: http://dx.doi.org/10.1145/1836135.1836147 Google ScholarDigital Library
- William Delamare, Celine Coutrix, and Laurence Nigay. 2012. Pointing in the Physical World for Light Source Selection. In Proceedings of the Designing Interactive Lighting Workshop. http://hal.archives-ouvertes.fr/hal-00757683/Google Scholar
- William Delamare, Céline Coutrix, and Laurence Nigay. 2013. Mobile Pointing Task in the Physical World: Balancing Focus and Performance while Disambiguating. In Proceedings of the 15th International Conference on Human-Computer Interaction with Mobile Devices & Services - Mobile HCI '13. ACM Press, 89--98. DOI: http://dx.doi.org/10.1145/2493190.2493232 Google ScholarDigital Library
- Jean-Daniel Fekete, Niklas Elmqvist, and Yves Guiard. 2009. Motion-Pointing: Target Selection using Elliptical Motions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '09. ACM Press, 289--298. DOI: http://dx.doi.org/10.1145/1518701.1518748 Google ScholarDigital Library
- Euan Freeman, Stephen Brewster, and Vuokko Lantz. 2014. Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions. In Proceedings of the 16th International Conference on Multimodal Interaction ICMI '14. ACM Press, 419--426. DOI:http://dx.doi.org/10.1145/2663204.2663280 Google ScholarDigital Library
- Euan Freeman, Stephen Brewster, and Vuokko Lantz. 2015. Interactive Light Feedback: Illuminating Above-Device Gesture Interfaces. In Proceedings of INTERACT '15 in LNCS 2929. 478--481. DOI: http://dx.doi.org/10.1007/978--3--319--22723--8_42Google ScholarCross Ref
- William W. Gaver, Jacob Beaver, and Steve Benford. 2003. Ambiguity as a Resource for Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '03. ACM Press, 233--240. DOI: http://dx.doi.org/10.1145/642651.642653 Google ScholarDigital Library
- Ivan Golod, Felix Heidrich, Christian Möllering, and Martina Ziefle. 2013. Design Principles of Hand Gesture Interfaces for Microinteractions. In Proceedings of the 6th International Conference on Designing Pleasurable Products and Interfaces DPPI '13. ACM Press, 11--20. DOI: http://dx.doi.org/10.1145/2513506.2513508 Google ScholarDigital Library
- Chris Harrison and Scott E. Hudson. 2009. Abracadabra: Wireless, High-Precision, and Unpowered Finger Input for Very Small Mobile Devices. In Proceedings of the 22nd Symposium on User Interface Software and Technology - UIST '09. ACM Press, 121--124. DOI: http://dx.doi.org/10.1145/1622176.1622199 Google ScholarDigital Library
- Khalad Hasan, David Ahlström, and Pourang Irani. 2013. AD-Binning: Leveraging Around Device Space for Storing, Browsing and Retrieving Mobile Device Content. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '13. ACM Press, 899--908. DOI: http://dx.doi.org/10.1145/2466110.2466115 Google ScholarDigital Library
- Khalad Hasan, David Ahlström, and Pourang Irani. 2015. SAMMI: A Spatially-Aware Multi-Mobile Interface for Analytic Map Navigation Tasks. Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services MobileHCI '15 (2015), 36--45. DOI: http://dx.doi.org/10.1145/2785830.2785850 Google ScholarDigital Library
- Michael J. Hove, Merle T. Fairhurst, Sonja A. Kotz, and Peter E. Keller. 2013. Synchronizing with auditory and visual rhythms: an fMRI assessment of modality differences and modality appropriateness. NeuroImage 67 (2013), 313--321. DOI: http://dx.doi.org/10.1016/j.neuroimage.2012.11.032Google ScholarCross Ref
- Michael J. Hove, Michael J. Spivey, and Carol L. Krumhansl. 2010. Compatibility of Motion Facilitates Visuomotor Synchronization. Journal of experimental psychology. Human perception and performance 36, 6 (2010), 1525--1534. DOI: http://dx.doi.org/10.1037/a0019059Google ScholarCross Ref
- Takayuki Iwamoto, Mari Tatezono, and Hiroyuki Shinoda. 2008. Non-contact method for producing tactile sensation using airborne ultrasound. In Proceedings of EuroHaptics 2008. Springer, 504--513. http://www.springerlink.com/index/X41J595757401387.pdf Google ScholarDigital Library
- Brett Jones, Rajinder Sodhi, David Forysth, Brian Bailey, and Giuliano Maciocci. 2012. Around Device Interaction for Multiscale Navigation. In Proceedings of the 14th International Conference on Human Computer Interaction with Mobile Devices and Services MobileHCI '12. ACM Press, 83--92. http://dl.acm.org/citation.cfm?id=2371574.2371589 Google ScholarDigital Library
- Rick Kjeldsen and Jacob Hartman. 2001. Design Issues for Vision-Based Computer Interaction Systems. In Proceedings of the Workshop on Perceptive User Interfaces - PUI '01. ACM Press, 1--8. DOI: http://dx.doi.org/10.1145/971478.971511 Google ScholarDigital Library
- Sven Kratz and Michael Rohs. 2009. HoverFlow: Expanding the Design Space of Around-Device Interaction. In Proceedings of the 11th International Conference on Human Computer Interaction with Mobile Devices and Services - MobileHCI '09. ACM Press, Article 4. http://dl.acm.org/citation.cfm?id=1613864 Google ScholarDigital Library
- David Ledo, Saul Greenberg, Nicolai Marquardt, and Sebastian Boring. 2015. Proxemic-Aware Controls: Designing Remote Controls for Ubiquitous Computing Ecologies. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services - MobileHCI '15. ACM Press, 187--198. DOI: http://dx.doi.org/10.1145/2785830.2785871 Google ScholarDigital Library
- Paul P. Maglio, Teenie Matlock, Christopher S. Campbell, Shumin Zhai, and Barton A. Smith. 2000. Gaze and Speech in Attentive User Interfaces. In Proceedings of the International Conference on Multimodal Interaction in LNCS 1948 ICMI 2000. Springer, 1--7. DOI: http://dx.doi.org/10.1007/3--540--40063-X Google ScholarDigital Library
- Sylvain Malacria, Eric Lecolinet, and Yves Guiard. 2010. Clutch-Free Panning and Integrated Pan-Zoom Control on Touch-Sensitive Surfaces: The CycloStar Approach. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '10. ACM Press, 2615--2624. DOI: http://dx.doi.org/10.1145/1753326.1753724 Google ScholarDigital Library
- Nicolai Marquardt, Till Ballendat, Sebastian Boring, Saul Greenberg, and Ken Hinckley. 2012. Gradual Engagement: Facilitating Information Exchange between Digital Devices as a Function of Proximity. In Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces ITS '12. ACM Press, 31--40. DOI: http://dx.doi.org/10.1145/2396636.2396642 Google ScholarDigital Library
- Microsoft. Kinect for Xbox 360. http://www.xbox.com/en-US/xbox-360/accessories/kinectGoogle Scholar
- Cecily Morrison, Neil Smyth, Robert Corish, Kenton O'Hara, and Abigail Sellen. 2014. Collaborating with Computer Vision Systems: An Exploration of Audio Feedback. In Proceedings of the 2014 Conference on Designing Interactive Systems DIS '14. ACM Press, 229--238. DOI: http://dx.doi.org/10.1145/2598510.2598519 Google ScholarDigital Library
- Sarah Morrison-Smith and Jaime Ruiz. 2014. Using Audio Cues to Support Motion Gesture Interaction on Mobile Devices. In CHI '14 Extended Abstracts on Human Factors in Computing Systems. ACM Press, 1621--1626. DOI: http://dx.doi.org/10.1145/2559206.2581236 Google ScholarDigital Library
- Heiko Müller, Andreas Löcken, Wilko Heuten, and Susanne Boll. 2014. Sparkle: An Ambient Light Display for Dynamic Off-Screen Points of Interest. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction - NordiCHI '14. ACM Press, 51--60. DOI: http://dx.doi.org/10.1145/2639189.2639205 Google ScholarDigital Library
- Nest. 2012. The Nest Thermostat. (2012). http://nest.com/thermostatGoogle Scholar
- Kenton O'Hara, Gerardo Gonzalez, Abigail Sellen, Graeme Penney, Andreas Varnavas, Helena Mentis, Antonio Criminisi, Robert Corish, Mark Rouncefield, Neville Dastur, and Tom Carrell. 2014. Touchless Interaction in Surgery. Commun. ACM 57, 1 (2014), 70--77. DOI: http://dx.doi.org/10.1145/2541883.2541899 Google ScholarDigital Library
- Qian Qin, Michael Rohs, and Sven Kratz. 2011. Dynamic Ambient Lighting for Mobile Devices. In Adjunct Proceedings of the 24th Symposium on User Interface Software and Technology - UIST '11 Adjunct. ACM Press, 51--52. DOI:http://dx.doi.org/10.1145/2046396.2046418 Google ScholarDigital Library
- Bruno H. Repp. 2006. Rate Limits of Sensorimotor Synchronization. Advances in Cognitive Psychology 2, 2 (2006), 163--181. DOI: http://dx.doi.org/10.2478/v10053-008-0053--9Google ScholarCross Ref
- Dominik Schmidt, David Molyneaux, and Xiang Cao. 2012. PICOntrol: Using a Handheld Projector for Direct Control of Physical Devices through Visible Light. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology - UIST '12. ACM Press, 379--388. DOI: http://dx.doi.org/10.1145/2380116.2380166 Google ScholarDigital Library
- Julia Schwarz, Charles Claudius Marais, Tommer Leyvand, Scott E. Hudson, and Jennifer Mankoff. 2014. Combining Body Pose, Gaze, and Gesture to Determine Intention to Interact in Vision-Based Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '14. ACM Press, 3443--3452. DOI: http://dx.doi.org/10.1145/2556288.2556989 Google ScholarDigital Library
- Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges. 2014. In-air Gestures Around Unmodified Mobile Devices. In Proceedings of the 27th Symposium on User Interface Software and Technology - UIST '14. ACM Press, 319--329. DOI: http://dx.doi.org/10.1145/2642918.2647373 Google ScholarDigital Library
- Srinath Sridhar, Anna Maria Feit, Christian Theobalt, and Antti Oulasvirta. 2015. Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '15. ACM Press, to appear. DOI:http://dx.doi.org/10.1145/2702123.2702136 Google ScholarDigital Library
- Breanna E. Studenka and Howard N. Zelaznik. 2011. Synchronization in repetitive smooth movement requires perceptible events. Acta Psychologica 136, 3 (2011), 432--441. DOI: http://dx.doi.org/10.1016/j.actpsy.2011.01.011Google ScholarCross Ref
- Breanna E. Studenka, Howard N. Zelaznik, and Ramesh Balasubramaniam. 2012. The distinction between tapping and circle drawing with and without tactile feedback: an examination of the sources of timing variance. Quarterly Journal of Experimental Psychology 65, 6 (2012), 1086--1100. DOI: http://dx.doi.org/10.1080/17470218.2011.640404Google ScholarCross Ref
- Colin Swindells, Kori M. Inkpen, John C. Dill, and Melanie Tory. 2002. That one there! Pointing to establish device identity. In Proceedings of the 15th Annual Symposium on User Interface Software and Technology -- UIST '02. ACM Press, 151--160. DOI: http://dx.doi.org/10.1145/571985.572007 Google ScholarDigital Library
- Robrecht P. R. D. van der Wel, Dagmar Sternad, and David A. Rosenbaum. 2009. Moving the Arm at Different Rates: Slow Movements are Avoided. Journal of Motor Behavior 26, 5 (jan 2009), 29--36. DOI: http://dx.doi.org/10.1080/00222890903267116Google Scholar
- Jo Vermeulen, Kris Luyten, Karin Coninx, Nicolai Marquardt, and Jon Bird. 2015. Proxemic Flow: Dynamic Peripheral Floor Visualizations for Revealing and Mediating Large Surface Interactions. In Proceedings of INTERACT '15 in LNCS 9299. 264--281. DOI: http://dx.doi.org/10.1007/978--3--319--22723--8{_}22Google ScholarCross Ref
- Robert Walter, Gilles Bailly, and Jörg Müller. 2013. StrikeAPose: Revealing Mid-Air Gestures on Public Displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '13. ACM Press, 841--850. DOI: http://dx.doi.org/10.1145/2470654.2470774 Google ScholarDigital Library
- Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only ANOVA Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '11. ACM Press, 143--146. DOI: http://dx.doi.org/10.1145/1978942.1978963 Google ScholarDigital Library
- Xing-dong Yang, Khalad Hasan, Neil Bruce, and Pourang Irani. 2013. Surround-See: Enabling Peripheral Vision on Smartphones during Active Use. In Proceedings of the 26th Symposium on User Interface Software and Technology - UIST '13. ACM Press, 291--300. DOI: http://dx.doi.org/10.1145/2501988.2502049 Google ScholarDigital Library
Index Terms
- Do That, There: An Interaction Technique for Addressing In-Air Gesture Systems
Recommendations
Gunslinger: Subtle Arms-down Mid-air Interaction
UIST '15: Proceedings of the 28th Annual ACM Symposium on User Interface Software & TechnologyWe describe Gunslinger, a mid-air interaction technique using barehand postures and gestures. Unlike past work, we explore a relaxed arms-down position with both hands interacting at the sides of the body. It features "hand-cursor" feedback to ...
Touch screen gestures for web browsing tasks
One-step gestures such as a single swipe are preferred over multi-step gestures.The most common binding was object-centric.Most gestures used one finger, but five-finger gesture are used for larger devices.Gestures should reflect the desired action and ...
Visual touchpad: a two-handed gestural input device
ICMI '04: Proceedings of the 6th international conference on Multimodal interfacesThis paper presents the Visual Touchpad, a low-cost vision-based input device that allows for fluid two-handed interactions with desktop PCs, laptops, public kiosks, or large wall displays. Two downward-pointing cameras are attached above a planar ...
Comments