Topi Kaaresoja
Stephen Brewster
ABSTRACT
Multimodal interaction is becoming common in many kinds of devices, particularly mobile phones. If care is not taken in design and implementation, there may be latencies in the timing of feedback in the different modalities may have unintended effects on users. This paper introduces an easy to implement multimodal latency measurement tool for touchscreen interaction. It uses off-the-shelf components and free software and is capable of measuring latencies accurately between different interaction events in different modalities. The tool uses a high-speed camera, a mirror, a microphone and an accelerometer to measure the touch, visual, audio and tactile feedback events that occur in touchscreen interaction. The microphone and the accelerometer are both interfaced with a standard PC soundcard that makes the measurement and analysis simple. The latencies are obtained by hand and eye using a slow-motion video player and an audio editor. To validate the tool, we measured four commercial mobile phones. Our results show that there are significant differences in latencies, not only between the devices, but also between different applications and modalities within one device. In this paper the focus is on mobile touchscreen devices, but with minor modifications our tool could be also used in other domains.
- Mine, M. R., Characterization of End-to-End Delays in Head-Mounted Display Systems. University of North Carolina, Chapel Hill, 1993.Google Scholar
- He, D., Liu, F., Pape, D., Dawe, G. and Sandin, D., Video-Based Measurement of System Latency. In Proceedings of the IPT2000, International Immersive Projection Technology Workshop (Ames IA, USA, 2000) (2000).Google Scholar
- Miller, D. and Bishop, G., Latency meter: a device for easily monitoring VE delay. In Proceedings of the Stereoscopic Displays and Virtual Reality Systems IX (San Jose, CA, USA, 2002) (2002).Google Scholar
- Wright, M., Cassidy, R. J. and Zbyszynski, M. F., Audio and Gesture Latency Measurements on Linux and OSX. In Proceedings of the International Computer Music Conference (Miami, FL, USA, 2004) (2004), 423--429.Google Scholar
- Lehtosalo, A., Haptics Technologies and Electromechanical Latency Measurements in Tactile Feedback System. Master's Thesis, Tampere University of Technology, Tampere, Finland, 2009.Google Scholar
- Levitin, D. J., MacLean, K., Mathews, M. and Chu, L., The Perception of Cross-Modal Simultaneity. Invited paper. In Proceedings of the 3rd International Conference on Computing Anticipatory Systems (Liège, Belgium., 1999) (1999).Google Scholar
- Adelstein, B. D., Begault, D. R., Anderson, M. R. and Wenzel, E. M., Sensitivity to haptic-audio asynchrony. In Proceedings of the 5th International Conference on Multimodal Interfaces (Vancouver, Canada, 2003). ACM (2003), 73--76. Google ScholarDigital Library
- Liang, J., Shaw, C. and Green, M., On Temporal-Spatial Realism in the Virtual Reality Environment. In Proceedings of the UIST'91, Symposium of User Interface Software and Technology (Hilton Head, South Carolina, US, 1991) (1991). Google ScholarDigital Library
- Freed, A., Chaudhary, A. and Davila, B., Operating Systems Latency Measurement and Analysis for Sound Synthesis and Processing Applications. In Proceedings of the International Computer Music Conference (Thessaloniki, Creece, 1997). ICMA (International Computer Music Association) (1997).Google Scholar
- MacMillan, K., Droettboom, M. and Fujinaga, I., Audio Latency Measurements of Desktop Operating Systems. In Proceedings of the International Computer Music Conference (Habana, Cuba, 2001). ICMA (International Computer Music Association) (2001).Google Scholar
- Nelson, M. and Thom, B., Interactive MIDI: Real-time Performance Evaluation. In Proceedings of the New Interfaces for Musical Expression (Hamamatsu, Japan, 2004) (2004).Google Scholar
Index Terms
- Feedback is... late: measuring multimodal delays in mobile device touchscreen interaction
Recommendations
The Future Is a Foreign Country: We'll Do Things Differently There
The iPhone's touchscreen has changed the way today's users expect to interact with technology. But what about tomorrow? The author considers additional ways to alter or enhance device interfaces using other human senses.
Playing with tactile feedback latency in touchscreen interaction: two approaches
INTERACT'11: Proceedings of the 13th IFIP TC 13 international conference on Human-computer interaction - Volume Part IIA great deal of research has investigated the potential parameters of tactile feedback for virtual buttons. However, these studies do not take the possible effects of feedback latencies into account. Therefore, this research investigates the impact of ...
Frictional widgets: enhancing touch interfaces with programmable friction
CHI EA '11: CHI '11 Extended Abstracts on Human Factors in Computing SystemsTouch interactions occur through flat surfaces that lack the tactile richness of physical interfaces. We explore the design possibilities offered by augmenting touchscreens with programmable surface friction. Four exemplar applications -- an alarm clock,...
Comments