Skip to main content
Log in

Adapting smartwatch interfaces to hand gestures during movements: offset models and the C-shaped pattern of tapping

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Interacting with smartwatches is fairly common when users are moving. Although novel interaction gestures like flick of the wrist are implemented, basic touch gestures such as tapping and swiping still dominate. Using these gestures during a variety of movements could be challenging, and it is still not clear how the interface of smartwatches should tailor to users’ gestures during movements and how the usage of smartwatches influences the pattern of users’ movements. Therefore, this study investigates the interrelationship among users’ interaction gestures, movements, and gait features. An experiment was conducted among 47 participants, who used smartwatches through tapping, swiping, and wrist flicking to complete daily tasks in stand, strolling, normal walking, rushing, and jogging. They were tracked through built-in accelerometer and angle sensors. Four findings were derived from the experiment. First, rushing and jogging significantly decrease the effectiveness and efficiency of tapping. To reduce the tapping deviation, offset models were proposed and tested. Second, there is a C-shaped pattern on the round screen where tapping targets achieves higher accuracy than other areas. Third, the tapping performance could be improved by setting target sizes. Target sizes at 0.7 cm in stand, 1.1 cm in strolling, and 1.1 cm in walking achieve a high level of accuracy (95%), while target sizes at 1.5 cm in rushing and jogging achieve a middle level of accuracy (90%). Finally, tapping, swiping, and wrist flicking when users are moving significantly reduce their gait symmetry and step length. They do not imply significant influence on gait intensity, regularity, and overall stability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  • Agarwal R, Karahanna E (2000) Time flies when you’re having fun: cognitive absorption and beliefs about information technology usage. MIS Q 24(4):665–694. https://doi.org/10.2307/3250951

    Article  Google Scholar 

  • Apple (2018a) Themes-watchOS-human interface guidelines. https://developer.apple.com/design/human-interface-guidelines/watchos/overview/themes/. Accessed 25 Nov 2018

  • Apple (2018b) Home screen icons-icons and images-watchOS-human interface guidelines. https://developer.apple.com/design/human-interface-guidelines/watchos/icons-and-images/home-screen-icons/. Accessed 2 Dec 2018

  • Ashbrook D, Lyons K, Starner T (2008) An investigation into round touchscreen wristwatch interaction. In: Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. ACM, New York, pp 311–314. https://doi.org/10.1145/1409240.1409276

  • Barnard L, Yi JS, Jacko JA, Sears A (2005) An empirical comparison of use-in-motion evaluation scenarios for mobile computing devices. Int J Hum Comput Stud 62(4):487–520. https://doi.org/10.1016/j.ijhcs.2004.12.002

    Article  Google Scholar 

  • Bragdon A, Nelson E, Li Y, Hinckley K (2011) Experimental analysis of touch-screen gesture designs in mobile environments. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 403–412. https://doi.org/10.1145/1978942.1979000

  • Buschek D, Rogers S, Murray-Smith R (2013) User-specific touch models in a cross-device context. In: Proceedings of the 15th international conference on human–computer interaction with mobile devices and services. ACM, New York, pp 382–391. https://doi.org/10.1145/2493190.2493206

  • Chaparro BS, He J, Turner C, Turner K (2015) Is touch-based text input practical for a smartwatch? In: HCI international 2015—posters’ extended abstracts. Springer, Cham, pp 3–8. https://doi.org/10.1007/978-3-319-21383-5_1

  • Conradi J, Alexander T (2014a) Analysis of visual performance during the use of mobile devices while walking. In: Engineering psychology and cognitive ergonomics. Springer, Cham, pp 133–142. https://doi.org/10.1007/978-3-319-07515-0_14

  • Conradi J, Alexander T (2014b) Analysis of visual performance during the use of mobile devices while walking. In: International conference on engineering psychology and cognitive ergonomics. Springer, Berlin, pp 133–142. http://link.springer.com/chapter/10.1007/978-3-319-07515-0_14

  • Conradi J, Busch O, Alexander T (2015) Optimal touch button size for the use of mobile devices while walking. Procedia Manuf 3:387–394. https://doi.org/10.1016/j.promfg.2015.07.182

    Article  Google Scholar 

  • Damos D (1991) Multiple task performance. CRC Press, Boca Raton

    Google Scholar 

  • Feher C, Elovici Y, Moskovitch R, Rokach L, Schclar A (2012) User identity verification via mouse dynamics. Inf Sci 201:19–36

    Article  Google Scholar 

  • Gibson MP (2018) Precision pointing with fat fingers. https://www.technologyreview.com/s/407947/precision-pointing-with-fat-fingers/. Accessed 3 Dec 2018

  • Gil H, Lee D, Im S, Oakley I (2017) TriTap: identifying finger touches on smartwatches. In: Proceedings of the 2017 CHI conference on human factors in computing systems. ACM, New York, pp 3879–3890. https://doi.org/10.1145/3025453.3025561

  • Goel M, Findlater L, Wobbrock J (2012) WalkType: using accelerometer data to accommodate situational impairments in mobile touch screen text entry. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 2687–2696. https://doi.org/10.1145/2207676.2208662

  • Google (2018) Designing for watches—wear OS. https://designguidelines.withgoogle.com/wearos/wear-os-by-google/designing-for-watches.html#. Accessed 25 Nov 2018

  • Han T, Li J, Hasan K, Nakamura K, Gomez R, Balakrishnan R, Irani P (2018) PageFlip: leveraging page-flipping gestures for efficient command and value selection on smartwatches. In: Proceedings of the 2018 CHI conference on human factors in computing systems. ACM, New York, pp 1–12. https://doi.org/10.1145/3173574.3174103

  • Harrison C, Hudson SE (2009) Abracadabra: wireless, high-precision, and unpowered finger input for very small mobile devices. In: Proceedings of the 22nd annual ACM symposium on user interface software and technology. ACM, New York, pp 121–124. https://doi.org/10.1145/1622176.1622199

  • Hayes ST, Hooten ER, Adams JA (2014) Tablet interaction accuracy and precision: seated vs. walking. In: Proceedings of the human factors and ergonomics society annual meeting, vol 58, no 1. SAGE Publications, Los Angeles, CA, pp 432–436. https://doi.org/10.1177/1541931214581090

  • Hussain I, Chen L, Mirza HT, Wang L, Chen G, Memon I (2016) Chinese-based spearcons: improving pedestrian navigation performance in eyes-free environment. Int J Hum Comput Interact 32(6):460–469. https://doi.org/10.1080/10447318.2016.1160669

    Article  Google Scholar 

  • IBM (2016) Design language|interaction: touch (2016, January 15). [CTZZZ]. http://www.ibm.com/design/language/experience/interaction/touch. Accessed 22 Nov 2018

  • Kamata Y, Nakamura T, Tada M, Sueda S, Pai DK, Toyama Y (2016) How the lumbrical muscle contributes to placing the fingertip in space: a three dimensional cadaveric study to assess fingertip trajectory and metacarpophalangeal joint balancing. J Hand Surg (Eur Vol) 41(4):386–391

    Article  Google Scholar 

  • Kamper DG, Cruz EG, Siegel MP (2003) Stereotypical fingertip trajectories during grasp. J Neurophysiol 90(6):3702–3710

    Article  Google Scholar 

  • Kane SK, Wobbrock JO, Smith IE (2008) Getting off the treadmill: evaluating walking user interfaces for mobile devices in public spaces. In: Proceedings of the 10th international conference on human computer interaction with mobile devices and services. ACM, New York, pp 109–118. https://doi.org/10.1145/1409240.1409253

  • Kubo Y, Shizuki B, Tanaka J (2016) B2B-Swipe: swipe gesture for rectangular smartwatches from a bezel to a bezel. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, New York, pp 3852–3856. https://doi.org/10.1145/2858036.2858216

  • Lafreniere B, Gutwin C, Cockburn A, Grossman T (2016) Faster command selection on touchscreen watches. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, New York, pp 4663–4674. https://doi.org/10.1145/2858036.2858166

  • Lewis JR (2002) Psychometric evaluation of the PSSUQ using data from five years of usability studies. Int J Hum Comput Interact 14(3–4):463–488. https://doi.org/10.1080/10447318.2002.9669130

    Article  Google Scholar 

  • Lim JJ, Feria C (2012) Visual search on a mobile device while walking. In: Proceedings of the 14th international conference on human–computer interaction with mobile devices and services. ACM, New York, pp 295–304. https://doi.org/10.1145/2371574.2371618

  • Lin M, Goldman R, Price KJ, Sears A, Jacko J (2007) How do people tap when walking? An empirical investigation of nomadic data entry. Int J Hum Comput Stud 65(9):759–769. https://doi.org/10.1016/j.ijhcs.2007.04.001

    Article  Google Scholar 

  • McMillan D, Brown B, Lampinen A, McGregor M, Hoggan E, Pizza S (2017) Situating wearables: smartwatch use in context. In: Proceedings of the 2017 CHI conference on human factors in computing systems. ACM, New York, pp 3582–3594. https://doi.org/10.1145/3025453.3025993

  • Menz HB, Lord SR, Fitzpatrick RC (2003) Acceleration patterns of the head and pelvis when walking on level and irregular surfaces. Gait Posture 18(1):35–46. https://doi.org/10.1016/S0966-6362(02)00159-5

    Article  Google Scholar 

  • Mizobuchi S, Chignell M, Newton D (2005) Mobile text entry: relationship between walking speed and text input task difficulty. In: Proceedings of the 7th international conference on Human computer interaction with mobile devices and services. ACM, New York, pp 122–128. https://doi.org/10.1145/1085777.1085798

  • Mo F, Zhou J, Yi S (2017) Adapting the navigation interface of smart watches to user movements. Int J Hum Comput Interact 33(6):460–474. https://doi.org/10.1080/10447318.2017.1279826

    Article  Google Scholar 

  • Moe-Nilssen R, Helbostad JL (2004) Estimation of gait cycle characteristics by trunk accelerometry. J Biomech 37(1):121–126. https://doi.org/10.1016/S0021-9290(03)00233-1

    Article  Google Scholar 

  • Motti VG, Caine K (2015) Micro interactions and multi dimensional graphical user interfaces in the design of wrist worn wearables. In: Proceedings of the human factors and ergonomics society annual meeting, vol 59, no 1. SAGE Publications, Los Angeles, pp 1712–1716. https://doi.org/10.1177/1541931215591370

  • Motti VG, Caine K (2016) Smart wearables or dumb wearables?: understanding how context impacts the UX in wrist worn interaction. In: Proceedings of the 34th ACM international conference on the design of communication. ACM, New York, p 10. https://doi.org/10.1145/2987592.2987606

  • Musić J, Murray-Smith R (2016) Nomadic input on mobile devices: the influence of touch input technique and walking speed on performance and offset modeling. Hum Comput Interact 31(5):420–471. https://doi.org/10.1080/07370024.2015.1071195

    Article  Google Scholar 

  • Muthumari G, Shenbagaraj R, Pepsi MBB (2014) Mouse gesture based authentication using machine learning algorithm. In: 2014 IEEE international conference on advanced communications, control and computing technologies. IEEE, New York, pp 492–496. https://doi.org/10.1109/ICACCCT.2014.7019492

  • Ng A, Williamson JH, Brewster SA (2014) Comparing evaluation methods for encumbrance and walking on interaction with touchscreen mobile devices. In: Proceedings of the 16th international conference on human–computer interaction with mobile devices and services. ACM, New York, pp 23–32. https://doi.org/10.1145/2628363.2628382

  • Ng A, Williamson J, Brewster S (2015) The effects of encumbrance and mobility on touch-based gesture interactions for mobile phones. In: Proceedings of the 17th international conference on human–computer interaction with mobile devices and services. ACM, New York, pp 536–546. https://doi.org/10.1145/2785830.2785853

  • O’Donovan KJ, Greene BR, McGrath D, O’Neill R, Burns A, Caulfield B (2009) SHIMMER: a new tool for temporal gait analysis. In: 2009 Annual international conference of the IEEE engineering in medicine and biology society, pp 3826–3829. https://doi.org/10.1109/IEMBS.2009.5335140

  • Oakley I, Lee D, Islam MR, Esteves A (2015) Beats: tapping gestures for smart watches. ACM Press, New York, pp 1237–1246. https://doi.org/10.1145/2702123.2702226

    Book  Google Scholar 

  • Oakley I, Lindahl C, Le K, Lee D, Islam MR (2016) The flat finger: exploring area touches on smartwatches. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, New York, pp 4238–4249. https://doi.org/10.1145/2858036.2858179

  • Schildbach B, Rukzio E (2010) Investigating selection and reading performance on a mobile phone while walking. In: Proceedings of the 12th international conference on human computer interaction with mobile devices and services. ACM, New York, pp 93–102. https://doi.org/10.1145/1851600.1851619

  • Siek KA, Rogers Y, Connelly KH (2005) Fat finger worries: how older and younger users physically interact with PDAs. In: IFIP conference on human–computer interaction. Springer, Berlin, pp 267–280. https://doi.org/10.1007/11555261_24

  • Statistic (2017) U.S. wearable user penetration 2017, by age. https://www.statista.com/statistics/739398/us-wearable-penetration-by-age/. Accessed 7 Dec 2018

  • Van Noort FL (2015) Effect of running on throughput in pointing tasks: a Fitts’ law experiment. Master’s thesis

  • Weir D, Rogers S, Murray-Smith R, Löchtefeld M (2012) A user-specific machine learning approach for improving touch accuracy on mobile devices. In: Proceedings of the 25th annual ACM symposium on user interface software and technology. ACM, New York, pp 465–476. https://doi.org/10.1145/2380116.2380175

  • Xiao R, Laput G, Harrison C (2014) Expanding the input expressivity of smartwatches with mechanical pan, twist, tilt and click. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 193–196. https://doi.org/10.1145/2556288.2557017

  • Zhang Y, Rau P-LP (2015) Playing with multiple wearable devices: exploring the influence of display, motion and gender. Comput Hum Behav 50:148–158. https://doi.org/10.1016/j.chb.2015.04.004

    Article  Google Scholar 

  • Zhang Y, Mao M, Rau P-LP, Choe P, Bela L, Wang F (2013) Exploring factors influencing multitasking interaction with multiple smart devices. Comput Hum Behav 29(6):2579–2588. https://doi.org/10.1016/j.chb.2013.06.042

    Article  Google Scholar 

  • Zhang C, Yang J, Southern C, Starner TE, Abowd GD (2016) WatchOut: extending Interactions on a smartwatch with inertial sensing. In: Proceedings of the 2016 ACM international symposium on wearable computers. ACM, New York, pp 136–143. https://doi.org/10.1145/2971763.2971775

  • Zhong R, Rau P-LP, Yan X (2018) Application of smart bracelet to monitor frailty-related gait parameters of older Chinese adults: a preliminary study. Geriatr Gerontol Int 18(9):1366–1371. https://doi.org/10.1111/ggi.13492

    Article  Google Scholar 

Download references

Acknowledgements

Thanks Dr. Runting Zhong (Tsinghua University) for providing advice on gait analysis, and thanks Editing Services for proofreading the article. This work was supported by funding from the National Natural Science Foundation of China (Grants no. 71661167006) and the Chongqing Municipal Natural Science Foundation (cstc2016jcyjA0406).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jia Zhou.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1: Instructions for movement conditions

The study used psychological hints to control speed of participants (Zhong et al. 2018). The method is closer to the real walking scenario when compared to PWS.

Standing: Please stand free and relaxed without moving your limbs.

Strolling: Please stroll, and walk slowly, just like you are waiting for someone in the meantime.

Normal walking: Please walk just like your common speed.

Rushing: Please rush, and walk in haste, just like you are late for a meeting.

Jogging: Please jog and run at a slow speed.

Appendix 2: Measurement of interaction performance and gait features

See Tables 9, 10 and 11.

Table 9 Measurement of tapping performance
Table 10 Measurement of swiping performance
Table 11 Gait characteristics

Appendix 3: Mathematics of the offset models

The vector fields are used as offset models. The screen domain is represented by D, and thus the vector fields are represented as \(\vec{v}_{{\left( {x_{t} ,y_{t} } \right)}}\), where (xtyt) ∊ D, and xt and yt denote the x- and y-coordinates of the target point; and xr and yr denote the x- and y-coordinates of the real tapping point. The \(\vec{v}_{{(x_{t} ,y_{t} )}} = \left( {x_{r}^{{x_{t} }} - x_{t} ,y_{r}^{{y_{t} }} - y_{t} } \right)\) is defined and calculated in the study where \(x_{r}^{{x_{t} }}\) and \(y_{r}^{{y_{t} }}\) denote the x- and y-coordinates of the real tapping point corresponding to the target point (xtyt). We assume that the real tapping point and the target point exhibit one-to-one correspondence. Therefore, the opposite of the vector \(- \vec{v}_{{(x_{t} ,y_{t} )}} = \left( {x_{t} - x_{r}^{{x_{t} }} ,y_{t} - y_{r}^{{y_{t} }} } \right) = \left( {x_{t}^{{x_{r} }} - x_{r} ,y_{t}^{{y_{r} }} - y_{r} } \right) = \vec{v}_{{(x_{r} ,y_{r} )}} ,\) where \(x_{r}^{{x_{t} }}\) and \(y_{r}^{{y_{t} }}\) denote the x- and y-coordinates, respectively, of the target point corresponding to the real tapping point.

With respect to a specific real tapping points (xryr) that users just tap, the following expression is used to offset its deviation:

$$\left( {x_{t}^{{x_{r} }} ,y_{t}^{{y_{r} }} } \right) = (x_{r} ,y_{r} ) - \vec{v}_{{(x_{t} ,y_{t} )}} ,\quad \text{where}\; { (}x_{t} ,y_{t} )\in D \, \text{and} \, \left( {x_{t}^{{x_{r} }} ,y_{t}^{{y_{r} }} } \right) \in D.$$

The expression denotes a real tapping point on the screen, and the other side of its vector denotes the target point that the user aims to tap.

Appendix 4: C-shaped area in different simulated diameters

On the one hand, by adjusting the diameter, we can simulate the results of tapping different size of target sizes. Tapping error rates is measured in this way.

On the other hand, in these offset models, when we adjust the diameter, more tapping behaviors are included. The offset model becomes more conforming to real situations. However, tapping points with very large deviations may be considered as errors, such as deviating half a screen from the target point. Therefore, we chose 1.5 cm as the final parameter for offset models (the diameter of the screen is around 3.5 cm). Offset models with other simulated diameters are shown as bellows.

figure a

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mo, F., Zhou, J. Adapting smartwatch interfaces to hand gestures during movements: offset models and the C-shaped pattern of tapping. J Ambient Intell Human Comput 12, 8099–8117 (2021). https://doi.org/10.1007/s12652-020-02545-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-020-02545-3

Keywords

Navigation