Don Samitha Elvitigala
Yunfan Wang
Yongquan Hu
Aaron J Quigley
ABSTRACT
Everyday, billions of people use footwear for walking, running, or exercise. Of emerging interest are “smart footwear”, which help users track gait, count steps or even analyse performance. However, such nascent footwear lack fine-grain ground surface context awareness, which could allow them to adapt to the conditions and create usable functions and experiences. Hence, this research aims to recognize the walking surface using a radar sensor embedded in a shoe, enabling ground context-awareness. Using data collected from 23 participants from an in-the-wild setting, we developed several classification models. We show that our model can detect five common terrain types with an accuracy of 80.0% and further ten terrain types with an accuracy of 66.3%, while moving. Importantly, it can detect the gait motion types such as ‘walking’, ‘stepping up’, ‘stepping down’, ‘still’, with an accuracy of 90%. Finally, we present potential use cases and insights for future work based on such ground-aware smart shoes.
Supplemental Material
Available for Download
zip
3606738.zip (173.5 MB)
Supplemental File
- Gianluca Agresti and Simone Milani. 2019. Material identification using RF sensors and convolutional neural networks. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3662–3666.Google Scholar
Cross Ref
- W Al-Nuaimy, Y Huang, M Nakhkash, MTC Fang, VT Nguyen, and A Eriksen. 2000. Automatic detection of buried utilities and solid objects with GPR using neural networks and pattern recognition. Journal of applied Geophysics 43, 2-4 (2000), 157–165.Google ScholarCross Ref
- Rafael Alencar. 2017. Resampling strategies for imbalanced datasets. Retrieved December 08, 2022 from https://www.kaggle.com/code/rafjaa/resampling-strategies-for-imbalanced-datasets/notebook.Google Scholar
- Kamiar Aminian, Benoit Mariani, Anisoara Paraschiv-Ionescu, Constanze Hoskovec, C Büla, J Penders, Carlo Tacconi, and F Marcellini. 2011. Foot worn inertial sensors for gait assessment and rehabilitation based on motorized shoes. In 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 5820–5823.Google ScholarCross Ref
- John W Arthur. 2008. The fundamentals of electromagnetic theory revisited. IEEE Antennas and Propagation Magazine 50, 1 (2008), 19–65.Google ScholarCross Ref
- J Bruder, J Carlo, J Gurney, and J Gorman. 2003. IEEE standard for letter designations for radar-frequency bands. IEEE Aerospace & Electronic Systems Society (2003), 1–3.Google Scholar
- Hakan Brunzell. 1999. Detection of shallowly buried objects using impulse radar. IEEE Transactions on Geoscience and Remote sensing 37, 2 (1999), 875–886.Google ScholarCross Ref
- Carv. 2022. Carv Digital Ski Coach. Retrieved September 01, 2022 from https://getcarv.com/products/carv.Google Scholar
- Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321–357.Google ScholarCross Ref
- Fulong Chen, Rosa Lasaponara, and Nicola Masini. 2017. An overview of satellite synthetic aperture radar remote sensing in archaeology: From site detection to monitoring. Journal of Cultural Heritage 23 (2017), 5–11.Google ScholarCross Ref
- Wenyao Chen, Yimeng Feng, Mark Cardamis, Cheng Jiang, Wei Song, Oula Ghannoum, and Wen Hu. 2022. Soil moisture sensing with mmWave radar. In Proceedings of the 6th ACM Workshop on Millimeter-Wave and Terahertz Networks and Sensing Systems. 19–24.Google ScholarDigital Library
- Mi-Hyeon Cheon, Dong-Gyun Hong, and Donghwa Lee. 2017. Surface crack detection in concrete structures using image processing. In International Conference on Robot Intelligence Technology and Applications. Springer, 559–564.Google Scholar
- Youngjun Cho, Nadia Bianchi-Berthouze, Nicolai Marquardt, and Simon J Julier. 2018. Deep thermal imaging: Proximate material type recognition in the wild through deep learning of spatial surface temperature patterns. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Asim Kumar Roy Choudhury. 2014. 2 - Object appearance and colour. In Principles of Colour and Appearance Measurement, Asim Kumar Roy Choudhury (Ed.). Woodhead Publishing, 53–102. https://doi.org/10.1533/9780857099242.53Google ScholarCross Ref
- Ashutosh Dhekne, Mahanth Gowda, Yixuan Zhao, Haitham Hassanieh, and Romit Roy Choudhury. 2018. Liquid: A wireless liquid identifier. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. 442–454.Google ScholarDigital Library
- DIGITSOLE. 2022. Digitsole Smart Insoles. Retrieved September 01, 2022 from https://digitsole.com/.Google Scholar
- Don Samitha Elvitigala, Denys JC Matthies, Löic David, Chamod Weerasinghe, and Suranga Nanayakkara. 2019. GymSoles: Improving Squats and Dead-Lifts by Visualizing the User’s Center of Pressure. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.Google ScholarDigital Library
- Don Samitha Elvitigala, Denys JC Matthies, Chamod Weerasinghe, and Suranga Nanayakkara. 2021. GymSoles++: Combining Google Glass with Smart Insoles to Improve Body Posture when Performing Squats. In The 14th PErvasive Technologies Related to Assistive Environments Conference. 48–54.Google Scholar
- [19] evalu. 2020. Retrieved September 01, 2022 from https://www.evalu.com.Google Scholar
- Cheng Fang, Di Wang, Dezhen Song, and Jun Zou. 2021. Fingertip Pulse-Echo Ultrasound and Optoacoustic Dual-Modal and Dual Sensing Mechanisms Near-Distance Sensor for Ranging and Material Sensing in Robotic Grasping. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 14105–14111.Google Scholar
- Corinna Feeken, Merlin Wasmann, Wilko Heuten, Dag Ennenga, Heiko Müller, and Susanne Boll. 2016. ClimbingAssist: direct vibro-tactile feedback on climbing technique. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. 57–60.Google ScholarDigital Library
- Feetme. 2020. Smart medical werable to improve mobility. Retrieved September 01, 2022 from https://feetme.fr/en.Google Scholar
- III Frampton E. Ellis. US10016015B2, USA. Footwear soles with computer controlled configurable structures.Google Scholar
- Martin Frey. 2007. CabBoots: shoes with integrated guidance system. In Proceedings of the 1st international conference on Tangible and embedded interaction. 245–246.Google ScholarDigital Library
- C Lee Giles and Walter J Wild. 1982. Fresnel reflection and transmission at a planar boundary from media of equal refractive indices. Applied Physics Letters 40, 3 (1982), 210–212.Google ScholarCross Ref
- Michael Gillham, Gareth Howells, Sarah Spurgeon, and Ben McElroy. 2013. Floor covering and surface identification for assistive mobile robotic real-time room localization application. Sensors 13, 12 (2013), 17501–17515.Google ScholarCross Ref
- Paul R Gray, Paul J Hurst, Stephen H Lewis, and Robert G Meyer. 2009. Analysis and design of analog integrated circuits. John Wiley & Sons.Google Scholar
- NA Hoog, TE van den Berg, and HS Bindra. 2022. A 60 GHz pulsed coherent radar for online monitoring of the withering condition of leaves. Sensors and Actuators A: Physical 343 (2022), 113693.Google ScholarCross Ref
- Ayanna Howard and Homayoun Seraji. 2001. Vision-based terrain characterization and traversability assessment. journal of robotic systems 18, 10 (2001), 577–587.Google Scholar
- SS Hubbard, JE Peterson Jr, EL Majer, PT Zawislanski, KH Williams, J Roberts, and Frank Wobber. 1997. Estimation of permeable pathways and water content using tomographic radar data. The leading EDGE 16, 11 (1997), 1623–1630.Google Scholar
- Keigo Iizuka and Alois P Freundorfer. 1983. Detection of nonmetallic buried objects by a step frequency radar. Proc. IEEE 71, 2 (1983), 276–279.Google ScholarCross Ref
- Anoosha Iqbal. 2017. Intel powered 90Fun Smart Sports Shoes – perfect partner for athletes!Sports Wearable. Retrieved September 01, 2022 from https://www.sportswearable.net/intel-powered-90fun-smart-sports-shoes-perfect-partner-athletes/.Google Scholar
- Harish Jonnalagadda. 2017. Xiaomi Mi Smart Shoes review: Obsessively track your workouts. Retrieved September 01, 2022 from https://www.androidcentral.com/xiaomi-mi-smart-shoes-review.Google Scholar
- Umar S Khan, Waleed Al-Nuaimy, and Fathi E Abd El-Samie. 2010. Detection of landmines and underground utilities from acoustic and GPR images with a cepstral approach. Journal of Visual Communication and Image Representation 21, 7 (2010), 731–740.Google ScholarDigital Library
- Satyam Kumar. 2021. 5 Techniques to work with Imbalanced Data in Machine Learning. Retrieved December 08, 2022 from https://towardsdatascience.com/5-techniques-to-work-with-imbalanced-data-in-machine-learning-80836d45d30c.Google Scholar
- Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1–19.Google ScholarDigital Library
- Yawen Liu, Shihan Lu, and Heather Culbertson. 2022. Texture Classification by Audio-Tactile Crossmodal Congruence. In 2022 IEEE Haptics Symposium (HAPTICS). IEEE, 1–7.Google ScholarCross Ref
- Denys JC Matthies, Thijs Roumen, Arjan Kuijper, and Bodo Urban. 2017. CapSoles: who is walking on what kind of floor?. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1–14.Google ScholarDigital Library
- Walterio W Mayol-Cuevas, J Juarez-Guerrero, and S Munoz-Gutierrez. 1998. A first approach to tactile texture recognition. In SMC’98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No. 98CH36218), Vol. 5. IEEE, 4246–4250.Google ScholarCross Ref
- Anita Meier, Denys JC Matthies, Bodo Urban, and Reto Wettach. 2015. Exploring vibrotactile feedback on the body and foot for the purpose of pedestrian navigation. In Proceedings of the 2nd international Workshop on Sensor-based Activity Recognition and Interaction. 1–11.Google ScholarDigital Library
- moticon. 2020. Sensor insoles for research. Retrieved September 01, 2022 from https://www.moticon.de.Google Scholar
- Henrique Oliveira and Paulo Lobato Correia. 2014. CrackIT—An image processing toolbox for crack detection and characterization. In 2014 IEEE international conference on image processing (ICIP). IEEE, 798–802.Google ScholarCross Ref
- MJD Otis and BAJ Ménélas. 2014. Method to determine physical properties of ground, foot-worn sensor therefore, and method to advise a user of a risk of falling based thereon. Patent No. WO 2014/032181 A1 (2014).Google Scholar
- Joseph A Paradiso, Stacy J Morris, Ari Y Benbasat, and Erik Asmussen. 2004. Interactive therapy with instrumented footwear. In CHI’04 Extended Abstracts on Human Factors in Computing Systems. 1341–1343.Google Scholar
- Amal Kumar Raychaudhuri. 2022. Reflection and Refraction of Electromagnetic Waves. Springer Singapore, Singapore, 123–136. https://doi.org/10.1007/978-981-16-8139-4_11Google ScholarCross Ref
- Parisa Momen Roodaki, Fateme Taghian, Saeed Bashirzadeh, and Mohsen Jalaali. 2011. A survey of millimeter-wave technologies. In 2011 international conference on electrical and control engineering. IEEE, 5726–5728.Google ScholarCross Ref
- Maximilian Schrapel, Jonathan Liebers, Michael Rohs, and Stefan Schneegass. 2020. Skiables: Towards a Wearable System Mounted on a Ski Boot for Measuring Slope Conditions. In Proceedings of the 19th International Conference on Mobile and Ubiquitous Multimedia. 320–322.Google ScholarDigital Library
- Joseph A Shaw. 2013. Radiometry and the Friis transmission equation. American journal of physics 81, 1 (2013), 33–37.Google Scholar
- Lynn T Staheli. 1991. Shoes for children: a review.Pediatrics 88, 2 (1991), 371–375.Google Scholar
- Okeke Stephen, Uchenna Joseph Maduh, and Mangal Sain. 2021. A machine learning method for detection of surface defects on ceramic tiles using convolutional neural networks. Electronics 11, 1 (2021), 55.Google ScholarCross Ref
- Matti Strese, Clemens Schuwerk, Albert Iepure, and Eckehard Steinbach. 2016. Multimodal feature-based surface material classification. IEEE transactions on haptics 10, 2 (2016), 226–239.Google Scholar
- stridalyzer. 2020. Stridalyzer Performance. Retrieved September 01, 2022 from https://retisense.com/runners-athletes/.Google Scholar
- J. van Brakel. 2014. Robust Peak Detection Algorithm Using z-Scores. Retrieved December 08, 2022 from https://stackoverflow.com/questions/22583391/peak-signal-detection-in-realtime-timeseries-data/.Google Scholar
- Jaap J van Netten, MJ Jannink, Juha M Hijmans, JH Geertzen, and Klaas Postema. 2010. Use and usability of custom-made orthopedic shoes. J Rehabil Res Dev 47, 1 (2010), 73–81.Google ScholarCross Ref
- Chenshu Wu, Feng Zhang, Beibei Wang, and KJ Ray Liu. 2020. mSense: Towards mobile material sensing with a single millimeter-wave radio. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020), 1–20.Google ScholarDigital Library
- Kaishun Wu. 2016. Wi-metal: Detecting metal by using wireless networks. In 2016 IEEE International Conference on Communications (ICC). IEEE, 1–6.Google ScholarCross Ref
- Hui-Shyong Yeo, Barrett Ens, and Aaron Quigley. 2017. Tangible UI by object and material classification with radar. In SIGGRAPH Asia 2017 Emerging Technologies. 1–2.Google Scholar
- Hui-Shyong Yeo, Gergely Flamich, Patrick Schrempf, David Harris-Birtill, and Aaron Quigley. 2016. Radarcat: Radar categorization for input & interaction. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 833–841.Google ScholarDigital Library
- Hugh D Young, Roger A Freedman, and A Lewis Ford. 2012. University physics with modern physics 13th edition. Haettu 11 (2012), 2020.Google Scholar
- Diana Zhang, Jingxian Wang, Junsu Jang, Junbo Zhang, and Swarun Kumar. 2019. On the feasibility of wi-fi based material sensing. In The 25th Annual International Conference on Mobile Computing and Networking. 1–16.Google ScholarDigital Library
- Qin Zou, Yu Cao, Qingquan Li, Qingzhou Mao, and Song Wang. 2012. CrackTree: Automatic crack detection from pavement images. Pattern Recognition Letters 33, 3 (2012), 227–238.Google ScholarDigital Library
Index Terms
- RadarFoot: Fine-grain Ground Surface Context Awareness for Smart Shoes
Recommendations
A Review of Smart Technologies Embedded in Shoes
AbstractTechnological advancements in wearable devices have revolutionized smart shoes. Smart shoes are sometimes referred to as intelligent shoes or computer-based shoes. They are capable of recognizing and recording data from day-to-day activities by ...
Walking pattern analysis using deep learning for energy harvesting smart shoes with IoT
AbstractWearable Health Devices (WHDs) benefit people to monitor their health status and have become a necessity in today’s world. The smart shoe is the type of WHD, that provides comfort, convenience, and fitness tracking. Hence smart shoes can be ...
Smart home context awareness based on Smart and Innovative Cities
EANN '15: Proceedings of the 16th International Conference on Engineering Applications of Neural Networks (INNS)In the emerging Smart Cities - Smart Homes computing paradigms developing a formalization for context information is increasingly important. In the present paper, basedon the EU FIRE research project "Social and Smart" we aim to formalize and build a ...
Comments