Shibo Zhang
Josiah Hester
Skip Abstract Section
Skip Supplemental Material SectionAbstract
We present the design, implementation, and evaluation of a multi-sensor, low-power necklace, NeckSense, for automatically and unobtrusively capturing fine-grained information about an individual's eating activity and eating episodes, across an entire waking day in a naturalistic setting. NeckSense fuses and classifies the proximity of the necklace from the chin, the ambient light, the Lean Forward Angle, and the energy signals to determine chewing sequences, a building block of the eating activity. It then clusters the identified chewing sequences to determine eating episodes. We tested NeckSense on 11 participants with and 9 participants without obesity, across two studies, where we collected more than 470 hours of data in a naturalistic setting. Our results demonstrate that NeckSense enables reliable eating detection for individuals with diverse body mass index (BMI) profiles, across an entire waking day, even in free-living environments. Overall, our system achieves an F1-score of 81.6% in detecting eating episodes in an exploratory study. Moreover, our system can achieve an F1-score of 77.1% for episodes even in an all-day-long free-living setting. With more than 15.8 hours of battery life, NeckSense will allow researchers and dietitians to better understand natural chewing and eating behaviors. In the future, researchers and dietitians can use NeckSense to provide appropriate real-time interventions when an eating episode is detected or when problematic eating is identified.
Supplemental Material
Available for Download
zip
zhang-1.zip (32.5 KB)
Supplemental movie, appendix, image and software files for, NeckSense: A Multi-Sensor Necklace for Detecting Eating Activities in Free-Living Conditions
- Rawan Alharbi, Angela Pfammatter, Bonnie Spring, and Nabil Alshurafa. 2017. WillSense: Adherence Barriers for Passive Sensing Systems That Track Eating Behavior. In Proceedings of the CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA). https://doi.org/10.1145/3027063.3053271Google Scholar
Digital Library
- Rawan Alharbi, Tammy Stump, Nilofar Vafaie, Angela Pfammatter, Bonnie Spring, and Nabil Alshurafa. 2018. I Can't Be Myself: Effects of Wearable Cameras on the Capture of Authentic Behavior in the Wild. ACM Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) 2, 3, Article 90 (2018), 40 pages. https://doi.org/10.1145/3264900Google Scholar
- Rawan Alharbi, Mariam Tolba, Lucia C Petito, Josiah Hester, and Nabil Alshurafa. 2019. To Mask or Not to Mask? Balancing Privacy with Visual Confirmation Utility in Activity-Oriented Wearable Cameras. ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) 3, 3, Article 72 (2019), 29 pages. https://doi.org/10.1145/3351230Google Scholar
- Rawan Alharbi, Nilofar Vafaie, Kitty Liu, Kevin Moran, Gwendolyn Ledford, Angela Pfammatter, Bonnie Spring, and Nabil Alshurafa. 2017. Investigating barriers and facilitators to wearable adherence in fine-grained eating detection. In IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). 407--412. https://doi.org/10.1109/PERCOMW.2017.7917597Google ScholarCross Ref
- Nabil Alshurafa, Annie Wen Lin, Fengqing Zhu, Roozbeh Ghaffari, Josiah Hester, Edward Delp, John Rogers, and Bonnie Spring. 2019. Counting Bites With Bits: Expert Workshop Addressing Calorie and Macronutrient Intake Monitoring. Journal of Medical Internet Research (JMIR) 21, 12 (2019). https://doi.org/10.2196/14904Google Scholar
- Nabil Alshurafa, Wenyao Xu, Jason J. Liu, Ming-Chun Huang, Bobak Mortazavi, Majid Sarrafzadeh, and Christian Roberts. 2013. Robust human intensity-varying activity recognition using Stochastic Approximation in wearable sensors. In IEEE International Conference on Body Sensor Networks. https://doi.org/10.1109/BSN.2013.6575515Google ScholarCross Ref
- Oliver Amft and Gerhard Tröster. 2006. Methods for detection and classification of normal swallowing from muscle activation and sound. In IEEE Pervasive Health Conference and Workshops. 1--10. https://doi.org/10.1109/PCTHEALTH.2006.361624Google ScholarCross Ref
- Oliver Amft and Gerhard Tröster. 2009. On-body sensing solutions for automatic dietary monitoring. IEEE Pervasive Computing 8, 2 (2009), 62--70. https://doi.org/10.1109/MPRV.2009.32Google ScholarDigital Library
- Abdelkareem Bedri, Richard Li, Malcolm Haynes, Raj Prateek Kosaraju, Ishaan Grover, Temiloluwa Prioleau, Min Yan Beh, Mayank Goel, Thad Starner, and Gregory Abowd. 2017. EarBit: Using Wearable Sensors to Detect Eating Episodes in Unconstrained Environments. ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) 1, 3, Article 37 (2017). https://doi.org/10.1145/3130902Google Scholar
- Abdelkareem Bedri, Apoorva Verlekar, Edison Thomaz, Valerie Avva, and Thad Starner. 2015. Detecting mastication: A wearable approach. In ACM International Conference on Multimodal Interaction. 247--250. https://doi.org/10.1145/2818346.2820767Google ScholarDigital Library
- Brooke Bell, Ridwan Alam, Nabil Alshurafa, Edison Thomaz, Abu Mondol, Kayla de la Haye, John Stankovic, John Lach, and Donna Spruijt-Metz. 2020. Automatic, wearable-based, in-field eating detection approaches for public health research: a scoping review. npj Digital Medicine 3, Article 38 (2020). https://doi.org/10.1038/s41746-020-0246-2Google Scholar
- France Bellisle. 2004. Impact of the daily meal pattern on energy balance. Scandinavian Journal of Nutrition 48, 3 (2004), 114--118. https://doi.org/10.1080/11026480410000454Google ScholarCross Ref
- Shengjie Bi, Tao Wang, Nicole Tobias, Josephine Nordrum, Shang Wang, George Halvorsen, Sougata Sen, Ronald Peterson, Kofi Odame, Kelly Caine, Ryan Halter, Jacob Sorber, and David Kotz. 2018. Auracle: Detecting Eating Episodes with an Ear-Mounted Sensor. ACM Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) 2, 3, Article 92 (2018). https://doi.org/10.1145/3264902Google ScholarDigital Library
- Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In ACM International Conference on Knowledge Discovery and Data Mining (KDD). 785--794. https://doi.org/10.1145/2939672.2939785Google ScholarDigital Library
- Keum San Chun, Sarnab Bhattacharya, and Edison Thomaz. 2018. Detecting Eating Episodes by Tracking Jawbone Movements with a Non-Contact Wearable Sensor. ACM Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) 2, 1, Article 4 (2018), 21 pages. https://doi.org/10.1145/3191736Google Scholar
- CM Doak, TLS Visscher, CM Renders, and JC Seidell. 2006. The prevention of overweight and obesity in children and adolescents: a review of interventions and programmes. Obesity Reviews 7, 1 (2006), 111--136. https://doi.org/10.1111/j.1467-789X.2006.00234.xGoogle ScholarCross Ref
- Yujie Dong, Adam Hoover, Jenna Scisco, and Eric Muth. 2012. A new method for measuring meal intake in humans via automated wrist motion tracking. Applied Psychophysiology and Biofeedback 37, 3 (2012). https://doi.org/10.1007/s10484-012-9194-1Google Scholar
- Yujie Dong, Jenna Scisco, Wilson Mike, Muth Eric, and Hoover Adam. 2014. Detecting periods of eating during free-living by tracking wrist motion. IEEE Journal of Biomedical and Health Informatics (JBHI) 18, 4 (2014). https://doi.org/10.1109/JBHI.2013.2282471Google Scholar
- Muhammad Farooq, Megan A McCrory, and Edward Sazonov. 2017. Reduction of energy intake using just-in-time feedback from a wearable sensor system. Obesity 25, 4 (2017), 676--681. https://doi.org/10.1002/oby.21788Google ScholarCross Ref
- Muhammad Farooq and Edward Sazonov. 2016. Detection of chewing from piezoelectric film sensor signals using ensemble classifiers. In Annual International Conference of the Engineering in Medicine and Biology Society (EMBC). https://doi.org/10.1109/EMBC.2016.7591833Google ScholarCross Ref
- Muhammad Farooq and Edward Sazonov. 2016. Segmentation and characterization of chewing bouts by monitoring temporalis muscle using smart glasses with piezoelectric sensor. IEEE Journal of Biomedical and Health Informatics (JBHI) 21, 6 (2016), 1495--1503. https://doi.org/10.1109/JBHI.2016.2640142Google ScholarCross Ref
- Juan M Fontana, Muhammad Farooq, and Edward Sazonov. 2014. Automatic ingestion monitor: A novel wearable device for monitoring of ingestive behavior. IEEE Transactions on Biomedical Engineering (TBE) 61, 6 (2014), 1772--1779. https://doi.org/10.1109/TBME.2014.2306773Google ScholarCross Ref
- Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189--1232. http://www.jstor.org/stable/2699986Google Scholar
- Beat Gfeller. 2011. Finding longest approximate periodic patterns. In Workshop on Algorithms and Data Structures (WADS). Springer, 463--474. https://doi.org/10.1007/978-3-642-22300-6_39Google ScholarCross Ref
- Berit Lilienthal Heitmann and Lauren Lissner. 1995. Dietary underreporting by obese individuals-is it specific or non-specific? British Medical Journal (BMJ) 311, 7011 (1995), 986--989. https://doi.org/10.1136/bmj.311.7011.986Google ScholarCross Ref
- Steve Hodges, Lyndsay Williams, Emma Berry, Shahram Izadi, James Srinivasan, Alex Butler, Gavin Smyth, Narinder Kapur, and Ken Wood. 2006. SenseCam: A retrospective memory aid. In International Conference on Ubiquitous Computing (UbiComp). Springer, 177--193. https://doi.org/10.1007/11853565_11Google ScholarDigital Library
- AK Illner, H Freisling, H Boeing, Inge Huybrechts, SP Crispim, and N Slimani. 2012. Review and evaluation of innovative technologies for measuring diet in nutritional epidemiology. International Journal of Epidemiology (IJE) 41, 4 (2012), 1187--1203. https://doi.org/10.1093/ije/dys105Google ScholarCross Ref
- Luis G Jaimes, Juan Calderon, Juan Lopez, and Andrew Raij. 2015. Trends in mobile cyber-physical systems for health just-in time interventions. In IEEE SoutheastCon. 1--6. https://doi.org/10.1109/SECON.2015.7132887Google Scholar
- Adrienne S Juarascio, Megan N Parker, Madeline A Lagacey, and Kathryn M Godfrey. 2018. Just-in-time adaptive interventions: A novel approach for enhancing skill utilization and acquisition in cognitive behavioral therapy for eating disorders. International Journal of Eating Disorders 51, 8 (2018), 826--830. https://doi.org/10.1002/eat.22924Google ScholarCross Ref
- Haik Kalantarian, Nabil Alshurafa, Tuan Le, and Majid Sarrafzadeh. 2015. Monitoring eating habits using a piezoelectric sensor-based necklace. Computers in Biology and Medicine 58 (2015), 46--55. https://doi.org/10.1016/j.compbiomed.2015.01.005Google ScholarDigital Library
- Haik Kalantarian, Nabil Alshurafa, and Majid Sarrafzadeh. 2014. A wearable nutrition monitoring system. In IEEE International Conference on Wearable and Implantable Body Sensor Networks (BSN). 75--80. https://doi.org/10.1109/BSN.2014.26Google ScholarDigital Library
- Rebecca M Leech, Anthony Worsley, Anna Timperio, and Sarah A McNaughton. 2015. Characterizing eating patterns: a comparison of eating occasion definitions. The American Journal of Clinical Nutrition (AJCN) 102, 5 (2015). https://doi.org/10.3945/ajcn.115.114660Google Scholar
- Jindong Liu, Edward Johns, Louis Atallah, Claire Pettitt, Benny Lo, Gary Frost, and Guang-Zhong Yang. 2012. An intelligent food-intake monitoring system using wearable sensors. In International Conference on Wearable and Implantable Body Sensor Networks (BSN). https://doi.org/10.1109/BSN.2012.11Google ScholarDigital Library
- BessH Marcus, Neville Owen, LeighAnnH Forsyth, NickA Cavill, and Fred Fridinger. 1998. Physical activity interventions using mass media, print media, and information technology. American Journal of Preventive Medicine (AJPM) 15, 4 (1998), 362--378. https://doi.org/10.1016/S0749-3797(98)00079-8Google ScholarCross Ref
- Mark Mirtchouk, Drew Lustig, Alexandra Smith, Ivan Ching, Min Zheng, and Samantha Kleinberg. 2017. Recognizing Eating from Body-Worn Sensors: Combining Free-living and Laboratory Data. ACM Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) 1, 3, Article 85 (2017), 20 pages. https://doi.org/10.1145/3131894Google Scholar
- Inbal Nahum-Shani, Eric B Hekler, and Donna Spruijt-Metz. 2015. Building health behavior models to guide the development of just-in-time adaptive interventions: A pragmatic framework. Health Psychology 34, S (2015), 1209. https://doi.org/10.1037/hea0000306Google Scholar
- Inbal Nahum-Shani, Shawna N Smith, Ambuj Tewari, Katie Witkiewitz, Linda M Collins, Bonnie Spring, and Susan Murphy. 2014. Just in time adaptive interventions (jitais): An organizing framework for ongoing health behavior support. Methodology Center Technical Report 2014 (2014), 14--126. https://doi.org/10.1007/s12160-016-9830-8Google Scholar
- Linda Neuhauser and Gary L Kreps. 2003. Rethinking communication in the e-health era. Journal of Health Psychology 8, 1 (2003), 7--23. https://doi.org/10.1177/1359105303008001426Google ScholarCross Ref
- Linda Neuhauser and Gary L Kreps. 2010. eHealth communication and behavior change: promise and performance. Social Semiotics 20, 1 (2010), 9--27. https://doi.org/10.1080/10350330903438386Google ScholarCross Ref
- Jun Nishimura and Tadahiro Kuroda. 2008. Eating habits monitoring using wireless wearable in-ear microphone. In IEEE International Symposium on Wireless Pervasive Computing (ISWPC). https://doi.org/10.1109/ISWPC.2008.4556181Google ScholarCross Ref
- Gillian O'Loughlin, Sarah Jane Cullen, Adrian McGoldrick, Siobhan O'Connor, Richard Blain, Shane O'Malley, and Giles D Warrington. 2013. Using a wearable camera to increase the accuracy of dietary analysis. American Journal of Preventive Medicine 44, 3 (2013), 297--301. https://doi.org/10.1016/j.amepre.2012.11.007Google ScholarCross Ref
- Vasileios Papapanagiotou, Christos Diou, Lingchuan Zhou, Janet van den Boer, Monica Mars, and Anastasios Delopoulos. 2017. A Novel Chewing Detection System Based on PPG, Audio, and Accelerometry. IEEE Journal of Biomedical and Health Informatics (JBHI) 21, 3 (2017), 607--618. https://doi.org/10.1109/JBHI.2016.2625271Google ScholarCross Ref
- Sarah E Pember and Adam P Knowlden. 2017. Dietary change interventions for undergraduate populations: Systematic review and recommendations. American Journal of Health Education 48, 1 (2017), 48--57. https://doi.org/10.1080/19325037.2016.1250018Google ScholarCross Ref
- J.M.C. Po, Jules Kieser, Luigi M Gallo, A.J. Tésenyi, P Herbison, and Mauro Farella. 2011. Time-frequency analysis of chewing activity in the natural environment. Journal of Dental Research 90, 10 (2011), 1206--1210. https://doi.org/10.1177/0022034511416669Google ScholarCross Ref
- Tauhidur Rahman, Alexander Travis Adams, Mi Zhang, Erin Cherry, Bobby Zhou, Huaishu Peng, and Tanzeem Choudhury. 2014. BodyBeat: a mobile system for sensing non-speech body sounds. In Annual International Conference on Mobile Systems, Applications, and Services (MobiSys). https://doi.org/10.1145/2594368.2594386Google ScholarDigital Library
- Sasank Reddy, Andrew Parker, Josh Hyman, Jeff Burke, Deborah Estrin, and Mark Hansen. 2007. Image Browsing, Processing, and Clustering for Participatory Sensing: Lessons from a DietSense Prototype. In Workshop on Embedded Networked Sensors (EmNets). 5. https://doi.org/10.1145/1278972.1278975Google ScholarDigital Library
- Polly Ryan and Diane Ruth Lauver. 2002. The efficacy of tailored interventions. Journal of Nursing Scholarship 34, 4 (2002), 331--337. https://doi.org/10.1111/j.1547-5069.2002.00331.xGoogle ScholarCross Ref
- Jenna Scisco, Eric Muth, and Adam Hoover. 2014. Examining the utility of a bite-count-based measure of eating activity in free-living human beings. Journal of the Academy of Nutrition and Dietetics (JAND) 114, 3 (2014), 464--469. https://doi.org/10.1016/j.jand.2013.09.017Google ScholarCross Ref
- Sougata Sen, Vigneshwaran Subbaraju, Archan Misra, Rajesh Balan, and Youngki Lee. 2018. Annapurna: Building a Real-World Smartwatch-Based Automated Food Journal. In IEEE International Symposium on "A World of Wireless, Mobile and Multimedia Networks" (WoWMoM). https://doi.org/10.1109/WoWMoM.2018.8449755Google ScholarCross Ref
- Sougata Sen, Vigneshwaran Subbaraju, Archan Misra, Rajesh Krishna Balan, and Youngki Lee. 2015. The case for smartwatch-based diet monitoring. In IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom workshops). https://doi.org/10.1109/PERCOMW.2015.7134103Google ScholarCross Ref
- Donna Spruijt-Metz and Wendy Nilsen. 2014. Dynamic models of behavior for just-in-time adaptive interventions. IEEE Pervasive Computing 13, 3 (2014), 13--17. https://doi.org/10.1109/MPRV.2014.46Google ScholarCross Ref
- STMicroelectronics. 2019. STM32 Nucleo expansion board. Date accessed: 11/11/2019.Google Scholar
- Akio Tada and Hiroko Miura. 2018. Association of mastication and factors affecting masticatory function with obesity in adults: a systematic review. BMC oral health 18, 1 (2018), 76. https://doi.org/10.1186/s12903-018-0525-3Google Scholar
- Jason Tang, Charles Abraham, Elena Stamp, and Colin Greaves. 2015. How can weight-loss app designers' best engage and support users? A qualitative investigation. British Journal of Health Psychology (BJHP) 20, 1 (2015), 151--171. https://doi.org/10.1111/bjhp.12114Google ScholarCross Ref
- Edison Thomaz, Irfan Essa, and Gregory D. Abowd. 2015. A Practical Approach for Recognizing Eating Moments with Wrist-mounted Inertial Sensing. In ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp). https://doi.org/10.1145/2750858.2807545Google Scholar
- Edison Thomaz, Aman Parnami, Irfan Essa, and Gregory D. Abowd. 2013. Feasibility of Identifying Eating Moments from First-Person Images Leveraging Human Computation. In Proceedings of International SenseCam & Pervasive Imaging Conference (SenseCam). https://doi.org/10.1145/2526667.2526672Google Scholar
- Edison Thomaz, Cheng Zhang, Irfan Essa, and Gregory D. Abowd. 2015. Inferring Meal Eating Activities in Real World Settings from Ambient Sounds: A Feasibility Study. In International Conference on Intelligent User Interfaces (IUI). https://doi.org/10.1145/2678025.2701405Google Scholar
- James T Tufano and Bryant T Karras. 2005. Mobile eHealth interventions for obesity: a timely opportunity to leverage convergence trends. Journal of Medical Internet Research (JMIR) 7, 5 (2005). https://doi.org/10.2196/jmir.7.5.e58Google Scholar
- Koji Yatani and Khai N. Truong. 2012. BodyScope: A Wearable Acoustic Sensor for Activity Recognition. In ACM Conference on Ubiquitous Computing (UbiComp). https://doi.org/10.1145/2370216.2370269Google Scholar
- Rui Zhang and Oliver Amft. 2017. Monitoring chewing and eating in free-living using smart eyeglasses. IEEE Journal of Biomedical and Health Informatics (JBHI) (2017). https://doi.org/10.1109/JBHI.2017.2698523Google Scholar
- Shibo Zhang, Rawan Alharbi, Matthew Nicholson, and Nabil Alshurafa. 2017. When Generalized Eating Detection Machine Learning Models Fail in the Field. In ACM International Joint Conference on Pervasive and Ubiquitous Computing and ACM International Symposium on Wearable Computers (UbiComp). https://doi.org/10.1145/3123024.3124409Google Scholar
Index Terms
- NeckSense: A Multi-Sensor Necklace for Detecting Eating Activities in Free-Living Conditions
Recommendations
Auracle: Detecting Eating Episodes with an Ear-mounted Sensor
In this paper, we propose Auracle, a wearable earpiece that can automatically recognize eating behavior. More specifically, in free-living conditions, we can recognize when and for how long a person is eating. Using an off-the-shelf contact microphone ...
Detecting Eating Episodes by Tracking Jawbone Movements with a Non-Contact Wearable Sensor
Eating is one of the most fundamental human activities, and because of the important role it plays in our lives, it has been extensively studied. However, an objective and usable method for dietary intake tracking remains unrealized despite numerous ...
Leveraging Sound and Wrist Motion to Detect Activities of Daily Living with Commodity Smartwatches
Automatically recognizing a broad spectrum of human activities is key to realizing many compelling applications in health, personal assistance, human-computer interaction and smart environments. However, in real-world settings, approaches to human action ...
Comments