ABSTRACT
Determining when an individual is eating can be useful for tracking behavior and identifying patterns, but to create nutrition logs automatically or provide real-time feedback to people with chronic disease, we need to identify both what they are consuming and in what quantity. However, food type and amount have mainly been estimated using image data (requiring user involvement) or acoustic sensors (tested with a restricted set of foods rather than representative meals). As a result, there is not yet a highly accurate automated nutrition monitoring method that can be used with a variety of foods. We propose that multi-modal sensing (in-ear audio plus head and wrist motion) can be used to more accurately classify food type, as audio and motion features provide complementary information. Further, we propose that knowing food type is critical for estimating amount consumed in combination with sensor data. To test this we use data from people wearing audio and motion sensors, with ground truth annotated from video and continuous scale data. With data from 40 unique foods we achieve a classification accuracy of 82.7% with a combination of sensors (versus 67.8% for audio alone and 76.2% for head and wrist motion). Weight estimation error was reduced from a baseline of 127.3% to 35.4% absolute relative error. Ultimately, our estimates of food type and amount can be linked to food databases to provide automated calorie estimates from continuously-collected data.
- Oliver Amft, Holger Junker, and Gerhard Troster. 2005. Detection of Eating and Drinking Arm Gestures Using Inertial Body-worn Sensors. In Proceedings of the Ninth IEEE International Symposium on Wearable Computers (ISWC '05). IEEE Computer Society, Washington, DC, USA, 160--163. DOI: http://dx.doi.org/10.1109/ISWC.2005.17 Google ScholarDigital Library
- Oliver Amft, Martin Kusserow, and Gerhard Troster. 2009. Bite weight prediction from acoustic recognition of chewing. IEEE Transactions on Biomedical Engineering 56, 6 (2009), 1663--1672.Google ScholarCross Ref
- Oliver Amft, Mathias Stäger, Paul Lukowicz, and Gerhard Tröster. 2005. Analysis of Chewing Sounds for Dietary Monitoring. In Proceedings of the 7th International Conference on Ubiquitous Computing (UbiComp'05). 56--72. http://dx.doi.org/10.1007/11551201_4 Google ScholarDigital Library
- Oliver Amft and Gerhard Tröster. 2009. On-Body Sensing Solutions for Automatic Dietary Monitoring. IEEE Pervasive Computing 8, 2 (2009), 62--70. Google ScholarDigital Library
- Abdelkareem Bedri, Apoorva Verlekar, Edison Thomaz, Valerie Avva, and Thad Starner. 2015. Detecting Mastication: A Wearable Approach. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI '15). ACM, New York, NY, USA, 247--250. DOI: http://dx.doi.org/10.1145/2818346.2820767 Google ScholarDigital Library
- Oscar Beijbom, Neel Joshi, Dan Morris, Scott Saponas, and Siddharth Khullar. 2015. Menu-Match: Restaurant-Specific Food Logging from Images. In Proceedings of the 2015 IEEE Winter Conference on Applications of Computer Vision (WACV '15). 844--851. http://dx.doi.org/10.1109/WACV.2015.117 Google ScholarDigital Library
- Vinay Bettadapura, Edison Thomaz, Aman Parnami, Gregory D. Abowd, and Irfan Essa. 2015. Leveraging Context to Support Automated Food Recognition in Restaurants. In Proceedings of the 2015 IEEE Winter Conference on Applications of Computer Vision (WACV '15). IEEE Computer Society, Washington, DC, USA, 580--587. DOI:http://dx.doi.org/10.1109/WACV.2015.83 Google ScholarDigital Library
- Jingyuan Cheng, Oliver Amft, and Paul Lukowicz. 2010. Active Capacitive Sensing: Exploring a New Wearable Sensing Modality for Activity Recognition. In Proceedings of the 8th International Conference on Pervasive Computing (Pervasive'10). Springer-Verlag, Berlin, Heidelberg, 319--336. DOI: http://dx.doi.org/10.1007/978-3-642-12654-3_19 Google ScholarDigital Library
- Jingyuan Cheng, Bo Zhou, Kai Kunze, Carl Christian Rheinländer, Sebastian Wille, Norbert Wehn, Jens Weppner, and Paul Lukowicz. 2013. Activity Recognition and Nutrition Monitoring in Every Day Situations with a Textile Capacitive Neckband. In Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication (UbiComp '13 Adjunct). 155--158. http://doi.acm.org/10.1145/2494091.2494143 Google ScholarDigital Library
- Pei-Yu. Y. Chi, Jen-Hao. H. Chen, Hao-Hua. H. Chu, and Jin-Ling. L. Lo. 2008. Enabling Calorie-Aware Cooking in a Smart Kitchen. In Proceedings of the 3rd International Conference on Persuasive Technology (PERSUASIVE '08). Springer-Verlag, Berlin, Heidelberg, 116--127. DOI: http://dx.doi.org/10.1007/978-3-540-68504-3_11 Google ScholarDigital Library
- Yujie Dong, Adam Hoover, Jenna Scisco, and Eric Muth. 2012. A new method for measuring meal intake in humans via automated wrist motion tracking. Applied psychophysiology and biofeedback 37, 3 (2012), 205--215.Google Scholar
- Yujie Dong, Jenna Scisco, Mike Wilson, Eric Muth, and Andrew Hoover. 2014. Detecting periods of eating during free-living by tracking wrist motion. IEEE Journal of Biomedical and Health Informatics 18, 4 (2014), 1253--1260.Google ScholarCross Ref
- Adam Drewnowski, Susan Ahlstrom Renderson, Alissa Driscoll, and Barbara J Rolls. 1997. The Dietary Variety Score: assessing diet quality in healthy young and older adults. Journal of the American Dietetic Association 97, 3 (1997), 266--271. DOI: http://dx.doi.org/10.1016/S0002-8223(97)00070-9Google ScholarCross Ref
- Joey Hagedorn, Joshua Hailpern, and Karrie G. Karahalios. 2008. VCode and VData: Illustrating a New Framework for Supporting the Video Annotation Workflow. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '08). 317--321. http://doi.acm.org/10.1145/1385569.1385622 Google ScholarDigital Library
- R. J. Hill and P. S. W. Davies. 2001. The validity of self-reported energy intake as determined using the doubly labelled water technique. British Journal of Nutrition 85, 04 (2001), 415--430.Google ScholarCross Ref
- Azusa Kadomura, Cheng-Yuan Li, Koji Tsukada, Hao-Hua Chu, and Itiro Siio. 2014. Persuasive Technology to Improve Eating Behavior Using a Sensor-embedded Fork. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). 319--329. http://doi.acm.org/10.1145/2632048.2632093 Google ScholarDigital Library
- Haik Kalantarian and Majid Sarrafzadeh. 2015. Audio-based detection and evaluation of eating behavior using the smartwatch platform. Computers in Biology and Medicine 65 (10 2015), 1--9. DOI: http://dx.doi.org/10.1016/j.compbiomed.2015.07.013 Google ScholarDigital Library
- J. Lester, D. Tan, S. Patel, and A. J. B. Brush. 2010. Automatic classification of daily fluid intake. Pervasive Health (3 2010), 1--8. DOI:http: //dx.doi.org/10.4108/ICST.PERVASIVEHEALTH2010.8906Google Scholar
- Christopher Merck, Christina Maher, Mark Mirtchouk, Min Zheng, Yuxiao Huang, and Samantha Kleinberg. 2016. Multimodality Sensing for Eating Recognition. In Proceedings of the 10th International Conference on Pervasive Computing Technologies for Healthcare.Google ScholarDigital Library
- Austin Meyers, Nick Johnston, Vivek Rathod, Anoop Korattikara, Alex Gorban, Nathan Silberman, Sergio Guadarrama, George Papandreou, Jonathan Huang, and Kevin P Murphy. 2015. Im2Calories: towards an automated mobile vision food diary. In Proceedings of the IEEE International Conference on Computer Vision. 1233--1241. Google ScholarDigital Library
- Jon Noronha, Eric Hysen, Haoqi Zhang, and Krzysztof Z. Gajos. 2011. Platemate: Crowdsourcing Nutritional Analysis from Food Photographs. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). 1--12. http://doi.acm.org/10.1145/2047196.2047198 Google ScholarDigital Library
- Sebastian Päßler and Wolf-Joachim Fischer. 2011. Acoustical method for objective food intake monitoring using a wearable sensor system. In Pervasive Health.Google Scholar
- Sebastian Päßler, Matthias Wolff, and Wolf-Joachim Fischer. 2012. Food intake monitoring: an acoustical approach to automated food intake activity detection and classification of consumed food. Physiological measurement 33, 6 (2012), 1073.Google Scholar
- Shah Atiqur Rahman, Christopher Merck, Yuxiao Huang, and Samantha Kleinberg. 2015. Unintrusive Eating Recognition using Google Glass. In Pervasive Health.Google Scholar
- Tauhidur Rahman, Alexander T. Adams, Mi Zhang, Erin Cherry, Bobby Zhou, Huaishu Peng, and Tanzeem Choudhury. 2014. BodyBeat: A Mobile System for Sensing Non-speech Body Sounds. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '14). 2--13. http://doi.acm.org/10.1145/2594368.2594386 Google ScholarDigital Library
- Jenna L. Scisco, Eric R. Muth, and Adam W. Hoover.Google Scholar
- Sougata Sen, Vigneshwaran Subbaraju, Archan MISRA, Rajesh Krishna Balan, and Youngki Lee. 2015. The Case for Smartwatch-based Diet Monitoring. In Workshop on Sensing Systems and Applications Using Wrist Worn Smart Devices.Google Scholar
- Edison Thomaz, Irfan Essa, and Gregory D. Abowd. 2015. A Practical Approach for Recognizing Eating Moments with Wrist-mounted Inertial Sensing. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 1029--1040. DOI:http://dx.doi.org/10.1145/2750858.2807545 Google ScholarDigital Library
- Edison Thomaz, Aman Parnami, Jonathan Bidwell, Irfan Essa, and Gregory D. Abowd. 2013a. Technological Approaches for Addressing Privacy Concerns when Recognizing Eating Behaviors with Wearable Cameras. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '13). 739--748. http://doi.acm.org/10.1145/2493432.2493509 Google ScholarDigital Library
- Edison Thomaz, Aman Parnami, Irfan Essa, and Gregory D. Abowd. 2013b. Feasibility of Identifying Eating Moments from First-person Images Leveraging Human Computation. In Proceedings of the 4th International SenseCam and Pervasive Imaging Conference (SenseCam '13). 26--33. http://doi.acm.org/10.1145/2526667.2526672 Google ScholarDigital Library
- Edison Thomaz, Cheng Zhang, Irfan Essa, and Gregory D. Abowd. 2015. Inferring Meal Eating Activities in Real World Settings from Ambient Sounds: A Feasibility Study. In Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI '15). 427--431. http://doi.acm.org/10.1145/2678025.2701405 Google ScholarDigital Library
- Koji Yatani and Khai N. Truong. 2012. BodyScope: A Wearable Acoustic Sensor for Activity Recognition. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (UbiComp '12). 341--350. http://doi.acm.org/10.1145/2370216.2370269 Google ScholarDigital Library
- Bo Zhou, Jingyuan Cheng, P. Lukowicz, A. Reiss, and O. Amft. 2015a. Monitoring Dietary Behavior with a Smart Dining Tray. IEEE Pervasive Computing 14, 4 (10 2015), 46--56. DOI:http://dx.doi.org/10.1109/MPRV.2015.79Google Scholar
- Bo Zhou, Jingyuan Cheng, Mathias Sundholm, Attila Reiss, Wuhuang Huang, Oliver Amft, and Paul Lukowicz. 2015b. Smart table surface: A novel approach to pervasive dining monitoring. In 2015 IEEE International Conference on Pervasive Computing and Communications (PerCom). 155--162. DOI: http://dx.doi.org/10.1109/PERCOM.2015.7146522Google ScholarCross Ref
- Examining the Utility of a Bite-Count--Based Measure of Eating Activity in Free-Living Human Beings. Journal of the Academy of Nutrition and Dietetics 114, 3 (2014), 464--469.Google Scholar
Index Terms
- Automated estimation of food type and amount consumed from body-worn audio and motion sensors
Recommendations
Recognizing Eating from Body-Worn Sensors: Combining Free-living and Laboratory Data
Automated dietary monitoring solutions that can find when, what, and how much individuals consume are needed for many applications such as providing feedback to individuals with chronic disease. Advances in body-worn sensors have led to systems with high ...
Multimodality sensing for eating recognition
PervasiveHealth '16: Proceedings of the 10th EAI International Conference on Pervasive Computing Technologies for HealthcareWhile many sensors can monitor physical activity, there is no device that can unobtrusively measure eating at the same level of detail. Yet, tracking and reacting to food consumption is key to managing many chronic diseases such as obesity and diabetes. ...
Adaptive implicit interaction for healthy nutrition and food intake supervision
HCII'11: Proceedings of the 14th international conference on Human-computer interaction: towards mobile and intelligent interaction environments - Volume Part IIIThe current work is going to provide you information about our solution in the challenge of nutrition and food intake supervision, which has been developed lately. We will give an overview of the system and the implemented mechanisms, which were needed ...
Comments