Improving real-life, heart rate based estimates of emotion by taking metabolic heart rate into account – a perspective and an example in cooking

• ¹ Netherlands Organisation for Applied Scientific Research (TNO), Netherlands
• ² University of Twente, Netherlands
• ³ Noldus Information Technology, Netherlands
• ⁴ United States Army Research, Development and Engineering Command, United States
• ⁵ Eaglescience Software BV, Netherlands
• ⁶ Unilever (Netherlands), Netherlands
• ⁷ Wageningen University & Research, Netherlands

Introduction Physiological variables such as heart rate carry information about mental state (cognitive and emotional state - here we simply refer to ‘emotion’). An important advantage over verbal methods as a way to probe emotion, is that heart rate is an implicit measure that can be measured continuously. Extracting information about emotion from heart rate in real life is challenged by the concurrent effect of physical activity on heart rate caused by metabolic need. “Non-metabolic heart rate”, which refers to the heart rate that is caused by factors other than physical activity, would be a more sensitive and more universally applicable correlate of emotion than heart rate itself. To precisely determine non-metabolic heart rate is a challenge. We explored evidence from the literature that non-metabolic heart rate, as it has been determined up until now, indeed reflects emotion. We focused on methods using accelerometry since these sensors are readily available in devices suitable for daily life usage. We found no convincing evidence that these existing methods lead to estimates of non-metabolic heart rate that reflect emotion. This is probably caused by the fact that intensity of motion signals as recorded from accelerometers does not correspond one-on-one with muscle activity in cases that forces are exerted without much movement (isometric muscle activity). In addition, the placement of the sensors do not always match the currently moving body parts, leading to invalid estimates. Studies in the field of energy expenditure as measured through variables extracted from respiratory gas exchange, show that energy expenditure can be well estimated using accelerometry by first identifying the type of action that is performed, rather than (only) using the intensity of motion signals. We therefore suggest that for real-life cases, estimating the type and intensity of activities based on accelerometry (and other information), and in turn use those to determine the non-metabolic heart rate, is the most promising route to determining non-metabolic heart rate precisely enough in order for it to be useful in estimating emotional state. In a study on estimating emotion during real-life cooking, we determined non-metabolic heart rate by correcting heart rate in an activity specific way. Rather than using a model that estimates specific activities based on accelerometry, we made an effort to estimate a baseline metabolic heart rate from participants performing movements that occur during cooking, without the emotion that occurs during cooking. The aim was to investigate the potential of non-metabolic heart rate to reflect emotionally salient phases during the process of cooking a dish. Methods 74 participants cooked and tasted a stir-fried chicken curry dish following a standardized protocol, presented by timed audio commands (schematically depicted in Figure 1). Heart rate as well as other physiological variables (not discussed here) were recorded. Before and after cooking and tasting, participants performed ‘dry cooking’ sessions in which they followed a standardized protocol to make cooking movements without actual food involved. As described above, this was done to acquire baseline physiology caused by physical activity only, without being associated with affective processes related to cooking. Half of the participants cooked using A-brand products and fresh ingredients, and half cooked with less attractive looking products and dried or canned ingredients. Differences between these ‘pleasantness’ conditions are not discussed here. Results Heart rate showed clearly reproducible patterns associated with (dry) cooking phase as defined by the audio commands (Figure 2). Heart rate was relatively high for two specific intervals during cooking (adding curry paste from a sachet and taking a bite), also after subtracting metabolic heart rate as estimated by the corresponding phases in the dry cooking sessions (Figure 3). The emotional saliency of these phases was confirmed by verbal retrospective reports of valence and arousal. Discussion When monitoring affective experience, physiological variables may be helpful since, in contrast to verbal reports, they do not interfere with the experience itself and they are of a continuous nature. However, when interpreting physiological variables in this context, it is important to take into account effects due to physical activity. We here provide handles emerging from the literature of how to proceed in the case of heart rate, and showed how this could work in the case of cooking. Our method to correct heart rate for effects caused by physical activity suggests that corrected heart rate can be used as continuous, implicit measure to identify emotionally salient phases during cooking and eating, and may be a valuable addition to verbal reports. Figure 1: Depiction of experimental procedure (auditory commands) over time and setup. Figure 2: Heart rate during cooking. The dashed lines represent audio commands, some of which have been labeled in the figure. The green and red curve represent two different groups of participants. Figure 3: Non-metabolic heart rate during several cooking phases