Modeling and identification of emotional aspects of locomotion

https://doi.org/10.1016/j.jocs.2012.10.001Get rights and content

Abstract

The studies of emotional facial expressions and emotional body language are currently receiving a lot of attention in the cognitive sciences. In this project, we study implicit bodily expression of emotions during standard motions, such as walking forwards.

An underlying assumption of our work is that all human motion is optimal in some sense and that different emotions induce different objective functions, which result in different deformations of normal motion.

We created a 3D rigid-body model of a human of which we use the forward dynamics simulation in an optimal control context. We performed two kinds of optimizations: (i) reconstruction of dynamic quantities, such as joint torques, of pre-recorded data of emotional walking motions and (ii) forward optimization that generates neutral and varied walking motions using different objective functions. Optimizations are performed with the software package MUSCOD-II, which uses a direct multiple-shooting discretization scheme. The results of this work form the foundation for further analysis of emotional motions using inverse optimal control methods.

Highlights

► In this study we investigate of full-body emotional walking motions. ► We use a model based approach using the dynamics simulation of a rigid-body model. ► We use optimal control methods to reconstruct dynamic quantities from mo-cap data. ► We also generate walking motions by optimization using different criteria. ► Results of this work are the base for further analysis using inverse optimal control.

Introduction

We encounter emotions every day. Most of us “speak” and “read” emotional body language whether we want or not. Our perception is very sensitive to human motion and already small cues suffice for us to recognize gender and emotion of a walking person, without even seeing the face. Yet there is little known about why we express emotions in these specific ways or how emotions influence our motor system.

Already Darwin [5] asked whether facial expressions of emotions are inherent or they are learned during lifetime. He also proposed that expressions which can be found in both humans and animals, such as the sneer, are due to a common genetic ancestors. More recently Ekman and Friesen [6] created the Facial Action Coding System (FACS), which is able to categorize and encode almost every anatomically possible facial expression. It is actively used by animators or psychologists. A similar system for emotional body language does not exist. But to create one would be a tedious task due to the huge amount of muscles.

But emotions are not only expressed in faces. Already simple visualizations such as point light displays of a walking person suffices for us to be able to recognize emotions Atkinson et al. [1]. This is demonstrated by the computer program “BML Walker” described in Troje [20]. It allows specification of physical properties such as sex, weight, and also emotional aspects nervous-relaxed and happy-sad via sliders. A point light display of a motion that fits to the chosen parameter is generated on-the-fly. Generation of emotional motions therefore seems to be possible. But how can we use this to gain insights into emotions?

Research of emotions is also being done in neuroscience. The somatic marker hypothesis [4] states that our decision making is influenced not only by logical reasoning, but also by our emotions. Furthermore, the perception of emotional body language is being done by neural systems that are responsible for perceiving action and triggering of behaviour [8]. This supports the idea that emotional body language is not only static information of the current state of a person, instead it also conveys cues about his or her action. From this perspective, it comes naturally to look at emotions in human motions.

Kinematic analysis of emotional walking motions was done in Omlor and Giese [14]. They used motion capture data and used a new non-linear blind-source separation method. It efficiently finds only a few source components that approximate high-dimensional emotional walking. This allows one to simulate different emotional styles on a kinematic skeleton. Furthermore these spatio-temporal primitives are specific for different emotions and indicate emotion-specific primitives in the human motor system.

Only little research of emotional body language has so far been done by using (rigid-body) dynamics. In Liu et al. [12] a learning method was used to create stylized motions for physics-based animation system. It allows one to learn a parameter vector θ from motion-capture data, which can then be applied to different motions. The vector θ contains informations such as elasticity parameters for shoe contact, muscles and tendons and also preferences for certain muscles. The method was used to learn emotional styles, but focus of this research is on generation of stylized animations and not on emotional body language itself.

With our research, we want to explore emotional body language on a kinetic level. One of the assumptions of our work is that human motions are optimal in some sense. Optimization has already been successfully used to create human like motions such as running [19] or platform diving [9]. We therefore use a model-based approach to combine kinematic motion-capture recordings with a rigid-body model by using optimal control methods. This allows us to look at emotional body language on a level of forces and torques that act in or on the body. One of the key questions we have is, whether it is possible to relate certain emotions to specific goals in the form of objective functions. In this paper we want to show our preliminary results that will form the base for future work.

Section snippets

Emotional human walking

In our research we investigate full-body 3D walking gaits that are varied by emotions. However to be able to discuss alterations of a walk, we first have to describe a neutral walking gait.

Experiments

In 2010 we performed a case-study with 2 lay actors (one male, one female) at the Laboratoire de Physiologie de la Perception et de l’Action (LPPA) Paris. Recordings of full-body walking motions were made with a VICON motion capture system. We used a standard Plug-in-Gait model for the marker placements. The subjects were asked to walk through the motion capture area, which is about 6 steps in length. There were three different modes of recordings: neutral, passive emotion, and acted emotion.

Dynamics of human walking

We use a model-based approach for our research similar to previous efforts such as Schultz and Mombaur [19] and Koschorreck and Mombaur [9]. For us the term model is primarily associated with a physical rigid-body model of the human locomotor system that is subject to external forces like gravity, ground reaction forces and internal forces (torques in the joints). The purpose of our model is to enable us to test various hypothesises that we pose on emotions that are listed in Table 1. In this

Using optimal control to approximate measurement data by a dynamical model

The data from our experiments only contain kinematic marker data and therefore does not allow any analysis of the dynamics within the body. Using optimal control methods we can use the data from the motion capture as input and reconstruct dynamics quantities such as joint torques for the previously described model. To do so we formulated an optimal control problem which uses a least-squares objective function which tries to approximate the motion of our model to a recorded motion.

Optimal

Optimizing walking motions using different criteria

Instead of trying to reconstruct pre-recorded motions we also computed walking motions without any input from motion capture data, similar to the running motions generated in Schultz and Mombaur [19]. The idea is to replace the least-squares functional in Eq. (3a) with an appropriate objective function that generates a gait with a certain emotion. A general optimal control problem is then formulated as:

minx(·),u(·),p,ti0tfΦL(t,x(t),u(t),p)dt+ΦM(tf)subject to:x˙(t)=fi(t,x(t),u(t),p)x(ti+1+)=hi(x

Conclusion and outlook

We have investigated reconstruction of pre-recorded motion capture data by using optimal control methods. Furthermore we generated optimal human gaits for different objective functions. The results of the reconstructions showed that a more realistic foot model is required. This is currently in progress and should allow better approximations of the recorded motions. The forward optimization allows us to test different objective function that results in gaits of different style. Already using a

Acknowledgements

The authors gratefully acknowledge the financial support and the inspiring environment provided by the Heidelberg Graduate School of Mathematical and Computational Methods for the Sciences, funded by DFG (Deutsche Forschungsgemeinschaft).

Martin L. Felis received his diploma in mathematics with specialization in optimal control from the University of Heidelberg in 2009. He has been since then a PhD student of the Heidelberg Graduate School of Mathematical and Computational Methods for the Sciences and a member of the research group Optimization in Robotics and Biomechanics. His research interests include, multi-body dynamics, screw theory, animation, and optimal-control methods.

References (21)

  • P. de Leva

    Adjustments to zatsiorsky-seluyanov's segment inertia parameters

    Journal of Biomechanics

    (1996)
  • L. Omlor et al.

    Extraction of spatio-temporal primitives of emotional body expressions

    Neurocomput

    (2007)
  • A.P. Atkinson et al.

    Emotion perception from dynamic and static body expressions in point-light and full-light displays

    Perception

    (2004)
  • G.A. Bliss

    Lectures on the Calculus of Variations. Number 504 in Phoenix Books; 504; Phoenix Science Series; Phoenix Books

    (1976)
  • S.H. Collins et al.

    Dynamic arm swinging in human walking

    Proceedings of the Royal Society B: Biological Sciences

    (2009)
  • A.R. Damasio et al.

    The somatic marker hypothesis and the possible functions of the prefrontal cortex [and discussion]

    Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences

    (1996)
  • C. Darwin

    The Expression of the Emotions in Man and Animals

    (1872)
  • P. Ekman et al.

    Facial Action Coding System: A Technique for the Measurement of Facial Movement

    (1978)
  • P. Ekman et al.

    Constants across cultures in the face and emotion

    Journal of Personality and Social Psychology

    (1971)
  • B. de Gelder

    Towards the neurobiology of emotional body language

    Nature Review Neuroscience

    (2006)
There are more references available in the full text version of this article.

Cited by (8)

  • Design of optimized soft soles for humanoid robots

    2017, Robotics and Autonomous Systems
    Citation Excerpt :

    Recent studies show that humanoid walking on flat floors reached a good level of maturity. Indeed, humanoid robots can walk with human-in-loop [2], with human stylized gaits [3,4] or with emotional behavior gaits [5]. In the latter works, the precise control of the contact interaction between the humanoid robot feet and the ground plays an important role.

  • Generation of movements with boundary conditions based on optimal control theory

    2017, Robotics and Autonomous Systems
    Citation Excerpt :

    Systems like this were extended to even more complex systems by building skeleton models [16]. Such skeleton models can be used for even very complex cases like modelling and identification of emotional movements [17]. In our study, instead of building models for control of specific motions for particular robots, we present a general framework for generation of on-line trajectories (similar to DMPs, GMMs and PMPs) based on optimal control theory (i.e., robot unspecific).

  • Adjustments to de Leva-anthropometric regression data for the changes in body proportions in elderly humans

    2015, Journal of Biomechanics
    Citation Excerpt :

    The mass distribution to the segments of the elderly human body, although lacking the geometric properties of the body segments, has been addressed by Jensen and Fletcher (1994). The de Leva data has been used in many different biomechanical applications involving the dynamics of human motions and modeling in terms of rigid body systems, e.g. to perform inverse dynamic analysis and estimate the energy expenditure in human motions (Ackermann and Schiehlen, 2006), to examine causes for injury during sudden motions (McLean et al., 2004), to study emotional body expressions in human walking (Felis et al., 2013), or to investigate foot placement during human locomotion after perturbations (Millard et al., 2012). It has also been used in computer graphics to ensure generated animations of human character motions that obey the laws of physics (Popović and Witkin, 1999) and in orthopedics medicine to describe the effect of foot orthoses on the kinematics and kinetics of human gait (Nester et al., 2003).

  • Assuring consistency in mixed models

    2014, Journal of Computational Science
    Citation Excerpt :

    Section 5 present the mixed model and Section 6 concludes the paper. Modeling and validation has been used extensively in the literature [3–13]. Hassan and Logrippo [20] proposed a method to detect inconsistencies of multiple security policies mixed together in one system and to report the inconsistencies at the time when the secrecy system is designed.

  • Parameter identification method for process control systems based on the Newton iteration

    2017, 2017 6th International Symposium on Advanced Control of Industrial Processes, AdCONIP 2017
View all citing articles on Scopus

Martin L. Felis received his diploma in mathematics with specialization in optimal control from the University of Heidelberg in 2009. He has been since then a PhD student of the Heidelberg Graduate School of Mathematical and Computational Methods for the Sciences and a member of the research group Optimization in Robotics and Biomechanics. His research interests include, multi-body dynamics, screw theory, animation, and optimal-control methods.

Katja Mombaur is professor at the Interdisciplinary Center of Scientific Computing (IWR) at the University of Heidelberg since 2010, and leads the research group on Optimization in Robotics & Biomechanics as well as the IWR Robotics Lab. She also holds an associate researcher status at LAAS-CNRS Toulouse, where she spent two years as a visiting researcher in 2008–2010. She studied Aerospace Engineering at the University of Stuttgart, Germany, and the ENSAE in Toulouse, France, and got her Diploma in 1995. For the next two years she worked at IBM Germany. She received her PhD degree in Applied Mathematics from the University of Heidelberg, Germany, in 2001. In 2002, she was a postdoctoral researcher at Seoul National University, South Korea. From 2003 to 2008, she worked as a lecturer and researcher at IWR, University of Heidelberg. Her research interests include modeling of complex mechanical systems and cognitive processes in biomechanics and robotics, as well as optimization and control techniques for fast motions.

Hideki Kadone received his diploma in information science and technology with specialization in mechano-informatics from the University of Tokyo in 2008. He was since then a postdoctoral researcher at the Laboratory of Physiology of Perception and Action at College de France and is now a researcher at Center for Cybernics Research in University of Tsukuba. His research interests include, emotion and gaze in locomotion and its application to robotic systems.

Alain Berthoz is currently honorary professor at the College de France, member of the French Academy of Sciences and Academy of Technology, the Academia Europae, American Academy of Arts and Sciences and other Academies (Belgium, Bulgaria). He was the founder and director of the Laboratory of Physiology of Perception and Action of CNRS/College de France. He is an engineer, psychologist and neurophysiologist and has made his carreer at the CNRS as a scientist in the field of cognitive neuroscience. He is an author of more than 250 papers in international journals and the author of several books, such as The Brain's Sense of Movement (Harvard Univ Press), Emotion and Reason (Oxford Univ. Press), Principles of Simplexity (Yale Univ. Press; 2011).

View full text