Automatic facial expression analysis: a survey

Abstract

Over the last decade, automatic facial expression analysis has become an active research area that finds potential applications in areas such as more engaging human–computer interfaces, talking heads, image retrieval and human emotion analysis. Facial expressions reflect not only emotions, but other mental activities, social interaction and physiological signals. In this survey, we introduce the most prominent automatic facial expression analysis methods and systems presented in the literature. Facial motion and deformation extraction approaches as well as classification methods are discussed with respect to issues such as face normalization, facial expression dynamics and facial expression intensity, but also with regard to their robustness towards environmental changes.

Introduction

Facial expression analysis goes well back into the nineteenth century. Darwin [1] demonstrated already in 1872 the universality of facial expressions and their continuity in man and animals and claimed among other things, that there are specific inborn emotions, which originated in serviceable associated habits. In 1971, Ekman and Friesen [2] postulated six primary emotions that possess each a distinctive content together with a unique facial expression. These prototypic emotional displays are also referred to as so called basic emotions. They seem to be universal across human ethnicities and cultures and comprise happiness, sadness, fear, disgust, surprise and anger. In the past, facial expression analysis was primarily a research subject for psychologists, but already in 1978, Suwa et al. [3] presented a preliminary investigation on automatic facial expression analysis from an image sequence. In the nineties, automatic facial expression analysis research gained much inertia starting with the pioneering work of Mase and Pentland [4]. The reasons for this renewed interest in facial expressions are multiple, but mainly due to advancements accomplished in related research areas such as face detection, face tracking and face recognition as well as the recent availability of relatively cheap computational power. Various applications using automatic facial expression analysis can be envisaged in the near future, fostering further interest in doing research in different areas, including image understanding, psychological studies, facial nerve grading in medicine [5], face image compression and synthetic face animation [6], video-indexing, robotics as well as virtual reality. Facial expression recognition should not be confused with human emotion recognition as is often done in the computer vision community. While facial expression recognition deals with the classification of facial motion and facial feature deformation into abstract classes that are purely based on visual information, human emotions are a result of many different factors and their state might or might not be revealed through a number of channels such as emotional voice, pose, gestures, gaze direction and facial expressions. Furthermore, emotions are not the only source of facial expressions, see Fig. 1. In contrast to facial expression recognition, emotion recognition is an interpretation attempt and often demands understanding of a given situation, together with the availability of full contextual information.