Elsevier

Pattern Recognition Letters

Volume 26, Issue 15, November 2005, Pages 2374-2389
Pattern Recognition Letters

A facial expression recognition system based on supervised locally linear embedding

https://doi.org/10.1016/j.patrec.2005.04.011Get rights and content

Abstract

In this paper, a facial expression recognition system based on supervised locally linear embedding (SLLE) is introduced. The system consists of three modules: face detection, feature extraction with SLLE and classification. In face detection module, two independent characteristics, skin color characteristic and motion characteristic are used to detect face region, and a trained SVM is used to verify candidate regions. In feature extraction module, SLLE, a supervised learning algorithm that can compute low dimensional, neighborhood-preserving embeddings of high dimensional data is used to reduce data dimension and extract features. In classification module, minimum-distance classifier is used to recognize different expressions. The experiments show that the proposed method is superior to PCA-based method.

Introduction

In recent years, there has been a growing interest in improving all aspects of the intelligent human–computer interaction. It would be desirable to make use of more natural communication modes in human–computer interaction. As we know, facial expression plays an important role in human–human interaction, so the automatic recognition of natural facial expressions is a necessary step. The best known facial expression model was given in the study by Ekman and Friesen (1975). Ekman has argued that there are a neutral facial expression and six basic facial expressions corresponding to happiness, sadness, surprise, anger, disgust and fear. These seven classes of facial expressions are considered in our work.

In recent years, much has been written about automatic facial expression analysis and recognition methods. Since tracking of the face can provide information for expression recognition, many researchers have focus on this area. One of most popular approaches is optical flow tracking. Tian et al., 2001, Fasel, 2002 proposed the approaches based on neural networks. Yacoob and Davis, 1996, Black and Yacoob, 1997 presented the rule-based models. Wang et al., 1998, Gokturk et al., 2002 proposed the template-based methods. Bourel et al. (2002) proposed the state-based models using k-nearest neighbor classifier. Otsuka and Ohya, 1996, Muller et al., 2002 gave the probabilistic models such as hidden Markov models etc. But these methods depend heavily on how well the movement of the muscle is tracked on the human face.

In face recognition area, eigenface method is a well-known technique. In the training stage, a large number of images are used to generate a small set of feature images using principal component analysis (PCA). Lo et al., 2001, Bartlett, 1998, Zhao et al., 1999 used the PCA-based methods to recognize facial expression. It means that each facial expression has its own expression space as a subspace of the image space. Recognition of facial expression is then based on distance measure between the input image and each of the expression spaces. Since the manifold underlying the facial expression cannot be extracted to be “smooth”, or having a homogeneous density, PCA-based method has inherent limitations as a linear method.

Recently, Roweis and Saul (2000) proposed locally linear embedding algorithm (LLE), an unsupervised learning algorithm that can compute low dimensional, neighborhood-preserving embeddings of high dimensional data. The basic idea of LLE is global minimization of the reconstruction error of the set of all local neighbors in the data set, because the construction of a local embedding from a fixed number of nearest neighbors appears more appropriate than from a fixed subspace. The supervised LLE (SLLE) algorithm proposed by Ridder et al. (2002) used class label information when computing neighbors to improve the performance of classification.

In this paper, a facial expression recognition system based on SLLE is presented. In which, SLLE is used to reduce dimension and extract features. When followed by a simple minimum-distance classifier, the better performance compared with PCA is obtained in the experiments.

The rest of this paper is organized as follows: Section 2 introduces the steps and principle of SLLE algorithm. Section 3 shows the system structure and a detailed presentation of three modules. Section 4 presents the experimental results. Finally, a summary is given in Section 5.

Section snippets

Supervised locally linear embedding algorithm

LLE has been used for face posed analysis by Hadid et al. (2002) and hyperspectral image processing by Kim and Finkel (2003) since it has been proposed. LLE maps its inputs into a single global coordinate system of lower dimension, attempting to discover nonlinear structure in high dimensional data by exploiting the local symmetries of linear reconstructions. Its optimizations do not involve local minima though capable of generating highly nonlinear embeddings.

The LLE algorithm, summarized in

Description of the facial expression recognition system

Similar to other statistical models, the facial expression recognition system consists of training and testing stage. Fig. 5 shows the flowchart of the system. SLLE serves as a basis of extracting the feature vector of the subject’s face expression. Inherently, the learned models serve as the training data for minimum-distance classifier. The feature vector extracted with SLLE is subsequently used by the minimum-distance classifier to assign particular probability of each expression.

Here we

Experiments and results

Two image databases are used to test the validity of facial expression recognition method based on SLLE algorithm in our experiments. One is JAFFE facial expression database, a database of static facial expression images used by Lyons et al. (1998). Another is the motion image series.

Conclusion and future work

The automatic recognition of facial expression plays a very important role in intelligent interaction between the human and computer. In this paper, SLLE, a novel nonlinear dimension reduction algorithm, is used as a method of feature extraction in the facial expression recognition system because of its ability of preserving the original local geometry. When followed by a simple minimum-distance classifier, a better result than PCA-based method is obtained.

Nevertheless, the SLLE algorithm still

References (21)

  • Bartlett, M., 1998. Face Image Analysis by Unsupervised Learning and Redundancy Reduction. PhD thesis, University of...
  • M.J. Black et al.

    Recognizing facial expressions in image sequences using local parameterized models of image motion

    Internat. J. Comput. Vision

    (1997)
  • Bourel, F., Chibelushi, C.C., Low, A.A., 2002. Robust facial expression recognition using a state-based model of...
  • de Ridder, D., Duln, R.P.W., 2002. Locally linear embedding for classification. Available from:...
  • P. Ekman et al.

    Unmasking the Face

    (1975)
  • Fasel, B., 2002. Head-pose invariant facial expression recognition using convolutional neural networks. In: Proc....
  • Gokturk, S.B., Bouguet, J.-Y., Tomasi, C., Girod, B., 2002. Model-based face tracking for view-independent facial...
  • Hadid, A., Koouropteva, O., Picetikainen, M., 2002. Unsupervised learning using locally linear embedding: Experiments...
  • Kim, D.H., Finkel, L.H., 2003. Hyperspectral image processing using locally linear embedding. In: 1st Internat. IEEE...
  • Xufeng Ling et al.

    Face detection in color image series

    Chin. J. Infrared Laser Eng.

    (2001)
There are more references available in the full text version of this article.

Cited by (0)

View full text