A facial expression recognition system based on supervised locally linear embedding
Introduction
In recent years, there has been a growing interest in improving all aspects of the intelligent human–computer interaction. It would be desirable to make use of more natural communication modes in human–computer interaction. As we know, facial expression plays an important role in human–human interaction, so the automatic recognition of natural facial expressions is a necessary step. The best known facial expression model was given in the study by Ekman and Friesen (1975). Ekman has argued that there are a neutral facial expression and six basic facial expressions corresponding to happiness, sadness, surprise, anger, disgust and fear. These seven classes of facial expressions are considered in our work.
In recent years, much has been written about automatic facial expression analysis and recognition methods. Since tracking of the face can provide information for expression recognition, many researchers have focus on this area. One of most popular approaches is optical flow tracking. Tian et al., 2001, Fasel, 2002 proposed the approaches based on neural networks. Yacoob and Davis, 1996, Black and Yacoob, 1997 presented the rule-based models. Wang et al., 1998, Gokturk et al., 2002 proposed the template-based methods. Bourel et al. (2002) proposed the state-based models using k-nearest neighbor classifier. Otsuka and Ohya, 1996, Muller et al., 2002 gave the probabilistic models such as hidden Markov models etc. But these methods depend heavily on how well the movement of the muscle is tracked on the human face.
In face recognition area, eigenface method is a well-known technique. In the training stage, a large number of images are used to generate a small set of feature images using principal component analysis (PCA). Lo et al., 2001, Bartlett, 1998, Zhao et al., 1999 used the PCA-based methods to recognize facial expression. It means that each facial expression has its own expression space as a subspace of the image space. Recognition of facial expression is then based on distance measure between the input image and each of the expression spaces. Since the manifold underlying the facial expression cannot be extracted to be “smooth”, or having a homogeneous density, PCA-based method has inherent limitations as a linear method.
Recently, Roweis and Saul (2000) proposed locally linear embedding algorithm (LLE), an unsupervised learning algorithm that can compute low dimensional, neighborhood-preserving embeddings of high dimensional data. The basic idea of LLE is global minimization of the reconstruction error of the set of all local neighbors in the data set, because the construction of a local embedding from a fixed number of nearest neighbors appears more appropriate than from a fixed subspace. The supervised LLE (SLLE) algorithm proposed by Ridder et al. (2002) used class label information when computing neighbors to improve the performance of classification.
In this paper, a facial expression recognition system based on SLLE is presented. In which, SLLE is used to reduce dimension and extract features. When followed by a simple minimum-distance classifier, the better performance compared with PCA is obtained in the experiments.
The rest of this paper is organized as follows: Section 2 introduces the steps and principle of SLLE algorithm. Section 3 shows the system structure and a detailed presentation of three modules. Section 4 presents the experimental results. Finally, a summary is given in Section 5.
Section snippets
Supervised locally linear embedding algorithm
LLE has been used for face posed analysis by Hadid et al. (2002) and hyperspectral image processing by Kim and Finkel (2003) since it has been proposed. LLE maps its inputs into a single global coordinate system of lower dimension, attempting to discover nonlinear structure in high dimensional data by exploiting the local symmetries of linear reconstructions. Its optimizations do not involve local minima though capable of generating highly nonlinear embeddings.
The LLE algorithm, summarized in
Description of the facial expression recognition system
Similar to other statistical models, the facial expression recognition system consists of training and testing stage. Fig. 5 shows the flowchart of the system. SLLE serves as a basis of extracting the feature vector of the subject’s face expression. Inherently, the learned models serve as the training data for minimum-distance classifier. The feature vector extracted with SLLE is subsequently used by the minimum-distance classifier to assign particular probability of each expression.
Here we
Experiments and results
Two image databases are used to test the validity of facial expression recognition method based on SLLE algorithm in our experiments. One is JAFFE facial expression database, a database of static facial expression images used by Lyons et al. (1998). Another is the motion image series.
Conclusion and future work
The automatic recognition of facial expression plays a very important role in intelligent interaction between the human and computer. In this paper, SLLE, a novel nonlinear dimension reduction algorithm, is used as a method of feature extraction in the facial expression recognition system because of its ability of preserving the original local geometry. When followed by a simple minimum-distance classifier, a better result than PCA-based method is obtained.
Nevertheless, the SLLE algorithm still
References (21)
- Bartlett, M., 1998. Face Image Analysis by Unsupervised Learning and Redundancy Reduction. PhD thesis, University of...
- et al.
Recognizing facial expressions in image sequences using local parameterized models of image motion
Internat. J. Comput. Vision
(1997) - Bourel, F., Chibelushi, C.C., Low, A.A., 2002. Robust facial expression recognition using a state-based model of...
- de Ridder, D., Duln, R.P.W., 2002. Locally linear embedding for classification. Available from:...
- et al.
Unmasking the Face
(1975) - Fasel, B., 2002. Head-pose invariant facial expression recognition using convolutional neural networks. In: Proc....
- Gokturk, S.B., Bouguet, J.-Y., Tomasi, C., Girod, B., 2002. Model-based face tracking for view-independent facial...
- Hadid, A., Koouropteva, O., Picetikainen, M., 2002. Unsupervised learning using locally linear embedding: Experiments...
- Kim, D.H., Finkel, L.H., 2003. Hyperspectral image processing using locally linear embedding. In: 1st Internat. IEEE...
- et al.
Face detection in color image series
Chin. J. Infrared Laser Eng.
(2001)