Infrared facial expression recognition via Gaussian-based label distribution learning in the dark illumination environment for human emotion detection

Highlights

• The natural correlation ambiguity is revealed, and a novel label distribution is constructed.

• An end-to-end learning framework of FER is proposed in both feature learning and classifier learning.

• Experimental results demonstrate that the proposed model achieves the best performance.

Abstract

Facial expression recognition task as a crucial step for emotion recognition remains an open challenge that due to individual expression correlation/ambiguity. In this paper, to tackle these challenges, a novel model with the correlation emotion label distribution learning is proposed for near-infrared (NIR) facial expression recognition which associates multiple emotions with each expression depend on the similarity of expressions. Firstly, the similarities of the seven basic expressions are calculated, and then guide the correlation emotion label distribution by predicting the latent label probability distribution of the expression. Furthermore, the proposed model can be learned in an end-to-end manner via a constructed convolutional neural network to classify the six basic facial expressions. Experimental results on Oulu_CASIA database demonstrate that the proposed method has achieved the superior performance on NIR expression recognition.

Introduction

Emotion recognition by facial expression in human–computer interactive system [1], [2] is one of the challenging research topics in the field of artificial intelligence which has attracted plenty of attention in recent years. However, it is difficult to achieve natural and harmonious emotional interaction with traditional interaction methods such as keyboard, mouse, screen, and pattern, which is far from meeting the requirements for artificial intelligence [2]. Human expression [3] is the most important carrier of emotions perception and the most direct and obvious way of expressing emotions. Thus, facial expression recognition (FER) has important theoretical significance for improving the emotional interaction ability of computer [4], [5]. Furthermore, facial expression is arguably the most natural, powerful and immediate signal to communicate emotional states and intentions [1]. However, automatic FER is still difficult in an unconstrained real-life situation with the widespread use of deep learning techniques [6], [7]. It encounters various challenges caused by occlusion, face pose variations, illumination changes, head motion, expressions ambiguity and so on. An ideal automatic FER system is supposed to be able to tackle these challenges.

It is well-known that the active near-infrared (NIR) (780–1100 nm) imaging [8] is an alternative method to overcome the problem of illumination variations and even robust in near darkness. In Fig. 1, it can be observed that the different facial features between the NIR images and visible (VIS) images, such as wrinkles and texture (shown by red arrows). The NIR images are clear and without shadows while some dark areas caused by self-occlusion can be found in the VIS images (Fig. 1(a)). Li et al. [9] firstly develop an active NIR imaging system to recognize human face in different illuminations.

Over the past two decades, many FER algorithms [10], [11], [12] have been proposed to classify the six basic emotions including the anger (An), disgust (Di), fear (Fe), happy (Ha), sadness (Sa) and surprise (Su) in each facial image with a predefined emotion category. However, it is difficult to obtain the ground-truth of facial expression in practice. Usually, approximate approaches are adopted to acquire facial expressions. For example, the Oulu_SACIA database is collected by asking the subjects to make a facial expression based on an expression sample displayed in picture sequences under a laboratory-controlled environment according to facial action coding system (FACS) [13]. However, there are factors that may cause inaccurate results. First of all, the expression is formed by the combination of multiple facial action modules. It is not guaranteed that the changes in facial action module of the subject are completely the same. Secondly, the movements of the same facial action module are existed in the different expressions. As a result, even when two facial images are labeled with the same emotion, they might belong to quite different real emotions. Moreover, most of the emotions appear in a combination, mixed or composite form of basic emotions according to the wheel of emotions theory of Plutchik [14]. Furthermore, human express their feelings through the facial appearance that is often a fusion or compound of different emotions rather than a single basic feeling. And each basic emotion plays a different role in the expression. In this sense, the facial appearance expression is ambiguous or correlative, i.e., multiple expression numbers might be utilized to describe the appearance of human face. Thus, the single-label learning methods [15], [16] for identifying a basic emotion of each expression may fail to describe the correlation/ambiguity in different emotions which may not be applicable to real-life expression recognition applications.

To address the problems, a multi-output Laplacian dynamic ordinal regression method is proposed by Rudovic et al. [17], which can estimate the probability of each emotion label as well as their intensities. However, it assumes that each expression with one correct emotion label and outputs the emotion with the highest probability as a result, which may fail in the mixture emotion situation. Moreover, multi-label learning (MLL) [18] is suitable for describing each expression image with several related emotions for FER tasks when each basic emotion is considered as a single label. Li et al. [19] develop the database of VIS multi-label facial expression, and preserve the manifold structures of emotion labels and the local affinity of deep features to learn the distinguishing features of multi-label expressions. However, MLL fails to learn the degree of each emotion describing the expression. Gan et al. [20] employ a CNN and softed label with a diverse ensemble that associate multiple emotions with each expression. It has achieved impressive results. However, this method is only suitable for the VIS facial images, and it often fails in the NIR facial images. Because the features of the NIR images (Fig. 1(g)–(l)) are essentially different from that of the VIS facial images (Fig. 1(a)–(f)).

Thus, a new emotion label distribution learning method is proposed for NIR FER which assigns the value to each basic emotion to describe facial expressions. While the ground-truth emotion of facial image is considered to be the most relevant label to the image, those emotions close to the ground-truth emotion could be utilized to describe the facial image with lower relevance. Our proposed method allows direct modeling of different importance of each label to the instance, and thus can better match the nature of many real practical applications.

In this paper, an attempt is made to reveal the correlation between different frontal facial expressions which is independent of the datasets and universal inspired by our observations. Specially, not only do we need to understand the emotions associated with facial expressions, but also need to learn the description degree that each emotion describes the expression. And then, a novel automatic FER framework is proposed based on deep constructed convolutional neural network (CNN) and label distribution. In the first stage, the expression feature similarity is calculated by using the cosine distance between the feature vectors which are learned from NIR FER datasets with frontal face images. Then, the expression label distribution construction is learned via an end-to-end CNN. The contributions of our study can be summarized as follows.

• Based on the expression feature similarities, the natural correlation/ambiguity among expressions is revealed, and a novel label distribution is constructed in this paper. To the best of our knowledge, the natural relationships among different expressions are revealed and modeled for the first time.

• A new end-to-end learning framework of FER is proposed which learns correlation emotion label distribution and regresses ground-truth expression in both feature learning and classifier learning.

• Experimental results on the active NIR public datasets demonstrate that the proposed model achieves better performance than the state-of-the-art methods.

The rest of this paper is organized as follows. The correlation or ambiguity of the different expressions is revealed in Section 2. The details of the method we proposed are presented in Section 3. Experimental results on the dataset and analysis are provided in Section 4. Finally, Section 5 concludes this paper.