MULTIPLE INSTANCE LEARNING FOR HUMAN EMOTION ANALYSIS USING GABOR FEATURES

: Facial expression analysis (FEA) or Human Emotion Analysis (HEA) is an essential tool for human computer interaction. The nonverbal messages of humans are expressed by facial expression. In this study, an HEA system to classify seven classes of human emotions like happy, sad, angry, disgust, fear, surprise and neutral is presented. It uses Gabor filter for feature extraction and Multiple Instance Learning (MIL) for classification. Gabor filter analyzes the facial images in a localized region to extract specific frequency content in specific directions. Then, MIL classifier is used for the classification of emotions into any one of the seven emotions. The evaluation of HEA system is carried on JApanese Female Facial Expression (JAFFE) database. The overall recognition rate of the HEA system using Gabor and MIL technique is 95%.


I. INTRODUCTION
The human beings express their emotions, intensions and other nonverbal messages by facial expression.Among the various behaviors of humans like speaking and gaits, only the facial expression shows the exact mind set of humans.Support Vector Machine (SVM) and active shape model based facial expression classification is discussed in [1].The facial expression images are given to active shape model.Then, textural information features are extracted from the region of interest area.SVM model is used for classification of facial expression images.An emotion care service system using SVM classification model is discussed in [2].The edges are detected by canny edge detection algorithm.SVM classifier is used for emotion recognition using facial features.
Action unit classification based facial expression is presented in [3].The Gabor filter and susan operator are used for feature extraction.The dynamic Bayesian network is used for classification of facial images.A facial expression recognition system using Zernike moments and fuzzy classification is discussed in [4].At first, pseudo Zernike moments are extracted from the facial images.Then, the classification is made by fuzzy inference system.
Electroencephalogram and gyroscope signals using facial expression classification is described in [5].Initially, the input facial images are pre-processed by data interval extraction in time and frequency domain.Then, the zero mean and forward Fourier transform is applied.The classification is made by linear discriminant analysis, SVM and naïve Bayes classifier.Deep extreme extraction network based facial expression classification is described in [6].The input facial expression images are classified by the deep convolutional neural network based Xception model.
A sparse representation of k-svd is used for facial expression classification in [7].At first, the k-svd is applied to input facial images for the sparse representation.Then, SVM classifier is trained to classify the facial images.Quantum Neural Network (QNN) based feature difference matrix for facial expression recognition is described in [8].The input facial images are divided into several expression blocks.The feature difference matrix is obtained from neural expression block.Then, the QNN is used for the classification of feature difference matrix.
Sparse representation based facial expression recognition is described in [9].At first, pyramid histogram of oriented gradients is extracted.The Sparse Representation Classifier (SRC) is used for the classification.Fuzzy logic and facial effective areas for facial expression recognition is described in [10].Features are extracted by calculating the openness of eyes, eyebrows, mouth and density of mouth.Then the fuzzy inference classifier is used for the classification.
Binarized statistical image features based facial expression recognition is discussed in [11].The local feature descriptors for the binarized statistical image features are extracted.The classification is made by SRC.SVM ensemble classification for facial expression recognition is described in [12].From the facial images, features are extracted by base level feature extraction technique.Then, higher level feature extraction is done after normalization.
The classification and regression of different facial expression is discussed in [13].The local binary pattern with global features and six distance vector are extracted for expression analysis using SVM classifier.The recognition of facial expression is discussed in [14] using SVM and geometric deformation features.The geometric features are extracted to classify the emotions by the SVM classifier.
In this study, MIL based HEA system using Gabor features is presented.The organization of this paper is as follows: The HEA system is described in section 2. The results and discussions about the performance of HEA system are discussed in section 3. The last section concludes the HEA system.

II. METHODS AND MATERIALS
The HEA system using Gabor filter and MIL classifier is shown in Fig. 1.The following seven human emotions; happy, sad, angry, fear, disgust, neutral and surprise are recognized by the HEA system.The JAFFE images are given to Gabor filter to extract the features.The extracted features are classified by MIL classifier.The detailed workflow of the HEA system is explained in the following sub-sections.

A. Gabor Filter
It is a linear filter and used for texture analysis in many applications and it finds if there is any specific frequency components at specific directions.In human visual system, the Gabor filter is analyzed by many scientists.This is appropriate for texture discrimination and representation.The sinusoidal plane wave is used to modulate the two-dimensional Gabor filter with Gaussian kernel function.It is related to wavelets as it provides multi-resolution of the given input data.The two-dimensional Gabor filter is defined by, where (u,v) is the spatial domain pixel position,  is the Gabor filter orientation,  is the radial center frequency, and  is the standard deviation of the Gaussian function in u-axis and v-axis.

Fig. 1 HER system using Gabor filter and MIL classifier
The feature extraction using Gabor filter is made by, The different frequencies and orientations of Gabor filter are used to extract features of emotions of humans from facial images.Gabor filter is also used in other fields like image fusion [15] and texture classification [16].In this study, features are extracted from the different emotional images like happy, sad, angry, neutral, surprise, disgust and fear using Gabor filter.The Gabor filter bank uses eight orientations and five frequencies for feature extraction.In filter bank, the one Gabor filter is placed at 60 degrees.The 60 pixels in the Gabor wavelet are analyzed to express the standard deviation.Figure 2 shows the Gabor filters used to extract emotional features.

B. MIL Classification
MIL is a machine learning algorithm with supervised learning.It is also used in image annotation [17] and similarity classification [18].In this study, the MIL classifier is used to classify the different facial expression class.The individually labeled set of instances is received by MIL and also a set of labeled bags received by learner which contains more instances.In binary classification, if labeled bag is negative then all instances are negative.Otherwise the labeled bag is positive if at least one instance is positive.There are no labels in the individual instance then MIL classifies the hidden bags or instances based on labeled bags.The MIL is defined as follows,

III. RESULTS AND DISCUSSION
JAFFE database [19] is employed for performance evaluation of HEA system designed in section 2. It can be freely downloadable [20].All face images are captured in gray scale format with frontal pose.It contains one neutral image with six basic emotions such as happiness, sadness, surprise, anger, disgust, fear and neutral.Figure 3 shows emotional images from 3 Japanese females from the database.The size of the tiff images is 256x256 pixels.All the images are considered by HEA system for the performance evaluation.
Among the total images in the database, 2/3 rd of images are used for training MIL classifier and remaining images are used for testing.The system is considered as a subject dependent recognition system.Figure 5 shows the recognition accuracy of HEA system using Gabor features and MIL classifier.

Fig. 4 Recognition accuracy of HEA system using Gabor features and MIL classifier
As the surprise emotion has lot of variations in the facial images, the HEA system successfully classified surprise emotion with no misclassification.Among the various emotions, the emotions such as Disgust and Sad has recognized with minimum accuracy.The overall accuracy of HEA system is 92.5% which shows

IV. CONCLUSION
In this study, a reliable automated HEA system is presented using Gabor filter based features and MIL classifier.It is implemented with two sequential modules.The first module extracts the appropriate features that discriminate human emotions using Gabor filter.MIL classifier is adopted for emotion recognition stage where the emotions are classified into seven classes.The HEA system can be used in applications such as affective computing, video surveillance and human computer interaction.In future, the system can be implemented to find the diagnostic information such as mania, schizophrenia and depression.Results show that the HEA system using Gabor filter and MIL classifier provides promising results for emotion classification.

Fig. 3
Fig. 3 JAFFE database images features extracts dominant features to classify the emotions successfully with the help of MIL classifier.