Self-aware face emotion accelerated recognition algorithm: a novel neural network acceleration algorithm of emotion recognition for international students

Construct dataset

Existing emotion datasets are biased toward facial information, such as IAPS (Lang, Bradley & Cuthbert, 2005), FlickrLDL, and TwitterLDL (Yang, Sun & Sun, 2017), which generally only have head information, and lack the person’s physical description and background information, making it difficult for the NN to respond to real emotional changes, and this type of data is scarce in reality. In this study, we created and annotated a dataset of 600 images of international students’ emotions. We could achieve a more accurate emotion recognition model by responding to the emotional changes of international students more effectively using this dataset. The dataset includes comprises students from various countries who gave their permission for their social media data to be distorted and deformed to achieve diversity in the dataset.

Portrait recognition algorithm selection

In this study, experiments were performed using several public datasets to assess the recognition abilities and performance of different convolutional neural networks (CNNs), and a selection was made to demonstrate the efficacy of our approach.

Novel emotion recognition algorithm

In this study, a self-aware accelerated facial emotion recognition algorithm (SFEARA) is suggested. The algorithm increases the real-time performance of the CNNs in facial emotion recognition (FER), preprocesses the image, and uses hybrid accuracy for the feature extraction of different multiple regions to lower reduce the computation and energy consumption in convolutional operations.

Related work

In this study, we focus on facial emotion recognition of international students, and through certain online social media image data of university students during their study period, our proposed algorithm can recognize their emotions faster and improve the real-time analysis capability of the entire emotion recognition system. In the existing public benchmarks, human expressions are generally classified into six categories, namely happy, anger, disgust, sad, surprise, fear, but there are new datasets released since 2020’s that include compound expressions and thus reach 11 categories. Presently, the application scenarios of expression recognition are primarily human–computer interaction, precise direction of advertising, and student status recording. FER can be divided into two categories: traditional recognition methods (Lyons et al., 1998) and deep learning methods (Zhang, Quan & Ren, 2016), both of which can perform emotion recognition. However, the energy consumption of various processing methods varies. This section discusses the existing methods for sentiment recognition, including traditional methods and deep learning methods.

Application of traditional emotion recognition in emotion recognition scenarios

Feature extraction in traditional emotion recognition relies heavily on mathematical methods and computer technology to organize and process data from digital images of facial expressions, extract expression features and remove non-expression noise. This feature extraction algorithm is for the image’s main features, which objectively reduce the image’s dimensionality, and therefore these feature extraction algorithms affect dimensionality reduction. At this point, the expression feature extraction algorithms are classified as static image-based feature extraction methods (e.g., holistic and local methods) and dynamic image-based feature extraction methods (e.g., optical flow, model, and geometric methods), with the distinction, made based on the state of the expressions and the object to be processed. Among them, static images show a single image of the expression when it occurs, whereas dynamic images show the movement of the expression between multiple images.

Several studies (Wang, Huang & Wang, 2011; Zou, 2008; Ying & Cai, 2009; Ma & Khorasani, 2004) have been conducted as holistic feature extraction methods using static images. The classical algorithms in holistic methods include PCA, independent component analysis (ICA), and linear discriminant analysis (LDA). Researchers have significantly contributed to this effort. Wang, Huang & Wang (2011), Zou (2008) extracted expression features using the FastICA algorithm, which not only inherits the ICA algorithm’s ability to extract hidden information between pixels but also can quickly complete the separation of expression features through iteration. Ying & Cai (2009) proposed a support vector discriminant analysis algorithm based on Fisher LDA and support vector machine that can maximize the inter-class separation of expression data with small sample data and does not require the decision function required to develop the support-vector machine (SVM) algorithm. Ma & Khorasani (2004) used the two which they then combined with neural networks to classify expression features. Because facial expressions on still images have both overall variations and local variations, the information contained in local variations, such as texture and folds of facial muscles, can aid in accurately determining expression properties. Therefore, studies by Kyperountas, Tefas & Pitas (2010), Zheng et al. (2006), Fu & Wei (2008) examined the local information of still images. Kyperountas, Tefas & Pitas (2010) used various feature extraction algorithms such as Gabor and wavelets, as well as a new classifier for experiments on still images. In Zheng et al. (2006), 34 face feature points are first manually labeled, then their the Gabor wavelet coefficients of the feature points are represented as marker map vectors, and finally the KCCA coefficients between the marker map vectors and the expression semantic vectors are calculated to achieve expressions classification.

Yacoob & Davis (1996) proposed using the optical flow field and gradient field between consecutive frames to represent the temporal and spatial variation of the image, respectively, to achieve the expression region tracking for each face image; then, the motion direction change of the feature region is used to represent the motion of facial muscles, which corresponds to different expressions. Tsalakanidou & Malassiotis (2010) proposed an ASM-based 3D face feature tracking method that tracks and models 81 face feature points to recognize certain composite action units. Wang & Yin (2007) proposed a method to recognize face movements and expressions using an image terrain feature model. The method is to procure face expression regions by tracking face feature points using AAM and manual annotation and then procure topographic features by computing the topographic histogram of face expression regions to achieve expression recognition. Sung & Kim (2008) proposed an AAM algorithm based on apparent 2D features and 3D shape features to achieve expression in an environment where the face position is shifted feature extraction. Other researchers have claimed that the generation and expression of expressions are heavily dependent on the changes in the facial organs that are represented. The major organs of the face and their folds are the regions where expression features are concentrated. Therefore, marking feature points on the facial organ regions and calculating the distance between feature points and the curvature of the curve where the feature points are situated becomes a method for extracting facial expressions using geometric forms. For example, the study by Kotsia & Pitas (2007) includes gridding the faces with different expressions using a deformation grid and using the change in grid node coordinates between the first and maximum frames of that expression sequence as geometric features for expression recognition.

The above methods are popular in emotion recognition scenarios, but because of their high computational effort and unsatisfactory accuracy, new solutions are urgently required to address the urgent need for emotion recognition scenarios. For example, wearing a face mask for social distancing related to COVID-19 reduces the recognition rate of facial expressions (Pazhoohi, Forby & Kingstone, 2021).

Application of convolutional neural network in sentiment recognition scenarios

Deep learning differs from traditional methods, especially CNN has a strong feature extraction capability for images, which can avoid the complex algorithm of traditional extraction feature methods and has higher recognition efficiency. Traditional face recognition extracts features from images through models, such as the commonly used 68 facial feature points, with low efficiency and accuracy, while deep learning can learn features more generally without a dedicated feature extraction step, greatly improving the efficiency and fault tolerance of recognition. Recently, using deep neural networks (DNNs) to solve FER problems has become prevalent because of the advancement of DNNs, which has aided in the advancement of sentiment analysis. As computational power increases, the structure of DNNs provides a good foundation for computer self-learning, allowing it to identify complex patterns in text, images, and sounds. Therefore, using DNNs to solve the FER problem is extremely appropriate. Among them, convolutional neural networks (CNNs) (Zhang, Quan & Ren, 2016) are popular in computer vision, and facial emotion recognition systems can be designed in a less time by feature extraction and classification during training, and such systems outperform traditional methods in real-time inference. With more research and practice, multiple network models such as AlexNet, GoogLeNet (Inception), VGGNet, and ResNet have performed well. These networks have been tested using a powerful dataset ImageNet, and therefore they are often used for image classification problems. For the construction of the network, the features of the image are usually extracted by a CNN layer, and then a fully connected layer performs nonlinear classification, similar to MLP (Multilayer perceptron), except those mechanisms, such as dropout, are added to avoid overfitting. In the last layer, different classifications correspond to different neural units, and the samples obtained from the variations of softmax belong to different types of probability distributions. Miller & Fellbaum (1991) proposed large datasets with 1 million images to annotate natural emotional facial expressions, which is quite applicable to computer vision, affective computing, social cognition, and neuroscience. Corneanu, Madadi & Escalera (2018) proposed a deep neural network architecture that solves both issues by combining them in the initial stage. Wang et al. (2022) proposed a new scheme for compressing datasets by aligning features (CAFE) that explicitly attempts to preserve the true feature distribution and the discriminative power of the resulting synthetic set, making it useful for various of architectures with a high degree of generality and improves performance by 11%. A very efficient CNN model optimization-based on AdaMax and ReLu algorithm (Sezer & Altan, 2021) was proposed to accurately identify faulty welds at the early stage. The robust hybrid classification model (Yağ & Altan, 2022) was designed with the flower pollination algorithm (FPA) and support vector machine (SVM) to accurately classify plant leaf diseases in real time, and got an 8.1% improvement in the same condition. The OCNN model (Tang et al., 2023) was improved by Octave convolution algorithm and DyReLU function, obtained a higher accuracy and little lower computational consumption.

Table 1 compares the results of the overall approach, the local approach and the CNN for image accuracy (acc) and single image running time (sirt) in sentiment process recognition for a single image.

Generally, recognizing sentiment using deep learning is the most suitable approach, but the real difficulty is caused by the performance of processing applications. Although these methods are extensively used in industry and academia, there are few studies that balance accuracy and efficiency, or greatly improve efficiency, especially in the field of education management of international students. Regarding the question “how to process input data quickly and obtain better results in the FER domain”, in this study, a novel neural network acceleration algorithm of emotion recognition for international students is proposed to respond to improve the quality of education management, and has achieved a significant improvement in efficiency with almost no decrease in accuracy and more complete metrics, such as single image runtime (sirt), average mask (am), and the sum of running time on the whole dataset (SumRuntime). This may indeed help universities to understand the psychological conditions of students and take timely action, and also lower the applicable threshold.

CNN overview

CNN has two parts; they are feature extraction and full connection classification. Figure 1 describes the forward processing of CNNs Feature extraction belongs to the convolutional layer while full connection classification belongs to the full connection layer (Goodfellow et al., 2016; Gu et al., 2018). In the convolutional layer, it analyzes the input data set in batches and obtains different abstract features. The input includes H, W, and channel (H and W are the image resolution ratio while the channel is the number of channels), e.g., r, g, b. Assuming that the convolutional layer uses the filters of dimension (Df:Hf:Wf) to extract feature map a, a_i:j is the value of the current feature map in row i and column j, as shown in Eq. (1):

(1) $$a_{i,j}=f\left(\sum _{d=1}^{D_f}\sum _{m=1}^{H_f}\sum _{n=1}^{W_f}\left(w_{d,m,n}\times x_{d,i,j}\right)+w_b\right).$$

D ${}_f$ stands for depth, H ${}_f$ stands for height, W ${}_f$ stands for width, and f is an activation function such as sigmoid, hyperbolic tangent (tanh), or rectified linear units (ReLU). Training primarily includes forward propagation, backward propagation, and kernel update. The inference process is forward propagation only. The network model is generated by the training of CNN, which requires to be fine-tuned and optimized before starting to perform inference on different data sets and then completing the inference.

Construct dataset

In this study, we used publicly available image sentiment datasets, including MIRFLICKR-25K (Huiskes & Lew, 2008), MIRFLICKR-1M (Huiskes, Thomee & Lew, 2010), and Fer2013 (Goodfellow et al., 2015). Images containing human faces were selected from these three datasets using random sampling, and emotions were classified as pleasant, normal, sad, and angry. The MIRFLICKR-25K dataset is a collection of 25,000 images downloaded from the public API of the social photography site Flickr, plus full human annotations. MIRFLICKR-1M was obtained using the same data collection method as MIRFLICKR-25K, but without manual annotation of all images, all of which are available under a Creative Commons Attribution License. The Fer2013 dataset is derived from the Kaggle Facial Expression Contest dataset and comprises seven categories of emotional images. We randomly selected 1,269 portrait images only with one of facial sentiment in pleasure, normal, sadness, and anger from MIRFLICKR-25K, of which 80% of the images were divided as training set and 20% as the test set. A total of 23,308 portrait images only with one of facial sentiment in pleasure, normal, sadness, and anger were randomly selected from the MIRFLICKR-1M dataset, of which 80% of the images were used as the training set, and 20% of the images were divided as the test set. They are checked and annotated by hand, also compared the features extracted from two datasets. If the contrast ratio exceeds 99%, it is considered the same image and an image is deleted. We just selected the images which have only one emotion to guarantee the images have the same annotations as Fer2013. The Fer2013 dataset comprises 35,886 images of faces with different expressions, of which there are 28,708 images in the training set and 3,589 images in each of the validation and test sets. We used the dataset’s test set to compare the inference results’ accuracy and efficiency. To study the required consistency, in this study, the three datasets were uniformly reclassified, grouping the emotion categories into pleasure, normal, sadness, and anger. We use the unified classification data of these three recognized categories for model training in preparation for the real-time inference session.

To some extent, text data on social media can accurately reflect college students’ emotional states. There are various types of social media, and this study examines “Twitter” and “Facebook”, the mainstream media society as well as face emotion recognition (FER). In addition to the above public dataset, 150 portrait images posted on Twitter and Facebook by college students from different countries (Japan/Taiwan/India/English/South Africa/America, etc.) were collected in this study to obtain a more suitable model for international students. These images are expanded from 150 to 600 images by image processing such as perspective transformation (Mezirow, 1978), affine transformation (Modenov & Parkhomenko, 1965) and Gaussian blur (Hummel, Kimia & Zucker, 1987). The expansion in this manner improves the robustness of the model trained by CNNs. To improve the classification accuracy, images in social media are generally equipped with text, and we use the SVM vector machine algorithm to classify the text in the images, and the text corresponds to the corresponding category of the images. This dataset can help us develop a robust emotional inference model for international students, which can be used to study the emotional changes of college students during their life and study. In this study, to annotate the dataset more accurately, the SVM vector machine algorithm is used to pre-annotate the emotion of the text in the life status posted by international students, and then the corresponding images are manually proofread to achieve accurate annotation of the ISSM DataSet. The generalizability of this study is confirmed by fine-tuning the trained model using ISSM DataSet and employing inference operations. For this purpose, we use 70% of the images in the ISSM DataSet as the fine-tuned training dataset and 30% as the inference dataset. The classification of images in each dataset is shown in Table 2.

Preprocessing algorithm selection

In this study, we focus on the emotion recognition of international students; to be able to improve the real-time emotion detection, we use a face detection algorithm to preprocess the data and improve the efficiency of emotion recognition by masking during fine-tuning and inference, and the accuracy will only be reduced within 3.3%. To suppress the uncertainties efficiently and prevent deep networks from over-fitting uncertain facial images, Wang et al. (2020) proposed a simple yet efficient SelfCure Network (SCN). We analyzed the most popular face recognition algorithms. Faster R-CNN, SSD-300, and multi-task convolutional neural network (MTCNN), and selected SSD-300 based on the results of accuracy and single image processing efficiency.

Faster R-CNN (Ren et al., 2017) integrates feature extraction, suggestion extraction, rect refine, and classification in a single network, allowing for comprehensive performance improvements, particularly in detection speed. First, Faster R-CNN uses a set of basic conv+relu+pooling layers to extract the feature maps of the image. Second, the RPN network is used to generate region proposals. This layer determines whether the anchor point is positive or negative by softmax, and then uses bounding box regression to correct the anchor point to obtain an accurate positive point. Third, this layer collects the input feature maps and proposals, combines this information to extract the proposed feature maps, and sends them to the subsequent fully connected layers to determine the target class. Finally, box regression, obtain the final accurate location of the detection box.

Single Shot MultiBox Detector (SSD) (Liu et al., 2015) removes the bounding box proposal and the subsequent pixel/feature resampling, which is considerably faster than Faster R-CNN and has higher detection accuracy than Faster R-CNN in terms of prediction on feature maps of different scales. The core of SSD prediction first determines the prediction box category (the one with the highest confidence) and confidence value, and filters out the prediction boxes belonging to the background and those with a low confidence threshold; then encodes the remaining prediction boxes to get the real position parameters (after decoding, it requires to clip to prevent the prediction box position from exceeding the decoding, the prediction boxes are sorted in the descending order as per the confidence level, and the top-k prediction boxes are retained. Finally, the non-maximum suppression algorithm is performed to filter out those prediction boxes with large overlap, and the remaining prediction boxes are the results.

Multi-task convolutional neural network (MTCNN) (Zhang et al., 2016) is used to balance performance and accuracy, avoiding the huge performance consumption associated with traditional ideas such as sliding window plus classifier. First, the test images are successively resized to obtain image pyramids. Second, a small model (P-Net) is used to generate target region candidate frames with certain probability. Third, a more complex model (R-Net) is used to finely classify and regress the frames in the region, and let this step proceed recursively. Finally, many candidate images are filtered out by R-Net and the results are input to (O-Net) to get the exact frame coordinates and landmark coordinates.

In this study, we measured the accuracy (acc), single image runtime (sirt), and average mask (am) of three face recognition networks in different datasets. Based on the experimental results of face detection accuracy, running time, and average mask as shown in Table 3, we selected the SSD recognition network with good accuracy and highest operational efficiency as the face preprocessing algorithm for SFEARA emotion recognition. Considering the effect of SSD preprocessing algorithm on overall performance, we selected the confidence as the key parameter and evaluated the best setting on our dataset, although we used the default parameters of SSD. The result showed that the confidence parameter setting in the 0.001–0.011 balances performance and precision best as Fig. 2, and its default value is 0.001.

Design of SFEARA

The traditional emotion recognition methods and CNNs methods at the present stage are inefficient in the face of high real-time inference scenarios and cannot adapt to the urgent requirements of the day. To improve the speed of facial emotion recognition, this article proposes a SFEARA. Figure 3 depicts the use of the SFEARA method in the inference process. The algorithm has three advantages:

1. the input information can be preprocessed and decoupled using edge devices;

2. the local feature library is less noisy, and independent local is more conducive to global learning;

3. local learning is adaptive at different stages, and local learning is actually more appropriate at later stages.

The algorithm comprises three main parts. First, we preprocess the image with SSD, whose process is shown in Fig. 4, which includes convolution operations, determination of prediction frames, and classification. Then, combine the face detector api in OpenCV (Nasrollahi & Moeslund, 2013) and the model provided by the model in Ren et al. (2014) to detect key points on the face and fuse the separated image data in a way as shown in Fig. 5. The approach is to detect key points of faces, filter non-face information, and qualify object frames twice, and by fusing in non-key regions, the result is more accurate information in face detection images. Next, a quadratic regression method based on LBF (local binary features) is proposed in Ren et al. (2014) to solve the face alignment problem; this method is based on the idea of local first and then whole. Finally, using the key point information identified by FacemarkLBF, after dividing the key and non-key regions, the CNN is finally used for inference classification, as shown in Fig. 6. The specific steps described by Algorithm 1.

As shown in Fig. 4, we use the object region coordinates obtained by SSD on the edge device and use the object region coordinates and the image to be predicted as the input to the algorithm in this study. Using the region information of object detection for face key point detection is an important of preprocessing step.

As shown in Fig. 5, we use the loaded face detector method to scan the object detection region data and determine the key points of faces. Because all face keypoint detection algorithms are a truncated face image, here we require to use the Haar face detector in OpenCV (Nasrollahi & Moeslund, 2013), and the program flow of process is as follows.

1. Use Haar-like features for detection.

2. Use an integral map to accelerate the solution of Haar-like feature values.

3. Use AdaBoost algorithm as a solid classifier for distinguish between human and non-human faces.

4. Cascade the strong classifier using filter cascade to improve the accuracy.

Then, avoid memory leaks by creating FaceMark objects using smart pointers (PTR). The face detector is then trained by loading a 300-W dataset. Each frame is run on the face detector to obtain the coordinates of the important regions of the face, and multiple rectangular containers are developed by limiting the coordinates of the key regions with a pixel threshold, whose main purpose is to limit the number of pixels in each container. The more containers there are, the fewer pixels are contained in a single container. Finally, the face information contained in the containers is used as the keypoint region. The keypoints are collected with different precisions to use them as input in subsequent inference. The keypoint regions inside the container are obtained by threshold restriction, and those outside the container are non-keypoint regions.

As shown in Fig. 6, we use CNNs to extract features from the preprocessing output, where the key regions of the features are inferred using a high precision model and the non-key regions of the features are inferred using a low progress model; Then, the results are fused and the prediction results are classified.

In forward propagation, the first layer of convolutional operations is computed with different accuracies based on the coordinates of critical and non-critical regions obtained using the above mentioned algorithm. When the region involved in the calculation is non-critical, we use low precision calculation, and when the region involved in the calculation is a critical region, a high precision calculation is used to calculate it. Based on the feature fusion of the convolution calculation, it is easy to determine the coordinates of the critical and non-critical regions after convolution. The tracking of the entire forward propagation of the convolution calculation can be achieved. The primary operation is to track the steps in the convolution process. The convolution kernel runs to the sensitive region, where we quantize the input features, converting from FP32 to INT8, and to the non-sensitive region, where we quantize the input features, converting from FP32 to INT4. This is a typical step in traditional quantization schemes because INT8 is sufficient for most inference workloads. Each time before computing the matrix-vector multiplier (MVM: matrix-vertex multiplier), the switch is made based on the coordinates of critical and non-critical regions. The computational accuracy is switched between critical and non-critical regions. When the coordinates of the filter run are in the critical region, we use int8 to represent the weight and feature values. The 8-bit convolution operation is performed in this region. When the coordinates of the filter run are in the non-critical region, we use int4 to represent the weight and eigenvalues and perform 4-bit convolution operations in this region.

Obviously, the computational consumption of the 4-bit operation is smaller than the 8-bit operation; this approach can effectively reduce the computation and improve inference performance. However, when the filter runs on the boundary of the mixed region, i.e., the critical region and the non-critical region, to ensure the feature information’s validity, we all use the int8 method for convolutional operation. This method can guarantee the accuracy of the NN, but we require to retrain the network model and fine-tune the weights such that it can be more adaptable to this bit change and reduce the impact on the accuracy of emotion recognition of faces.

Using the SFEARA algorithm, in this study, we focus on the person’s information using high precision and extract the background information using low precision. In this manner, people’s emotions in images can be recognized more quickly, and the computational effort and energy consumption in the emotion recognition process are reduced.