Abstract
The enormous use of facial expression recognition in various sectors of computer science elevates the interest of researchers to research this topic. Computer vision coupled with deep learning approach formulates a way to solve several real-world problems. For instance, in robotics, to carry out as well as to strengthen the communication between expert systems and human or even between expert agents, it is one of the requirements to analyze information from visual content. Facial expression recognition is one of the trending topics in the area of computer vision. In our previous work, a facial expression recognition system is delivered which can classify an image into seven universal facial expressions—angry, disgust, fear, happy, neutral, sad, and surprise. This is the extension of our previous research in which a real-time facial expression recognition system is proposed that can recognize a total of ten facial expressions including the previous seven facial expressions and additional three facial expressions—mockery, think, and wink from video streaming data. After model training, the proposed model has been able to gain high validation accuracy on a combined facial expression dataset. Moreover, the real-time validation of the proposed model is also promising.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ekman, P.: Facial expression and emotion. Am. Psychol. 48(4), 384 (1993)
Al-modwahi, AAM, Sebetela O, Batleng LN, Parhizkar B, Lashkari AH. Facial expression recognition intelligent security system for real time surveillance. In: Proceedings of the International Conference on Computer Graphics and Virtual Reality (CGVR). The Steering Committee of The World Congress in Computer Science, Computer âĂę; 2012. p. 1.
Li, H., Chutatape, O.: Automated feature extraction in color retinal images by a model based approach. IEEE Trans. Biomed. Eng. 51(2), 246–254 (2004)
Srinivas, S., Sarvadevabhatla, R.K., Mopuri, K.R., Prabhu, N., Kruthiventi, S.S., Babu, R.V.: A taxonomy of deep convolutional neural nets for computer vision. Front. Robot. AI. 2, 36 (2016)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. pp. 1097–1105 (2012)
Liu, M., Li, S., Shan, S., Chen, X.: Au-inspired deep networks for facial expression feature learning. Neurocomputing 159, 126–136 (2015)
Kim, B.K., Roh, J., Dong, S.Y., Lee, S.Y.: Hierarchical committee of deep convo- lutional neural networks for robust facial expression recognition. J. Multimodal User Interfaces. 10(2), 173–189 (2016)
Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2983–2991 (2015)
Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction ACM, pp. 435–442 (2015)
Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y., Dobaie, A.M.: Facial expression recogni- tion via learning deep sparse autoencoders. Neurocomputing 273, 643–649 (2018)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Luo, L., Xiong, Y., Liu, Y., Sun, X.: Adaptive gradient methods with dynamic bound of learning rate. In: Proceedings of the 7th International Conference on Learning Representations. New Orleans, Louisiana (2019)
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, IEEE, pp. 94–101 (2010)
Ng, H.W., Nguyen, V.D., Vonikakis, V., Winkler, S.: Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction ACM, pp. 443–449 (2015)
Koestinger, M., Wohlhart, P., Roth, P.M., Bischof, H.: Annotated facial landmarks in the wild: a large-scale, real-world database for facial landmark localization. In: IEEE International Conference on Computer Vision Workshops (ICCV Workshops), IEEE, pp. 2144–2151 (2011)
Ahmed, T.U., Hossain, S., Hossain, M.S., ul Islam, R., Andersson, K.: Facial expression recognition using convolutional neural network with data augmentation. In: 2019 Joint 8th International Conference on Informatics, Electronics Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision Pattern Recognition (icIVPR), pp. 336–341 (2019)
Perez, L., Wang, J.: The effectiveness of data augmentation in image classification using deep learning (2017). arXiv preprint arXiv:171204621
Gulli, A., Pal, S.: Deep learning with Keras. Packt Publishing Ltd (2017)
Shipman, J.W.: Tkinter 8.4 reference: a GUI for Python. New Mexico Tech Computer Center (2013)
Bradski, G., Kaehler, A.: Learning OpenCV: computer vision with the OpenCV library. “ O’Reilly Media, Inc. (2008)
Wilson, P.I., Fernandez, J.: Facial feature detection using Haar classifiers. J. Comput. Sci. Coll. 21(4), 127–133 (2006)
Bisong, E.: Google colaboratory. In: Building Machine Learning and Deep Learning Models on Google Cloud Platform, Springer, pp. 59–64 (2019)
Hossain, M.S., Rahaman, S., Kor, A.L., Andersson, K., Pattinson, C.: A belief rule based expert system for datacenter pue prediction under uncertainty. IEEE Trans. Sustain. Comput. 2(2), 140–153 (2017)
Ul Islam, R., Andersson, K., Hossain, M.S.: A web based belief rule based expert system to predict flood. In: Proceedings of the 17th International Conference on Information Integration and Web-based Applications & Services, ACM, p. 3 (2015)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Ahmed, T.U., Hossain, S., Hossain, M.S., Islam, R.U., Andersson, K. (2022). A Deep Learning Approach with Data Augmentation to Recognize Facial Expressions in Real Time. In: Kaiser, M.S., Ray, K., Bandyopadhyay, A., Jacob, K., Long, K.S. (eds) Proceedings of the Third International Conference on Trends in Computational and Cognitive Engineering. Lecture Notes in Networks and Systems, vol 348. Springer, Singapore. https://doi.org/10.1007/978-981-16-7597-3_40
Download citation
DOI: https://doi.org/10.1007/978-981-16-7597-3_40
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-7596-6
Online ISBN: 978-981-16-7597-3
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)