Skip to main content
SpringerLink
Log in
Book cover

Advances in Hybridization of Intelligent Methods pp 1–16Cite as

Deep Learning Approaches for Facial Emotion Recognition: A Case Study on FER-2013

  • Chapter
  • First Online:

Part of the book series: Smart Innovation, Systems and Technologies ((SIST,volume 85))

Abstract

Emotions constitute an innate and important aspect of human behavior that colors the way of human communication. The accurate analysis and interpretation of the emotional content of human facial expressions is essential for the deeper understanding of human behavior. Although a human can detect and interpret faces and facial expressions naturally, with little or no effort, accurate and robust facial expression recognition by computer systems is still a great challenge. The analysis of human face characteristics and the recognition of its emotional states are considered to be very challenging and difficult tasks. The main difficulties come from the non-uniform nature of human face and variations in conditions such as lighting, shadows, facial pose and orientation. Deep learning approaches have been examined as a stream of methods to achieve robustness and provide the necessary scalability on new type of data. In this work, we examine the performance of two known deep learning approaches (GoogLeNet and AlexNet) on facial expression recognition, more specifically the recognition of the existence of emotional content, and on the recognition of the exact emotional content of facial expressions. The results collected from the study are quite interesting.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Pantic, M.: Facial expression recognition. In: Encyclopedia of Biometrics, pp. 400–406. Springer, US (2009)

    Google Scholar 

  2. Ekman, P., Rosenberg, E.L. (eds.): What the face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System. Oxford University Press, Oxford, UK (2005)

    Google Scholar 

  3. Mehrabian, A.: Communication without words. Psychol. Today 2(4), 53–56 (1968)

    Google Scholar 

  4. Heylen, D.: Head gestures, gaze and the principles of conversational structure. Int. J. Humanoid Rob. 3(03), 241–267 (2006)

    Article  Google Scholar 

  5. Ochs, M., Niewiadomski, R., Pelachaud, C.: Facial Expressions of Emotions for Virtual Characters. The Oxford Handbook of Affective Computing, 261 (2014)

    Google Scholar 

  6. Liebold, B., Richter, R., Teichmann, M., Hamker, F.H., Ohler, P.: Human capacities for emotion recognition and their implications for computer vision. i-com, 14(2), pp. 126–137 (2015)

    Google Scholar 

  7. Clavel, C.: Surprise and human-agent interactions. Rev. Cogn. Linguist. 13(2), 461–477 (2015)

    Article  Google Scholar 

  8. Liebold, B., Ohler, P.: Multimodal emotion expressions of virtual agents, mimic and vocal emotion expressions and their effects on emotion recognition. In: Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp. 405–410. IEEE (2013)

    Google Scholar 

  9. Bahreini, K., Nadolski, R., Westera, W.: Towards multimodal emotion recognition in e-learning environments. Interact. Learning Environ. 24(3), 590–605 (2016)

    Article  Google Scholar 

  10. Akputu, K.O., Seng, K.P., Lee, Y.L.: Facial emotion recognition for intelligent tutoring environment. In: 2nd International Conference on Machine Learning and Computer Science (IMLCS’2013), pp. 9–13 (2013)

    Google Scholar 

  11. Shen, L., Wang, M., Shen, R.: Affective e—learning: Using “emotional” data to improve learning in pervasive learning environment. Educ. Technol. Soc. 12(2), 176–189 (2009)

    Google Scholar 

  12. Koutlas, A., Fotiadis, D.I.: An automatic region based methodology for facial expression recognition. In: IEEE International Conference on Systems Man and Cybernetics SMC, pp. 662–666 (2008)

    Google Scholar 

  13. Pantic, M., Rothkrantz, L.J.M.: Automatic analysis of facial expressions: The state of the art. Pattern Anal. Mach. Intell. IEEE Trans. 22(12), 1424–1445 (2000)

    Article  Google Scholar 

  14. Arca, S., Campadelli, P., Lanzarotti, R.: An automatic feature-based face recognition system. In: Proceedings of the 5th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’04) (2004)

    Google Scholar 

  15. Ekman, P.: Basic Emotions. Handbook of Cognition and Emotion, pp. 45–60 (1999)

    Google Scholar 

  16. Ortony, A., Clore, G., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988)

    Book  Google Scholar 

  17. Plutchik, R.: The nature of emotions. Am. Sci. 89(4), 344–350 (2001)

    Article  Google Scholar 

  18. Wang, W., Yang, J., Xiao, J., Li, S., Zhou, D.: Face recognition based on deep learning. In: International Conference on Human Centered Computing, pp. 812–820. International Publishing, Springer (2014)

    Google Scholar 

  19. Salakhutdinov, R., Hinton, G.: Deep boltzmann machines. In: Artificial Intelligence and Statistics, pp. 448–455 (2009)

    Google Scholar 

  20. Deng, L.: A tutorial survey of architectures, algorithms, and applications for deep learning. APSIPA Trans. Signal Inf. Process. 3, e2 (2014)

    Article  Google Scholar 

  21. Srivastava, N., Salakhutdinov, R.R.: Multimodal learning with deep boltzmann machines. In: Advances in Neural Information Processing Systems, pp. 2222–2230 (2012)

    Google Scholar 

  22. Căleanu, C.D.: Face expression recognition: A brief overview of the last decade. In: IEEE 8th International Symposium on Applied Computational Intelligence and Informatics (SACI), pp. 157–161 (2013)

    Google Scholar 

  23. Sariyanidi, E., Gunes, H., Cavallaro, A.: Automatic analysis of facial affect: A survey of registration, representation, and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015)

    Article  Google Scholar 

  24. Danelakis, A., Theoharis, T., Pratikakis, I.: A survey on facial expression recognition in 3D video sequences. Multimedia Tools Appl. 74(15), 5577–5615 (2015)

    Article  Google Scholar 

  25. Aung, D.M., Aye, N.A.: Facial expression classification using histogram based method. In: International Conference on Signal Processing Systems (2012)

    Google Scholar 

  26. Murthy, G.R.S., Jadon, R.S. Recognizing facial expressions using eigenspaces. In: IEEE International Conference on Computational Intelligence and Multimedia Applications. 3, pp. 201–207 (2007)

    Google Scholar 

  27. Thai, L.H., Nguyen, N.D.T., Hai, T.S.: A facial expression classification system integrating canny, principal component analysis and artificial neural network.(2011) arXiv preprint arXiv:1111.4052

  28. Perikos, I., Ziakopoulos, E., Hatzilygeroudis, I.: Recognizing emotions from facial expressions using neural network. In: IFIP International Conference on Artificial Intelligence Applications and Innovations, pp. 236–245. Springer, Heidelberg (2014)

    Google Scholar 

  29. Perikos, I., Ziakopoulos, E., & Hatzilygeroudis, I.: Recognize emotions from facial expressions using a SVM and neural network schema. In: Engineering Applications of Neural Networks, pp. 265–274. Springer International Publishing, (2015)

    Google Scholar 

  30. Anderson, K., McOwan, P.W.: A real-time automated system for the recognition of human facial expressions. IEEE Trans. Syst. Man Cybern. Part B (Cybern.), 36(1), 96–105 (2006)

    Google Scholar 

  31. Přinosil, J., Smékal, Z., Esposito, A.: Combining features for recognizing emotional facial expressions in static images. In: Esposito, A., Bourbakis, N.G., Avouris, N., Hatzilygeroudis, I. (eds.) Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction, pp. 56−69. Springer, Heidelberg (2008)

    Google Scholar 

  32. Shinohara, Y., Otsu, N.: Facial expression recognition using fisher weight maps. In: Proceedings Sixth IEEE International Conference on Automatic Face and Gesture Recognition, IEEE. pp. 499–504 (2004)

    Google Scholar 

  33. Yang, J., Zhang, D., Frangi, A.F., Yang, J.Y.: Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 131–137 (2004)

    Article  Google Scholar 

  34. Oh, S.K., Yoo, S.H., Pedrycz, W.: Design of face recognition algorithm using PCA-LDA combined for hybrid data pre-processing and polynomial-based RBF neural networks: Design and its application. Expert Syst. Appl. 40(5), 1451–1466 (2013)

    Article  Google Scholar 

  35. Mohammadi, M.R., Fatemizadeh, E., Mahoor, M.H.: PCA-based dictionary building for accurate facial expression recognition via sparse representation. J. Vis. Commun. Image Represent. 25(5), 1082–1092 (2014)

    Article  Google Scholar 

  36. Tang, Y.: Deep learning using linear support vector machines.(2013). arXiv preprint arXiv:1306.0239

  37. Ionescu, R.T., Popescu, M., Grozea, C.: Local learning to improve bag of visual words model for facial expression recognition. In: Workshop on Challenges in Representation Learning, ICML (2013)

    Google Scholar 

  38. Mollahosseini, A., Chan, D., & Mahoor, M.H. (2016, March). Going deeper in facial expression recognition using deep neural networks. In: IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE. pp. 1–10 (2016)

    Google Scholar 

  39. Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Zhou, Y.: Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing pp. 117–124. Springer, Heidelberg (2013)

    Google Scholar 

  40. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. (2014) arXiv preprint arXiv:1409.4842

  41. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ng, A.Y.: Large scale distributed deep networks. In: Advances in Neural Information Processing Systems, pp. 1223−1231 (2012)

    Google Scholar 

  42. Krizhevsky, A., Sutskever, I., Hinton. G.E.: ImageNet classification with deep convolutional neural networks. Part of: Adv. Neural Inf. Process. Syst. NIPS, 25 (2012)

    Google Scholar 

  43. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675−678. ACM (2014)

    Google Scholar 

  44. By Aphex34 (Own work) [CC BY-SA 4.0 (http://creativecommons.org/licenses/by-sa/4.0)], via Wikimedia Commons

  45. I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, Y. Zhou, C. Ramaiah, F. Feng, R. Li, X. Wang, D. Athanasakis, J. Shawe-Taylor, M. Milakov, J. Park, R. Ionescu, M. Popescu, C. Grozea, J. Bergstra, J. Xie, L. Romaszko, B. Xu, Z. Chuang, and Y. Bengio, Challenges in representation learning: A report on three machine learning contests, Neural Networks, vol. 64, pp. 59–63 (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering and Informatics, University of Patras, Patras, Greece

    Panagiotis Giannopoulos, Isidoros Perikos & Ioannis Hatzilygeroudis

  2. Technological Educational Institute of Western Greece, Patras, Greece

    Isidoros Perikos

Authors
  1. Panagiotis Giannopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Isidoros Perikos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ioannis Hatzilygeroudis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Panagiotis Giannopoulos .

Editor information

Editors and Affiliations

  1. School of Engineering, Department of Computer Engineering and Informatics, University of Patras, Patras, Greece

    Ioannis Hatzilygeroudis

  2. The Faculty of Engineering, Environment and Computing, School of Computing, Coventry University, Coventry, United Kingdom

    Vasile Palade

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Cite this chapter

Giannopoulos, P., Perikos, I., Hatzilygeroudis, I. (2018). Deep Learning Approaches for Facial Emotion Recognition: A Case Study on FER-2013. In: Hatzilygeroudis, I., Palade, V. (eds) Advances in Hybridization of Intelligent Methods. Smart Innovation, Systems and Technologies, vol 85. Springer, Cham. https://doi.org/10.1007/978-3-319-66790-4_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-66790-4_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-66789-8

  • Online ISBN: 978-3-319-66790-4

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation