Skip to main content
SpringerLink
Log in

A Review of Survey and Assessment of Facial Emotion Recognition (FER) by Convolutional Neural Networks

  • Conference paper
  • First Online:
Micro-Electronics and Telecommunication Engineering (ICMETE 2023)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 894))

  • 72 Accesses

Abstract

Computer vision and the area of artificial intelligence (AI) both heavily rely on the detection of facial expressions. This article concentrates on operations based on face images. It demonstrates how visual articulations are most important data facilitates, despite the limitless possibilities of how FER can be analyzed by using various instruments. This essay provides a succinct analysis of recent FER research. However, theoretical FER structure designs and their initial evaluations are displayed close by conventional FER approaches. The presentation of numerous FER views using the “start to finish” learning permission through critical associating authorization follows. As a result, this study will help in connecting a convolutional neural network (CNN) for some LSTM components (long transient memory). This paper concludes with a short poll, evaluation assessment, findings, and standards that serve as a standard for measurable connections between all of these FER studies and experiments. For students in FER, this audit can serve as a succinct manual that provides pertinent details and evaluation for recent tests. Additionally, knowledgeable examiners are searching for promising paths for future work.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Walecki R, Rudovic O (2017) Deep structured learning for facial expression intensity estimation. Image Vis Comput

    Google Scholar 

  2. Kaulard K, Cunningham DW, Bülthoff HH, Wallraven C (2012) The MPI facial expression database—a validated database of emotional and conversational facial expressions. PLoS ONE

    Google Scholar 

  3. Chu WS, Torre FD, Cohn JF (2017) Learning spatial and temporal cues for multi-label facial action unit detection. In: Proceedings of the 12th IEEE international conference on automatic face and gesture recognition, Washington, DC, USA, 30 May–3 June 2017

    Google Scholar 

  4. Gunawan AAS (2015) Face expression detection on Kinect using active appearance model and fuzzy logic. Procedia Comput Sci

    Google Scholar 

  5. Hickson S, Dufour N, Sud A, Kwatra V, Essa IA (2017) Eyemotion: classifying facial expressions in VR using eye-tracking cameras. arXiv

    Google Scholar 

  6. Assari MA, Rahmati M (2011) Driver drowsiness detection using face expression recognition. In: Proceedings of the IEEE international conference on signal and image processing applications, Kuala Lumpur, Malaysia, 16–18 Nov 2011

    Google Scholar 

  7. Chen CH, Lee IJ, Lin LY (2015) Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders. Res Dev Disabil

    Google Scholar 

  8. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I, The extended Cohn-Kanade Dataset (CK+): a complete dataset for action unit and emotion- specified expression. In: Proceedings of the IEEE conference on computer vision and

    Google Scholar 

  9. Kahou SE, Michalski, V, Konda K (2015) Recurrent neural networks for emotion recognition in video. In: Proceedings of the ACM on international conference on multimodal interaction, Seattle, WA, USA, 9–13 Nov 2015

    Google Scholar 

  10. Kim DH, Baddar W, Jang J, Ro YM (2017) Multi-objective based Spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans Affect Comput

    Google Scholar 

  11. Ekman P, Friesen WV (1978) Facial action coding system: investigator’s guide, 1st edn. Consulting Psychologists Press, Palo Alto, CA, USA

    Google Scholar 

  12. Hamm J, Kohler CG, Gur RC, Verma R (2011) Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders. J Neurosci Methods

    Google Scholar 

  13. Jeong M, Kwak SY, Ko BC, Nam JY (2017) Driver facial landmark detection in real driving situation. IEEE Trans Circ Syst Video Technol

    Google Scholar 

  14. Tao SY, Martinez AM (2014) Compound facial expressions of emotion. Natl Acad Sci

    Google Scholar 

  15. Benitez-Quiroz CF, Srinivasan R, Martinez AM (2016) EmotioNet: an accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 26 June–1 July 2016

    Google Scholar 

  16. Kolakowaska A (2013) A review of emotion recognition methods based on keystroke dynamics and mouse movements. In: Proceedings of the 6th international conference on human system interaction, Gdansk, Poland, 6–8 June 2013

    Google Scholar 

  17. Kumar S (2015) Facial expression recognition: a review. In: Proceedings of the national conference on cloud computing and big data, Shanghai, China, 4–6 Nov 2015

    Google Scholar 

  18. Ghayoumi MA (2017) Quick review of deep learning in facial expression. J Commun Comput

    Google Scholar 

  19. Suk M, Prabhakaran B (2014) Real-time mobile facial expression recognition system—a case study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, Columbus, OH, USA, 24–27 June 2014

    Google Scholar 

  20. Ghimire D, Lee J (2013) Geometric feature- based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors

    Google Scholar 

  21. Happy SL, George A, Routray A (2012) A real time facial expression classification system using local binary patterns. In: Proceedings of the 4th international conference on intelligent human computer interaction, Kharagpur, India, 27–29 Dec 2012

    Google Scholar 

  22. Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl

    Google Scholar 

  23. Siddiqi MH, Ali R, Khan AM, Park YT, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEETrans Image Proc

    Google Scholar 

  24. Khan RA, Meyer A, Konik H, Bouakaz S (2013) Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recognit Lett

    Google Scholar 

  25. Torre FD, Chu WS, Xiong X, Vicente F, Ding X, Cohn J (2015) IntraFace. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, Ljubljana, Slovenia, 4–8 May 2015

    Google Scholar 

  26. Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: Proceedings of the 3rd international conference on crime detection and prevention, London, UK, 3 Dec 2009

    Google Scholar 

  27. Mavadati SM, Mahoor MH, Bartlett K, Trinh P, Cohn J (2013) DISFA: a spontaneous facial action intensity database. IEEE Trans Affect Comput

    Google Scholar 

  28. Szwoch M, Pienia˛z˙ek P (2015) Facial emotion recognition using depth data. In: Proceedings of the 8th international conference on human system interactions, Warsaw, Poland, 25–27 June 2015

    Google Scholar 

  29. Maalej A, Amor BB, Daoudi M, Srivastava A, Berretti S (2011) Shape analysis of local facial patches for 3D facial expression recognition. Pattern Recognit

    Google Scholar 

  30. Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In: Proceedings of the international conference on automatic face and gesture recognition, Southampton, UK, 10–12 April 2006

    Google Scholar 

  31. Zhao G, Huang X, Taini M, Li SZ, Pietikäinen M (2011) Facial expression recognition from near-infrared videos. Image Vis Comput

    Google Scholar 

  32. Lyons MJ, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with Gabor wave. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, Nara, Japan, 14– 16 Apr 1998

    Google Scholar 

  33. B+. Available online: https://computervisiononline.com/dataset/1105138686 29 Nov 2017

  34. MI. Available, online: https://mmifacedb.eu/29 Nov 2017

  35. Walecki R, Rudovic O, Pavlovic V, Schuller B, Pantic M (2017) Deep structured learning for facial action unit intensity estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA, 21–26 July 2017

    Google Scholar 

  36. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput

    Google Scholar 

  37. Ko BC, Lee EJ, Nam JY (2016) Genetic algorithm based filter bank design for light convolutional neural network. Adv Sci Lett

    Google Scholar 

  38. Breuer R, Kimmel R (2017) A deep learning perspective on the origin of facial expressions

    Google Scholar 

  39. Jung H, Lee S, Yim J, Park S, Kim J (2015) Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE international conference on computer vision, Santiago, Chile, 7–12 Dec 2015

    Google Scholar 

  40. Zhao K, Chu WS, Zhang H (2016) Deep region and multi-label learning for facial action unit detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 26 June–1 July 2016

    Google Scholar 

  41. Shen P, Wang S, Liu Z (2013) Facial expression recognition from infrared thermal videos. Intell Auton Syst

    Google Scholar 

  42. Wei W, Jia Q, Chen G (2016) Real-time facial expression recognition for affective computing based on Kinect. In: Proceedings of the IEEE 11th conference on industrial electronics and applications, Hefei, China, 5–7 June 2016

    Google Scholar 

  43. Tian Y, Luo P, Luo X, Wang X, Tang X (2015) Pedestrian detection aided by deep learning semantic tasks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA, 8–10 June 2015

    Google Scholar 

  44. Deshmukh S, Patwardhan M, Mahajan A (2016) Survey on real-time facial expression recognition techniques. IET Biom

    Google Scholar 

  45. Dornaika F, Raducanu B (2007) Efficient facial expression recognition for human robot interaction. In: Proceedings of the 9th international work-conference on artificial neural networks on computational and ambient intelligence, San Sebastián, Spain, 20–22 June

    Google Scholar 

  46. Bartneck C, Lyons MJ (2007) HCI and the face: towards an art of the soluble. In: Proceedings of the international conference on human- computer interaction: interaction design and usability, Beijing, China, 22–27 July 2007

    Google Scholar 

  47. Zhan C, Li W, Ogunbona P, Safaei F (2008) A real-time facial expression recognition system for online games. Int J Comput Games Technol

    Google Scholar 

  48. Mourão A, Magalhães J (2013) Competitive affective gaming: winning with a smile. In: Proceedings of the ACM international conference on multimedia, Barcelona, Spain, 21–25 Oct 2013

    Google Scholar 

  49. Sandbach G, Zafeiriou S, Pantic M, Yin L (2012) Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis Comput

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Communication Engineering, ACE College of Engineering and Management, Agray, India

    Sanyam Agarwal

  2. Departmen of Computer Science Engineering, Thapar Institute of Engineering and Technology, Patialay, India

    Veer Daksh Agarwal & Lakshay Singla

  3. Department of Computer Science Engineering, SRM Institute of Science and Technology, Kattankulathur, Chennai, India

    Ishaan Agarwal

  4. Department of Electronics and Communication Engineering, IIMT University, Meeut, India

    Vipin Mittal

  5. Scientific Research Centre of the Islamic University, The Islamic University, Najaf, Iraq

    Ahmed Hussein Alkhayyat

Authors
  1. Sanyam Agarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Veer Daksh Agarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ishaan Agarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vipin Mittal
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lakshay Singla
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ahmed Hussein Alkhayyat
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Veer Daksh Agarwal .

Editor information

Editors and Affiliations

  1. Department of Electronics and Communication Engineering, ABES Engineering College, Ghaziabad, Uttar Pradesh, India

    Devendra Kumar Sharma

  2. National Taipei University of Business, Taoyuan, Taiwan

    Sheng-Lung Peng

  3. Department of Electronics and Communication Engineering, ABES Engineering College, Ghaziabad, Uttar Pradesh, India

    Rohit Sharma

  4. Department of Embedded Systems Engineering, Incheon National University, Incheon, Korea (Republic of)

    Gwanggil Jeon

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Agarwal, S., Agarwal, V.D., Agarwal, I., Mittal, V., Singla, L., Alkhayyat, A.H. (2024). A Review of Survey and Assessment of Facial Emotion Recognition (FER) by Convolutional Neural Networks. In: Sharma, D.K., Peng, SL., Sharma, R., Jeon, G. (eds) Micro-Electronics and Telecommunication Engineering. ICMETE 2023. Lecture Notes in Networks and Systems, vol 894. Springer, Singapore. https://doi.org/10.1007/978-981-99-9562-2_63

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-9562-2_63

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-9561-5

  • Online ISBN: 978-981-99-9562-2

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation