Skip to main content
SpringerLink
Log in

Human emotion recognition based on facial expressions via deep learning on high-resolution images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Detecting human emotion based on facial expression is considered a hard task for the computer vision community because of many challenges such as the difference of face shape from a person to another, difficulty of recognition of dynamic facial features, low quality of digital images, etc. In this paper, we propose a face-sensitive convolutional neural network (FS-CNN) for human emotion recognition. The proposed FS-CNN is used to detect faces on large scale images then analyzing face landmarks to predict expressions for emotion recognition. The FS-CNN is composed form two stages, patch cropping, and convolutional neural networks. The first stage is used to detect faces in high-resolution images and crop the face for further processing. The second stage is a convolutional neural network used to predict facial expression based on landmarks analytics, it was applied on pyramid images to process scale invariance. The proposed FS-CNN was trained and evaluated on the UMD Faces dataset. High performance was achieved with a mean average precision of about 95%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Afif M, Ayachi R, Said Y, Pissaloux E, Atri M (2018) Indoor image recognition and classification via deep convolutional neural network. In: International conference on the Sciences of Electronics, Technologies of Information and Telecommunications, pp. 364–371. Cham: Springer

  2. Afif M, Ayachi R, Said Y, Pissaloux E, Atri M (2020) An evaluation of RetinaNet on indoor object detection for blind and visually impaired persons assistance navigation. Neural Process Lett:1–15

  3. Arshad H, Khan MA, Sharif MI, Yasmin M, Tavares JMRS, Zhang Y-D, Satapathy SC (2020) A multilevel paradigm for deep convolutional neural network features selection with an application to human gait recognition. Exp Syst e12541

  4. Ayachi R, Afif M, Said Y, Atri M (2018) Strided convolution instead of max pooling for memory efficiency of convolutional neural networks. In: International conference on the Sciences of Electronics, Technologies of Information and Telecommunications (pp. 234–243). Cham: Springer

  5. Ayachi R, Afif M, Said Y, Atri M (2019) Traffic signs detection for real-world application of an advanced driving assisting system using deep learning. Neural Process Lett:1–15

  6. Ayachi R, Said v, Atri M (n.d.) To perform road signs recognition for autonomous vehicles using cascaded deep learning pipeline. Artif Intell Adv

  7. Baber J, Bakhtyar M, Ahmed KU, Noor W, Devi V, Sammad A (2019) Facial expression recognition and analysis of interclass false positives using CNN. In: Future of Information and Communication Conference (pp. 46–54). Cham: Springer

  8. Bansal A, Nanduri A, Castillo CD, Ranjan R, Chellappa R (2017) Umdfaces: An annotated face dataset for training deep networks. In: 2017 IEEE International Joint Conference on Biometrics (IJCB), pp. 464–473. IEEE

  9. Bargal SA, Barsoum E, Ferrer CC, Zhang C (2016) Emotion recognition in the wild from videos using images. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction (pp. 433–436)

  10. Bhowmik MK, Saha K, Majumder S, Majumder G, Saha A, Sarma AN, Bhattacharjee D, Basu DK, Nasipuri M (2011) Thermal infrared face recognition—a biometric identification technique for robust security system. Reviews refinements and new ideas in face recognition 7

  11. Bodla N, Singh B, Chellappa R, Davis LS (2017) Soft-NMS--improving object detection with one line of code. In: Proceedings of the IEEE international conference on computer vision (pp. 55615569)

  12. Dandıl E, Özdemir R (2019) Real-time facial emotion classification using deep learning. Data Sci Appl 2(1):13–17

    Google Scholar 

  13. Dhall A, Goecke R, Lucey S, Gedeon T (2012) Collecting large, richly annotated facial-expression databases from movies

  14. Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision (pp. 1440-1448)

  15. Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W et al. (2013) Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing (pp. 117–124). Berlin: Springer

  16. He K, Zhang X, Ren S, Sun J (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37(9):1904–1916

    Article  Google Scholar 

  17. Hsu R-L, Abdel-Mottaleb M, Jain AK (2002) Face detection in color images. IEEE Trans Pattern Anal Mach Intell 24(5):696–706

    Article  Google Scholar 

  18. Jaiswal S, Nandi GC (2019) Robust real-time emotion detection system using CNN architecture. Neural Comput Appl 1–10

  19. Jiao L, Zhang F, Liu F, Yang S, Li L, Feng Z, Qu R (2019) A survey of deep learning-based object detection. IEEE Access 7:128837–128868

    Article  Google Scholar 

  20. Jose E, Greeshma M, Mithun Haridas TP, Supriya MH (2019) Face recognition based surveillance system using facenet and mtcnn on jetson tx2. In: 2019 5th International Conference on Advanced Computing & Communication Systems (ICACCS), pp. 608–613. IEEE

  21. Jumani SZ, Ali F, Guriro S, Kandhro IA, Khan A, Zaidi A (2019) Facial expression recognition with histogram of oriented gradients using CNN. Indian J Sci Technol 12:24

    Google Scholar 

  22. Kavitha SN, Shahila K, Kumar SCP (2018) Biometrics Secured Voting System with Finger Print, Face and Iris Verification. In: 2018 Second International Conference on Computing Methodologies and Communication (ICCMC), pp. 743–746. IEEE

  23. Khan MA, Javed K, Khan SA, Saba T, Habib U, Khan JA, Abbasi AA (2020) Human action recognition using fusion of multiview and deep features: an application to video surveillance. Multimedia Tools Appl 1–27

  24. Khan MA, Zhang Y-D, Khan SA, Attique M, Rehman A, Seo S (2020) A resource conscious human action recognition framework using 26-layered deep convolutional neural network. Multimedia Tools Appl 1–23

  25. Kollias D, Zafeiriou S (2019) Exploiting multi-cnn features in cnn-rnn based dimensional emotion recognition on the omg in-the-wild dataset. arXiv preprint arXiv:1910.01417

  26. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems (pp. 1097–1105)

  27. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC 2016 Ssd: Single shot multibox detector. In: European conference on computer vision (pp. 21–37). Cham: Springer

  28. Liu Z, Luo P, Wang X, Tang X (2018) Large-scale celebfaces attributes (celeba) dataset. Retrieved August 15: 2018

  29. Mehendale N (2020) Facial emotion recognition using convolutional neural networks (FERC). SN Appl Sci 2(3):1–8

    Article  Google Scholar 

  30. Mehmood A, Khan MA, Sharif M, Khan SA, Shaheen M, Saba T, Riaz N, Ashraf I (2020) Prosperous human gait recognition: an end-to-end system based on pre-trained CNN features selection. Multimedia Tools Appl

  31. Ranjan R, Patel VM, Chellappa R (2017) Hyperface: a deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans Pattern Anal Mach Intell 41(1):121–135

    Article  Google Scholar 

  32. Rashid M, Khan MA, Alhaisoni M, Wang S-H, Naqvi SR, Rehman A, Saba T (2020) A sustainable deep learning framework for object recognition using multi-layers deep features fusion and selection. Sustainability 12(12):5037

    Article  Google Scholar 

  33. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems (pp. 91-99)

  34. UMD Faces Dataset (n.d.) available at : http://umdfaces.io/

  35. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  36. Wu W, Qian C, Yang S, Wang Q, Cai Y, Zhou Q (2018) Look at boundary: A boundary-aware face alignment algorithm. In: 2018 Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2129–2138. IEEE

  37. Yitzhak N, Gurevich T, Inbar N, Lecker M, Atias D, Avramovich H, Aviezer H (2020) Recognition of emotion from subtle and non-stereotypical dynamic facial expressions in Huntington's disease. Cortex 126:343–354

    Article  Google Scholar 

Download references

Acknowledgements

The authors wish to acknowledge the approval and the support of this research study by the grant N° ENG-2019-1-10-F-1113 from the Deanship of the Scientific Research in Northern Border University, Arar, KSA.

Author information

Authors and Affiliations

  1. Electrical Engineering Department, College of Engineering, Northern Border University, Arar, Saudi Arabia

    Yahia Said & Mohammad Barr

  2. Laboratory of Electronics and Microelectronics (LR99ES30), Faculty of Sciences of Monastir, University of Monastir, Monastir, Tunisia

    Yahia Said

Authors
  1. Yahia Said
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Barr
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yahia Said.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Said, Y., Barr, M. Human emotion recognition based on facial expressions via deep learning on high-resolution images. Multimed Tools Appl 80, 25241–25253 (2021). https://doi.org/10.1007/s11042-021-10918-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10918-9

Keywords

Search

Navigation