Skip to main content
SpringerLink
Log in

A survey on analysis of human faces and facial expressions datasets

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Facial expressions are the basic input for visual emotion detection. They are of great importance in computer vision society. In last decade, substantial amount of work has been done in the field of facial expressions datasets. This survey covers all of the publically available databases in detail and provides necessary information about these sets. This review delivers comprehensive support to researchers in selection of their desired dataset. The datasets are organized in decreasing order of their importance with respect to diversity in expressions, poses, number of images and resolution. This survey also provides comprehensive tabular comparison of different face based databases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27

Similar content being viewed by others

References

  1. Cabanac M (2002) What is emotion? Behav Process 60(2):69–83

    Google Scholar 

  2. Adolphs R, Andler D (2018) Investigating emotions as functional states distinct from feelings. Emot Rev 10(3):191–201

    Google Scholar 

  3. Ekman P (1992) An argument for basic emotions. Cognit Emot 6(3–4):169–200

    Google Scholar 

  4. Gunes H, Celiktutan O, Sariyanidi E (2019) Live human-robot interactive public demonstrations with automatic emotion and personality prediction. Philos Trans R Soc B 374(1771):20180026

    Google Scholar 

  5. Scott B, Clifford N (2003) Emotion in human–computer interaction. Human–computer interaction. CRC, Boca Raton, p 5

    Google Scholar 

  6. Donato G, Bartlett MS, Hager JC, Ekman P, Sejnowski TJ (1999) Classifying facial actions. IEEE Trans Pattern Anal Mach Intell 21(10):974–989

    Google Scholar 

  7. Sumathi CP, Santhanam T, Mahadevi M (2012) Automatic facial expression analysis a survey. Int J Comput Sci Eng Surv 3(6):47

    Google Scholar 

  8. Ioannou SV, Raouzaiou AT, Tzouvaras VA, Mailis TP, Karpouzis KC, Kollias SD (2005) Emotion recognition through facial expression analysis based on a neurofuzzy network. Neural Netw 18(4):423–435

    Google Scholar 

  9. Khan G, Siddiqi A, Ghani U, Waqar S et al (2019) Geometric positions and optical flow based emotion detection using mlp and reduced dimensions. IET Image Process 13:634–643

    Google Scholar 

  10. Jain N, Kumar S, Kumar A, Shamsolmoali P, Zareapoor M (2018) Hybrid deep neural networks for face emotion recognition. Pattern Recognit Lett 115:101–106

    Google Scholar 

  11. Fan Y, Lu X, Li D, Liu Y (2016) Video-based emotion recognition using cnn-rnn and c3d hybrid networks. In: Proceedings of the 18th ACM international conference on multimodal interaction. ACM, pp 445–450

  12. Dhanashree T, Mohole GP (2015) Application opening based on emotion using hog features. Int J Adv Res Comput Commun Eng 4

  13. Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Google Scholar 

  14. Ko B (2018) A brief review of facial emotion recognition based on visual information. Sensors 18(2):401

    Google Scholar 

  15. Bartlett Marian Stewart, Littlewort Gwen, Fasel Ian, Movellan Javier R (2003) Real time face detection and facial expression recognition: development and applications to human computer interaction. In: Computer vision and pattern recognition workshop, 2003. CVPRW’03. Conference on, vol 5, pp 53–53. IEEE

  16. Pumarola A, Agudo A, Martinez AM, Sanfeliu A, Moreno-Noguer F (2018) Ganimation: Anatomically-aware facial animation from a single image. In: Proceedings of the European conference on computer vision (ECCV), pp 818–833

    Google Scholar 

  17. Heisele B, Pontil M et al (2000) Face detection in still gray images. Technical report, DTIC Document

  18. Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3d facial expression database for facial behavior research. In: Automatic face and gesture recognition, 2006. FGR 2006. 7th international conference on. IEEE, pp 211–216

  19. Savran A, Alyüz N, Dibeklioğlu H, Çeliktutan O, Gökberk B, Sankur B, Akarun L (2008) Bosphorus database for 3d face analysis. In: European workshop on biometrics and identity management. Springer, pp 47–56

  20. Huang GB, Ramesh M, Berg T, Learned-Miller E (2007) Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical report, Technical Report 07-49, University of Massachusetts, Amherst

  21. Zhou Z-H, Geng X (2004) Projection functions for eye detection. Pattern Recognit 37(5):1049–1056

    MATH  Google Scholar 

  22. Lenc L, Král P (2015) Unconstrained facial images: database for face recognition under real-world conditions. In: Mexican international conference on artificial intelligence. Springer, pp 349–361

  23. Hancock P (2008) Psychological image collection at stirling (pics). http://pics.psych.stir.ac.uk

  24. Lundqvist D, Flykt A, Öhman A (1998) The karolinska directed emotional faces (kdef). CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, pp 91–630

  25. Bainbridge WA, Isola P, Oliva A (2013) The intrinsic memorability of face photographs. J Exp Psychol Gen 142(4):1323

    Google Scholar 

  26. Mollahosseini A, Hasani B, Mahoor MH (2017) Affectnet: a database for facial expression, valence, and arousal computing in the wild. arXiv preprint arXiv:1708.03985

  27. Zafeiriou S, Kollias D, Nicolaou MA, Papaioannou A, Zhao G, Kotsia I (2017) Aff-wild: valence and arousal in-the-wild challenge. In: IEEE CVPR workshop

  28. Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Google Scholar 

  29. Martinez AM (1998) The ar face database. CVC technical report, 24

  30. Ekman P, Friesen WV (1977) Facial action coding system

  31. Bettadapura V (2012) Face expression recognition and analysis: the state of the art. arXiv preprint arXiv:1203.6722

  32. Wallraven C, Cunningham DW, Breidt M, Bülthoff HH (2004) View dependence of complex versus simple facial motions. In: Proceedings of the 1st symposium on applied perception in graphics and visualization. ACM, pp 181–181

  33. Kleiner M, Wallraven C, Bülthoff HH (2004) The mpi videolabva system for high quality synchronous recording of video and audio from multiple viewpoints (tech. rep. no. 123). Tübingen, Germany: Max Planck Institute for Biological Cybernetics

  34. Zhao G, Huang X, Taini M, Li SZ, PietikäInen M (2011) Facial expression recognition from near-infrared videos. Image Vis Comput 29(9):607–619

    Google Scholar 

  35. Mohammad Mavadati S, Mahoor MH, Bartlett K, Trinh P, Cohn JF (2013) Disfa: a spontaneous facial action intensity database. IEEE Trans Affect Comput 4(2):151–160

    Google Scholar 

  36. Harmon-Jones E (2003) Clarifying the emotive functions of asymmetrical frontal cortical activity. Psychophysiology 40(6):838–848

    Google Scholar 

  37. Cornelius RR (1996) The science of emotion: research and tradition in the psychology of emotions. Prentice-Hall Inc, Upper Saddle River

    Google Scholar 

  38. Krause R (1987) Universals and cultural differences in the judgments of facial expressions of emotion. J Personal Soc Psychol 5(3):4–712

    MathSciNet  Google Scholar 

  39. Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) Deap: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31

    Google Scholar 

  40. Soleymani M, Lichtenauer J, Pun T, Pantic M (2012) A multimodal database for affect recognition and implicit tagging. IEEE Trans Affect Comput 3(1):42–55

    Google Scholar 

  41. Gunes H, Schuller B (2013) Categorical and dimensional affect analysis in continuous input: current trends and future directions. Image Vis Comput 31(2):120–136

    Google Scholar 

  42. Benitez-Quiroz CF, Srinivasan R, Martinez AM et al (2016) Emotionet: an accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: CVPR, pp 5562–5570

  43. Setty S, Husain M, Beham P, Gudavalli J, Kandasamy M, Vaddi R, Hemadri V, Karure JC, Raju R, Rajan B et al (2013) Indian movie face database: a benchmark for face recognition under wide variations. In: Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG), 2013 Fourth National Conference on. IEEE, pp 1–5

  44. Shankar S, Udupi VR (2014) A study of face databases used as benchmarks in face recognition. Int J Innov Sci Res 10:83–89

    Google Scholar 

  45. Kumar V, Namboodiri AM, Jawahar CV (2014) Face recognition in videos by label propagation. In: Pattern recognition (ICPR), 2014 22nd international conference on. IEEE, pp 303–308

  46. Aifanti N, Papachristou C, Delopoulos A(2010) The mug facial expression database. In: Image analysis for multimedia interactive services (WIAMIS), 2010 11th international workshop on. IEEE, pp 1–4

  47. Rahulamathavan Yogachandran, Phan Raphael CW, Chambers Jonathon A, Parish David J (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4(1):83–92

    Google Scholar 

  48. Sultan Zia M, Jaffar MA (2015) An adaptive training based on classification system for patterns in facial expressions using surf descriptor templates. Multimedia Tools Appl 74(11):3881–3899

    Google Scholar 

  49. Ahmad F, Najam A, Ahmed Z (2013) Image-based face detection and recognition: “state of the art”. arXiv preprint arXiv:1302.6379

  50. Hassanat ABA, Alkasassbeh M, Al-awadi M, Esra’a AA (2015) Colour-based lips segmentation method using artificial neural networks. In: Information and communication systems (ICICS), 2015 6th international conference on. IEEE, pp 188–193

  51. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: Computer vision and pattern recognition workshops (CVPRW), 2010 IEEE computer society conference on. IEEE, pp 94–101

  52. Dhall A et al (2012) Collecting large, richly annotated facial-expression databases from movies. IEEE Multimedia 19:34–41

    Google Scholar 

  53. Rivera AR, Castillo JR, Chae OO (2013) Local directional number pattern for face analysis: face and expression recognition. IEEE Ttrans Image Process 22(5):1740–1752

    MathSciNet  MATH  Google Scholar 

  54. Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, Bartlett M (2011) The computer expression recognition toolbox (cert). In: Automatic face & gesture recognition and workshops (FG 2011), 2011 IEEE international conference on. IEEE, pp 298–305

  55. Wolf L, Hassner T, Maoz I (2011) Face recognition in unconstrained videos with matched background similarity. In: Computer vision and pattern recognition (CVPR), 2011 IEEE conference on. IEEE, pp 529–534

  56. Wolf L, Levy N (2013) The svm-minus similarity score for video face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3523–3530

  57. Li H, Hua G, Lin Z, Brandt J, Yang J (2013) Probabilistic elastic matching for pose variant face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3499–3506

  58. Cui Z, Li W, Xu D, Shan S, Chen X (2013) Fusing robust face region descriptors via multiple metric learning for face recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3554–3561

  59. Méndez-Vázquez H, Martínez-Díaz Y, Chai Z (2013) Volume structured ordinal features with background similarity measure for video face recognition. In: Biometrics (ICB), 2013 international conference on. IEEE, pp 1–6

  60. Taigman Y, Yang M, Ranzato M, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1701–1708

  61. Hu J, Lu J, Tan YP (2014) Discriminative deep metric learning for face verification in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1875–1882

  62. Li H, Hua G, Shen X, Lin Z, Brandt J (2014) Eigen-pep for video face recognition. In: Asian conference on computer vision. Springer, pp 17–33

  63. Hu J, Lu J, Yuan J, Tan YP (2014) Large margin multi-metric learning for face and kinship verification in the wild. In: Asian conference on computer vision. Springer, pp 252–267

  64. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Automatic face and gesture recognition, 1998. Proceedings. Third IEEE international conference on. IEEE, pp 200–205

  65. Martínez AM (2003) Recognizing expression variant faces from a single sample image per class. In: Computer vision and pattern recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, vol 1. IEEE, pp I–I

  66. Cheng F, Jiangsheng Y, Xiong H (2010) Facial expression recognition in jaffe dataset based on gaussian process classification. IEEE Trans Neural Netw 21(10):1685–1690

    Google Scholar 

  67. Todorov A (2008) Evaluating faces on trustworthiness. Ann N Y Acad Sci 1124(1):208–224

    Google Scholar 

  68. Hamm AO, Weike AI (2005) The neuropsychology of fear learning and fear regulation. Int J Psychophysiol 57(1):5–14

    Google Scholar 

  69. Crognale S Shabbir Danish Whats your type? personalized prediction of facial attractiveness

  70. Arbabzadah F, Montavon G, Müller KR, Samek W (2016) Identifying individual facial expressions by deconstructing a neural network. In: German conference on pattern recognition. Springer, pp 344–354

  71. Mitchell TM (1999) Cmu face images data set. UCI machine learning repository. https://archive.ics.uci.edu/ml/datasets/CMU+Face+Images

  72. He X, Niyogi P (2003) Locality preserving projections. In: NIPS, vol 16

  73. Meila M, Jordan MI (2000) Learning with mixtures of trees. J Mach Learn Res 1(Oct):1–48

    MathSciNet  MATH  Google Scholar 

  74. Rowley HA, Baluja S, Kanade T (1998) Neural network-based face detection. IEEE Trans Pattern Anal Mach Intell 20(1):23–38

    Google Scholar 

  75. Stekas N, van den Heuvel D (2016) Face recognition using local binary patterns histograms (lbph) on an fpga-based system on chip (soc). In: Parallel and distributed processing symposium workshops, 2016 IEEE international. IEEE, pp 300–304

  76. Forczma P (2013) Recognition of occluded faces based on multi-subspace classification. In: Computer information systems and industrial management. Springer, pp 148–157

  77. Jianxin W, Zhou Z-H (2003) Efficient face candidates selector for face detection. Pattern Recognit 36(5):1175–1186

    Google Scholar 

  78. Spacek L (2008) Computer vision science research projects. Data Base Updat Friday 12:17–59

    Google Scholar 

  79. Karim TF, Lipu Molla SH, Rahman ML, Sultana F (2010) Face recognition using pca-based method. In: Advanced management science (ICAMS), 2010 IEEE international conference on, vol 3. IEEE, pp 158–162

  80. Mefraz KN, Riadh K, Shafiq AI, Boubakeur B (2012) A novel svm+ nda model for classification with an application to face recognition. Pattern Recognit 45(1):66–79

    Google Scholar 

  81. Yi D, Lei Z, Liao S, Li SZ (2014) Learning face representation from scratch. arXiv preprint arXiv:1411.7923

  82. Guillaumin M, Verbeek J, Schmid C (2009) Is that you? metric learning approaches for face identification. In: Computer Vision, 2009 IEEE 12th international conference on. IEEE, pp 498–505

  83. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823

  84. Vu N-S, Dee HM, Caplier A (2012) Face recognition using the poem descriptor. Pattern Recognit 45(7):2478–2488

    Google Scholar 

  85. Ahmad F, Ahmed Z, Najam A (2013) Soft biometric gender classification using face for real time surveillance in cross dataset environment. In: Multi topic conference (INMIC), 2013 16th International. IEEE, pp 131–135

  86. Jesorsky O, Kirchberg KJ, Frischholz RW (2001) Robust face detection using the hausdorff distance. In: International conference on audio-and video-based biometric person authentication. Springer, pp 90–95

  87. Stephen M, John M, Fred N (2010) The muct landmarked face database. Pattern Recognition Association of South Africa, p 201

  88. Lakshmiprabha NS, Bhattacharya J, Majumder S (2011) Face recognition using multimodal biometric features. In: Image information processing (ICIIP), 2011 international conference on. IEEE, pp 1–6

  89. Arigbabu OA, Ahmad SMS, Adnan WAW, Yussof S (2015) Integration of multiple soft biometrics for human identification. Pattern Recognit Lett 68:278–287

    Google Scholar 

  90. Westerlund T (2004) Fast face finding

  91. Nanaa K, Rizon M, Rahman MNA, Almejrad A, Aziz AZA, Mohamed SB (2013) Eye detection using composite cross-correlation. Am J Appl Sci 10(11):1448

    Google Scholar 

  92. Craw I, Cameron P (1992) Face recognition by computer. In: BMVC92. Springer, pp 498–507

  93. Weber M (1999) Caltech frontal face database

  94. James AP, Dimitrijev S (2008) Face recognition using local binary decisions. IEEE Signal Process Lett 15:821–824

    Google Scholar 

  95. Zhang T, Tang YY, Fang B, Shang Z, Liu X (2009) Face recognition under varying illumination using gradient faces. IEEE Trans Image Process 18(11):2599–2606

    MathSciNet  MATH  Google Scholar 

  96. Kaneda T, Bietsch K (2016) 2016 world population data sheet

Download references

Author information

Authors and Affiliations

  1. Al-khawarizmi Institute of Computer Sciences, University of Engineering and Technology, Lahore, Pakistan

    Gulraiz Khan, Sahar Samyan, Muhammad Usman Ghani Khan, Muhammad Shahid & Samyan Qayyum Wahla

Authors
  1. Gulraiz Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sahar Samyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muhammad Usman Ghani Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Muhammad Shahid
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Samyan Qayyum Wahla
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gulraiz Khan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khan, G., Samyan, S., Khan, M.U.G. et al. A survey on analysis of human faces and facial expressions datasets. Int. J. Mach. Learn. & Cyber. 11, 553–571 (2020). https://doi.org/10.1007/s13042-019-00995-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-019-00995-6

Keywords

Search

Navigation