Skip to main content
SpringerLink
Log in

Towards Facial Expression Robustness in Multi-scale Wild Environments

  • Conference paper
  • First Online:
Image Analysis and Processing – ICIAP 2023 (ICIAP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14233))

Included in the following conference series:

  • 485 Accesses

Abstract

Facial expressions are dynamic processes that evolve over temporal segments, including onset, apex, offset, and neutral. However, previous works on automatic facial expression analysis have mainly focused on the recognition of discrete emotions, neglecting the continuous nature of these processes. Additionally, facial images captured from videos in the wild often have varying resolutions due to fixed-lens cameras. To address these problems, our objective is to develop a robust facial expression recognition classifier that provides good performance in such challenging environments. We evaluated several state-of-the-art models on labeled and unlabeled collections and analyzed their performance at different scales. To improve performance, we filtered the probabilities provided by each classifier and demonstrated that this improves decision-making consistency by more than 10%, leading to accuracy improvement. Finally, we combined the models’ backbones into a temporal-sequence classifier, leveraging this consistency-performance trade-off and achieving an additional improvement of 9.6%.

This work is partially funded by the the Spanish Ministry of Science and Innovation under project PID2021-122402OB-C22, and by the ACIISI-Gobierno de Canarias and European FEDER funds under project, ProID2021010012, ULPGC Facilities Net, and Grant EIS 2021 04.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Akbar, M.T., Ilmi, M.N., Rumayar, I.V., Moniaga, J., Chen, T.K., Chowanda, A.: Enhancing game experience with facial expression recognition as dynamic balancing. Proc. Comput. Sci. 157, 388–395 (2019)

    Article  Google Scholar 

  2. Aminbeidokhti, M., Pedersoli, M., Cardinal, P., Granger, E.: Emotion recognition with spatial attention and temporal softmax pooling. In: Karray, F., Campilho, A., Yu, A. (eds.) Image Analysis and Recognition, pp. 323–331 (2019)

    Google Scholar 

  3. Baddar, W.J., Ro, Y.M.: Mode variational LSTM robust to unseen modes of variation: application to facial expression recognition. In: AAAI Conference on Artificial Intelligence. vol. 33, pp. 3215–3223 (2019)

    Google Scholar 

  4. Barrett, L.F., Adolphs, R., Marsella, S., Martinez, A.M., Pollak, S.D.: Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychol. Sci. Public Interest 20(1), 1–68 (2019)

    Article  Google Scholar 

  5. Bell, S., Bala, K.: Learning visual similarity for product design with convolutional neural networks. ACM Trans. Graph. 34(4), 98:1–98:10 (2015)

    Google Scholar 

  6. Deng, J., Guo, J., Zhou, Y., Yu, J., Kotsia, I., Zafeiriou, S.: RetinaFace: Single-stage dense face localisation in the wild. CoRR abs/1905.00641 (2019)

    Google Scholar 

  7. Dhall, A.: EmotiW 2019: automatic emotion, engagement and cohesion prediction tasks. In: 2019 International Conference on Multimodal Interaction, pp. 546–550 (2019)

    Google Scholar 

  8. Dhall, A., Asthana, A., Goecke, R., Gedeon, T.: Emotion recognition using PHOG and LPQ features. In: 2011 IEEE International Conference on Automatic Face Gesture Recognition, pp. 878–883 (2011)

    Google Scholar 

  9. Ekman, P., Friesen, W.: Unmasking the Face: A Guide to Recognizing Emotions from Facial Expressions. Prentice Hall, Hoboken (1975)

    Google Scholar 

  10. Fan, Y., Lam, J.C.K., Li, V.O.K.: Video-based emotion recognition using deeply-supervised neural networks. In: 20th ACM International Conference on Multimodal Interaction, pp. 584–588 (2018)

    Google Scholar 

  11. Feng, D., Ren, F.: Dynamic facial expression recognition based on two-stream-CNN with LBP-TOP. In: 2018 5th IEEE International Conference on Cloud Computing and Intelligence Systems, pp. 355–359 (2018)

    Google Scholar 

  12. Freire-Obregón, D., Castrillón-Santana, M.: An evolutive approach for smile recognition in video sequences. Int. J. Pattern Recogn. Artif. Intell. 29, 1550006 (2015)

    Google Scholar 

  13. Freire-Obregón, D., Castrillón-Santana, M., Barra, P., Bisogni, C., Nappi, M.: An attention recurrent model for human cooperation detection. Comput. Vis. Image Underst. 197–198, 102991 (2020)

    Article  Google Scholar 

  14. Freire-Obregón, D., Rosales-Santana, K., Marín-Reyes, P.A., Penate-Sanchez, A., Lorenzo-Navarro, J., Castrillón-Santana, M.: Improving user verification in human-robot interaction from audio or image inputs through sample quality assessment. Pattern Recogn. Lett. 149, 179–184 (2021)

    Article  Google Scholar 

  15. Happy, S.L., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1–12 (2015)

    Article  Google Scholar 

  16. Howard, A.G., et al.: MobileNets: Efficient convolutional neural networks for mobile vision applications. CoRR abs/1704.04861 (2017)

    Google Scholar 

  17. Hu, P., Cai, D., Wang, S., Yao, A., Chen, Y.: Learning supervised scoring ensemble for emotion recognition in the wild. In: 19th ACM International Conference on Multimodal Interaction, pp. 553–560 (2017)

    Google Scholar 

  18. Jain, V., Learned-Miller., E.: FDDB: A benchmark for face detection in unconstrained settings. Tech. rep., University of Massachusetts, Amherst (2010)

    Google Scholar 

  19. Kumar, V., Rao, S., Yu, L.: Noisy student training using body language dataset improves facial expression recognition. In: Computer Vision - ECCV 2020 Workshops, pp. 756–773 (2020)

    Google Scholar 

  20. Liu, C., Tang, T., Lv, K., Wang, M.: Multi-feature based emotion recognition for video clips. In: 20th ACM International Conference on Multimodal Interaction, pp. 630–634 (2018)

    Google Scholar 

  21. Luan, P., Huynh, V., Tuan Anh, T.: Facial expression recognition using residual masking network. In: IEEE 25th International Conference on Pattern Recognition, pp. 4513–4519 (2020)

    Google Scholar 

  22. Marras, M., Marín-Reyes, P., Lorenzo-Navarro, J., Castrillón-Santana, M., Fenu, G.: AveROBOT: an audio-visual dataset for people re-identification and verification in human-robot interaction. In: Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods, pp. 255–265 (2019)

    Google Scholar 

  23. Meng, D., Peng, X., Wang, K., Qiao, Y.: Frame attention networks for facial expression recognition in videos. In: 2019 IEEE International Conference on Image Processing, pp. 3866–3870 (2019)

    Google Scholar 

  24. Pantic, M., Patras, I.: Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Syst. Man Cybern. 36(2), 433–449 (2006)

    Google Scholar 

  25. Penate-Sanchez, A., Freire-Obregón, D., Lorenzo-Melián, A., Lorenzo-Navarro, J., Castrillón-Santana, M.: TGC20ReId: a dataset for sport event re-identification in the wild. Pattern Recog. Lett. 138, 355–361 (2020)

    Article  Google Scholar 

  26. Saeed, A., Al-Hamadi, A., Niese, R., Elzobi, M.: Effective geometric features for human emotion recognition. In: 2012 IEEE 11th International Conference on Signal Processing. vol. 1, pp. 623–627 (2012)

    Google Scholar 

  27. Savchenko, A.V.: Facial expression and attributes recognition based on multi-task learning of lightweight neural networks. In: 2021 IEEE 19th International Symposium on Intelligent Systems and Informatics, pp. 119–124 (2021)

    Google Scholar 

  28. Tan, M., Le, Q.V.: EfficientNet: Rethinking model scaling for convolutional neural networks. CoRR abs/1905.11946 (2019)

    Google Scholar 

  29. Uchida, M.C., et al.: Identification of muscle fatigue by tracking facial expressions. PLoS ONE 13(12), e0208834 (2018)

    Google Scholar 

  30. Vyas, A.S., Prajapati, H.B., Dabhi, V.K.: Survey on face expression recognition using CNN. In: 2019 5th International Conference on Advanced Computing Communication Systems (ICACCS), pp. 102–106 (2019)

    Google Scholar 

  31. Yang, S., Luo, P., Loy, C.C., Tang, X.: WIDER FACE: a face detection benchmark. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5525–5533. IEEE, Hawai, USA (2016)

    Google Scholar 

  32. Yao, A., Cai, D., Ping Hu, S.W., Sha, L., Chen, Y.: HoloNet: towards robust emotion recognition in the wild. In: 18th ACM International Conference on Multimodal Interaction, pp. 472–478 (2016)

    Google Scholar 

  33. Zhang, F., Zhang, T., Mao, Q., Xu, C.: Geometry guided pose-invariant facial expression recognition. IEEE Trans. Image Process. 29, 4445–4460 (2020)

    Article  MATH  Google Scholar 

  34. Zhang, N., Luo, J., Gao, W.: Research on face detection technology based on MTCNN. In: 2020 International Conference on Computer Network, Electronic and Automation, pp. 154–158 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. SIANI, Universidad de Las Palmas de Gran Canaria, Las Palmas de Gran Canaria, Spain

    David Freire-Obregón, Daniel Hernández-Sosa, Oliverio J. Santana, Javier Lorenzo-Navarro & Modesto Castrillón-Santana

Authors
  1. David Freire-Obregón
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Hernández-Sosa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Oliverio J. Santana
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Javier Lorenzo-Navarro
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Modesto Castrillón-Santana
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Freire-Obregón .

Editor information

Editors and Affiliations

  1. University of Udine, Udine, Italy

    Gian Luca Foresti

  2. University of Udine, Udine, Italy

    Andrea Fusiello

  3. University of York, York, UK

    Edwin Hancock

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Freire-Obregón, D., Hernández-Sosa, D., Santana, O.J., Lorenzo-Navarro, J., Castrillón-Santana, M. (2023). Towards Facial Expression Robustness in Multi-scale Wild Environments. In: Foresti, G.L., Fusiello, A., Hancock, E. (eds) Image Analysis and Processing – ICIAP 2023. ICIAP 2023. Lecture Notes in Computer Science, vol 14233. Springer, Cham. https://doi.org/10.1007/978-3-031-43148-7_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43148-7_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43147-0

  • Online ISBN: 978-3-031-43148-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation