Skip to main content
SpringerLink
Log in

To See Facial Expressions Through Occlusions via Adversarial Disentangled Features Learning with 3D Supervision

  • Conference paper
  • First Online:
Biometric Recognition (CCBR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12878))

Included in the following conference series:

  • 1418 Accesses

Abstract

Facial expression recognition (FER) is still a challenging problem if face images are contaminated by occlusions, which lead to not only noisy features but also loss of discriminative features. To address the issue, this paper proposes a novel adversarial disentangled features learning (ADFL) method for recognizing expressions on occluded face images. Unlike previous methods, our method defines an explicit noise component in addition to the identity and expression components to isolate the occlusion-caused noise features. Besides, we learn shape features with joint supervision of 3D shape reconstruction and facial expression recognition to compensate for the occlusion-caused loss of features. Evaluation on both in-the-lab and in-the-wild face images demonstrates that our proposed method effectively improves FER accuracy for occluded images, and can even deal with noise beyond occlusions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Mollahosseini, A., Hasani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)

    Article  Google Scholar 

  2. Li, S., Deng, W.: Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans. Image Proces. 28(1), 356–370 (2018)

    Article  MathSciNet  Google Scholar 

  3. Li, Y., Zeng, J., Shan, S., Chen, X.: Occlusion aware facial expression recognition using CON with attention mechanism. IEEE Trans. Image Process. 28(5), 2439–2450 (2019)

    Article  MathSciNet  Google Scholar 

  4. Pan, B., Wang, S., Xia, B.: Occluded facial expression recognition enhanced through privileged information. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 566–573 (2019)

    Google Scholar 

  5. Lu, Y., Wang, S., Zhao, W., Zhao, Y.: Wgan-based robust occluded facial expression recognition. IEEE Access 7, 93594–93610 (2019)

    Article  Google Scholar 

  6. Bai, M., Xie, W., Shen, L.: Disentangled feature based adversarial learning for facial expression recognition. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 31–35. IEEE (2019)

    Google Scholar 

  7. Halawa, M., Wöllhaf, M., Vellasques, E., SánchezSanz, U., Hellwich, O.: Learning disentangled expression representations from facial images. arXiv preprint arXiv:2008.07001 (2020)

  8. Lekdioui, K., Messoussi, R., Ruichek, Y., Chaabi, Y., Touahni, R.: Facial decomposition for expression recognition using texture/shape descriptors and SVM classifier. Sig. Process. Image Commun. 58, 300–312 (2017)

    Article  Google Scholar 

  9. Li, Jie, Liu, Zhengxi, Zhao, Qijun: Exploring shape deformation in 2D images for facial expression recognition. In: Sun, Zhenan, He, Ran, Feng, Jianjiang, Shan, Shiguang, Guo, Zhenhua (eds.) CCBR 2019. LNCS, vol. 11818, pp. 190–197. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31456-9_21

    Chapter  Google Scholar 

  10. Liu, F., Zhu, R., Zeng, D., Zhao, Q., Liu, X.: Disentangling features in 3D face shapes for joint face reconstruction and recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5216–5225 (2018)

    Google Scholar 

  11. Liu, F., Zhao, Q., Liu, X., Zeng, D.: Joint face alignment and 3d face reconstruction with application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 42(3), 664–678 (2020)

    Google Scholar 

  12. Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: a 3D solution. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  13. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101. IEEE (2010)

    Google Scholar 

  14. Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: 2005 IEEE International Conference on Multimedia and Expo, pp. 5-pp. IEEE (2005)

    Google Scholar 

  15. Zhao, G., Huang, X., Taini, M., othersäInen, M.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29(9), 607–619 (2011)

    Google Scholar 

  16. Amos, B., Ludwiczuk, B., Satyanarayanan, M., et al.: Openface: a general-purpose face recognition library with mobile applications. CMU Sch. Comput. Sci. 6(2) (2016)

    Google Scholar 

  17. Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 296–301. IEEE (2009)

    Google Scholar 

  18. Cao, C., Weng, Y., Zhou, S.: Facewarehouse: a 3D facial expression database for visual computing. IEEE Trans. Visual. Comput. Graph. 20(3), 413–425 (2013)

    Google Scholar 

  19. Zhu, X., Lei, Z., Yan, J., et al.: High-fidelity pose and expression normalization for face recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 787–796 (2015)

    Google Scholar 

  20. Dapogny, A., Bailly, K., Dubuisson, S.: Confidence-weighted local expression predictions for occlusion handling in expression recognition and action unit detection. Int. J. Comput. Vis. 126(2–4), 255–271 (2018)

    Article  MathSciNet  Google Scholar 

  21. Li, Y., Zeng, J., Shan, S., Chen, X.: Patch-gated CNN for occlusion-aware facial expression recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2209–2214. IEEE (2018)

    Google Scholar 

  22. Liu, M., Shan, S., Wang, R., Chen, X.: Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1749–1756 (2014)

    Google Scholar 

  23. Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2983–2991 (2015)

    Google Scholar 

  24. Kacem, A., Daoudi, M., et al.: A novel space-time representation on the positive semidefinite cone for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3180–3189 (2017)

    Google Scholar 

Download references

Acknowledgment

This work is supported by the National Natural Science Foundation of China (61773270).

Author information

Authors and Affiliations

  1. College of Computer Science, Sichuan University, Chengdu, China

    Wenxue Yuan, Qijun Zhao, Feiyu Zhu & Zhengxi Liu

Authors
  1. Wenxue Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qijun Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Feiyu Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhengxi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qijun Zhao .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Jianjiang Feng

  2. Fudan University, Shanghai, China

    Junping Zhang

  3. Shanghai Jiao Tong University, Shanghai, China

    Manhua Liu

  4. Shanghai University, Shanghai, China

    Yuchun Fang

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yuan, W., Zhao, Q., Zhu, F., Liu, Z. (2021). To See Facial Expressions Through Occlusions via Adversarial Disentangled Features Learning with 3D Supervision. In: Feng, J., Zhang, J., Liu, M., Fang, Y. (eds) Biometric Recognition. CCBR 2021. Lecture Notes in Computer Science(), vol 12878. Springer, Cham. https://doi.org/10.1007/978-3-030-86608-2_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86608-2_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86607-5

  • Online ISBN: 978-3-030-86608-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation