Skip to main content
SpringerLink
Log in

FER-YOLO: Detection and Classification Based on Facial Expressions

  • Conference paper
  • First Online:
Image and Graphics (ICIG 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12888))

Included in the following conference series:

Abstract

Due to the wide application prospect and market value of emotion recognition, it has become an important research topic in today’s society. Among them, facial expression recognition (FER) plays an important role in expressing human emotional information. Generally, the FER classification process includes face pre-processing (face detection, alignment, etc.), which adds extra workload. To this end, detection and classification are carried out simultaneously in this paper. We first manually annotated the RAF-DB dataset. We then designed an end-to-end FER network with better performance and applied it to facial expressions called FER-YOLO. FER-YOLO is built on the basis of YOLOv3. We combine the squeeze-and-excitation (SE) module with the backbone network and assign a certain weight to each feature channel so that FER-YOLO can focus on learning prominent facial features. We also discussed the performance changes caused by the lightweight enhanced feature extraction networks. Experimental results show that the proposed FER-YOLO network is 3.03% mAP higher than YOLOv3 on the RAF-DB dataset.

Southwest Jiaotong University.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Poria, S., Cambria, E., Bajpai, R., Hussain, A.: A review of affective computing: from unimodal analysis to multimodal fusion. Inf. Fusion 37, 98–125 (2017)

    Article  Google Scholar 

  2. Spezialetti, M., Placidi, G., Rossi, S.: Emotion recognition for human-robot interaction: recent advances and future perspectives. Front. Robot. AI 7, 145–155 (2020)

    Article  Google Scholar 

  3. Joesph, C., Rajeswari, A., Premalatha, B., Balapriya, C.: Implementation of physiological signal based emotion recognition algorithm. In: IEEE 36th International Conference on Data Engineering (ICDE) 2020, Dallas, TX, USA, pp. 2075–2079 (2020). https://doi.org/10.1109/ICDE48307.2020.9153878

  4. Cosentino, S., Randria, E.I.S., Lin, J.-Y., Pellegrini, T., Sessa, S., Takanishi, A.: Group emotion recognition strategies for entertainment robots. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2018, Madrid, Spain, pp. 813–818 (2018). https://doi.org/10.1109/IROS.2018.8593503

  5. Li, G., Wang, Y.: Research on Leamer’s emotion recognition for intelligent education system. In: IEEE 3rd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Chongqing, China, pp. 754–758 (2018). https://doi.org/10.1109/IAEAC.2018.8577590

  6. Rusli, N., Sidek, S.N., Yusof, H.M., Ishak, N.I., Khalid, M., Dzulkarnain, A.A.A.: Implementation of wavelet analysis on thermal images for affective states recognition of children with autism spectrum disorder. IEEE Access 8, 120818–120834 (2020)

    Article  Google Scholar 

  7. Zou, J., Cao, X., Zhang, S., Ge, B.: A facial expression recognition based on improved convolutional neural network. In: IEEE International Conference of Intelligent Applied Systems on Engineering (ICIASE), Fuzhou, China, pp. 301–304 (2019). https://doi.org/10.1109/ICIASE45644.2019.9074074

  8. Singh, S., Nasoz, F.: Facial expression recognition with convolutional neural networks. In: 10th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA, pp. 0324–0328 (2020). https://doi.org/10.1109/CCWC47524.2020.9031283

  9. Ma, H., Celik, T., Li, H.-C.: Lightweight attention convolutional neural network through network slimming for robust facial expression recognition. Signal Image Video Process. 1863–1711 (2021)

    Google Scholar 

  10. Mohan, K., Seal, A., Krejcar, O., Yazidi, A.: FER-net: facial expression recognition using deep neural net. Neural Comput. Appl. 33(15), 9125–9136 (2021). https://doi.org/10.1007/s00521-020-05676-y

    Article  Google Scholar 

  11. Xie, S., Hu, H.: Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Trans. Multimedia 21(1), 211–220 (2019)

    Article  MathSciNet  Google Scholar 

  12. Ma, H., Celik, T.: FER-Net: facial expression recognition using densely connected convolutional network. Electron. Lett. 55(4), 184–186 (2019)

    Article  Google Scholar 

  13. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 779–788 (2016). https://doi.org/10.1109/CVPR.2016.91

  14. Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, pp. 6517–6525 (2017). https://doi.org/10.1109/CVPR.2017.690

  15. Joseph, R., Ali, F.: YOLOv3: An Incremental Improvement. arXiv:1804.02767 (2018)

  16. Bochkovskiy, A., Wang, C.-Y., Liao, H.-Y.M.: YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv:2004.10934 (2020)

  17. Liu, W., et al.: SSD: single shot MultiBox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2

    Chapter  Google Scholar 

  18. Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. 422, 318–327 (2020). https://doi.org/10.1109/TPAMI.2018.2858826

    Article  Google Scholar 

  19. Hu, J., Shen, L., Albanie, S., Sun, G., Wu, E.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. 428, 2011–2023 (2020)

    Article  Google Scholar 

  20. Mamalet, F., Garcia, C.: Simplifying ConvNets for fast learning. In: International Conference on Artificial Neural Networks (ICANN 2012), Lausanne, Switzerland, pp. 58–65 (2012). https://doi.org/10.1007/978-3-642-33266-1_8

  21. Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, pp. 2584–2593 (2017). https://doi.org/10.1109/CVPR.2017.277

Download references

Acknowledgements

This work was supported by Sichuan Provincial Science and Technology Projects (2019JDJQ0023).

Author information

Authors and Affiliations

  1. Southwest Jiaotong University, Chengdu, 610031, China

    Hui Ma, Turgay Celik & Hengchao Li

  2. University of the Witwatersrand, Johannesburg, 2000, South Africa

    Turgay Celik

Authors
  1. Hui Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Turgay Celik
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hengchao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Peking University, Beijing, China

    Yuxin Peng

  2. Tsinghua University, Beijing, China

    Shi-Min Hu

  3. Tampere University, Tampere, Finland

    Moncef Gabbouj

  4. Zhejiang University, Hangzhou, China

    Kun Zhou

  5. Technion – Israel Institute of Technology, Haifa, Israel

    Michael Elad

  6. Tsinghua University, Beijing, China

    Kun Xu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ma, H., Celik, T., Li, H. (2021). FER-YOLO: Detection and Classification Based on Facial Expressions. In: Peng, Y., Hu, SM., Gabbouj, M., Zhou, K., Elad, M., Xu, K. (eds) Image and Graphics. ICIG 2021. Lecture Notes in Computer Science(), vol 12888. Springer, Cham. https://doi.org/10.1007/978-3-030-87355-4_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87355-4_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87354-7

  • Online ISBN: 978-3-030-87355-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation