Skip to main content
SpringerLink
Log in

Pyr-HGCN: Pyramid Hybrid Graph Convolutional Network for Gait Emotion Recognition

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14429))

Included in the following conference series:

  • 358 Accesses

Abstract

Gait emotion recognition (GER) plays a crucial role in identifying human emotions. Most previous methods apply Spatial-Temporal Graph Convolutional Networks (ST-GCN) to recognize emotions. However, these methods suffer from two serious problems: (1) they ignore the fact that the similarity between emotions with the similar emotional intensity. Consequently, fine-grained information from the low-layer network, which is essential for accurate emotion recognition, is lost. (2) They ignore that the expression of emotion is a continuous process, that is, failing to model the temporal dimension effectively. To address these issues, a novel Pyramid Hybrid Graph Convolutional Network (Pyr-HGCN) is proposed for GER. Specifically, we first introduce and enhance the pyramid structure in GER to compensate for the missing fine-grained information of the ST-GCN structure. Additionally, we design a novel Spatial-Temporal Hybrid Convolution (STHC) block, which can indirectly and simultaneously capture complex spatio-temporal correlations in long-term regions. Extensive experiments and visualizations were performed on several benchmarks, with an accuracy improvement of 0.01 to 0.02 demonstrating the effectiveness of our approach against state-of-the-art competitors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Arunnehru, J., Kalaiselvi Geetha, M.: Automatic human emotion recognition in surveillance video. In: Intelligent techniques in signal processing for multimedia security, pp. 321–342. Springer (2017)

    Google Scholar 

  2. Bhatia, Y., Bari, A.H., Gavrilova, M.: A lstm-based approach for gait emotion recognition. In: 2021 IEEE 20th International Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC). pp. 214–221. IEEE (2021)

    Google Scholar 

  3. Bhattacharya, U., Mittal, T., Chandra, R., Randhavane, T., Bera, A., Manocha, D.: Step: Spatial temporal graph convolutional networks for emotion perception from gaits. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 1342–1350 (2020)

    Google Scholar 

  4. Bhattacharya, U., Roncal, C., Mittal, T., Chandra, R., Kapsaskis, K., Gray, K., Bera, A., Manocha, D.: Take an emotion walk: Perceiving emotions from gaits using hierarchical attention pooling and affective mapping. In: European Conference on Computer Vision. pp. 145–163. Springer (2020)

    Google Scholar 

  5. Chen, Y., Zhang, Z., Yuan, C., Li, B., Deng, Y., Hu, W.: Channel-wise topology refinement graph convolution for skeleton-based action recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13359–13368 (2021)

    Google Scholar 

  6. Daoudi, M., Berretti, S., Pala, P., Delevoye, Y., Bimbo, A.D.: Emotion recognition by body movement representation on the manifold of symmetric positive definite matrices. In: International Conference on Image Analysis and Processing. pp. 550–560. Springer (2017)

    Google Scholar 

  7. Destephe, M., Henning, A., Zecca, M., Hashimoto, K., Takanishi, A.: Perception of emotion and emotional intensity in humanoid robots gait. In: 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO). pp. 1276–1281. IEEE (2013)

    Google Scholar 

  8. Gavrilova, M.L., Ahmed, F., Bari, A.H., Liu, R., Liu, T., Maret, Y., Sieu, B.K., Sudhakar, T.: Multi-modal motion-capture-based biometric systems for emergency response and patient rehabilitation. In: Research Anthology on Rehabilitation Practices and Therapy, pp. 653–678. IGI global (2021)

    Google Scholar 

  9. Han, X., Gao, Y., Lu, Z., Zhang, Z., Niu, D.: Research on moving object detection algorithm based on improved three frame difference method and optical flow. In: 2015 Fifth International Conference on Instrumentation and Measurement, Computer, Communication and Control (IMCCC). pp. 580–584. IEEE (2015)

    Google Scholar 

  10. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7132–7141 (2018)

    Google Scholar 

  11. Jin, L., Wang, X., Nie, X., Wang, W., Guo, Y., Yan, S., Zhao, J.: Rethinking the person localization for single-stage multi-person pose estimation. IEEE Transactions on Multimedia (2023)

    Google Scholar 

  12. Li, B., Zhu, C., Li, S., Zhu, T.: Identifying emotions from non-contact gaits information based on microsoft kinects. IEEE Trans. Affect. Comput. 9(4), 585–591 (2016)

    Article  Google Scholar 

  13. Li, F., Zhu, A., Liu, Z., Huo, Y., Xu, Y., Hua, G.: Pyramidal graph convolutional network for skeleton-based human action recognition. IEEE Sens. J. 21(14), 16183–16191 (2021)

    Article  Google Scholar 

  14. Li, Q., Han, Z., Wu, X.M.: Deeper insights into graph convolutional networks for semi-supervised learning. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)

    Google Scholar 

  15. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2117–2125 (2017)

    Google Scholar 

  16. Liu, Z., Zhang, H., Chen, Z., Wang, Z., Ouyang, W.: Disentangling and unifying graph convolutions for skeleton-based action recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 143–152 (2020)

    Google Scholar 

  17. Martinez, J., Hossain, R., Romero, J., Little, J.J.: A simple yet effective baseline for 3d human pose estimation. In: Proceedings of the IEEE international conference on computer vision. pp. 2640–2649 (2017)

    Google Scholar 

  18. Narayanan, V., Manoghar, B.M., Dorbala, V.S., Manocha, D., Bera, A.: Proxemo: Gait-based emotion learning and multi-view proxemic fusion for socially-aware robot navigation. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 8200–8207. IEEE (2020)

    Google Scholar 

  19. Randhavane, T., Bhattacharya, U., Kapsaskis, K., Gray, K., Bera, A., Manocha, D.: Learning perceived emotion using affective and deep features for mental health applications. In: 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). pp. 395–399. IEEE (2019)

    Google Scholar 

  20. Sheng, W., Li, X.: Multi-task learning for gait-based identity recognition and emotion recognition using attention enhanced temporal graph convolutional network. Pattern Recogn. 114, 107868 (2021)

    Article  Google Scholar 

  21. Shi, L., Zhang, Y., Cheng, J., Lu, H.: Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12026–12035 (2019)

    Google Scholar 

  22. Smith, F.W., Smith, M.L.: Decoding the dynamic representation of facial expressions of emotion in explicit and incidental tasks. Neuroimage 195, 261–271 (2019)

    Article  Google Scholar 

  23. Teepe, T., Gilg, J., Herzog, F., Hörmann, S., Rigoll, G.: Towards a deeper understanding of skeleton-based gait recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1569–1577 (2022)

    Google Scholar 

  24. Teepe, T., Khan, A., Gilg, J., Herzog, F., Hörmann, S., Rigoll, G.: Gaitgraph: Graph convolutional network for skeleton-based gait recognition. In: 2021 IEEE International Conference on Image Processing (ICIP). pp. 2314–2318. IEEE (2021)

    Google Scholar 

  25. Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., Hu, Q.: Eca-net: Efficient channel attention for deep convolutional neural networks. In: Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11531–11539 (2020)

    Google Scholar 

  26. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19 (2018)

    Google Scholar 

  27. Xia, R., Ding, Z.: Emotion-cause pair extraction: A new task to emotion analysis in texts. arXiv preprint arXiv:1906.01267 (2019)

  28. Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Thirty-second AAAI conference on artificial intelligence (2018)

    Google Scholar 

  29. Yin, Y., Jing, L., Huang, F., Yang, G., Wang, Z.: Msa-gcn: Multiscale adaptive graph convolution network for gait emotion recognition. arXiv preprint arXiv:2209.08988 (2022)

  30. Zhuang, Y., Lin, L., Tong, R., Liu, J., Iwamot, Y., Chen, Y.W.: G-gcsn: Global graph convolution shrinkage network for emotion perception from gait. In: Proceedings of the Asian Conference on Computer Vision (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Computer Science, Chongqing University, Chongqing, 400000, China

    Li Jing, Guangchao Yang & Yunfei Yin

Authors
  1. Li Jing
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guangchao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yunfei Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yunfei Yin .

Editor information

Editors and Affiliations

  1. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

  2. Xiamen University, Xiamen, China

    Hanzi Wang

  3. Beijing University of Posts and Telecommunications, Beijing, China

    Zhanyu Ma

  4. Sun Yat-sen University, Guangzhou, China

    Weishi Zheng

  5. Peking University, Beijing, China

    Hongbin Zha

  6. Chinese Academy of Sciences, Beijing, China

    Xilin Chen

  7. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  8. Xiamen University, Xiamen, China

    Rongrong Ji

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jing, L., Yang, G., Yin, Y. (2024). Pyr-HGCN: Pyramid Hybrid Graph Convolutional Network for Gait Emotion Recognition. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14429. Springer, Singapore. https://doi.org/10.1007/978-981-99-8469-5_15

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8469-5_15

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8468-8

  • Online ISBN: 978-981-99-8469-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation