Skip to main content
SpringerLink
Log in

Speech Emotion Recognition Method Based on Cross-Layer Intersectant Fusion

  • Conference paper
  • First Online:
Data Science and Information Security (IAIC 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2059))

Included in the following conference series:

  • 44 Accesses

Abstract

Speech emotion recognition (SER) is a key technology in human-computer interaction (HCI) systems. Although the existing neural-based methods have achieved some satisfactory results in recognition accuracy, the failure of effective in-depth fusion of multi-scale features hinders the improvement of the accuracy of SER. In this paper, we address this issue from the two aspects of extracting exhaustive features and fusing features of multi-scale. In particular, we propose a recognition network based on Cross-Layer Intersectant Fusion, termed CLIF. It mainly consists of multi-scale feature extraction and cross-layer intersectant fusion. The former takes acoustic features as input and extracts feature maps with different receptive field ranges layer by layer through deepening convolution structures. Among these features, the lower level has more original information but also contains noise. The higher level has emotional semantics that is easier to classify but loses the perception of the details of the original acoustic features. Therefore, we use the cross-layer intersectant fusion module to achieve efficient utilization of low-level and high-level features. The experimental results demonstrate that the proposed CLIF is superior to the existing state-of-the-art speech emotion recognition algorithm. The overall recognition accuracies of CLIF can achieve 82.17% and 93.26% on IEMOCAP and CASIA datasets respectively.

Supported in part by Youth Innovation Promotion Association, Chinese Academy of Sciences, in part by South China Sea Nova project of Hainan Province, in part by the Important Science and Technology Project of Hainan Province under Grant ZDKJ2020010, and in part by Frontier Exploration Project Independently Deployed by Institute of Acoustics, Chinese Academy of Sciences under Grant QYTS202015 and Grant QYTS202115.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://pytorch.org/vision/stable/models.html.

References

  1. Busso, C., et al.: IEMOCAP: interactive emotional dyadic motion capture database. Lang. Resour. Eval. 42, 335–359 (2008). https://doi.org/10.1007/s10579-008-9076-6

    Article  Google Scholar 

  2. Cao, Q., Hou, M., Chen, B., Zhang, Z., Lu, G.: Hierarchical network based on the fusion of static and dynamic features for speech emotion recognition. In: 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2021, pp. 6334–6338. IEEE (2021)

    Google Scholar 

  3. Chen, L., Su, W., Feng, Y., Wu, M., She, J., Hirota, K.: Two-layer fuzzy multiple random forest for speech emotion recognition in human-robot interaction. Inf. Sci. 509, 150–163 (2020)

    Article  Google Scholar 

  4. Cummins, N., Scherer, S., Krajewski, J., Schnieder, S., Epps, J., Quatieri, T.F.: A review of depression and suicide risk assessment using speech analysis. Speech Commun. 71, 10–49 (2015)

    Article  Google Scholar 

  5. Kim, J., An, Y., Kim, J.: Improving speech emotion recognition through focus and calibration attention mechanisms. arXiv preprint arXiv:2208.10491 (2022)

  6. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings, San Diego, CA, USA, 7–9 May 2015 (2015). http://arxiv.org/abs/1412.6980

  7. Kockmann, M., Burget, L., et al.: Application of speaker-and language identification state-of-the-art techniques for emotion recognition. Speech Commun. 53(9–10), 1172–1185 (2011)

    Article  Google Scholar 

  8. Li, P., Song, Y., McLoughlin, I.V., Guo, W., Dai, L.R.: An attention pooling based representation learning method for speech emotion recognition. International Speech Communication Association (2018)

    Google Scholar 

  9. Li, Y., Zhao, T., Kawahara, T., et al.: Improved end-to-end speech emotion recognition using self attention mechanism and multitask learning. In: Interspeech, pp. 2803–2807 (2019)

    Google Scholar 

  10. Liu, J., Liu, Z., Wang, L., Guo, L., Dang, J.: Speech emotion recognition with local-global aware deep representation learning. In: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2020, pp. 7174–7178. IEEE (2020)

    Google Scholar 

  11. Liu, Y., Sun, H., Guan, W., Xia, Y., Zhao, Z.: Discriminative feature representation based on cascaded attention network with adversarial joint loss for speech emotion recognition. In: Proceedings of the Interspeech, pp. 4750–4754 (2022)

    Google Scholar 

  12. McFee, B., et al.: librosa: audio and music signal analysis in Python. In: Proceedings of the 14th Python in Science Conference, vol. 8, pp. 18–25 (2015)

    Google Scholar 

  13. Muppidi, A., Radfar, M.: Speech emotion recognition using quaternion convolutional neural networks. In: 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2021, pp. 6309–6313. IEEE (2021)

    Google Scholar 

  14. Nediyanchath, A., Paramasivam, P., Yenigalla, P.: Multi-head attention for speech emotion recognition with auxiliary learning of gender recognition. In: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2020, pp. 7179–7183. IEEE (2020)

    Google Scholar 

  15. Neumann, M., Vu, N.T.: Improving speech emotion recognition with unsupervised representation learning on unlabeled speech. In: 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2019, pp. 7390–7394. IEEE (2019)

    Google Scholar 

  16. Nwe, T.L., Foo, S.W., De Silva, L.C.: Speech emotion recognition using hidden Markov models. Speech Commun. 41(4), 603–623 (2003)

    Article  Google Scholar 

  17. Pao, T.L., Chen, Y.T., Yeh, J.H., Li, P.J.: Mandarin emotional speech recognition based on SVM and NN. In: 18th International Conference on Pattern Recognition, ICPR 2006, vol. 1, pp. 1096–1100. IEEE (2006)

    Google Scholar 

  18. Rajamani, S.T., Rajamani, K.T., Mallol-Ragolta, A., Liu, S., Schuller, B.: A novel attention-based gated recurrent unit and its efficacy in speech emotion recognition. In: 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2021, pp. 6294–6298. IEEE (2021)

    Google Scholar 

  19. Sun, L., Fu, S., Wang, F.: Decision tree SVM model with fisher feature selection for speech emotion recognition. EURASIP J. Audio Speech Music Proces. 2019(1), 1–14 (2019)

    Article  Google Scholar 

  20. Tarantino, L., Garner, P.N., Lazaridis, A., et al.: Self-attention for speech emotion recognition. In: Interspeech, pp. 2578–2582 (2019)

    Google Scholar 

  21. Van Der Maaten, L.: Accelerating t-SNE using tree-based algorithms. J. Mach. Learn. Res. 15(1), 3221–3245 (2014)

    MathSciNet  Google Scholar 

  22. Wang, S., et al.: Advances in data preprocessing for biomedical data fusion: an overview of the methods, challenges, and prospects. Inf. Fus. 76, 376–421 (2021)

    Article  Google Scholar 

  23. Wang, X., Wang, M., Qi, W., Su, W., Wang, X., Zhou, H.: A novel end-to-end speech emotion recognition network with stacked transformer layers. In: 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2021, pp. 6289–6293. IEEE (2021)

    Google Scholar 

  24. Xu, M., Zhang, F., Cui, X., Zhang, W.: Speech emotion recognition with multiscale area attention and data augmentation. In: 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2021, pp. 6319–6323. IEEE (2021)

    Google Scholar 

  25. Xu, M., Zhang, F., Zhang, W.: Head fusion: improving the accuracy and robustness of speech emotion recognition on the IEMOCAP and RAVDESS dataset. IEEE Access 9, 74539–74549 (2021)

    Article  Google Scholar 

  26. Zhang, J., Jia, H.: Design of speech corpus for mandarin text to speech. In: The Blizzard Challenge 2008 Workshop (2008)

    Google Scholar 

  27. Zhang, S., Zhang, S., Huang, T., Gao, W.: Speech emotion recognition using deep convolutional neural network and discriminant temporal pyramid matching. IEEE Trans. Multimedia 20(6), 1576–1590 (2017)

    Article  Google Scholar 

  28. Zhang, Y.D., et al.: Advances in multimodal data fusion in neuroimaging: overview, challenges, and novel orientation. Inf. Fus. 64, 149–187 (2020)

    Article  Google Scholar 

  29. Zhao, Z., Bao, Z., Zhang, Z., Cummins, N., Wang, H., Schuller, B.: Attention-enhanced connectionist temporal classification for discrete speech emotion recognition. ISCA (2019)

    Google Scholar 

  30. Zhu, W., Li, X.: Speech emotion recognition with global-aware fusion on multi-scale feature representation. In: 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2022, pp. 6437–6441. IEEE (2022)

    Google Scholar 

  31. Zou, H., Si, Y., Chen, C., Rajan, D., Chng, E.S.: Speech emotion recognition with co-attention based multi-level acoustic information. In: 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2022, pp. 7367–7371. IEEE (2022)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Hainan Acoustics Laboratory, Institute of Acoustics, Chinese Academy of Sciences, Haikou, 570105, China

    Kaiqiao Wang, Peng Liu, Songbin Li & Jingang Wang

  2. Lingshui, Marine Information, Hainan Observation and Research Station, Lingshui, 572423, China

    Kaiqiao Wang, Peng Liu & Songbin Li

  3. University of Chinese Academy of Sciences, Beijing, 100049, China

    Jingang Wang

  4. The University of Melbourne, Melbourne, VIC, 3010, Australia

    Cheng Zhang

Authors
  1. Kaiqiao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Songbin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jingang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cheng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peng Liu .

Editor information

Editors and Affiliations

  1. Huazhong University of Science and Technology, Wuhan, Hubei, China

    Hai Jin

  2. Chinese Academy of Science, Shenzhen, China

    Yi Pan

  3. Nanjing University of Science and Technology, Nanjing, China

    Jianfeng Lu

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, K., Liu, P., Li, S., Wang, J., Zhang, C. (2024). Speech Emotion Recognition Method Based on Cross-Layer Intersectant Fusion. In: Jin, H., Pan, Y., Lu, J. (eds) Data Science and Information Security. IAIC 2023. Communications in Computer and Information Science, vol 2059. Springer, Singapore. https://doi.org/10.1007/978-981-97-1280-9_21

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-1280-9_21

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-1279-3

  • Online ISBN: 978-981-97-1280-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation