Skip to main content
SpringerLink
Log in

EEG-based emotion recognition via capsule network with channel-wise attention and LSTM models

  • Regular Paper
  • Published:
CCF Transactions on Pervasive Computing and Interaction Aims and scope Submit manuscript

Abstract

Emotion is a kind of psychological and physical state that people produce to objective things. Accurate recognition of emotions is very important in the field of human-machine interface. It is still challenging to extract discriminative features used for electroencephalogram (EEG) emotion recognition which contain the subtle spatial feature of EEG signal and the temporal representations of EEG signal. In order to overcome these challenges, the paper proposes a model based on the capsule network for multi-channel EEG emotion recognition, which combines the attention mechanism and the LSTM network. First, the channel-wise attention mechanism is used to adaptively assign different weights to each channel, then the CapsNet is used to extract the spatial features of the EEG channel, and LSTM is used to extract temporal features of the EEG sequences. The paper proposed method achieves average accuracy of 97.17%, 97.34% and 96.50% on the valence, arouse and dominance of the DEAP dataset, respectively. These results show that the proposed method is better than the current state-of-the-art method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Alhagry, S., Fahmy, A.A., El-Khoribi, R.A.: Emotion recognition based on EEG using LSTM recurrent neural network. Emotion 8(10), 355–358 (2017)

    Google Scholar 

  • Anuradha, R., Saranya, N., Priyadharsini, M., et al.: Assessment of extended MNIST (EMNIST) dataset using capsule networks. In: 2019 International Conference on Intelligent Sustainable Systems (ICISS), pp. 263–266 (2019)

  • Catarina, C.K., Izelle, L., Sophie, C.A., et al.: Diminished facial EMG responses to disgusting scenes and happy and fearful faces in Huntington’s disease. Cortex 106, 185–99 (2018)

    Article  Google Scholar 

  • Chen, J., Zhang, P., Mao, Z., et al.: Accurate EEG-based emotion recognition on combined features using deep convolutional neural networks. IEEE Access 7, 44317–44328 (2019)

    Article  Google Scholar 

  • Christensen, L., Abdullah, M.: EEG Emotion Detection Review, pp. 1–7 (2018)

  • Du, X., Ma, C., Zhang, G., et al.: An efficient LSTM network for emotion recognition from multi-channel EEG signals. IEEE Trans. Affect. Comput. 99, 1–1 (2020). https://doi.org/10.1109/TAFFC.2020.3013711

    Article  Google Scholar 

  • Ekman, P.: An argument for basic emotions. Cognit. Emotion 6, 169–200 (1992)

    Article  Google Scholar 

  • Fabiano, D., Canavan, S.: Emotion recognition using fused physiological signals. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), Cambridge, pp. 42–48. IEEE (2019)

  • Goshvarpour, A.: Evaluation of novel entropy-based complex wavelet sub-bands measures of PPG in an emotion recognition system. J. Med. Biol. Eng. 40(3), 451–461 (2020)

    Article  Google Scholar 

  • Goshvarpour, A., Goshvarpour, A.: The potential of photoplethysmogram and galvanic skin response in emotion recognition using nonlinear features. Australas. Phys. Eng. Sci. Med. 1, 43 (2020)

    Google Scholar 

  • Hassan, M.M., Alam, M.G.R., Uddin, M.Z., et al.: Human emotion recognition using deep belief network architecture. Int. J. Inf. Fusion 51, 10–18 (2019)

    Article  Google Scholar 

  • Hemanth, D.J.: EEG signal based modified Kohonen neural networks for classification of human mental emotions. J. Artif. Intell. Syst. 2, 1–13 (2020)

    Article  Google Scholar 

  • Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  • Hollosi, J., Pozna, C.R.: Improve the accuracy of neural networks using capsule layers. In: 2018 IEEE 18th International Symposium on Computational Intelligence and Informatics (CINTI), pp. 15–18 (2018)

  • Huang, H., Hu, Z., Wang, W., et al.: Multimodal emotion recognition based on ensemble convolutional neural network. IEEE Access 8, 3265–3271 (2020)

    Article  Google Scholar 

  • Hwang, S., Ki, M., Hong, K., et al.: Subject-independent EEG-based emotion recognition using adversarial learning. In: 2020 8th International Winter Conference on Brain-Computer Interface (BCI), pp. 1–4 (2020)

  • Jin, Y.X., Luo, Y., Yu, Y.: Research on EEG emotion recognition based on deep forest. Softw. Guide 18(07), 53–55 (2019)

    Google Scholar 

  • Kan, W., Li, Y.: Emotion recognition from EEG signals by using LSTM recurrent neural networks. J. Nanjing Univ. (Natural Science) 55(01), 110–116 (2019)

    Google Scholar 

  • Katsigiannis, S., Ramzan, N.: DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J. Biomed. Health Inform. 22(1), 98–107 (2017)

    Article  Google Scholar 

  • Keshari, T., Palaniswamy, S.: Emotion recognition using feature-level fusion of facial expressions and body gestures. In: 2019 International Conference on Communication and Electronics Systems (ICCES), pp. 1184–1189 (2019)

  • Kim, B.H., Jo, S., et al.: Deep physiological affect network for the recognition of human emotions. IEEE Trans. Affect. Comput. 2, 230–243 (2020)

    Google Scholar 

  • Kim, M.K., Kim, M.Y., Oh, E., et al.: A review on the computational methods for emotional state estimation from the human EEG. Comput. Math. Methods Med. 2013, 573734 (2013)

    MathSciNet  MATH  Google Scholar 

  • Koelstra, S., Muhl, C., Soleymani, M.: Deap: a database for emotion analysis using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2011)

    Article  Google Scholar 

  • Lee, M.S., Lee, Y.K., Pae, D.S., et al.: Fast emotion recognition based on single pulse PPG signal with convolutional neural network. Appl. Sci. 9(16), 3355 (2019)

    Article  Google Scholar 

  • Li, X., Cai, E.J., Tian, Y.X.: An improved electroencephalogram feature extraction algorithm and its application in emotion recognition. J. Biomed. Eng. 34(4), 510–517 (2017)

    Google Scholar 

  • Liao, J., Zhong, Q., Zhu, Y., et al.: Multimodal physiological signal emotion recognition based on convolutional recurrent neural network. MSE 782(3), 032005 (2020)

    Google Scholar 

  • Liu, W., Qiu, J.L., Zheng, W.L., et al.: Multimodal emotion recognition using deep canonical correlation analysis (2019)

  • Liu, Y., Ding, Y., Li, C., et al.: Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Comput. Biol. Med. 123, 103927 (2020)

    Article  Google Scholar 

  • Luo, J.Y.: Overview of emotion recognition based on EEG signals. Mod. Comput. 08, 68–71 (2020)

    Google Scholar 

  • Ma, J., Tang, H., Zheng, W.L., et al.: Emotion recognition using multimodal residual LSTM network. In: Proceedings of the 27th ACM International Conference on Multimedia, Nice, vol. 8, pp. 176–183. New York (2019)

  • Ma, X., Zhong, H., Li, Y., et al.: Forecasting transportation network speed using deep capsule networks with nested LSTM models. IEEE Trans. Intell. Transport. Syst. 99, 1–12 (2020)

    Google Scholar 

  • Mehrabian, A.: Comparison of the PAD and PANAS as models for describing emotions and for differentiating anxiety from depression. J. Psychopathol. Behav. Assess. 19(4), 331–357 (1997)

    Article  Google Scholar 

  • Mert, A., Akan, A.: Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal. Appl. 21(1), 81–89 (2016)

    Article  MathSciNet  Google Scholar 

  • Nie, D., Wang, X.W., Duan, R.N., et al.: A survey on EEG based emotion recognition. Chin. J. Biomed. Eng. 31(4), 595–606 (2012)

    Google Scholar 

  • Pan, L., Yin, Z., She, S., et al.: Emotional state recognition from peripheral physiological signals using fused nonlinear features and team-collaboration identification strategy. Entropy 22(5), 511 (2020)

    Article  Google Scholar 

  • Sabour S., Frosst N., Hinton G.E.: Dynamic routing between capsules. (2017). arXiv:1710.09829

  • Samarth, T., Shrinivas, A., Ranti, D.S., et al.: Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI’ 17). AAAI Press, pp. 4746–4752 (2017)

  • Sun, Z.G., Xue, Q.D., Wang, X.J., Huang, X.L.: A survey of emotion recognition method based on EEG signals. Beijing Biomed. Eng. 39(2), 186–194 (2020)

    Google Scholar 

  • Tao, W., Li, C., Song, R., et al.: EEG-based emotion recognition via channel-wise attention and self attention[J]. IEEE Trans. Affect. Comput. Early Access (2020). https://doi.org/10.1109/TAFFC.2020.3025777

    Article  Google Scholar 

  • Verma, G.K., Tiwary, U.S.: Affect representation and recognition in 3d continuous valence-arousal-dominance space. Multimed. Tools Appl. 76(2), 2159–2183 (2017)

    Article  Google Scholar 

  • Wang, B.: A Research of Emotion Recognition Based on Respiratory Signals. M.A. dissertation, Southwest University (2010)

  • Wang, Y.: Research and Application on Emotion Recognition Algorithm Based on Multi-model Eye Movement Information. M.A. dissertation, Anhui University, Anhui(2019)

  • Wang, X.W., Nie, D., Lu, B.L.: Emotional state classification from EEG data using machine learning approach. Neurocomputing 129, 94–106 (2014)

    Article  Google Scholar 

  • Wu, S.Y., Xu, X., Shu, L., et al.: Estimation Of valence of emotion using two frontal EEG channels. In: 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Kansas, Nov 13–16, 2017. IEEE, pp. 1127–1130 (2017)

  • Yang, Y., Wu, Q., Qiu, M., et al.: Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. In: 2018 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 1–7 (2018)

  • Zhang, X.N., Luo, P.C., Hu, X.W., et al.: Research on classification performance of small-scale dataset based on capsule network. In: Proceedings of the 2018 4th International Conference on Robotics and Artificial Intelligence, pp. 24–28 (2018)

  • Zhang, D.,Yao, L., Zhang X., Wang S., et al.: Eeg-based intention recognition from spatio-temporal representations via cascade and parallel convolutional recurrent neural networks (2017). arXiv:1708.06578

  • Zhang, G.H., Yu, M.J., Chen, G., et al.: A review of EEG features for emotion recognition (in Chinese). Sci. Sin. (Informationis) 49, 1097–1118 (2019)

    Google Scholar 

Download references

Acknowledgements

The financial support for this work provided by the Cooperative Education Fund of China Ministry of Education (201702113002, 201801193119) and the Scientific Research Fund of Hunan Provincial Education Department (20A191) are greatly appreciated by the authors.

Author information

Authors and Affiliations

  1. Hunan University of Science and Technology, Xiang Tan, China

    Lina Deng & Xiaoliang Wang

  2. Deakin University, Geelong, Australia

    Frank Jiang & Robin Doss

Authors
  1. Lina Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoliang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frank Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Robin Doss
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoliang Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Deng, L., Wang, X., Jiang, F. et al. EEG-based emotion recognition via capsule network with channel-wise attention and LSTM models. CCF Trans. Pervasive Comp. Interact. 3, 425–435 (2021). https://doi.org/10.1007/s42486-021-00078-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42486-021-00078-y

Keywords

Search

Navigation