Skip to main content
SpringerLink
Log in

Self-supervised group meiosis contrastive learning for EEG-based emotion recognition

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The progress of EEG-based emotion recognition has received widespread attention from the fields of human-machine interaction and cognitive science. However, recognizing emotions with limited labelled data is still challenging. To address this issue, this paper proposes a self-supervised group meiosis contrastive learning (SGMC) framework for EEG-based emotion recognition. First, to reduce the dependence of emotion labels, SGMC introduces a contrastive learning task according to the alignment of video clips based on the similar EEG response across subjects. Moreover, the model adopts a group projector to extract group-level representations from the group samples to further decrease the subject difference and random effects in EEG signals. Finally, a novel genetics-inspired data augmentation method, named meiosis is developed, which takes advantage of the alignment of video clips among a group of EEG samples to generate augmented groups by pairing, cross exchanging, and separating. The experiments show that SGMC exhibits competitive performance on the publicly available DEAP and SEED datasets. It is worth of noting that the SGMC shows a high ability to recognize emotion even when using limited labelled data. Moreover, the results of feature visualization suggest that the model might have learned video-level emotion-related feature representations to improve emotion recognition. The hyper-parametric analysis further shows the effect of the group size during emotion recognition. Finally, the comparisons of both the symmetric function and the ablation models and the analysis of computational efficiency are carried out to examine the rationality of the SGMC architecture. The code is provided publicly online.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Algorithm 2
Algorithm 3
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Availability of data and materials

All data generated or analyzed during this study are included in this article.

Code Availability

The code is provided publicly online on https://github.com/kanhaoning/Self-supervised-group-meiosis-contrastive-learning-for-EEG-based-emotion-recognition.

References

  1. Kalaganis FP, Adamos DA, Laskaris NA (2018) Musical neuropicks: a consumer-grade bci for on-demand music streaming services. Neurocomputing 280:65–75

    Article  Google Scholar 

  2. Pandey P, Swarnkar R, Kakaria S et al (2019) Understanding consumer preferences for movie trailers from EEG using machine learning. In: The 6th Annual Conference of Cognitive Science, pp 98–101

  3. Subasi A, Tuncer T, Dogan S et al (2021) EEG-based emotion recognition using tunable q wavelet transform and rotation forest ensemble classifier. Biomed Signal Process Control 68(102):648

    Google Scholar 

  4. Tuncer T, Dogan S, Baygin M et al (2021) Tetromino pattern based accurate EEG emotion classification model. Artif Intell Med 123(102):210

    Google Scholar 

  5. Dogan A, Barua PD, Baygin M et al (2022) (2022) Automated accurate emotion classification using clefia pattern-based features with EEG signals. Int J Healthc Manag. https://doi.org/10.1080/20479700.2022.2141694

    Article  Google Scholar 

  6. Zhang X, Yao L, Wang X et al (2019) A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers. J Neural Eng. https://doi.org/10.1088/1741-2552/ABC902

    Article  Google Scholar 

  7. Song T, Zheng W, Song P et al (2020) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11:532–541

    Article  Google Scholar 

  8. Du X, Ma C, Zhang G et al (2022) An efficient lstm network for emotion recognition from multichannel EEG signals. IEEE Trans Affect Comput 13:1528–1540

    Article  Google Scholar 

  9. Tao W, Li C, Song R et al (2020) EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans Affect Comput 14(1):382–393

    Article  Google Scholar 

  10. Becker H, Fleureau J, Guillotel P et al (2020) Emotion recognition based on high-resolution EEG recordings and reconstructed brain sources. IEEE Trans Affect Comput 11:244–257

    Article  Google Scholar 

  11. Zhang G, Yu M, Liu YJ et al (2021) SparseDGCNN: Recognizing emotion from multichannel EEG signals. IEEE Trans Affect Comput 14(1):537–548

    Article  Google Scholar 

  12. Wang Z, Zhang JW, He Y et al (2022) EEG emotion recognition using multichannel weighted multiscale permutation entropy. Appl Intell 52:12,064–12,076

  13. Ye Z, Xie X, Liu Y et al (2021) Towards a better understanding of human reading comprehension with brain signals. The ACM Web Conference 2022:380–391

  14. Zhang X, Zhao Z, Tsiligkaridis T et al (2022) Self-supervised contrastive pre-training for time series via time-frequency consistency. In: 2022 Neural Information Processing Systems (NeurIPS)

  15. Banville HJ, Chehab O, Hyvärinen A et al (2020) Uncovering the structure of clinical EEG signals with self-supervised learning. J Neural Eng 18(4):46,020

  16. Jiang X, Zhao J, Du B et al (2021) Self-supervised contrastive learning for EEG-based sleep staging. In: 2021 International Joint Conference on Neural Networks (IJCNN), pp 1–8

  17. Dmochowski JP, Bezdek MA, Abelson B et al (2014) Audience preferences are predicted by temporal reliability of neural processing. Nat Commun 5(1):4567

    Article  Google Scholar 

  18. Dmochowski JP, Sajda P, Dias J et al (2012) Correlated components of ongoing EEG point to emotionally laden attention - a possible marker of engagement? Front Hum Neurosci 6:112

    Article  Google Scholar 

  19. Shen X, Liu X, Hu X et al (2022) Contrastive learning of subject-invariant EEG representations for cross-subject emotion recognition. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2022.3164516

    Article  Google Scholar 

  20. Ma Y, Zhao W, Meng M et al (2023) Cross-subject emotion recognition based on domain similarity of EEG signal transfer learning. IEEE Trans Neural Syst Rehabilitation Eng 31:936–943

    Article  Google Scholar 

  21. Tian C, Ma Y, Cammon J et al (2023) Dual-encoder vae-gan with spatiotemporal features for emotional EEG data augmentation. IEEE Trans Neural Syst Rehabilitation Eng 31:2018–2027

    Article  Google Scholar 

  22. Tuncer T, Dogan S, Subasi A (2021) Ledpatnet19: Automated emotion recognition model based on nonlinear led pattern feature extraction function using EEG signals. Cogn Neurodyn 16:779–790

    Article  Google Scholar 

  23. Peng Y, Liu H, Li J et al (2022) Cross-session emotion recognition by joint label-common and label-specific EEG features exploration. IEEE Trans Neural Syst Rehabilitation Eng 31:759–768

    Article  Google Scholar 

  24. Alhagry S, Fahmy AA, El-Khoribi RA (2017) Emotion recognition based on EEG using LSTM recurrent neural network. Int J Adv Comput Sci Appl 8(10):355–358

    Google Scholar 

  25. Li Y, Wang L, Zheng W et al (2021) A novel bi-hemispheric discrepancy model for EEG emotion recognition. IEEE Trans Cogn Dev Syst 13:354–367

    Article  Google Scholar 

  26. Tripathi S, Acharya SG, Sharma RD et al (2017) Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. In: 2017 AAAI, pp 4746–4752

  27. Salama ES, El-Khoribi RA, Shoman M et al (2018) EEG-based emotion recognition using 3D convolutional neural networks. Int J Adv Comput Sci Appl 9(8):329–337

    Google Scholar 

  28. Yang Y, Wu Q, Qiu M et al (2018) Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp 1–7

  29. Maria MA, Akhand MAH, Hossain ABMA et al (2023) A comparative study on prominent connectivity features for emotion recognition from EEG. IEEE Access 11:37,809–37,831

  30. Cheah KH, Nisar H, Yap VV et al (2021) Optimizing residual networks and VGG for classification of EEG signals: identifying ideal channels for emotion recognition. J Healthc Eng. https://doi.org/10.1155/2021/5599615

    Article  Google Scholar 

  31. Gu Y, Zhong X, Qu C et al (2023) A domain generative graph network for EEG-based emotion recognition. IEEE J Biomed Health Inform 27:2377–2386

    Article  Google Scholar 

  32. Niu W, Ma C, Sun X et al (2023) A brain network analysis-based double way deep neural network for emotion recognition. IEEE Trans Neural Syst Rehabilitation Eng 31:917–925

    Article  Google Scholar 

  33. Lu W, Tan TP, Ma H (2023) Bi-branch vision transformer network for EEG emotion recognition. IEEE Access 11:36,233–36,243

  34. Song Y, Zheng Q, Liu B et al (2022) EEG conformer: Convolutional transformer for EEG decoding and visualization. IEEE Trans Neural Syst Rehabilitation Eng 31:710–719

    Article  Google Scholar 

  35. Gidaris S, Singh P, Komodakis N (2018) Unsupervised representation learning by predicting image rotations. In: 2018 International Conference on Learning Representations (ICLR)

  36. Sermanet P, Lynch C, Chebotar Y et al (2018) Time-contrastive networks: Self-supervised learning from video. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp 1134–1141

  37. Mikolov T, Sutskever I, Chen K et al (2013) Distributed representations of words and phrases and their compositionality. In: 2013 Conference and Workshop on Neural Information Processing Systems (NIPS), pp 3113–3119

  38. Devlin J, Chang MW, Lee K et al (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. In: 2019 North American Association for Computational Linguistics (NAACL), pp 4171–4186

  39. Saeed A, Salim FD, Ozcelebi T et al (2021) Federated self-supervised learning of multisensor representations for embedded intelligence. IEEE Internet Things J 8:1030–1040

    Article  Google Scholar 

  40. Sarkar P, Etemad A (2020) Self-supervised learning for ecg-based emotion recognition. In: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 3217–3221

  41. Zheng Z, Feng X, Yu H et al (2022) Unsupervised few-shot image classification via one-vs-all contrastive learning. Appl Intell 53(7):7833–7847

    Article  Google Scholar 

  42. Zhang Z, Zhong Sh, Liu Y (2022) GANSER: a self-supervised data augmentation framework for EEG-based emotion recognition. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2022.3170369

    Article  Google Scholar 

  43. Qi C, Su H, Mo K et al (2017) PointNet: deep learning on point sets for 3D classification and segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 77–85

  44. Chen T, Kornblith S, Norouzi M et al (2020) A simple framework for contrastive learning of visual representations. In: The 37th International Conference on Machine Learning (ICML), pp 1597–1607

  45. Paszke A, Gross S, Massa F et al (2019) Pytorch: an imperative style, high-performance deep learning library. In: 2019 Conference and Workshop on Neural Information Processing Systems (NeurIPS), pp 8024–9035

  46. Kingma D, Ba J (2015) Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR)

  47. Ma JX, Tang H, Zheng WL et al (2019) Emotion recognition using multimodal residual LSTM network. In: The 27th ACM International Conference on Multimedia, pp 176–183

  48. Gao Z, Wang X, Yang Y et al (2021) A channel-fused dense convolutional network for EEG-based emotion recognition. IEEE Trans Cogn Dev Syst 13:945–954

    Article  Google Scholar 

  49. Tao W, Li C, Song R et al (2020) EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans Affect Comput 14(1):382–393

    Article  Google Scholar 

  50. Dong Y, Ren F (2020) Multi-reservoirs EEG signal feature sensing and recognition method based on generative adversarial networks. Comput Commun 164:177–184

    Article  Google Scholar 

  51. Li Y, Zheng W, Cui Z et al (2018) EEG emotion recognition based on graph regularized sparse linear regression. Neural Process Lett 49:555–571

    Article  Google Scholar 

  52. Song T, Zheng W, Song P et al (2020) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11:532–541

    Article  Google Scholar 

  53. van der Maaten L, Hinton GE (2008) Visualizing data using t-SNE. J Mach Learn Res 9:2579–2605

    MATH  Google Scholar 

  54. Zhang H, Cisse M, Dauphin YN et al (2018) Mixup: beyond empirical risk minimization. In: 2018 International Conference on Learning Representations (ICLR)

Download references

Funding

The work is supported by Beijing Natural Science Foundation (No. 4222022).

Author information

Authors and Affiliations

  1. Faculty of Science, Beijing University of Technology, 100 Pingleyuan, 100124, Beijing, China

    Haoning Kan, Zihe Liu & Heqian Wang

  2. Beijing-Dublin International College, Beijing University of Technology, 100 Pingleyuan, 100124, Beijing, China

    Jiale Yu

  3. Faculty of Information Technology, Beijing University of Technology, 100 Pingleyuan, 100124, Beijing, China

    Jiajin Huang & Haiyan Zhou

Authors
  1. Haoning Kan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiale Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiajin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zihe Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Heqian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Haiyan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Haoning Kan, Jiajing Huang and Haiyan Zhou designed and developed the model. Haoning Kan, Jiale Yu, Jiajing Huang and Zihe Liu performed the experiments and analyzed the data. Haoning Kan, Jiale Yu, Heqiang Wang and Haiyan Zhou wrote the article.

Corresponding author

Correspondence to Haiyan Zhou.

Ethics declarations

Conflicts of interest

The authors declared that they have no conflicts of interest to this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kan, H., Yu, J., Huang, J. et al. Self-supervised group meiosis contrastive learning for EEG-based emotion recognition. Appl Intell 53, 27207–27225 (2023). https://doi.org/10.1007/s10489-023-04971-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04971-0

Keywords

Search

Navigation