Short communicationDecoding human brain activity with deep learning
Introduction
Recent developments in brain-computer interfaces have promise for the brain activity decoding [1]. Most studies focus on learning a latent space to classify the brain response, which is relatively easy to implement [[2], [3], [4], [5]]. However, the research on understanding how the human brain works is more complex. Perhaps, we can create something more meaningful, illuminating, and complex by studying what happens in the brain. For example, we can study the brain activity caused by watching different images [6].
Decoding the human brain activity evoked by visual stimuli would have a great impact in brain-inspired computing and computer vision [7]. Many scientific communities have promoted research on analyzing how the human brain interacts with the outside world [8,9]. In some studies, deep learning was applied to decode and reconstruct the brain’s visual activity [10]. Current research on reconstructing the images from the brain activity is mostly focused on functional magnetic resonance imaging (fMRI). For example, Shen et al. trained a deep neural network to reconstruct the stimulus image from the brain activity as captured by fMRI, finally, the generated images resembled the stimulus images [11,12]. Du et al. used a deep generative representation with Bayesian inference to combine an external stimulus with the human brain response via fMRI [13]. These methods depend on high sensitivity of fMRI. However, superiority of these methods is offset by the difficulty of manipulating fMRI scanners and their high cost.
To overcome these drawbacks, some researchers turned their attention to electroencephalograms (EEGs) because of their lower cost [14,15]. Additionally, EEG can provide a higher temporal resolution than fMRI. However, the EEG signal has a lower signal-to-noise ratio and lower spatial resolution, and the signal processing algorithm must achieve a higher accuracy. In 2017, PeRCeiVe Lab at the University of Central Florida developed an automated visual classifier by learning a brain activity manifold for specific visual categories with recurrent neural networks [16]. They have learned the visual category representations from EEG signals, but only obtained an average accuracy of 83%. In our opinion, it is not sufficient and may affect sharpness and fidelity of generated images. Afterwards, they compared variational autoencoders and generative adversarial networks in the task of reconstructing the image [17]. Their promising results strongly demonstrate that visually relevant features extracted from EEG can effectively generate images semantically consistent with visual stimuli. However, we think that the quality of generated images can be improved.
At present, great challenges remain in decoding the human brain activity using EEG by reconstructing the visual stimuli. However, this task has a long-term research value. In our opinion, the success factors for a method of decoding and reconstructing the brain’s visual responses are as follows:
- 1)
the latent feature manifold extracted by the decoding algorithm must represent the category of visual stimuli as precisely as possible;
- 2)
an excellent generative model must use the EEG feature manifold to learn stimuli-related image distribution;
- 3)
a suitable evaluation method must be used to judge the trueness and sharpness of the generated images.
In this paper, we make three specific contributions:
- 1)
we propose an LSTM-CNN architecture that consists of an LSTM network followed by a convolutional layer to extract visually related latent representations of EEG signals;
- 2)
we employ an improved spectral normalization generative adversarial network (SNGAN) to conditionally generate images using the learned EEG features;
- 3)
we compute the classification accuracy of the EEG latent representations and analyze the quality of the generated images. We show that our approach is superior to the existing methods.
Section snippets
Materials and methods
In this paper, we design a method to decode the brain activity evoked by visual stimuli. Our study is feasible for several reasons. First, the EEG signals are recorded when human subjects are shown images on a screen. The EEG signals convey visual information that can be used to recognize different images and understand their content. Second, EEG is a multichannel and time-domain signal with underlying noise components. It must be possible to extract low-dimensional and meaningful features to
Results and discussion
This section examines the feasibility of our method for decoding the human brain activity based on EEG and deep learning. We provide the results and discussion from two aspects: 1) we analyze the performance of the Encoder module to evaluate how it learns to extract the latent EEG visual representation; 2) to test our improved SNGAN model, we assess the quality of EEG-driven image by two methods.
Conclusion
In this paper, we propose a method of reading the mind on the visual level based on deep learning. It consists of two phases: 1) an LSTM-CNN model is designed to extract the EEG visual representation; 2) using the learned EEG features, an improved SNGAN network is used to conditionally generate images that depict the same visual categories as the stimuli. Our results find that the proposed LSTM-CNN algorithm is able to reach a competitive performance in discriminating the object classes using
Declaration of Competing Interest
The authors declare that there are no conflicts of interest.
Acknowledgments
We sincerely appreciate the editors and reviewers for their valuable suggestions and questions. This work was supported by the Science and technology development project of Jilin province, China [grant numbers 20190302034GX]; the Fundamental Research Funds for the Central Universities of China [grant numbers 451170301193].
References (29)
- et al.
A broadband method of quantifying phase synchronization for discriminating seizure EEG signals
Biomed. Signal Process. Control
(2019) - et al.
Wave2Vec: deep representation learning for clinical temporal data
Neurocomputing
(2019) - et al.
Single-trial classification of EEG in a visual object task using ICA and machine learning
J. Neurosci. Methods
(2014) - et al.
EEG signals classification: motor imagery for driving an intelligent wheelchair
IEEE. Lat. Am. Trans.
(2018) - et al.
A hybrid BCI (brain-computer interface) based on multi-mode EEG for words typing and mouse control
J. X. Univ. Technol.
(2016) - et al.
Control of an electrical prosthesis with an ssvep-based BCI
IEEE Trans. Biomed. Eng.
(2008) - et al.
LMD based features for the automatic seizure detection of EEG signals using SVM
IEEE Trans. Neural Syst. Rehabil. Eng.
(2017) - et al.
The neural basis of visual body perception
Nat. Rev. Neurosci.
(2007) - et al.
Perceived shape similarity among unfamiliar objects and the organization of the human object vision pathway
J. Neurosci.
(2008) - et al.
Reconstructing speech from human auditory cortex
PLoS Biol.
(2012)
Cortical regions involved in perceiving object shape
J. Neurosci.
Generic decoding of seen and imagined objects using hierarchical visual features
Nat. Commun.
Deep image reconstruction from human brain activity
bioRxiv
Sharing deep generative representation for perceived image reconstruction from human brain activity
Proceedings of the International Joint Conference on Neural Networks
Cited by (36)
Visual image reconstruction based on EEG signals using a generative adversarial and deep fuzzy neural network
2024, Biomedical Signal Processing and ControlDM-RE2I: A framework based on diffusion model for the reconstruction from EEG to image
2023, Biomedical Signal Processing and ControlImages Reconstruction from Functional Magnetic Resonance Imaging Patterns Based on the Improved Deep Generative Multiview Model
2023, NeuroscienceCitation Excerpt :The goal of neural decoding is to classify, recognize or reconstruct the perceived stimuli according to the brain activities information (Cowen et al., 2014a). Decoding human brain activities evoked by visual stimulus images is an important research topic in the field of neuroscience, and the related research have a significant impact on brain heuristic computing and computer vision (Zheng et al., 2020). However, high-level semantic features and low-level visual details are needed for the accurate reconstruction of visual stimulus images through the brain activities.
A GAN model encoded by CapsEEGNet for visual EEG encoding and image reproduction
2023, Journal of Neuroscience MethodsCitation Excerpt :After obtaining rich EEG features from deep neural network by self-learning, it is natural to use generative network to complete the image generation liking in the field of computer vision. The current works (Palazzo et al., 2017; Kavasidis et al., 2017; Zheng et al., 2020a; Fares et al., 2020) based on the Spampinato dataset using the GAN and its variants to learn the latent spaces from the brain signals have generated the meaningful and human-readable representations. Previous studies have achieved very good results so far, but they still have three main drawbacks.
Domain adaptation for epileptic EEG classification using adversarial learning and Riemannian manifold
2022, Biomedical Signal Processing and ControlBoosting-LDA algriothm with multi-domain feature fusion for motor imagery EEG decoding
2021, Biomedical Signal Processing and ControlCitation Excerpt :By reasonably designing classifiers, EEG signals can be accurately divided into different categories, which directly affects the interaction effect of BCI system, and the performance of classifiers determines the quality of BCI system [25,26]. There are many classification and recognition algorithms applied to EEG signals, such as linear discriminant analysis (LDA), decision tree (DT) [27], multilayer perceptron, random forest (RF) [28], neural network (NN) [29], Naive Bayesian (NB) [30], k-nearest neighbor (KNN) [31], support vector machine (SVM) [32] and so on. Because deep learning can avoid more complex feature extraction and feature selection, it is increasingly used in the field of EEG signal decoding.