Skip to main content

Advertisement

Log in

Portable deep-learning decoder for motor imaginary EEG signals based on a novel compact convolutional neural network incorporating spatial-attention mechanism

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Due to high computational requirements, deep-learning decoders for motor imaginary (MI) electroencephalography (EEG) signals are usually implemented on bulky and heavy computing devices that are inconvenient for physical actions. To date, the application of deep-learning techniques in independent portable brain-computer-interface (BCI) devices has not been extensively explored. In this study, we proposed a high-accuracy MI EEG decoder by incorporating spatial-attention mechanism into convolution neural network (CNN), and deployed it on fully integrated single-chip microcontroller unit (MCU). After the CNN model was trained on workstation computer using GigaDB MI datasets (52 subjects), its parameters were then extracted and converted to build deep-learning architecture interpreter on MCU. For comparison, EEG-Inception model was also trained using the same dataset, and was deployed on MCU. The results indicate that our deep-learning model can independently decode imaginary left-/right-hand motions. The mean accuracy of the proposed compact CNN reaches 96.75 ± 2.41% (8 channels: Frontocentral3 (FC3), FC4, Central1 (C1), C2, Central-Parietal1 (CP1), CP2, C3, and C4), versus 76.96 ± 19.08% of EEG-Inception (6 channels: FC3, FC4, C1, C2, CP1, and CP2). To the best of our knowledge, this is the first portable deep-learning decoder for MI EEG signals. The findings demonstrate high-accuracy deep-learning decoding of MI EEG in a portable mode, which has great implications for hand-disabled patients. Our portable system can be used for developing artificial-intelligent wearable BCI devices, as it is less computationally expensive and convenient for real-life application.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Xygonakis I, Athanasiou A, Pandria N et al (2018) Decoding motor imagery through common spatial pattern filters at the EEG source space. Comput Intell Neurosci 2018:7957408. https://doi.org/10.1155/2018/7957408

    Article  PubMed  PubMed Central  Google Scholar 

  2. Qi FF, Li YQ, Wu W (2015) RSTFC: a novel algorithm for spatio-temporal filtering and classification of single-trial EEG. IEEE Trans Neural Netw Learn Syst 26(12):3070–3082. https://doi.org/10.1109/TNNLS.2015.2402694

    Article  PubMed  Google Scholar 

  3. Lesenfants D, Vanthornhout J, Verschueren E, Francart T (2019) Data-driven spatial filtering for improved measurement of cortical tracking of multiple representations of speech. J Neural Eng 16(6):066017. https://doi.org/10.1088/1741-2552/ab3c92

    Article  CAS  PubMed  Google Scholar 

  4. Wang L, Huang WJ, Yang Z, Zhang C (2020) Temporal-spatial-frequency depth extraction of brain-computer interface based on mental tasks. Biomed Signal Process Control 58:101845. https://doi.org/10.1016/j.bspc.2020.101845

    Article  Google Scholar 

  5. Tautan AM, Rossi AC, de Francisco R, Ionescu B (2021) Dimensionality reduction for EEG-based sleep stage detection: comparison of autoencoders, principal component analysis and factor analysis. Biomed Eng-Biomed Tech 66(2):125–136. https://doi.org/10.1515/bmt-2020-0139

    Article  Google Scholar 

  6. Garcia-Laencina PJ, Rodriguez-Bermudez G, Roca-Dorda J (2014) Exploring dimensionality reduction of EEG features in motor imagery task classification. Expert Syst Appl 41(11):5285–5295. https://doi.org/10.1016/j.eswa.2014.02.043

    Article  Google Scholar 

  7. Haufe S, Dahne S, Nikulin VV (2014) Dimensionality reduction for the analysis of brain oscillations. Neuroimage 101:583–597. https://doi.org/10.1016/j.neuroimage.2014.06.073

    Article  PubMed  Google Scholar 

  8. Artoni F, Delorme A, Makeig S (2018) Applying dimension reduction to EEG data by principal component analysis reduces the quality of its subsequent independent component decomposition. Neuroimage 175:176–187. https://doi.org/10.1016/j.neuroimage.2018.03.016

    Article  PubMed  Google Scholar 

  9. Demuru M, La Cava SM, Pani SM, Fraschini M (2020) A comparison between power spectral density and network metrics: an EEG study. Biomed Signal Process Control 57:101760. https://doi.org/10.1016/j.bspc.2019.101760

    Article  Google Scholar 

  10. Wang RF, Wang J, Yu HT, Wei XL, Yang C, Deng B (2015) Power spectral density and coherence analysis of Alzheimer’s EEG. Cogn Neurodyn 9(3):291–304. https://doi.org/10.1007/s11571-014-9325-x

    Article  PubMed  Google Scholar 

  11. Petroff OA, Spencer DD, Goncharova II, Zaveri HP (2016) A comparison of the power spectral density of scalp EEG and subjacent electrocorticograms. Clin Neurophysiol 127(2):1108–1112. https://doi.org/10.1016/j.clinph.2015.08.004

    Article  PubMed  Google Scholar 

  12. Jiang AM, Shang J, Liu XF, Tang YB, Kwan HK, Zhu YP (2020) Efficient CSP algorithm with spatio-temporal filtering for motor imagery classification. IEEE Trans Neural Syst Rehabil Eng 28(4):1006–1016. https://doi.org/10.1109/TNSRE.2020.2979464

    Article  PubMed  Google Scholar 

  13. Mishuhina V, Jiang XD (2021) Complex common spatial patterns on time-frequency decomposed EEG for brain-computer interface. Pattern Recog 115:107918. https://doi.org/10.1016/j.patcog.2021.107918

    Article  Google Scholar 

  14. Guo Y, Zhang Y, Chen ZQ, Liu Y, Chen W (2020) EEG classification by filter band component regularized common spatial pattern for motor imagery. Biomed Signal Process Control 59:101917. https://doi.org/10.1016/j.bspc.2020.101917

    Article  Google Scholar 

  15. Cote-Allard U, Fall CL, Drouin A, Campeau-Lecours A, Gosselin C, Glette K, Laviolette F, Gosselin B (2019) Deep learning for electromyographic hand gesture signal classification using transfer learning. IEEE Trans Neural Syst Rehabil Eng 27(4):760–771. https://doi.org/10.1109/TNSRE.2019.2896269

    Article  PubMed  Google Scholar 

  16. Kannathal N, Choo ML, Acharya UR, Sadasivan PK (2005) Entropies for detection of epilepsy in EEG. Comput Methods Programs Biomed 80(3):187–194. https://doi.org/10.1016/j.cmpb.2005.06.012

    Article  CAS  PubMed  Google Scholar 

  17. Liang ZH, Wang YH, Sun X, Li D, Voss LJ, Sleigh JW, Hagihira S, Li X (2015) EEG entropy measures in anesthesia. Front Comput Neurosci 9:16. https://doi.org/10.3389/fncom.2015.00016

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Molina V, Lubeiro A, de Luis GR, Gomez-Pilar J, Martin-Santiago O, Igleaias-Tejedor M, Holgado-Madera P, Segarra-Echeverria R, Recio-Barbero M, Nunez P, Haidar MK, Fernandez-Sevillano J, Sanz-Fuentenebro J (2020) Deficits of entropy modulation of the EEG: a biomarker for altered function in schizophrenia and bipolar disorder? J Psychiatry Neurosci 45(5):322–333. https://doi.org/10.1503/jpn.190032

    Article  PubMed  PubMed Central  Google Scholar 

  19. Xu Q, Zhou H, Wang YJ, Huang J (2009) Fuzzy support vector machine for classification of EEG signals using wavelet-based features. Med Eng Phys 31(7):858–865. https://doi.org/10.1016/j.medengphy.2009.04.005

    Article  PubMed  Google Scholar 

  20. Jrad N, Congedo M, Phlypo R, Rousseau S, Flamary R, Yger F, Rakotomamonjy A (2011) sw-SVM: sensor weighting support vector machines for EEG-based brain-computer interfaces. J Neural Eng 8(5):056004. https://doi.org/10.1088/1741-2560/8/5/056004

    Article  CAS  PubMed  Google Scholar 

  21. Kumar B, Gupta D (2021) Universum based Lagrangian twin bounded support vector machine to classify EEG signals. Comput Methods Prog Biomed 208:106244. https://doi.org/10.1016/j.cmpb.2021.106244

    Article  Google Scholar 

  22. Wang JR, Bi LZ, Fei WJ, Guan CT (2021) Decoding single-hand and both-hand movement directions from noninvasive neural signals. IEEE Trans Biomed Eng 68(6):1932–1940. https://doi.org/10.1109/TBME.2020.3034112

    Article  PubMed  Google Scholar 

  23. Erla S, Faes L, Tranquillini E, Orrico D, Nollo G (2011) k-Nearest neighbour local linear prediction of scalp EEG activity during intermittent photic stimulation. Med Eng Phys 33(4):504–512. https://doi.org/10.1016/j.medengphy.2010.12.003

    Article  PubMed  Google Scholar 

  24. Li M, Xu HP, Liu XW, Lu SF (2018) Emotion recognition from multichannel EEG signals using K-nearest neighbor classification. Technol Health Care 26:509–519. https://doi.org/10.3233/THC-174836

    Article  PubMed  PubMed Central  Google Scholar 

  25. Bablani A, Edla DR, Dodia S (2018) Classification of EEG data using k-nearest neighbor approach for concealed information Test. 8th Int Conf Adv Comput Commun (Icacc-2018) 143:242–249

  26. Edla DR, Mangalorekar K, Dhavalikar G, Dodia S (2018) Classification of EEG data for human mental state analysis using random forest classifier. Procedia Comput Sci 132:1523–1532. https://doi.org/10.1016/j.procs.2018.10.392

    Article  Google Scholar 

  27. Fraiwan L, Lweesy K, Khasawneh N, Wenz H, Dickhaus H (2012) Automated sleep stage identification system based on time-frequency analysis of a single EEG channel and random forest classifier. Comput Methods Programs Biomed 108(1):10–19. https://doi.org/10.1016/j.cmpb.2011.11.005

    Article  PubMed  Google Scholar 

  28. Al-Saegh A, Dawwd SA, Abdul-Jabbar JM (2021) Deep learning for motor imagery EEG-based classification: a review. Biomed Signal Process Control 63:102172. https://doi.org/10.1016/j.bspc.2020.102172

    Article  Google Scholar 

  29. Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ (2018) EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng 15(5):056013. https://doi.org/10.1088/1741-2552/aace8c

    Article  PubMed  Google Scholar 

  30. Izzuddin TA, Safri NM, Othman MA (2021) Compact convolutional neural network (CNN) based on SincNet for end-to-end motor imagery decoding and analysis. Biocybernetics Biomed Eng 41(4):1629–1645. https://doi.org/10.1016/j.bbe.2021.10.001

    Article  Google Scholar 

  31. Mumtaz W, Rasheed S, Irfan A (2021) Review of challenges associated with the EEG artifact removal methods. Biomed Signal Process Control 68:102741. https://doi.org/10.1016/j.bspc.2021.102741

    Article  Google Scholar 

  32. Craik A, He YT, Contreras-Vidal JL (2019) Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng 16(3):031001. https://doi.org/10.1088/1741-2552/ab0ab5

    Article  PubMed  Google Scholar 

  33. Gao Z, Dang WD, Wang XM, Hong X, Hou LH, Ma K, Perc M (2021) Complex networks and deep learning for EEG signal analysis. Cogn Neurodyn 15(3):369–388. https://doi.org/10.1007/s11571-020-09626-1

    Article  PubMed  Google Scholar 

  34. Chiarelli AM, Croce P, Merla A, Zappasodi F (2018) Deep learning for hybrid EEG-fNIRS brain-computer interface: application to motor imagery classification. J Neural Eng 15(3):036028. https://doi.org/10.1088/1741-2552/aaaf82

    Article  PubMed  Google Scholar 

  35. Wang P, Jiang AM, Liu XF, Shang J, Zhang L (2018) LSTM-based EEG classification in motor imagery tasks. IEEE Trans Neural Syst Rehabil Eng 26(11):2086–2095. https://doi.org/10.1109/TNSRE.2018.2876129

    Article  PubMed  Google Scholar 

  36. Kant P, Laskar SH, Hazarika J, Mahamune R (2020) CWT based transfer learning for motor imagery classification for brain computer interfaces. J Neurosci Methods 345:108886. https://doi.org/10.1016/j.jneumeth.2020.108886

    Article  PubMed  Google Scholar 

  37. Han Y, Wang B, Luo J, Li L, Li X (2022) A classification method for EEG motor imagery signals based on parallel convolutional neural network. Biomed Signal Process Control 71:103190. https://doi.org/10.1016/j.bspc.2021.103190

    Article  Google Scholar 

  38. Xiao X, Fang Y (2021) Motor imagery EEG signal recognition using deep convolution neural network. Front Neurosci 15:655599. https://doi.org/10.3389/fnins.2021.655599

    Article  PubMed  PubMed Central  Google Scholar 

  39. Santamaría-Vázquez E, Martínez-Cagigal V, Vaquerizo-Villar F, Hornero R (2020) EEG-Inception: a novel deep convolutional neural network for assistive ERP-based brain-computer interfaces. IEEE Trans Neural Syst Rehabil Eng 28(12):2773–2782. https://doi.org/10.1109/TNSRE.2020.3048106

    Article  PubMed  Google Scholar 

  40. Tabar YR, Halici U (2017) A novel deep learning approach for classification of EEG motor imagery signals. J Neural Eng 14(1):016003. https://doi.org/10.1088/1741-2560/14/1/016003

    Article  PubMed  Google Scholar 

  41. Dose H, Moller JS, Iversen HK, Puthusserypady S (2018) An end-to-end deep learning approach to MI-EEG signal classification for BCIs. Expert Syst Appl 114:532–542. https://doi.org/10.1016/j.eswa.2018.08.031

    Article  Google Scholar 

  42. Amin SU, Alsulaiman M, Muhammad G, Mekhtiche MA, Hossain MS (2019) Deep learning for EEG motor imagery classification based on multi-layer CNNs feature fusion. Futur Gener Comput Syst- Int J Escience 101:542–554. https://doi.org/10.1016/j.future.2019.06.027

    Article  Google Scholar 

  43. Zhang RL, Zong Q, Dou LQ, Zhao XY, Tang YF, Li ZY (2021) Hybrid deep neural network using transfer learning for EEG motor imagery decoding. Biomed Signal Process Control 63:102144. https://doi.org/10.1016/j.bspc.2020.102144

    Article  Google Scholar 

  44. Karacsony T, Hansen JP, Iversen HK, Puthusserypady S (2019) Brain computer interface for neuro-rehabilitation with deep learning classification and virtual reality feedback. Proc 10th Augmented Human Int Conf 2019 (Ah2019). https://doi.org/10.1145/3311823.3311864

  45. Lun XM, Yu ZL, Chen T, Wang F, Hou YM (2020) A simplified CNN classification method for MI-EEG via the electrode pairs signals. Front Human Neurosci 14:338. https://doi.org/10.3389/fnhum.2020.00338

    Article  Google Scholar 

  46. Zhang H, Zhao X, Wu ZX, Sun B, Li T (2021) Motor imagery recognition with automatic EEG channel selection and deep learning. J Neural Eng 18(1):016004. https://doi.org/10.1088/1741-2552/abca16

    Article  Google Scholar 

  47. Cho H, Ahn M, Ahn S, Kwon M, Jun SC (2017) EEG datasets for motor imagery brain-computer interface. Gigascience 6(7):1–8. https://doi.org/10.1088/1741-2552/abca16

    Article  PubMed  PubMed Central  Google Scholar 

  48. Tsuchimoto S, Shibusawa S, Lwama S, Hayashi M, Okuyama K, Mizuguchi N, Kato K, Ushiba J (2021) Use of common average reference and large-Laplacian spatial-filters enhances EEG signal-to-noise ratios in intrinsic sensorimotor activity. J Neurosci Methods 353:109089. https://doi.org/10.1016/j.jneumeth.2021.109089

    Article  PubMed  Google Scholar 

  49. Belwafi K, Gannouni S, Aboalsamh H, Mathkour H, Belghith A (2019) A dynamic and self-adaptive classification algorithm for motor imagery EEG signals. J Neurosci Methods 327:108346. https://doi.org/10.1016/j.jneumeth.2019.108346

    Article  PubMed  Google Scholar 

  50. Ramos-Arguelles F, Morales G, Egozeue S, Pabon RM, Alonso MT (2009) Basic techniques of electroencephalography: principles and clinical applications. Anales Del Sistema Sanitario De Navarra 32:69–82. https://doi.org/10.23938/ASSN.0148

  51. Kaya I (2020) A brief summary of EEG artifact handling. arXiv, 2001:00693. https://doi.org/10.5772/intechopen.99127

  52. Wen JH, Thibeau-Sutre E, Diaz-Melo M, Samper-Gonzalez J, Routier A, Bottani S, Dormont D, Durrleman S, Burgos N, Colliot O (2020) Convolutional neural networks for classification of Alzheimer’s disease: overview and reproducible evaluation. Med Image Anal 63:101694. https://doi.org/10.1016/j.media.2020.101694

    Article  PubMed  Google Scholar 

  53. Gu BZ, Ge RJ, Chen Y, Luo LM, Coatrieux G (2021) Automatic and robust object detection in X-Ray baggage inspection using deep convolutional neural networks. IEEE Trans Industr Electron 68(10):10248–10257. https://doi.org/10.1109/TIE.2020.3026285

    Article  Google Scholar 

  54. Wang Y, Luo XB, Ding L, Fu S, Wei X (2019) Detection based visual tracking with convolutional neural network. Knowl-Based Syst 175:62–71. https://doi.org/10.1016/j.knosys.2019.03.012

    Article  Google Scholar 

  55. Schirrmeister RT, Springenberg JT, Fiederer LDJ, Glasstetter M, Eggensperger K, Tangermann M, Hutter F, Burgard W, Ball T (2017) Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp 38(11):5391–5420. https://doi.org/10.1002/hbm.23730

    Article  PubMed  PubMed Central  Google Scholar 

  56. Suchetha M, Madhumitha R, SornaMeena M, Sruthi R (2021) Sequential convolutional neural networks for classification of cognitive tasks from EEG signals. Appl Soft Comput 111:107664. https://doi.org/10.1016/j.asoc.2021.107664

    Article  Google Scholar 

  57. Amin SU, Alsulaiman M, Muhammad G, Mekhtiche MA, Hossain MS (2019) Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion. Futur Gener Comput Syst 101:542–554. https://doi.org/10.1016/j.future.2019.06.027

    Article  Google Scholar 

  58. Zhang R, Zong Q, Dou L, Zhao X, Tang Y, Li Z (2021) Hybrid deep neural network using transfer learning for EEG motor imagery decoding. Biomed Signal Process Control 63:10144. https://doi.org/10.1016/j.bspc.2020.102144

    Article  Google Scholar 

  59. David R, Duke J, Jain A, Reddi VJ, Jeffries N, Li J, Kreeger N, Nappier I, Natraj, M, Regev S, Rhodes R, Wang T, Warden P (2021) TensorFlow lite micro: embedded machine learning on TinyML systems. ArXiv, 2010:08678. https://doi.org/10.48550/arXiv.2010.08678

  60. Ren HG, Bello S, Alam MN, Manivannan N, Shagam T, D’Souza N (2021) Optimizing inference performance of a fundus image quality neural network model for edge computing using TensorFlow Lite. Investig Ophthalmol Visual Sci 62:113

    Google Scholar 

  61. Tariq M, Trivailo PM, Simic M (2018) EEG-based BCI control schemes for lower-limb assistive-robots. Front Human Neurosci 12:312. https://doi.org/10.3389/fnhum.2018.00312

    Article  Google Scholar 

  62. Luu T, Nakagome S, He Y, Contreras-Vidal J (2017) Real-time EEG-based brain-computer interface to a virtual avatar enhances cortical involvement in human treadmill walking. Sci Rep 7:8895. https://doi.org/10.1038/s41598-017-09187-0

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  63. Wang J, Wang M (2021) Review of the emotional feature extraction and classification using EEG signals. Cognitive Robotics 1:29–40. https://doi.org/10.1016/j.cogr.2021.04.001

    Article  Google Scholar 

  64. Al-Qazzaz NK, Alyasseri ZAA, Abdulkareem KH, Ali NS, Al-Mhiqani MN, Guger C (2021) EEG feature fusion for motor imagery: a new robust framework towards stroke patients rehabilitation. Comput Biol Med 137:104799. https://doi.org/10.1016/j.compbiomed.2021.104799

    Article  PubMed  Google Scholar 

  65. Roy Y, Banville H, Albuquerque I, Gramfort A, Falk TH, Faubert J (2019) Deep learning-based electroencephalography analysis: a systematic review. J Neural Eng 16(5):051001. https://doi.org/10.1088/1741-2552/ab260c

    Article  PubMed  Google Scholar 

  66. Mahmood M, Mzurikwao D, Kim YS, Lee Y, Mishra S, Herbert R, Duarte A, Ang CS, Yeo WH (2019) Fully portable and wireless universal brain-machine interfaces enabled by flexible scalp electronics and deep learning algorithm. Nature Mach Intell 1(9):412–422. https://doi.org/10.1038/s42256-019-0091-7

    Article  Google Scholar 

  67. Kranczioch C, Zich C, Schierholz I, Sterr A (2014) Mobile EEG and its potential to promote the theory and application of imagery-based motor rehabilitation. Int J Psychophysiol 91(1):10–15. https://doi.org/10.1016/j.ijpsycho.2013.10.004

    Article  PubMed  Google Scholar 

  68. Netzer E, Frid A, Feldman D (2020) Real-time EEG classification via coresets for BCI applications. Eng Appl Artif Intell 89:103455. https://doi.org/10.1016/j.engappai.2019.103455

    Article  Google Scholar 

  69. Hou YM, Zhou L, Jia SY, Lun XM (2020) A novel approach of decoding EEG four-class motor imagery tasks via scout ESI and CNN. J Neural Eng 17(1):016048. https://doi.org/10.1088/1741-2552/ab4af6

    Article  PubMed  Google Scholar 

Download references

Funding

This study is supported by the Natural Science Foundation of Zhejiang Province (LY20E070005, LY17E070007), “Pioneer” and “Leading Goose” R&D Program of Zhejiang (2022C01119), and National Natural Science Foundation of China (51207038, 62171169).

Author information

Authors and Affiliations

Authors

Contributions

Zhanxiong Wu: methodology, result analysis, writing, and supervision. Xudong Tang: EEG data preprocess, investigation. Jinhui Wu: CNN model construction and training. Jiye Huang: hardware, programming, and debugging. Jian Shen: EEG data analyzing. Hui Hong: review and language editing.

Corresponding author

Correspondence to Zhanxiong Wu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Z., Tang, X., Wu, J. et al. Portable deep-learning decoder for motor imaginary EEG signals based on a novel compact convolutional neural network incorporating spatial-attention mechanism. Med Biol Eng Comput 61, 2391–2404 (2023). https://doi.org/10.1007/s11517-023-02840-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-023-02840-z

Keywords

Navigation