Skip to main content
SpringerLink
Log in

Multi-scale Learning for Multimodal Neurophysiological Signals: Gait Pattern Classification as an Example

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Neurophysiological signals are manifestations of the underlying brain activity, and they contain an abundance of neural information. The decoding and understanding of these signals is useful to develop robotic exoskeletons, benefitting device-aided motor rehabilitation. To this date, numerous efforts have been carried out to explore the relations between neurophysiological signals and locomotor capacity. Most of these studies focused on a single modality of neurophysiological signal and ignored its multiple modalities. In this study, the modalities from two kinds of biosensors were fused [electroencephalogram (EEG) and electromyogram (EMG)], and a novel deep learning model was proposed (multi-scale learning, MSL) to classify four walking patterns. The EEG and EMG data were collected during a walking experiment, where different walking conditions with and without exoskeleton-aided assistance were implemented (i.e. free-walking and exoskeleton-aided walking at zero, low, and high assistive forces). The performance achieved by the MSL model was compared to that of existing models, and the results show that multimodal MSL achieved the highest performance in terms of classification accuracy (89.33%). Moreover, the comparisons in our study show that an improved classification performance was obtained when a full 62-channel EEG setting was used compared to using a subset of 20 channels located on the sensorimotor region. This work contributes to the improvement of neurophysiological signal decoding and promotes the development of rehabilitation technologies as well as exoskeleton-aided applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Ren WJ, Han M (2019) Classification of EEG signals using hybrid feature extraction and ensemble extreme learning machine. Neural Process Lett 50(2):1281–1301

    Article  MathSciNet  Google Scholar 

  2. Xi XG, Yang C, Shi JH, Luo ZZ, Zhao YB (2019) Surface electromyography-based daily activity recognition using wavelet coherence coefficient and support vector machine. Neural Process Lett 50(3):2265–2280

    Article  Google Scholar 

  3. Young AJ, Ferris DP (2017) State of the art and future directions for lower limb robotic exoskeletons. IEEE Trans Neural Syst Rehabil Eng 25(2):171–182

    Article  Google Scholar 

  4. Gordleeva SY, Lobov SA, Grigorev NA, Savosenkov AO, Shamshin MO, Lukoyanov MV, Khoruzhko MA, Kazantsev VB (2020) Real-time EEG-EMG human-machine interface-based control system for a lower-limb exoskeleton. IEEE Access 8:84070–84081

    Article  Google Scholar 

  5. Ma J, Zhang Y, Cichocki A, Matsuno F (2014) A novel EOG/EEG hybrid human-machine interface adopting eye movements and ERPs: application to robot control. IEEE Trans Biomed Eng 62(3):876–889

    Article  Google Scholar 

  6. Rashed-Al-Mahfuz M, Moni MA, Uddin S, Alyami SA, Summers MA, Eapen V (2021) A deep convolutional neural network method to detect seizures and characteristic frequencies using epileptic electroencephalogram (eeg) data. IEEE J Transl Eng Health Med 9:2000112–2000112

    Google Scholar 

  7. Amiri S, Fazel-Rezai R, Asadpour V (2013) A review of hybrid brain-computer interface systems. Adv Hum Comput Interact 2013

  8. Giraudet L, Imbert J-P, Berenger M, Tremblay S, Causse M (2015) The neuroergonomic evaluation of human machine interface design in air traffic control using behavioral and EEG/ERP measures. Behav Brain Res 294:246–253

    Article  Google Scholar 

  9. da Silva-Sauer L, Valero-Aguayo L, de la Torre-Luque A, Ron-Angevin R, Varona-Moya S (2016) Concentration on performance with p300-based BCI systems: a matter of interface features. Appl Ergon 52:325–332

    Article  Google Scholar 

  10. Cho H, Ahn M, Ahn S, Kwon M, Jun SC (2017) EEG datasets for motor imagery brain-computer interface. GigaScience 6(7):gix034

    Article  Google Scholar 

  11. Ramoser H, Muller-Gerking J, Pfurtscheller G (2000) Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabil Eng 8(4):441–446

    Article  Google Scholar 

  12. Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemometr Intell Lab Syst 2(1–3):37–52

    Article  Google Scholar 

  13. Comon P (1994) Independent component analysis, a new concept? Signal Process 36(3):287–314

    Article  Google Scholar 

  14. Phinyomark A, Phukpattaranont P, Limsakul C (2012) Feature reduction and selection for EMG signal classification. J Expert Syst Appl 39(8):7420–7431

    Article  Google Scholar 

  15. Phinyomark A, Quaine F, Charbonnier S, Serviere C, Tarpin-Bernard F, Laurillau Y (2013) EMG feature evaluation for improving myoelectric pattern recognition robustness. Expert Syst Appl 40(12):4832–4840

    Article  Google Scholar 

  16. Wang H, Zhang L, Yao LX (2021) Application of genetic algorithm based support vector machine in selection of new EEG rhythms for drowsiness detection. Expert Syst Appl 171:7

    Google Scholar 

  17. Griffin D, Lim J (1984) Signal estimation from modified short-time. IEEE Trans Acoust Speech Signal Process 32(2):236–243

    Article  Google Scholar 

  18. Newson JJ, Thiagarajan TC (2019) EEG frequency bands in psychiatric disorders: a review of resting state studies. Front Hum Neurosci 12:24

    Article  Google Scholar 

  19. Li J, Dimitrakopoulos GN, Thangavel P, Chen G, Sun Y, Guo Z, Yu H, Thakor N, Bezerianos A (2019) What are spectral and spatial distributions of EEG-EMG correlations in overground walking? An exploratory study. IEEE Access 7:143935–143946

    Article  Google Scholar 

  20. Alharthi AS, Yunas SU, Ozanyan KB (2019) Deep learning for monitoring of human gait: a review. IEEE Sens J 19(21):9575–9591

    Article  Google Scholar 

  21. Zhang J, Wang B, Zhang C, Xiao Y, Wang MY (2019) An EEG/EMG/EOG-based multimodal human-machine interface to real-time control of a soft robot hand. Front Neurorobot 13:7

    Article  Google Scholar 

  22. Said AB, Mohamed A, Elfouly T, Harras K, Wang ZJ (2017) Multimodal deep learning approach for joint EEG-EMG data compression and classification. In: 2017 IEEE wireless communications and networking conference (WCNC)

  23. Wang T, Bezerianos A, Cichocki A, Li J (2020) Multi-kernel capsule network for schizophrenia identification. IEEE Trans Cybern 10:10. https://doi.org/10.1109/TCYB.2020.3035282

    Article  Google Scholar 

  24. Zhang P, Wang X, Zhang W, Chen J (2019) Learning spatial-spectral-temporal EEG features with recurrent 3D convolutional neural networks for cross-task mental workload assessment. IEEE Trans Neural Syst Rehabil Eng 27(1):31–42

    Article  Google Scholar 

  25. Lamos M, Marecek R, Slavicek T, Mikl M, Rektor I, Jan J (2018) Spatial-temporal-spectral EEG patterns of BOLD functional network connectivity dynamics. J Neural Eng 15(3):036025

    Article  Google Scholar 

  26. Burges Christopher JC (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 2(2):121–167

    Article  Google Scholar 

  27. Breiman L (2001) Random forests. Mach Learn Arch 45(1):5–32

    Article  Google Scholar 

  28. Thomas C, Hart PE (1967) Nearest neighbor pattern classification. IEEE Trans Inf Theory 13(1):21–27

    Article  Google Scholar 

  29. Wenyu L, Feng D, Sheng Shili XCG, Rensong L, Zhiwen Z, Xue J (2018) A human-vehicle collaborative simulated driving system based on hybrid brain-computer interfaces and computer vision. IEEE Trans Cognit Dev Syst 10(3):810–822

    Article  Google Scholar 

  30. Hu HF, Liao ZK, Xiao X (2019) Action recognition using multiple pooling strategies of CNN features. Neural Process Lett 50(1):379–396

    Article  Google Scholar 

  31. Lipton ZC, Berkowitz J, Elkan C (2015) A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv:1506.00019

  32. Zhang R, Nie F, Li X, Wei X (2019) Feature selection with multi-view data: a survey. Inf Fusion 50:158–167

    Article  Google Scholar 

  33. Wang G, Han H, Shan S, Chen X (2020) Cross-domain face presentation attack detection via multi-domain disentangled representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  34. Xie E, Zang Y, Shao S, Gang Yu, Yao C, Li G (2019) Scene text detection with supervised pyramid context network. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 9038–9045

  35. Han ZY, Zhao J, Leung H, Ma A, Wang W (2021) A review of deep learning models for time series prediction. IEEE Sens J 21(6):7833–7848

    Article  Google Scholar 

  36. Alexander C, Yongtian H, Contreras-Vidal Jose L (2019) Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng 16(3):031001

    Article  Google Scholar 

  37. Tsinganos P, Cornelis B, Cornelis J, Jansen B, Skodras A (2018) Deep learning in EMG-based gesture recognition. In: Proceedings of the 5th international conference on physiological computing systems, pp 107–114

  38. Kuan GS, Abbass Hussein A, Chen TK, Abdullah A-M, Nitish T, Anastasios B, Junhua L (2018) Spatio-spectral representation learning for electroencephalographic gait-pattern classification. IEEE Trans Neural Syst Rehabil Eng 26(9):1858–1867

    Article  Google Scholar 

  39. Dai M, Zheng D, Na R, Wang S, Zhang S (2019) EEG classification of motor imagery using a novel deep learning framework. Sensors 19(3):551

    Article  Google Scholar 

  40. Zhai X, Jelfs B, Chan RHM, Tin C (2017) Self-recalibrating surface EMG pattern recognition for neuroprosthesis control based on convolutional neural network. Front Neurosci 11:379

    Article  Google Scholar 

  41. Neep H, Zhu CJ, Chung TA, Alex S (1997) Classification of EEG signals using the wavelet transform. Signal Process 59(1):61–72

    Article  Google Scholar 

  42. Li J, Thakor N, Bezerianos A (2018) Unilateral exoskeleton imposes significantly different hemispherical effect in parietooccipital region, but not in other regions. Sci Rep 8:126

    Article  Google Scholar 

  43. Li J, Chen G, Thangavel P, Yu H, Thakor N, Bezerianos A, Sun Y (2016) A robotic knee exoskeleton for walking assistance and connectivity topology exploration in EEG signal. In: 6th IEEE international conference on biomedical robotics and biomechatronics (BioRob), proceedings of the IEEE RAS-EMBS international conference on biomedical robotics and biomechatronics, pp 1068–1073

  44. He P, Wilson G, Russell C (2004) Removal of ocular artifacts from electro-encephalogram by adaptive filtering. Med Biol Eng Comput 42(3):407–412

    Article  Google Scholar 

  45. Li J, Chen Y, Taya F, Lim J, Wong K, Sun Y, Bezerianos A (2017) A unified canonical correlation analysis-based framework for removing gradient artifact in concurrent EEG/fMRI recording and motion artifact in walking recording from EEG signal. Med Biol Eng Comput 55(9):1669–1681

    Article  Google Scholar 

  46. Torrence C, Compo GP (1998) A practical guide to wavelet analysis. Bull Am Meteorol Soc 79(1):61–78

    Article  Google Scholar 

  47. Krizhevsky A, Sutskever I, Geoffrey HE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 875:84–90

    Article  Google Scholar 

Download references

Author information

Author notes

  1. F. Duan and Y. Lv have contributed equally to this work.

Authors and Affiliations

  1. College of Artificial Intelligence, Nankai University, Jinnan, Tianjin, China

    Feng Duan & Yizhi Lv

  2. Computational Engineering Applications Unit, Head Office for Information Systems and Cybersecurity, Riken, Wako-Shi, Japan

    Zhe Sun

  3. The School of Computer Science and Electronic Engineering, University of Essex, Colchester, CO4 3SQ, UK

    Junhua Li

Authors
  1. Feng Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yizhi Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhe Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junhua Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Feng Duan, Zhe Sun or Junhua Li.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Duan, F., Lv, Y., Sun, Z. et al. Multi-scale Learning for Multimodal Neurophysiological Signals: Gait Pattern Classification as an Example. Neural Process Lett 54, 2455–2470 (2022). https://doi.org/10.1007/s11063-021-10738-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-021-10738-w

Keywords

Search

Navigation