An EEG/EOG-based hybrid brain-neural computer interaction (BNCI) system to control an exoskeleton for the paralyzed hand

Introduction

While there is major progress in the development of assistive devices built for instance to compensate for a lost or paralyzed limb, for example, lightweight and versatile prostheses or exoskeletons [3, 33] (Figure 1A), intuitive and reliable control of such bio-robotic devices is an enormous challenge. Most available bio-robotic devices use electromyography (EMG), as muscle activity-related EMG activity is well assessable, and EMG control can be quickly learned [16]. While many examples demonstrate the versatility of such EMG control, e.g., to control arm prostheses, exoskeletons [11, 25], or robotic legs [10] and wheelchairs [6], EMG control depends on the availability of sufficient muscle activity often compromised or absent in certain patient populations, e.g., in amputees, stroke survivors, or individuals suffering from severe spinal cord or brachial plexus injuries (BPI). Also, tremor and muscle fatigue can substantially reduce applicability and reliability of EMG control [14, 26]. The development of brain-machine interfaces (BMI) that translate electric or metabolic brain activity into control signals of machines or robots promised to overcome these limitations and dependence of muscle activity [7, 17–19, 23, 24, 29] but is associated with other challenges far from being mastered yet [31]. While implantation of invasive BMI systems led to impressive results, e.g., enabling a quadriplegic patient to drink a cup of coffee [5], the risk of the necessary surgical procedures and instability of decoding performance are still to be evaluated in larger clinical trials. Also, many patients refuse surgical procedures, and the cost-benefit ratio often depends on the patients’ individual situation [4]. Therefore, noninvasive BMI technology might represent a possible alternative but does not reach high reliability and is susceptible to signal artifacts, particularly when pursued in daily life environments. Furthermore, brain activity, e.g., recorded by electroencephalography (EEG), is highly nonstationary [13] requiring frequent recalibration that interferes with fluent, self-paced (asynchronous) control. Systems that combine or fuse different input signals to control computers or external devices were recently conceptualized as hybrid BMI [20] or hybrid brain/neural-computer interaction (BNCI) systems [8] and are currently broadly investigated [28].

Figure 1

(A) Lightweight and versatile hand exoskeleton allowing for grasping motions developed by The BioRobotics Institute, Scuola Superiore di Studi Universitari e Perfezionamento Sant’Anna, Pisa, Italy [25]. (B) Illustration of a hybrid EEG/EOG brain/neural-computer interaction (BNCI) system setup. (C) Scheme of a BNCI control loop.

As most neurological disorders or trauma leading to paralysis leave the ability to move the eyes intact, e.g., spinal cord injuries or BPI, fusion of biosignals related to the movement of the eyes and noninvasive recordings of brain activity were recently suggested [27]. Such hybrid BNCI system combining electrooculography (EOG) and EEG (Figure 1B and C) promises to increase applicability and reliability of assistive technology, e.g., to control a hand exoskeleton [2] (Figure 1A). Here we introduce such a system and provide proof-of-principle by evaluating its applicability and performance in healthy volunteers and an individual with complete finger paralysis due to a BPI. Upon a visual signal (cue), participants were instructed to either look at their hand, imaging grasping motions or to relax without any intention to grasp. The intention to grasp (i.e., motor intention) was inferred by detection of motor imagery related modulations of EEG sensorimotor rhythm (SMR, 8–15 Hz) quantified as event-related desynchronization and synchronization (ERD/ERS) [22] and became translated into grasping motions of the exoskeleton. In a second condition, false positive detection of motor intention could be interrupted by full left or right eye movements. While previous BNCI studies focused on EEG/EMG fusion [12], this is the first study that investigated the applicability and performance of an EEG/EOG approach in a patient with complete finger paralysis who controlled grasping motions of a hand exoskeleton.

Materials and methods

Biosignal recordings and experimental setup

For BNCI control, EEG was recorded from seven sites (F3, T3, C3, P3, CZ, GRND, and REF) using a wireless 32-channel EEG system (MOVE^®, BrainProducts GmbH, Gilching, Bayern, Germany). EEG was sampled at 200 Hz and bandpass filtered between 0.4 Hz and 70 Hz. A lightweight and robust hand exoskeleton was used allowing for grasping motions of the hand. The exoskeleton was linked to a BNCI system using a custom version of BCI2000, a multipurpose standard BCI platform (www.bci2000.org), integrating different biosignals and translating them into control signals of the exoskeleton (Figure 1C). All participants used the BNCI system under two conditions (condition 1/condition 2). Each condition consisted of three runs with 20 trials in each run (resulting in a total of 60 trials per condition). At the beginning of the study, electric brain activity related to 20 externally paced imagined grasping movements was evaluated to identify the frequency and amplitude of SMR modulation in preparation and execution of motor imagery (calibration run). Then, EEG was recorded for 18 s at rest to calculate the baseline amplitude and variance of SMR in the absence of any motion or initiation of movement. Inability to modulate SMR would result in exclusion from the study. Time-frequency representations (TFRs) were computed for each participant to ensure that the BNCI is controlled by EEG and not EMG or facial muscles (Figure 2). SMR-ERD was computed for the EEG channel closest to the motor cortex (according to the international 10-20 convention: HV, C3 controlling right hand motions; BPI, C4 controlling left hand motions) using the power method described by Pfurtscheller and Aranibar [22]. Based on the SMR amplitude and variance at rest, a threshold for detection of motor intention during continuous direct brain control was defined. As threshold, a value at two standard deviations of SMR variance at rest was used.

Figure 2

Time-frequency representation (TFR) of rest trials (left panel) and task trials (right panel) in a representative participant. Sensorimotor rhythm (SMR, 8–15 Hz) event-related desynchronization (ERD, dark blue) was detectable shortly after onset of the task (0 s) and translated into grasping motions of a hand exoskeleton. Grasping motions were initiated if the ERD detection threshold was exceeded within a sample block of 100 ms. A full closing motions was reached if ERD were detected in 43 successive sample blocks.

Results

In four participants and the BPI patient, 11 Hz was identified as the best frequency to detect motor imagery related modulation of SMR-ERD. One participant showed best motor imagery-related SMR modulation at 9 Hz. In the BNCI calibration runs across participants, SMR-ERD reached maximum peaks of 42.1±5.3% relative to the mean signal power during rest as assessed during the calibration runs and had an average value of 26.2±4.3%. The level of significance between the signal power at rest and signal power during task across participants was p=0.0106.

Across all participants, an average 8.4±3.6% of the trials had to be discarded due to EMG activity. For condition 1, evaluation of percent of time the exoskeleton was moving during each trial resulted in 75.45±16.99% in HV and 61.78±22.17% in BPI. During the trials in which no motions should be elicited, the exoskeleton moved for 32.57±21.23% (HV) and 29.93±25.77% (BPI), respectively, of the total time (Figure 3, Table 1). Accordingly, HV achieved an overall BNCI control performance of 70.24±16.71%, while BPI achieved 65.93±24.27% during condition 1 (Table 1).

Figure 3

Percent of time the exoskeleton was moving during trials and inter-trial intervals in healthy volunteers (HV, black circles) and in the individual with severe brachial plexus injury (BPI, red crosses) under condition 1 (EEG control only) and condition 2 (hybrid EEG/EOG control).

During condition 2, the percent of time the exoskeleton was moving during trials reached 76.38±10.67% in HV and 63.48±16.88% in BPI, while the exoskeleton moved during 14.50±10.0% (HV) and 11.43±8.5% (BPI) of the trials in which the exoskeleton should not be moved (Figure 4). The EOG detection threshold was exceeded in 42.21±21.66% (HV) and 35.38±41.37% (BPI) of these trials (Table 2). This corresponded to a BNCI control performance of 80.65±11.28 (HV) and 76.03±18.32% (BPI), respectively. During condition 2, comparison of BNCI control performance between the conditions (EEG, EEG/EOG) and groups (HV, BPI) using a repeated-measures ANOVA (rmANOVA) revealed a main effect for “condition” (F(1, 32)=7.4724, p<0.01, **) but not for “group” (F(1, 32)=0.0592, p=0.809) and no interaction between the factors (F(1, 32)=0.0598, p=0.808). A post hoc t-test indicated no difference in BNCI performance between HV and BPI under condition 1, while performance improved significantly in both HV and BPI, between condition 1 and condition 2 (p<0.05, *). Also, there was no difference in BNCI performance between HV and BPI under condition 2, indicating that EOG improved BNCI control performance similarly in HV and the paralyzed participant. After the experiment, all participants reported that they experienced the exoskeleton’s movement contingent upon their imagination of grasping motions and that control under condition 1 (EEG only) was more difficult and exhausting compared to condition 2 (hybrid EEG/EOG control). They further stated that they felt more in control of the device under condition 2 with the system promptly responding to their eye’s movement and that they did not experience any changes in control within sequential runs in either condition.

Figure 4

Percent of time the exoskeleton was moving during trials and inter-trial intervals in healthy volunteers (HV, black circles) and in the individual with severe brachial plexus injury (BPI, red crosses) under condition 1 (EEG control only) and condition 2 (hybrid EEG/EOG control).

Discussion

We have introduced a novel hybrid BNCI system merging EEG and EOG signals to control a hand exoskeleton and tested this system in a group of five healthy volunteers and a patient with complete flaccid finger paralysis after severe traumatic BPI. Following calibration, participants used motor imagery of hand grasping motions to control the hand exoskeleton using EEG only (condition 1). While all participants were able to control the BNCI system in a semi-autonomous control mode in which visual cues were presented to evaluate the degree of control at an average performance of 70.24±16.71% (HV) and 65.93±24.27% (BPI), inclusion of EOG control significantly improved performance to 80.65±11.28 (HV) and 76.03±18.32% (BPI), respectively (Figure 5).

Figure 5

Performance of hybrid BNCI control of healthy volunteers (HV) and the patient with complete finger paralysis due to a severe brachial plexus injury (BPI) in each trial. Trials were averaged across HV participants. Fusion of EEG/EOG signals significantly improved BNCI control performance in HV and BPI.

Whereas synchronous, externally cued EEG-based BNCI control with repeated re-calibration of the system is associated with frequent interruptions of control (e.g., every 5 or 10 s), such operation mode can provide significantly higher performances ranging up to 80–90% [32] per trial in average but limits the system’s practicality in daily life where continuous control during an extended period of time (in the range of a few minutes) is desirable. Therefore, a semi-autonomous paradigm was used in which participants controlled the BNCI system without any re-calibration for approximately 10 min in total, while a visual cue was presented to indicate when to perform a grasping motion. To assess the performance of the system in fully asynchronous mode, a real grasping task without a visual cue needs to be performed. Currently, there is no BMI or BNCI system available reaching 100% classification accuracy or control performance of a robotic device or hand exoskeleton. This does not only limit the applicability of these systems but becomes a serious safety issue when unintended commands or movements are performed. Due to the susceptibility of EMG to fatigue, our findings suggest that inclusion of a biosignal providing high reliability and accessibility such as EOG might improve assistive systems in daily life environments. Future research is necessary to show if such novel approach can indeed improve control of assistive devices in daily life environments. It is for instance unclear whether the EOG error correction mechanism has a negative impact on the ability to manipulate objects that require large eye deflections.

While not tested here, additional inclusion of EMG might further improve the system’s degrees of freedom and reliability.

While we aimed to provide proof-of-principle that a patient with complete chronic finger paralysis can improve control of exoskeleton-driven grasping motions when using a hybrid EEG/EOG BNCI system as compared to EEG control only, it was not the aim of this study to evaluate possible functional benefits related to the use of this system. Most likely, only patients with sufficient shoulder and elbow function, e.g., after high cervical spinal cord injury, will benefit from such device.

To compensate for the susceptibility to fatigue or decreasing EMG signal quality over time, adaptive weighting of fused biosignals might further enhance daily-life applicability. While the ability to modulate SMR was comparable between the severely paralyzed BPI patient and healthy volunteers, individuals with brain lesions, e.g., following stroke or a traumatic brain injury, in which this ability is compromised might not have achieved such good control. It was shown, however, that the majority of chronic stroke patients with severe paralysis could learn to modulate ipsilesional SMR [1]. This suggests that the introduced EEG/EOG BNCI system might be also applicable in these patient populations, e.g., in the context of BMI-related neurorehabilition that strives to increase ipsilesional brain activity by providing closed-loop sensory feedback [30]. Alternatively, in stroke patients in which voluntary modulation of ipsilesional brain activity is insufficient for control, signals from the contralesional, unaffected motor areas might be used [9]. Besides restoring the patients’ ability for grasping motions using a hand exoskeleton, it is unclear, though, whether such application would lead to motor recovery.

Conclusion

Hybrid BNCI systems that fuse biosignals from different sources, e.g., EEG and EOG, can lead to better performance in control of a hand exoskeleton than systems that use brain signals alone. Future studies should investigate to what extend such hybrid BNCIs can increase applicability of assistive devices in real life scenarios and identify the main factors influencing user acceptance.