Abstract
Functional near-infrared spectroscopy (fNIRS) is a type of functional brain imaging. Brain-computer interfaces (BCIs) based on fNIRS have recently been implemented. Most existing fNIRS-BCI studies have involved off-line analyses, but few studies used online performance testing. Furthermore, existing online fNIRS-BCI experimental paradigms have not yet carried out studies using different imagined movements of the same side of a single limb. In the present study, a real-time fNIRS-BCI system was constructed to identify two imagined movements of the same side of a single limb (right forearm and right hand). Ten healthy subjects were recruited and fNIRS signal was collected and real-time analyzed with two imagined movements (leftward movement involving the right forearm and right-hand clenching). In addition to the mean and slope features of fNIRS signals, the correlation coefficient between fNIRS signals induced by different imagined actions was extracted. A support vector machine (SVM) was used to classify the imagined actions. The average accuracy of real-time classification of the two imagined movements was 72.25 ± 0.004%. The findings suggest that different imagined movements on the same side of a single limb can be recognized real-time based on fNIRS, which may help to further guide the practical application of online fNIRS-BCIs.
Introduction
BCIs aim to bypass peripheral nerves and muscles to establish direct communication and/or control between brains and external devices. This technology is expected to provide an alternative new communication and/or control platform for patients with severe movement disabilities or for healthy people in specific situations with ad-hoc needs for BCIs [1, 2]. At present, non-invasive BCIs are mainly based on EEG rather than fNIRS.
fNIRS uses near-infrared light to measure hemodynamic responses in the cerebral cortex. fNIRS indirectly reflects changes in neural activity by calculating changes in local concentrations of HbO and HbR [3], [4], [5], [6]. Compared with EEG, fNIRS allows for a certain tolerance to movement. fNIRS can measure hemodynamic activity in brain tissue under the middle position between the transmitting probe and the receiving probe, which yields a better spatial localization compared to that of EEG. In addition, since fNIRS involves an optical signal, its measurement is not affected by electromagnetic interference [7]. Therefore, using fNIRS signals to drive BCIs has potential advantages [8], [9], [10], [11], [12], [13].
Currently, most fNIRS-BCI studies have involved off-line analyses and have adopted paradigms mainly consisting of the imagined movements of different limbs, such as the imagined differences between the following: the raising of left and right arms and the grasping of left and right hands [10]; flexion movements of left and right wrists [11]; movements of left and right hands [12]; movements of the right hand and foot [14]; the percussive movements of left and right fingers [15]; movements involving the left and right hands and feet [16]; left- and right-hand movements and mental calculations [17]; left- and right-hand movements and mental arithmetic and mental counting [18]; clench force and speed [19] and the movement of right or left ankle [20]. In addition, mental arithmetic, mental counting, and puzzle solving have been divided into three kinds of mental activities [21]. Handshake, ball grabbing, hand pricking, and cold temperature have been classified into four kinds of mental activities [22]. Auditory studying of English speech, non-English speech, annoying sounds, and nature sounds have also been studied [23]. The common fNIRS features extracted in the previous fNIRS-BCI studies have included the mean value, peak value, slope, and other features of the oxyhemoglobin signal [11, 12, 18, 23, 24] that belong to the signal’s own attributes and do not involve correlations between signals induced by different tasks. However, such correlations may contribute to the recognition of different tasks.
Currently, few studies have employed online fNIRS-BCI. These studies have used the following paradigms: online decoding deception task [25]; answering “yes” and “no” through mental activity task [26]; real-time monitoring of brain activity states [27]; and monitoring and prediction of low, medium, and high mental states [28]. In addition, studies using online fNIRS-BCI movement imagination have mainly included imagination of the following: left- and right-hand movements in which mental arithmetic and mental counting were used [29]; continuous imagination of left movement of the right forearm (two classification problems with rest states) [30]; self-paced right hand finger-tapping [31]; and mental arithmetic used as the cognitive training task [32]. However, no previous studies have investigated the paradigm of real-time recognition of different movements on the same side of a single limb based on fNIRS. The existing fNIRS features extracted in online fNIRS-BCI studies have mainly used similar features to those selected in offline studies.
Based on the above analysis, the present study employed a real-time fNIRS-BCI to develop a new experimental paradigm that required subjects to imagine two different actions on the same side of a single limb (leftward movement involving the right forearm, as well as right-hand clenching), after which we tested the separability between the two kinds of imagined actions via fNIRS signals. In addition, in contrast to traditional fNIRS features, we extracted the correlation coefficient between different imagined actions via fNIRS signals as a new feature. Finally, the validity of the proposed paradigm and the extracted new features were verified by the real-time fNIRS-BCI system.
Materials and methods
Subjects, experimental paradigm, experimental process, and data collection
Subjects
In this experiment, 10 male subjects (20–28 years old) with the mean ± standard deviation age of 23.7 ± 2.32 years were recruited, and all of them were undergraduates and postgraduates. All subjects were right-handed and had no history of mental illness or motor disorders. Before signing the experimental informed consent form, the subjects were informed of the content, nature, and purpose of the experiment, after which they all voluntarily signed the experimental informed consent form. This study was approved by the medical ethics committee of the Affiliated Hospital of Kunming University of Science and Technology.
Experimental paradigm
The task involved imagined movements of the right upper limb, as follows: (1) leftward movement involving the right forearm; and (2) right-hand clenching. The experimental paradigm is shown in Figure 1. The experiment was divided into two parts: baseline time and task time. The duration of the baseline time was 60 s. Each trial time included 2 s of prompt time, 10 s of movement imagination time, and 20 s of rest time for a total of 32 s. There were 40 trials in each experiment and 20 trials for each task. Each subject was recorded a total of two experiments separated by short breaks.
Experimental paradigm. An experiment included a total of 40 trials preceded by 60 s of rest. The schematic diagram of one trial is shown. After 60 s of baseline time, a beep cue occurred for 2 s, which indicated subjects to perform the motor imagery tasks. The type of task was randomly displayed on the computer screen. After performing the tasks, participants were provided 20 s to rest between trials.
At the beginning of the experiment, the subjects were required to be awake and relaxed during the baseline period, and then the first trial was performed. First, one of the two imagined actions of the right upper limb was randomly prompted. After the prompt disappeared, an asterisk was displayed in the center of the screen. The subjects were required to imagine the action prompted from a first-person perspective. Then, the screen prompted the subjects to rest and to then enter the next trial after the rest period. During the experiment, the whole process was accompanied with voice prompts.
Experimental process
The experiment was conducted in a spacious and quiet room. The subjects sat on a comfortable chair and faced the display screen at 65–70 cm away from their eyes. Each subject received task cue through the display screen. Before the formal experiment, subjects were trained to imagine two movements of the right upper limb (leftward movement involving the right forearm and right-hand clenching), during which a pre-experiment was carried out. The subjects were required to be familiar with the whole experimental process. Before the experiment, the subjects were required to relax for at least 5 min to stabilize their heart rate and blood pressure. During the experiment, the subjects were required to keep still and relax, avoid body movements as much as possible, reduce interference to near-infrared signals, and complete the required tasks according to the experimental paradigm in Figure 1.
Signal acquisition
The real-time system mainly includes two parts: server and client. The server is used to collect and store fNIRS signals. The function of the client mainly includes the training of classifier, Bluetooth connection, stimulus presentation, server connection, real-time signal reading and real-time task data processing. The framework of the whole system is shown in Figure 2. The entire system is consisted of NirSmart device (Danyang Huichuang Medical Equipment Co., LTD, China) used as the server of the BCI system and the PC with MATLAB (Mathworks, inc.) served as the client.
The framework of the real-time BCI system.
The Nirsmart fNIRS equipment used two wavelengths (760 and 850 nm) of near-infrared light, consisting of six incident light-source probes and eight detection-fiber probes. The arrangement of fNIRS light-source probes and detection probes is shown in Figure 3. There were eight channels placed on the left scalp and eight channels placed on the right scalp, for a total of 16 channels. The left and right regions were symmetrical and covered the motor area of the cerebral cortex. In Figure 3, S represents the light source probe, D represents the detection probe, the line between the light source probe and the detection probe represents the channel, and the number beside each channel represents the channel number. The sampling rate of the device was set to 20 Hz. Finally, during the experiment, all lights in the room were turned off to reduce noise from environmental light sources.
Channel configurations.
(A) is the locations of emitters (LEDs) and detectors (photodiodes) on the cortical motor area; the S indicates the detectors, the D represents the emitters, the numbers denote the channel numbers. (B) is a picture of the cap on a participant.
The client sent predefined commands to the robot via Bluetooth communication. The software interface of the BCI system is shown in Figure 4. The fNIRS device provided the data interface based on TCP/IP communication. The MATLAB program on the client called the interface function to connect to the server and the real-time data acquisition function to obtain the collected data.
The software interface of the BCI system.
Signal processing
The interference components of fNIRS signals collected in the experiment mainly included baseline drift, movement artifacts, physiological interference, and high-frequency noise. Among them, the baseline drift mainly derived from the baseline change caused by changes in equipment temperature during recordings. Movement artifacts mainly originated from changes in the light path caused by the relative movement between the light pole and the head. Physiological interference mainly included cardiac interference (0.7–1.5 Hz), respiratory interference (0.13–0.33 Hz), and Mayer waves (0.1 Hz). High-frequency noise mainly came from power-derived frequency interference. First, data preprocessing was carried out to eliminate high-frequency physiological interference, after which the time series of each channel was divided into epochs of rest, imagined leftward movement involving the right forearm, and imagined right-hand clenching. Features were extracted from the time series of each experiment and were stored for further analysis. The first experimental dataset of each subject was used as training data, and the second experimental dataset was classified real-time.
Data pre-processing
First, a 3rd order Butterworth band-pass infinite impulse response filter with cutoff frequencies of 0.02 and 0.1 Hz was used to filter the collected fNIRS data. This band-pass filter removed high-frequency physiological noise caused by heartbeats (1–1.5 Hz), respiration (0.13–0.33 Hz), and Mayer waves (0.1 Hz). After the above processing, the baseline drift was further removed.
The fNIRS signal collected in this experiment is the original light intensities, which was converted into the relative value of HbO and HbR using the Modified Beer–Lambert Law (MBLL) [33]. The differential path length factors is 6. The molar absorption coefficients of HbO and HbR are 1486.587 cm−1/(mol L−1) and 3843.707 cm−1/(mol L−1) respectively when the wavelength is 760 nm, and the molar absorption coefficients of HbO and HbR are 2526.391 cm−1/(mol L−1) and 1798.643 cm−1/(mol L−1) respectively when the wavelength is 850 nm. The improvement algorithm about the correlation of inverse variation with HbO and HbR was used to remove movement artifacts [34].
Feature selection
After the fNIRS signals being preprocessed, the following features are extracted from the time-domain signal: signal mean and signal slope, which are commonly used in near-infrared research. The mean value of the HbO signal represents the degree of activity of the corresponding brain area, while the slope of the HbO signal can reflect changes in the degree of brain activity. The signal mean value is as follows (1):
where x N is the sampling point during the task, and N is the sampling number; the signal slope was first fitted with MATLAB, and then the maximum slope of the curve was calculated.
In addition to the signal mean and signal slope, the correlation coefficients between fNIRS signals induced by different imagined movement tasks were extracted as features, as shown in formula (2):
where Cov (X, Y) is the covariance function of random variables X and Y, and σ X and σ Y are the standard deviations of the two random variables, respectively; and E(X) and E(Y) are the average of the two random variables.
In this experiment, there were two kinds of different imagination tasks. After preprocessing, two types of the correlation coefficient were calculated. One was the correlation coefficient between the signals of different imagining tasks for each trial in each channel. The other was the correlation coefficient between the signals of the same imagining task for each trial in each channel. Then, the mean value of the correlation coefficient of each kind of imagined movement task in each channel was taken as the feature. For all of 16 channels, the number of correlation coefficient features from each trial of the imagination tasks was 32.
Classification
In this study, a SVM classification method was used. The SVM Library in MATLAB was used to train the classifier by a cross-validation grid search to determine the best kernel and features, and the best kernel was a Gaussian radial basis function [35]. The cost function parameters and free parameters of the Gaussian kernel were optimized to obtain the optimal classification performance. The SVM model was trained by the data collected in the offline experiment, and the trained SVM model was used to classify the data collected in the real-time experiment. SVMs have been used in various fNIRS-BCI studies for classification purposes and have been shown to work favorably [12, 25, 26, 30].
HbO response curve and brain activation map
In this study, the brain topographical distribution of HbO concentrations was drawn. Two groups (offline and real-time) of data were collected from each subject according to the paradigm shown in Figure 1 and were used to calculate the mean value of HbO concentration during two different actions in the same channel and the mean value of the HbO concentration during the baseline period. Then, the mean value of the HbO concentrations during the two actions was subtracted from the mean value of the HbO concentration during the baseline period to obtain the differential value of the HbO concentration between the two actions. Then, the two channels corresponding to the same channel of all subjects were calculated. The mean value of the total concentration difference of HbO during imagination action was normalized. Finally, the topographic map of the concentration difference of the two imagined actions HbO concentrations in the cortical motor area was drawn. In addition, in order to compare the response of HbO concentrations between the two different imagined actions, the total average HbO concentration curve corresponding to the two imagined tasks of all subjects was drawn.
Statistical analysis
T tests were used to test for differences in the HbO signals in the left and right brain channels during imagined leftward movement involving the right forearm and imagined clenching involving the right hand, and to test the corresponding channel HbO signals during the two imagined movements and the accuracy of the two feature combination classifications in Table 1. T tests were mainly used for normal distributions with small sample sizes and unknown overall standard deviations (σ). We used the ttest2 function in MATLAB to perform t tests.
Accuracy rate of offline and real-time classification of 10 subjects (%).
| Subject no. | Mean + slope (offline) | Mean + slope + correlation coefficient (real-time) |
|---|---|---|
| S1 | 65 | 75 |
| S2 | 62.5 | 72.5 |
| S3 | 57.5 | 70 |
| S4 | 60 | 67.5 |
| S5 | 55 | 57.5 |
| S6 | 65 | 77.5 |
| S7 | 62.5 | 75 |
| S8 | 75 | 80 |
| S9 | 65 | 72.5 |
| S10 | 67.5 | 75 |
| Average | 72.25 ± 0.0031 | 72.25 ± 0.004 |
Results
Table 1 shows the off-line and real-time classification accuracies of the 10 included subjects. The real-time average classification accuracy was 72.25 ± 0.004%, which yielded some degree of differentiation between the two kinds of imagined actions. There was a significant difference between the two classification results, as revealed by a paired t-test (H=1, p=0.004).
Figure 5 shows the brain activation map of the fNIRS signals during the two kinds of imagined motor actions. The data analyzed in this map represent the mean values of the HbO concentrations during the two kinds of imagined motor actions from all subjects. It can be seen from the HbO topographic map in Figure 5 that the activation degree of the left brain motor area was significantly higher than that of the right motor area during the two kinds of imagined actions. In the left motor area the activation degree of the left-movement imagined action was higher than that of clenching. Table 2 shows the test results of the difference between the right brain channel and the left brain channel during the imagined leftward movement of the right forearm. The results showed that 84.44% of the channel HbO signals in the left and right channels were different from each other. Table 3 shows the test results of the HbO signal difference between the right channels and left channels during the imagination of right-hand clenching. The results showed that 84.44% of the channel HbO signals in the left and right channels were different from each other.
Brain activation map of fNIRS signals during imagined leftward movement involving the right forearm, as well as right-hand clenching.
(A) HbO, HbR, and HbT topographic map during imagined leftward movement of the right forearm; (B) HbO, HbR, and HbT topographic map of imagined right-hand clenching.
HbO signal-difference test between right and left channels during imagined leftward movement involving the right forearm (H=1 denotes a significant difference; H=0 denotes no significant difference).
| Right-channel numbers | Left-channel numbers | |||||||
|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | |
| 9 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 0 |
| 10 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 1 |
| 11 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 0 |
| 12 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 |
| 13 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 1 |
| 14 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 15 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 16 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
HbO signal-difference test between right channels and left channels during the imagination of right-hand clenching (H=1 denotes a significant difference; H=0 denotes no significant difference).
| Right-channel numbers | Left-channel numbers | |||||||
|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | |
| 9 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 0 |
| 10 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 |
| 11 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 |
| 12 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 13 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 1 |
| 14 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 15 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 16 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Figure 6 shows the response curve of the HbO concentration during two kinds of imagined movements. The data analyzed in this figure represent the mean values of HbO concentrations during the two imagined actions from all subjects. It can be seen in Figure 6 that the maximum value of the HbO concentration and the time to reach the maximum value were different during the two kinds of imagined actions. Table 4 shows the test results of the HbO signal difference between imagined leftward movement involving the right forearm and imagined right-hand clenching. The results showed that there were significant differences in the HbO signals of all 16 channels during the two kinds of imagined actions.
HbO concentration-response curve of imagined leftward movement involving the right forearm and imagined right-hand clenching. (A) HbO concentration-response curve of imagined leftward movement involving the right forearm. (B) HbO concentration-response of imagined right-hand clenching.
Testing of the HbO signal differences in different channels during imagination of leftward movement involving the right forearm and imagined right-hand clenching (H=1 denotes a significant difference, H=0 denotes no significant difference).
| Channel number | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Result (H) | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Discussion
Traditional BCI research based on fNIRS has mainly focused on off-line analysis of different imagined limb movements using limited classification features. Therefore, the present study focused on real-time classification of two kinds of imagined movements on the same side of a single limb (imagining leftward movement involving the right forearm and right-hand clenching). Additionally, in terms of feature extraction, the correlation coefficients between HbO signals of different imagination tasks were calculated as new classification features and were used for real-time classification. The purpose of this paper is to explore the feasibility of online recognition of different movements of a single limb on the same side based on fNIRS signals.
Under the designed experimental paradigm, the average classification accuracy of the two kinds of imagined movements was 72.25 ± 0.004% by using the combined characteristics of the mean value, slope, and correlation coefficient (Table 1). This result suggests that the use of fNIRS can be used to identify different imagined movements on the same side of a single limb. The accuracy of the classification obtained in the present study is comparable with that of previous studies [11, 12, 36, 37]. However, the experimental paradigm, extracted features, number of classification categories, and classification methods used in previous studies are different from those used in the present study. In addition, the classification accuracy of the present study was also comparable with that reported in previous studies that employed online fNIRS-BCIs [29, 30]. However, the experimental paradigms of these studies (not entirely consisting of imagined movements) were also different from that of our present study.
Traditional fNIRS-BCI studies have mainly extracted the mean value and slope of the HbO signal as the primary feature to identify imagined movements [11, 12, 18, 23, 24]. In the present study, the correlation coefficients of the HbO signals during imagined movements were also extracted as the feature. The average recognition rate of the combination of the mean, slope, and correlation coefficient features (72.25 ± 0.004%) was higher than that of the combination of the mean and slope features (63.5 ± 0.0031%). This result is due to the correlation coefficient being a quantitative description of the correlation between two signals, which is refined into descriptive data. The coefficient not only reflects the relationship between signals but also represents the correlation between signals without the limitation of signal units [38]. Although the mean value feature of the HbO signal was easy to calculate, which can intuitively and concisely represent the centralized trend of a single signal, it makes use of the information of all the data of the signal, and its value is easily affected by the singular data [39]. Meanwhile, the mean value feature of the HbO signal only describes the attributes of the signal itself (i.e., a single trial in a channel only describes one feature). For the slope characteristics of the HbO signal, the slope of a point on the signal curve reflects the speed of the signal change at this point. The slopes of different points are different, so it is difficult to select the slope of an effective point as the classification feature. Generally, the largest value of the slope of all signal sampling points is selected as the classification feature [11, 12, 18]; at the same time, the slope characteristics of the HbO signal only describe its own attributes (i.e., a single trial in a channel is described by only one feature). In comparison, our present study extracted the correlation coefficient of the HbO signals during imagined movements as the classification feature. That is, we calculated the correlation coefficient of each trial (whose category label was known) and other trials (whose category label was known) in the same channel, and then divided all the correlation coefficients calculated into imagined leftward movement involving the right forearm and imagined right-hand clenching according to the type of trial label and other trial label types. Finally, the average of these two kinds of correlation coefficients was calculated (i.e., two statistics were used to represent the trial from this channel). For a single trial in a single channel, the mean value and slope characteristics were only represented by one statistic, while the correlation coefficient characteristics were represented by two statistics. Previous studies have shown that a single feature is not sufficient to fully characterize the signal pattern, and that the corresponding recognition accuracy is not high. However, use of multiple features to describe tasks has been shown to significantly improve the recognition accuracy [40]. The results of this study show that the average recognition rate was increased by 9% by introducing the correlation coefficient feature, which may be due to increasing signal distinguishability.
Comparison of the classification accuracies of each subject in Table 1 revealed that their classification accuracies were different, which may have been caused by individual differences among the included subjects [41]. Differences in head shape and distances between scalps and cortices may represent potential sources of the inter-subject differences in classification accuracies. In addition, differences in hemodynamic responses related to individual imagined movements may also lead to inter-subject differences in classification accuracies. However, none of the subjects’ classification accuracies were lower than the opportunity level. In addition, there can also be intra-individual differences in the classification accuracies in different experiments, which may be attributed to variability in hemodynamic responses related to different imagined actions, and may also be caused by random background activities or other unknown sources [42].
Previous studies have shown that fNIRS can recognize imagined movements of different limbs. For example, Kaiser et al. [14] used fNIRS to recognize imagined movements of the right hand and two feet; Jiao et al. [12] recognized imagined movements of the same movement of the left and right hands; Naseer and Ahikhong [11] successfully distinguished imagined movements of left and right wrist bending. However, the number of instructions provided by the above BCIs based on fNIRS recognition of different imagined movements of limbs has been limited (two to three instructions were usually provided). Recognition of different imagined limb movements on the same side based on fNIRS can increase the number of instructions, but there has been little research in this area. In the present study, we classified two different imagined movements on the same side of a single limb real-time and obtained moderate results (72.25 ± 0.004%). Previous studies have shown that different imagined movements can be transformed into different external device-control instructions [43, 44]. If combined with the recognition of different imagined movements of the same limb as we found in the present study, it is expected to increase the number of instructions of fNIRS-BCIs.
In addition to increasing the number of instructions of fNIRS-BCIs, in order to promote the practical application of this kind of BCI, it is necessary to build an effective online fNIRS-BCI and to train subjects in imagining movements online through neural feedback to further improve the performance of the system. The key technology of the practical application of online fNIRS-BCI is to train the subjects’ motor imagination strategy based on neural feedback to regulate the HbO. Neurofeedback learning not only has the capacity to induce brain plasticity but can also improve the motor skills of subjects [45]. Neurofeedback learning plays an important role in the process of individual growth, such as in children learning how to perform actions. At the same time, neurofeedback is also used for rehabilitation training after motor impairments (such as brain injury and limb injury) in adults. From the perspective of motor-function rehabilitation, online decoding of neural responses induced from imagined movements may be particularly useful. Neural feedback control and rehabilitation neurobiology rely on training or learning to modify the overall function of the nervous system. The nervous system completes corresponding actions through effective feedback, reward, and progressive training [45]. Especially in the context of neural rehabilitation for retraining after impairment in motor function, providing online real-time feedback on the performance of motor imagination can enhance the training effect [46], [47], [48], [49], [50]. The use of fNIRS online feedback may be able to directly reflect the execution of imagined movements by patients. Direct comparison of the actual effect and the target effect may help therapists to evaluate the efficacies of such treatments.
It is an important step to verify the activation area of the brain during imagined movements [23]. Figure 6 shows that the HbO concentration in the left brain was higher than that in the right brain during the imagined leftward movement involving the right forearm and right-hand clenching, which indicates that the imagination of right-hand movement is mainly controlled by the left motor area, and that activation of the left brain causes an increase in HbO concentration. The results of Nishiyori et al. [37] showed that unilateral limb movement is controlled by neurons in the contralateral brain motor area (i.e., unilateral limb movement is opposite to the activated brain motor area). This can also be explained by hemispheric dominance (i.e., activation of the ipsilateral brain motor area is inhibited during unilateral limb movement) [51]. The results of our present study confirm these previous results. Research by Power and Naito et al. [36, 52] showed that the same brain region can decode different imagination tasks. In our present study in Figure 5, two kinds of imagined actions activated the left motor region of the brain. In addition, the activation map in Figure 5 can only provide information about the location and intensity of brain activation and cannot reveal the information about the peak value or peak time of the HbO response [22]. Figure 6 shows that the peak value and the peak time to reach the peak value of the HbO response signal of the different imagined actions were different [53], and the time of leftward movement involving the right forearm and right-hand clenching was between 9–10 s and 8–9 s, respectively. Therefore, the peak value of the HbO signal and the time to reach the peak value can be used to classify different imagined actions.
Finally, it should be noted that the subjects in this study were 20–28 years old, but it has been reported that there is a difference in hemodynamic response between the elderly and the young [54], [55], [56]. In addition, all subjects in this study were healthy, and their hemodynamic responses may be different from those of patients with amyotrophic lateral sclerosis or other neurological diseases [52]. In the future, we will conduct experiments on the elderly and on patients with neurological disease, which will further contribute to the versatile promotion of fNIRS-BCIs. In addition, our present study consisted of a 10 s task period and a 20 s rest period, with periodic alternations. This regular interval may induce physiological noise from respiratory, cardiac, and blood pressure signals. Our future studies need to take these potential caveats into account, from which we plan to adopt irregular intervals to reduce interference from other signals, which may improve classification accuracies and increase the stability of fNIRS-BCIs. In our work the baseline is based on resting state, whereas using opposite side action (left) as the baseline may improve the actual brain activation. Besides, Different baseline return from the previous task-evoked hemodynamic response can make different impact on the subsequent signal. A new method using vector phase analysis approach to detect baseline state was proposed. Significant differences in hemodynamic signal change between the optimal- and suboptimal-baseline blocks detected using the proposed method showed that it can optimize mental workload estimation [57]. Therefore, exploring the method of choosing baseline is the other work we are going to do next.
Conclusions
In the present study, fNIRS was used to identify different imagined movements (leftward movement involving the right forearm and right-hand clenching) of a single limb. The correlation coefficient between different imagination tasks was introduced as a new feature and was combined with traditional mean and slope features. The average classification accuracy was 72.25 ± 0.004%. This result shows that it is feasible to recognize different movements of a single limb real-time based on fNIRS signals. Our findings may provide additional control commands for fNIRS-BCIs and further promote the practical application of this kind of BCI.
Our future studies will focus on the following: (1) to further improve the classification accuracy of the identification of imagined movements to build an online fNIRS-BCI; and (2) to improve the speed and stability of online fNIRS-BCIs based on imagined movements, so as to better control unilateral prostheses for performing multiple actions.
Funding source: National Natural Science Foundation of China
Award Identifier / Grant number: 61763022
Award Identifier / Grant number: 81771926
Award Identifier / Grant number: 82172058
Award Identifier / Grant number: 62006246
-
Research funding: This work was supported by the National Natural Science Foundation of China (Nos. 61763022, 81771926, 82172058, and 62006246).
-
Author contributions: Yunfa Fu supervised all the process from the beginning. Yu Li carried out the data processing, and wrote the first draft of the manuscript. Anmin Gong and Fan Wang verified the data analyses in the revision process. Qian Qian has suggested the theoretical aspects of the study. Lei Su conducted experiments. Lei Zhao corrected the manuscript. All authors read and approved the final manuscript.
-
Competing interests: No conflict of interest exits in the submission of this manuscript, and manuscript is approved by all authors for publication.
-
Informed consent: Before signing the experimental informed consent form, the subjects were informed of the content, nature, and purpose of the experiment, after which they all voluntarily signed the experimental informed consent form.
-
Ethical approval: This study was approved by the medical ethics committee of the Affiliated Hospital of Kunming University of Science and Technology.
References
1. Wolpaw, JR, Birbaumer, N, Mcfarland, DJ, Pfurtscheller, G, Vaughan, TM. Brain-computer interfaces for communication and control. Clin Neurophysiol 2002;113:767–91. https://doi.org/10.1016/s1388-2457(02)00057-3.Search in Google Scholar
2. Hochberg, LR, Bacher, D, Jarosiewicz, B, Masse, NY, Simeral, JD, Vogel, J, et al.. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 2012;485:372–5. https://doi.org/10.1038/nature11076.Search in Google Scholar PubMed PubMed Central
3. Hoshi, Y. Functional near-infrared spectroscopy: current status and future prospects. J Biomed Opt 2007;12:062106. https://doi.org/10.1117/1.2804911.Search in Google Scholar PubMed
4. Wylie, GR, Graber, HL, Voelbel, GT, Kohl, AD, Deluca, J, Pei, Y, et al.. Using co-variations in the Hb signal to detect visual activation: a near infrared spectroscopic imaging study. NeuroImage 2009;47:473–81. https://doi.org/10.1016/j.neuroimage.2009.04.056.Search in Google Scholar PubMed PubMed Central
5. Ghafoor, U, Lee, JH, Hong, KS, Park, SS, Yoo, HR. Effects of acupuncture therapy on MCI patients using functional near-infrared spectroscopy. Front Aging Neurosci 2019;11:237. https://doi.org/10.3389/fnagi.2019.00237.Search in Google Scholar PubMed PubMed Central
6. Hong, KS, Yaqub, MA. Application of functional near-infrared spectroscopy in the healthcare industry. J Innov Opt Health Sci 2019;12. https://doi.org/10.1142/s179354581930012x.Search in Google Scholar
7. Moghimi, S, Kushki, A, Power, S, Guerguerian, AM, Chan, T. Automatic detection of a prefrontal cortical response to emotionally rated music using multi-channel near-infrared spectroscopy. J Neural Eng 2012;9:026022. https://doi.org/10.1088/1741-2560/9/2/026022.Search in Google Scholar PubMed
8. Naseer, N, Hong, KS. fNIRS-based brain-computer interfaces: a review. Front Hum Neurosci 2015;9:3. https://doi.org/10.3389/fnhum.2015.00003.Search in Google Scholar PubMed PubMed Central
9. Jöbsis, FF. Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science 1978;198:1264–7.10.1126/science.929199Search in Google Scholar PubMed
10. Janani, A, Sasikala, M. Evaluation of classification performance of functional near infrared spectroscopy signals during movement execution for developing a brain-computer interface application using optimal channels. J Near Infrared Spectrosc 2018;26:209–21. https://doi.org/10.1177/0967033518787331.Search in Google Scholar
11. Naseer, N, Hong, KS. Classification of functional near-infrared spectroscopy signals corresponding to the right- and left-wrist motor imagery for development of a brain–computer interface. Neurosci Lett 2013;553:84–9. https://doi.org/10.1016/j.neulet.2013.08.021.Search in Google Scholar PubMed
12. Jiao, X, Zhang, Z, Jiang, J, Wang, C, Fu, J. The brain-computer interface using functionnal near-infrared spectroscopy. Shanghai Jiaotong Daxue Xuebao 2017;51:1456–63.Search in Google Scholar
13. Yamamoto, K, Takahashi, H, Sugimachi, T, Nakano, K, Suda, Y. The study of driver’s brain activity and behaviour on DS test using fNIRS. IFAC-PapersOnLine 2019;51:244–9. https://doi.org/10.1016/j.ifacol.2019.01.045.Search in Google Scholar
14. Kaiser, V, Bauernfeind, G, Kreilinger, A, Kaufmann, T, Kübler, A, Neuper, C, et al.. Cortical effects of user training in a motor imagery based brain–computer interface measured by fNIRS and EEG. NeuroImage 2014;85:432–44. https://doi.org/10.1016/j.neuroimage.2013.04.097.Search in Google Scholar PubMed
15. Zhang, S, Zheng, Y, Wang, D, Wang, L, Ma, J, Zhang, J, et al.. Application of a common spatial pattern-based algorithm for an fNIRS-based motor imagery brain-computer interface. Neurosci Lett 2017;655:35–40. https://doi.org/10.1016/j.neulet.2017.06.044.Search in Google Scholar PubMed
16. Olivas-Padilla, BE, Chacon-Murguia, MI. Classification of multiple motor imagery using deep convolutional neural networks and spatial filters. Appl Soft Comput 2019;75:461–72. https://doi.org/10.1016/j.asoc.2018.11.031.Search in Google Scholar
17. Hong, KS, Naseer, N, Kim, YH. Classification of prefrontal and motor cortex signals for three-class fNIRS–BCI. Neurosci Lett 2015;587:87–92. https://doi.org/10.1016/j.neulet.2014.12.029.Search in Google Scholar PubMed
18. Naseer, N, Hong, KS. Decoding answers to four-choice questions using functional near infrared spectroscopy. J Near Infrared Spectrosc 2015;23:23–31. https://doi.org/10.1255/jnirs.1145.Search in Google Scholar
19. Yin, X, Xu, B, Jiang, C, Fu, Y, Wang, Z, Li, H, et al.. NIRS-based classification of clench force and speed motor imagery with the use of empirical mode decomposition for BCI. Med Eng Phys 2015;37:280–6. https://doi.org/10.1016/j.medengphy.2015.01.005.Search in Google Scholar PubMed
20. Al-Quraishi, MS, Elamvazuthi, I, Tang, TB, Al-Qurishi, M, Ebrahim, M. Bimodal data fusion of simultaneous measurements of EEG and fNIRS during lower limb movements. Brain Sci 2021;11:713. https://doi.org/10.3390/brainsci11060713.Search in Google Scholar PubMed PubMed Central
21. Zafar, A, Khan, MJ, Park, J, Hong, KS. Initial-dip based quadcopter control: application to fNIRS-BCI. IFAC-PapersOnLine 2018;51:945–50. https://doi.org/10.1016/j.ifacol.2018.09.072.Search in Google Scholar
22. Hong, KS, Bhutta, MR, Liu, X, Shin, YI. Classification of somatosensory cortex activities using fNIRS. Behav Brain Res 2017;333:225–34. https://doi.org/10.1016/j.bbr.2017.06.034.Search in Google Scholar PubMed
23. Hong, KS, Santosa, H. Decoding four different sound-categories in the auditory cortex using functional near-infrared spectroscopy. Hear Res 2016;333:157–66. https://doi.org/10.1016/j.heares.2016.01.009.Search in Google Scholar PubMed
24. Kirilina, E, Jelzow, A, Heine, A, Niessing, M, Wabnitz, H, Brühl, R, et al.. The physiological origin of task-evoked systemic artefacts in functional near infrared spectroscopy. Neuroimage 2012;61:70–81. https://doi.org/10.1016/j.neuroimage.2012.02.074.Search in Google Scholar PubMed PubMed Central
25. Hu, XS, Hong, KS, Ge, SS. fNIRS-based online deception decoding. J Neural Eng 2012;9:026012. https://doi.org/10.1088/1741-2560/9/2/026012.Search in Google Scholar PubMed
26. Naseer, N, Hong, MJ, Hong, KS. Online binary decision decoding using functional near-infrared spectroscopy for the development of brain-computer interface. Exp Brain Res 2014;232:555–64. https://doi.org/10.1007/s00221-013-3764-1.Search in Google Scholar PubMed
27. Li, Y, Zhang, L, Long, K, Gong, H, Lei, H. Real-time monitoring prefrontal activities during online video game playing by functional near-infrared spectroscopy. J Biophot 2018: e201700308. https://doi.org/10.1002/jbio.201700308.Search in Google Scholar PubMed
28. Çakır, MP, Vural, M, Koç, SÖ, Toktaş, A. Real-time monitoring of cognitive workload of airline pilots in a flight simulator with fNIR optical brain imaging technology. In: International Conference on Augmented Cognition, Toronto, Canada; 2016.10.1007/978-3-319-39955-3_14Search in Google Scholar
29. Khan, MJ, Hong, KS, Bhutta, MR, Naseer, N. FNIRS based dual movement control command generation using prefrontal brain activity. In: 2014 International Conference on Robotics and Emerging Allied Technologies in Engineering (iCREATE), Islamabad, Pakistan; 2014.10.1109/iCREATE.2014.6828373Search in Google Scholar
30. Abibullaev, B, An, J, Lee, SH, Moon, J. Design and evaluation of action observation and motor imagery based BCIs using NIRS. Measurement 2016. https://doi.org/10.1016/j.measurement.2016.12.001.Search in Google Scholar
31. Ung, WC, Funane, T, Katura, T, Sato, H, Tang, TB, Hani, et al.. Effectiveness evaluation of real-time scalp signal separating algorithm on near-infrared spectroscopy neurofeedback. IEEE J Biomed Health 2018;22:1148–56. https://doi.org/10.1109/jbhi.2017.2723024.Search in Google Scholar PubMed
32. Ung, WC, Meriaudeau, F, Kiguchi, M, Tang, TB. Functional near infrared spectroscopy adaptive cognitive training system (FACTS) for cognitive underload and overload prevention: a feasibility study. IEEE Access 2020;8:172939–50. https://doi.org/10.1109/access.2020.3024867.Search in Google Scholar
33. Asgher, U, Ahmad, R, Naseer, N, Ayaz, Y, Amjad, MK. Assessment and classification of mental workload in the prefrontal cortex (PFC) using fixed-value modified beer-lambert law. IEEE Access 2019;7:143250–62. https://doi.org/10.1109/access.2019.2944965.Search in Google Scholar
34. Xu, C, Bray, S, Reiss, AL. Functional near infrared spectroscopy (NIRS) signal improvement based on negative correlation between oxygenated and deoxygenated hemoglobin dynamics. Neuroimage 2010;49:3039–46.10.1016/j.neuroimage.2009.11.050Search in Google Scholar PubMed PubMed Central
35. Swami, A, Jain, R. Scikit-learn: machine learning in Python. J Mach Learn Res 2012;12:2825–30.Search in Google Scholar
36. Power, SD, Kushki, A, Chau, T. Automatic single-trial discrimination of mental arithmetic, mental singing and the no-control state from prefrontal activity: toward a three-state NIRS-BCI. BMC Res Notes 2012;5:141. https://doi.org/10.1186/1756-0500-5-141.Search in Google Scholar PubMed PubMed Central
37. Nishiyori, R, Bisconti, S, Ulrich, B. Motor cortex activity during functional motor skills: an fNIRS study. Brain Topogr 2016;29:42–55. https://doi.org/10.1007/s10548-015-0443-5.Search in Google Scholar PubMed
38. Song, T. Discussions on the functions of the statistics of correlation coefficient and its applications: taking SPSS as analysis tool. Statistical Thinktank 2008:27–31.Search in Google Scholar
39. Chen, Z, Luo, L. Statistics (high-quality course three-dimensional teaching material series). Beijing: Science Press; 2006.Search in Google Scholar
40. Sheng, XV. Research on three-dimensional object recognition [Ph.D. thesis]. Chengdu: University of Electronic Science and Technology of China; 2010.Search in Google Scholar
41. Yarkoni, T, Braver, TS. Cognitive neuroscience approaches to individual differences in working memory and executive control: conceptual and methodological issues. New York: Springer; 2010.10.1007/978-1-4419-1210-7_6Search in Google Scholar
42. Hu, XS, Hong, KS, Ge, SS. Reduction of trial-to-trial variability in functional near-infrared spectroscopy signals by accounting for resting-state functional connectivity. J Biomed Opt 2013;18:017003. https://doi.org/10.1117/1.jbo.18.1.017003.Search in Google Scholar
43. Dura-Bernal, S, Zhou, X, Neymotin, SA, Przekwas, A, Joseph, T, Francis, WWL. Cortical spiking network interfaced with virtual musculoskeletal arm and robotic arm. Front Neurorob 2015. https://doi.org/10.3389/fnbot.2015.00013.Search in Google Scholar PubMed PubMed Central
44. Mehmet, K, Ozcan, GH, Resit, C. Toward building hybrid biological/in silico neural networks for motor neuroprosthetic control. Front Neurorob 2015. https://doi.org/10.3389/fnbot.2015.00008.Search in Google Scholar PubMed PubMed Central
45. Dobkin, BH. Brain–computer interface technology as a tool to augment plasticity and outcomes for neurological rehabilitation. J Physiol 2007;579:637–42. https://doi.org/10.1113/jphysiol.2006.123067.Search in Google Scholar PubMed PubMed Central
46. Buch, E, Weber, C, Cohen, LG, Braun, C, Birbaumer, N. Think to move: a neuromagnetic brain-computer interface (BCI) system for chronic stroke. Stroke 2008;39:910–7. https://doi.org/10.1161/strokeaha.107.505313.Search in Google Scholar
47. Davidson, P, Wolpert, D. Widespread access to predictive models in the motor system: a short review. J Neural Eng 2005;2:S313–9. https://doi.org/10.1088/1741-2560/2/3/s11.Search in Google Scholar PubMed
48. De Vries, S, Mulder, T. Motor imagery and stroke rehabilitation: a critical discussion. J Rehabil Med 2007;39:5–13. https://doi.org/10.2340/16501977-0020.Search in Google Scholar PubMed
49. Diamond, PT. Rehabilitative management of post-stroke visuospatial inattention. Disabil Rehabil 2001;23:407–12. https://doi.org/10.1080/09638280010008834.Search in Google Scholar PubMed
50. Neuper, C, Scherer, R, Wriessnegger, S, Pfurtscheller, G. Motor imagery and action observation: modulation of sensorimotor brain rhythms during mental control of a brain–computer interface. Clin Neurophysiol 2009;120:239–47. https://doi.org/10.1016/j.clinph.2008.11.015.Search in Google Scholar PubMed
51. Vidal, AC, Banca, P, Pascoal, AG, Cordeiro, G, Sargento-Freitas, J, Castelo-Branco, M. Modulation of cortical interhemispheric interactions by motor facilitation or restraint. Neural Plast 2014;2014. https://doi.org/10.1155/2014/210396210396.Search in Google Scholar PubMed PubMed Central
52. Naito, M, Michioka, Y, Ozawa, K, Ito, Y, Kanazawa, T. A communication means for totally locked-in ALS patients based on changes in cerebral blood volume measured with near-infrered light. IEICE T Inf Syst 2007;90:1028–37. https://doi.org/10.1093/ietisy/e90-d.7.1028.Search in Google Scholar
53. Nguyen, HD, Hong, KS, Shin, YI. Bundled-optode method in functional near-infrared spectroscopy. PLoS One 2016;11:e0165146. https://doi.org/10.1371/journal.pone.0165146.Search in Google Scholar PubMed PubMed Central
54. Wang, G, Erpelding, N, Davis, KD. Sex differences in connectivity of the subgenual anterior cingulate cortex. Pain 2014;155:755–63. https://doi.org/10.1016/j.pain.2014.01.005.Search in Google Scholar PubMed
55. Brodoehl, S, Klingner, C, Witte, OW. Age-dependent modulation of the somatosensory network upon eye closure. Behav Brain Res 2016;298:52–6. https://doi.org/10.1016/j.bbr.2015.10.035.Search in Google Scholar PubMed
56. Wang, B, Zhang, M, Bu, L, Xu, L, Wang, W, Li, Z. Posture-related changes in brain functional connectivity as assessed by wavelet phase coherence of NIRS signals in elderly subjects. Behav Brain Res 2016;312:238–45. https://doi.org/10.1016/j.bbr.2016.06.037.Search in Google Scholar PubMed
57. Lim, LG, Wei, CU, Chan, YL, Lu, CK, Tang, TB. Optimizing mental workload estimation by detecting baseline state using vector phase analysis approach. IEEE T Neur Sys Reh 2021;29:597–606. https://doi.org/10.1109/tnsre.2021.3062117.Search in Google Scholar
© 2022 Yunfa Fu et al., published by De Gruyter, Berlin/Boston
This work is licensed under the Creative Commons Attribution 4.0 International License.
