Hand Sign Classification Employing Myoelectric Signals of Forearm

Electromyogram (EMG) signals are generated in muscles, when the muscles contract and a joint is flexed or extended. EMG signals can be measured from a skin surface with noninvasive electrodes, and they include some information on motions such as muscle torque or joint angles. Hence, it is possible to achieve more intuitive human-machine interface using EMG signals than conventional interfaces such as joysticks, data gloves, motion captures. Various interfaces using EMG signals have been proposed to control robot hands (Graupe et al.; Jacobson et al.; Yoshikawa et al., 2009; Ibe at al.). Some methods for hand motion identification have been reported since the 1990s based on soft-computing approaches, e. g. artificial neural networks (Fukuda et al.; Hudgins et al.), fuzzy logic (Karlik & Tokhi; Chan et al.), support vector machine (Yoshikawa et al., 2007; Oskoei & Huosheng), and so on (Chen et al.; Huang et al.). These approaches have improved accuracy of motion discrimination and the number of discriminated motions. However, they need complicated processes and huge amount of calculations.


Introduction
Electromyogram (EMG) signals are generated in muscles, when the muscles contract and a joint is flexed or extended. EMG signals can be measured from a skin surface with noninvasive electrodes, and they include some information on motions such as muscle torque or joint angles. Hence, it is possible to achieve more intuitive human-machine interface using EMG signals than conventional interfaces such as joysticks, data gloves, motion captures. Various interfaces using EMG signals have been proposed to control robot hands (Graupe et al.;Jacobson et al.;Yoshikawa et al., 2009;Ibe at al.). Some methods for hand motion identification have been reported since the 1990s based on soft-computing approaches, e. g. artificial neural networks (Fukuda et al.; Hudgins et al.), fuzzy logic (Karlik & Tokhi; Chan et al.), support vector machine (Yoshikawa et al., 2007;Oskoei & Huosheng), and so on (Chen et al.; Huang et al.). These approaches have improved accuracy of motion discrimination and the number of discriminated motions. However, they need complicated processes and huge amount of calculations.
The purpose of our study is to design an uncomplicated system to identify finger motion and to develop innovative human-machine interfaces. We began with the investigation of the forearm muscle EMG (Tsujimura et al.; Yamamoto et al.). We supposed that not only finger muscles but forearm ones work when the knuckles display hand signs. For this purpose, an EMG measurement system is constructed first to detect surface EMG signals of a forearm and to convert them to more manageable types of features. We next evaluate the correlation between the forearm EMG signals and finger motions. It discloses the activity pattern of each forearm muscle corresponding to specific hand sign. The identification algorithm of hand signs is then designed based on the optimized criterion of muscle activity. Finally, identification of finger gesture is experimented to demonstrate the effectiveness of our proposed method.

Measurement system design
Block diagram of our EMG measurement system is shown in Fig. 1. Surface EMG signals are measured with three electrodes placed on a forearm. The EMG signals are preprocessed and converted into integrated EMG (IEMG) signals through the EMG measurement instrument. An IEMG signal has been used as an index of a muscle activity level in exercise physiology (Milner-Brown & Stein). Both EMG and IEMG are introduced into a PC to evaluate the averaged IEMG (AIEMG) features. An estimation algorithm of finger gesture is installed in the PC. After determining criterions of muscle activity, the proposed system identifies motions of fingers.   Figure 2 illustrates the forearm muscles whose EMG signals are measured by the instrument, and the positions of three electrodes placed on the forearm. The center figure shows forearm cross section of anatomical muscle placement.
Extensor pollicis brevis monitored with the electrode 1 (channel 1) is involved in finger extension. Extensor digitorum monitored with the electrode 2 (channel 2) is also involved in finger extension. Flexor digitorum profundus monitored with the electrode 3 (channel 3) is involved in finger flexion.
The EMG signals are measured with bipolar surface electrodes consisting of two parallel silver bars. These signals are amplified and converted into IEMG signals with rectification smoothing (the cutoff frequency 2.4 Hz) by means of a differential amplifier (Universal-EMG, Oisaka development Ltd.). The EMG signals are sampled at 10 kHz through a 16-bit A/D converter (PCI-3176, Interface Co.) and taken in a data-collection computer (Core i7 2.8 GHz, Windows 7).

Electromyogram features
The AIEMG feature is a periodic average of EMG signals during a designated interval (Yoshikawa et al., 2007). It is extracted from the IEMG in the 100 ms frame, which is shifted for 12.5 ms (80 Hz). Hamming window functions are applied to the signals in each frame. Since the measured data is converted into digital quantity by the PC interface, the AIEMG is calculated in terms of the moving average of the IEMG magnitude as where AIEMG(k), and IEMG(t) represent the AIEMG feature of the k-th averaging frame and the IEMG magnitude of the t-th sample within the frame, respectively. Number of samples in a frame is denoted by N.
The AIEMG eliminates momentary noise such as a spike, because it is a kind of low-pass filters. The larger you take the sampling number, the smoother the AIEMG signal becomes. If you choose N=1, the AIEMG is the same as the IEMG signal.

Forearm EMG signals regarding finger motion
Not only muscles of fingers but of forearms work when you use your fingers. First of all, we have investigated the relationship between finger motion and the forearm EMG signals.
Although we have obtained fundamental responses with regard to each single finger motion, this paper focuses only on typical gestures of composite finger configurations.
"Rock-paper-scissors" is a hand game played by two or more people. Each player changes his hand into one of three basic hand-signs representing rock, paper, or scissors as shown in Fig. 3. Each of the hand signs beats one of the other two, and loses to the other in the game.
Our purpose in this paper is to distinguish the hand-signs by analyzing the forearm EMG signals.

EMG and IEMG signals
We have evaluated the forearm EMG signals with our measurement system. When displaying "rock" by clenching fist, the EMG and IEMG signals were measured as shown in Figs. 4 and 5. Figure 4 (a1), (a2), and (a3) indicate the EMG signals measured with channels 1, 2, and 3, respectively, where the horizontal axis expresses time, and the vertical represents magnitude of the EMG signal. The IEMG signals are also evaluated as shown in Fig. 5. Figure 5 (b1), (b2), and (b3) indicate the IEMG signals measured with channels 1, 2, and 3, respectively, where the horizontal axis expresses time and the vertical represents magnitude of the IEMG signal. Those waveforms shown in Figs. 4 and 5 indicate that channel 1 is inactive, channel 2 is less active, and only channel 3 is active. It can be considered that flexor digitorum profundus is mainly working when you shape "rock" with your hand.
"Scissors" are represented by two fingers extended and separated. The EMG and IEMG signals were measured regarding "scissors" as shown in Figs. 6 and 7, respectively. These figures show that channel 2 is solely active and channels 1 and 3 are almost inactive. Results support that extensor digitorum contributes to showing "scissors." An open hand signifies "paper." The EMG and IEMG signals are shown with regard to "paper" in Figs. 8 and 9, respectively. They indicate both channels 1 and 2 are active and channel 3 is less active. It is surmised that "paper" is formed owing to both extensor pollicis brevis and extensor digitorum. rock scissors paper

AIEMG signals
We next calculated the averaged IEMG (AIEMG) signals from IEMG according to eq. (1). They are stable and noiseless compared to the IEMG, because the feature prevents instantaneous noise of IEMG signals. It is important to adopt the optimum sampling number, N, to obtain the ideal AIEMG feature. When the number is too small, the signal intensity fluctuates and it is difficult to obtain consistent feature at every measurement. If it is too large, the AIEMG signal becomes blunt and thus its original waveform is lost. Figure 10 shows the examples of AIEMG derived from IEMG signals indicated in Fig. 5, with regard to the number of sampling, N as 1, 50, 100, 200, 500, 1000, 2000, and 5000. Figure 10(c1_1), for instance, displays the AIEMG signal measured by channels 1 with sampling number of 1, where the horizontal axis expresses time, and the vertical represents magnitude of the AIEMG signal. Figures 11 and 12 are representations of the AIEMG signals regarding channels 2 and 3, respectively, whose sampling number is 200.

Muscle activity corresponding to finger sign
When evaluating the experimental results, we have Table 1 which indicates contribution of muscles to gesticulation by hands. Extensor pollicis brevis does its part only in displaying "paper" among three signs. Extensor digitorum works when forming "scissors" and "paper." Flexor digitorum profundus contributes only to indication of "rock." This table helps us to classify displayed finger signs based only on the forearm surface EMG signals in real time. If obtaining any of the specific EMG signal combinations shown in the table, we can deduce one of the finger signs among three.
Note that an electrode does not necessarily catch signals only when the corresponding muscle works. Thus, it is necessary to differentiate active signals from inactive to identify muscle motion precisely.

Criterion of muscle activity
We have next investigated an identification of finger signs by analyzing the AIEMG of a forearm. The EMG signals were detected by three electrodes put on the forearm skin of the subjects, and active signals were distinguished from inactive ones according to the following principle. An algorithm for identifying finger motion was designed to refer the active/inactive combination described above.
We have measured the forearm EMG signals in advance to determine the criterion for each muscle to discriminate between active and inactive signals. The criteria were separately settled with regard to several sampling numbers by observing activity of the muscles. The activity is evaluated by the peak voltage of each AIEMG signal.
Ten trials were conducted by gesturing each of the hand shapes for each sampling number.
Experimental results are arranged in Fig. 13. Figure 13(d1_1), for example, indicates the magnitude of 30 AIEMG waveforms detected by channel 1 with the sampling number of 1. The vertical axis represents the peak voltage of the AIEMG signal when the subject made gestures of "rock," "scissors," and "paper." It implys the activity of extensor pollicis brevis. We determined the criterion index, CI1 for channel 1 at N=1 as 0.58 V, which is illustrated by a bold line in the figure.
The activity of muscles can be estimated according to the criteria as follows. Provided that the magnitude of a measured signal is larger than the criterion, the corresponding muscle is presumed to be active. Otherwise it is considered to be inactive.
All the data regarding "rock" and "scissors" were smaller than the line, while those for "paper" were larger in this figure. That is why we could surmise that extensor pollicis brevis is active only for "paper." In the same way, the output AIEMG signals of channels 2 and 3 are arranged as for N=1 in Figs. 11 (d2_1) and (d3_1), respectively. The criterion indices, CI2 and CI3 for channel 2 and 3 were determined as 0.76 and 0.61 V, respectively. By acquired AIEMG signals of extensor digitorum, channel 2 indicated the muscle is active for "scissors" and "paper." Channel 3, representing the activity of flexor digitorum profundus, confirmed that the muscle is active only in the case of "rock." Note that these experimental data support the classification patterns of finger signs shown in Table 1.
With respect to other sampling numbers, similar features were observed as shown in Fig.11(d1_50) -(d3_5000). Their corresponding criteria were determined as shown in Table 2. (4) Figure 13. Discrimination between active and inactive muscle

Finger gesture estimation
A flowchart of the gesture estimation algorithm is shown in Fig. 14. First of all, the AIEMG features are calculated by eq. (1). Next, the peak intensity, AIEMG2 of the AIEMG signal detected by channel 2 is compared with the criterion index, CI2. Then, the intensities, AIEMG3 and AIEMG1 are weighed against the criterion indices, CI3 and CI1, respectively. This process checks the combination of the measured AIEMG signals against the activation patterns of muscles corresponding to the finger signs shown in Table 1.
We finally carried out the experiments of finger sign estimation based on the algorithm. Identification rate was evaluated after 40 trials were conducted for each finger sign. Several sampling numbers for AIEMG feature were investigated. Experimental results are indicated in Table 3 and Fig. 15, which show percentages of correct answers with regard to each sampling number. They suggest that N=200 is the optimum sampling number among our investigations after all. Detailed analysis clarified that larger sampling number, e. g. N=1000, deforms the signals into blunt waveforms and thus it occasionally prevents discriminating between active and inactive signals. On the other hand, the AIEMG signals contain some transient noise when the sampling number is too small. It caused misjudgment on discrimination of activity of extensor pollicis brevis, for instance, that is evaluated with electrode 1 (channel 1). It can be considered to be one of the reasons why the identification rate of "paper" is inferior to the others.   Table 3. Accuracy rate of finger sign identification

Conclusion
We have studied the estimation method of hand signs employing the electromyogram. This paper focused on the forearm EMG signals caused by the finger motion. It relys on the proposition that the specific muscles of forearms work even if you only move your fingers.
First of all, the EMG measurement system was designed to detect signals from the surface skin of forearms. We constructed three-channel myoelectric signal processing system by assigning three forearm muscles; extensor pollicis brevis, extensor digitorum, and flexor digitorum profundus. It provided EMG and IEMG signals, and also calculated AIEMG features.
Fundamental experiments were carried out next to acquire data regarding the relationship between the finger motion and the forearm EMG signals. Investigation on myoelectric responses revealed that the specified forearm muscles were activated with respect to the corresponding finger motion.
The disclosed principles were applied to identification of typical hand signs such as "rock," "paper," and "scissors" in terms of the well-known hand game. We obtained correlative experimental data of hand signs and the myoelectric signals.
We found out the activity pattern of forearm muscles with regard to each hand sign as follows. If shaping "rock," flexor digitorum profundus is mainly working. When "scissors" are indicated, extensor digitorum is activated. Display of "paper" is owing to both extensor pollicis brevis and extensor digitorum.
We established the following principles in consequence to deduce the hand sign from the activities of forearm muscles. Extensor pollicis brevis is active in displaying "paper." Extensor digitorum operates when forming "scissors" and "paper." Flexor digitorum profundus contributes only to indication of "rock." We then designed the classification algorithm based on the results. Because the myoelectric signals fluctuated and depended on the measurement conditions in reality, we determined the criterion of each muscle's activity by statistical treatment and we evaluated the averaged IEMG signals. The AIEMG functioned as a kind of low-pass filters, and its performance was dependent on the sampling number. We investigated the results of AIEMG features and determined the optimum sampling number was 200.
Finally, we conducted some experiments on real-time discrimination of three typical hand signs. The identification accuracy was no less than 97 % with respect to any hand sign when choosing the optimum sampling number.
Experimental results proved that the proper AIEMG feature was successful in inferring the shape of hands. We have confirmed the validity and effectiveness of our proposed estimation system at last. Thus, the method to estimate hand signs has been established based on the activity of forearm muscles instead of finger muscles.

Future directions
Our study was just started with myoelectrical analysis on the forearm muscles and finger motion. Proposed technique was applied to the hand game called "rock-paper-scissors." But it was configured so as to verify the validity of the method rather than to demonstrate its usefulness.
We are planning to improve the technique to human-robot interface system. The advanced input device for computers is one of the applications, which is more intuitive than the dataglove. Such versatile system is neccesary to distinguish many finger motions based on vague myoelectric information.
This paper evaluated the myoelectric signals processed simply by statistic evaluation for noise reduction or feature extraction. Higher performance may be expected by introducing some meta-heuristic method or intelligent method, e. g. neural networks, or suppot vector machine.
The muscle activity was determined on the basis of dualistic taxology according to the criterion established beforehand in this paper. It is necessary to determine the criterion adaptively to the measument conditions to apply to realtime robotic systems.
Practical systems will be realized by integrated measurement method combined with the other perceptual devices.
The proposed technique will be improved not only to the engineering applications but the medical ones, such as the bio-feedback system in rehabilitation or the phyisical support system for handicapped persons.