Muscle Synergy and Musculoskeletal Model-Based Continuous Multi-Dimensional Estimation of Wrist and Hand Motions

In this study, seven-channel electromyography signal-based two-dimensional wrist joint movement estimation with and without handgrip motions was carried out. Electromyography signals were analyzed using the synergy-based linear regression model and musculoskeletal model; they were subsequently compared with respect to single and combined wrist joint movements and handgrip. Using each one of wrist motion and grip trial as a training set, the synergy-based linear regression model exhibited a statistically significant performance with 0.7891 ± 0.0844 Pearson correlation coefficient (r) value in two-dimensional wrist motion estimation compared with 0.7608 ±  0.1037 r value of the musculoskeletal model. Estimates on the grip force produced 0.8463 ± 0.0503 r value with 0.2559 ± 0.1397 normalized root-mean-square error of the wrist motion range. This continuous wrist and handgrip estimation can be considered when electromyography-based multi-dimensional input signals in the prosthesis, virtual interface, and rehabilitation are needed.


Introduction
Owing to advances in surface electromyography (EMG) signal-based models and algorithms, numerous techniques have been proposed for prosthesis controls and clinical controllers. ough in a discreet fashion, several studies had attempted to convert hand motion into input signals to control prosthetic machines [1], virtual hands [2], and exoskeletons [3], with the aim of estimating both the wrist motion and hand gesture. Nishikawa et al. [1], Sebelius et al. [2], and Kita et al. [4] classified several gestures, such as hand gestures and wrist motions, using algorithms like machine learning, Gaussian mixture models (GMMs), and other linear classifiers (e.g., k-NN and Bayes).
Continuous estimations are applied in response to feedback from real users who require various movements suitable for daily life [5]. Vogel et al. [6] used standard supervised machine learning algorithms to create a mapping between arm/forearm muscle activities and 6-dimensional (6D) position/orientation; this has extended the four rotational degree-of-freedom (DOF) models for the joints of the shoulder and elbow [7]. An algorithm for simultaneous estimation of the three DOFs of the wrist was also proposed [8]; it showed promise of applicability to unilateral amputees by employing a bilateral mirror-training strategy [9]. However, these continuous estimations did not consider the combined motions of the wrist and fingers.
Under the flexor muscles are the multiple finger muscles that lie deep inside the forearm [10]. Many researchers use their expertise to minimize the interference of surface EMG (sEMG), without fully solving crosstalk. For this reason, the sEMG electrode is not free from the inclusion of both external and internal muscle signals. ese mixed signals can influence the estimates obtained via the other.
Muscle synergy is defined as a set of muscles recruited by a neural command [11]. A muscle synergy generates a primitive motion, and complex motions are produced by the combination of several synergies [12,13]. Real-time classification for upper limb motion was conducted using a machine learning technique [14]. However, synergies were differentially weighted according to task constraints [15]; therefore, in this study, two different synergy calculations were attempts: deriving wrist and grip synergies simultaneously and deriving each synergy separately. Besides, synergy model performance with the change of synergy was analyzed, and the choice of number of wrist synergy was checked.
e musculoskeletal model (MSM) is a second-order computational motor control model with nonlinear dynamics. It estimates a one-degree-of-freedom joint angle for flexion and extension considering muscle elasticity and viscosity [16]. Kawase et al. [17] developed a simplified computational model that investigated the estimation of three different joint angles (i.e., elbow, wrist, and finger) with a little influence between finger joint and wrist position estimation.
To realize the prosthetic hand for daily use, wrist motion and grip motion have to be controlled simultaneously; however, few papers treat this problem [17] because of the crosstalk of muscle activation measurement.
is study aims to estimate wrist motion with and without grip motion and compared the estimation performance between optimized MSM and synergy models.

Materials and Methods
2.1. Subjects. Ten healthy subjects (males, aged 28.0 ± 5.7, 9 right-handed, 1 left-handed, none ambidextrous) participated in the experiment. ey did not have a history of any form of neurological disorder. ey used their dominant hand (either left or right hand) during the conduct of this experiment.

Experimental Protocol.
e study protocol was approved by the ethics committee of the Tokyo Institute of Technology (2014042) and was carried out in accordance with the Declaration of Helsinki. Written consent was obtained from each subject before the experiment. Table 1 indicates the muscle groups chosen to estimate the wrist and grip movements. Five muscles are associated with wrist motion (i.e., ECR, ECU, FCU, FCR, and APL) and two with grip action (i.e., FDS and FDP). Previous wrist-based experiments analyzed ECR, ECU, FCU, and FCR [17,18], which are the flexor and extensor muscles of the wrist with different deviations (radial and ulnar). In addition to these muscles, in particular APL, an extensor of the thumb was included to trace the radial movement of the wrist. e FDS and FDP-the flexor muscles of the finger-were included to estimate the grip force with a synergy-based model. Figure 1 shows the placement of the EMG sensor on the forearm. e EMG signals were measured using Trigno ™ EMG system. In the experiment, two tasks were conducted. e first task was a wrist motion, which measured motion in different movement conditions. For the second task, isometric grip force was measured in different grip force levels. us, the trials were divided into two tasks to check the wrist movement at a certain grip condition and grip force at a certain posture.
In the first task, EMG was measured using wireless Trigno ™ EMG system sensors, and wrist joint angles were measured using the IM sensors of the system. Figure 2 shows the placement of the IM sensors, which were attached to the back of the hand and the back of the forearm; they were attached to detect the relative wrist joint angle from the forearm. Subjects placed their forearms on the table fastened by a wrist binder. ereafter, they performed four wrist motions: flexion, extension, radial deviation, and ulnar deviation. ese motions were conducted while the hand was free (no gripping action) and in gripping mode (normal strength). Wrist motions were conducted under three conditions based on the subject's comfort: comfortable maximum limit (with and without grip), half of comfortable maximum limit (without grip only), and stiffened movement with force exertion (without grip only). e subjects performed each motion three times per trial. ree trials were conducted for each condition. ereafter, the gripping action (without wrist motion) was conducted in the center position (Figure 2) for which the subjects performed strong grips and weak grips. ree trials were conducted for the gripping experiment.  In the second task, EMG and grip force were measured, with the latter done using ReachMAN robot [19]. e subjects adjusted the angle of the grip to their best fit while maintaining a center position posture, as demonstrated in Figure 2, after which the grip force was measured. ree levels of grip strength were performed: strongest, half, and a quarter of gripping power. e strongest grip force (in newton N) varied for every subject with an average of 16.2 ± 3.2 N.

Data Acquisition.
e data were sampled separately per signal category using lab streaming layer (LSL) in MATLAB 2018b program base [20]. e EMG signals were sampled at 2000 Hz, IMU sensors at 74 Hz, and ReachMAN force sensor at 100 Hz. e seven EMG signal channels were filtered and normalized before computing the synergy set. e EMG signals were rectified and filtered using a second-order Butterworth low-pass filtering with 5 Hz cutoff frequency [21]. Figure 3 shows the conversion of an EMG signal. e filtered EMG signals are called "quasi-tension" because it showed a high correlation between the joint torques of its muscle [21].
A recurring issue during experiments and analyses was the fact that the magnitude of EMG signals for each channel had to be changed every time the sensor was detached then attached again. To resolve this, all signals were normalized by the peak activation level of the whole task, including a range of joint angles and maximum effort of trials [22]. Normalization was performed after quasi-tension signal filtering. In this experiment, the combined hand motion tasks, co-activating both grip and wrist motions, were chosen; hence, the normalized quasi-tension signals, which were obtained by filtering and normalizing the EMG signals, were resampled into the other sampling rates and measured together.

Wrist Angle Derivation.
e Madgwick IMU algorithm was implemented to estimate the two-dimensional wrist joint angle [23]. IMU sensors were placed on the back of the hand and forearm to track the orientational difference between the hand and forearm.
Subjects performed self-paced movements without visual feedback; consequently, most of them performed diagonal movements even if only vertical and horizontal movements had been requested. To compensate for this, the two angles obtained by the IMU algorithm were normalized by each angle's absolute maximum value; the sum and difference obtained can be seen in Figure 4. In each model, these calculations were estimated and the summation was recalculated to estimate the angle; furthermore, considering the EMG crosstalk error and wide range of wrist angle movements, a comfortable maximum limit trial was mainly used as the train data.

Synergy-Based Linear Regression Model.
A synergybased linear regression model was used to estimate wrist and grip values. To reduce computational costs in a model calculation, a simplified version of the nonnegative matrix method, i.e., the hierarchical alternating least square (HALS) method, was used [24]. Apart from the computational cost, HALS also has a wide capability: it can work with a large number of components [24], in contrast to the canonical NMF method [25], which is only applicable if the number of the sources is greater than the number of components; it can work in conditions where the number of components is large [24]. is feature of HALS is appropriate when multiple hand gestures need to be applied. Hence, this computation method is valid even when the number of combined synergy set exceeds the number of measured EMG signals. e HALS decomposes the normalized quasi-tension as follows: where E is the normalized quasi-tension signals in an m by n matrix with m being the number of time series and n the number of EMG channel inputs; S � [s 1 , . . . , s j ] is the synergy set, where j is the number of synergies and s j � [c 1 , . . . c n ] T representing a single set of synergy, where c n is the coactivation coefficient of EMG n; and furthermore, M is the coactivation coefficients of the synergy in m by j matrix: When the synergy model is derived, learning algorithm procedures are used to iterate (2) and (3) several times, where k (1, 2, . . ., j) denotes the label of synergies. e matrices S and M were computed using one set of single wrist motion data and single grip motion data.
In the analysis, wrist synergies with varying numbers from one to six were calculated from a wrist movement trial to confirm the validity number of synergies; thereafter, a single grip synergy was taken from a grip trial. More often  than note, the variance account for (VAF) became the standard means of choosing the muscle synergy number [26][27][28]. In the same context, this study applies the number of synergies that matches over 0.9 VAF to all the subjects to ensure the synergy model consistency. e gains of the wrist motion synergy for the angles were derived using linear regression to compute the normalized sum and difference of the wrist angle (flexion-extension, radial-ulnar deviation) θ i from the following equation: where a 0,i denotes the angle bias, a j,i s are the regression coefficients for each synergy coefficient m j , and ε denotes the random noise error. A combined synergy set with regression coefficients were used to estimate both grip motion and wrist motion task; hence, the synergies and gains from a combination of two trials were applied to all other tasks. e computation of the wrist and grip synergies was conducted in two different ways. A facial image study showed that NMF learns the object in part-based representation [29]. In the case of grip motion, the muscles of all channels work together; therefore, multitrial-based muscle synergy was calculated in two ways: simultaneously from jointed wrist and grip trials (SLRM1) and separately per trial (SLRM2). e synergy sets were derived from comfortable maximum limit trials and grip trials.

Musculoskeletal Model (MSM).
e musculoskeletal model was used to compare the angle estimates of the synergy-based model. e MSM succeeded in estimating the  joint angles of the elbow, wrist, and index finger with little influence from a change in wrist position [17]. e performances of SLRM and MSM were compared with each other to ascertain how good that of SLRM is; furthermore, the train set of MSM was taken from a comfortable maximum limit, which are the same trials used in SLRM. To optimize the MSM performance, MSM was derived from two different muscle numbers, namely, MSMS1 and MSMS2; the former used all measured muscles, while the latter used five wrist muscles. Kawase et al. constructed a one-degree-of-freedom model per joint [17]; to fit the model into this experiment, two wrist joint angles were converted as depicted in Figure 4.

Statistical
Analysis. An exhaustive cross-validation was used to test the performance of each model per subject, with indices used to estimate performance. e Pearson correlation coefficient (r) and normalized root-mean-square error (nRMSE) are defined as follows: where n is the number of samples, y is a reference, x is an estimate, and a is defined as the normalization coefficient. nRMSE chooses a to be 90, the limit of the wrist angle range. All statistical analyses were conducted using t test2 function of MATLAB 2018b.

Synergy Number
Optimization. e reproducibility check of SLRM1 and SLRM2 in the different number of synergies was tested in VAF. e number of grip synergies was fixed to one, both in SLRM1 and SLRM2 to ensure that SLRM1 computed the synergy one more from the joint trials. Wrist motion (average)  e nRMSE of the grip motion implies a statistical significance (p < 0.001, Student's t-test). From the results, MSM2 was chosen as the representative MSM model. e time series of the wrist angle in two dimensions is shown in Figure 6. Subjects were asked to rotate their wrists in four directions. ey were able to move freely at their own pace, moving in an inclined diagonal direction at different angles. Because of this tendency, both models appear to have the underlying assumption that a subject moved in a diagonal direction even if they performed a gradual movement, as shown in Figure 6(c). Angle Y corresponds to radial-ulnar deviation having radial deviation as positive. e blue-colored line represents the IMUreference angle derived from two IMU sensors by differentiating relative orientation in the Euler angle. e red-colored estimate is a musculoskeletal model-(MSM-) based estimation having 5 input signals. Yellow-colored estimate stands for synergy-based linear regression model-(SLRM-) based estimation deriving synergy derived separately per trial. An example of (a) a comfortable maximum limit trial, (b) half of a comfortable maximum trial, (c) a stiffened movement trial, (d) a grip-trial having twelve times gripping, and (e) combined movement of a comfortable maximum limit with grip.   Tables 2 and 3. Wrist motion performances r on average are 0.7891 ± 0.0844 in SLRM and 0.7608 ± 0.1037 in MSM, implying a statistically significant difference (p < 0.001, Student's t-test). Similarly, nRMSE also shows a significant difference between SLRM and MSM (p < 0.01, Student's t-test). is trend continued during wrist motion trials without a grip (comfortable maximum limit trial, comfortable half limit trial, and stiffened movement trial).
However, when grip motion was added, there was no statistically significant difference in r; however, differences were still apparent in nRMSE. e r of the comfortable maximum with grip trials were 0.7562 ± 0.0631 in SLRM and 0.7579 ± 0.0877 in MSM, and nRMSEs were 0.1654 ± 0.0.271 in SLRM and 0.1804 ± 0.0511 in MSM (p < 0.001, Student's t-test). Finally, in the grip trial, where r measurement was inappropriate because the wrist motion in the trial is just an indication of a perturbation, here, SLRM had 0.1458 ± 0.0251 and MSM had 0.1864 ± 0.0770 in nRMSE, implying a statistically significant difference (p < 0.001, Student's t-test).
e stiffened movement trials also had the same trend in r values, being 0.8406 ± 0.0344 in SLRM and 0.8142 ± 0.0589  Figure 7 shows the SLRMbased time series grip force estimate and resulting angle estimate perturbation. Subjects were constrained to a grip device during the task to ensure that there was no actual wrist motion during the entirety of task. Hence, wrist motion estimation during grip motion was checked for strong distortion in angle estimate. e results showed that during the gripping task, instability of the wrist angle estimation occurred in the presence of a strong force activation, as may be seen in Figure 7(d). For the half and quarter grip force task, angle estimation was less than 30°, as shown in Figures 7(e) and 7(f ). e r value for grip force estimate and nRMSE of the X-Y angle estimate, compared with zero angle (no movement), were computed as indicated in Table 4. Sub8 data were omitted in this task because the EMG signal was saturated during the ADC converting process using NIDAQ (±5 voltage). e SLRM-based gripforce estimate from nine subjects was 0.8463 ± 0.0503 in r with 0.2559 ± 0.1397 nRMSE in wrist movement estimation.

Discussion
is study tested both MSM and SLRM in two different conditions to optimize each model. Figure 9 shows the VAF from SLRM1 and SLRM2 with the different number of wrist motion synergies. e VAF of SLRM1 converged in three wrist motion synergies and overfitted thereafter. is trend was the same as obtained in other studies whether in patients or healthy subjects [26][27][28]. For SLRM2, the VAF converged  in four wrist motion synergies and overfitted up to five. ere was also a decrease in VAF in SLRM2. d' Avella et al. suggested that when the number of extracted synergies is greater than the generator synergies combination, each additional synergy captures an equal amount of noise-generated variation [30]. In that sense, fifth and sixth muscle synergies are noise-derived synergies. is study measured five wrist motion muscles and two finger motion muscles; thus, the sixth muscle synergy in SLRM2 formulated synergy set with finger motion muscles, which made reproducibility impossible. Such noise-derived synergies contaminated the estimation performance. e wrist joint angle estimation performance of SLRM2 with varying numbers of wrist synergies are shown in Figure 10. e highest performance was obtained in four wrist motion synergies; the additional number of synergies deteriorates the estimation performance.
In the SLRM, wrist movement estimation performance showed statistical significance depending on the synergy extraction method both in r and nRMSE. is study aims to use NMF for prosthetic and interface purposes; therefore, synergies were modulated per subject and trial. Separate synergy sets preserve multi-EMG coactivation in the grip synergy, which enables synergy to cluster movement type. In a joint-trial-based synergy set, the NMF divided multi-EMG signals into several part-based groups, resulting in tying the grip synergy to be a combination of EMG signals not used in other synergies. erefore, the joint-trial-based synergy set could not discriminate movement type. is tendency can be seen in Figure 11. Synergy 5 had the most distinctive shape, representing grip synergy. In a time-domain reaction, SLRM1 responded to grip activation regardless of grip force. In SLRM2, it also increased other synergy coefficients albeit, relatively, by a small amount when a strong grip was assumed. is result could support extracting a unique synergy for each motion [31].     e FDS and FDP are flexor muscles of the fingers from an anatomical point of view. ose two muscles are separated by synergy analysis, which can be observed from Synergy 5, as shown in Figure 10. For the wrist motion with and without grip task, the other five muscles are appropriate to estimate joint angles. For the MSM, the minimum number of muscles exhibited good performance. e estimation performance between SLRM and MSM had a statistically significant difference in the entire wristonly movement task both in r and nRMSE values; however, there were no other r values in wrist motion with grip. e results show the robustness of the wrist movements with respect to finger movement of the MSM, which is the same as those obtained by Kawase et al.'s experiment [17]. e estimation performance of SLRM also showed comparable  Figure 11: Time series synergy coefficient data with corresponding synergy matrix in the synergy-based linear regression model in the grip task. Synergies one to four represent wrist synergy, and synergy five represents grip synergy. e purple-colored region represents threetime strong grip actions and the green-colored region represents three-time weak grip actions. performance with previous results in the literature on trajectory [6] and joint force [9] estimation. e advantage of SLRM is that it estimates not only continuous wrist movements but also complex movements in addition to grip motion. In this experiment, we confirmed that the estimation performance of SLRM for complex motion was equivalent to MSM performance.
Within the wrist-only motion trials, both SLRM and MSM had the lowest performance in comfortable half performance.
is is most likely due to the nonlinearity between EMG signals and arm motion or contamination of movement artifact and baseline noise to the EMG (representatively, Sub3). erefore, the nonlinear regression techniques used in the previous studies [6,9] could be comparable with the synergy-based model having an alternative to using linear regression. e linear envelope filtering used in the EMG signal analysis was proven to have highly correlated signals with joint torque induced by the target muscle [21]. Grip synergy, which is a coactivation of these filtered EMG signals, also showed a high grip force estimation performance of 0.8463 ± 0.0503 without resort to further conversion or regression techniques. However, in the strong grip trial, it was confirmed that the wrist angle estimation was distorted by gripping EMG signals. is strong grip distortion indicates the necessity to investigate the limits of the SLRM in grip force estimation. Within the current experiment results, it is difficult to determine whether each subject distorted the wrist angle estimation with a similar absolute force level or with a specific ratio of the maximum force.

Conclusions
In this study, we explored a model for estimating wrist joint angle with and without grip action. In the first task, in which we examined wrist angle estimation of SLRM and MSM, the SLRM exhibited a relatively higher performance in wrist motion. In the grip task, SLRM showed robustness in angle estimation when the grip force is half or a quarter of its maximum force level. In addition, SLRM can provide the extent of grip force exerted in the center position with little perturbation. ese characteristics of SLRM are useful for combined wrist and grip action; however, obtaining limiting grip force required to crash the wrist angle estimation, and vice versa, was beyond the scope of this study. Further studies are required to obtain the simultaneous estimation of both parameters necessary for daily usage.
Data Availability e experimental results from the individual participants are provided as a table within the article. Data of the raw EMG, IMU, and force sensors are available upon request to the corresponding author.

Conflicts of Interest
e authors declare that there are no conflicts of interest regarding the publication of this paper.