Long-term upper-extremity prosthetic control using regenerative peripheral nerve interfaces and implanted EMG electrodes

For clarity, participant 1 (P1) and participant 2 (P2), who had transradial amputations, underwent RPNI surgery for the treatment of their neuroma pain and phantom pain. One-year post-RPNI surgery, eight indwelling bipolar electrodes were implanted to record EMG from the RPNIs and several residual muscles. P1 had one RPNI created on each median and ulnar nerve, whereas P2 had one RPNI created on the median nerve and two RPNIs created on the divided ulnar nerve. Electrodes were placed in the following residual muscles: flexor pollicis longus, flexor digitorum profundus (FDP) index finger, FDP small finger, flexor carpi radialis (FCR), extensor digitorum communis (EDC), and extensor pollicis longus (EPL). P2 received electrodes in all residual muscles mentioned above except for FDP small finger to accommodate the additional ulnar RPNI.

Percutaneous connectors were attached to a neural signal processor (NeuroPort, Blackrock Microsystems), which recorded EMG signals at 30 ksps and filtered them between 3 and 7000 Hz (unity gain) for offline analysis. A Matlab target xPC (Mathworks) further filtered the EMG from 100 to 500 Hz, down-sampled the recording to 1 kSps, and decoded EMG into movement commands. EMG SNR from each electrode channel was calculated by taking the root mean square (RMS) of the EMG during volitional phantom finger movements and dividing by the RMS of the electrode's noise floor seen during rest. For P1 and P2, 12 sessions (across 1 year) and 27 sessions (across 3 years) were analyzed for signal quality and SNR.

P2 completed 16 experiment sessions over 604 d to assess the stability of a decoder performance over time. P1 also completed real-time control experiments in a previous study, often using weeks-old calibration data [63]. However, he withdrew from the clinical trial before the structured protocol to assess decoder stability was developed for this study. A Hidden Markov Model-Naïve Bayes (HMM-NB) classifier was trained using six channels (median RPNI, ulnar RPNI 1, ulnar RPNI 2, FDL, FDP index, EDC) to distinguish four functional grips: rest, fist, pinch, and point. FCR and EPL channels were excluded because they did not add information to the four movements. P2 completed one calibration session in a neutral arm position (arm supported on table) to train the HMM-NB, during which she mimicked the virtual display with her phantom hand for five repetitions of each movement (figure 1(a)). The Mean-Absolute-Value (MAV) time domain feature [25] was extracted from each channel in 50 ms non-overlapping time bins during each repetition. The calibration procedure and model training took less than 5 min to complete. The trained HMM-NB model parameters were fixed and reused for subsequent decoding sessions.

Zoom In Zoom Out Reset image size

Briefly, the HMM-NB models dynamic EMG patterns evoked during finger movements as a series of latent states. The HMM-NB can leverage the high-resolution signals from intramuscular electrodes to accurately decode movements with faster processing windows than similar single-state models. Implementation of the HMM used here is detailed in previous work [63]. Decoder performance was quantified using a real-time posture matching discrete task in a virtual environment. A virtual cued hand was displayed, while the participant controlled a second virtual hand to match the cued grip. In these sessions, performance was measured in four arm positions detailed below (figure 1(b)):

Arm at side: arm relaxed in a natural position parallel to trunk;

Arm in front: shoulder flexed 50°, elbow flexed 130°;

Arm across: forearm parallel to floor, across chest such that the prostheses fingers extended 6" beyond the participant's midline;

Arm raised: shoulder flexed 80°, elbow flexed 100°.

P2 used the HMM-NB to directly switch between the three postures and rest. The virtual task required her to hold a cued posture for 1 s continuously, with a timeout period of 5 s, before switching to a new pseudo-randomly cued posture. In each decoding session, she completed 20–30 trials of the virtual task in each arm position, with approximately 3 min of rest in between positions. Researchers used a goniometer to confirm her shoulder and elbow flexion angles for each arm position. Decoder performance was quantified by measuring transition errors between the start of EMG onset and the end of each trial. A transition error was counted if the decoder predicted any grip outside the cued grip during any 50 ms timestep. Decoder accuracy accounts for both the occurrence and duration of transition errors and was calculated as a percentage of correctly classified timesteps out of the total timesteps as follows:

$$\begin{equation*}{A_c} = \frac{{\mathop \sum \nolimits_{i \in {T_c}} \left[ {{x_i} = c} \right]}}{{n\left( {{T_c}} \right)}},\end{equation*}$$

where A_c is the accuracy for movement c, x_i is the logged decoder output at timestep i, and T is the set of timesteps analyzed.

To visualize the EMG features, infomax independent component analysis (ICA) with dimensionality reduction (principal component analysis (PCA)) was used to decompose the six channels into 2D space. The ICA-PCA technique was performed using the function runica() in the EEGLAB package [64, 65] (v4.5b; http://sccn.ucsd.edu/eeglab/).

P2 also completed a bilateral Coffee Task that required use of all four functional grasps mentioned above using an extra small iLimb Quantum™ (Ossur, Reykjavik, Iceland). During the Coffee Task, she used a cup with water (simulated with beads), coffee pod, sugar, and a coffee brewer (Keurig™ mini (Reading, MA)) to simulate brewing a cup of coffee (figure 1(c)). The participant was asked to complete the full task continuously to quantify completion time. She also repeated the task in five segments, in which each segment measured the accuracy of transitioning into a specified grip (e.g. make a fist to pick up the cup with water). P2 repeated each segment five times across five trials with a total maximum number of possible transition errors of 125 (5 segments * 5 repetitions * 5 trials).

P2's controller for the coffee task had an additional hand open movement (hand abduction) that allowed her to open the hand and then switch to a new grip. A grip selection filter and proportional control strategy were adapted from previous work [28, 66]. Designed to prevent sudden movements, which could occur with a discrete controller, the grip selection filter had a 250 ms threshold to actuate a new grip and the proportional controller attenuated her proportional control signal with a 500 ms velocity ramp. Calibration data for the hand open movement was also collected on the same calibration day as the four grips mentioned above. The coffee task was completed 590 and 611 d post-decoder training. In our previous study, P2 was able to use the three-grasp HMM, without an active open, to perform a simple object movement task. However, we concluded that a proportional adjustment of grip aperture would provide better control for a wider range of activities [63].

The same calibration method and MAV feature extraction was used to collect data for the nine-movement offline analysis for five sessions over a span of 276 d for P1 and seven sessions over 463 d for P2. In addition to the HMM-NB, single-state Naïve Bayes (NB) and linear discriminant analysis (LDA) classifiers were trained with Matlab 2021a functions to predict rest, thumb, index, middle, ring, small finger flexion, wrist flexion, finger abduction, and finger adduction. Decoders were trained on the first day of collected data and not updated for subsequent sessions. Accuracy on the first day was measured using leave-one-out cross validation. Decoder accuracy was calculated on a per-trial basis by predicting movement from trial-averaged EMG for NB and LDA, and the mode of the HMM-NB output. This method attempts to ignore transient errors and was chosen to measure algorithm performance in an open loop setting.

To highlight the contribution of RPNIs to movement decoding, a post-hoc analysis compared the performance of movement classifiers with input from only residual muscles to classifiers with input from residual muscles and RPNIs for both P1 and P2. In an offline analysis, LDA classifiers were trained and tested on multiple sessions for three movement sets: the four functional grips used in the arm posture test (n = 3 sessions across 173 d for P1, n = 4 sessions across 288 d for P2), the above nine movement analysis, and six intrinsic movements (n = 4 sessions across 212 d for P1, n = 7 sessions across 463 d for P2). The third movement set focused on distinguishing thumb and finger movements controlled by intrinsic hand muscles (thumb opposition, finger abduction, finger adduction) from thumb flexion, index flexion, and rest. Decoders were retrained for each session, accuracy was calculated as described above, then averaged across all sessions. RPNIs were inferred to be the most valuable for predicting the movements that were most negatively affected by their removal as a decoder input.

Trends in decoder performance and SNR were assessed with a linear regression model. A non-zero slope, i.e. change in SNR or decoder performance over time, was evaluated with an F-test. If a significant change was detected, the sign of the slope then determined an increasing or decreasing linear trend over time. Online decoding accuracy between arm positions used a one-way ANOVA with Bonferroni correction for multiple comparisons. Offline comparison of decoding algorithms used a Wilcoxon rank sum test. All statistical comparisons were analyzed with a significance level of α = 0.05.

To quantify the long-term stability of intramuscular EMG signals over time, participants completed a virtual posture switching task monthly up to 12 and 35 months for participant 1 (P1), and participant 2 (P2), respectively. The SNR of participants P1 and P2's RPNIs remained consistently high, ranging from 15 to 250 across sessions (figure 2(a)). These SNRs were captured up to 276 and 1054 d post-electrode implantation for P1 and P2, respectively. The median (interquartile range; IQR) SNR of RPNIs was 47.61 (103.89) for P1 and 24.49 (18.53) for P2. Comparatively, the median SNR of residual muscles was 83.72 (111.9) for P1 and 25.23 (40.72) for P2. Interestingly, P1's RPNIs had a significant linear increase in SNR amplitude over time (p < 0.05, F-test), whereas P2's RPNIs did not have significant increasing or decreasing linear trends in SNR amplitudes (p = 0.97, 0.12, 0.07 for median RPNI, ulnar RPNI 1, and ulnar RPNI 2, respectively, F-test) across sessions.

Zoom In Zoom Out Reset image size

Though all electrode channels showed high amplitude SNRs on each day, the SNR magnitude varied between sessions. Breaking down the SNR into its numerator (RMS of EMG) and denominator (RMS of electrode noise floor) components, we first investigated if the variability was being driven by the RMS of the electrode noise floor. For both P1 and P2, the RMS of the electrode noise floor across all channels remained stable with a median (IQR) of 1.43 µV (0.44 µV). On the other hand, the EMG RMS range was higher across both RPNIs and residual muscles, with median of 148.2 µV (45.1 µV) for P1 and 39.6 µV (40.5 µV) for P2 (figure 2(b)). Therefore, the variability of SNR was likely driven by the variability of the EMG signal rather than shifts in electrode noise.

In a real-world environment, users prefer not to recalibrate their control system every day to achieve high accuracy. To quantify the long-term prosthesis performance using intramuscular signals, we evaluated our participant's ability to use the same decoder parameters over a period of 16 months. P2 used a four-grip (Rest, Fist, Two-finger pinch, Index finger point) Hidden Markov Model based classifier (HMM-NB) to complete a posture switching virtual task. She completed the task in multiple arm positions without recalibration from the original training session (see methods). Across all sessions, P2 successfully maintained a 1 s hold within a 5-s timeout period on 100% of all trials without recalibration up to 604 d following electrode implantation. Decoder accuracy measured every 50 ms, which accounts for transition errors, remained above 94% with no significant linear decrease in performance across sessions (p = 0.11, F-test) (figure 3). Across different arm positions, there was a significant difference in the occurrence of transition errors per trial between arm at the side and arm raised (p < 0.05, one-way ANOVA with Bonferroni correction for multiple comparisons). The number of transition errors between other postures was not significant (p = 0.052 between arm raised and arm front, p = 0.38 between arm raised and arm across, p = 1.00 between arm to the side, arm front, and arm across).

Zoom In Zoom Out Reset image size

The mean decoder accuracy per timestep across all 16 sessions was 95.6%, 94.1%, 96.3%, and 96.2% for arm at the side, arm raised, arm in front, and arm across, respectively (figure 4(a)). The few grip misclassifications that did occur were between fist and pinch, and fist and point. To better understand the cause for the misclassifications, we decomposed the six-channel EMG inputs into 2D space using ICA with dimensionality reduction using principal component analysis (ICAPCA) to visualize the signal features [69]. This revealed that each grip maintained visually separable clusters across all sessions, but the clusters remained in proximity with one another (figure 4(b)). This analysis is particularly helpful for understanding transition errors for models such as LDA or NB which rely on differences between classes to make decisions. Results also explained why the classifier was able to decode point and pinch with 100% accuracy since they had the greatest cluster separation, but was susceptible to misclassifying between fist and pinch, and fist and point. Confusion between fist and pinch accounted for 41% of transition inaccuracies, and this was most apparent during the arm raised position. Lastly, of the 1673 trials performed, 82.5% of trials had prediction speeds of less than 250 ms from EMG onset, which is well below the 300 ms threshold of perceived delay between muscle activation and prosthetic hand movement [70] (figure 4(c)).

Zoom In Zoom Out Reset image size

Next, we determined whether the high decoder classification accuracy in the virtual task translated to real world applications where the participant moved around a physical workspace wearing her prosthesis. Specifically, P2 completed a physical task using the same four-grip controller described above without recalibration. A grip-sequence based activity of daily living was created to demonstrate robustness and intuitive control of the four-grip classifier. We asked P2 to perform a 'coffee task' that required grip transitions between fist, pinch, point, and open (finger abduction). P2 was able to switch between each grip to smoothly transition from one segment of the task to the next (movie S1). To quantify grip classification accuracy, P2 was asked to attempt each segment of the coffee task up to five times or until she achieved the correct grip. P2 completed the task with 99% accuracy (n = 5 trials, 125 total movements) with only one task error (1 in 125) of dropping the coffee pod. Similarly, grip accuracy was 99% (n = 5 trials, 125 movements) with one grasp error of transitioning into a fist instead of a point. From start to finish, P2 completed the coffee task in 61.3 s ± 6.34 on average (n = 5 trials). The continuous and segmented coffee task sessions were completed 590 and 611 d post-decoder training, respectively.

Lastly, we wanted to determine how well pattern recognition systems performed without recalibration with more than four movements. In an offline analysis, we trained eight additional movements (thumb flexion (T), index finger flexion (I), middle finger flexion (M), ring finger flexion (R), small finger flexion (S), wrist flexion (WF), finger abduction (Ab), finger adduction (Ad)) and we simulated performance across three different classifiers—HMM-NB, single-state NB, and LDA, in both P1 and P2. The average decoding accuracy across all days post-decoder training was 75.9%, 75.2%, and 81.2% (HMM-NB, NB, LDA, respectively; n = 5 sessions) for P1, and 76.5%, 65.9%, and 79.9% for P2 (n = 7 sessions). All decoders except P2's LDA had a small but significant linear decrease in performance across sessions (p < 0.05, F-test) (figure 5(a)). LDA had the highest average performance across days, suggesting that it may be more robust to changes in EMG activation strength, however the difference was not statistically significant (p = 0.220, Wilcoxon rank sum test). Unsurprisingly, these results indicate that predicting an increased number of movements without recalibration may be more challenging. Most classification errors occurred between adjacent postures (abduction and adduction for P1, middle and ring fingers for P2) (figure 5(b)). These movements were either primarily controlled by muscles that were not implanted, or shared ulnar nerve function. Nevertheless, accuracy was greater than 80% for five movements across 276 d for P1, and six movements across 463 d for P2.

Zoom In Zoom Out Reset image size

P1 and P2 both had transradial amputations and electrodes implanted into six and five residual innervated finger and wrist muscles which were used for movement decoding in addition to their RPNIs. To understand the contribution of RPNIs to finger and grasp prediction, we compared the performance of movement classifiers offline to see which movements were most impacted without input from RPNIs. For P1, grip accuracy only decreased by 0.7% when removing RPNI signals, indicating his implanted residual muscles may be sufficient to control those movements (figures 6(a) and (b)). Removing RPNIs decreased overall accuracy by 3.2% for the nine movement dataset (figures 6(c) and (d)), and 9.6% for an intrinsic movement dataset (figures 6(e) and (f)). For P2, grip accuracy decreased by 9.2% (figures 7(a) and (b)), accuracy of the nine movement dataset decreased by 21.3% (figures 7(c) and (d)), and accuracy of the intrinsic movement dataset decreased by 9.2% (figures 7(e) and (f)) when RPNIs were removed. In both patients, RPNIs most strikingly contributed to accurate prediction of thumb opposition while also improving the distinction of finger adduction. For P2, RPNIs also greatly improved prediction of middle, ring, and small finger movements as well as distinguishing fist from pinch. These movements are all either controlled by lost intrinsic hand muscles or extrinsic muscles that were not implanted (for instance, FDP small finger was implanted for P1 but not P2). These results are consistent with recent online control comparisons [71], and indicate that RPNIs provide valuable control signals when muscles are lost due to amputation.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size