Inferring Grasp Intentions from Arm Trajectories via Deep Learning to Enable Functional Movement in Quadriplegia Revised Title Determining Grasp Selection from Arm Trajectories via Deep Learning to Enable Functional Hand Movement in Tetraplegia

Background: Cervical spinal cord injury severely affects grasping ability of its survivors. Fortunately, many individuals with tetraplegia retain residual arm movements that allow them to reach for objects. We propose a wearable technology that utilizes arm movement trajectory information and deep learning methods to determine grasp selection. Furthermore, we combined this approach with neuromuscular stimulation to determine if self-driven functional hand movement could be enabled in spinal cord injury participants. Methods: Two cervical SCI participants performed arbitrary and natural reaching movements toward target objects in three-dimensional space, which were recorded using an inertial sensor worn on their wrist. Time series classiers were trained to recognize the trajectories using either a Dynamic Time Warping (DTW) algorithm or a Long Short-Term Memory (LSTM) recurrent neural network. As an initial proof-of-concept, we demonstrate real-time classication of the arbitrary movements using DTW only (due to its implementation simplicity), which when used in combination with a high density neuromuscular stimulation sleeve with textile electrodes, enabled participants to perform functional grasping. Results: Participants were able to consistently perform arbitrary two-dimensional and three-dimensional arm movements which could be classied with high accuracy. Furthermore, it was found that natural reaching trajectories for two different target objects (requiring two different grasp types) were distinct and also discriminable with high accuracy. In oine comparisons, LSTM (mean accuracies 99%) performed signicantly better than DTW (mean accuracies 86% and 83%) for both arbitrary and natural reaching movements, respectively. Type I and II errors occurred more frequently for DTW (up to 60% and 15%, respectively), whereas it stayed under 5% for LSTM. Also, DTW achieved online accuracy of 79%. Conclusions: We demonstrate the feasibility of utilizing arm trajectory information to determine grasp selection using a wearable inertial sensor along with DTW and deep learning methods. Importantly, this technology can be successfully used to control neuromuscular stimulation and restore functional independence to individuals living with paralysis.


Introduction
In the United States alone, every year there are more than 17,700 new cases of spinal cord injury (1). A majority of these injuries results in incomplete (48%) and complete (12%) tetraplegia, which severely affects arm and hand movements of the survivors and undermines their quality of life. Neuromuscular stimulation offers a viable solution to assist with arm and hand movements to increase independence, but often users nd it challenging to e ciently control such stimulation devices for everyday use. Therefore, several different modalities have been developed to extract user intent for controlling neuromuscular stimulation devices in order to restore grasping. These modalities range from conventional push button or shoulder position control (2,3), to implanted muscle sensors (4), and most recently brain implants (5,6).
Grasping an object is often preceded by reaching for the object. In fact, previous studies have shown that grasping intentions of amputees and able-bodied participants could be inferred from their muscle activity (electromyogram signals) during reaching (7). Tetraplegia is most often caused by damage to the C5 vertebra and importantly, individuals with C5 and below level injury retain su cient control over their deltoid and biceps muscles, which allows them to reach for objects (8,9). Therefore, we proposed to develop a non-invasive approach that can determine grasp choice in tetraplegia from a user's natural reaching and also arbitrary arm trajectories. Unlike previous studies that used multi-channel surface electromyography for classifying reaching movements (7), here we used a low-cost, wearable and easyto-setup inertial sensor. Further, we combined our trajectory recognition algorithms with a custom-built neuromuscular stimulator to determine if functional movement could be achieved in tetraplegic SCI participants.
In recent years, inertial measurement units (IMU) are extensively being used for human computer interactions, particularly for gesture recognition and wearable sensing (10). With advancement in portable computing devices, sophisticated machine learning algorithms such as recurrent neural networks, can be readily deployed for deciphering IMU data (11). For an in-depth review on neuromuscular stimulator and inertial sensing for upper-limb movements, we refer the interested reader to several review papers discussing these technologies (2,(12)(13)(14). In this study, we compared a well-known pattern recognition algorithm called Dynamic Time Warping (DTW) with a recurrent neural network for time series classi cation called Long Short-Term Memory (LSTM). With these methods, we classi ed arbitrary and natural reaching trajectories for different target objects (requiring different grasp types) in threedimensional space (3D). We hypothesized that arbitrary arm movements could be classi ed with high accuracy and that natural reaching movements associated with different target objects, and therefore grasp types, would be distinct and discriminable as well. We further hypothesized that although DTWbased techniques are easily deployable and computationally inexpensive, LSTM networks could outperform DTW within sessions and across multiple days since they are well-suited to classifying timeseries patterns of variable length and lag (15).
In Section 2, methods for the paper describing experimental setup, study protocol, and training of machine learning algorithms are presented. Section 3 presents results from o ine and online validation of the algorithms, based on data from two SCI participants and discusses its signi cance.

A Participants
Two participants with tetraplegia were recruited for the study after providing informed consent. The study protocols were approved by the Institutional Review Board of Northwell Health (Great Neck, NY). Participant 1 was a 32 year old male, injured 6 years prior, with a C4/C5 ASIA (American Spinal Injury Association) B injury. He participated in 10 sessions (2 hours/session), out of which 7 sessions were used to record 2D and 3D arm movement trajectories. During the remaining 3 sessions, grasp selections were decoded online (in real-time) and used to drive a custom neuromuscular stimulator with textilebased electrodes housed in a sleeve (16). This in turn allowed the participant to perform functional movements (e.g. eat a granola bar). Participant 2 was a 28 year old male, injured 10 years prior, with a C4/C5 ASIA A injury. He participated in 3 sessions, which involved 2 training and 1 online testing session.

B Experiment Setup and Data Collection
Participants were seated with their hands initially resting on a table. A wireless sensor module was attached to the wrist of their arm using a Velcro strap (Fig. 1A). While both participants were bilaterally impaired, each still possessed residual movement that allowed reaching with at least one of their arms, which was then used for the study. The sensor module consisted of a 32-bit ARM microcontroller unit (MCU) from Adafruit (Feather Huzzah32) and a Bosch SensorTec BNO055 9-axis IMU. The IMU has a built-in processor and algorithms to estimate its orientation and perform gravity compensation in realtime to produce linear acceleration in three orthogonal directions. Linear acceleration along the X, Y, and Z axes was available externally via an I2C interface. A During the experiments, verbal cues associated with arbitrary (2D & 3D) and natural reaching (3D) movement trajectories were randomly called out to the participant. For the arbitrary movements, the participants were instructed to perform the movements within approximately 1 second and to start from a location that allowed them to complete the movement. It is important to note that for the natural reaching trajectories the participants were instructed to reach for two different target objects on the table (a water bottle and a pen), normally associated with two different grasps (cylindrical and pincer/claw grasp respectively), and to stop just in front of the object. Due to their hand impairment, they could not complete the grasping action, but their natural trajectory information was collected to determine if there were two repeatable and discriminable patterns associated with each target object and associated grasp.
The two different natural reaching (3D) movements were called: 'bottle-reach' and 'pen-reach' (refer to Table 1). Under the arbitrary reaching movement category, four 2D movements and one 3D movement, called corkscrew, were trained. The four 2D trajectories (performed in the horizontal X-Y plane) corresponded to well-known English and Greek letters: S, ε (epsilon or E), γ (gamma), and M. Experiments were conducted in blocks of 18-20 trials and su cient breaks were given between blocks to minimize participant fatigue. Initially, the participants were asked to perform only S and ε trajectories because these were simple to learn and didn't cause fatigue. Later, once the participants became comfortable with moving their arm, we included additional 2D. Thus, in our nal datasets there was a higher percentage of 2D trajectories (especially, S and ε) than the remaining trajectories.
During online (real-time) testing of 2D movement recognition, participants also wore a custom-built fabric sleeve with 128 textile-based electrodes over their forearm to receive neuromuscular stimulation. Neuromuscular stimulation was provided by an 8-channel proprietary, battery-operated, voltage-controlled stimulator. The stimulation parameters were set at 500µs pulse-width, 20Hz pulse frequency, and 0 -110V output. Additional details of the stimulator design will be presented in a forthcoming publication. The stimulation channels were mapped to individual or multiple electrodes on the fabric sleeve, in order to evoke various nger exion and extension type movements. The mapping process itself is described in detail in a previous paper (12). By grouping multiple stimulation channels and sequencing their activation pro le, we could program different hand movements such as opening and closing (and different grasp types).
In a proof-of-concept demonstration in Fig.1C, an SCI participant uses a single 2D trajectory (M) to initiate a neuromuscular stimulation sequence, which allowed the participant to grasp and eat a granola bar, with his paralyzed hand. The stimulation sequence included the appropriate spatial electrode pattern for opening the hand for 5 seconds, followed by the pattern to evoke a cylindrical grasp for 5 seconds to hold the granola bar. Although not implemented here, a second trajectory (e.g. corkscrew) can be used to replace the object back onto the table. The durations for opening and grasping of the hand were selected, to allow enough time for the participant to place his hand around the object and feed himself.

C Data Processing and Machine Learning
The 3-axis linear acceleration obtained from the IMU was band-pass ltered (Butterworth, 8 th order, 0.2 -6Hz) and processed o ine for identifying training samples. The magnitude of the 3-axis acceleration vector was used to identify onset of movement by setting a threshold of 0.95g. The movement onsets were then used to segment the acceleration data over time along the X, Y, and Z axes into windows ranging -0.1s to 0.9s with respect to onset. Each trial was visually con rmed to be free from any noise artifacts or if it exceeded the 1s window and such trials were excluded from further analysis. Next, to determine how discriminative the 2D and 3D trajectories were, two time series classi ers based on either a Dynamic Time Warping (DTW) distance measure or Long Short Term Memory (LSTM) network algorithms were trained separately for 2D and 3D trajectories.
The DTW algorithm optimally aligns a sample trajectory with respect to a previously determined template trajectory such that the Euclidean distance between the two trajectories is minimized. This is achieved by iteratively expanding or shrinking the time axis until an optimal match is obtained. For multivariate data such as acceleration, the algorithm simultaneously minimizes the distance along the different dimensions using dependent time warping (17). In our DTW-based classi er, this algorithm was used to compute the optimal distance between a test sample and pre-de ned templates associated with the 2D and 3D trajectories. Ultimately, the template with the smallest optimal distance to the test sample, was selected as the classi er's output. The template for each trajectory type was chosen as the training sample with the least aggregate DTW score (i.e. sum of individual scores) to every other training sample of the same type.
To implement the LSTM network we used MATLAB R2019b's Deep Learning Toolbox with default values for most parameters. Speci cally, an LSTM network comprising of a single bidirectional layer with 10 hidden units was used. This transformed the 2D or 3D linear acceleration data into inputs for a fully connected layer whose outcome was binary, i.e. 0 or 1. Next, a softmax layer was used to determine the probability of multiple output classes. Finally, the network output mode was set at 'last', so as to generate a decision only after the nal time step has passed. This allowed the LSTM classi er to behave similarly to DTW and classify trajectory windows. During training of the LSTM network weights, an adaptive moment estimation (ADAM) solver was used with a gradient threshold of 1 and maximum number of epochs of 200. Since all the training and validation data were 1 second long, zero padding was not used.
During real-time classi cation of arm trajectories, the linear acceleration signals were ltered and processed in real-time using a MATLAB script that looped at 50Hz. Within the loop, the acceleration data was divided into 1 second long segments with 98% overlap. To demonstrate an initial proof-of-concept, only the DTW-based classi er was implemented, due to its simple and computationally e cient implementation in MATLAB. For online prediction, the incoming acceleration windows were compared with 2D template trajectories of each type and if the optimal distance between trajectories were below 10 units (empirically determined), then positive decision was made. This would then trigger our custom neuromuscular stimulator to perform a complete movement sequence of opening and closing of the hand.

Results And Discussion
Over 250 training samples across 7 movement trajectories were recorded for participant 1 and 96 samples from 5 movement trajectories were recorded for participant 2. Trials with noisy sensor data or incorrect labels were visually identi ed and removed from the training set. Table 1 shows the distribution of samples across different arbitrary and natural reaching trajectories for both the participants. The top row shows a graphical representation of trajectories of each type by reconstructing a participant's hand position in space, which was obtained by double integration of linear acceleration. Given the unequal distribution of samples in our dataset, a 5-fold strati ed cross-validation scheme was selected for evaluating DTW and LSTM based classi ers. Fig. 2 shows the mean ± standard deviation (SD) classi cation accuracy for the 2 participants. In the o ine scenario LSTM performed better than DTW for 2D trajectories, achieving 99 ± 2% (median 100%, IQR 0%) accuracy versus 86 ± 12% (median 89%, IQR 29%), respectively. For 3D trajectories also, LSTM outperformed DTW and obtained 99 ± 2% (median 100%, IQR 0%) accuracy over 83 ± 16% (median 83%, IQR 31%). Using two-sided Wilcoxon signed-rank test, LSTM based classi cation accuracy was signi cantly better than DTW for both 2D (W = 28, p = 0.016) and 3D (W = 21, p = 0.0313) movements. Also shown in Fig. 2, is the online performance of DTW based classi er for 2D arbitrary trajectories. During online classi cation, we either compared between 2 trajectories (e.g. S v/s ε) or between a single trajectory and rest (e.g. M v/s rest) and achieved 79 ± 5% (median 80%, IQR 8%) accuracy. To further evaluate each classi er's performance for type I and II errors, we calculated their cumulative confusion matrices by combining the confusion matrices from each fold per participant.
The resulting confusion matrices for both classi ers and for both types of trajectories are shown in Fig. 3.
For DTW-based classi er, type I error occurred largely for M (60%) trajectory, followed by corkscrew (37.8%) and ε (20.2%) trajectories. In terms of type II errors, DTW-based classi er misclassi ed bottlereach (14.5%), pen-reach (13.8%) and S (10%) trajectories more often as compared to rest of the classes. For LSTM-based classi er the type I and II errors were very low and ranged from 0 -4.5% for all trajectories. In order to test the classi ers' performance across multiple days, we used the existing data for S and ε trajectories from three sessions. Initially, we trained the DTW and LSTM classi ers using only 50% of trials (i.e. 20 trials) from session 1 and then tested them on remaining 50% trials from that session. Next, we used the same model to test trials from sessions 3 and 5 that were recorded several days apart. As shown in Fig. 4, DTW's performance dropped signi cantly with time and was below chance level (50%) by session 5. Whereas, LSTM's performance remained consistently high for all the sessions. This further con rms the superiority of LSTM based classi er over DTW for classifying arm trajectories across multiple days, without requiring frequent recalibrations. Fig. 4. Comparison of LSTM vs. DTW performance across multiple sessions. In this example, data recorded for two 2D trajectories S and ε over three sessions were used for analysis.
A potential limitation of this study is that the LSTM-based classi er has not been validated during online testing. This is still under development and will be reported in a future publication of this study. Nonetheless, LSTM's highly robust o ine performance, suggests that its online performance will be better than DTW's online performance. Another limitation is that a reasonable degree of residual arm movements should be preserved in order for the deep learning algorithms to reliably infer grasp intentions. However, given that most tetraplegics include individuals with C5 and lower level injury that retain su cient arm movements, a majority of SCI survivors will be able to operate this technology. Finally, the current study does not perform an in-depth objective evaluation of grasp performance (e.g. success rate, time to grasp, etc.) or subjective assessment (e.g. ease of use, intuitiveness, etc.) and will be pursued in a follow-up study. Nevertheless, during post-session discussions with the participants, both felt the device was intuitive and easy to learn and utilize.

Conclusions
This study demonstrates the feasibility of utilizing information from arbitrary and natural arm trajectories to determine grasp selection in tetraplegia. Furthermore, it was shown machine learning methods can be used to automatically recognized user selections and initiate neuromuscular stimulator patterns for associated grasp types. This approach has clinical viability and could be deployed in rehabilitation centers for use in not only SCI patients, but also individuals living with paralysis from stroke, multiple sclerosis, traumatic brain injury, or other injuries or diseases. Importantly, the rewarding experience of being able to control your own movements, may lead to increased patient engagement during therapy and ultimately, lead to better motor recovery. and is being conducted in adherence to IRB study # 17-0070. All study participants provided written informed consent prior to the initiation of any research procedures.

Consent for publication
All study participants provided written informed consent for publication as part of their consent for participation in the research study. Additionally, separate audio and visual authorization was provided for the utilization of images and video in publication Availability of data and materials Data from the publication is available from the corresponding author upon request.
Competing interests CEB holds patents in related elds and is the founder of Neuvotion, LLC, a company focused on movement restoration.

Funding
This study was supported by donors and the Feinstein Institutes for Medical Research at Northwell Health.
Authors' contributions CB conceived the study and edited the manuscript; KK and NB designed the software with inputs from CB; NB, KK, and RR performed/assisted with the study; NB performed data analysis and wrote the manuscript. All authors read and approved the nal manuscript.