Adaptive Visuo-haptic Perception through Muscle Coactivation

While the nervous system can coordinate muscles’ activation to shape the mechanical interaction with the environment, it is unclear if and how the arm’s coactivation inﬂuences visuo-haptic perception and motion planning. Here we show that the nervous system can voluntarily coactivate muscles to improve the quality of the haptic percept. Subjects tracked a randomly moving visual target, to which they were physically coupled by a virtual elastic band whose stiffness increased with wrist coactivation. Subjects initially relied on vision alone to track the target, but with practice they learned to combine the visual and haptic percepts in a Bayesian manner to improve their tracking performance. This improvement cannot be explained by the stronger mechanical guidance from the elastic band. These results suggest that with practice the nervous system can learn to integrate a novel haptic percept with vision in an optimal fashion. Post-hoc conﬁrmed the normalized cocontraction in the ﬁrst block across all visual noise levels, but different between the and highest visual levels in block (p 0.02). result came as consequence of a decrease in between pre and training trial < for σ normalized cocontraction a larger margin the noise lower. yielded a cocontraction in the ﬁnal three blocks that increased monotonically with the level of visual noise p < suggests


Introduction
Could the nervous system (NS) adapt the limbs' viscoelasticity to improve haptic perception? Muscles stiffen with activation, and the NS coordinates muscles' activation to shape the mechanical interaction with the environment 1, 2 . However, it is unclear how this affects visuo-haptic perception 3 . It is not known whether the NS would consider this information and adapt the limb's viscoelasticity to plan motion.
Examples of such adaptive sensing in human sensorimotor control include the pupil's dilation to improve visual sensitivity and the influence of γ-motorneuron's on muscle spindle sensitivity to optimise proprioception 4 . In both of these cases sensing can be studied independently from motor actions. In contrast, haptic sensing, synthesizing information from sensors in the skin, muscles and tendons during contact with the environment, cannot be separated from the mechanical interaction. This makes it challenging to investigate the haptic percept and dissociate it from the mechanical effect of adapting the limb's viscoelasticity.
To investigate whether the NS can actively control the viscoelasticity of a limb to improve the haptic percept, we observed how subjects used wrist flexion and extension to track a moving target to which they were physically coupled by an elastic band (Fig.1A). The stiffness of the virtual elastic band increased proportionally to the cocontraction of the wrist antagonistic muscle pair. In order to study the visuo-haptic perception, different amounts of visual noise were imposed on the target on select trials (Fig.1B). If subjects tried to improve the haptic percept by increasing the wrist's viscoelasticity, cocontraction should be greater in trials with higher visual noise. To assess whether adaptive haptic sensing was employed to plan motion, we compared four different models of visuo-haptic perception testing which combination of visual and haptic information could best explain the task performance.

Results
Figs. 2A and 2B show the square root of square error to the target over a trial or tracking error (defined in the Methods through eq.7) and the mean normalised coactivation (eq.5) as a function of the block number, separately for each level of visual noise. The tracking error was large but gradually decreased over blocks ( Fig. 2A). The effect of training was measured by comparing the error in the first and last blocks. A two-way repeated measures ANOVA showed that both the visual noise (p<0.001, F(3,40)=8.67) and the training (p=0.002, F(1,12)=14.34) significantly affected the error. Post-hoc comparisons using Tukey's honestly significant difference (HSD) test confirmed a decrease of tracking error with training for the two higher levels of visual noise (p=0.027 and p<0.001 respectively) with an overall comparable performance for all visual noise levels in the last block (p>0.86).
To understand how muscle activation depends on the level of visual noise and training, we then looked at the mean  cocontraction measured over a whole trial. The cocontraction was normalized to allow comparison between different participants. A two-way repeated measures ANOVA revealed a significant effect of both visual noise level (p<0.001, F(3,40)=8.3) and training (p=0.01, F(1,12)=8.9) on the normalized cocontraction. Post-hoc comparisons confirmed that the normalized cocontraction in the first block was comparable across all visual noise levels, but different between the lowest and highest visual noise levels in the last block (p=0.02). This result came as consequence of a decrease in activation between pre and post training trial (p<0.020 for σ v = 1.01 • ) so that the normalized cocontraction decreased by a larger margin when the visual noise was lower. This yielded a normalized cocontraction in the final three blocks that increased monotonically with the level of visual noise (slope=0.0185±0.002, one-sample t-test t(2)=13.15, p<0.006). This suggests that the NS adapts the body's stiffness to modify the haptic guidance in response to visual noise on the target. The participants improved their performance with training, and also modified the amount of cocontraction depending on the level of visual noise imposed on the target. While learning is known to be generally accompanied by a decrease of muscle cocontraction 5, 6 , this would not explain the distinct cocontraction levels observed at the end of the learning phase for the different visual noise conditions. Why did subjects increase cocontraction as a function of visual noise?
To address this question, we analyzed how the sensory information of the target's motion from vision and haptics was combined by the NS. We considered the tracking performance as a linear function of the amount of visual error e v and haptic error e h : and tested four models expressing different combinations of the visual and haptic information. The visual dominance model expresses that only information of the target's motion from vision is used to track the target, such that the performance is fully explained by the reliability of vision alone (α=1, β = 0). The haptic dominance model expresses the converse, where haptic information alone is used to track the target (α= 0, β = 1). In the averaging model, visual and haptic information are weighted equally to track the target (α =β = 1/2). The Bayesian integration model assumes that a stochastically optimal weighted average is used to minimize the prediction error considering information from vision and haptic sensing: with σ v and σ h the standard deviations of visual and haptic noise, respectively. Each model yielded a different prediction of the task performance for a given visual noise level and cocontraction value. By comparing the predicted task error and the empirically measured error, we could assess which model best explained the sensory combination of vision and haptics. We first analyzed how each model's predicted performance depended on the visual noise condition (Fig.3A). Unsurprisingly, the visual dominance model predicted greater error as visual noise increased. Conversely, the haptic dominance model predicted decreasing error with visual noise since the cocontraction was greater with higher visual noise. The averaging model predicted a roughly constant error irrespective of the visual noise value. The predicted performance from the Bayesian integration model was the nearest to the empirical data. A two way repeated measure ANOVA of the difference between the model and experimental data revealed a significant influence of the different models on the estimation (p<0.001, F(3,56)=5.75), while the effect of visual noise level had a negligible effect. Post-hoc comparisons both confirmed the similarity of the Bayesian integration model predicted with experimental data (p>0.1), and their difference to predictions from the other models (p<0.001).
We then examined how the prediction error (computed as the maximum likelihood estimate of the difference between the task error predicted by each model and the empirical error from the experimental data) changed as a function of the trials' blocks. The visual dominance model yielded the smallest prediction error in the first block (Fig.3B). However, its prediction faltered in later blocks. The haptic dominance model yielded the worst prediction of all four models, with the prediction error growing in later blocks. The averaging model layed in between the visual and haptic dominance models, which indicates that the information from the two sensory channels was not weighted equally by the NS. The Bayesian integration model was the best model in all blocks but the first. This suggests that subjects may have relied mainly on vision in the first block to track the target, but gradually learned to integrate the haptic percept from the second block onwards to improve performance.

Discussion
The field of morphological computation has analysed how animals' biomechanical design facilitates their functions 7,8 , and human motor control has revealed how the NS controls the limbs' viscoelasticity to shape the mechanical interaction with the environment 1, 2, 9 . Different from these morphological and motor adaptations, our results provide evidence that the NS actively controls the limbs' viscoelasticity to improve perception. Our results showed that muscle coactivation was adapted to increase the accuracy of visuo-haptic perception by considering the reliability of visual and haptic information. While previous studies have provided qualitative evidence of adaptive sensing in human motor control 4, 10, 11 , we could quantify the performance improvement resulting from the increase in muscle cocontraction. This enabled us to identify the stochastically optimal mechanism used by the NS to regulate cocontraction and improve visuo-haptic perception.
The association of various sensory modalities and their stochastically optimal integration has been reported in numerous studies (see 3,12 ), where the reliability of each sensory input was manipulated externally by the experimenter. In contrast, the reliability of the haptic information in our experiment was controlled by the participants themselves. Importantly, while the subjects could have coactivated their wrist indiscriminately to yield superior performance through haptic guidance to the target, they only increased coactivation to maintain the same performance level independently on the visual noise level. This points to a trade-off between task performance and effort, a common metric used to explain motor behaviors from reaching to interaction tasks [13][14][15] .
The time course of the prediction of each model offers some insight into the learning process. While the visual dominance model was superior in the first block, it gradually diverged in proceeding blocks, suggesting that vision was used predominantly only early on. The Bayesian integration model was best from the second block onwards, suggesting that the NS had learned to adapt the wrist's cocontraction to sense the haptic percept relating to the target's trajectory and integrate it with vision. This requires the NS to understand the relationship between the muscle cocontraction and the mechanical guidance to the target. At this point, it is unclear how the brain models or learns this relationship. While some recent studies have examined the role of cocontraction and its effects on motor learning 16,17 , questions remain concerning the NS' ability to recognize sources of additional sensory information pertaining to a task, how it decides to integrate it with existing sensory channels, and how this 4/8 decision-making process influences motion planning and the formation of motor memories.

Experimental apparatus
The experiments described below were approved by the Joint Research Compliance Office at Imperial College London and performed in accordance with the Declaration of Helsinki's ethical standards. Participants without known sensorimotor impairments were recruited to take part in the main experiment and in the two control experiments. Each participant gave written informed consent prior to participation. 30/31 participants were right-handed, as was assessed using the Edinburgh Handedness Inventory 18 .
Each participant was seated comfortably on a height-adjustable chair next to the Hi5 robotic interface 19 , and held a handle with their dominant wrist. A screen placed in front of the participant provided visual feedback of the task with a cursor indicating the current wrist position (Fig.1A). The Hi5 handle is connected to a current-controlled DC motor (MSS8, Mavilor) that can exert torques of up to 15 Nm, and was equipped with a differential encoder (RI 58-O, Hengstler) to measure the wrist angle and a force sensor (TRT-100, Transducer Technologies) to measure the exerted torque in the range [0,11.29] Nm. The Hi5 system is controlled at 1kHz using Labview Real-Time v14.0 (National Instruments) and a data acquisition board (DAQ-PCI-6221, National Instruments), while data was recorded at 100 Hz.
The activation of two antagonist wrist muscles, the flexor carpi radialis (FCR) and extensor carpi radialis longus (ECRL), were recorded with surface electrodes using a medically certified non-invasive electromyography system (g.LADYBird + g.GAMMABox + g.BSamp, g.Tec). The raw muscle activity was high-pass filtered at 100 Hz, rectified, then low-pass filtered at 10 Hz using a second-order Butterworth filter to yield the filtered muscle activity.

Normalized coactivation
Every experiment started with an EMG normalization to map the raw muscle activity (in mV) to a corresponding torque value (in Nm). The subject placed their wrist in the most comfortable middle posture, which was set to 0 • . Constrained at that posture, they were then instructed to sequentially (i) flex, or extend the wrist to exert a torque, or (ii) maximally co-contract in order to keep the wrist position stable during a 3 Hz sinusoidal positional disturbance of 10 • amplitude. Each phase was 4 s long with a 5 s rest period between consecutive contraction phases to avoid fatigue, which was used as a reference activity in the relaxed condition. This was repeated four times at flexion/extension torque levels {1,2,3,4} Nm and {-1,-2,-3,-4} Nm, respectively. For each subject, the recorded muscle activity was linearly regressed against the torque values to estimate the relationship between them. The torque of the flexor muscle was modelled from the filtered EMG signal u f as and similarly for the torque of the extensor muscle τ e . Muscle cocontraction was then computed as The average coactivation over all participants (as shown in Figs.2B,D) was computed from each participant's normalised coactivation, calculated as with u min and u max the minimum and maximum of the means of all trials of the specific participant.

Tracking with visual and haptic feedback
In the main experiment, subjects were required to track a visual target (in degrees) moving with q * (t) ≡ 18.5 sin πt 1.547 sin πt 2.875 , 0 ≤ t ≤ 20 s using flexion-extension movements (Fig.1A). After each 20 s long trial, the target disappeared and the participant was required to place the hand on the starting position at the center of the screen. The next trial then started after a 5 s rest period and a 3 s countdown. Each subject was instructed to take small breaks when feeling (mental or physical) fatigue during the course of the experiment. The tracking error , T ≡ 20 s (7)

5/8
was displayed at the end of each trial, where q(t) is the angle of the wrist. The tracking task consisted of a free phase (in which no interaction torque was exerted on the wrist) to get the subject accustomed with the Hi5 interface, followed by an interaction phase in which the subject's wrist position was connected to a haptic reference trajectory q h (t) with an elastic force where the connection stiffness κ< 0.25 Nm / • linearly increased with the cocontraction u(t).
Subjects were informed of the possibility to regulate the coupling stiffness by co-contracting or relaxing wrist's muscles, and of the transition of free trials to interaction a phase of interaction trials. They were instructed not to resist large torques provided by the motor.

Main experiment
15 right-handed subjects (7 females, aged 23.46±2.39 years old) carried out the main experiment. The target trajectory was displayed on the screen either as a 8 mm diameter circle or as a "cloud" of eight normally distributed dots around the target (Fig.1A), depending on the experimental condition. The cloud of dots were defined by three parameters, randomly picked from independent Gaussian distributions: the vertical distance to the target position η ∈ N(0, 15 mm), the angular distance to the target position η q ∈ N(0, 0<σ v <8.00 • ), and the angular velocity ηq ∈ N(0, 4 • /s). The amplitude of visual noise was controlled by the angular distance deviation, while both the vertical and the angular velocity deviations were kept constant. The dots were updated sequentially so that each dot was replaced every 100 ms. The experimental protocol consisted of 10 free trials followed by 32 interaction trials split into 8 blocks. The 4 trials of each block used a different value of σ v ∈ {1.01 • , 3.34 • , 5.67 • , 8.00 • } presented in a random order in each block (Fig.1B). We assume that the ordering of the blocks has no effect. In the interaction trials q h ≡ q * thus providing the subjects with an unbiased haptic reference of the visual target.    (Fig.4A, 20 ) and decreased with learning, yielding consistent performance improvement among the visual noise conditions (p=6.60e-05 for initial-last trials, paired t-test). This was modelled as

Control experiments
where α v = −11.67, β v = 0.13 were identified from a least-squares linear regression with data from the last two trials. Next, a haptic control experiment was performed to measure how the tracking error depended on the elasticity of the virtual band. In this control experiment, no visual feedback was provided while the wrist was connected to the target reference trajectory with 7 selected levels of elasticity. Another eight participants (7/8 right-handed, 26.75±1.28 years old, including 4 females) participated in this experiment. The experiment was structured in 5 blocks of 7 elasticity κ ∈ {0.011, 0.016, 0.025, 0.037, 0.055, 0.081, 0.120} Nm/ • presented in random order. In the interaction phase totalling 35 trials, the subjects experienced an elastic force to the target. As expected, the tracking error decreased with increasing elasticity (Fig.4B), which was modelled as a quadratic function e(κ) = α h + β h log 10 (κ) + γ h log 10 (κ) 2 (10) where α h = 4.42, β h = −7.59, γ h = 4.15 were identified from a least-square fit. This relation between tracking error and coupling stiffness did not change with learning.

Statistical modeling
In order to compute eq.(2), we need to calculate the standard deviation due to the connection compliance similarly to the visual noise. This is done by equalizing the error function due to visual noise eq.(9) with the tracking error due to compliance eq.(10): e(κ) ≡ e(σ v ), yielding for ith subject and jth visual noise level. In eq.(1) the visual error e v was generated from a Gaussian distribution with σ v ∈{1.01 • , 3.34 • , 5.67 • , 8.00 • } around the average error of the last two blocks (see Fig.4A).