Parietal neural prosthetic control of a computer cursor in a graphical-user-interface task

Objective. To date, the majority of Brain–Machine Interfaces have been used to perform simple tasks with sequences of individual targets in otherwise blank environments. In this study we developed a more practical and clinically relevant task that approximated modern computers and graphical user interfaces (GUIs). This task could be problematic given the known sensitivity of areas typically used for BMIs to visual stimuli, eye movements, decision-making, and attentional control. Consequently, we sought to assess the effect of a complex, GUI-like task on the quality of neural decoding. Approach. A male rhesus macaque monkey was implanted with two 96-channel electrode arrays in area 5d of the superior parietal lobule. The animal was trained to perform a GUI-like ‘Face in a Crowd’ task on a computer screen that required selecting one cued, icon-like, face image from a group of alternatives (the ‘Crowd’) using a neurally controlled cursor. We assessed whether the crowd affected decodes of intended cursor movements by comparing it to a ‘Crowd Off’ condition in which only the matching target appeared without alternatives. We also examined if training a neural decoder with the Crowd On rather than Off had any effect on subsequent decode quality. Main results. Despite the additional demands of working with the Crowd On, the animal was able to robustly perform the task under Brain Control. The presence of the crowd did not itself affect decode quality. Training the decoder with the Crowd On relative to Off had no negative influence on subsequent decoding performance. Additionally, the subject was able to gaze around freely without influencing cursor position. Significance. Our results demonstrate that area 5d recordings can be used for decoding in a complex, GUI-like task with free gaze. Thus, this area is a promising source of signals for neural prosthetics that utilize computing devices with GUI interfaces, e.g. personal computers, mobile devices, and tablet computers.

On pages 8 and 9 of this article the words 'trials' and 'trial' were incorrectly published as 'trails' and 'trail' due to editorial errors.
The corrected sentences are given below. Page 6: The fraction of trials that were completed successfully did not significantly differ between Manual and Brain Control blocks (p = 0.75).
Page 8/9: One possible explanation for this outcome is the increased difficulty and imperfect accuracy of Brain Control, i.e. on trials where cursor control is worse, the subject would need to gaze at the cursor longer to maintain closed-loop control. An alternative explanation is extraneous saccades reduced decode accuracy, and the subject learned to make fewer saccades to maintain cursor control. To distinguish these possibilities, we computed the correlation between numbers of saccades to peripheral targets (as above) in each trial to the TTA across all n = 1718 trials. A correlation of r = −0.12 (p < 10e−7, T test) supports the former account and rules out the latter. Trials with many saccades to locations not occupied by the cursor were amongst the shortest, while the longest trials involved prolonged periods of gazing at the cursor, presumably to accommodate feedback control.

Introduction
Neural prosthetics hold great promise for allowing disabled individuals to regain agency over their environment by directly manipulating robotic limbs or computer interfaces. When tested in the laboratory, computer or motor-based interfaces tend to only involve series of individual targets (Velliste et al 2008, Collinger et al 2012, Gilja et al 2012, Hauschild et al 2012. When multiple targets have been used, the effects of the greater task complexity themselves were not evaluated (O'Doherty et al 2011, Ifft et al 2013 or the cognitive component of the tasks have been minor (Hochberg et al 2006, Shanechi et al 2012, limiting clinical usefulness. These studies do not replicate the function of a modern GUI interface, i.e. selecting a remembered or desired target from a group of alternatives. Such function would be clinically relevant and useful to patients with motor deficits. To this end, we designed a task for NHPs that incorporated these behavioral elements and assessed its effect on an interface driven by neural activity in the posterior parietal cortex (PPC). The 'Face in a Crowd' task required selecting a single, icon-like stimulus, the 'face', from a group, the 'crowd'. The correct target was indicated by an initial sample face stimulus. The targets were refined so as to naturally require a visual search of several saccades to locate the matching stimulus without imposing any artificial constraints on eye movements, i.e. during free gaze. After visually locating the matching target, it was selected by manipulating a manually or neurally controlled computer cursor. This task created a NHP analog to human use of a GUI. In the 'Crowd Off' task condition, No Crowd appeared, reducing behavior to a traditional center-out task.
Increasing task complexity from one to many possible targets may seem like a simple change, but the associated cognitive and behavioral requirements are not: more eye movements to and between visual stimuli, more complex decision making, and greater demands on working memory and attention. Prosthetics driven by neural activity in motor cortex may be influenced by these variables, as motor (Rao and Donoghue 2014) and premotor areas (Pesaran et al 2008) exhibit strong transients to the onset of visual stimuli. PPC, which has also been used to drive cortical prosthetics (Musallam et al 2004, Mulliken et al 2008, Hauschild et al 2012, Ifft et al 2013 in NHPs, is well known to be sensitive to many of the these behavioral variables (Colby and Goldberg 1999, Buneo and Andersen 2006, Pesaran et al 2010, Louie et al 2011. It is therefore relevant to determine if these added task demands and their neural correlates interfere with the signals upon which a useful neural prosthetic would depend.
Area 5d, a subregion of PPC, was chosen as the substrate for Brain Control due to its selectivity for arm kinematics (Crammond and Kalaska 1989;Graziano et al 2000, Bremner and Andersen 2012, Cui and Andersen 2012. Neural decoders were repeatedly trained to transform neural activity from this region into cursor commands during both the Crowd On and Crowd Off task conditions to determine whether the crowd's presence during training or thereafter adversely affected decoding performance. Some of the behavioral variables mentioned above, e.g. attention and working memory, are difficult to measure directly. Eye movements are closely related to them (Soto et al 2005) and much more readily recorded. Therefore, we examined eye movements during the various phases of the Face in a Crowd task as well as during a saccade-only task to (a) assess the degree of eye tuning in the recorded population of neurons, (b) ensure the Face in a Crowd task required a visual search, and (c) examine whether task performance under Brain Control was impaired as a result. Furthermore, we sought to determine if cursor movement under Brain Control could be dissociated from eye movements as during natural hand eye coordination.

Methods
A male rhesus monkey participated in this study. All procedures were approved by the California Institute of Technology Institutional Animal Care and Use Committee and were performed in accordance with NIH guidelines.

Behavioral setup
The monkey was seated in a chair and viewed all visual stimuli on a vertical LCD monitor placed about 40 cm from the eyes. The NHP's head was held in place by a surgically implanted headpost. When Brain Control was performed, both arms were gently restrained to prevent large arm movements. Eye position was recorded using the ISCAN system (ISCAN, Woburn, MA). Hand position was tracked at 120 Hz with a magnetic six DOF trakStar sensor (Ascension Technology Corporation, Milton, VT) affixed to the hand. View of the hand was blocked by an opaque plate placed at neck height. Stimulus presentation was performed with the PsychoPy psychophysics library for Python (Peirce 2007). Task control and recordings were performed with the Simulink real-time system (The MathWorks, Boston, MA).

Neural recordings
The monkey was implanted with two 96-channel electrode Cereport arrays (Blackrock Microsystems, Salt Lake City, UT) on the convexity of the superior parietal lobule near the posterior half of the IPS, i.e. the approximate location of neurons functionally ascribed to area 5d in previous studies (Cui andAndersen 2012, Bremner and. The Cereport (formerly known as the 'Utah' array) has been commonly used in human neuroprosthetic studies and was thus used in the present study to more closely mimic clinical techniques for recording extracellular potentials (Rothschild 2010). Neural activity was amplified, digitized, and recorded with the Cerebus neural signal processor. In the Central software suite (Blackrock Microsystems), thresholds for action potential detection for each channel were set at − 4.5 times the root-mean-square of the raw signal sampled over a 1 s window on a daily basis. In real-time, the time of threshold crossings were transmitted to Matlab software (The Mathworks, Boston, MA) and counted in non-overlapping, 50 ms time bins. No spike sorting was used, as spike sorting itself presents a significant difficulty to maintain from day to day in human trials (Franke et al 2012), and has been reported to confer little benefit upon BMI performance (Fraser et al 2009). From the two arrays combined, approximately 105 active channels were reliably recorded with spiking activity of some kind as judged by the experimenter. Active channels that met the simple criterion of firing at an average rate of 1 crossing per second during the Training Block were used in online decoding. This resulted in 85 ± 2 channels being used for decoding each day.

Behavioral task
A green cursor of radius 0.7 cm was continuously presented on the screen. The cursor was controlled either by the monkey's hand moving in the horizontal plane above a flat, tabletop surface immediately in front of his body (Manual Control mode), or by the output of a neural decoder (Brain Control mode) with hands gently restrained on the table surface in a relaxed position with elbows bent at approximately 90°. The purpose of the task was to create an animal paradigm that mimics human use of a GUI. The Crowd task naturally required a period of visual search for a cued stimulus via repeated saccades followed by a cursor movement to, and selection of, the chosen target. We chose visual stimuli/targets consisting of images of various human faces taken from the Psychological Image Collection at Stirling (PICS) database (http://pics.stir.ac.uk). Face targets consisted of a photographic head-on image of one of three facial expressions of 12 individuals. One individual was chosen for use as the 'goal' face or individual for the current study. All faces were normalized for size with a red surrounding mask that obscured the overall shape of the head and hair. The faces were also normalized for total brightness. These manipulations made the stimuli subtle enough in their differences that they required fixation for correct identification of the goal individual. The goal individual's expression varied from trial to trial but not within a trial. The outer diameter of all the face stimuli, red mask included, was 3 cm. Acceptance windows for all targets and the cursor were identical in size to their respective visual representations.
A trial began when a sample face cue (Target 1) of the goal individual appeared at the center of the screen (figure 1). The subject moved the cursor to overlap the cue for a contiguous Hold Period of 400 ms. If overlap was broken during the Hold Period before 400 ms elapsed, an entire new 400 ms Hold Period would need to be performed. This rule was applied for all Hold Periods in the task. For the Crowd On condition, after the Hold Period, Target 1 disappeared, and a 'crowd' of face stimuli of eight individuals appeared. One of the eight faces in the crowd (Target 2) was an identical match to the initial cue face, Target 1. Each face in the crowd was situated on a circle of radius 9 cm centered on the middle of the screen and separated by 45°on the circle (figure 1). The monkey then had 20 s to locate the matching face and move the cursor to overlap it for another Hold Period of 400 ms. After this second Hold Period, a juice reward was delivered via a tube placed in front of the monkey's mouth. Simultaneously, all targets disappeared and a reward beep was sounded. A new trial began after an intertrial interval (ITI) of 0.5 s. Failure to locate, select, and Hold Target 2 within the 20 s period resulted in termination of the trial: the disappearance of all targets, an auditory cue signifying trial failure, and a penalty ITI of 5-10 s. Overlap with an incorrect target for 400 ms or more also resulted in termination of the Figure 1. Face in the Crowd task trial structure: the timeline pictured schematizes the phases of the task and associated events. The start of each phase of the task is marked with a tick, labeled, and pictured above with a screenshot of the task display. The behavioral measures used and their corresponding temporal extents are also indicated below the timeline. Target 1 is the Cue Face, and Target 2 is the Match Face. In the Crowd On condition, Target 2 is accompanied by seven other faces of different individuals. In Crowd Off, it appears alone. The green dot represents the cursor.
trial. An overlap of less than 400 ms with an incorrect Target in the crowd was permitted. The cursor was continuously controlled during the trials and ITI. In the 'No Crowd' task condition, Target 2 appeared somewhere on the same circle described above, but with no other face stimuli present (figure 1).

Performance measures
Task performance was assessed by the fraction of trials successfully completed and by measuring the time required to perform the various stages of the task (figure 1). Time to Acquire, or TTA, spanned the time between Target 2 onset (with or without the crowd) and initial contact with Target 2. This period included the time required to visually locate the matching face whether the crowd was present or not. The time from initial acquisition of Target 2 to Reward, or Time to Hold (TTH), measured how long the subject took to 'settle' the cursor down on the Target. Time to Hold could be no shorter than 400 ms, but could be longer if overlap of the cursor and Target 2 was broken and reestablished before completing the trial. Time to Reward, or TTR, captured the time from Target 2 onset to Reward and so would be the sum of TTA and TTH for a given trial.
In order to assess the effect of Brain Control on cursor control without the influence of task difficulty, we calculated the change in Time to Acquire, or ΔTTA, by subtracting average daily TTA in Manual Control from each subsequent Brain Control trial TTA, i.e.
where superscripts MC or BC indicate Manual Control or Brain Control and k indicates Brain Control trial k. The average TTA in Manual Control, TTA MC , was computed per day and per task condition (Crowd On or Off) and was only subtracted from Brain Control trials with the corresponding task condition on the same day. This calculation isolated the difference in TTA that was attributable solely to the use of Brain Control rather than Manual Control by eliminating time consumed by other aspects of the task, e.g. searching for and reacting to the presence of the correct target. This calculation thereby gave a direct indication of the effectiveness of Brain Control of the cursor independent of the influence of other, task-related factors. This measure was then examined as a function of assessment condition and training condition (described below) in subsequent analyses. P values reported are the result of a non-parametric, twosample Kolmogorov-Smirnov test for differences in distributions. Where reported, interquartile range (IQR) was computed by taking the difference between the third quartile (Q3) and the first quartile (Q1) of the data. Reported Rsquared values were computed by taking the square of the Pearson's correlation coefficient (R) between the variables specified.

Decoder training
Each day began with the monkey performing 160 trials of both the Crowd and No Crowd task conditions alternating every 20 trials under Manual Control. This allowed assessment of daily variation in basic task performance without the influence of Brain Control quality. Next, the monkey's hands were gently restrained.
A previously computed neural decoder, a Training Decoder, was used by the NHP to manipulate the cursor during an initial 250 s Training Block in either the Crowd On or Crowd Off condition. The Training Decoder and computer assistance functioned like a set of 'training wheels' on a bicycle, allowing the NHP to use neural activity to drive the cursor, though not fully independently.
The Training Decoder was computed in a previous behavioral session using the same methods described here. The task used during training of the Training Decoder was the Crowd Off task condition of the Face in a Crowd task. We attempted to use the same Training Decoder for every Training Block in the current study in order to keep initial conditions for each Training Block as similar as possible; however, after one to four days, Training Decoders stopped generating useful output even with substantial assistance during the Training. When that occurred, the most recently computed decoder (trained with the Crowd Off) was substituted in as a Training Decoder. The dataset for the current study spanned seven days and 23 decoders. The decoders trained in the first four days all used the same Training Decoder during training. The next Training Decoder was used for two days, and the third for one day. All analyses described below were repeated on a restricted data set using only decoders trained with the first Training Decoder (days 1-4). The results of those analyses did not differ substantially from the results described below.
Furthermore, using a Training Decoder (itself trained on the Crowd Off task condition) to compute a new decoder with the Crowd On task condition could be considered a sort of worst-cast scenario in which the task type changes from one Training Block to the next. We reasoned that if we find no impairment of decoding function caused by the 'switch' to the Crowd On condition, there is no reason to expect an impairment would arise if the two tasks were fully segregated with respect to training and assessment. It would be feasible that it might provide an advantage, but our main goal in the study was to examine whether or not these task contexts reduced decode performance.
During the Training Block, output of the Training Decoder was assisted by removing some fraction of the error in cursor movement in each time bin. Error was defined as the component of the instantaneous movement vector that did not point directly at the Target. When no target was present on the screen, any movement of the cursor was considered error. Typically, the assistance level was adjusted such that 30% of the error was removed in each time bin. During Training, the ITI was set to 0 s. The neural activity and cursor kinematics during the Training Block were subsequently used as input to compute a new decoder.
When computing the new decoder after performing the Training Block, the noisy velocities of the cursor during the Training Block were reoriented to point towards the instantaneous goal to more accurately capture the assumed intentions of the subject (Gilja et al 2012). For each time bin, the intention of the NHP was assumed to be either move the cursor towards the current correct face target or hold the cursor steady if the cursor already overlapped the correct target. Reaction times were accounted for by assuming an intention to hold the cursor steady until 200 ms after Target 2 onset in the No Crowd condition and after initial cue onset and 400 ms after onset of Target 2 and the other targets in the Crowd condition. These values were chosen based on average reaction times during Manual Control.
Once the new decoder was computed, it was then used for Brain Control by the NHP without assistance to perform the task in both Crowd On and Off conditions in ten alternating blocks of 20 trials, yielding a total of 200 assessment trials per decoder. This process of training and assessment of performance was repeated so that the effect on performance of the crowd (during Training and/or Decoding) could be measured. Twenty-three decoders were trained and assessed across seven days. The task condition of the first Training Block on a given day was alternated to remove any order effects.

Decoder calculation
For transforming neural activity into cursor position and velocity, we used a linear decoding model coupled with a linear state space model of the cursor dynamics. The final decoder form closely resembled that described by Gilja and colleagues (Gilja et al 2012).

Saccade task
To assess the correlation between eye kinematics and neural activity, the monkey was trained to perform a task in which a trial consisted of repeated fixation on a series of four yellow circular targets placed on a 2 × 2 equally spaced grid measuring 14 cm square. After fixating a target for the required period of 500 ms, the target would turn from yellow to gray. After successfully fixating on all four targets in any sequence, the targets would all disappear and juice reward would be delivered. An ITI of 0.5 s followed. The position of the hand (which was not required to perform the task) was recorded along with neural signals during this task, though the hand rarely moved.
Two days, each consisting of approximately 1500 trials, were recorded. R-squared values between (a) linear predictions of eye kinematics based on neuronal firing with (b) actual eye kinematics were computed and validated using Leave One Out Cross-Validation (LOOCV) on 20 equally sized segments of the data.
Additionally, segments of neural data were used to decode the spatial locations of the endpoints of saccades during this task. Saccades that began on one of the four targets and ended on any of the other three targets were preselected from the data. Each saccade was labeled by the target at which the saccade ended. Observations were comprised of total number of spikes summed for each neural channel across a window beginning 0.150 s before a saccade onset and 0.300 s after. Linear discriminant analysis was used to classify the neural data into one of four possible targets/ categories. LOOCV was used to obtain a measure of the performance of neural classification of saccade targets, whereby all observations save one were used as training data. The class of the 'left out' trial was then predicted using the classifier. This was repeated using each available trial as the excluded trial. Performance was computed as the percentage of 'left out' trials that were correctly classified. A permutation test, whereby target labels were randomly shuffled and LOOCV repeated, was used to generate a null distribution of performance in order to assess whether classification of the actual data exceeded chance levels (n = 10 3 permutations).

Saccade task
Ideally, the neurons recorded would not at all be sensitive to eye movements. However, area 5d neurons show some eye position tuning (Bremner and Andersen 2012). To quantify the degree of eye position tuning in the population recorded for the current study, we recorded neural activity while the NHP performed a task involving saccades only. Cross-validated R-squared values between neural activity and eye movements were computed. Though highly significant for x and y position (p x = 1.5e −4 and p y = 6.4e −3 ), R-squared values of 0.05 and 0.02, respectively, were obtained, indicating a measurable but small relationship with eye position. P values were not significant (≫0.05) for eye velocity.
When trying to decode the goal of individual saccades from amongst the four possible targets in the saccade task based only on neural data, 41.54% correctness was achieved for n = 674 saccades. Though modest, this performance significantly exceeded chance level of 25% (p < 10 −5 , permutation test). Thus, the neural activity correlated with saccades could not account for the 98% success rate achieved with the neural cursor during the Face in the Crowd task.

Behavioral task-Manual Control
During the Manual Control block of each day, the monkey was able to successfully complete >99% of the trials correctly, i.e. selecting the correct face before time ran out. This performance indicated that the animal had no difficulty in reliably finding the matching Face in the Crowd. Basic task performance statistics under Manual Control for all days (Crowd On n = 567 trials, Crowd Off n = 560 trials) revealed the desired effect (figure 2). As expected, the presence of the Crowd significantly increased TTA (Crowd On Median = 0.82, IQR = 0.32, Crowd Off Median = 0.48, IQR = 0.11, p < 10e −16 ) and TTR (Crowd On Median = 1.23, IQR = 0.34, Crowd Off Median = 0.89, IQR = 0.12, p <10e −16 ), but not TTH (Crowd On Median = 0.40, IQR = 1.4e −14 , Crowd Off Median = 0.40, IQR = 2.8e −14 , p = 0.08). This is because the Crowd causes a visual search which delays acquisition time, but not the time required to hold the target once it has been initially contacted. The delay in acquisition time of course results in slower overall task performance as measured by TTR.
To verify that TTA was increased because the crowd required the animal to search and identify the target face, we examined eye behavior prior to movement onset and between movement onset and target acquisition for each day. Eye positions for each trial were rotated to place Target 2 at the 3 o'clock position. Two-dimensional histograms of eye position for each task condition clearly reveal the monkey's tendency to visually search though the faces in the Crowd On task condition before initiating his hand movement ( figure 3). Histograms for the period between movement onset and Target 2 acquisition looked similar, indicating that the monkey continued scanning the faces even during and after movement to select the correct target ( figure 4, left panel).

Behavioral task-Brain Control
During the Brain Control sessions, the monkey was able to successfully complete > 98% of the trials correctly, i.e. selecting the correct face before time ran out (video 1). The fraction of trails that were completed successfully did not significantly differ between Manual and Brain Control blocks (p = 0.75).
Each decoder was trained during a Training Block either with the Crowd On or Crowd Off and then assessed for performance with the Crowd On or Off. This comprised a 2 × 2 factorial design with the 'main effects' being training condition and assessment condition.
The task condition in which the decoders were trained, the training condition, did not significantly influence the TTA achieved (Crowd On Median = 1.65, IQR = 1.00, n = 1508 trials, Crowd Off Median = 1.69, IQR = 1.05, p = 0.22, n = 1871 trials, figure 5, left panel). We also examined whether or not the task condition used during Training of a decoder had any systematic effect on the β weights for any channel or dimension (x or y velocity). After using the Bonferonni method to correct for multiple comparisons, there were no significant differences in the decoder weights among all channel/dimension combinations as a function of task condition. Thus, the task condition used during training did not seem to affect subsequent performance or the decoders themselves.
But, as in Manual Control and as expected, the presence of the crowd during assessment blocks slowed task performance (Crowd On Median = 1.75, IQR = 1.05, n = 1656 trials; Crowd Off Median = 1.55, IQR = 1.00, n = 1723 trials; p < 10e −16 , figure 5, left panel). While this effect on performance was almost certainly due in part to the visual search required when the crowd was present, it was also possible that the presence of the crowd impaired Brain Control of the cursor by interfering with the neural signals used to determine cursor position. To test this possibility, we devised a second measure, ΔTTA, to directly assess the quality of Brain Control under the various training and assessment conditions.
The ΔTTA was computed for all trials to quantify how Brain Control affected the acquisition of Target 2 relative to Manual Control in isolation from other factors. We then used the same statistical comparisons that were computed for the unadjusted TTA values to determine whether or not the Crowd's presence during training or assessment influenced the ΔTTA. While the comparison revealed a significant effect of training condition (Crowd On Median = 0.99, IQR = 1.00, n = 1508 trials; Crowd Off Median = 1.04, IQR = 1.03, n = 1871 trials; p = 0.03, figure 5, right panel), and the trend favored the Crowd On condition, the small difference in the medians suggested only a negligible advantage. We found a significant influence of assessment condition on ΔTTA (Crowd On Median = 0.95, IQR = 1.04, n = 1656 trials; Crowd Off Median = 1.08, IQR = 0.97, n = 1723 trials; p = 1.3e −6 , figure 5, right panel), indicating that decoding with the Crowd On yielded slightly better Brain Control quality. Again, however, the magnitude of this difference was only 45 ms, so we considered this difference small enough to be negligible.
1.5 ( 3 1)) are considered outliers, where Q1 and Q3 are the 25th and 75th percentiles, respectively. The Crowd On condition resulted in an increase in acquisition and reward time relative to the Crowd Off condition. Time to Hold was not significantly affected, as the vast majority of the hold times in both conditions were the minimum possible value of 400 ms.
Taken together, these results indicated that the additional eye movements and various behavioral demands of the Crowd On task condition did not interfere directly with decoding of a neurally controlled cursor. The more complex task condition might have conferred a small albeit negligible advantage to performance of neural decoding.
These results suggest the NHP was able to simultaneously gaze freely around the screen while independently controlling cursor position. However, an alternative hypothesis that saccades did negatively impact cursor control would also account for this result if the NHP simply learned to minimize extraneous saccades when the Crowd was present, To rule out this possibility, we once again examined 2D histograms of eye position during a phase in the trials when the cursor was actively being transported to the target ( figure 4, right panel). These histograms include eye positions across a one second window ending on acquisition of Target 2. This window was chosen to capture the time when the cursor is still in motion in both Manual (left panel) as well as Brain Control (right panel). For Brain Control, the trials with straightest cursor trajectories were preselected for this analysis by choosing the fastest 15% of trials. For this The data demonstrated the NHP's tendency to gaze around the screen before moving the cursor when the Crowd was On. When the Crowd was Off, the NHP was able to initiate his hand movement to the target even before making a saccade to it. This explains why the left panel does not capture the position of the target at the 3 o'clock position. Here, data was taken during a 1 s window ending on target acquisition, guaranteeing that the cursor was in motion. We compared this epoch between Manual and Brain Control to confirm that the NHP was able to freely gaze at the targets even while maintaining straight cursor motion. For Brain Control, the trials with the straightest cursor trajectories were preselected by only analyzing the fastest 15% of trials. Average cursor trajectory during the same period for each control type is superimposed on the images in cyan. Video 2 also demonstrates dissociation of neural cursor movement from eye movements.
representative set, it is clear that, on average, even during active cursor movement in both Manual and Brain Control, the animal made many saccades to the faces around the screen. Additionally, ample saccades during cursor movement are evident in videos of task performance under Brain Control wherein playback speed was slowed and the animal's gaze position was added post-hoc (video 2).
Though the NHP was able to gaze around the screen with the Crowd On during Brain Control as well as in Manual Control, it should be noted that the overall number of saccades was reduced in Brain Control. We compared the number of saccades landing on a peripheral target (and not the correct target or the cursor) in a one second window ending on initial acquisition of Target 2 for each trial. We then compared the occurrence of these saccades in Manual (mean = 2.04 saccades, s.d. = 1.54) versus Brain Control (mean = 0.82 saccades, s.d. = 1.18) trials, revealing that there were significantly more in Manual Control (p < 10 −16 ). One possible explanation for this outcome is the increased difficulty and imperfect accuracy of Brain Control, i.e. on trails where cursor control is worse, the subject would need to gaze at the cursor longer to maintain closed-loop control. An alternative explanation is extraneous saccades reduced decode accuracy, and the subject learned to make fewer saccades to maintain cursor control. To distinguise these possibilities, we computed the correlation between numbers of saccades to peripheral targets (as above) in each trail to the TTA across all n = 1718 trails. A correlation of r = −0.12 (p < 10e −7 , T test) Outliers have been excluded for clarity. Assessment condition, indicated by the label on the abscissa, denotes the task used during full Brain Control with no assistance. Training condition, indicated by filled (Crowd On) or empty (Crowd Off) bands, denotes task used during training of decoders. For Time to Acquire (left panel), only assessment condition reached significance (p < 10 −16 ). For ΔTime to Acquire, both the training condition (p = 0.03) and assessment condition (p = 1.3e −6 ) were significantly better in the Crowd On conditions, though only by small margins. No pairwise comparisons, indicated by lines joining adjacent medians, reached statistical significance. All group statistics are listed in table 1. supports the former account and rules out the latter. Trails with many saccades to locations not occupied by the cursor were amongst the shortest, while the longest trails involved prolonged periods of gazing at the cursor, presumably to accommodate feedback control.
In a separate session, the Brain Control task was run with long (1 s) ITIs to determine if the animal was able to move the cursor back to the middle of the screen, the location where Target 1 appears at the start of each trial, before there is any overt visual cue to do so. The animal's performance clearly demonstrated his ability to move the cursor back to the middle of the screen before Target 1 appeared in anticipation of the upcoming trial (video 3). Additionally, in separate sessions, we confirmed that the decoders trained in this center-out style task could generalize to a 3 × 3 grid of the same face targets spaced evenly on an 18 cm × 18 cm square. While performance in terms of trial length was inherently slower than the circular, center-out task (p < 10 −16 ) due to the longer cursor movements required (TTA: Median = 2.405 s, IQR = 1.433, TTR: Median = 3.226 s IQR = 1.540), control of the cursor itself was qualitatively no different (video 4).
While the subject's arms were prevented from making large movements during Brain Control, he was still able to make small wrist and finger movements, though these movements as measured by the tracking sensor did not directly influence cursor position. On most days both hands were observably and measurably still (figure 6), however on other days small, residual movements were made during performance of the task under Brain Control. We used the measured hand kinematics (in the horizontal plane that would typically be used to control the cursor in Manual Control) to predict the kinematics of the neural cursor. Cross-validated Rsquared values never exceeded 0.03 for either dimension for any day, indicating little influence of residual hand movements on the decoded cursor.

Discussion
Despite the known sensitivity of motor control areas to numerous cognitive and motor variables (Colby and Goldberg 1999, Buneo and Andersen 2006, Pesaran et al 2010, Louie et al 2011, Rao and Donoghue 2014, we showed robust use of a neurally controlled cursor driven by signals from the parietal cortex in a context cognitively and visually richer than those created to date for use by primates. The task created a primate model of human use of GUI interfaces, e.g. tablet computers or smartphones. The performance measures indicated that training and decoding with the Crowd On did not impair neural cursor control, but may have actually conferred a small (albeit negligible) advantage. These small differences may simply have arisen as a result of motivational factors, i.e. more 'interesting' stimuli being present on the screen with the Crowd On.
By targeting area 5d for implant, we were able to obtain neural signals that reflected intended movements of the limb. Though residual eye-related signals were measurable in a control task, it was clear that they did not interfere with the functioning of the interface, whether during use of an existing decoder or during training. To our knowledge, this study is the first confirmation that unconstrained gaze does not interfere with prosthetic control, even in a visually complex task environment. Furthermore, we demonstrated the ability of the subject to decouple gaze position, i.e. sensing, from control of the cursor, i.e. the motor intention. This is a crucial capability for providing natural, intuitive control.
This capability was further emphasized given the ability of the subject to manipulate the cursor even in the absence of overt visual targets during the trials with extended ITIs. This result suggests that neural activity in parietal cortex can capture motor intentions without the need for overt visual representations of movement goals.
We hope in upcoming clinical work that human subjects will be able to control parietal neuroprosthetics by naturally manipulating their internal representation of the limb. Or with training, perhaps patients will mentally manipulate the cursor or end effector directly without remapping imagined actions or using other indirect strategies. This capability could reasonably be expected to occur given the observed mechanisms of tool use and/or extension of the body schema in parietal neurons by Iriki and colleagues (Iriki et al 2001).
These findings as a whole strengthen the case for the use of the parietal cortex in human clinical neuroprosthetic applications. They suggest that a human subject controlling a neural cursor driven by spiking activity in the parietal cortex could elicit similar results: robust 2D control that is insensitive to the visual and behavioral nuances of a modern computing interface.