Independent circuits in basal ganglia and cortex for the processing of reward and precision feedback
Introduction
Humans and other animals must be able to evaluate actions as a function of the quality of their outcome. Decades of neurophysiological and neuroimaging studies have demonstrated that the meso-cortico-striatal pathway is central to this function (McClure et al., 2004, O'Doherty, 2004, Schultz, 2000, Schultz, 2006, Schultz, 2013). Neurons in this system respond to explicit reward (Apicella et al., 1991, Knutson et al., 2003), signal errors in the prediction of reward (Schultz et al., 1997, Bayer and Glimcher, 2005), and drive selection of reward cues and approach toward these objects (Berridge and Robinson, 1998, Flagel et al., 2011, Hickey and Peelen, 2015). The ventral striatum (VS), a target of midbrain and cortical projections, has received particular attention in this context. This structure plays a core role in instrumental learning (O'Doherty et al., 2004) and reward-contingent behavior (Tricomi et al., 2004) and is sensitive to various types of external reward feedback (Knutson and Cooper, 2005).
The well-known sensitivity of the VS to reward feedback has led to the widely-held notion that this structure is in fact dedicated to the processing of reward. However, recent functional magnetic resonance (fMRI) findings have shown that the VS, together with other reward-related structures, is also activated by simple cognitive feedback such as that indicating performance accuracy (Rodriguez et al., 2006, Daniel and Pollmann, 2010, Tricomi and Fiez, 2008, Ullsperger and Von Cramon, 2003, Han et al., 2010, Wolf et al., 2011).
Feedback-related responses in the striatum have been observed in a variety of tasks, ranging from information-integration learning (Daniel and Pollmann, 2010) to perceptual training (Tricomi et al., 2006). A handful of studies have observed striatal activation following accurate responses even when no explicit feedback is provided at all (Daniel and Pollmann, 2012, Satterthwaite et al., 2012, Guggenmos et al., 2016). In this situation, the VS responds most strongly when participants are completing a challenging task (Satterthwaite et al., 2012, Dobryakova et al., 2016) or when they are confident about their performance (Daniel and Pollmann, 2012).
In addition to the VS, other striatal and cortical structures have been associated with both reward and performance processing. On one hand, the putamen - a key node in the motor feedback loop - responds to aspects of task performance that extend beyond purely motor execution processes. A number of studies have shown putamen activation in response to performance feedback (Cincotta and Seger, 2007, Eppinger et al., 2013), reward prediction errors (Garrison et al., 2013, Daniel and Pollmann, 2012, Sommer and Pollmann, 2016), performance evaluation and perceived competence, even in the absence of external feedback or reward (Daniel and Pollmann, 2010, Daniel and Pollmann, 2012, Guggenmos et al., 2016, Sommer and Pollmann, 2016). On the other hand, regions such as orbitofrontal cortex (OFC) and posterior cingulate cortex (PCC) have been extensively linked to the processing of external reward (Liu et al., 2011). This suggests that performance feedback and internal signals of precision may target specific subcomponents of the reward system and striatal nuclei in particular. Reward-associated cortical areas, in contrast, may be sensitive to explicit primary and secondary reward feedback.
A number of studies have addressed the possibility that the dopaminergic system, and the striatum in particular, may contribute not only to the analysis of external rewards but also to the processing of internally-generated signals reflecting valuation of accurate performance (Satterthwaite et al., 2012, Daniel and Pollmann, 2012, Pascucci and Turatto, 2013, Pascucci et al., 2015; see Daniel and Pollmann, 2014 for a review). For example, Daniel and Pollmann (2010) directly compared neural correlates of monetary reward with cognitive feedback during two parallel category-learning tasks. The authors found that both types of reinforcer activate the dopaminergic system in similar ways, but that a core structure of the VS, the nucleus accumbens (NAc), responded more strongly when learning was paired with monetary reward. Similarly, Delgado et al. (2004) found that VS activation in response to the outcome of a gambling task was greater after reward-related feedback than after accuracy feedback, and Murayama et al. (2010) showed that the removal of external reward from a previously enjoyable task decreased the sensitivity of reward-related structures to task performance.
Taken together, this evidence suggests that reward incentives may be crucial in driving dopaminergic responses to performance outcomes. Tricomi and colleagues (Tricomi et al., 2006) have proposed that non-reward incentives like performance feedback become effective only under specific circumstances. As a result, motivational context and individual variability become important in predicting striatal sensitivity to different types of feedback (Tricomi et al., 2006, Delgado et al., 2004).
There is thus ambiguity in our understanding of striatal sensitivity to reward or performance feedback. One reason for this ambiguity is that existing studies investigating the role of non-reward information in striatal activation have understandably tended either to omit reward from the experimental design (Rodriguez et al., 2006, Murayama et al., 2010, Daniel and Pollmann, 2012, Satterthwaite et al., 2012) or have associated explicit reward to one task and accuracy feedback to another (Daniel and Pollmann, 2010, Delgado et al., 2004). Under these circumstances, it is unclear whether observed striatal sensitivity to task accuracy reflects a fundamental function of the area. It may be that this system always analyzes the quality of task performance, even when this kind of evaluation is not required by task instructions and is not required to achieve rewarding outcome. But it may alternatively be the case that, in the absence of external feedback, the dopaminergic system becomes sensitive to the next best learning signal, namely task accuracy.
Here we test these contrasting hypotheses. While in the fMRI scanner, we had human participants perform a simple video game that involved firing a bullet at a target. Each trial of this game resulted in one of five outcomes: a perfect hit, when the bullet hit the center of the target; a good hit, when the bullet hit the side of the target; a near miss, when the bullet hit the extreme edge of the target; a near hit, when the bullet just missed the target; and a bad miss, when the bullet landed far from the target (see Fig. 1B). Participants knew that hits resulted in monetary reward, but, critically, they were unaware that the game was rigged: the outcome of each trial was determined prior to task execution. This provided us the ability not only to manipulate whether a trial resulted in a hit, and thus whether reward was received, but also to vary the quality of the hit, and therefore the perceived precision of performance.
We used parametric analyses of the resulting fMRI data to isolate activity caused by the manipulation of explicit reward from activity caused by manipulation of task precision, and we used functional connectivity analysis to identify segregated networks supporting the processing of explicit reward feedback and task precision.
Section snippets
Subjects
Twenty healthy volunteers (mean age = 24 ± 3, 14 female) were recruited from the University of Trento and paid at the end of the experiment. All participants gave written informed consent. The study was conducted under the approval of the local institutional ethics committee.
Visual stimulation
Stimuli were back-projected onto a screen by a liquid-crystal projector at a frame rate of 60 Hz and a screen resolution of 1280 × 1024 pixels (mean luminance: 109 cd/m2). Participants viewed the stimuli binocularly through
Precision
In the parametric modulation analysis of the reward-first GLM, the Precision modulator could account only for variance not already partitioned to the Reward manipulation. This revealed a single significant cluster of 66 voxels in the right posterior putamen (peak activity at x = 27, y = −4, z = −7, T = 6.44; see Fig. 2A, green color scale, and Table 1). The sensitivity of this caudal portion of the striatum to precision feedback was corroborated by results from the parametric GLM. The analysis
Discussion
We investigated brain areas involved in the processing of reward and performance feedback when both signals were present in the same task. To date, effects of accuracy feedback on striatal activity have been investigated in two ways: 1) with external reward explicitly omitted from an experimental design (Rodriguez et al., 2006, Murayama et al., 2010, Daniel and Pollmann, 2012, Satterthwaite et al., 2012), and 2) with reward and accuracy feedback alternated in separate blocks of trials (Daniel
References (77)
- et al.
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Neuron
(2005) - et al.
What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?
Brain Res. Rev.
(1998) - et al.
Absolute coding of stimulus novelty in the human substantia nigra/VTA
Neuron
(2006) - et al.
Striatal activations signal prediction errors on confidence in the absence of external feedback
Neuroimage
(2012) - et al.
A universal role of the ventral striatum in reward-based learning: evidence from human studies
Neurobiol. Learn. Mem.
(2014) - et al.
Modulation of ventral striatal activity by cognitive effort
Neuroimage
(2017) - et al.
Prediction error in reinforcement learning: a meta-analysis of neuroimaging studies
Neurosci. Biobehav. Rev.
(2013) - et al.
Modeling regional and psychophysiologic interactions in fMRI: the importance of hemodynamic deconvolution
Neuroimage
(2003) - et al.
Posterior cingulate cortex mediates outcome-contingent allocation of behavior
Neuron
(2008) - et al.
Neural mechanisms of incentive salience in naturalistic human vision
Neuron
(2015)
Stimulus–response and response–outcome learning mechanisms in the striatum
Behav. Brain Res.
Distinct striatal regions for planning and executing novel and automated movement sequences
Neuroimage
Actor–critic models of the basal ganglia: new anatomical and computational perspectives
Neural Netw.
A region of mesial prefrontal cortex tracks monetarily rewarding outcomes: characterization with rapid event-related fMRI
Neuroimage
Novelty increases the mesolimbic functional connectivity of the substantia nigra/ventral tegmental area (SN/VTA) during reward anticipation: evidence from high-resolution fMRI
Neuroimage
Common and distinct networks underlying reward valence and processing stages: a meta-analysis of functional neuroimaging studies
Neurosci. Biobehav. Rev.
Temporal prediction errors in a passive learning task activate human striatum
Neuron
Saccade reward signals in posterior cingulate cortex
Neuron
A generalized form of context-dependent psychophysiological interactions (gPPI): a comparison to standard approaches
Neuroimage
Valid conjunction inference with the minimum statistic
Neuroimage
Reward representations and reward-related learning in the human brain: insights from neuroimaging
Curr. Opin. Neurobiol.
Neural responses during anticipation of a primary taste reward
Neuron
Neurons in posterior cingulate cortex signal exploratory decisions in a dynamic multioption choice task
Curr. Biol.
Posterior cingulate cortex: adapting behavior to a changing world
Trends Cognit. Sci.
Role of dopamine neurons in reward and aversion: a synaptic plasticity perspective
Neuron
Being right is its own reward: load and performance related ventral striatum activation to correct responses during a working memory task in youth
Neuroimage
Updating dopamine reward signals
Curr. Opin. Neurobiol.
Modulation of caudate activity by action contingency
Neuron
Feedback signals in the caudate reflect goal achievement on a declarative memory task
Neuroimage
Frontal pole cortex: encoding ends at the end of the endbrain
Trends Cognit. Sci.
Responses to reward in monkey dorsal and ventral striatum
Exp. Brain Res.
Behavioral inhibition, behavioral activation, and affective responses to impending reward and punishment: the BIS/BAS Scales
J. Pers. Soc. Psychol.
Dissociation between striatal regions while learning to categorize via feedback and via observation
J. Cognit. Neurosci.
Effects of expectations for different reward magnitudes on neuronal activity in primate striatum
J. Neurophysiol.
Comparing the neural basis of monetary reward and cognitive feedback during information-integration category learning
J. Neurosci.
Motivation-dependent responses in the human caudate nucleus
Cereb. Cortex
Contributions of the putamen to cognitive function
Reduced striatal responses to reward prediction errors in older compared with younger adults
J. Neurosci.
Cited by (12)
Dorsal striatum does not mediate feedback-based, stimulus-response learning: An event-related fMRI study in patients with Parkinson's disease tested on and off dopaminergic therapy
2019, NeuroImageCitation Excerpt :Further, learning efficiency and VS activation were reduced for PD patients on relative to off dopaminergic therapy, suggesting that VS, a VTA-innervated structure, was overdosed by exogenous dopamine. This result fits with the larger literature implicating VS in forms of implicit learning (Tricomi et al., 2009; Sommer and Pollmann, 2016; Vo et al., 2016; Pascucci et al., 2017; Vo et al., 2018), such as reward (Camara et al., 2010), stimulus-stimulus (MacDonald et al., 2011), sequence (Ghilardi et al., 2007), motor sequence (Feigin et al., 2003), and category learning (Shohamy et al., 2006). In contrasts where DS activation emerged, cortical regions previously implicated in decision making and categorization judgments were also revealed.
Decision-making under ambiguity and risk and executive functions in Parkinson’s disease patients: A scoping review of the studies investigating the Iowa Gambling Task and the Game of Dice
2023, Cognitive, Affective and Behavioral NeuroscienceFamily income buffers the relationship between childhood adverse experiences and putamen volume
2023, Developmental NeurobiologyThe Neural Correlates of Reinforcement Sensitivity Theory: A Systematic Review of the (f)MRI Literature
2022, Psychology and Neuroscience