The contribution of surprise to the prediction based modulation of fMRI responses
Introduction
The extensively studied neural repetition suppression (RS) phenomenon (for review see Grill-Spector and Martin (2006)) has been recently associated with predictive coding (PC) theories (Friston, 2005) of neural functions (Summerfield et al., 2008). RS describes decreased neuronal response after the repetition of a given stimulus and is used to study the selective properties of neuronal populations (Malach, 2012). Similar to RS, fulfilled expectations also lead to reduced neural activity when compared to incorrect predictions, i.e. surprising events, and this phenomenon has recently been termed as expectation suppression (, ). Yet, the relationship between RS and expectation suppression as well as their underlying neural mechanisms are still unclear.
Summerfield et al. (2008) found that the magnitude of RS depends on the probability of stimulus repetitions (Prep): the RS was enhanced in the fusiform face area (FFA; Kanwisher et al., 1997) when faces were presented in blocks in which repetitions were frequent (therefore expected) as compared to when presented in blocks with low repetition probability. Authors suggested that higher-order contextual expectations modulated, via feedback connections, repetition-related processes and interpreted this result in the context of PC models (, ). According to PC, the visual cortex operates under a hierarchical structure where higher areas send predictions about sensory inputs to lower level areas, which then compute the difference between predictions and the actual sensory input (termed as prediction error - ɛ). To re-estimate and update predictions, ɛ is forwarded from lower to higher areas of the processing system. Consequently, surprising/incorrectly predicted events generate higher neural activity in comparison with correctly predicted events, maximizing the efficiency of neuronal processing (, , ). Summerfield et al. (2008) interpreted the enhanced magnitude of RS for expected stimuli as the reduced neuronal activity induced by a smaller ɛ (following Henson (2003) claim of a link between RS and ɛ). This effect of expectation on RS was later replicated for faces (Grotheer et al., 2014, , , ) and for stimuli of high expertise (Grotheer and Kovács, 2014), while such Prep modulations were not found for object-related RS (, ) and for unfamiliar characters (Grotheer and Kovács, 2014). It should be noted that all of these above studies used blocks with high and low repetition probabilities to manipulate expectations, for example in blocks with high likelihood of repetition, repeated trials are predicted and alternating trials are surprising while the opposite is true for blocks with low repetition probabilities. Therefore, this mixed design does not allow the independent testing of expectation and repetition effects.
Recently, a MEG study (Todorovic and Lange, 2012) could however, manipulate RS and expectation suppression independently, evoking expectations on a trial-by-trial basis using a preceding cue. Pairs of identical or different tones were presented; the expectations of the subjects were generated by the first tone of each pair, which signaled the likelihood of repetitions with 75% accuracy. The results indicated that expectation suppression and RS have different temporal windows, though an expectancy modulation on repetition effects was also observed. The different mechanisms behind expectation suppression and RS is supported further by Grotheer and Kovács (2015), where pairs of female/male faces were used as stimuli and their gender was signaling the different repetition probabilities (for example female faces were repeated with high while male faces were repeated with a low probability). This fMRI study showed that RS and expectation suppression are additive, rather than interacting in the FFA and the occipital face area (OFA; Gauthier et al., 2000).
However, none of these previous studies could clarify whether the addition of expectation suppression and RS effects is due to a decrease of the response for correctly expected stimuli or an increase of the response to the surprising, unexpected stimuli (Kovács and Vogels (2014) raised this issue and suggested the inclusion of a “neutral” condition with equal probabilities for alternating and repeated trials, in which no expectations are induced (see also , ). Fig.1 illustrates the possible hypotheses regarding RS and expectation modulations of the neural responses, considering the inclusion of the neutral condition. We reasoned that if the previously observed expectation effects are due to a genuine response reduction, then these trials should lead to lower BOLD signal when compared to the unpredicted, neutral trials as well. However, if the prediction effects are due to the enhanced response in the surprising trials (alternating and repeated) then these should lead to larger BOLD responses when compared to the unpredicted (neutral) as well as to the correctly predicted trials. Thus, a main effect of expectation conditions and a subsequent post-hoc analysis would clarify from which expectation condition the BOLD signal of unpredicted, neutral trials differs most – from the correctly predicted (suggesting the role of expectation in predictions) or from the surprising trials (suggesting the role of surprise in predictions). Here we used the methods, task and paradigm of Grotheer and Kovács (2015) with the additional trials of the neutral, unpredicted condition, to study under which circumstances these top-down (suppressing or enhancing) modulations operate.
Anticipating our results, we found significant RS and expectation effect in the FFA. Further, we observed a significant increase of neuronal responses for the surprising, unexpected events, relative to the neutral and unpredicted events in the alternation trials. The relationship of RS and surprise differed between hemispheres: rFFA revealed a dependence of RS on surprise, whereas lFFA showed the independence of these two processes. Overall our results emphasize the role of surprise in predicted processes.
Section snippets
Participants
24 healthy subjects participated in the experiment after giving written, informed consent in accordance with the protocols approved by the Ethical Committee of the Friedrich Schiller University Jena. No participant had any history of neurological or psychiatric illness and all had normal or corrected to normal vision. Due to technical issues, 2 participants were excluded from the analysis and for 1 participant only 2 of the 3 functional runs were acquired. Thus, 22 subjects (8 male; 2
Behavior
Mean accuracy for gender judgement was 91% (±SD: 8%) across all trial types (Exp_Rep: 93(7)%, Exp_Alt: 94(5)%, Sur_Rep: 84(18)%, Sur_Alt: 88(13)%, Neu_Rep: 92(6)%, Neu_Alt: 90(11)%). The participant’s performances did not differ between trial types (F(1,21)=0.97, p=0.34, ηp2=0.04). However, participants had a significantly lower performance in trials when their predictions were incorrect (main effect of expectation condition: F(2,42)=3.8, p=0.03) as compared to trials with correct predictions
Discussion
Our major result is that surprising events lead to significantly larger activity as compared to unpredicted, neutral events, thereby supporting the hypothesis outlined on Fig. 1A and emphasizing the role of surprise in predictive coding processes.
Predictive coding models assume that ɛ relies on the discrepancy between observed and predicted sensory states (Friston, 2012), supporting the finding of surprise related enhancement of the activity. However according to theories of PC (Friston 2012)
Conflict of interest
The authors declare no competing financial interests.
Acknowledgements
Supported by a Deutsche Forschungsgemeinschaft Grant (KO 3918/1-2; 2-2). HP and ZV was supported by a grant from the Hungarian Brain Research Program (KTIA_13_NAP-A-I/18).
We thank the anonymous reviewer for pointing out the inability of univariate analysis techniques to separate ɛ and representational unit activities.
References (33)
- et al.
Cortical oscillations and sensory predictions
Trends Cognit. Sci.
(2012) Prediction, perception and agency
Int. J. Psychophysiol.
(2012)- et al.
Repetition and the brain: neural models of stimulus-specific effects
Trends Cognit. Sci.
(2006) - et al.
Repetition probability effects for inverted faces
NeuroImage
(2014) - et al.
The relationship between stimulus repetitions and fulfilled expectations
Neuropsychologia
(2015) Neuroimaging studies of priming
Prog. Neurobiol.
(2003)- et al.
Less is more: expectation sharpens representations in the primary visual cortex
Neuron
(2012) - et al.
Stimulus repetition probability effects on repetition suppression are position invariant for faces
Neuroimage
(2012) Targeting the functional properties of cortical neurons using fMR-adaptation
Neuroimage
(2012)- et al.
A blueprint for target motion: fMRI reveals perceived sequential complexity to modulate premotor cortex
Neuroimage
(2002)
Expectation (and attention) in visual cognition
Trends Cognit. Sci.
The asymmetry of the fusiform face area is a stable individual characteristic that underlies the left-visual-field superiority for faces
Neuropsychologia
The problem of functional localization in the human brain
Nat. Rev. Neurosci.
That baby caught my eye… attention capture by infant faces
Emotion
Prediction, cognition and the brain
Front. Hum. Neurosci.
Neural correlates of high-level adaptation-related aftereffects
J. Neurophysiol.
Cited by (27)
The effect of context congruency on fMRI repetition suppression for objects
2023, NeuropsychologiaEvaluating the evidence for expectation suppression in the visual system
2021, Neuroscience and Biobehavioral ReviewsCitation Excerpt :Across the studies reviewed here, it appears that the exact design of the neutral condition may be potentially relevant to whether or not ES is observed. The majority of experiments compared an expected stimulus condition, in which a specific image or stimulus attribute could be predicted to appear with more than 50 % probability, with a neutral condition in which two stimuli could each appear with 50 % probability (e.g., Summerfield and Koechlin, 2008; Egner et al., 2010; Rahnev et al., 2011; St John-Saaltink et al., 2015; Amado et al., 2016; Hindy et al., 2020). The majority of these studies did not report statistically significant ES effects.
Visual mismatch responses index surprise signalling but not expectation suppression
2021, CortexCitation Excerpt :Our findings also suggest that, in some contexts, the notion of expectation suppression (Grotheer & Kovacs, 2015; Pajani et al., 2017; Summerfield et al., 2008; Todorovic & de Lange, 2012) may be a misnomer when used to explain neural response differences between expected and surprising stimulus contexts. Studies using non-oddball designs and probabilistic cueing have similarly reported much larger effects of surprise on fMRI BOLD signals than of fulfilled expectations (Egner et al., 2010; Amado et al., 2016; reviewed in; Kovacs & Vogels, 2014). Previous studies using roving stimulus variants of oddball designs have reported expectation-related suppression of prediction error-related neural responses (e.g., Stefanics et al., 2018), however these designs tend to conflate expectation and repetition effects, and the observed neural response reductions could instead be attributed to repetition suppression.
Measures of repetition suppression in the fusiform face area are inflated by co-occurring effects of statistically learned visual associations
2020, CortexCitation Excerpt :Importantly, prior to the fMRI scanning session participants underwent 4 training sessions on consecutive days, during which they were presented with 6 predictable alternating face pairs (i.e., the first face of a pair was always followed by a given same-sex face) to create specific face associations for the alternating trials. Because previous fMRI studies that presented face stimuli (Amado et al., 2016; Egner et al., 2010; Summerfield et al., 2008) found the most pronounced effects of stimulus repetition and perceptual expectations in the fusiform face area (FFA; Kanwisher et al., 1997) we focused our analyses on this region. Our design allowed us to control for effects of stimulus novelty, enabling a more accurate estimate of stimulus predictability effects in repetition designs.