Research reportCortical and subcortical contributions to state- and strength-based perceptual judgments
Introduction
How do we detect changes in the environment? Imagine you are shown two photographs of a park and asked whether they are exactly the same or if something about the park was different in the two images. In some cases, you may be able to detect a specific difference—for example, a water fountain that is in one picture but not in the other. Alternatively, you may know that the pictures are different, but are unable to provide details about any specific change.
Thus, there are two kinds of information that can be used for perceptual change detection, which have been referred to as state-based and strength-based perception (Aly and Yonelinas, 2012; for related distinctions, see Fernandez-Duque and Thornton, 2000, Rensink, 2000, Rensink, 2004, Dehaene et al., 2006, Howe and Webb, 2014). State- and strength-based perception have been studied by asking individuals to make same/different confidence judgments on pairs of images (e.g., pairs of scenes, faces, fractals, or objects; Aly and Yonelinas, 2012, Aly et al., 2013, Aly et al., 2014). Receiver-operating characteristic (ROC; Green and Swets, 1966, Macmillan and Creelman, 2005) analyses are then used to estimate the contributions of two kinds of perceptual decisions.
State-based perception is associated with high-confidence responses that are rarely in error; it is a discrete state that either occurs or does not, and when it does occur, it is associated with accurate awareness of specific details that differentiate two images. The probability of state-based perception is reflected in the upper x-intercept of ROCs (Fig. 1). Strength-based perception, on the other hand, is associated with a wider range of confidence responses; it is a continuously-graded signal associated with a feeling that something has changed, with little to no ability to report what that change was. The discriminability afforded by strength-based perception is related to the curvilinearity of ROCs (Fig. 1).
In previous studies, we have found that these two kinds of perception can be doubly dissociated, have different temporal dynamics, and are associated with distinct kinds of conscious experiences (Aly and Yonelinas, 2012, Aly et al., 2013, Aly et al., 2014). For example, state-based perception makes a greater contribution to tasks involving detection of discrete object changes (e.g., a water fountain that is present in one scene but absent in another), is associated with a rapid temporal onset, and subjective experiences are those of consciously perceiving specific, detailed differences. In contrast, strength-based perception makes a greater contribution to tasks involving global or relational change detection (e.g., a subtle manipulation of the distances between component parts of a scene), is associated with a gradual temporal onset, and subjective experiences are those of feeling as if a change has occurred but being unable to pinpoint what that change was (Aly and Yonelinas, 2012; also see Fernandez-Duque and Thornton, 2000; Rensink, 2000, Rensink, 2004; Dehaene et al., 2006; Galpin et al., 2008; Busch et al., 2009, Busch et al., 2010; Howe and Webb, 2014; but see Simons et al., 2005).
Thus, previous behavioral work on state- and strength-based perception has shown that perceptual decisions can be made on the basis of functionally dissociable processes or representations. State- and strength-based perception may reflect differences at early- to mid-level stages of perceptual representation (i.e., what information is represented in visual cortex, depending on the focus of attention) or later stages of decision-making (i.e., what information is used to inform the perceptual decision). While current data do not allow adjudication between these possibilities, it is clear that independent sources of information can be used to guide perceptual judgments.
In a previous neuropsychological study, we investigated the contribution of the hippocampus and surrounding medial temporal lobe (MTL) cortex to state- and strength-based perception (Aly et al., 2013). We tested patients with selective lesions to the hippocampus, bilaterally, and patients with more extensive unilateral MTL lesions that included the hippocampus and surrounding cortex. On each trial, patients and healthy controls were presented with a pair of scenes that were either identical or differed in that the center of one scene was expanded or contracted relative to the other (Fig. 1A). These changes alter the relational or configural information within the scenes without adding or removing any specific objects. Participants made same/different confidence judgments using a 1–6 scale, and these confidence responses were used to plot ROCs (Green and Swets, 1966, Macmillan and Creelman, 2005). The ROCs were in turn used to estimate state- and strength-based perception (see Fig. 1B for hypothetical data). The upper x-intercept of an ROC provides the probability that state-based perception has occurred, while the degree of curvilinearity is proportional to the contribution of strength-based perception (Aly and Yonelinas, 2012; see also Yonelinas, 1994).
Using this approach, we found that the patients were selectively impaired in strength-based perception (graded judgments of the overall configural or relational match/mismatch between images) but showed intact state-based perception (related to the ability to identify specific detailed differences between scenes; Aly and Yonelinas, 2012). This was true for patients with selective hippocampal lesions as well as those with more extensive MTL lesions. These data suggested that the hippocampus is critical for detecting configural or relational match/mismatch between complex scenes, but is not needed for state-based judgments based on identification of specific, item-level differences.
The MTL is just one of several regions that are likely to be critical for perceptual judgments on complex scenes. In a previous fMRI study (Aly et al., 2014), we examined whole-brain data to determine whether activity in different brain regions was differentially correlated with state- or strength-based perception. Individuals performed a task similar to that used in the MTL patient study, in which they viewed pairs of images and made same/different confidence judgments. These judgments were made using a scale that allowed individuals to report when state-based perception occurred, or, if it did not occur, to rate the confidence associated with strength-based perception. Activity in the supramarginal gyrus, posterior cingulate cortex, and precuneus was related to the occurrence of state-based perception, and was not modulated by varying confidence of strength-based perception. Activity in the fusiform gyrus, however, was sensitive to strength-based, but not state-based, perception. The lateral occipital complex showed both effects: that is, this region showed a graded increase in activity as confidence in strength-based perception increased, and showed an additional increase in activity for state-based judgments.
This study provides some insight into how state- and strength-based perception are supported by different brain regions, but, as with any fMRI study, it only indicates which regions are correlated with these different kinds of judgments, and does not indicate whether their activity is necessary for state- or strength-based perception. Thus, in the current study, we took a neuropsychological approach to determine which regions make necessary contributions to state- and strength-based perception.
In addition to this first exploratory aim, we also set out to test competing hypotheses about the role of lateral parietal cortex in state- vs. strength-based perception. The previous fMRI study (Aly et al., 2014) motivated the hypothesis that lateral parietal cortex — specifically, the supramarginal gyrus — might be critical for state- but not strength-based perception. Moreover, the “global neuronal workspace” model (Dehaene et al., 2006) proposes that an extended parietal-frontal network is critically involved in the threshold for conscious access; that is, this network shows a neural “ignition” that is related to conscious awareness of specific visual information. Insofar as state-based perception reflects a discrete signal indicating conscious awareness of detailed visual information, this would suggest a role for parietal regions in state-based perception (also see Lamme, 2003).
There are, however, reasons to predict that parietal cortex might be critical for strength-based perception. Our prior patient study implicated the hippocampus (and more generally, the MTL) in strength-based perception (Aly et al., 2013; also see Elfman et al., 2014). Due to the anatomical and functional connectivity between the hippocampus/MTL and parietal cortex (e.g., Kahn et al., 2008, Kravitz et al., 2011, Libby et al., 2012, Ranganath and Ritchey, 2012), one prediction is that patients with damage that includes parietal regions will show impairments in strength-based perception. Additionally, our findings relating strength-based perception to graded changes in confidence (Aly and Yonelinas, 2012) are reminiscent of the graded signals in monkey LIP neurons, which reflect continuous integration of sensory evidence in the service of perceptual decision-making (e.g., Shadlen and Newsome, 2001, Mazurek et al., 2003, Gold and Shadlen, 2007, Bollimunta et al., 2012). Although at different levels of analysis and different timescales, this parallel suggests that neural signals in parietal cortex may be related to perceptual judgments based on signals that vary in strength (for related fMRI work in humans, see Heekeren et al., 2006, Ploran et al., 2007, Ploran et al., 2011, Kayser et al., 2010, Liu and Pleskac, 2011).
Thus, our aims were twofold: (1) to explore which regions in the brain (outside of the MTL) are necessary for state-based and strength-based perception, and (2) to test competing hypotheses about the role of lateral parietal cortex in state- vs. strength-based perception. In order to examine these issues, we tested perceptual judgments in 11 stroke patients with right hemisphere lesions, which — considered as a group — included parietal, occipital, and temporal cortical regions, insula, thalamus, basal ganglia, and white matter in the vicinity of these cortical and subcortical structures (Fig. 2). Inclusion of patients with damage in heterogeneous regions allowed us to investigate the contributions of distinct brain areas to state- and strength-based perception, in addition to examining the specific hypotheses about the role of parietal cortex. Such an approach offers an important advance over our previous patient study, in which we only tested individuals with damage to the medial temporal lobe (Aly et al., 2013). We focus on right hemisphere structures because previous work has indicated that the right, more than the left, hemisphere plays a necessary role in visuospatial perception and attention (Mesulam, 1981).
We used a perceptual change detection task in which patients and healthy controls viewed pairs of scenes, presented sequentially, and indicated their confidence that the two were the same or different (Fig. 3). Differences consisted of a relational manipulation that slightly contracted or expanded the scenes relative to one another, changing the distances between component parts without adding or removing any particular object. Confidence ratings were used to plot ROCs and estimate the contributions of state- and strength-based perception.
In addition to the main behavioral analyses in which we examined state- and strength-based perception in the entire patient group, we conducted follow-up analyses in order to determine the roles of different right hemisphere regions in state- and strength-based perception. Specifically, we examined lesion overlap images for subgroups of patients depending on their behavioral performance. Such an analysis enabled us to test whether parietal cortical regions played a unique role in state- vs. strength-based perception: if this is indeed the case, patients who do not have damage in the parietal cortex should perform differently from those who do. Thus, we felt that this analysis would be useful in providing further insights into the neural correlates of state- and strength-based perception, and would be important in guiding future studies.
Section snippets
Participants
The study was approved by the University of Liège Psychology ethics review board. All patients and healthy control participants gave their written informed consent prior to their inclusion in this study.
The patient group consisted of 11 patients with right hemisphere damage as a result of stroke. Patients were recruited at Centre Neurologique et de Réadaptation Fonctionnelle Fraiture, Hôpital Sainte-Ode, and University hospitals from Liège and Brussels in Belgium. All patients but one were
Results
Performance was examined by plotting confidence-based ROC curves. The leftmost point on the ROC is the probability of a hit (y-axis) and false alarm (x-axis) for the most confident “same” response, and subsequent points are the cumulative probabilities for hits and false alarms as responses of decreasing confidence are added. Parameter estimates of state- and strength-based perception are obtained using maximum likelihood estimation to find the curve that best fits the observed ROC points (Aly
Discussion
Perceptual judgments can be based on different kinds of information (Fernandez-Duque and Thornton, 2000, Rensink, 2000, Rensink, 2004, Dehaene et al., 2006, Galpin et al., 2008, Busch et al., 2009, Busch et al., 2010, Aly and Yonelinas, 2012, Howe and Webb, 2014). A useful distinction is between state-based judgments in which individuals have conscious access to specific, detailed information, and strength-based judgments, which are based on a graded sense of overall match/mismatch (Aly and
Conclusions
Perceptual change detection can be based on different kinds of information: conscious access to local, detailed information (state-based perception), or graded signals reflecting a sense of relational match/mismatch (strength-based perception). In the current study, we show that right temporo-parietal cortical regions play a critical and selective role in strength-based perception, while the integrity of the right thalamus, putamen, and adjacent white matter is necessary for intact state- and
Funding
National Institute of Mental Health Grant MH059352 to APY.
Inter-University Attraction Pole P7/11, F.R.S.-FNRS (CB is an F.R.S.-FNRS research associate; travel funding 2012/V 3/5/110-IB/JN-758).
Acknowledgments
We would like to thank Joy Geng for valuable feedback on earlier drafts of the manuscript.
References (71)
- et al.
Detecting changes in scenes: the hippocampus is critical for strength-based perception
Neuron
(2013) - et al.
Parietal cortex and attention
Curr. Opin. Neurobiol.
(2004) - et al.
Unilateral neglect of representational space
Cortex
(1978) - et al.
Spatial normalization of brain images with focal lesions using cost function masking
NeuroImage
(2001) - et al.
Sensory and cognitive functions of the basal ganglia
Curr. Opin. Neurobiol.
(1997) - et al.
Cognitive contributions of the ventral parietal cortex: an integrative theoretical account
Trends Cogn. Sci.
(2012) - et al.
Conscious, preconscious, and subliminal processing: a testable taxonomy
Trends Cogn. Sci.
(2006) - et al.
Impaired extra-dimensional shift performance in medicated and unmedicated Parkinson’s disease: evidence for a specific attentional dysfunction
Neuropsychologia
(1989) - et al.
A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data
NeuroImage
(2005) - et al.
Re-evaluating the role of TPJ in attentional control: contextual updating?
Neurosci. Biobehav. Rev.
(2013)
Paying attention to the thalamic reticular nucleus
Trends Neurosci.
Spatial cognition: evidence from visual neglect
Trends Cogn. Sci.
Impaired perception of mnemonic oldness, but not mnemonic newness, after parietal lobe damage
Neuropsychologia
The anatomy of spatial neglect
Neuropsychologia
Why visual attention and awareness are different
Trends Cogn. Sci.
Attenuated boundary extension produces a paradoxical memory advantage in amnesic patients
Curr. Biol.
Seeing, sensing, and scrutinizing
Vis. Res.
Age-specific CT and MRI templates for spatial normalization
NeuroImage
Gain control in the visual thalamus during perception and cognition
Curr. Opin. Neurobiol.
Distractibility in early Parkinson’s disease
Cortex
The nature of recollection and familiarity: A review of 30 years of research
J. Mem. Lang.
A simple test of visual neglect
Neurology
Neural correlates of state- and strength-based perceptions
J. Cogn. Neurosci.
Bridging consciousness and cognition in memory and perception: evidence for both state and strength processes
PLoS One
Sensitivity of clinical and behavioral tests of spatial neglect after right hemisphere stroke
J. Neurol. Neurosurg. Psychiatry
Doors and People: A Test of Visual and Verbal Recall and Recognition
Neural dynamics of choice: single-trial analysis of decision-related activity in parietal cortex
J. Neurosci.
ERP effects of change localization, change identification, and change blindness
NeuroReport
Electrophysiological evidence for different types of change detection and change blindness
J. Cogn. Neurosci.
Spatial neglect and attention networks
Annu. Rev. Neurosci.
Function of the thalamic reticular complex: the searchlight hypothesis
Proc. Natl. Acad. Sci. USA
The FAB: a frontal assessment battery at bedside
Neurology
Neurocomputational account of memory and perception: thresholded and graded signals in the hippocampus
Hippocampus
Change detection without awareness: do explicit reports underestimate the representation of change in the visual system?
Vis. Cogn.
Sensing without seeing in comparative visual search
Conscious. Cogn.
Cited by (4)
Visual working memory impairments for single items following medial temporal lobe damage
2019, NeuropsychologiaCitation Excerpt :In addition to functional independence, neuropsychological, neuroimaging, and neurocomputational studies converge in showing that the dual processes underlying perception, working memory, and long-term memory are also anatomically distinct. For instance, in perception and working memory, sensing relies on the hippocampus but perceiving does not (Aly et al., 2013; Aly et al., 2014; Aly et al., 2014; Goodrich and Yonelinas, 2016). Similarly, in long-term memory, remembering is hippocampally dependent but knowing is not (Düzel et al., 2001; Duzel et al., 1997; Eichenbaum et al., 2007; Elfman et al., 2014).
Eye movements dissociate between perceiving, sensing, and unconscious change detection in scenes
2022, Psychonomic Bulletin and Review