Neural correlates of active vision: An fMRI comparison of natural reading and scene viewing
Introduction
Visual perception and visual cognition are active processes in which saccadic eye movements play a central role (Dodge, 1903, Buswell, 1935, Findlay and Gilchrist, 2003, Henderson and Ferreira, 2004, Henderson, 2013, Rayner, 2009, Yarbus, 1967). For example, in both natural scene viewing and reading, the eyes move from location to location to enable the acquisition of information as it is needed in real time. Current theoretical approaches and computational models of reading and scene viewing therefore attempt to account for how attention and particularly eye movements are controlled during these activities (e.g., Reichle et al., 1998; Engbert et al., 2005; Nuthmann et al., 2010; Torralba et al., 2006). An important issue in understanding active vision is the degree to which eye movement control systems are domain-specific (i.e., dedicated to a particular active vision domain such as reading or scene perception) versus domain-general (i.e., dedicated to all active vision tasks in which eye movements are controlled in the service of perception and cognition). Evidence from eyetracking shows strong correlations in basic eye movement measures within individuals across tasks like reading, scene viewing, and search, suggesting common neurocognitive eye movement control processes across domains (Andrews and Coppola, 1999, Castelhano and Henderson, 2008, Henderson and Luke, 2014, Rayner et al., 2007). Several computational models of eye movement control that have been developed to account for one domain have been extended to other domains, implicitly endorsing the domain-general hypothesis. For example, the E–Z Reader and SWIFT models of eye movement control in reading have both been extended to non-reading tasks (Reichle et al., 2012, Trukenbrod and Engbert, 2012), and the CRISP model of eye movement control in scene perception has been extended to reading (Nuthmann et al., 2010, Nuthmann and Henderson, 2012). Yet there has been very little direct investigation of the degree to which the eye movement control system is similar or different across domains. In the present study we examined this issue by investigating the similarities and differences in the cortical eye movement control network during natural reading and scene viewing.
Past studies using single-step eye movement paradigms as in the gap and anti-saccade tasks have provided evidence for a frontal-parietal eye movement control network that includes as core components the frontal eye fields (FEF), supplementary eye fields (SEF), and intraparietal sulcus (IPS). Additional cortical areas often observed in single-step eye movement studies include dorsolateral prefrontal cortex (DLPFC), likely associated with maintaining task goals in these tasks. Recently, Hillen et al. (2013) found blood oxygenation level dependent (BOLD) activation in FEF, SEF, and IPS during a target-search task through texts and pseudo-texts that required sequences of saccadic eye movements. These results indicate that the eye-movement network observed for single-saccade tasks is also engaged during sequential scanning. In addition, Choi et al. (2014) compared BOLD activation during natural reading using normal texts and pseudo-reading using nonword control texts. No secondary task was administrated when participants read paragraphs of normal texts. Choi et al. also found common brain areas such as FEF, SEF and IPS associated with the control of eye movements in both reading and pseudo-reading.
Whether this eye movement network is also functional in other active vision tasks such as scene viewing is currently unknown. The present study investigated similarities and differences in the eye movement control network in natural reading and scene viewing by comparing the natural reading condition from Choi et al. (2014) to a scene viewing condition. The data for natural reading in the current study is the same as that reported in Choi et al. (2014). The purpose of the present report is to compare the similarities and differences in the neural correlates of eye movements during natural reading from Choi et al. (2014) with the neural correlates of eye movements in scene viewing to investigate the domain-specificity of eye movement control systems.
A secondary issue investigated here was whether differential activation in cortical regions associated with reading and scene viewing can be observed during active viewing. The majority of neuroimaging studies of both reading and scene viewing have used brief presentation (rapid serial visual presentation or RSVP) methods in which saccadic eye movements are not possible or are discouraged1. In general, reading via RSVP tends to activate a left-lateralized frontal–temporal–parietal network associated with language processing (Friederici, 2011, Martin, 2003, Price, 2012). This activation typically involves an occipital-temporal network including the purported visual word form area (VWFA), middle and superior temporal gyrus/sulcus (MTG, STG/STS), and inferior frontal gyrus (IFG). In comparison, scene viewing tends to activate a right-lateralized frontal-temporal-parietal network associated with visual scene analysis (Epstein, 2005, Epstein, 2008, Grill-Spector, 2003, Maguire, 2001). This network notably includes temporal–occipital junction (TOJ) associated with visual objects, and areas of fusiform gyrus and the parahippocampal cortex associated with scene analysis (Bar et al., 2001, Epstein et al., 2000, Henderson et al., 2008, Henderson et al., 2007, Henderson et al., 2011, Park and Chun, 2009, Walther et al., 2009). Here we investigated whether active vision tasks involving extended viewing with eye movements in reading versus scene viewing lead to the type of differential activation that is typically observed in brief presentations of words and scenes. Similar activation in the hypothesized eye movement network in spite of differential activation in domain-specific processing networks associated with reading and scene viewing would provide additional evidence for the domain-generality of eye movement control.
To investigate these issues, we combined simultaneous high-resolution eyetracking and fMRI data acquisition while subjects read passages and viewed photographs of real-world scenes. Each stimulus was presented for 12 s and was viewed actively. Eyetracking was used to ensure that subjects were engaging naturally with the stimuli. We hypothesized that the eye movement control network identified in single-saccade tasks also supports these active vision tasks, but that reading and scene viewing may also draw on independent eye movement sub-systems given that these tasks use partially independent cognitive operations. We also hypothesized that actively engaging in reading and scene viewing would produce activation similar to that observed previously in brief-presentation RSVP studies of reading and scene perception.
Section snippets
Subjects
Thirty-three subjects (12 male) participated in this study. They were all right-handed, native speakers of English, aged 18–35 years (Mean Age: 21.45). Thirty subjects were students from the University of South Carolina and three were recruited from the Columbia, South Carolina community. All subjects gave informed consent and were screened for MRI safety. They all reported normal or corrected-to-normal vision and were given $10 per hour for participation in the study. Two did not finish the
Eye-movement analysis
The eye-movement data were analyzed to obtain fixations and saccades in the reading and scene viewing conditions (Table 1). All fixations meeting the following criteria were included in this analysis: A fixation could not have a blink immediately before or after, and had to have a duration between 50 and 1500 ms. Data during track losses were also rejected. In total, 13.4% of fixations (11.2% for the reading condition and 16.7% for the scene viewing condition) were excluded. The summary eye
Discussion
An important issue in understanding active vision is the degree to which eye movement control is domain-specific (i.e., dedicated to a particular active vision task such as reading or scene perception) versus domain-general (i.e., dedicated to all active vision tasks in which eye movements are controlled in the service of perception and cognition). The current study was designed to investigate this issue by examining the similarities and differences in brain activity elicited by two active
Conclusions
The present block-design fMRI study demonstrated common and differential brain activation in two active vision tasks that required sequences of saccadic eye movements, reading and scene viewing. As expected, language-related cortical areas including IFG, STG, and MTG were more active in the reading condition relative to the scene viewing condition, whereas ventral occipito-medial areas including PPA and RSC produced more activation for the scene viewing condition. In addition, anterior areas of
Acknowledgments
This research was supported by the National Science Foundation of the USA (BCS-1151358). We thank William Brixius, Jennifer Olejarczyk, and Joseph Schmidt for their help with data collection.
References (80)
- et al.
Idiosyncratic characteristics of saccadic eye movements when viewing different visual environments
Vis. Res.
(1999) - et al.
Cortical mechanisms specific to explicit visual object recognition
Neuron
(2001) - et al.
Inhibition and generation of saccades: rapid event-related fMRI of prosaccades, antisaccades and no-go trials
Neuroimage
(2006) - et al.
Specialization within the ventral stream: the case for the visual word form area
Neuroimage
(2004) AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages
Comput. Biomed. Res.
(1996)- et al.
The unique role of the visual word form area in reading
Trends Cogn. Sci.
(2011) - et al.
An effect of context on saccade-related behavior and brain activity
NeuroImage
(2007) Parahippocampal and retrosplenial contributions to human spatial navigation
Trends Cogn. Sci.
(2008)- et al.
Reworking the language network
Trends Cogn. Sci.
(2014) - et al.
Orbitofrontal cortex and memory formation
Neuron
(2002)
The neural basis of object perception
Curr. Opin. Neurobiol.
On Broca, brain, and binding: a new framework
Trends Cogn. Sci.
Full scenes produce more activation than close-up scenes and scene-diagnostic objects in parahippocampal and retrosplenial cortex: an fMRI study
Brain Cogn.
Towards natural stimulation in fMRI— issues of data analysis
NeuroImage
The visual word form area: expertise for reading in the fusiform gyrus
Trends Cogn. Sci.
Neurophysiology and neuroanatomy of reflexive and volitional saccades: evidence from studies of humans
Brain Cogn.
Different roles of the parahippocampal place area (PPA) and retrosplenial cortex (RSC) in panoramic scene perception
What is that little voice inside my head? Inner speech phenomenology, its role in cognitive performance, and its relation to self-monitoring
Behav. Brain Res.
A review and synthesis of the first 20 years of PET and fMRI studies of heard speech, spoken language and reading
NeuroImage
The myth of the visual word form area
NeuroImage
The interactive account of ventral occipitotemporal contributions to reading
Trends Cogn. Sci.
Eye movements during information processing tasks: Individual differences and cultural effects
Vis. Res.
A new method for improving functional-to-structural MRI alignment using local Pearson correlation
NeuroImage
Working memory effects on semantic processing: Priming differences in pars orbitalis
Neuroimage
The visual what for area: words and pictures in the left fusiform gyrus
Neuroimage
Eye fields in the frontal lobes of primates
Brain Res. Rev.
Posterior parietal cortex and episodic encoding: insights from fMRI subsequent memory effects and dual-attention theory
Neurobiol. Learn. Mem.
The amodal system for conscious word and picture identification in the absence of a semantic task
Neuroimage
Hierarchical coding of letter strings in the ventral stream: dissecting the inner organization of the visual word-form system
Neuron
Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing
Neuroimage
Functional neural correlates of psychometric schizotypy: an fMRI study of antisaccades
Psychophysiology
Fact vs fiction-how paratextual information shapes our reading processes
Soc. Cogn. Affect. Neurosci.
Functional brain mapping during free viewing of natural scenes
Hum. Brain Map.
Differential sensitivity to words and shapes in ventral occipito-temporal cortex
Cerebr. Cortex
Functional MRI of language: New approaches to understanding the cortical organization of semantic processing
Annu. Rev. Neurosci.
How People Look at Pictures: A Study of Psychology and Perception of Art
Stable individual differences across images in human saccadic eye movements
Can. J. Exp. Psychol.
The neural substrates of natural reading: a comparison of normal and nonword text using eyetracking and fMRI
Front. Hum. Neurosci.
Language-specific tuning of visual cortex? Functional properties of the visual word form area
Brain
Real-time 3D image registration of functional MRI
Magn. Reson. Imag.
Cited by (19)
Individual differences in first-pass fixation duration in reading are related to resting-state functional connectivity
2021, Brain and LanguageCitation Excerpt :Our findings can be linked to a large body of previous research on the neurobiology of reading. In task-state neuroimaging studies, the involvement of the early visual network, the VWFA, and the eye-movement control/dorsal attention network in reading have been well indicated (Bolger, Perfetti, & Schneider, 2005; Choi et al., 2014; Choi & Henderson, 2015; Zhou et al., 2016). In resting-state neuroimaging studies, the RSFCs of these regions/networks have also been found to be related to several behavioural measurements of reading, such as reaction times of single characters (Wang et al., 2012), reading fluency (Zhou et al., 2015; Zhou et al., 2016), and lexical decision scores (Zhou et al., 2016).
Linguistic networks associated with lexical, semantic and syntactic predictability in reading: A fixation-related fMRI study
2019, NeuroImageCitation Excerpt :This region, like the DLPFC, is linked to cognitive control of actions, and is connected to the DLPFC as well as the frontal eye fields (Grahn et al., 2008). Like the DLPFC, it is commonly activated in studies of connected-text reading (Choi and Henderson, 2015; Henderson et al., 2015) and so may also be involved in allowing predictability in reading to influence eye movements. There was also a significant cluster of activation in the bilateral orbitofrontal cortex.
Language structure in the brain: A fixation-related fMRI study of syntactic surprisal in reading
2016, NeuroImageCitation Excerpt :In our study, subjects read natural stories (e.g., The Emperor's New Clothes) presented as paragraphs in an MRI scanner while both eye movements and BOLD signals were recorded. Because effects in an fMRI study may be influenced by inclusion of a secondary task, our general approach is to understand complex cognitive processes in a natural context, an approach that has been used with auditory narratives (Brennan et al., 2010; Wilson et al., 2008), natural reading (Altmann et al., 2014; Choi et al., 2014; Choi and Henderson, 2015), film viewing (Bartels and Zeki, 2004; Hasson et al., 2004; Nishimoto et al., 2011), and active photograph viewing (Choi and Henderson, 2015; Henderson and Choi, 2015; Stansbury et al., 2013). Here we asked participants simply to read naturally for comprehension.
Neural correlates of subsequent memory-related gaze reinstatement
2022, Journal of Cognitive Neuroscience