Audio-visual synchrony modulates the ventriloquist illusion and its neural/spatial representation in the auditory cortex
Introduction
Our sensory systems are interconnected so as to integrate stimuli in different modalities across space and time to achieve coherent percepts of environmental events. Recent neurophysiological studies of multisensory integration in humans have indicated that cross-modal interactions occur not only in multisensory cortical areas such as the parietal- and superior temporal cortex, but also in regions normally considered to be unisensory (for review, see Driver and Noesselt, 2008). Studies in both human and nonhuman primates have shown that the auditory cortical processing can be influenced by visual (e.g. Kayser et al., 2007, Kayser et al., 2008, Kayser et al., 2009) as well as by tactile (e.g. Bruns and Röder, 2010, Lakatos et al., 2007, Murray et al., 2005, Occelli et al., 2012) input. These cross modal interactions within sensory-specific regions may be mediated by feedback connections from higher-level multisensory regions or by direct connections between regions that are normally considered to be unisensory (Altmann et al., 2007, Falchier et al., 2002, Noesselt et al., 2007).
Behaviorally, crossmodal interactions can lead to illusory percepts such as the ventriloquist illusion, whereby sounds are mislocalized towards a spatially discrepant visual event (for review, see Vroomen and de Gelder, 2004). Crucially, this effect depends upon audio-visual temporal synchrony (Slutsky and Recanzone, 2001), being less effective in creating the illusion when the visual stimulus precedes the sound; this decrease of illusory shifting of sound localization due to audio-visual asynchrony follows an almost linear function of stimulus onset asynchrony with larger asynchronies leading to less ventriloquist illusions. More recently, a formalized theory of the window of integration has been proposed (TWIN theory; e.g., Colonius and Diederich, 2012), which includes a mathematical model of the behavioral effects. However, little is known about the neural mechanisms by which a concurrent visual stimulus produces a shift in perceived sound location in humans. Several studies have examined neural interactions between spatially disparate auditory and visual stimuli (Busse et al., 2005, Gondan et al., 2005, Teder-Salejarvi et al., 2005) without finding evidence for a visual influence on the auditory cortex that could be directly linked to the illusion of a shifted auditory percept.
In a combined event-related potential (ERP)/fMRI experiment, Bonath et al. (2007) identified a putative neural correlate of the ventriloquist illusion. ERP recordings showed that a long-latency N260 component was reduced in amplitude over the hemisphere ipsilateral to the side of the illusory shift of sound location. The neural generator of this N260 component was localized by source modeling within auditory belt regions of the superior temporal lobe. To corroborate this electrophysiological source localization, a parallel fMRI experiment with identical parameters as used in the ERP experiment showed that leftward and rightward ventriloquist illusions for central sounds paired with lateralized visual stimuli correlated with a reduced BOLD-signal within the auditory cortex ipsilateral to the side of the perceived sound location. Previous fMRI studies had shown that this same auditory belt region was spatio-topically organized, with a bias for contralateral sound processing (Deouell et al., 2007, Krumbholz et al., 2005, Warren and Griffiths, 2003). Based on these findings, Bonath et al. (2007) hypothesized that this lateralized biasing of the auditory cortex activity represents a neural code for the ventriloquist illusion. The current study aimed to test this hypothesis by manipulating the strength of the ventriloquist illusion using asynchronous audio-visual stimulus parings. One possible outcome would be that the reduction of the ventriloquist effect for asynchronous pairings, as reported by Slutsky and Recanzone (2001), would be associated with a decrease of the lateralized BOLD response within the posterior auditory cortex reported by Bonath et al. (2007). Alternatively, the neural coding of the ventriloquist illusion might involve different sub-regions within the auditory cortex for synchronous versus asynchronous audio-visual pairings (see Hoefer et al., 2013 for similar findings associated with asynchronous vs. synchronous audio–tactile interplay). There is indeed reason to believe that different auditory regions may have wider or narrower audio-visual temporal response windows (Camalier et al., 2012, Inui et al., 2005). Such an anatomical separation of neural responses for synchronous and asynchronous ventriloquist illusions would suggest that the brain flexibly recruits auditory regions with audio-visual temporal integration windows best suited to solve the task at hand. To investigate these questions, audio-visual stimulus combinations were presented either synchronously or asynchronously while subjects performed an auditory localization task. Both perceived shifts of central sounds towards the periphery and shifts of lateralized sounds towards the center were examined while fMRI analysis was carried out in sensory-specific auditory cortex.
Section snippets
Subjects
Twenty right-handed subjects (eleven females, age range 20–34 years old) participated in the experiment. All subjects were paid and gave written informed consent in accord with the local ethics committee. Subjects had no history of neurological or psychiatric illness and normal or corrected-to-normal vision and normal hearing.
Stimulus presentation
Visual and auditory stimuli were presented using Presentation 9.11 (Neurobehavioral Systems, Inc., CA). Pure sounds (500 Hz, 80 dB, 30 ms duration) were presented from
Behavioral results
Subjects correctly localized left (AL), center (AC) and right (AR) sounds on 77% (+/− 3.4% SE) of the trials. For the analysis of auditory accuracy, a one-way repeated measurement ANOVA with the factor ‘sound position’ did not reveal a significant main effect (see Fig. 2A; F(1.6; 31) = 0.8 p = 0.43, Greenhouse–Geisser correction). Thus, subjects localized sounds from the different speakers equally well. Fig. 2B depicts the auditory localizations (left, center, right) for the audio-visual stimulus
Discussion
The present study used fMRI to explore the neural basis of the ventriloquist illusion and its modulation by temporal delays between visual and auditory stimulations. While subjects were in the scanner brief sounds were presented to left, right, or central locations, accompanied by flashes at different locations to induce the ventriloquist effect. The subject's task was to report the perceived sound location and ignore the flashes. Strong ventriloquist effects were observed for all stimulus
Conclusion
In sum, the present results demonstrate that for central sounds illusory sound localization towards the position of a flash in either the left or to the right hemifield is related to reduced BOLD-responses within the auditory cortex ipsilateral to the perceived sound location. For peripheral sounds, on the other hand, illusory sound localization towards the center is related to increased BOLD-responses within the auditory cortex ipsilateral to the sound location. Moreover, these results show
Funding
This research was supported by grants from the DFG No 451/2-1 (T.N.;B.B.), the U.S. National Science Foundation (BCS-1029084) and the National Institute of Mental Health (1P50MH86385) (S.A.H.), and the SFB 779 TP A03 (K.K.; B.B.).
Author contributions
BB and ST acquired and analyzed the data. BB, TN, and SAH designed the study. TN, BB, KK, CT, and SAH wrote the manuscript.
The authors declare no competing financial interests.
References (55)
- et al.
Processing of location and pattern changes of natural sounds in the human auditory cortex
Neuroimage
(2007) - et al.
Response preferences for “what” and “where” in human non-primary auditory cortex
Neuroimage
(2006) Ventriloquism: A case of cross-modal perceptual grouping
- et al.
Utilizing the ventriloquist-effect to investigate audio-visual binding
Neuropsychologia
(2007) - et al.
Neural basis of the ventriloquist illusion
Curr. Biol.
(2007) - et al.
Anatomical connections suitable for the direct processing of neuronal information of different modalities via the rodent primary auditory cortex
Hear. Res.
(2009) - et al.
Neural correlates of sound externalization
Neuroimage
(2013) - et al.
Cerebral responses to change in spatial location of unattended sounds
Neuron
(2007) - et al.
Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments
Neuron
(2008) - et al.
Multisensory integration: vision boosts information through suppression in auditory cortex
Curr. Biol.
(2010)
Spatial sensitivity of neurons in the anterior, posterior, and primary fields of cat auditory cortex
Hear. Res.
Tactile stimulation and hemispheric asymmetries modulate auditory perception and neural responses in primary auditory cortex
Neuroimage
Processing of binaural spatial information in human auditory cortex: neuromagnetic responses to interaural timing and level differences
Neuropsychologia
Endoscopic eye tracking system for fMRI
J. Neurosci. Methods
Neuronal oscillations and multisensory interaction in primary auditory cortex
Neuron
Human auditory cortical mechanisms of sound lateralization: II. Interaural time differences at sound onset
Hear. Res.
Audiotactile integration is reduced in congenital blindness in a spatial ventriloquism task
Neuropsychologia
Spatial processing in human auditory cortex: the effects of 3D, ITD, and ILD stimulation techniques
Brain Res. Cogn. Brain Res.
Human auditory cortical mechanisms of sound lateralization: I. Interaural time differences within sound
Hear. Res.
Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing
Int. J. Psychophysiol.
Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations
Neuropsychologia
Neuromagnetic recordings reveal the temporal dynamics of auditory spatial processing in the human cortex
Neurosci. Lett.
Encoding of sound-source location and movement: activity of single neurons and interactions between adjacent neurons in the monkey auditory cortex
J. Neurophysiol.
Tactile capture of auditory localization: an event-related potential study
Eur. J. Neurosci.
The spread of attention across modalities and space in a multisensory object
Proc. Natl. Acad. Sci. U. S. A.
Neural latencies across auditory cortex of macaque support a dorsal stream supramodal timing advantage in primates
Proc. Natl. Acad. Sci. U. S. A.
Models of the Time Window of Integration
Cited by (23)
Reduced multisensory integration of self-initiated stimuli
2019, CognitionThe role of auditory cortex in the spatial ventriloquism aftereffect
2017, NeuroImageCitation Excerpt :Bruns and colleagues hypothesized that the relatively early ERP modulation associated with the VAE suggests an involvement of low-level auditory cortical processing. Functional magnetic resonance imaging (fMRI) studies of the VE provided evidence that an asymmetry between the activity in left- and right-hemispheric auditory areas might represent a mechanism by which sound azimuth is coded, with auditory areas being more sensitive to contralateral than ipsilateral sounds (Bonath et al., 2014, 2007; Callan et al., 2015). Importantly, in the VE situation it has been shown that the spatially disparate visual stimulus influences this asymmetry by attenuating auditory cortex activity in one hemisphere, such that the asymmetry corresponded with the perceived sound location.
Metacognition in Multisensory Perception
2016, Trends in Cognitive SciencesDistinct computational principles govern multisensory integration in primary sensory and association cortices
2016, Current BiologyCitation Excerpt :Likewise, IPS3–4 was the only region where perceptual report was decoded significantly better than chance. Previous studies focusing selectively on the auditory ventriloquist illusion have highlighted the importance of the planum temporale in audiovisual integration at the perceptual level [21, 38]. Indeed, our study also revealed a high correlation between neural and behavioral weights indices in higher auditory areas.
How prior expectations shape multisensory perception
2016, NeuroImage
- 1
These authors contributed equally to this work.