Research paperMultisensory interactions in primate auditory cortex: fMRI and electrophysiology
Introduction
Our ability to recognize sounds or to understand speech profits considerably from the information provided by the other sensory systems. The best known example for such multisensory benefits of hearing is the cocktail party: with loud music playing and people cheering and chatting we can much better understand somebody speaking when we watch the movements of his lips at the same time. In such situations, the visual information can boost hearing capabilities by an equivalent of about 10–20 dB sound intensity (Ross et al., 2007, Sumby and Polack, 1954), although the exact gain depends on the signal to noise ratio and the kind of target signal itself. However, multisensory input not only improves our understanding of speech, and recent work employing a wide range of behavioral paradigms clearly shows that multisensory stimuli are often processed faster, are easier to recognize and are remembered better than unisensory stimuli (Driver and Spence, 1998, Hershenson, 1962, Lehmann and Murray, 2005, McDonald et al., 2000, Seitz et al., 2006, Vroomen and de Gelder, 2000). As a consequence, multisensory stimulation not only eases our perception during every day interactions, but can also be exploited in learning and rehabilitation programs seeking to improve impoverished sensory capabilities (Oakland et al., 1998, Seitz et al., 2006).
This importance of multisensory input for our ability to hear begs the questions of where and how the brain combines the acoustic and non-acoustic information. Earlier studies had found little evidence for multisensory interactions at early stages of processing and promoted a hierarchical view, suggesting that sensory information converges only in higher association areas such as the superior temporal and intra-parietal sulci and regions in the frontal lobe (Benevento et al., 1977, Bruce et al., 1981, Felleman and Van Essen, 1991, Hikosaka et al., 1988, Hyvarinen and Shelepin, 1979, Jones and Powell, 1970). More recent work, in contrast, emphasizes the importance of lower-level regions and suggests that multisensory interactions already occur at the first processing stages in auditory cortex (Foxe and Schroeder, 2005, Ghazanfar and Schroeder, 2006, Schroeder et al., 2004). In the following we will review the evidence for multisensory influences in primate (human and monkey) auditory cortex, and place particular emphasis on the complementary nature of the imaging and electrophysiological approaches typically employed to localize and characterize multisensory interactions.
While the majority of studies reporting multisensory influences are based on functional magnetic resonance imaging (fMRI) experiments, the neuronal basis underlying this signal is still controversial. As we discuss, the uncertain nature of the fMRI-BOLD signal makes it extremely difficult to extrapolate from functional imaging data to neuronal activity. Hence, functional imaging neither unequivocally resolves whether neurons in a particular region of the brain have access to information from several modalities, nor characterizes potential multisensory interactions at the neuronal level. We argue that electrophysiological studies, which provide direct and spatio-temporally refined measurements of neuronal activity, are required to provide sensitive tests of multisensory interactions in a given brain region.
Section snippets
Functional imaging of multisensory influences in auditory cortex
To address ‘where’ along the auditory processing pathways influences from other sensory modalities first arise, one needs to localize those regions where neuronal activity to acoustic stimulation can be modulated by stimulation of another modality. A simple and direct way of doing so is to compare neuronal responses to the presentation of an acoustic stimulus to the response when the same stimulus is paired with, for instance, a visual stimulus. A seemingly efficient, non-invasive way of doing
Predicting neuronal activity from functional imaging data
To interpret the results of functional imaging in the context of neuronal activity we need to better understand the neuronal substrate that couples the hemodynamic responses to neuronal activity – hence the neural correlate of the imaging signal. The fMRI-BOLD signal reflects cerebral blood flow (CBF) and tissue oxygenation, both of which change in proportion to the local energy demands in or near an image voxel. Following current understanding, this energy demand originates mostly due to peri
Multisensory influences at the level of neuronal activity
Several electrophysiological studies have demonstrated multisensory influences in auditory regions in a number of species, a range of paradigms, and most importantly, to different degrees in the different electrophysiological signals. Recordings of local field potentials or current source densities in auditory regions revealed a widespread influence of visual or somatosensory stimuli on acoustic responses (Ghazanfar et al., 2008, Ghazanfar et al., 2005, Lakatos et al., 2007, Schroeder and Foxe,
Conclusions
The notion that auditory fields in or near primary auditory cortex receive inputs from other modalities and possibly integrate this with the acoustic information has become increasingly popular over the last years (Ghazanfar and Schroeder, 2006). A good deal of this evidence comes from functional imaging experiments. However, as we argue here, this technique provides good means to localize regions of interest, but does not warrant direct conclusions about the properties of individual or
Acknowledgements
This work was supported by the Max Planck Society and the Alexander von Humboldt Foundation.
References (102)
- et al.
The neural basis of functional brain imaging signals
Trends Neurosci.
(2002) - et al.
FMRI and its interpretations: an illustration on directional selectivity in area V5/MT
Trends Neurosci.
(2008) - et al.
Auditory–visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey
Exp. Neurol.
(1977) - et al.
Reduction of information redundancy in the ascending auditory pathway
Neuron
(2006) - et al.
Volumetric vs. surface-based alignment for localization of auditory cortex activation
Neuroimage
(2005) - et al.
Crossmodal attention
Curr. Opin. Neurobiol.
(1998) - et al.
Merging the senses into a robust percept
Trends Cogn. Sci.
(2004) - et al.
Mirror-symmetric tonotopic maps in human primary auditory cortex
Neuron
(2003) - et al.
Multisensory auditory–somatosensory interactions in early cortical processing revealed by high-density electrical mapping
Brain Res. Cogn. Brain Res.
(2000) - et al.
Is neocortex essentially multisensory?
Trends Cogn. Sci.
(2006)
Neurophysiology of the BOLD fMRI Signal in Awake Monkeys
Curr. Biol.
Distribution of visual and somatic functions in the parietal associative area 7 of the monkey
Brain Res.
Integration of touch and sound in auditory cortex
Neuron
Spike-phase coding boosts and stabilizes the information carried by spatial and temporal spike patterns
Neuron
Neuronal oscillations and multisensory interaction in primary auditory cortex
Neuron
Differential patterns of multisensory interactions in core and belt areas of human auditory cortex
Neuroimage
The role of multisensory memories in unisensory object discrimination
Brain Res. Cogn. Brain Res.
Magnetoencephalographic correlates of audiotactile interaction
Neuroimage
Representation of the cochlear partition of the superior temporal plane of the macaque monkey
Brain Res.
Information theory in auditory research
Hear. Res.
Coupling between neuronal firing rate, gamma LFP, and BOLD fMRI is related to interneuronal correlations
Curr. Biol.
Pyramidal neuron as two-layer neural network
Neuron
Cortical processing of complex sounds
Curr. Opin. Neurobiol.
Cytochrome oxidase, acetylcholinesterase, and NADPH-diaphorase staining in human supratemporal and insular cortex: evidence for multiple auditory areas
Neuroimage
The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex
Brain Res. Cogn. Brain Res.
Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing
Int. J. Psychophysiol.
Touch activates human auditory cortex
Neuroimage
Sound facilitates visual learning
Curr. Biol.
Integration of letters and speech sounds in the human brain
Neuron
Audiovisual synchrony perception for music, speech, and object actions
Brain Res.
FMRI retinotopic mapping – step by step
Neuroimage
Low-frequency local field potentials and spikes in primary visual cortex convey independent visual information
J. Neurosci.
Visual speech perception without primary auditory cortex activation
Neuroreport
Information theory and neural coding
Nat. Neurosci.
Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque
J. Neurophysiol.
Reading speech from still and moving faces: the neural substrates of visible speech
J. Cogn. Neurosci.
Response amplification in sensory-specific cortices during crossmodal binding
Neuroreport
Activation of auditory cortex during silent lipreading
Science
Neuronal Responses to Visual Stimuli in Auditory Cortical Areas of Monkeys Performing an Audio–visual Detection Task
Patterns of calcium-binding proteins support parallel and hierarchical organization of human auditory areas
Eur. J. Neurosci.
Compartments within human primary auditory cortex: evidence from cytochrome oxidase and acetylcholinesterase staining
Eur. J. Neurosci.
Comparison of spatial normalization procedures and their impact on functional maps
Hum. Brain Mapp.
fMRI of human visual cortex
Nature
Distributed hierarchical processing in the primate cerebral cortex
Cereb. Cortex
GABAergic regulation of cerebral microvascular tone in the rat
J. Cereb. Blood Flow Metab.
The case for feedforward multisensory convergence during early cortical processing
Neuroreport
Auditory–somatosensory multisensory processing in auditory association cortex: an fMRI study
J. Neurophysiol.
Architectonic analysis of the auditory-related areas of the superior temporal region in human brain
J. Comp. Neurol.
Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys
J. Neurosci.
Cited by (0)
- 1
Present address: The Institute of Neuroscience, University of Newcastle, Newcastle upon Tyne, NE2 4HH, UK.