In everyday experience we are exposed to complex stimuli that impact sensorial receptors in more than one sensory modality. For example, human social interactions through language necessarily imply integration of auditory information with visual information referring to facial expression and gesture. In such a context, selective attention seems to operate favoring some of those stimuli on the basis of preexisting contextual information and physical or psychological salience, among other criteria. Although most studies of attentional mechanisms are still conducted using stimulation of a single sensory modality, which allows simpler hypothesis testing and minimizes confounding factors, in the last decade we have witnessed great developments in the study of so-called crossmodal attention. Evidence has accrued that both conflicting and congruent information presented to different sensory modalities interferes or influences the processing at each modality at different levels. Behavioral and electrophysiological studies support the existence of early crossmodal integration between almost all sensory modalities and stable crossmodal plasticity as a long-term effect. In this context, event-related potentials have been extensively used, especially owing to some advantages that this technique offers for the study of attentional mechanisms, such as its high temporal resolution. Attentional modulation of event-related potentials elicited by crossmodal stimuli partially resembles what has been described using monomodal stimulation. For example, an early negativity named “Nd” increases its amplitude when the eliciting stimulus is attended. In the present chapter, we attempt to revise the most recent developments in this cutting-edge research area and their implications for the theoretical framework in which we understand attention and attentional processes.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Alsius, A., Navarra, J., Campbell, R., & Soto-Faraco, S. (2005). Audiovisual integration of speech falters under high attention demands. Current Biology, 15(9), 839–843
Alsius, A., Navarra, J., & Soto-Faraco, S. (2007). Attention to touch weakens audiovisual speech integration. Experimental Brain Research, 183(3), 399–404
Aschersleben, G., & Bertelson, P. (2003). Temporal ventriloquism: Crossmodal interaction on the time dimension. 2. Evidence from sensorimotor synchronization. International Journal of Psychophysiology, 50(1–2), 157–163
Barraclough, N. E., Xiao, D., Baker, C. I., Oram, M. W., & Perrett, D. I. (2005). Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. Journal of Cognitive Neuroscience, 17(3), 377–391
Bernstein, L. E., Auer, E. T., Jr., Wagner, M., & Ponton, C. W. (2008). Spatiotemporal dynamics of audiovisual speech processing. Neuroimage, 39(1), 423–435
Bertelson, P., & Aschersleben, G. (2003). Temporal ventriloquism: Crossmodal interaction on the time dimension. 1. Evidence from auditory-visual temporal order judgment. International Journal of Psychophysiology, 50(1–2), 147–155
Bertelson, P., Vroomen, J., de Gelder, B., & Driver, J. (2000). The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception and Psychophysics, 62(2), 321–332
Bizley, J. K., Nodal, F. R., Bajo, V. M., Nelken, I., & King, A. J. (2007). Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cerebral Cortex, 17(9), 2172–2189
Bonath, B., Noesselt, T., Martinez, A., Mishra, J., Schwiecker, K., Heinze, H. J., et al. (2007). Neural basis of the ventriloquist illusion. Current Biology, 17(19), 1697–1703
Budinger, E., Heil, P., Hess, A., & Scheich, H. (2006). Multisensory processing via early cortical stages: Connections of the primary auditory cortical field with other sensory systems. Neuroscience, 143(4), 1065–1083
Bushara, K. O., Grafman, J., & Hallett, M. (2001). Neural correlates of auditory-visual stimulus onset asynchrony detection. Journal of Neuroscience, 21(1), 300–304
Busse, L., Roberts, K. C., Crist, R. E., Weissman, D. H., & Woldorff, M. G. (2005). The spread of attention across modalities and space in a multisensory object. Proceedings of the National Academy of Sciences of the United States of America, 102(51), 18751–18756
Calvert, G. A. (2001). Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex, 11(12), 1110–1123
Calvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Williams, S. C., McGuire, P. K., et al. (1997). Activation of auditory cortex during silent lipreading. Science, 276(5312), 593–596
Calvert, G. A., Hansen, P. C., Iversen, S. D., & Brammer, M. J. (2001). Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage, 14(2), 427–438
Cappe, C., & Barone, P. (2005). Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey. European Journal of Neuroscience, 22(11), 2886–2902
Colavita, F. B., & Weisberg, D. (1979). A further investigation of visual dominance. Perception and Psychophysics, 25(4), 345–347
Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews Neuroscience, 3(3), 201–215
De Gelder, B., & Bertelson, P. (2003). Multisensory integration, perception and ecological validity. Trends in Cognitive Science, 7(10), 460–467
Dematte, M. L., Osterbauer, R., & Spence, C. (2007). Olfactory cues modulate facial attractiveness. Chemical Senses, 32(6), 603–610
Doyle, M. C., & Snowden, R. J. (2001). Identification of visual stimuli is improved by accompanying auditory stimuli: The role of eye movements and sound location. Perception, 30(7), 795–810
Driver, J. (1996). Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature, 381(6577), 66–68
Driver, J., & Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron, 57(1), 11–23
Driver, J., & Spence, C. (1998). Crossmodal attention. Current Opinion in Neurobiology, 8(2), 245–253
Eimer, M. (2004). Multisensory integration: How visual experience shapes spatial perception. Current Biology, 14(3), R115–117
Eimer, M., & Driver, J. (2000). An event-related brain potential study of cross-modal links in spatial attention between vision and touch. Psychophysiology, 37(5), 697–705
Eimer, M., & Driver, J. (2001). Crossmodal links in endogenous and exogenous spatial attention: Evidence from event-related brain potential studies. Neuroscience Biobehavioral Review, 25(6), 497–511
Eimer, M., & Van Velzen, J. (2002). Crossmodal links in spatial attention are mediated by supramodal control processes: Evidence from event-related potentials. Psychophysiology, 39(4), 437–449
Eimer, M., van Velzen, J., & Driver, J. (2002). Cross-modal interactions between audition, touch, and vision in endogenous spatial attention: ERP evidence on preparatory states and sensory modulations. Journal of Cognitive Neuroscience, 14(2), 254–271
Eimer, M., van Velzen, J., & Driver, J. (2004). ERP evidence for cross-modal audiovisual effects of endogenous spatial attention within hemifields. Journal of Cognitive Neuroscience, 16(2), 272–288
Eimer, M., van Velzen, J., Forster, B., & Driver, J. (2003). Shifts of attention in light and in darkness: An ERP study of supramodal attentional control and crossmodal links in spatial attention. Brain Research. Cognitive Brain Research, 15(3), 308–323
Escera, C., Yago, E., Corral, M. J., Corbera, S., & Nunez, M. I. (2003). Attention capture by auditory significant stimuli: Semantic analysis follows attention switching. European Journal of Neuroscience, 18(8), 2408–2412
Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002). Anatomical evidence of multimo- dal integration in primate striate cortex. Journal of Neuroscience, 22(13), 5749–5759
Fort, A., Delpuech, C., Pernier, J., & Giard, M. H. (2002). Dynamics of cortico-subcortical cross- modal operations involved in audio-visual object detection in humans. Cerebral Cortex, 12(10), 1031–1039
Foxe, J. J., & Schroeder, C. E. (2005). The case for feedforward multisensory convergence during early cortical processing. Neuroreport, 16(5), 419–423
Foxe, J. J., Wylie, G. R., Martinez, A., Schroeder, C. E., Javitt, D. C., Guilfoyle, D., et al. (2002). Auditory-somatosensory multisensory processing in auditory association cortex: An fMRI study. Journal of Neurophysiology, 88(1), 540–543
Gazzaniga, M. S., Ivry, R. B., & Mangun, G. R. (2002). Cognitive neuroscience: The biology of the mind (2nd ed.). New York: Norton
Ghazanfar, A. A., & Schroeder, C. E. (2006). Is neocortex essentially multisensory? Trends in Cognitive Science, 10(6), 278–285
Giard, M. H., & Peronnet, F. (1999). Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience, 11(5), 473–490
Guest, S., Catmur, C., Lloyd, D., & Spence, C. (2002). Audiotactile interactions in roughness perception. Experimental Brain Research, 146(2), 161–171
Hackett, T. A., De La Mothe, L. A., Ulbert, I., Karmos, G., Smiley, J., & Schroeder, C. E. (2007). Multisensory convergence in auditory cortex, II. Thalamocortical connections of the caudal superior temporal plane. The Journal of Comparative Neurology, 502(6), 924–952
Harter, M. R., & Anllo-Vento, L. (1991). Visual-spatial attention: Preparation and selection in children and adults. Electroencephalography and Clinical Neurophysiology Supplement, 42, 183–194
Herrmann, C. S., & Knight, R. T. (2001). Mechanisms of human attention: Event-related potentials and oscillations. Neuroscience Biobehavior Review, 25(6), 465–476
Kayser, C., & Logothetis, N. K. (2007). Do early sensory cortices integrate cross-modal information? Brain Structure and Function, 212(2), 121–132
Kayser, C., Petkov, C. I., & Logothetis, N. K. (2008). Visual Modulation of Neurons in Auditory Cortex. Cerebral Cortex, 18(7), 1560–1574
Koppen, C., & Spence, C. (2007). Audiovisual asynchrony modulates the Colavita visual dominance effect. Brain Research, 1186, 224–232
Lakatos, P., Chen, C. M., O'Connell, M. N., Mills, A., & Schroeder, C. E. (2007). Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron, 53(2), 279–292
Luck, S. J., Woodman, G. F., & Vogel, E. K. (2000). Event-related potential studies of attention. Trends in Cognitive Science, 4(11), 432–440
Macaluso, E. (2006). Multisensory processing in sensory-specific cortical areas. Neuroscientist,12(4), 327–338
Macaluso, E., Eimer, M., Frith, C. D., & Driver, J. (2003). Preparatory states in crossmodal spatial attention: Spatial specificity and possible control mechanisms. Experimental Brain Research, 149(1), 62–74
Macaluso, E., Frith, C. D., & Driver, J. (2000). Modulation of human visual cortex by crossmodal spatial attention. Science, 289(5482), 1206–1208
Macaluso, E., George, N., Dolan, R., Spence, C., & Driver, J. (2004). Spatial and temporal factors during processing of audiovisual speech: A PET study. Neuroimage, 21(2), 725–732
McDonald, J. J., Teder-Salejarvi, W. A., Di Russo, F., & Hillyard, S. A. (2003). Neural substrates of perceptual enhancement by cross-modal spatial attention. Journal of Cognitive Neuroscience, 15(1), 10–19
McDonald, J. J., Teder-Salejarvi, W. A., & Hillyard, S. A. (2000). Involuntary orienting to sound improves visual perception. Nature, 407(6806), 906–908
McDonald, J. J., Teder-Salejarvi, W. A., & Ward, L. M. (2001). Multisensory integration and crossmodal attention effects in the human brain. Science, 292(5523), 1791
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746–748
Miller, J. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14(2), 247–279
Miller, J. (1991). Channel interaction and the redundant-targets effect in bimodal divided attention. Journal of Experimental Psychology: Human Perception and Performance, 17(1), 160–169
Mishra, J., Martinez, A., Sejnowski, T. J., & Hillyard, S. A. (2007). Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. Journal of Neuroscience, 27(15), 4120–4131
Molholm, S., Martinez, A., Shpaner, M., & Foxe, J. J. (2007). Object-based attention is multisen- sory: co-activation of an object's representations in ignored sensory modalities. European Journal of Neuroscience, 26(2), 499–509
Molholm, S., Ritter, W., Murray, M. M., Javitt, D. C., Schroeder, C. E., & Foxe, J. J. (2002). Multisensory auditory-visual interactions during early sensory processing in humans: a high- density electrical mapping study. Brain Research. Cognitive Brain Research, 14(1), 115–128
Pekkola, J., Ojanen, V., Autti, T., Jaaskelainen, I. P., Mottonen, R., & Sams, M. (2006). Attention to visual speech gestures enhances hemodynamic activity in the left planum temporale. Human Brain Mapping, 27(6), 471–477
Populin, L. C., & Yin, T. C. (2002). Bimodal interactions in the superior colliculus of the behaving cat. J Neurosci, 22(7), 2826–2834
Posner, M. I. (1980). Orienting of attention. Quartely Journal of Experimental Psychology, 32(1), 3–25
Posner, M. I., & Dehaene, S. (2000). Attentional networks. In M. S. Gazzaniga (Ed.), Cognitive neuroscience a reader (pp. 156–164). Oxford: Blackwell Publishers Ltd
Puce, A., Allison, T., Bentin, S., Gore, J. C., & McCarthy, G. (1998). Temporal cortex activation in humans viewing eye and mouth movements. Journal of Neuroscience, 18(6), 2188–2199
Raab, D. H. (1962). Statistical facilitation of simple reaction times. Transaction of New York Academy of Science, 24, 574–590
Rockland, K. S., & Ojima, H. (2003). Multisensory convergence in calcarine visual areas in macaque monkey. International Journal of Psychophysiology, 50(1–2), 19–26
Rouger, J., Fraysse, B., Deguine, O., & Barone, P. (2008). McGurk effects in cochlear-implanted deaf subjects. Brain Research, 1188, 87–99
Saint-Amour, D., De Sanctis, P., Molholm, S., Ritter, W., & Foxe, J. J. (2007). Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia, 45(3), 587–597
Scholl, B. J. (2001). Objects and attention: The state of the art. Cognition, 80(1–2), 1–46
Schroeder, C. E., Lindsley, R. W., Specht, C., Marcovici, A., Smiley, J. F., & Javitt, D. C. (2001). Somatosensory input to auditory association cortex in the macaque monkey. Journal of Neurophysiology, 85(3), 1322–1327
Schroger, E., & Widmann, A. (1998). Speeded responses to audiovisual signal changes result from bimodal integration. Psychophysiology, 35(6), 755–759
Sekuler, R., Sekuler, A. B., & Lau, R. (1997). Sound alters visual motion perception. Nature, 385(6614), 308
Shams, L., Kamitani, Y., & Shimojo, S. (2000). Illusions. What you see is what you hear. Nature, 408(6814), 788
Shams, L., Kamitani, Y., & Shimojo, S. (2002). Visual illusion induced by sound. Brain Research.Cognitive Brain Research, 14(1), 147–152
Shimojo, S., & Shams, L. (2001). Sensory modalities are not separate modalities: Plasticity and interactions. Current Opinion in Neurobiology, 11(4), 505–509
Small, D. M., & Prescott, J. (2005). Odor/taste integration and the perception of flavor. Experimental Brain Research, 166(3–4), 345–357
Soto-Faraco, S., Lyons, J., Gazzaniga, M., Spence, C., & Kingstone, A. (2002). The ventriloquist in motion: Illusory capture of dynamic information across sensory modalities. Brain Research. Cognitive Brain Research, 14(1), 139–146
Spence, C., & Driver, J. (2000). Attracting attention to the illusory location of a sound: Reflexive crossmodal orienting and ventriloquism. Neuroreport, 11(9), 2057–2061
Stanford, T. R., & Stein, B. E. (2007). Superadditivity in multisensory integration: Putting the computation in context. Neuroreport, 18(8), 787–792
Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge, MA: MIT
Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18, 643–661
Sussman, E., Winkler, I., Huotilainen, M., Ritter, W., & Naatanen, R. (2002). Top-down effects can modify the initially stimulus-driven auditory organization. Brain Research. Cognitive Brain Research, 13(3), 393–405
Sussman, E., Winkler, I., & Schroger, E. (2003). Top-down control over involuntary attention switching in the auditory modality. Psychonomic Bulletin and Review, 10(3), 630–637
Talsma, D., Doty, T. J., & Woldorff, M. G. (2007). Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration? Cerebral Cortex, 17(3), 679–690
Talsma, D., & Woldorff, M. G. (2005). Selective attention and multisensory integration: Multiple phases of effects on the evoked brain activity. The Journal of Cognitive Neuroscience, 17(7), 1098–1114
Teder-Salejarvi, W. A., Di Russo, F., McDonald, J. J., & Hillyard, S. A. (2005). Effects of spatial congruity on audio-visual multimodal integration. The Journal of Cognitive Neuroscience, 17(9), 1396–1409
Teder-Salejarvi, W. A., McDonald, J. J., Di Russo, F., & Hillyard, S. A. (2002). An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Brain Research. Cognitive Brain Research, 14(1), 106–114
Teder-Salejarvi, W. A., Munte, T. F., Sperlich, F., & Hillyard, S. A. (1999). Intra-modal and cross- modal spatial attention to auditory and visual stimuli. An event-related brain potential study. Brain Research. Cognitive Brain Research, 8(3), 327–343
van Atteveldt, N. M., Formisano, E., Goebel, R., & Blomert, L. (2007). Top-down task effects overrule automatic multisensory responses to letter-sound pairs in auditory association cortex. Neuroimage, 36(4), 1345–1360
Vroomen, J., Bertelson, P., & de Gelder, B. (2001). The ventriloquist effect does not depend on the direction of automatic visual attention. Perception and Psychophysics, 63(4), 651–659
Vroomen, J., & de Gelder, B. (2000). Sound enhances visual perception: Cross-modal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance, 26(5), 1583–1590
Walla, P. (2008). Olfaction and its dynamic influence on word and face processing: Cross-modal integration. Progess in Neurobiology, 84(2), 192–209
Watanabe, K., & Shimojo, S. (1998). Attentional modulation in perception of visual motion events. Perception, 27(9), 1041–1054
Watanabe, K., & Shimojo, S. (2001). When sound affects vision: Effects of auditory grouping on visual motion perception. Psychology Science, 12(2), 109–116
White, T. L., & Prescott, J. (2007). Chemosensory cross-modal stroop effects: Congruent odors facilitate taste identification. Chemical Senses, 32(4), 337–341
Zampini, M., Torresan, D., Spence, C., & Murray, M. M. (2007). Auditory-somatosensory multi- sensory interactions in front and rear space. Neuropsychologia, 45(8), 1869–1877
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Ortega, R., López, V. (2009). Crossmodal Attention–The Contribution of Event-Related-Potential Studies. In: Aboitiz, F., Cosmelli, D. (eds) From Attention to Goal-Directed Behavior. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-70573-4_6
Download citation
DOI: https://doi.org/10.1007/978-3-540-70573-4_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-70572-7
Online ISBN: 978-3-540-70573-4
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)