Skip to main content
SpringerLink
Log in

When audiovisual correspondence disturbs visual processing

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Multisensory integration is known to create a more robust and reliable perceptual representation of one’s environment. Specifically, a congruent auditory input can make a visual stimulus more salient, consequently enhancing the visibility and detection of the visual target. However, it remains largely unknown whether a congruent auditory input can also impair visual processing. In the current study, we demonstrate that temporally congruent auditory input disrupts visual processing, consequently slowing down visual target detection. More importantly, this cross-modal inhibition occurs only when the contrast of visual targets is high. When the contrast of visual targets is low, enhancement of visual target detection is observed, consistent with the prediction based on the principle of inverse effectiveness (PIE) in cross-modal integration. The switch of the behavioral effect of audiovisual interaction from benefit to cost further extends the PIE to encompass the suppressive cross-modal interaction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Alais D, van Boxtel JJ, Parker A, van Ee R (2010) Attending to auditory signals slows visual alternations in binocular rivalry. Vis Res 50:929–935

    Article  PubMed  Google Scholar 

  • Albrecht DG, Hamilton DB (1982) Striate cortex of monkey and cat: contrast response function. J Physiol 48:217–237

    CAS  Google Scholar 

  • Alsius A, Munhall KG (2013) Detection of audiovisual speech correspondences without visual awareness. Psychol Sci 24(4):423–431

    Article  PubMed  Google Scholar 

  • Brainard DH (1997) The psychophysics toolbox. Spatial Vis 10:433–436

    Article  CAS  Google Scholar 

  • Chen YC, Yeh SL, Spence C (2011) Crossmodal constraints on human perceptual awareness: auditory semantic modulation of binocular rivalry. Front Psychol 2:212. doi:10.3389/fpsyg.2011.00212

    PubMed  PubMed Central  Google Scholar 

  • Chong SC, Blake R (2006) Exogenous attention and endogenous attention influence initial dominance in binocular rivalry. Vis Res 46:1794–1803

    Article  PubMed  Google Scholar 

  • Chong SC, Tadin D, Blake R (2005) Endogenous attention prolongs dominance durations in binocular rivalry. J Vis 5:1004–1012

    Article  PubMed  Google Scholar 

  • Conrad V, Bartels A, Kleiner M, Noppeney U (2010) Audiovisual interactions in binocular rivalry. J Vis 10:27

    Article  PubMed  Google Scholar 

  • Conrad V, Vitello MP, Noppeney U (2012) Interactions between apparent motion rivalry in vision and touch. Psychol Sci 23:940–948

    Article  PubMed  Google Scholar 

  • Convento S, Vallar G, Galantini C, Bolognini N (2013) Neuromodulation of early multisensory interactions in the visual cortex. J Cogn Neurosci 25(5):685–696

    Article  PubMed  Google Scholar 

  • Cox D, Hong SW (2015) Semantic-based crossmodal processing during visual suppression. Front Psychol 6:722. doi:10.3389/fpsyg.2015.00722

    Article  PubMed  PubMed Central  Google Scholar 

  • Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66:188–1404

    Article  Google Scholar 

  • Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on ‘‘sensory specific” brain regions, neural responses, and judgments. Neuron 57:11–23

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Fiebelkorn IC, Foxe JJ, Butler JS, Molholm S (2011) Auditory facilitation of visual-target detection persists regardless of retinal eccentricity and despite wide audiovisual misalignments. Exp Brain Res 213:167–174

    Article  PubMed  Google Scholar 

  • Gayet S, Van der Stigchel S, Paffen CLE (2014) Breaking continuous flash suppression: competing for consciousness on the pre-semantic battlefield. Front Psychol 5:460. doi:10.3389/fpsyg.2014.00460

    Article  PubMed  PubMed Central  Google Scholar 

  • Guzman-Martinez E, Ortega L, Grabowecky M, Mossbridge J, Suzuki S (2012) Interactive coding of visual spatial frequency and auditory amplitude-modulation rate. Curr Biol 22:383–388

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Iurilli G, Ghezzi D, Olcese U, Lassi G, Nazzaro C, Tonini R, Tucci V, Benfenati F, Medini P (2012) Sound-driven synaptic inhibition in primary visual cortex. Neuron 73:814–828

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Kadunce DC, Vaughan JW, Wallace MT, Benedek G, Stein BE (1997) Mechanisms of within- and cross-modality suppression in the superior colliculus. J Neurophysiol 78(6):2834–2847

    CAS  PubMed  Google Scholar 

  • Kang M-S, Blake R, Woodman GF (2011) Semantic analysis does not occur in the absence of awareness induced by interocular suppression. J Neurosci 31:13535–13545

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Keetels M, Stekelenburg J, Vroomen J (2007) Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism. Exp Brain Res 180:449–456

    Article  PubMed  PubMed Central  Google Scholar 

  • Ling S, Carrasco M (2006) When sustained attention impairs perception. Nat Neurosci 9:1243–1245

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Lippert M, Logothetis NK, Kayser C (2007) Improvement of visual contrast detection by a simultaneous sound. Brain Res 1173:102–109

    Article  CAS  PubMed  Google Scholar 

  • Lunghi C, Alais D (2013) Touch interacts with vision during binocular rivalry with a tight orientation tuning. PLoS One 8:e58754

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Lunghi C, Alais D (2015) Congruent tactile stimulation reduces the strength of visual suppression during binocular rivalry. Sci Rep 5:9413. doi:10.1038/srep09413

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Lunghi C, Binda P, Morrone MC (2010) Touch disambiguates rivalrous perception at early stages of visual analysis. Curr Biol 20:R143–R144

    Article  CAS  PubMed  Google Scholar 

  • Lunghi C, Morrone MC, Alais D (2014) Auditory and tactile signals combine to influence vision during binocular rivalry. J Neurosci 34(3):784–792

    Article  CAS  PubMed  Google Scholar 

  • Meredith MA, Stein BE (1986) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662

    CAS  PubMed  Google Scholar 

  • Moors P, Huygelier H, Wagemans J, de Wit L, van Ee R (2015) Suppressed visual looming stimuli are not integrated with auditory looming signals: evidence from continuous flash suppression. Perception 6:48–62

    Article  Google Scholar 

  • Noesselt T, Rieger JW, Schoenfeld MA, Kanowski M, Hinrichs H, Heinze HJ, Driver J (2007) Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. J Neurosci 27:11431–11441

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Noesselt T, Bergmann D, Hake M, Heinze HJ, Fendrich R (2008) Sound increases the saliency of visual events. Brain Res 1220:157–163

    Article  CAS  PubMed  Google Scholar 

  • Noesselt T, Tyll S, Boehler CN, Heinze HJ, Driver J (2010) Sound-induced enhancement of low-intensity vision: multisensory influences on human sensory-specific cortices and thalamic bodies relate to perceptual enhancement of visual detection sensitivity. J Neurosci 30:13609–13623

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Paffen CL, Alais D, Verstraten FA (2006) Attention speeds binocular rivalry. Psychol Sci 17:752–756

    Article  PubMed  Google Scholar 

  • Pelli DG (1997) The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spatial Vis 10:437–442

    Article  CAS  Google Scholar 

  • Plass J, Guzman-Martinez E, Ortega L, Grabowecky M, Suzuki S (2014) Lipreading without awareness. Psychol Sci 25(9):1835–1837

    Article  PubMed  PubMed Central  Google Scholar 

  • Romei V, Murray MM, Cappe C, Thut G (2009) Preperceptual and stimulus-selective enhancement of low-level human visual cortex excitability by sounds. Curr Biol 19:1799–1805

    Article  CAS  PubMed  Google Scholar 

  • Romei V, Gross J, Thut G (2012) Sounds reset rhythms of visual cortex and corresponding human visual perception. Curr Biol 22:807–813

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Stanford TR, Quessy S, Stein BE (2005) Evaluating the operations underlying multisensory integration in the cat superior colliculus. J Neurosci 25:6499–6508

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge

    Google Scholar 

  • Stein T, Sterzer P (2014) Unconscious processing under interocular suppression: getting the right measure. Front Psychol 5:387. doi:10.3389/fpsyg.2014.00387

    PubMed  PubMed Central  Google Scholar 

  • Stein BE, London N, Wilkonson LK, Price DD (1996) Enhancement of perceived visual intensity by auditory stimuli: a psychophysical analysis. J Cogn Neurosci 8(6):497–506

    Article  CAS  PubMed  Google Scholar 

  • Talsma D, Senkowski D, Soto-Faraco S, Woldorff MG (2010) The multifaceted interplay between attention and multisensory integration. Trends Cogn Sci 14(9):400–410

    Article  PubMed  PubMed Central  Google Scholar 

  • Tsuchiya N, Koch C (2005) Continuous flash suppression reduces negative afterimages. Nat Neurosci 8(8):1096–1101

    Article  CAS  PubMed  Google Scholar 

  • Tsuchiya N, Koch C, Gilroy L, Blake R (2006) Depth of interocular suppression associated with continuous flash suppression, flash suppression, and binocular rivalry. J Vis 6:1068–1078

    Article  PubMed  Google Scholar 

  • Van der Burg E, Olivers CNL, Bronkhorst AW, Theeuwes J (2008) Pip and Pop: nonspatial Auditory Signals Improve Spatial Visual Search. J Exp Psychol Hum Percep Perform 34:1053–1065

    Article  Google Scholar 

  • van Ee R, van Boxtel JJ, Parker AL, Alais D (2009) Multisensory congruency as a mechanism for attentional control over perceptual selection. J Neurosci 29:11641–11649

    Article  PubMed  Google Scholar 

  • Vroomen J, de Gelder B (2000) Sound enhances visual perception: cross-modal effects of auditory organization on vision. J Exp Psychol Hum Percep Perform 26(5):1583–1590

    Article  CAS  Google Scholar 

  • Werner S, Noppeney U (2010) Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cereb Cortex 20:1829–1842

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Florida Atlantic University, 777 Glades Rd., BS 12, Room 209, Boca Raton, FL, 33431, USA

    Sang Wook Hong

  2. Center for Complex Systems and Brain Sciences, Florida Atlantic University, Boca Raton, FL, USA

    Sang Wook Hong

  3. Department of Psychology and Brain Sciences, Dartmouth College, HB 6207 Moore Hall, Hanover, NH, 03755, USA

    Won Mok Shim

  4. Center for Neuroscience Imaging Research, Institute for Basic Science, Sungkyunkwan University, Suwon, Korea

    Won Mok Shim

  5. Department of Biomedical Engineering, Sungkyunkwan University, Suwon, Korea

    Won Mok Shim

Authors
  1. Sang Wook Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Won Mok Shim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sang Wook Hong.

Ethics declarations

Conflict of interest

The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hong, S.W., Shim, W.M. When audiovisual correspondence disturbs visual processing. Exp Brain Res 234, 1325–1332 (2016). https://doi.org/10.1007/s00221-016-4591-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-016-4591-y

Keywords

Search

Navigation