Abstract
This chapter reviews our research investigating how the processing of unisensory visual objects is impacted by past experiences with multisensory auditory–visual versions of these objects. Single-trial exposure to multisensory objects during an orthogonal old/new discrimination task can incidentally modulate memory performance as well as brain activity. Performance was enhanced when the preceding multisensory stimulus had been a semantically congruent pairing, whereas performance was impaired by pairings with meaningless pure tones. Importantly, these opposing effects occurred despite identical performance patterns with the initial exposure to the objects (and presumably encoding thereof). In terms of brain activity, responses were enhanced to visual stimuli that had been previously encountered in a multisensory context within regions of the lateral occipital cortex beginning just 60 ms post-stimulus onset. Our collective findings indicate that single-trial learning of multisensory object associations occurs incidentally, is distinguishable from encoding processes, persists despite many intervening stimuli, and manifests as differential brain activity at early processing stages within visual cortices. The consequences of multisensory interactions thus persist over time to impact memory retrieval and object discrimination.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
An alternative viewpoint is that the V+ condition is not a pure repetition, but rather a novel context (see Section 2.3 for discussion).
References
Amedi A, von Kriegstein K, van Atteveldt NM, Beauchamp MS, Naumer MJ (2005) Functional imaging of human crossmodal identification and object recognition. Exp Brain Res 166:559–71
Amedi A, Jacobson G, Hendler T, Malach R, Zohary E (2002) Convergence of visual and tactile shape processing in the human lateral occipital complex. Cereb Cortex 12:1202–1212
Amedi A, Malach R, Hendler T, Peled S, Zohary E (2001) Visuohaptic object-related activation in the ventral visual pathway. Nat Neurosci 4:324–330
Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7:1190–1192
Besle J, Bertrand O, Giard MH (2009) Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex. Hear Res 258(1–2):143–151
Braeutigam S, Bailey AJ, Swithenby SJ (2001) Task-dependent early latency (30–60 ms) visual processing of human faces and other objects. Neuroreport 12:1531–36
Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during cross-modal binding. Neuroreport 10:2619–23
Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–57
Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio–visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–38
Cappe C, Morel A, Barone P, Rouiller EM (2009) The thalamocortical projection systems in primate: an anatomical support for multisensory and sensorimotor integrations. Cereb Cortex. 19(9):2025–2037, doi:10.1093/cercor/bhn228
Chen YC, Yeh SL (2008) Visual events modulated by sound in repetition blindness. Psychonom Bull Rev 15:404–408
Colombo M, Gross CG (1994) Responses of inferior temporal cortex and hippocampal neurons during delayed matching to sample in monkeys (Macaca fascicularis). Behav Neurosci 108:443–55
Debruille JB, Guillem F, Renault B (1998) ERPs and chronometry of face recognition: following-up Seeck et al. and George et al. Neuroreport 9:3349–53
Deibert E, Kraut M, Kremen S, Hart JJ (1999) Neural pathways in tactile object recognition. Neurology 52:1413–7
Desimone, R (1996) Neural mechanisms for visual memory and their role in attention. Proc Natl Acad Sci USA 93:13494–9
Doehrmann O, Naumer MJ (2008) Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res 1242:136–150
Doniger GM, Foxe JJ, Murray MM, Higgins BA, Snodgrass JG, Schroeder CE, Javitt DC (2000) Activation timecourse of ventral visual stream object-recognition areas: high density electrical mapping of perceptual closure processes. J Cogn Neurosci 12:615–621
Doniger GM, Foxe JJ, Schroeder CE, Murray MM, Higgins BA, Javitt DC (2001) Visual perceptual learning in human object recognition areas: a repetition priming study using high-density electrical mapping. Neuroimage 13:305–313
Fort A, Delpuech C, Pernier J, Giard MH (2002a) Dynamics of cortico-subcortical cross-modal operations involved in audio–visual object detection in humans. Cereb Cortex 12:1031–9
Fort A, Delpuech C, Pernier J, Giard MH (2002b) Early auditory–visual interactions in human cortex during nonredundant target identification. Brain Res Cogn Brain Res 14:20–30
Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE (2000) Multisensory auditory–somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res Cogn Brain Res 10:77–83
George N, Jemel B, Fiore N, Renault B (1997) Face and shape repetition effects in humans: a spatio-temporal ERP study. Neuroreport 8:1417–23
Giard MH, Peronnet, F (1999) Auditory–visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–90
Gibson JR, Maunsell JHR (1997) Sensory modality specificity of neural activity related to memory in visual cortex. J Neurophysiol 78:1263–75
Gottfried JA, Smith APR, Rugg MD, Dolan RJ (2004) Remembrance of odors past: human olfactory cortex in cross-modal recognition memory. Neuron 42:687–95
Goulet S, Murray EA (2001) Neural substrates of crossmodal association memory in monkeys: the amygdala versus the anterior rhinal cortex. Behav Neurosci 115:271–284
Grill-Spector K, Henson RN, Martin A (2006) Repetition and the brain, neural models of stimulus specific effects. Trends Cogn Sci 10:14–23
Guo J, Guo A (2005) Crossmodal interactions between olfactory and visual learning in Drosophila. Science 309:307–310
Haenny PE, Maunsell JHR, Schiler PH (1988) State dependent activity in monkey visual cortex: II. Retinal and extraretinal factors in V4. Exp Brain Res 69:245–259
Halgren E, Raij T, Marinkovic K, Jousmaki V, Hari R (2000) Cognitive response profile of the human fusiform face area as determined by MEG. Cereb Cortex 10:69–81
Hamilton W (1859) Lectures on Metaphysics and Logic. Gould & Lincoln, Boston
Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:2425–2430
Hein G, Doehrmann O, Müller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27:7881–7887
Henson RN, Rugg MD (2003) Neural response suppression, haemodynamic repetition effects, and behavioural priming. Neuropsychologia 41:263–270
Holmes NP (2009) The principle of inverse effectiveness in multisensory integration: Some statistical considerations. Brain Topogr 21(3–4):168–176
James TW, Humphrey GK, Gati JS, Servos P, Menon RS, Goodale MA (2002) Haptic study of three-dimensional objects activates extrastriate visual areas. Neuropsychologia 40:1706–1714
Landis T, Lehmann D, Mita T, Skrandies W (1984) Evoked potential correlates of figure and ground Int J Psychophysiol 1:345–348
Laurienti PJ, Kraft RA, Maldjian JA, Burdette JH, Wallace MT (2004) Semantic congruence is a critical factor in multisensory behavioral performance. Exp Brain Res 158:405–414
Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19:213–223
Lehmann S, Murray MM (2005) The role of multisensory memories in unisensory object discrimination. Brain Res Cogn Brain Res 24:326–334
MacSweeney M, Woll B, Campbell R, McGuire PK, David AS, Williams SC, Suckling J, Calvert GA, Brammer MJ (2002) Neural systems underlying British Sign Language and audio–visual English processing in native users. Brain 125:1583–93
Martuzzi R, Murray MM, Michel CM, Thiran JP, Maeder PP, Clarke S, Meuli RA (2007) Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cereb Cortex 17:1672–1779
Maunsell JHR, Sclar G, Nealey TA, Depriest DD (1991) Extraretinal representations in area V4 in the macaque monkey. Vis Neurosci 7:561–573
Michel CM, Murray MM, Lantz G, Gonzalez S, Spinelli L, Grave de Peralta R (2004) EEG source imaging. Clin Neurophysiol 115:2195–222
Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual–auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14:452–465
Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14:115–128
Mouchetant-Rostaing Y, Giard MH, Delpuech C, Echallier JF, Pernier J (2000a) Early signs of visual categorization for biological and non-biological stimuli in humans. Neuroreport 11:2521–2525
Mouchetant-Rostaing Y, Giard MH, Bentin S, Aguera PE, Pernier J (2000b) Neurophysiological correlates of face gender processing in humans. Eur J Neurosci 12:303–310
Murray MM, Camen C, Spierer L, Clarke S (2008a) Plasticity in representations of environmental sounds revealed by electrical neuroimaging. Neuroimage 39:847–856
Murray MM, Brunet D, Michel CM (2008b) Topographic ERP analyses: a step-by-step tutorial review. Brain Topogr 20:249–264
Murray MM, Camen C, Gonzalez Andino SL, Bovet P, Clarke S (2006) Rapid brain discrimination of sounds of objects. J Neurosci 26:1293–1302
Murray MM, Foxe JJ, Wylie GR (2005a) The brain uses single-trial multisensory memories to discriminate without awareness. Neuroimage 27:473–478
Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Schroeder CE, Foxe JJ (2005b) Grabbing your ear: rapid auditory–somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex 15:963–974
Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D, Andino SG, Schnider A (2004) Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage 21:125–135
Naumer MJ, Doehrmann O, Müller NG, Muckli L, Kaiser J, Hein G (2009) Cortical plasticity of audio-visual object representations. Cereb Cortex. 19(7):1641–1653, doi: 10.1093/cercor/ bhn200
Nyberg L, Habib R, McIntosh AR, Tulving E (2000) Reactivation of encoding-related brain activity during memory retrieval. Proc Natl Acad Sci USA 97:11120–11124
Raij T, Uutela K, Hari R (2000) Audiovisual integration of letters in the human brain. Neuron 28:617–625
Ranganath C, Rainer G (2003) Neural mechanisms for detecting and remembering novel events. Nat Rev Neurosci 4:193–202
Ritter W, Simson R, Vaughan Jr HG, Macht M (1982) Manipulation of event-related potential manifestations of information processing stages. Science 218:909–911
Rolls ET, Treves A, Tovee MJ (1997) The representational capacity of the distributed encoding of information provided by populations of neurons in primate temporal visual cortex. Exp Brain Res 114:149–162
Romei V, Murray MM, Merabet LB, Thut G (2007) Occipital transcranial magnetic stimulation has opposing effects on visual and auditory stimulus detection: implications for multisensory interactions. J Neurosci 27:11465–11472
Rousselet GA, Fabre-Thorpe M, Thorpe SJ (2002) Parallel processing in high-level categorization of natural images. Nat Neurosci 5:629–630
Rugg MD, Mark RE, Walla P, Schloerscheidt AM, Birch CS, Allan K (1998) Dissociation of the neural correlates of implicit and explicit memory. Nature 392:595–598
Schnider A, Valenza N, Morand S, Michel CM (2002) Early cortical distinction between memories that pertain to ongoing reality and memories that don’t. Cereb Cortex 12:54–61
Seeck M, Michel CM, Mainwaring N, Cosgrove R, Blume H, Ives J, Landis T, Schomer DL (1997) Evidence for rapid face recognition from human scalp and intracranial electrodes. Neuroreport 8:2749–2754
Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cogn Sci 12:411–417
Spence C, Nicholls ME, Driver J (2001) The cost of expecting events in the wrong sensory modality. Percept Psychophys 63:330–336
Sperdin HF, Cappe C, Foxe JJ, Murray MM (2009) Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed. Front Integr Neurosci 3:2. doi:10.3389/neuro.07.002.2009
Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MA
Stoesz MR, Zhang M, Weisser VD, Prather SC, Mao H, Sathian K (2003) Neural networks active during tactile form perception: common and differential activity during macrospatial and microspatial tasks. Int J Psychophysiol 50:41–49
Thierry G, Martin CD, Downing P, Pegna AJ (2007) Controlling for interstimulus perceptual variance abolishes N170 face selectivity. Nat Neurosci 10:505–511
Tsivilis D, Otten LJ, Rugg MD (2001) Context effects on the neural correlates of recognition memory: an electrophysiological study. Neuron 31:497–505
Tulving E, Schacter DL (1990) Priming and human memory systems. Science 247:301–306
Vanni S, Revonsuo A, Saarinen J, Hari R (1996) Visual awareness of objects correlates with activity of right occipital cortex. Neuroreport 8:183–186
Von Kriegstein K, Giraud AL (2006) Implicit multisensory associations influence voice recognition. PLoS Biol 4:e326
Wang Y, Celebrini S, Trotter Y, Barone P (2008) Visuo-auditory interactions in the primary visual cortex of the behaving monkey: electrophysiological evidence. BMC Neurosci 9:79
Wheeler ME, Petersen SE, Buckner RL (2000) Memory’s echo: vivid remembering reactivates sensory-specific cortex. Proc Natl Acad Sci USA 97:11125–11129
Wiggs CL, Martin A (1998) Properties and mechanisms of perceptual priming. Curr Opin Neurobiol 8:227–233
Zangaladze A, Epstein CM, Grafton ST, Sathian K (1999) Involvement of visual cortex in tactile discrimination of orientation. Nature 401:587–590
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer Science + Business Media, LLC
About this chapter
Cite this chapter
Murray, M.M., Sperdin, H.F. (2010). Single-Trial Multisensory Learning and Memory Retrieval. In: Kaiser, J., Naumer, M. (eds) Multisensory Object Perception in the Primate Brain. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-5615-6_11
Download citation
DOI: https://doi.org/10.1007/978-1-4419-5615-6_11
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4419-5614-9
Online ISBN: 978-1-4419-5615-6
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)