An fMRI study of the binding of audio-visual information: the dissociation between object and space processing

Abstract

Recent neuroimaging research highlighted the contribution of different heteromodal cortical regions during various forms of crossmodal binding. Interestingly, crossmodal effects during audiovisual speech and object recognition have been found in the superior temporal sulcus (STS), while crossmodal effects during the execution of spatial tasks have been found over the intraparietal sulcus (IpS), suggesting an underlying “what/where” segregation. In this fMRI study, we investigated if a such “what/where” functional segregation, similar to the one observed for both visual and auditory cortical processing, exists in the crossmodal domain, when visual and auditory stimuli have to be matched in order to perform either a recognition or a localization task. In order to directly compare the specific involvement of these two heteromodal regions, we scanned ten male right handed subjects during the execution of two crossmodal matching tasks. Participants were simultaneously presented with a black and white picture, chosen from the Snodgrass and Vanderwart’s set, and an environmental sound, selected from a free internet archive set. The pictures and the sounds could be presented in either the same or the opposite hemifield and represent either the same or a different object. Subjects were required to report whether the two attended stimuli were matching or mismatching with respect to either spatial position (localization task) or semantic content (recognition task) by pressing two buttons of a response pad. The group analysis showed that the crossmodal localization task elicited more activity than the recognition task in the left and right precuneus, in the right parietal cortex, including the IpS, and in the right superior occipital cortex. Regions responding to the crossmodal recognition task more than to the localization task were found in the inferior occipital gyrus, bilaterally, and in the left lateral temporal cortex, including the anterior part of the STS and the superior temporal gyrus. An individual subject analysis was performed on the left superior temporal sulcus and on the right intraparietal sulcus, in order to take account for the intersubject anatomical variability in the relative location and extension of cortical sulci. The 2 × 2 (TASK × CONGRUENCY) within subject ANOVA confirmed the results of the group analysis, showing a main effect of task in the right IpS [F(1,9) = 14.33, P < 0.005] and in the left STS [F(1,9) = 20.52, P < 0.001] but no effect of congruency. Our results are consistent with a “what–where” functional segregation in the crossmodal audiovisual domain, and suggest the role of the IpS in the binding of “spatial” audiovisual information and of the STS in the binding of “semantic” audiovisual information. The dissociation between object and spatial processing streams seems to be an organizing principle of the cortical functioning.