Perception of Facial Expressions and Voices and of their Combination in the Human Brain

doi:10.1016/S0010-9452(08)70177-1

Cortex

Volume 41, Issue 1, 2005, Pages 49-59

https://doi.org/10.1016/S0010-9452(08)70177-1 Get rights and content

Abstract

Using positron emission tomography we explored brain regions activated during the perception of face expressions, emotional voices and combined audio-visual pairs. A convergence region situated in the left lateral temporal cortex was more activated by bimodal stimuli than by either visual only or auditory only stimuli. Separate analyses for the emotions happiness and fear revealed supplementary convergence areas situated mainly anteriorly in the left hemisphere for happy pairings and in the right hemisphere for fear pairings indicating different neuro-anatomical substrates for multisensory integration of positive versus negative emotions. Activation in the right extended amygdala was obtained for fearful faces and fearful audio-visual pairs but not for fearful voices only. These results suggest that during the multisensory perception of emotion, affective information from face and voice converge in heteromodal regions of the human brain.

References (76)

R Adolphs
Neural systems for recognizing emotion
Current Opinion in Neurobiology
(2002)
R Adolphs et al.
Intact recognition of emotional prosody following amygdala damage
Neuropsychologia
(1999)
TW Buchanan et al.
Recognition of emotional prosody and verbal components of spoken language: An fMRI study
Cognitive Brain Research
(2000)
GA Calvert et al.
Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex
Current Biology
(2000)
AR Damasio
Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition
Cognition
(1989)
RJ Davidson et al.
The functional neuroanatomy of emotion and affective style
Trends in Cognitive Sciences
(1999)
B de Gelder et al.
The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responses
Neuroscience Letters
(1999)
MJ Farah et al.
Parietal lobe mechanisms of spatial attention: Modality-specific or supramodal?
Neuropsychologia
(1989)
JS Morris et al.
Saying it with feeling: Neural responses to emotional vocalizations
Neuropsychologia
(1999)
FK Nahm et al.
Cross-modal associations and the human amygdala
Neuropsychologia
(1993)

G Pourtois et al.

Facial expressions modulate the time course of long latency auditory brain potentials

Cognitive Brain Research

(2002)

CJ Price et al.

Cognitive conjunction: A new approach to brain activation experiments

Neuroimage

(1997)

M Streicher et al.

Cross-modal recognition of familiar and unfamiliar objects by the monkey: The effects of ablation of polysensory neocortex or of the amygdaloid complex

Behavioral Brain Research

(1987)

DH Zald

The human amygdala and the emotional evaluation of sensory stimuli

Brain Research Reviews

(2003)

R Adolphs et al.

Neural Systems for Recognition of Emotional Prosody: A 3-D Lesion Study

Emotion

(2002)

R Adolphs et al.

Cortical systems for the recognition of emotion in facial expressions

Journal of Neuroscience

(1996)

R Adolphs et al.

Preferences for visual stimuli following amygdala damage

Journal of Cognitive Neuroscience

(1999)

AK Anderson et al.

Intact recognition of vocal expressions of fear following bilateral lesions of the human amygdala

Neuroreport

(1998)

P Belin et al.

Voice-selective areas in human auditory cortex

Nature

(2000)

P Bertelson

Ventriloquism: A case of crossmodal perceptual grouping

P Bertelson et al.

The ventriloquist effect does not depend on the direction of deliberate visual attention

Perception and Psychophysics

(2000)

Bertelson P and de Gelder B. The Psychology of Multi-modal Perception. J Driver and C Spence (Eds). London: Oxford...

JR Binder et al.

Human brain language areas identified by functional magnetic resonance imaging

Journal of Neuroscience

(1997)

JC Borod et al.

Relationships among facial, prosodic, and lexical channels of emotional perceptual processing

Cognition and Emotion

(2000)

GA Calvert et al.

Response amplification in sensory-specific cortices during crossmodal binding

Neuroreport

(1999)

B de Gelder

Crossmodal perception: There is more to seeing than meets the eye

Science

(2000)

B de Gelder et al.

Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures

Proceedings of the National Academy of Sciences of the United Sates of America

(2002)

B de Gelder et al.

Perceiving Emotions by Ear and by Eye

Cognition and Emotion

(2000)

B de Gelder et al.

Hearing smiles and seeing cries: The bimodal perception of emotions

Bulletin of the Psychonomic Society

(1995)

RJ Dolan et al.

Crossmodal binding of fear in voice and face

Proceedings of the National Academy of Sciences of the United Sates of America

(2001)

J Driver et al.

Multisensory perception: Beyond modularity and convergence

Current Biology

(2000)

P Ekman et al.

Pictures of facial affect

(1976)

HD Ellis et al.

Intra- and inter-modal repetition priming of familiar faces and voices

British Journal of Psychology

(1997)

PT Fox et al.

Enhanced detection of focal brain response using intersubject averaging and change-distribution analysis of substracted PET images

Journal of Cerebral Blood Flow and Metabolism

(1988)

KJ Friston et al.

Statistical Parametric Maps in Functional Imaging: A General Linear Approach

Human Brain Mapping

(1995)

G Gainotti

Neuropsychological theories of emotion

MS George et al.

Brain regions involved in recognizing facial emotion or identity: An oxygen-15 PET study

Journal of Neuropsychiatry and Clinical Neurosciences

(1993)

MS George et al.

Understanding emotional prosody activates right hemisphere regions

Archives of Neurology

(1996)

Cited by (154)

Creative flow as optimized processing: Evidence from brain oscillations during jazz improvisations by expert and non-expert musicians
2024, Neuropsychologia
Using a creative production task, jazz improvisation, we tested alternative hypotheses about the flow experience: (A) that it is a state of domain-specific processing optimized by experience and characterized by minimal interference from task-negative default-mode network (DMN) activity versus (B) that it recruits domain-general task-positive DMN activity supervised by the fronto-parietal control network (FPCN) to support ideation. We recorded jazz guitarists' electroencephalograms (EEGs) while they improvised to provided chord sequences. Their flow-states were measured with the Core Flow State Scale. Flow-related neural sources were reconstructed using SPM12. Over all musicians, high-flow (relative to low-flow) improvisations were associated with transient hypofrontality. High-experience musicians’ high-flow improvisations showed reduced activity in posterior DMN nodes. Low-experience musicians showed no flow-related DMN or FPCN modulation. High-experience musicians also showed modality-specific left-hemisphere flow-related activity while low-experience musicians showed modality-specific right-hemisphere flow-related deactivations. These results are consistent with the idea that creative flow represents optimized domain-specific processing enabled by extensive practice paired with reduced cognitive control.
Oxytocin has sex-specific effects on trust and underlying neurophysiological processes
2023, Psychoneuroendocrinology
The neuropeptide oxytocin (OT) regulates mammalian social approach behavior across sexes. Yet most OT studies in humans exclusively investigated men. Here, we studied sex differences in OT’s effects on human trust behavior in 144 heterosexual participants (73 women, 71 men). Participants received 24 international units of intranasal OT or placebo treatment and played a trust game in the role of the investor while undergoing electroencephalography. Trustees were represented by photos of the other sex gradually varying in their pre-rated intensities of facial features signaling attractiveness and threat. On a behavioral level, we observed that OT increased trust in men and reduced it in women when trustees showed weak signals of attractiveness and threat. Correspondingly, on the neurophysiological level, we noted that OT intensified the P100 in male participants, but dampened it in female ones. Our findings demonstrate OT’s sex- and context-specific effects on social approach behavior and an underlying early visual attention-related brain process. This evidence demonstrates the need to consider psychobiological mechanisms of sexual dimorphism in human OT research.
Exploring structural and functional alterations in drug-naïve obsessive-compulsive disorder patients: An ultrahigh field multimodal MRI study
2023, Asian Journal of Psychiatry
Brain structural and functional alterations have been reported in obsessive-compulsive disorder (OCD) patients; however, these findings were inconsistent across studies due to several limitations, including small sample sizes, different inclusion/exclusion criteria, varied demographic characteristics and symptom dimensions, comorbidity, and medication status. Prominent and replicable neuroimaging biomarkers remain to be discovered.
This study explored the gray matter structure, neural activity, and white matter microstructure differences in 40 drug-naïve OCD patients and 57 matched healthy controls using ultrahigh field 7.0 T multimodal magnetic resonance imaging, which increased the spatial resolution and detection power. We also evaluated correlations among different modalities, imaging features and clinical symptoms.
Drug-naïve OCD patients exhibited significantly increased gray matter volume in the frontal cortex, especially in the orbitofrontal cortex, as well as volumetric reduction in the temporal lobe, occipital lobe and cerebellum. Increased neural activities were observed in the cingulate gyri and precuneus. Increased temporal-middle cingulate and posterior cingulate-precuneus functional connectivities and decreased frontal-middle cingulate connectivity were further detected. Decreased fractional anisotropy values were found in the cingulum-hippocampus gyrus and inferior fronto-occipital fascicle in OCD patients. Moreover, significantly altered imaging features were related to OCD symptom severity. Altered functional and structural neural connectivity might influence compulsive and obsessive features, respectively.
Altered structure and function of the classical cortico-striato-thalamo-cortical circuit, limbic system, default mode network, visual, language and sensorimotor networks play important roles in the neurophysiology of OCD.
Odor-evoked hedonic contexts influence the discrimination of facial expressions in the human brain
2021, Biological Psychology
The influence of odor valence on expressive-face perception remains unclear. Here, three “valenced” odor contexts (pleasant, unpleasant, control) were diffused while scalp electroencephalogram (EEG) was recorded in 18 participants presented with expressive faces alternating at a 6-Hz rate. One facial expression (happiness, disgust or neutrality) repeatedly arose every 6 face pictures to isolate its discrimination from other expressions at 1 Hz and harmonics in the EEG spectrum. The amplitude of the brain response to neutrality was larger in the pleasant vs. control odor context, and fewer electrodes responded in the unpleasant odor context. The number of responding electrodes was reduced for disgust in both odor contexts. The response to happiness was unchanged between odor conditions. Overall, these observations suggest that valenced odors influence the neural discrimination of facial expressions depending on both face and odor hedonic valence, especially for the emotionally ambiguous neutral expression.
Conversational facial signals combine into compositional meanings that change the interpretation of speaker intentions
2024, Scientific Reports
Variable selection in additive models via hierarchical sparse penalty
2024, Canadian Journal of Statistics

View all citing articles on Scopus

View full text

Perception of Facial Expressions and Voices and of their Combination in the Human Brain

Abstract

Current Opinion in Neurobiology

Neuropsychologia

Cognitive Brain Research

Current Biology

Cognition

Trends in Cognitive Sciences

Neuroscience Letters

Neuropsychologia

Neuropsychologia

Neuropsychologia

Cognitive Brain Research

Neuroimage

Behavioral Brain Research

Brain Research Reviews

Neural Systems for Recognition of Emotional Prosody: A 3-D Lesion Study

Emotion

Cortical systems for the recognition of emotion in facial expressions

Journal of Neuroscience

Preferences for visual stimuli following amygdala damage

Journal of Cognitive Neuroscience

Intact recognition of vocal expressions of fear following bilateral lesions of the human amygdala

Neuroreport

Voice-selective areas in human auditory cortex

Nature

Ventriloquism: A case of crossmodal perceptual grouping

The ventriloquist effect does not depend on the direction of deliberate visual attention

Perception and Psychophysics

Human brain language areas identified by functional magnetic resonance imaging

Journal of Neuroscience

Relationships among facial, prosodic, and lexical channels of emotional perceptual processing

Cognition and Emotion

Response amplification in sensory-specific cortices during crossmodal binding

Neuroreport

Crossmodal perception: There is more to seeing than meets the eye

Science

Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures

Proceedings of the National Academy of Sciences of the United Sates of America

Perceiving Emotions by Ear and by Eye

Cognition and Emotion

Hearing smiles and seeing cries: The bimodal perception of emotions

Bulletin of the Psychonomic Society

Crossmodal binding of fear in voice and face

Proceedings of the National Academy of Sciences of the United Sates of America

Multisensory perception: Beyond modularity and convergence

Current Biology

Pictures of facial affect

Intra- and inter-modal repetition priming of familiar faces and voices

British Journal of Psychology

Enhanced detection of focal brain response using intersubject averaging and change-distribution analysis of substracted PET images

Journal of Cerebral Blood Flow and Metabolism

Statistical Parametric Maps in Functional Imaging: A General Linear Approach

Human Brain Mapping

Neuropsychological theories of emotion

Brain regions involved in recognizing facial emotion or identity: An oxygen-15 PET study

Journal of Neuropsychiatry and Clinical Neurosciences

Understanding emotional prosody activates right hemisphere regions

Archives of Neurology