Integration of face features in expression discrimination studied with psychophysics and fMRI

We studied facial cue integration by varying the expression (angry or happy) intensity in eyes and mouth separately and tested whether observers can integrate information when estimating the expression of the whole face. In addition, we tested whether conflicting taskirrelevant cues impairs discrimination performance. Our results show that participants were able to integrate the two facial features, and that the integration was obligatory. In fMRI, we found higher BOLD-activity for incongruent than congruent expressions in fusiform face area and medial frontal area.


Introduction
In order to perceive an expression from a face, the information from different face parts (e.g. eyes and mouth) needs to be evaluated since different facial features carry different amounts of information for different expressions. Classic studies using compositefaces have shown that the expression in the mouth affects the detection of the expression in the eyes, even when the mouth is task-irrelevant (Murphy, Gray, & Cook, 2017;Young, Hellawell, & Hay, 1987). In these studies, the amount of information in different features have not been varied, but instead expressions at full intensity has been used. Cue integration can be studied by independently varying the strength of the cues. Previous studies have shown (optimal) integration of cues from different senses (Ernst & Banks, 2002;Helbig & Ernst, 2007) as well as optimal and obligatory integration of different visual cues (Saarela & Landy, 2015). In face perception, a few studies have applied this paradigm to study facial identities (Dobs, Ma, & Reddy, 2017;Gold et al., 2014;Gold, Mundy, & Tjan, 2012;Hotaling, Cohen, Shiffrin, & Busemeyer, 2015;Shen & Palmeri, 2015). Here, we varied the intensity of expression in the eyes and mouth, asking how these two cues are integrated. In addition, we tested whether task-irrelevant, congruent or incongruent, cues affect integration. Finally, we measured fMRI to look into the neural correlates of the facial feature integration.

Methods
We varied expression intensity in the upper and lower half of a face separately, by morphing both halves on a dimension from happy to neutral to angry. The face parts were smoothly cropped and combined, resulting in images looking like original photographs, with no visible transitions between the halves. Expression intensities were defined individually for each subject in a separate expression discrimination experiment. In a Cue Conflict experiment, participants (n=8+8) categorized the expression in the top or bottom half of a face as angry or happy. The task-irrelevant half was either congruent (same expression), neutral, or incongruent (different expression). In a Cue Integration task, all stimuli had congruent expression, while the intensity, and thus available information, varied between the halves (e.g. more in top than bottom half, or all in the top half). The Cue Conflict task was repeated in fMRI with 100% eyes and 50% mouth expression intensities.

Results
In the expression discrimination task, similar discrimination accuracy was found for isolated mouth and whole face, but the discrimination of isolated eyes was reduced (Figure 1). These discrimination thresholds were used to individually scale the cue intensities in the following cue integration and cue conflict experiments.
In the cue integration experiment, the observers were able to integrate the information from mouth and eyes regions, if both cues were informative. However, when all expression information was in the eyes or mouth and the other cue was neutral (i.e., single cue condition), the 906 This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0 discrimination performance was surprisingly poor. Thus, the neutral part seemed to affect the participants decisions, even though they knew it was uninformative for the task. Figure 1: Expression discrimination thresholds. The first study (n=8) contained face images from 60 identities. In the second study (n=8) the amount of identities was reduced to 16.
In fMRI, incongruent faces (e.g. happy eyes with angry mouth) elicited larger BOLD-amplitudes than congruent faces in right fusiform face area (FFA) and in bilateral medial frontal cortex (Figure 3). Contrasting whole faces and face parts (isolated mouth or isolated eyes) revealed larger BOLD-amplitudes for face parts in left and right occipital face area, and larger amplitudes for whole faces in bilateral medial frontal cortex and right early visual cortex.

Discussion
In the cue integration experiment, the observers were able to integrate information from both cues, but only if both cues contained information of the expressions. In the Cue Conflict experiment, task-irrelevant but congruent face part increased discrimination accuracy, and task-irrelevant but incongruent face part reduced discrimination accuracy, when compared to neutral face part. This suggests obligatory integration of facial features. Unlike earlier studies, we controlled for information in the different face parts, and showed that both congruent and incongruent faces differed from faces with neutral task-irrelevant part. This shows that a neutral task-irrelevant half is sufficient to cause the face-composite effect. In fMRI, we found higher BOLDresponses for incongruent than congruent faces only in few areas, in a medial frontal patch and right FFA, suggesting their involvement in integrating facial features to whole faces.