Multimodal evidence on shape and surface information in individual face processing
Introduction
The segregation of shape and surface information defines a fundamental aspect of visual processing and cortical organization (Livingstone and Hubel, 1988; Van Essen and Deyoe, 1995) both in the human (Cant et al., 2008; Lafer-Sousa et al., 2016; Vinberg and Grill-Spector, 2008) and the monkey brain (Conway et al., 2007). Accordingly, this distinction has played a prominent role in accounts of face recognition (Bruce and Young, 1998). Extensive research has documented the importance of both types of information in face perception (Biederman and Kalocsai, 1997; Jiang et al., 2006; O'Toole et al., 1999; Russell et al., 2007; Russell and Sinha, 2007; Vuong et al., 2005), but the relative weight of shape and surface properties has been heavily debated, with either the former (Jiang et al., 2011; Lai et al., 2013) or the latter (Bruce et al., 1991; Bruce and Langton, 1994; Hole et al., 2002; Kaufmann and Schweinberger, 2008; Russell et al., 2006) considered dominant. Arguably, this debate arises from a lack of specificity in identifying the shape and surface features critical for individual face processing (Burton et al., 2015). Thus, the current research aims to uncover the nature of the information involved in individual face processing along with its accompanying neural profile.
To address the challenge above, here, we appeal to neural-based image reconstruction (Shen et al., 2018; Miyawaki et al., 2008; Naselaris et al., 2009; Nishimoto et al., 2011a, Nishimoto et al., 2011b; Thirion et al., 2006), namely, the endeavor of reconstructing the appearance of visual objects from neural activity prompted by their processing. While this endeavor has relied primarily on functional magnetic resonance imaging (fMRI), more recently, additional modalities have been used successfully as well. For instance, facial image reconstruction has been carried out using single-cell recordings (Chang and Tsao, 2017), electroencephalography (EEG) data (Nemrodov et al., 2018) and behavioral data (Chang et al., 2017b; Zhan et al., 2017), in addition to fMRI (Cowen et al., 2014; Lee and Kuhl, 2016; Nestor et al., 2016). Thus, in theory, image reconstruction can provide a powerful platform for investigating shape/surface processing in face individuation via multiple behavioral and neuroimaging modalities. Concretely, image reconstruction can be used to uncover, assess and compare facial shape and surface information recovered from distinct modalities.
To this end, we rely on data assessing individual face processing gleaned from behavioral (Nestor et al., 2013), EEG (Nemrodov et al., 2018) and fMRI data (Nestor et al., 2016). Specifically, for each modality, we aim to recover the shape and surface content of a common set of face stimuli as perceived by human observers. In addition, the same procedure is conducted with an image-based theoretical observer (TO) allowing us to compare the informational content of multiple empirical and TO reconstructions.
To achieve these goals, we appeal to an influential approach for analyzing face images into shape and surface properties (Craw and Cameron, 1991; Kramer et al., 2016; Tiddeman et al., 2001; Vetter and Troje, 1995). Specifically, this approach involves marking the positions of a set of fiducial points (e.g., the corners of the eyes or the tip of the nose) that deliver shape information. Then, faces are warped to a standard shape (i.e., a preset configuration of fiducial points) yielding ‘shape-free’ images that deliver surface information. To be clear, shape derived in this manner encompasses two sources of information: configural information, conceived as metric distances between different face parts (Maurer et al., 2002; Tanaka and Gordon, 2011), and local information associated with the geometric structure of specific face parts such as eye shape or mouth shape (Cabeza and Kato, 2000; Gold et al., 2012; Rakover, 2002). In contrast, surface contains information about the reflectance properties of a face (e.g., hue, specularity, albedo) that also play a role in individual face recognition (Hancock et al., 1996; Russell et al., 2007; Taschereau-Dumouchel et al., 2010) – such information is alternatively referred to as ‘texture’, ‘pigmentation’ or ‘surface reflectance’.
The appeal to shape-surface decomposition allows us to address a number of related questions. First, can image reconstruction separately recover facial shape and surface information from different modalities and, if so, how well? Second, what is the spatiotemporal profile of shape and surface processing? Third, what specific shape/surface features are recovered through reconstruction? And fourth, do different modalities reveal similar or complementary information about face representations? More generally, the present work evaluates and confirms the ability of a novel methodological paradigm to exploit multimodal evidence in an effort to elucidate the representational content of individual face processing.
In summary, the current work embarks on a comprehensive investigation of facial shape and surface processing by appealing to powerful and innovative image-reconstruction methodology as applied to multimodal data. Accordingly, this work serves a twofold purpose by shedding light on the psychological and neural profile of facial shape/surface processing and by clarifying the informational content responsible for the success of image reconstruction.
Section snippets
Stimuli
A common subset of 108 stimulus images was identified across three different studies investigating empirical and computational aspects of unfamiliar face recognition (see 2.3 Experimental procedures). Images of 54 individuals displaying neutral and happy facial expressions were selected from three databases: AR (Martinez and Benavente, 1998), FEI (Thomaz and Giraldi, 2010) and Radboud (Langner et al., 2010). All images featured young adult Caucasian males with frontal view, gaze and
Representational similarity
Estimates of pairwise face similarity were computed across 108 images (54 identities x 2 expressions, neutral and happy) for each of four data types: (i) behavioral, based on similarity ratings; (ii) EEG, based on neural discriminability across occipitotemporal (OT) electrodes; (iii) fMRI, based on neural discriminability across multiple fusiform gyrus (FG) areas, and (iv) TO, based on pixelwise image similarity. In particular, we note that fMRI estimates relied jointly on patterns of
Discussion
The present study examined the representational basis of shape and surface information underlying individual face processing. This investigation capitalized on a robust approach to image reconstruction to uncover and relate relevant representational structures captured by distinct modalities of data acquisition. The ability to retrieve such information successfully and consistently with the aid of this novel approach enabled us to address a number of key questions as follows.
First, we examined
Competing interests
The authors declare no competing financial or non-financial interests.
Acknowledgments
This research was supported by the Natural Sciences and Engineering Research Council of Canada of Canada (A.N. and M.N.), by the National Institute of Health (to M.B.), and by a Research Competitiveness Fund Award (A.N.).
References (102)
- et al.
Contributions of feature shapes and surface cues to the recognition and neural representation of facial identity
Cortex
(2016) - et al.
The role of face shape and pigmentation in other-race face perception: an electrophysiological study
Neuropsychologia
(2010) - et al.
Artificial faces are harder to remember
Comput. Hum. Behav.
(2015) - et al.
The importance of skin color and facial structure in perceiving and remembering others: an electrophysiological study
Brain Res.
(2011) - et al.
Robust representations for face recognition: the power of averages
Cognit. Psychol.
(2005) - et al.
Early adaptation to repeated unfamiliar faces across viewpoint changes in the right hemisphere: evidence from the N170 ERP component
Neuropsychologia
(2009) - et al.
Recognizing an individual face: 3D shape contributes earlier than 2D surface reflectance information
Neuroimage
(2009) - et al.
A principal component analysis of facial expressions
Vis. Res.
(2001) - et al.
Specialized color modules in macaque extrastriate cortex
Neuron
(2007) - et al.
Neural portraits of perception: reconstructing face images from evoked brain activity
Neuroimage
(2014)