Elsevier

NeuroImage

Volume 184, 1 January 2019, Pages 813-825
NeuroImage

Multimodal evidence on shape and surface information in individual face processing

https://doi.org/10.1016/j.neuroimage.2018.09.083Get rights and content

Highlights

  • Face shape and surface information is recovered from behavioral, EEG and fMRI data

  • Surface information is recovered better than shape from empirical data

  • Shape information is recovered more consistently across modalities

  • Shape and surface exhibit similar spatiotemporal profiles of neural processing

  • Eye shape and skin tone play key roles in individual face representation

Abstract

The significance of shape and surface information for face perception is well established, yet their relative contribution to recognition and their neural underpinnings await clarification. Here, we employ image reconstruction to retrieve, assess and visualize such information using behavioral, electroencephalography and functional magnetic resonance imaging data.

Our results indicate that both shape and surface information can be successfully recovered from each modality but that the latter is better recovered than the former, consistent with its key role for face representations. Further, shape and surface information exhibit similar spatiotemporal profiles, rely on the extraction of specific visual features, such as eye shape or skin tone, and reveal a systematic representational structure, albeit with more cross-modal consistency for shape than surface. More generally, the present work illustrates a novel approach to relating and comparing different modalities in terms of perceptual information content.

Thus, our results help elucidate the representational basis of individual face recognition while, methodologically, they showcase the utility of image reconstruction and clarify its reliance on diagnostic visual information.

Introduction

The segregation of shape and surface information defines a fundamental aspect of visual processing and cortical organization (Livingstone and Hubel, 1988; Van Essen and Deyoe, 1995) both in the human (Cant et al., 2008; Lafer-Sousa et al., 2016; Vinberg and Grill-Spector, 2008) and the monkey brain (Conway et al., 2007). Accordingly, this distinction has played a prominent role in accounts of face recognition (Bruce and Young, 1998). Extensive research has documented the importance of both types of information in face perception (Biederman and Kalocsai, 1997; Jiang et al., 2006; O'Toole et al., 1999; Russell et al., 2007; Russell and Sinha, 2007; Vuong et al., 2005), but the relative weight of shape and surface properties has been heavily debated, with either the former (Jiang et al., 2011; Lai et al., 2013) or the latter (Bruce et al., 1991; Bruce and Langton, 1994; Hole et al., 2002; Kaufmann and Schweinberger, 2008; Russell et al., 2006) considered dominant. Arguably, this debate arises from a lack of specificity in identifying the shape and surface features critical for individual face processing (Burton et al., 2015). Thus, the current research aims to uncover the nature of the information involved in individual face processing along with its accompanying neural profile.

To address the challenge above, here, we appeal to neural-based image reconstruction (Shen et al., 2018; Miyawaki et al., 2008; Naselaris et al., 2009; Nishimoto et al., 2011a, Nishimoto et al., 2011b; Thirion et al., 2006), namely, the endeavor of reconstructing the appearance of visual objects from neural activity prompted by their processing. While this endeavor has relied primarily on functional magnetic resonance imaging (fMRI), more recently, additional modalities have been used successfully as well. For instance, facial image reconstruction has been carried out using single-cell recordings (Chang and Tsao, 2017), electroencephalography (EEG) data (Nemrodov et al., 2018) and behavioral data (Chang et al., 2017b; Zhan et al., 2017), in addition to fMRI (Cowen et al., 2014; Lee and Kuhl, 2016; Nestor et al., 2016). Thus, in theory, image reconstruction can provide a powerful platform for investigating shape/surface processing in face individuation via multiple behavioral and neuroimaging modalities. Concretely, image reconstruction can be used to uncover, assess and compare facial shape and surface information recovered from distinct modalities.

To this end, we rely on data assessing individual face processing gleaned from behavioral (Nestor et al., 2013), EEG (Nemrodov et al., 2018) and fMRI data (Nestor et al., 2016). Specifically, for each modality, we aim to recover the shape and surface content of a common set of face stimuli as perceived by human observers. In addition, the same procedure is conducted with an image-based theoretical observer (TO) allowing us to compare the informational content of multiple empirical and TO reconstructions.

To achieve these goals, we appeal to an influential approach for analyzing face images into shape and surface properties (Craw and Cameron, 1991; Kramer et al., 2016; Tiddeman et al., 2001; Vetter and Troje, 1995). Specifically, this approach involves marking the positions of a set of fiducial points (e.g., the corners of the eyes or the tip of the nose) that deliver shape information. Then, faces are warped to a standard shape (i.e., a preset configuration of fiducial points) yielding ‘shape-free’ images that deliver surface information. To be clear, shape derived in this manner encompasses two sources of information: configural information, conceived as metric distances between different face parts (Maurer et al., 2002; Tanaka and Gordon, 2011), and local information associated with the geometric structure of specific face parts such as eye shape or mouth shape (Cabeza and Kato, 2000; Gold et al., 2012; Rakover, 2002). In contrast, surface contains information about the reflectance properties of a face (e.g., hue, specularity, albedo) that also play a role in individual face recognition (Hancock et al., 1996; Russell et al., 2007; Taschereau-Dumouchel et al., 2010) – such information is alternatively referred to as ‘texture’, ‘pigmentation’ or ‘surface reflectance’.

The appeal to shape-surface decomposition allows us to address a number of related questions. First, can image reconstruction separately recover facial shape and surface information from different modalities and, if so, how well? Second, what is the spatiotemporal profile of shape and surface processing? Third, what specific shape/surface features are recovered through reconstruction? And fourth, do different modalities reveal similar or complementary information about face representations? More generally, the present work evaluates and confirms the ability of a novel methodological paradigm to exploit multimodal evidence in an effort to elucidate the representational content of individual face processing.

In summary, the current work embarks on a comprehensive investigation of facial shape and surface processing by appealing to powerful and innovative image-reconstruction methodology as applied to multimodal data. Accordingly, this work serves a twofold purpose by shedding light on the psychological and neural profile of facial shape/surface processing and by clarifying the informational content responsible for the success of image reconstruction.

Section snippets

Stimuli

A common subset of 108 stimulus images was identified across three different studies investigating empirical and computational aspects of unfamiliar face recognition (see 2.3 Experimental procedures). Images of 54 individuals displaying neutral and happy facial expressions were selected from three databases: AR (Martinez and Benavente, 1998), FEI (Thomaz and Giraldi, 2010) and Radboud (Langner et al., 2010). All images featured young adult Caucasian males with frontal view, gaze and

Representational similarity

Estimates of pairwise face similarity were computed across 108 images (54 identities x 2 expressions, neutral and happy) for each of four data types: (i) behavioral, based on similarity ratings; (ii) EEG, based on neural discriminability across occipitotemporal (OT) electrodes; (iii) fMRI, based on neural discriminability across multiple fusiform gyrus (FG) areas, and (iv) TO, based on pixelwise image similarity. In particular, we note that fMRI estimates relied jointly on patterns of

Discussion

The present study examined the representational basis of shape and surface information underlying individual face processing. This investigation capitalized on a robust approach to image reconstruction to uncover and relate relevant representational structures captured by distinct modalities of data acquisition. The ability to retrieve such information successfully and consistently with the aid of this novel approach enabled us to address a number of key questions as follows.

First, we examined

Competing interests

The authors declare no competing financial or non-financial interests.

Acknowledgments

This research was supported by the Natural Sciences and Engineering Research Council of Canada of Canada (A.N. and M.N.), by the National Institute of Health (to M.B.), and by a Research Competitiveness Fund Award (A.N.).

References (102)

  • X. Gao et al.

    The neural representation of face space dimensions

    Neuropsychologia

    (2013)
  • S. Gilaie-Dotan et al.

    Perceptual shape sensitivity to upright and inverted faces is reflected in neuronal adaptation

    Neuroimage

    (2010)
  • R.J. Harris et al.

    Brain regions involved in processing facial identity and expression are differentially selective for surface and edge information

    Neuroimage

    (2014)
  • J.V. Haxby et al.

    The distributed human neural system for face perception

    Trends Cognit. Sci.

    (2000)
  • R.J. Itier et al.

    Inversion and contrast polarity reversal affect both encoding and recognition processes of unfamiliar faces: a repetition study using ERPs

    Neuroimage

    (2002)
  • M.L. Itz et al.

    Neural correlates of facilitations in face learning by selective caricaturing of facial shape or reflectance

    Neuroimage

    (2014)
  • F. Jiang et al.

    Neural correlates of shape and surface reflectance information in individual faces

    Neuroscience

    (2009)
  • J.M. Kaufmann et al.

    Distortions in the brain? ERP effects of caricaturing familiar and unfamiliar faces

    Brain Res.

    (2008)
  • M. Lai et al.

    The role of skin texture and facial shape in representations of age and identity

    Cortex

    (2013)
  • D. Maurer et al.

    The many faces of configural processing

    Trends Cognit. Sci.

    (2002)
  • T. Minami et al.

    The effects of facial color and inversion on the N170 event-related potential (ERP) component

    Neuroscience

    (2015)
  • Y. Miyawaki et al.

    Visual image reconstruction from human brain activity using a combination of multiscale local image decoders

    Neuron

    (2008)
  • T. Naselaris et al.

    Bayesian reconstruction of natural images from human brain activity

    Neuron

    (2009)
  • D. Nemrodov et al.

    The time course of individual face recognition: a pattern analysis of ERP signals

    Neuroimage

    (2016)
  • S. Nishimoto et al.

    Reconstructing visual experiences from brain activity evoked by natural movies

    Curr. Biol.

    (2011)
  • S. Nishimoto et al.

    Reconstructing visual experiences from brain activity evoked by natural movies

    Curr. Biol.

    (2011)
  • A.J. O'Toole et al.

    Three-dimensional shape and two-dimensional surface reflectance contributions to face recognition: an application of three-dimensional morphing

    Vis. Res.

    (1999)
  • J.J. Richler et al.

    Holistic processing of faces happens at a glance

    Vis. Res.

    (2009)
  • B. Thirion et al.

    Inverse retinotopy: inferring the visual content of images from brain activation patterns

    Neuroimage

    (2006)
  • C.E. Thomaz et al.

    A new ranking method for principal components analysis and its application to face image analysis

    Image Vis Comput.

    (2010)
  • Q.C. Vuong et al.

    The role of surface pigmentation for recognition revealed by contrast reversal in faces and Greebles

    Vis. Res.

    (2005)
  • N. Abudarham et al.

    Reverse engineering the face space: discovering the critical features for face identification

    J. Vis.

    (2016)
  • R. Ahdid et al.

    A Survey on Facial Feature Points Detection Techniques and Approaches

    (2016)
  • J.J. Attick et al.

    Statistical approach to shape from shading: reconstruction of 3D face surfaces from single 2D images

    Plast. Reconstr. Surg.

    (1996)
  • V. Axelrod et al.

    Successful decoding of famous faces in the fusiform face area

    PloS One

    (2015)
  • I. Biederman et al.

    Neurocomputational bases of object and face recognition

    Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci.

    (1997)
  • V. Bruce et al.

    The use of pigmentation and shading information in recognising the sex and identities of faces

    Perception

    (1994)
  • V. Bruce et al.

    In the Eye of the Beholder: the Science of Face Perception

    (1998)
  • V. Bruce et al.

    Recognising facial surfaces

    Perception

    (1991)
  • A.M. Burton et al.

    Arguments against a configural processing account of familiar face recognition

    Perspect. Psychol. Sci. J. Assoc. Psychol. Sci.

    (2015)
  • R. Cabeza et al.

    Features are also important: contributions of featural and configural processing to face recognition

    Psychol. Sci.

    (2000)
  • J.S. Cant et al.

    Independent processing of form, colour, and texture in object perception

    Perception

    (2008)
  • J.D. Carlin et al.

    Adjudicating between face-coding models with individual-face fMRI responses

    PLoS Comput. Biol.

    (2017)
  • T.A. Carlson et al.

    Reaction time for object categorization is predicted by representational distance

    J. Cognit. Neurosci.

    (2014)
  • L. Chang et al.

    The code for facial identity in the primate brain

    Cell

    (2017)
  • L. Chang et al.

    The representation of colored objects in macaque color patches

    Nat. Commun.

    (2017)
  • C.H. Chang et al.

    Memory and perception-based facial image reconstruction

    Sci. Rep.

    (2017)
  • R.M. Cichy et al.

    Similarity-based fusion of MEG and fMRI reveals spatio-temporal dynamics in human cortex during visual object recognition

    Cerebr. Cortex

    (2016)
  • J.A. Collins et al.

    Knowledge is power: how conceptual knowledge transforms visual cognition

    Psychon. Bull. Rev.

    (2014)
  • I. Craw et al.

    Parameterising images for recognition and reconstruction

  • Cited by (0)

    View full text