Electrophysiological correlates of the composite face illusion: Disentangling perceptual and decisional components of holistic face processing in the human brain

Abstract

When the bottom halves of two faces differ, people’s behavioral judgment of the identical top halves of those faces is impaired: they report that the top halves are different, and/or take more time than usual to provide a response. This behavioral measure is known as the composite face effect (CFE) and has traditionally been taken as evidence that faces are perceived holistically. Recently, however, it has been claimed that this effect is driven almost entirely by decisional, rather than perceptual, factors (Richler, Gauthier, Wenger, & Palmeri, 2008). To disentangle the contribution of perceptual and decisional brain processes, we aimed to obtain an event-related potential (ERP) measure of the CFE at a stage of face encoding (Jacques & Rossion, 2009) in the absence of a behavioral CFE effect. Sixteen participants performed a go/no-go task in an oddball paradigm, lifting a finger of their right or left hand when the top half of a face changed identity. This change of identity of the top of the face was associated with an increased ERP signal on occipito-temporal electrode sites at the N170 face-sensitive component (∼160 ms), the later decisional P3b component, and the lateralized readiness potential (LRP) starting at ∼350 ms. The N170 effect was observed equally early when only the unattended bottom part of the face changed, indicating that an identity change was perceived across the whole face in this condition. Importantly, there was no behavioral response bias for the bottom change trials, and no evidence of decisional biases from electrophysiological data (no P3b and LRP deflection in no-go trials). These data show that an early CFE can be measured in ERPs in the absence of any decisional response bias, indicating that the CFE reflects primarily the visual perception of the whole face.

Introduction

The efficiency with which human adults are able to recognize hundreds or perhaps even thousands of faces has been proposed to depend on the ability to process faces in a holistic manner. To process a face holistically means that rather than processing local facial features (eye, nose, mouth, ears…) independently, the face is processed as a single global perceptual representation, i.e. as a whole. This implies that a modification to a subset of features, or even to one feature, is capable of altering the perception of the overall face (Galton, 1883). This dependant relationship between the processing of local features and of the entire face has been demonstrated in a number of classical behavioral experiments (e.g., Sergent, 1984, Tanaka and Farah, 1993, Tanaka and Sengco, 1997). However, the most compelling evidence favoring the idea that faces are processed holistically comes from the composite face effect (CFE, Young, Hellawell, & Hay 1987).

The CFE was first described (Young et al., 1987) as an increase in the time needed to name the top part of a familiar face (cut below the eyes) when it is aligned with the bottom part of another face, relative to the time needed to name the top part when the same top and bottom parts are laterally offset (i.e. misaligned). With respect to unfamiliar faces, this effect emerges as a result of a visual illusion: identical top halves of faces are perceived as being slightly different when they are aligned with different bottom halves (see Fig. 1; see also Rossion, 2008, Rossion and Boremanse, 2008). This visual illusion nicely demonstrates that facial features (here the two halves of the face) cannot be perceived independently from one another, but rather that the face is perceived as a whole.

Since originally being reported, the CFE has been observed consistently in matching tasks that require discrimination of individual, unfamiliar, composite faces (see Hole (1994) for a first demonstration). In these tasks, participants are more likely to judge two identical top halves of a face as being different when they are presented with different bottom parts that are aligned with the top parts, in comparison to when the top and bottom parts are misaligned. This happens despite the fact that the bottom parts of the face are irrelevant for the task, (e.g., Goffaux and Rossion, 2006, Le Grand et al., 2004, Michel et al., 2006, Robbins and McKone, 2007, Taubert and Alais, 2009) and that eye gaze remains fixed on the top parts of the faces (de Heering, Rossion, Turati, & Simion, 2008). When faces are inverted, the composite effect either disappears or is strongly attenuated (e.g., Goffaux and Rossion, 2006, Hole, 1994, Hole et al., 1999, Rossion and Boremanse, 2008, Young et al., 1987).

Overall, the existing evidence demonstrating interdependent processing of the top and bottom halves of composite faces has led researchers to propose that the functional locus of the CFE is perceptual (Farah et al., 1998, Rossion, 2008, Rossion, 2009). That is, that faces are perceived holistically. However, the perceptual nature of holistic face processing in general (Wenger & Ingvalson, 2002), and of the CFE in particular, has recently been challenged (Richler, Gauthier, Wenger, & Palmeri, 2008; see also Cheung et al., 2008, Richler, Tanaka, et al., 2008). Specifically, Richler, Gauthier, et al. (2008) used the multidimensional generalization of signal detection theory known as general recognition theory (GRT; e.g., Ashby and Townsend, 1986, Kadlec and Townsend, 1992, Thomas, 1995) to analyze data from a behavioral same/different discrimination task with composite faces. Using GRT, the authors inferred that the composite effect is driven both by perceptual, as well as decisional (i.e. response related) factors, with the most consistent evidence pointing to a decisional locus of the effect. Although the authors admitted that signal detection theory could not speak to how decisional factors may influence the outcome of a composite face task, they suggested that decisional biases may be gained through experience. In particular, the authors postulated that extensive experience with faces may cause people to develop a deeply ingrained assumption that face parts change together. The strength of this assumption would make it difficult to override during an experiment, even when participants are instructed to selectively attend only to the top half of the face. This bias, in turn, would affect the percept of the face, as measured by the composite face task. Richler, Gauthier and colleagues (2008) suggested that such a decisional bias, because it relates to a domain of expertise, may be deeply ingrained and relatively immune to task influences.

It is reasonable to assume that decisional response biases may arise as a consequence of the perception of the composite face illusion. That is, when observers are presented with two identical top halves of faces each paired with a different bottom half, they tend to make more errors (i.e. respond “different”) and/or take more time when having to match the face halves. However, the claim that holistic face processing is driven by decisional factors (Richler, Gauthier, et al., 2008) seems difficult to reconcile with the available neuroimaging and electrophysiological evidence from studies of the CFE. For example, fMRI studies have shown that following adaptation to an aligned composite face, there was a significantly larger response to the same top part of a face when it was aligned with a different bottom part as compared to when it was aligned with the same bottom part. This “neural CFE” was found particularly in the right middle fusiform gyrus (rMFG), as well as less strongly in the right inferior occipital gyrus (IOG) (Schiltz et al., 2010, Schiltz and Rossion, 2006) two areas of the human visual cortex that have been shown to respond preferentially to faces (“FFA” and “OFA” respectively; Gauthier et al., 2000, Kanwisher et al., 1997, Puce et al., 1995, Sergent et al., 1992). These results, which were not found when faces were spatially misaligned or inverted, suggests that neurons in these visual areas integrate information from the two aligned face halves into a single representation of the whole face.

Most recently, evidence from scalp event-related potentials (ERPs) showed that following adaptation, top halves of faces with different aligned bottoms produced larger responses than the same top halves of faces with the same bottoms, as early as 160 ms post-stimulus onset (Jacques & Rossion, 2009),¹ on the face-sensitive N170 component (Bentin, McCarthy, Perez, Puce, & Allison, 1996; see Rossion and Jacques, 2008, Rossion and Jacques, 2010). Again, this was not the case when the faces where spatially misaligned. Since the face-sensitive N170 component at occipito-temporal recording sites is independent of decision making (e.g., Philiastides & Sajda, 2006) and reflects the earliest stage at which individual face representations are accessed (Jacques et al., 2007, Jacques and Rossion, 2006), these findings suggest that the initial perceptual representation of an individual face in the human brain is inherently holistic.

Together, fMRI and ERP evidence suggest that information from the two face halves are integrated quite early following stimulus onset, and that this operation is performed by areas of the visual cortex that are known to respond preferentially to faces. However, although the recent ERP findings (Jacques & Rossion, 2009) seem to indicate that there is little room for decisional factors to come into play before a face is processed holistically in face-sensitive areas of the visual cortex, this issue deserves further consideration. Importantly, in the critical condition of the ERP composite face effect, i.e. when the unattended bottom half of the face changes in the aligned condition, observers make errors and are slower to make their behavioral decision than when the same face halves are spatially misaligned (Jacques & Rossion, 2009). This implies that the ERP effects related to the composite face illusion have been observed in a paradigm in which behavioral decisional biases are also, concomitantly, observed. The unresolved question is thus whether electrophysiological evidence for holistic perception of faces can be found in the absence of any kind of decisional bias. Importantly, should this be the case, it would refute the idea that decisional biases give rise to the perceptual effect and are the driving factor in the manifestation of the CFE. Addressing this issue was the focus of the present study.

For this study, we adapted a face identity adaptation paradigm in the context of a visual oddball ERP paradigm. High-density electroencephalogram (EEG) was recorded as participants viewed sequentially presented aligned composite faces. A face with same top and bottom halves was presented on 78% of the trials. On 11% of the trials, the top half of the face was changed, and on the remaining 11% of the trials the bottom half of the face was changed. Participants attended to the top of the face and responded by lifting their finger only when the top half of the face changed (Go response; see Fig. 2).

With this new paradigm, we expected first to replicate the findings of Jacques and Rossion (2009). That is, we expected to observe an increase of N170 amplitude on trials in which the top half of the face changed (relative to no change), and crucially, we also expected to find the same modulation for trials in which the bottom half of the face changed, reflecting the perception of the composite illusion.

Second, and most importantly, we aimed to disentangle perceptual from decision-related processes of the CFE. With respect to this goal, the strength of the present paradigm lies in the fact that active responses were recruited for top change trials but not for bottom change or same trials. Therefore, in addition to the early ERP components associated with perceptual face processes, we expected to observe response-related components, but only for top change trials. In particular, we expected to observe a P3b, an event-related evoked potential peaking on parietal sites that has been proposed to reflect a bridge between perceptual processing and response processing (Verleger, Jaskowski, & Wascher, 2005). Amplitude of the P3b has been described as having an inverse relationship with the probability of the target stimulus (Duncan-Johnson & Donchin, 1977). We thus expected to observe a sizable P3b in the present study, due to the quite low probability of the occurrence of a top change stimulus. Although the exact function of the P3b is not entirely clear, it has been suggested that this component reflects a process of monitoring whether or not the decision to classify some stimulus is appropriately transformed into action (Verleger et al., 2005).

The P3b overlaps with the electrophysiological correlate of the overt go response, i.e. the lateralized readiness potential (LRP) associated with unilateral hand movement (Verleger, Paehge, Kolev, Yordanova, & Jaskowski, 2006). In the present experiment we used a go/no-go paradigm with half of the responses being provided by each hand. This allowed for the ERP effects related to perception to be separated from later processes, including decisional and motor responses, via an analysis of the LRP along the time-course of holistic face processing. The LRP is an electrophysiological potential, measured over central cites corresponding to motor cortex, that reflects the preparation of motor processes prior to the ballistic point of no return (Coles, 1989, Miller and Hackley, 1992; for its use in face processing experiments see e.g., Martens, Leuthold, & Schweinberger, 2010). Prior to a movement made in response to a stimulus, EEG activity becomes more negative at electrodes over motor cortex contralateral to the hand that will be used to respond. This excess contralateral negativity in the EEG is reflected in the LRP. Crucially, the LRP is also sensitive to low levels of response activation that are not associated with the ultimate overt behavior. This means that the LRP can develop based on an initial indication that a response may be needed, even if the response is subsequently aborted and never actually executed (e.g., de Jong et al., 1988, Hackley and Miller, 1995, Miller, 1998). Thus, the generation of response preparation, as reflected by the LRP, when a response is not required and ultimately not made, indicates that participants are initially biased to act in a particular way based on a particular decisional strategy, even though the actual response is ultimately made on the basis of a different strategy (Coles, 1989, Gratton et al., 1992, Miller and Hackley, 1992). In other words, the presence of an LRP on no-go trials would indicate that there is a discrepancy between a decision based on initial stimulus analysis, which may activate a response preparation, and the final decision regarding the response that is executed following the completion of stimulus analysis.

By measuring the LRP we were able to test for the existence of a response bias based on a deeply ingrained assumption that all face parts change together (Richler, Gauthier, et al., 2008). Specifically, we hypothesized that if adults inherently assume that all face parts change together, in the current experiment they should begin to prepare a manual response as soon as they detect any change within the face. However, on trials in which the bottom (i.e. the irrelevant, unattended) part of the face changes, to arrive at a correct response (i.e. no response), participants would need to override this bias, and abort their response preparation. The preparation and subsequent abortion of the response on ultimately correct trials should be reflected by the onset and subsequent return to baseline of an LRP on bottom change trials. Rather, if the composite effect is not driven by this type of decisional bias, no LRP should be observed at all on bottom change (composite illusion) trials. The absence of an LRP on bottom change trials would indicate that subjects do not need to overcome any decisional bias to arrive at a correct (non) response when the bottom of the face changes.

To summarize, there were three possible outcomes in the present study. First, an early sensitivity to a change in the bottom half of the face (N170 time-window) could be observed at occipito-temporal sites despite no behavioral response bias, no evidence of a decision-related component (P3b), and no effect of response hand lateralization (LRP) in this condition. This would demonstrate that holistic processing of a face is profoundly perceptual in nature and exists even in the absence of decisional/response biases. Second, the early perceptual effect may be absent, and P3b and LRP may be observed for the no-go bottom change trials. This outcome would support the view that holistic face processing, as assessed in the composite task, has a primarily decisional locus (Richler, Gauthier, et al., 2008). Finally, one may observe both an early effect at occipito-temporal sites associated with a perceptual locus, and P3b, LRP responses for the no-go bottom change trials. The presence of these decision-related components in the bottom change condition would not contradict the view that the origin of the effect is perceptual, provided that the effect is also clearly observed at posterior occipito-temporal sites starting at an earlier time-window. In this case, one could then measure and attempt to relate the perceptual and decision-related components, in order to better understand the cause and consequence of the holistic processing effect measured by the composite face paradigm.

Finally, there were two other novel aspects of the present study worth mentioning. First, in the study by Jacques and Rossion (2009), the condition in which the whole face was repeated (i.e. “same”) was analyzed in comparison to two conditions, one in which only the bottom half of the face changed, and another in which both the top and bottom halves changed simultaneously. In the present study, rather, whenever there was a change in the face, it was only either the top (attended) half or the bottom half which was modified. Hence, the comparison between the top change and bottom change conditions was more adequately balanced in terms of the overall amount of physical change (compared to the “same” condition). Second, we recorded EEG from a much higher density array of electrodes (128 vs. 58) with a more complete sampling of the visual regions of the brain, allowing us to make more precise topographical maps of the effects of interest.