The other-race effect describes the phenomenon of people being less accurate and slower in discriminating between faces from ethnic and racial groups other than their own (e.g., Herrmann et al., 2007; Lindsay, Jack, & Christian, 1991; Malpass & Kravitz, 1969; Walker & Tanaka, 2003). This difficulty could have important effects on the formation of social biases, with potentially severe consequences (e.g., mistaken eyewitness identification).

The dominant explanation is that other-race faces engage holistic perception to a lesser degree than own-race faces (Bukach, Cottle, Ubiwa, & Miller, 2012; Michel, Corneille, & Rossion, 2007). For example, the recognition of an own-race face may invoke an integrated face memory. In contrast, other-race faces may be encoded as sets of analytically separate facial features, requiring slower and relatively more error-prone feature-by-feature comparison.Footnote 1

Two tasks have been used to support this explanation of the other-race effect. Both tasks invoke the notion of a failure of selective attention to facial features to infer holistic encoding; that is, if the perceptual system fails to attend to an individual facial feature, then one concludes that this is due to holistic processing of faces. In the part-to-whole paradigm (Tanaka & Simonyi, 2016), participants are required to recognize a facial feature that is presented in different facial contexts. Holistic encoding is inferred from the recognition advantage of a target facial feature in a well-learned relative to a new facial context. The main idea is that a holistically processed face fails to selectively filter the irrelevant new context. The composite task (Hole, 1994) also relies on measuring a failure of selective attention to infer holistic encoding. In the complete composite-task design (Richler & Gauthier, 2013), participants match two faces using one half of a face (e.g., the top half) while ignoring the other half (e.g., the bottom half). The congruency and alignment of the top and bottom halves are manipulated. On congruent trials, the top and bottom of the test face are both the same or both different from the reference face; on incongruent trials, one half remains the same and the other half is different from the reference face. Holistic processing is inferred from better matching performance on the congruent than on the incongruent trials when faces are aligned, suggesting that participants cannot selectively ignore the irrelevant face half.

Several concerns about the lack of convergent validity of these tasks have been voiced (DeGutis, Wilmer, Mercado, & Cohan, 2013; Richler, Cheung, & Gauthier, 2011). For instance, closely related research focused in classification has shown that selective attention for upright faces is possible (Amishav & Kimchi, 2010; Fifić & Townsend, 2010; Fitousi, 2015; but see Richler, Palmeri, & Gauthier, 2013). Similar contradictory findings have also been observed in other-race effect studies using both the part-to-whole paradigm (DeGutis, Mercado, Wilmer, & Rosenblatt, 2013; Michel, Rossion, Han, Chung, & Caldara, 2006; Mondloch et al., 2010; Tanaka, Kiefer, & Bukach, 2004) and the composite task (Curby, Johnson, & Tyson, 2012; Harrison, Gauthier, Hayward, & Richler, 2014; Horry, Cheong, & Brewer, 2015). The effect appears to emerge in some studies only when the ignored half of a test face is always different from the reference face (Hugenberg & Corneille, 2009; Michel, Caldara, & Rossion, 2006; Michel et al., 2007; but see Mondloch et al., 2010). Inferences from these tasks are also potentially affected by aggregating data across individuals with qualitatively different strategies (Ashby, Maddox, & Lee, 1994; Estes & Maddox, 2005; Liew, Howe, & Little, 2016).

We claim that although the failure of selective attention is a necessary component of holistic perception, it is not sufficient to explain such perception. This is because analytic face encoding can also predict a failure of selective attention under very reasonable conditions (Fifić & Townsend, 2010). The main idea here is that under different organizations of mental processes during facial processing, it is possible to observe the failure of selective attention under analytic encoding. In this article, we address the issues above by focusing on identifying the organization of mental processes underlying holistic and analytic processing in individuals. Namely, we consider the logically independent concepts of processing architecture (serial, parallel, or coactive), stopping rule (self-terminating or exhaustive), and processing dependencies (independence or dependence).

Holistic encoding is most consistent with the notion of coactive processing (Fifić & Townsend, 2010). Coactive processing can be thought of as an interdependent, parallel, exhaustive processing system in which information is pooled together into a single decision process (Fifić & Townsend, 2010; Townsend & Wenger, 2004). By contrast, analytic encoding could entail several distinct types of mental architectures: for example, parallel self-terminating processing in which facial feature detectors race to accumulate evidence to some threshold value, with the first-finishing detector determining the decision. Although a parallel self-terminating system has the weakly holistic property of concurrent feature processing,Footnote 2 this property is combined with a strongly analytic property, in the self-terminating stopping rule. Fifić and Townsend argued that the parallel self-terminating mental architecture could also generate the failure-of-selective-attention effect in the part-to-whole paradigm, challenging the validity of using a failure of selective attention as a distinguishing feature of holistic and analytic encoding.

Systems factorial technology applied in the other-race effect study

We propose a study of the other-race effect using Systems Factorial Technology (SFT). SFT is a suite of methodological tools for discriminating between systems possessing different combinations of fundamental properties: (a) serial, parallel, or coactive processing; (b) exhaustive or self-terminating stopping rules; (c) stochastic independence or dependence; and (d) how processing changes with workload (e.g., with capacity; Little, Altieri, Fifić, & Yang, 2017; Townsend & Nozawa, 1995; Townsend & Wenger, 2004). These properties can, in turn, be linked to holistic and analytic encoding hypotheses in the manner defined above, and in the present case we will focus on the first two properties. The advantage of using SFT over other methodologies is that detailed time course information can be used to differentiate many different processing systems, making this method better equipped to test the hypothesis that holistic processing is stronger for own-race than for other-race faces.

In the present study, SFT was used to explore the underlying face-processing system encoding two second-order relational facial features: the eye-to-eye and nose-to-mouth separations. The two features are manipulated at different levels of saliency (H = high and L = low) by varying their differences from a constant reference source, and then factorially combining them in a 2 × 2 design. Each saliency level is manipulated so that the mean processing time is always slower for a low-salience condition than for a high-salience condition. Perceptually high saliency is defined as a manipulation that makes a single facial feature “stand out,” thus allowing for faster identification/discrimination.

We applied two statistics from SFT to analysis of the response times (RTs) from these factorially combined stimuli. The first statistic is the mean interaction contrast (MIC), which is expressed as

$$ \mathrm{MIC} = {\overline{RT}}_{\mathrm{HH}}-{\overline{RT}}_{\mathrm{HL}}-{\overline{RT}}_{\mathrm{LH}} + {\overline{RT}}_{\mathrm{LL}}, $$

where \( \overline{RT} \) represents mean response time. Each subscript represents the two facial features (in this case, eye separation and the distance between the nose and the mouth), and a capital L or H indicates the saliency level. Each term on the equation’s right-hand side indicates a single experimental face situation in which two features vary in saliency level. For example, the term \( {\overline{RT}}_{\mathrm{HH}} \) is the mean response time for a face that has both facial features at a high saliency level (see Fig. 1). In this study (as in Fifić & Townsend, 2010) a high-salience feature was defined as either a narrower eye-to-eye gap (e1 value in Fig. 1) or a narrower nose-to-mouth gap (n1 value in Fig. 1); in contrast, a low-salience feature was defined as a larger gap between the eyes (e2) or a larger nose-to-mouth gap (n2). Note that both low-salience features (e2, n2) are more similar to those features of the reference face (e3 and n3, correspondingly) than are the high-salience features (e1 and n1), and thus perceptually stand out less.

Fig. 1
figure 1

Face stimuli used in the own-race experiment (a) and the other-race experiment (b)

Full size image

The MIC value can be used to diagnose underlying processing properties. When MIC = 0, the two factors are linearly additive, suggesting that two processes are conducted in serial. When MIC > 0, the two factors have an overadditive relationship, implying a coactive or parallel self-terminating process. On the other hand, when MIC < 0, the two factors have an underadditive relationship, implying a parallel exhaustive process.

The second statistic that we applied is the survivor interaction contrast (SIC). When the above mean response time terms are replaced with their corresponding survivor function S(t), the following expression is obtained:

$$ \mathrm{SIC}= S{(t)}_{\mathrm{HH}}- S{(t)}_{\mathrm{HL}}- S{(t)}_{\mathrm{LH}}+ S{(t)}_{\mathrm{LL}}. $$

The resulting SIC statistic is a function over time (for t > 0; see Fig. 2). Testing SIC allows us to infer the processing models that cannot be inferred from the MIC. In an OR task, each combination of architecture and stopping rule predicts a qualitatively distinct functional shape for the SIC function (see Fig. 2; Townsend & Nozawa, 1995). By combining MIC and SIC, one can make a strong inference regarding the mental architecture and the decisional stopping rule.

Fig. 2
figure 2

Survivor interaction contrast (SIC) predictions for each of the candidate architectures and stopping rules

Full size image

Experiment

If own-race face perception is assumed to involve a strong holistic process, then our expectation would be that we should find coactivity. Conversely, if other-race processing involves strong analytic encoding, then we would expect to find serial processing of facial features. By contrast, if face perception involves weaker forms of either holistic or analytic encoding, we would expect to diagnose mental architectures that combine some of holistic (parallel, exhaustive, interdependent) and analytic (serial, self-terminating, independent) properties. We note that our goal was not to test the presence of the other-race effect (i.e., slower or more error-prone responding for other-race than for own-race faces) but to examine the architecture and stopping rule underlying processing of each type, to elucidate theories of own- and other-race face processing.

Method

Participants

Ten Taiwanese students (two male, eight female; ages 19–26, mean age 21.7) from National Cheng Kung University were reimbursed NTD 1,200 for their participation in two experiments. Own-race faces were presented in Experiment 1 and other-race faces were presented in Experiment 2 (see Fig. 1 for an example) with the order counterbalanced across participants. Participants S1–S5 completed the own-race experiment first; participants S6–S10 completed the other-race experiment first. All the participants had normal or corrected-to-normal vision.

Apparatus

The experiments were conducted in a darkened room on a 2.40-GHz Intel Pentium IV processor running E-Prime 1.1 (Schneider, Eschman, & Zuccolotto, 2002). The stimuli were presented on a 19-in. CRT monitor (CTX VL951T) with a refresh rate of 85 Hz and resolution of 1,024 × 768. A chinrest was used to prevent head movements.

Design, stimuli, and procedure

At the beginning of each session, a reference face was presented, and participants were asked to memorize the reference face, because classification would be based on comparison between the eye-to-eye and nose-to-mouth separations of the reference face and the test face (see Fig. 1). Stimulus selection is described in the supplementary materials. On each trial, after presentation of a fixation point for 350 ms, a test face (12° in height and 8° in width) was presented centrally until a response was made. Participants were instructed to quickly and accurately make a classification decision and to respond Category A (“Amy” for a Caucasian face or “美玲” for an Asian face) either when the separation of two eyes of the test face was narrower than that of the reference face or when the separation between the nose and mouth of the test face was narrower than that of the reference face, and otherwise to respond Category B (“Mary” for a Caucasian face or “淑芬” for an Asian face). The intertrial interval was 500 ms.

There were four Category A test stimuli (HH, HL, LH, and LL)Footnote 3 and one Category B stimulus (XX). The presentation of each category was balanced, with half being from Category A and the other half from Category B. The four test stimuli in Category A were equally probable within each block. There were a total of five sessions, and each session lasted for about an hour. On each session, participants first completed a practice block of 40 trials and then ten blocks of 80 test trials.

Data analysis

The data were analyzed using the SFT approach (Townsend & Nozawa, 1995). Two critical assumptions of SFT are that (1) the salience manipulation on a given dimension was effective—that is, RT(H) < RT(L) for some time t; and (2) the salience manipulation on one dimension affects that dimension and not the other. This assumption is termed effective selective influence (Dzhafarov, 1999; Schweickert, Giorgini, & Dzhafarov, 2000; Townsend & Thomas, 1994). These tests are presented in the supplementary materials.

We next computed MIC and SIC, to infer the mental architecture and the stopping rule. To test whether MIC equaled zero, we first analyzed the mean RTs (for Category A) using a two-way ANOVA. Second, a nonparametric bootstrapping method was used to simulate 1,000 samples for each condition and construct the 95% confidence interval (CI) for MIC (see Van Zandt, 2000, for details), to test whether the bootstrapped 95% CI for MIC included zero. We also adopted a nonparametric bootstrapping method to construct the 95% CI for SIC, to confirm whether SIC values were zero for all times t (i.e., consistent with serial self-terminating processing). The empirical survivor functions and bootstrapped MIC results are presented in the supplementary materials. Finally, we also applied a number of statistical tests directly to the SIC function (see, e.g., Houpt, Blaha, McIntire, Havig, & Townsend, 2014; Houpt &Townsend, 2010). These results, which were generally consistent with our other results, are also reported in the supplementary materials.

Results and discussion

Own-race faces (Asian faces)

Table 1 summarizes the mean performance (accuracy and mean RT) in each condition and the MIC for each participant. We found no correlation between accuracy and mean RTs (r = .41, p = .23), suggesting no speed–accuracy trade-off in the classification task performance. Accuracy was high in all conditions except the LL condition, suggesting that there was no response bias toward responding with one of the categories (Category A or Category B). These results also supported effective selective influence on classification accuracy. We then conducted a two-way analysis of variance (ANOVA) on the group-level classification RTs using the Salience of Eye Separation (high/low) and the Salience of Lip Position (high/low) as factors; the results showed that the main effects of both factors were significant [eye separation: F(1, 18888) = 97.81, p < .001; lip position: F(1, 18888) = 464.78, p < .001], suggesting that the salience manipulations were effective at the mean RT level (see the supplementary materials for the individual participant results and tests of selective influence at the distributional level).

Table 1 Mean performance in classifying own-race (Asian woman) faces
Full size table

The results showed that all the participants had similar patterns of MIC and SIC (see Table 1 and Fig. 2 ), suggesting that they adopted similar decision strategies. All participants had positive MICs (see Table 1). In addition, the results from Fig. 3 show that the simulated 95% CIs for SICs were positive at all times t. The results of the two-way ANOVA on the group-level RTs showed a significant interaction between the two factors [F(1, 18888) = 52.85, p < .001], suggesting that nonserial processing took place in the classifying of own-race faces. Seven participants had a significant interaction, and the other three participants (S5, S8, and S9) had a marginally significant interaction (ps < .1) (see Supplemental Table S2). The simulated 95% CIs for MIC from the nonparametric bootstrapped samples also confirmed this pattern of results (see Supplemental Fig. S3). Eight of the participants had 95% CIs for MIC that did not include 0, and the remaining two participants (S8 and S9) had 95% CIs for MIC that slightly included 0. Taken together, the MIC and SIC results support an inference of a parallel self-terminating processing strategy.

Fig. 3
figure 3

Results for the SIC (thick solid lines) and its 95% confidence interval (CIs) (dotted lines) for each participant in the own-race experiment

Full size image

Other-race (Caucasian) faces

Table 2 summarizes the mean performance (in terms of accuracy and mean RT) of each condition along with the MIC for each participant. The correlation between accuracy and mean RT (r = .46, p = .08) was not significant at the .05 level, suggesting no speed–accuracy trade-off. Overall, accuracy was high in all the conditions except the LL condition, supporting effective selective influence on classification accuracy. In addition, the results of a two-way ANOVA on the group-level RTs showed that the main effects of the two factors were significant [eye separation, F(1, 18258) = 37.50, p < .001; lip position, F(1, 18258) = 597.54, p < .001], suggesting that the effective selective-influence assumption was satisfied at the mean RT level (see the supplementary materials for the individual results and tests of selective influence at the distributional level).

Table 2 Mean performance in classifying other-race (Caucasian woman) faces
Full size table

Most participants (nine out of ten) had MICs around 0 (see Table 2). In addition, the results from Fig. 4 show that the simulated 95% CIs for their SICs include 0 at all times t. The results of two-way ANOVAs on RTs showed that the interaction effect did not reach the significance level [F(1, 16299) = 0.136, p = .71], suggesting that these participants adopted serial processing in classifying other-race faces. See Supplemental Table S3 for the individual results. We term this group of participants Group SS in Table 2. The simulated 95% CIs for MIC confirmed this pattern of results (see Supplemental Fig. S4). Combining MIC and SIC, we inferred that these participants adopted a serial self-terminating processing strategy. Because we found that the survivor functions of the HH and LH conditions overlapped and that the survivor functions of the HL and LL conditions overlapped (see Supplemental Fig. S2), we can further infer that the participants processed the lip position first and exclusively (i.e., the eye separation was not processed unless the information on lip position was not sufficient for decision making).Footnote 4

Fig. 4
figure 4

Results for the SIC (thick solid lines) and its 95% CIs (dotted lines) for each participant in the other-race experiment

Full size image

One participant (S10) had a positive MIC [see Table 2; F(1, 1995) = 33.65, p < .001], suggesting nonserial processing. Combining MIC and SIC (see Fig. 4 and Supplemental Fig. S4), we inferred that participant S10 adopted a parallel self-terminating processing strategy for the other-race faces.

Conclusion

Using the strong inference techniques made available via SFT, we identified that whereas own-race faces are processed in a parallel self-terminating fashion, other-races faces are processed in a serial self-terminating fashion. Specifically, we found that other-race faces were differentiated on the basis of a single feature, whereas both features were used for own-race faces.

We used a Taiwanese sample; consequently, it is worthwhile to consider cross-cultural results on analytic versus holistic face processing. Several studies have suggested that culture shapes visual perception as well as cognition (Boduroglu, Shah, & Nisbett, 2009; Chua, Boland, & Nisbett, 2005; Goh & Park, 2009; Kitayama, Duffy, Kawamura, & Larsen, 2003; Masuda & Nisbett, 2006), with Western participants adopting a more analytic processing style and Eastern participants adopting a more holistic processing style. Our results provide a more nuanced conclusion, in that we found no clear evidence of holistic encoding in our sample for either own- or other-race faces. Nonetheless, future research should examine this result using a sample of Western participants.

In addition to the absence of holistic processing in our data, our results indicate that eye separation and nose-to-mouth separation are processed in a similar fashion to other, more basic, separable perceptual dimensions, such as size and shape (see, e.g., Fifić, Nosofsky, & Townsend, 2008; Little, Nosofsky, & Denton, 2011; Little, Nosofsky, Donkin, & Denton, 2013; Moneer, Wang, & Little, 2016). One goal for future research is to extend the methodology to include measures of capacity and to provide tests of different processing architectures under an exhaustive AND stopping rule. In the former case, as compared to a benchmark of unlimited-capacity, independent parallel processing, we might expect to find fixed capacity coupled with serial processing in our other-race face condition (Townsend & Nozawa, 1995). This would provide stronger, converging evidence to confirm the present results. On the other hand, in an AND task, in which participants are forced to take account of both features, it may be more likely for participants to adopt more configural or Gestalt-like processing, which in turn might be reflected by coactivity (see, e.g., Blaha, 2017). We note, however, that Fifić and Townsend (2010) did not consistently find coactivation in their AND task using similar eye separation and nose-to-mouth dimensions.

The notable take-home message is that we did not find any evidence for coactivity in the processing of own-race faces, contrary to the strong-holistic-encoding hypothesis. Serial processing, often as a result of controlled attention (Chang, Little, & Yang, 2016; Shiffrin & Schneider, 1977), may operate more slowly than more automatic, parallel processing. Hence, parallel processing of own-race face features is sufficient to explain the own-race processing benefit that has been seen in several studies.