Attentional conditions differentially affect early, intermediate and late neural responses to fearful and neutral faces

Abstract The processing of fearful facial expressions is prioritized by the human brain. This priority is maintained across various information processing stages as evident in early, intermediate and late components of event-related potentials (ERPs). However, emotional modulations are inconsistently reported for these different processing stages. In this pre-registered study, we investigated how feature-based attention differentially affects ERPs to fearful and neutral faces in 40 participants. The tasks required the participants to discriminate either the orientation of lines overlaid onto the face, the sex of the face or the face’s emotional expression, increasing attention to emotion-related features. We found main effects of emotion for the N170, early posterior negativity (EPN) and late positive potential (LPP). While N170 emotional modulations were task-independent, interactions of emotion and task were observed for the EPN and LPP. While EPN emotion effects were found in the sex and emotion tasks, the LPP emotion effect was mainly driven by the emotion task. This study shows that early responses to fearful faces are task-independent (N170) and likely based on low-level and configural information while during later processing stages, attention to the face (EPN) or—more specifically—to the face’s emotional expression (LPP) is crucial for reliable amplified processing of emotional faces.


Introduction
Emotional facial expressions constitute a significant part of communication as they transfer crucial non-verbal signals to others. Therefore, their processing is assumed to be prioritized when compared to neutral facial expressions. In line with influences (Rossion and Caharel, 2011;Schindler et al., 2019b). Findings on how fearful faces modulate the P1 are mixed, with some studies reporting larger amplitudes for emotional compared to neutral expressions (e.g. see Mühlberger et al., 2009;Foti et al., 2010;Müller-Bardorff et al., 2018), while others do not find such an effect (e.g. see MacNamara et al., 2011;Wieser et al., 2012;Smith et al., 2013). The N170 is viewed as a structural and configural encoding component (Eimer, 2011) and found to be reliably modulated by fearful expressions (Hinojosa et al., 2015). The subsequent early posterior negativity (EPN) has previously been related to early attentional selection processes (Schupp et al., 2004;Wieser et al., 2010). It has been observed as a differential negativity in studies contrasting emotional with neutral expressions (Schupp et al., 2004;Schindler et al., 2017). Several studies have reported its enlargement for fearful stimuli (e.g. see Walentowska and Wronka, 2012;Wieser et al., 2012;Schindler et al., 2019a), while in other studies no differences between fearful and neutral faces were observed (e.g. see Santos et al., 2008;Herbert et al., 2013). Finally, the late positive potential (LPP) is indicative of controlled attentional processes and stimulus evaluation (Schupp et al., 2006;Hajcak et al., 2009), particularly when the appraisal of affective meaning is involved (Schupp et al., 2006;Wessing et al., 2013). With respect to emotional facial stimuli, some studies have found fearful faces to elicit larger late positivities than neutral faces (e.g. see Wieser et al., 2012;Herbert et al., 2013;Grunewald et al., 2019), but others do show no differential effects (e.g. see Peltola et al., 2018;Schindler et al., 2019a).
The attentional focus might be an underlying mechanism to explain the inconsistent findings regarding emotional ERP effects. Here, attentional instructions should amplify the emotional modulation of ERPs involved in the assumed processing stage. It is expected that early components should show a higher automaticity of emotional amplification while later components should predominantly depend on the attentional condition and its instruction (for a review, see Schindler and Bublatzky, 2020). A mechanistic explanation for this pattern would be that at early stages, emotion effects are mainly based either on feedforward processes within the visual cortex (Rossion and Caharel, 2011) or on rapid local feedback loops or rapid initial amygdala feedback (e.g. see Müller-Bardorff et al., 2018). In contrast, the LPP is hypothesized to reflect the activation of broad occipitoparietal regions (Sabatinelli et al., 2007(Sabatinelli et al., , 2014Liu et al., 2012) linked to higher cognition, such as stimulus evaluation and affective labelling (Schupp et al., 2006;Hajcak et al., 2009). Here, both emotional feedback from the amygdala and top-down signalling from frontoparietal attention networks might synergistically increase the processing of emotional stimuli (Pourtois et al., 2013). As these later processes are more vulnerable to competing tasks (e.g. see Schupp et al., 2007), they should only be affected by emotion if cognitive resources are available.
For the P1, preliminary evidence indicates that enlarged amplitudes for fearful faces are more often reported when faces serve as distracters, e.g. when an overlaid object has to be discriminated (e.g. Santos et al., 2008). Distraction tasks also seem to attenuate or abolish emotion effects for the consecutive components leading to absent N170 (Santos et al., 2008;Framorando et al., 2018;Li et al., 2018) or EPN modulations for fearful or angry expressions (Li et al., 2018;Wu et al., 2019). For the LPP, emotion effects for threatening faces seem to depend even more heavily on specific task sets. Several studies observe no significant differences during perceptual tasks (Müller-Bardorff et al., 2016), passive viewing tasks (Peltola et al., 2018) or tasks directing attention to the face but not to the expression (Syrjänen et al., 2018).
Among the many studies comparing specific tasks (e.g. Neath-Tavares and Itier, 2016;Wu et al., 2019), there are-to the best of our knowledge-only four studies to have realized a design with more than two task conditions (Rellecke et al., 2012;Valdés-Conroy et al., 2014;Itier and Neath-Tavares, 2017 ;Acunzo et al., 2019). Moreover, as some studies focused only on early ERPs (Itier and Neath-Tavares, 2017;Acunzo et al., 2019), the picture which these studies draw has remained rather incomplete. Overall, P1 emotion effects remain inconclusive (reporting taskindependent effects, Rellecke et al., 2012;Valdés-Conroy et al., 2014; reporting a lack of task-independent effects, Itier and Neath-Tavares, 2017;Acunzo et al., 2019), while emotion effects are consistently found for the N170 and EPN, being either taskinsensitive (Rellecke et al., 2012;Neath-Tavares, 2017) or task-modulated (N170, Valdés-Conroy et al., 2014). Surprisingly, analyses of the LPP provide conflicting interpretations, with one study reporting emotion effects are strongest during an emotion decision task (Rellecke et al., 2012) and the other study not supporting this finding (Valdés-Conroy et al., 2014). Taken together, a clear picture of task-dependent emotion effects cannot be derived from the literature, and studies are needed which systematically vary task instructions to directly test how neural responses across all relevant time windows depend on the attended feature.

The current design and hypothesis
In this pre-registered study (https://osf.io/qgwzd), we investigated feature-based attention effects on early (P1, N170), midlatency (EPN) and late (LPP) processing stages for fearful vs neutral faces. To this end, participants (N = 40) were presented fearful and neutral faces, always displayed with an overlay of thin horizontal or vertical lines. We used three attention tasks to gradually increase the attention to emotionally relevant features of the facial stimuli (line discrimination, sex discrimination, facial expression discrimination). Based on the line of argumentation outlined above, we predicted that the later the component of the ERP, the higher the relevance of the attentional focus on emotionally relevant features for finding emotional modulations of these components. In particular, we expected the emotional P1 modulation to be strongest in the perceptual task, while the N170 and especially the EPN should show stronger emotion effects in the sex and emotion decision task. LPP emotion effects were expected only in the emotion decision task.

Participants
In total, 42 participants were recruited from the University of Münster. They all gave written informed consent and received 10 euros per hour for participation. One participant was excluded due to a neurological disorder and one due to noisy EEG. According to the registered data sampling plan, this led to a final sample of 40 participants, for which power calculations using G * Power 3.1.7 (Faul et al., 2009) showed a power of >90% to detect medium effect sizes (f = 0.25). The resulting 40 participants (30 female) were 23.33 years old (s.d. = 3.08) on average. All participants had normal or corrected-to-normal eye vision, were right-handed and had no reported history of neurological or psychiatric disorders.

Stimuli
The facial stimuli were taken from the Radboud Faces Database, exhibiting well-standardized eye position and head orientation (Langner et al., 2010). Cut-out grey-scaled faces of 32 identities (16 male and 16 female), depicting neutral and fearful expressions, were chosen from this database. The faces were shown with an overlay of five thin horizontal or vertical lines, displayed within the boundaries of the face (horizontal lines 1.7 length; vertical lines 2.3 length; thickness 0.01; centred around x = 0.1, y = −0.1).

Procedure
While participants were prepared for the EEG, they responded to a demographic questionnaire as well as to the BDI-II and STAI Trait questionnaire (Spielberger et al., 1999;Hautzinger et al., 2009) and to a short version of the NEO-FFI (Körner et al., 2008). Participants were seated 60 cm in front of a gamma-corrected display (NBC Multisync E231W 23 ) running at 60 Hz with a Michelson contrast of 0.9979 (L min = 0.35 cd/m 2 ; L max = 327.43 cd/m 2 ). The background was set to medium grey (RGB 128,128,128). Participants were instructed to avoid eye movements and blinks during stimulus presentation. Participants started either with the perceptual decision, the sex decision or the emotion decision task, while task order and response buttons (x and m) were counterbalanced. In each trial, participants were confronted with a two-alternative forced choice task and had to decide whether the overlaid line orientation was horizontal or vertical, whether the sex was male or female or whether the expression was fearful or neutral. In all tasks, trial structure and stimuli were identical. Each trial started with the presentation of a fixation cross for 800-1000 ms after which a face was displayed for 100 ms. The face's display was followed by the presentation of another fixation cross for 1500 ms during which responses were recorded. Each face was repeated twice for a total of 64 fearful and 64 neutral faces presented in each task condition, summing up to a total of 384 trials. Of note, each identity was in total repeated six times with a neutral and six times with a fearful expression.

EEG recording and pre-processing
EEG signals were recorded from 64 BioSemi active electrodes using BioSemi's Actiview software (www.biosemi.com). Four additional electrodes measured horizontal and vertical eye movements. Recording sampling rate was at 512 Hz. Offline data were re-referenced to average reference and filtered with a high-pass forward filter of 0.01 (6 db/oct) as well as a 40 Hz low-pass zero-phase filter (24 db/oct). Recorded eye movements were corrected using the automatic eye-artefact correction method implemented in BESA (Ille et al., 2002). Remaining artefacts were rejected based on absolute threshold (120 μV), gradient (75) and low signal change (0.01). Noisy EEG sensors were interpolated using a spline interpolation procedure. The stimuli on the liquid crystal display (LCD) display in use were found to have a trigger delay of 15 ms, as measured by a photodiode. This delay was corrected during epoching. Filtered data were segmented from 200 ms before stimulus onset to 1000 ms after stimulus presentation. Baseline correction used the 200 ms before stimulus onset. On average, 5.29 electrodes were interpolated and 23.17% trials were rejected. On average this resulted in 50 fearful and 50 neutral faces kept for the perceptual, 49 fearful and 49 neutral trials kept for the sex and 48 fearful and 49 neutral faces kept for the emotion task across participants. For kept trials, no main effect of emotion (F (1,39) = 0.06, P = 0.814, η P 2 = 0.001) and of task (F (2,78) = 0.55, P = 0.579, η P 2 = 0.014) and no interaction were found (F (2,78) = 0.85, P = 0.430, η P 2 = 0.021).

Data analyses
All data were analysed using two (emotion, fearful, neutral) by three (task, perceptual, sex, emotion) repeated measures analyses of variance (ANOVAs). For analyses of the P1, N170 and EPN, laterality (left/right) was included as a factor. Partial eta-squared (η P 2 ) was used to describe effect sizes (Cohen, 1988). The pre-condition of sphericity was tested using Mauchly's test of sphericity, and in case of a violation, degrees of freedom were corrected in accordance with Greenhouse-Geisser. For behavioural data, reaction times above 100 ms and below 1500 ms were regarded as correct responses ('hit'). Please note that for two participants, responses were re-coded in one task condition (sex and perceptual task). EEG scalp data were statistically analysed using EMEGS (Peyk et al., 2011). Time windows were segmented into intervals from 80 to 100 ms for the P1, from 130 to 170 ms for the N170, from 250 to 350 ms for the EPN and from 400 to 600 ms for the LPP. Based on our registration, we measured the P1 and N170 over two symmetrical occipital clusters (left P9, P7, PO7; right P10, P8, PO8), the EPN over temporo-occipitally clusters (left P9, P7, TP7, T7; right P10, P8, TP8, T8) and the LPP over a centroparietal cluster (CP3, CP1, CPz, CP2, CP4, P3, P1, Pz, P2, P4). Analyses of covariance (ANCOVAs) with reaction time as a covariate were calculated to account for possible influences of reaction time differences on ERP modulations. We therefore corrected reaction time effects on ERP amplitudes by entering the respective RT data per condition as a within-subject covariate (as implemented in ezANOVA from the R-package 'ez'; see Lawrence and Lawrence, 2016). Finally, we tested average absolute activation in horizontal and vertical EOG channels, using repeated measures ANOVAs. The pre-registration can be retrieved from the Open Science Framework (https://osf.io/qgwzd).
Here, reaction times were significantly shorter in the perceptual task than in the sex (P < 0.001) and in the emotion task (P = 0.001), the latter two not differing (P = 0.395). In addition, a significant emotion × task interaction effect was found (F (2,78) = 12.71, P = 0.001, partial η 2 = 0.246). Post hoc test showed that compared to the perceptual task, larger emotion effects were found in the sex (P = 0.002) and emotion tasks (P < 0.001), the latter two not differing (P = 0.096). Here, fast responses to fearful expressions were found in the sex task and slower responses in the emotion task (see Figure 1). N170. Regarding the N170, there was a large main effect of emotion (F (1,39) = 90.81, P < 0.001, η P 2 = 0.700; see Figure 2) but no main effect of task (F (2,78) = 0.91, P = 0.405, η P 2 = 0.023). Another significant main effect could be found for the channel group (F (1,39) = 6.75, P = 0.013, η P 2 = 0.148). Regarding these significant main effects, fearful faces elicited larger N170 amplitudes than neutral ones, and larger N170 amplitudes were recorded over the right compared to the left electrode cluster. There was no interaction of emotion and task (F (2,78) = 1.73, P = 0.184, η P 2 = 0.043) and all further interactions remained insignificant as well (Fs < 0.36, Ps > 0.702).

Control analyses
ANCOVAs with reaction time as a covariate. Since reaction time was significantly affected by the task and the interaction of emotion and task, ANCOVAs with the average reaction time for each condition as a covariate were calculated (see Table 1). For these ANCOVAs, all main effects on the N170, EPN and LPP remained significant. Further the EPN and LPP interactions of emotion and task remained significant. We found main effects slightly decreasing and interactions slightly increasing in these ANCOVAs.

Discussion
In this pre-registered study, we investigated how attention tasks differentially affect the emotional modulation of early, midlatency and late ERP components towards fearful vs neutral   Notes: F and P columns refer to the ANOVA without covariates and F * and P * to the ANCOVA with reaction time as a covariate (P-values were Greenhouse-Geissercorrected whenever Mauchly's tests indicated a violation of the sphericity assumption). Significant main and interaction effects are highlighted in bold font. faces. We found emotion effects to be task-independent for the early N170 component. At the level of the EPN, the predicted increase of emotion effects, caused by growing attention to emotionally relevant features, could be validated. Finally, in line with our predictions, LPP emotion effects were only observed when the expression itself was rendered task-relevant. This study shows that feature-based attention does not modulate P1 and N170 effects to fearful vs neutral facial expressions. We predicted that emotion effects might be largest in the perceptual task, as effects are more often reported when faces serve as distracters (e.g. Santos et al., 2008;Li et al., 2018), or under conditions of perceptual load (e.g. Schindler et al., 2019b), and therefore might indicate early inhibitory processes (Klimesch et al., 2007). While we observed largest P1 amplitude differences in the perceptual task (see Figure 2), we found no significant emotion by task interaction effect. This might be due to a lack of statistical power (see Table 1) and adds to the notion that P1 emotion effects are highly variable and small in size (Schindler et al., 2019a; for a review see Schindler and Bublatzky, 2020) (reporting effects, Rellecke et al., 2012;Valdés-Conroy et al., 2014; reporting no effects, Itier and Neath-Tavares, 2017;Acunzo et al., 2019).
For the N170, previous studies comparing different attention conditions suggest either task-insensitive (Rellecke et al., 2012;Itier and Neath-Tavares, 2017) or task-modulated emotion effects (only for females: Valdés-Conroy et al., 2014). In our study, we found emotion effects for fearful compared to neutral expressions, not interacting with the attention task (in line with Rellecke et al., 2012;Itier and Neath-Tavares, 2017). This adds to the meta-analysis showing that fearful faces reliably potentiate N170 amplitudes (Hinojosa et al., 2015). The N170 is regarded as a reflection of structural encoding and high-level face information processing, where face-object differences have frequently been reported (e.g. Rousselet et al., 2008;Eimer, 2011;Rossion and Caharel, 2011;Ganis et al., 2012; but see also Thierry et al., 2007). Configural information appears to be of particular importance here and could be based on holistic (Piepers and Robbins, 2012;Rossion, 2013;Calvo and Beltrán, 2014) or on specific features such as the mouth (Schyns et al., 2007;Harris and Nakayama, 2008;Schyns et al., 2009;daSilva et al., 2016) or the eye region (Schyns et al., 2007;Schyns et al., 2009;Itier et al., 2011;Parkington and Itier, 2018). Research has found that emotional N170 modulations can be influenced by embedded context (e.g. by emotionally congruent or self-referential context; see Diéguez-Risco et al., 2015;Li et al., 2019), where specifically congruence between facial expressions and situational context is found to modulate this processing stage (Diéguez-Risco et al., 2015; but sometimes only in interaction with task and not expression; see Aguado et al., 2019). In contrast to such top-down effects, recent studies of ours showed that emotional modulations of the N170 are immune to a range of image manipulations (Schindler et al., 2019a;Bruchmann et al., 2020) or variations of a task's perceptual load difficulty (Schindler et al., 2019b). As the latter study did not include tasks manipulating the attention drawn to the face or the face's emotion, our current study add to the previous findings of the N170's insensitivity to task demands, suggesting a rather automatic extraction of emotionally relevant features.
The subsequent EPN was modulated by fearful faces, which is in line with a number of studies comparing fearful to neutral expressions (Mühlberger et al., 2009;Walentowska and Wronka, 2012;Wieser et al., 2012;Smith et al., 2013;Schindler et al., 2019a). These EPN modulations are interpreted to reflect a sensitivity to salient emotional information at this processing stage (Junghöfer et al., 2001), in line with the EPN's relation to early attentional selection processes (Schupp et al., 2004;Wieser et al., 2010). Emotion effects were significantly affected by an emotion × task interaction, showing increasing amplitude differences with increasing attention to emotionally relevant features. To explain these effects, we suggest that the EPN might represent a 'bottleneck' of elaborate emotion processing; more precisely, it might reflect a stage at which (task-oriented) attention processes compete with emotional differentiation (e.g. see Schupp et al., 2007;. This would account for the emotion effect building up with stronger attention to emotionally relevant features. In contrast, our findings suggest that during the late processing stages, explicit attention to the expression itself is necessary to elicit differential LPP effects. This attention to the expression has recently been suggested to be crucial (Schindler et al., 2019a for a review see Schindler and Bublatzky, 2020), which is supported by a study showing that LPP effects are strongest during an emotion decision task (Rellecke et al., 2012). This is further in line with the postulation that larger LPP amplitudes are related to stimulus evaluation and controlled attention processes (Schupp et al., 2006;Hajcak et al., 2009), particularly when involving the appraisal of affective meaning (Schupp et al., 2006;Wessing et al., 2013). A possible underlying mechanism which is supported by our study's results and can explain previous inconsistent LPP findings relates to the task requiring participants to deploy different aspects of feature-based attention. For late stages, we reason that top-down and bottom-up processes might interact with biologically relevant (threat-related) expressions benefitting from task relevance. Here, it might be a potentiation of an initial amygdaladependent feedback for fearful expressions with the task relevance inducing top-down signalling from frontoparietal attention networks which synergistically increases threatrelated processing (Pourtois et al., 2013). Such processes are vulnerable to competing tasks (e.g. see Schupp et al., 2007). Of note, we observed reaction time differences between task conditions, but also differently across the tasks between fearful and neutral faces. While we used ANCOVAs correcting ERP effects with respective conditional reaction times, there might be a trial-wise influence on ERP modulations and RTs which cannot be addressed by these analyses (see the limitations section).
Our study provides findings which can be integrated into a comprehensive model of facial emotion perception as a function of attention. Building on recent models of face processing (Haxby and Gobbini, 2011;Schweinberger and Neumann, 2016), specialized systems are suggested for the processing of basic visual facial features and for extended functions such as emotion processing and allocation of attentional resources. Importantly, the interplay of these systems flexibly varies across the visual processing stream, which finds its reflection in subsequent and partly overlapping ERP correlates. At the P1 time window, lowlevel analysis takes place, followed by configural face analyses during the N170 window. For the EPN, early attention processes integrate low-level information and task-relevant features, and at the stage of the LPP, expression differentiation is enhanced by relevance-this also includes evaluative, episodic, personal and biographical information available for the presented faces (see Schweinberger and Neumann, 2016;Schindler and Bublatzky, 2020).
Some further remarks shall be made on the behavioural responses: While we observed a ceiling effect for accuracy values, we found interactions regarding reaction times. No differences between fearful and neutral faces were found when attending to line orientation, but faster responses to fearful faces were made in the sex discrimination task, and slower responses in the emotion task. This fits recent findings highlighting the impact of task focus on reaction time differences and showing that, for example, a differentiation between fear-and angerrelated words occurs only when approach-withdrawal decisions are focused (Huete-Pérez et al., 2019).

Constraints on generality
With regard to our study's findings, there are some constraints which have to be mentioned. Our sample contained mostly female participants, and generalizing our findings to males should be taken with care since previous work has suggested sex differences with respect to the processing of emotional information (for a review, see Kret and De Gelder, 2012). Regarding ERPs, women exhibit larger mid-latency modulations for emotional vocalizations  and larger LPP responses to images conveying interpersonal touch in implicit tasks (Schirmer and McGlone, 2019) and show task-modulated N170 emotion effects (Valdés-Conroy et al., 2014). Furthermore, each emotional expression was repeated 6 times and each identity even 12 times in total. We used a homogenous stimulus to control for visual differences which have been shown to influence differential emotional modulations (displayed teeth, see daSilva et al., 2016; emotion-specific frequencies, see Bruchmann et al., 2020). While studies using pictorial scenes found this number of repetitions not altering differential emotion effects (Olofsson and Polich, 2007;Ferrari et al., 2013; for a recent review, see Ferrari et al., 2017), it is unclear if this also applies to (early) ERPs for emotional expressions. Furthermore, we found that interactions effects remained significant when using ANCOVAs with reaction times. However, we cannot exclude influences of single trials on our ERPs, and our findings need to be replicated with matched difficulty. This requires intense piloting to result in similar reaction times for classifying fearful and neutral expressions across the tasks. Finally, studies examining visual attention rely on eyemovement rejection to avoid condition differences on blinks or saccades (e.g. recommended by Luck, 2014). We pre-registered to use eye-movement correction to obtain a minimum number of trials per cell and examined average horizontal and vertical EOG activity. While we find no statistical differences, we cannot exclude influences of eye-related activity on ERPs. Further studies are needed with higher numbers of trials to use an eyemovement rejection approach and replicate our findings with matched task difficulty, also clarifying the influence of identity repetitions and participants' sex on the current ERP findings.

Conclusion
To summarize, this study shows that early N170 emotion effects are task-independent while EPN and LPP effects depend on the attended feature. These findings are vitally important for researchers who conduct ERP studies using facial expressions as they reveal a systematic pattern of emotional sensitivity varying with competing attention tasks and therefore enable the formulation of clear predictions.