A neural measure of the degree of face familiarity

However, ‘familiarity’ covers a wide range e from people we see every to those we barely know. Although face recognition is studied extensively, little is known about how the degree of familiarity affects neural face processing, despite the critical social importance of this dimension. Here we report the results of a multivariate cross-classiﬁcation EEG experiment, where we study the temporal representational dynamics of the degree of familiarity. Participants viewed highly variable face images of 20 identities. Importantly, we measured face familiarity using subjective familiarity ratings in addition to testing explicit knowledge and reaction times in a face matching task. A machine learning algorithm, trained to discriminate familiar and unfamiliar faces from a separate study, was used to predict the degree of face familiarity from the pattern of the EEG data. We found that the neural representations of the degree of familiarity emerge between 400 - 600 msec post-stimulus onset for famous persons. The correlation between decoding performance and behavioral familiarity was more reliable, occurred earlier and lasted longer when personally familiar and viewers ' own faces were included in the analysis. Our ﬁndings provide new insights into how the brain represents faces with various degrees of familiarity and show that the degree of familiarity can be decoded reliably from the EEG at a relatively late time window. These


Introduction
The recognition and identification of other humans is extremely important in our life. By seeing a person's face, we may identify them and at the same time, typically we also develop a "feeling of familiarity", informing us whether we have encountered the person in the past or not. Accordingly, theories describe recognition memory as consisting of two functionally and anatomically different processes: recollection and familiarity (Rugg & Yonelinas, 2003; of course, it is difficult to separate the two processes entirely but for a discussion of familiarity information in the absence of identity information see Ambrus et al., 2021). These models assume that while recollection is a discrete threshold-retrieval process, familiarity is a continuous measure of a signal detection process, determined by the amount, intensity, and variability of prior exposures.

1.1.
Face familiarity: behavioural, uni-and multivariate electrophysiological effects Several studies have shown large behavioral differences between familiar and unfamiliar faces in the last decades. For example, familiar faces are faster to categorise , easier to match, recognize or identify (Burton et al., 1999;Hancock et al., 2000;Johnston & Edmonds, 2009;Megreya & Burton, 2006;Ramon & Gobbini, 2018;Visconti di Oleggio Castello et al., 2017;Ramon, 2015;Ramon & Van Belle, 2016) and more robust to superficial within-person image changes in memory and matching tasks as compared to unfamiliar faces (Andrews et al., 2015;Jenkins et al., 2011). Accordingly, many studies have tried to identify the neural correlates of this familiarity effect (for a review of the neuroimaging results see Kov acs, 2020). Event-related potential (ERP) studies show that the earliest time window at which familiarity modulates the responses consistently is around 200e300 msec post-stimulus onset, corresponding to the N250 component over the occipito-temporal cortex (Caharel et al., , 2014Huang et al., 2017; for a review see Schweinberger & Neumann, 2016). More recently, a later component has also been identified, emerging between 400 and 600 msec post-stimulus onset and dubbed the "sustained familiarity effect" (SFE; Wiese et al., 2019a,b). The SFE was originally found for personally familiar faces (Wiese et al., 2019b) but was subsequently shown to exist for celebrity faces too, as long as they were known by viewers, and independently of whether viewers liked or disliked these people (Wiese et al., 2021). While not completely clear, the earlier, N250-related component is hypothesized to be purely perceptual while the processing within the later time window is explained by the integration of contextual, mnemonic, and affective components into the representation.
The application of machine learning algorithms to electrophysiological data enables us to decode information in the signal to a level that is not possible using univariate methods (Grootswagers et al., 2017). Thus, multivariate pattern analysis (MVPA) of the EEG/MEG data may give us more detailed information about the content of the differences described above. For example, Dobs et al. (2019), using MEG, observed generic familiarity information for famous faces within a time window closely corresponding to SFE. Karimi-Rouzbahani et al. (2021) found familiarity effects, peaking around 400 msec for one's own and personally familiar (but not for famous) faces. In a recent study we have evaluated the effect of experimental familiarization quality by using perceptual, extensive media and real-life familiarization methods (Ambrus et al., 2021) and found that the representation of familiarity within the same time window was strong after personal, weaker after media and absent after perceptual familiarization. The generic familiarity-related nature of the processing at this relatively late latency is also supported by another study from our laboratory. Dalski et al. (2022) used a novel, cross-experiment and crossparticipant decoding analysis and found that familiarity information is present in the EEG signal from 270 msec to 630 msec, consistent with earlier ERP findings (Barragan-Jason et al., 2015;Besson et al., 2017). The independence of this information from the method of familiarization suggests strongly that the underlying processing steps are related to a general face familiarity processing, a conclusion in line with prior MVPA (Ambrus et al., 2019) and ERP results (Wiese et al., 2021).

Familiarity degree
One issue which remains unclear is how the degree of familiarity is encoded in the brain. There are several factors affecting the degree of familiarity such as the frequency, the intensity, and the variability of prior exposures (Yonelinas, 2002). Unfortunately, studies testing the degree of familiarity in the past were typically confounded by the type of familiarity (Bortolon & Raffard, 2018;Gobbini & Haxby, 2007;Natu & O'Toole, 2011). For example, we know that personally familiar and famous faces differ in their neural processing (for review see Kov acs, 2020; Ramon & Gobbini, 2018). Therefore, previous ERP (Caharel et al., 2002;Wiese et al., 2021) and MVPA (Ambrus et al., 2021;Karimi-Rouzbahani et al., 2021) studies, testing the degree of familiarity and comparing highly personally familiar faces with less familiar faces of famous persons cannot reliably estimate how the degree of familiarity alone would affect neural responses. In addition, none of the previous studies have estimated familiarity in detail: they have either provided the results of subjective familiarity ratings only or simply assumed that personally familiar faces (and one's own face) are more familiar to the participants than famous faces. In the current study we provide a temporal characterization of face familiarity processing, which eliminates both shortcomings.
Our aim was to estimate how the degree of familiarity affects the previously described neural correlates of face familiarity processing for a large range of familiarity levels among a spectrum of famous faces, as well as for personally familiar and own faces. For this purpose, we first estimated familiarity behaviorally with three different measures: We used subjective familiarity ratings, together with the estimation of explicitly recalled information to calculate a behavioral index of familiarity. In addition, we also measured participants' face matching performance as it has been shown to be a sensitive measure of familiarity ( Second, in an EEG study, we provide a temporal characterization of face familiarity processing for the full range of familiarity levels within a given familiarity type (famous faces) with highly variable, "ambient" face stimuli (Jenkins et al., 2011). Finally, we also measured the EEG for personally familiar faces and one's own face, to compare the decoded information across different types of familiarity.

2.
Study 1: stimulus selection First, we identified a stimulus set reflecting a very broad range of familiarity for our participants. Due to the restrictions of the COVID-pandemic, we designed an online study where participants were instructed to record their subjective familiarity and the recalled explicit information of the stimuli.

Methods
We report how we determined our sample size, all data exclusions (if any), all inclusion/exclusion criteria, whether inclusion/exclusion criteria were established prior to data analysis, all manipulations, and all measures in the study.

Participants
Thirty-five participants (30 females, all right-handed) with an average age of 21.7 years (SD ¼ 2.7) took part in the online stimulus selection phase. Participants gave informed consent and received partial course credit or monetary compensation. This sample size was based on our previous studies, using MVPA of EEG data, leading to reliable familiarity decoding in the EEG signal (Ambrus et al., 2019;Dalski et al., 2022).
The study was conducted in accordance with the guidelines of the Declaration of Helsinki and was approved by the ethics committee of the Friedrich-Schiller-Universit€ at Jena.

Stimuli and procedure
Ambient, naturally variable faces of fifty-five persons were selected for this study, including nationally or internationally famous celebrities such as athletes, politicians, actors, and singers of both sexes, with a broad range of age (See Appendix 1 for a detailed of information of these celebrities). All images were collected using the Google image search engine. Participants were presented with the faces of these celebrities on the screen centrally, until they signaled their familiarity and answered the following questions [PsyToolkit (Stoet, 2010(Stoet, ,2017]. First, they were asked to indicate the level of familiarity with the faces on a Likert-scale (ranging from 0 to 9, 0: "I don't know them at all"; 9: "I know them very well"). Second, they were asked to enter the full name and occupation of the persons. Fig. 1a illustrates an example of the screen where the correct answer could be: "Angela Merkel", "Kanzlerin". Participants received 1 point for each correct answer. In addition, we also asked participants to list any biographical facts of the given persons (for example a potential answer could have been: "She comes from the former GDR" and "She is a physicist by profession") as well as any personally related episodes in relation with these persons which they could recall (for example "I saw her once in Berlin"). For the recalled biographical facts, we gave a maximum of 2 points and for recalled personal episodes, we gave an additional 1 point (maximum reachable points ¼ 5). After completing this survey participants were presented the next image in a random order. The study took 42 min on average.

Analysis
We created a "mnemonic familiarity index" (MFI) from the results of the familiarity rating and from the answers of the explicitly recalled memories the following way:  where LR is the Likert Rating score and MS is the total memory score of the participants obtained for the four declarative memory questions. The answers to the questions were evaluated by two native German scorers, independently. To ensure the reliability of the MS, we calculated the correspondence of the ratings of the two scorers. Spearman's correlation (R ¼ .997, p < .001) suggest that the evaluation of the MFI score was highly consistent between the two scorers. Altogether, the composed MFI signals familiarity from 0 to 100 and reflects both components of recognition memory (the subjective feeling of familiarity and the amount of explicitly recalled information about the persons) with an equal weight. We calculated MFI for each face identity and participant separately and averaged them across participants.

Results
Fig. 1b shows the average (±SE) MFI for the 55 tested celebrities. As the figure shows, MFI ranges from under 10 up to nearly 90, reflecting a very broad range of familiarity degrees across the stimulus set. Next, we selected 14 out of these 55 identities ( Fig. 1b, red dots) for the subsequent EEG experiment, covering as much of the familiarity spectrum as possible.

Participants
A new sample of 26 participants, who did not take part in Study 1, was recruited for the second, electrophysiological experiment. One participant was excluded from the analysis, due to a failure of the EEG recording. The remaining 25 participants (19 females; mean age ¼ 22.1 ¼ years; SD ¼ 3.4 years) were right-handed and had normal or corrected-to-normal vision. They had no history of any neurological disorders and were informed about the experimental tasks and gave their written informed consent. This sample size was based on our previous studies, using MVPA of EEG data, leading to reliable familiarity decoding in the EEG signal (Ambrus et al., 2019;Dalski et al., 2022). The experiment was conducted in accordance with the guidelines of the Declaration of Helsinki and was approved by the ethics committee of the Friedrich-Schiller-Universit€ at Jena.

Stimuli
The 14 celebrities selected based on the MFIs from Study 1 served as stimuli (7 females). In addition, prior to the EEG sessions participants provided pictures of themselves (own face) and pictures of three personally familiar persons. The personally familiar persons were either family members and relatives or close friends of the participants, reported as most familiar. In addition, two local celebrities from Hungary, unknown to our participants, were added to the stimulus set as unfamiliar identities. All images were ambient faces, presented in color, reflecting a large range of facial expressions and lighting conditions. The images were cropped and resized to 2.8 Â 3.9 deg (viewing distance: 108 cm), using GIMP 2.8.6 (for examples see Fig. 2). Thus, altogether a total of 20 identities served as stimulus material for the EEG experiment, including participants' own face (ID1), three personally familiar faces (ID2, ID3, ID4), faces with high MFI (above 60; ID5, ID6, ID7, ID8), moderate MFI (between 40 and 60; ID9, ID10, ID11, ID12), low familiarity (MFI between 10 and 40; ID13, ID14, ID15, ID16) as well as entirely unfamiliar faces (MFI below 10; ID17, ID18, ID19, ID20). For each identity we collected sixteen face images. For the EEG recording session we used 10 of these images (Ambrus et al., 2019) while for the face matching-task five previously unseen images were presented, per identity. The remaining, previously unseen image of the identities was used for the final familiarity evaluation phase of the study. Additionally, 20 unfamiliar faces (10 female; similar age and hair color as the target faces) were selected as "foil" images for the face matching task (Ambrus et al., 2017).

Procedure
The experiment included three phases: an EEG recording session, a subsequent face matching task and a final familiarity evaluation phase (Fig. 2).
3.1.3.1. EEG RECORDING SESSION. The EEG recording session was similar to that of Ambrus et al. (2019Ambrus et al. ( , 2021. A total of 1760 (1600 nontarget and 160 target) trials were presented in 8 runs, separated by self-spaced breaks. Each run included one presentation of the 10 images of the 20 identities. Additionally, 20 target trials were added to each run in a pseudorandom order where the image was identical to the previous one.
In each trial (Fig. 2b), a central fixation cross was presented for 250 msec, followed by the face stimulus for 600 msec and an Inter-Trial-Interval (ITI), selected randomly between 700 and 1000 msec. Participants were asked to press the space button when they saw a target image (1-back task; mean detection accuracy: 99.08 ± .67%). These target trials were set to ensure that participants maintained their attention and were excluded from the analysis. PsychoPy (Version 3.0) was used for stimulus presentation and behavioral response collection (Peirce, 2009). Stimuli were presented centrally on a uniform gray background (23.0-inch EIZO display, 1920 Â 1080 pixel resolution, refresh rate 60 Hz).
3.1.3.2. FACE MATCHING TASK. Previous studies found that face matching paradigms are a sensitive measure of face familiarity (Burton et al., 2010;Kramer et al., 2018), where performance correlates with the degree of familiarity well (Clutterbuck & Johnston, 2002, 2004, 2005. Thus, we conducted a face matching task after the EEG experiment where participants made same-different decisions about pairs of previously unseen images. Participants completed 800 trials, allocated into 4 blocks of 200 trials. For each identity 40 trials were presented. Each trial started with a central fixation cross for 250 msec, followed by a pair of face images for 1000 msec (Fig. 2c). The face pairs consisted of either two different images of a given identity ('same') or an unseen image of a previously seen identity and an image of a "foil" identity ('different' condition), with equal probability. Next, a response screen was presented until participants signaled their answer c o r t e x 1 5 5 ( 2 0 2 2 ) 1 e1 2 by a button-press, followed by an ITI of 700e1000 msec (see Fig. 2C). The participant's task was to judge if the pair of faces belonged to the same identity or not. The keys F and J were assigned to "same" and "different" responses, counterbalanced across participants. The experimental software was written in PsychoPy (Peirce, 2009). 3.1.3.3. FAMILIARITY EVALUATION. The experiment was concluded by a familiarity evaluation test session, identical to that of Study 1, with the exception that it was performed in the laboratory. Briefly, participants had to estimate the subjective familiarity of each identity on a 10-point Likert scale and answer the four explicit memory questions (for details see Fig. 1a). Finally, we calculated the MFI index of each participant and each identity separately, using the methods described at Study 1.

EEG recording and preprocessing
Participants were tested in a dimly lit, electrically shielded and sound-attenuated cabin with 108 cm between the screen and the eyes, secured via a chin rest. The EEG recording was performed continuously, using a 64-channel BioSemi Active II system (BioSemi, Amsterdam, The Netherlands) with a 512 Hz sample rate (band with: DC to 120 Hz). Electrooculogram (EOG) was recorded by four additional electrodes, placed over the outer canthi of both eyes, and above and below the left eye.
EEG preprocessing was identical to that of Ambrus et al. (2021). The preprocessing pipeline was implemented in MNE-python (Gramfort et al., 2013(Gramfort et al., , 2014. EEG was notchfiltered at 50 Hz, band-pass filtered between .1 and 40 Hz, segmented from À200 to 1000 msec relative to stimulus onset, and baseline corrected with respect to the first 200 msec. The resulting data was downsampled to 100 Hz to increase signalto-noise ratio in the multivariate analyses (Grootswagers et al., 2017).

Decoding analysis
Multivariate pattern analysis (MVPA) was performed using a cross-experiment, cross-participant classification approach (Fig. 3). This approach has recently been shown to demonstrate a robust and reliable, general neural signature of face familiarity signal that is independent of participants, stimulus identities or the type of familiarization (Dalski et al., 2022). During this procedure, ERP data from all participants of an experiment is concatenated to serve as the training set on which classifiers are fitted at each time-point. In the current study we used the publicly available dataset of Wiese et al. (2022), studying personally familiar and unfamiliar faces, as training dataset. These classifiers are then used to predict class membership in a dataset of a different experiment, for each participant separately. The benefit of this method over the typical within-experiment, within-participant MVPA is an increased and diverse training dataset and the reduction of confounding effects that are due to idiosyncratic, participant-level effects or uncontrolled stimulus properties.
Linear discriminant analysis [LDA; scikit-learn (Pedregosa et al., 2011)] classifiers were trained on then publicly available data of the Experiment 1 of Wiese et al. (2022 (Incidental Recognition condition; https://osf.io/7xtdy/)) to categorize two personally familiar and two unfamiliar identities (50 images each). Importantly, each of the 22 participants in this training dataset was exposed to a unique set of stimuli, and each presented image was trial-unique. Fig. 2 e A, The schematic paradigm of study. B, Schematic illustration of the trial structure of the EEG recording session. C, Schematic illustration of the trial structure of the face matching task ("same" trial). The familiarity evaluation phase was identical to that of Study 1 (Fig. 1a). c o r t e x 1 5 5 ( 2 0 2 2 ) 1 e1 2 Next, the LDA classifiers were used to assess prediction accuracy for the familiarity of the data of the current experiment, using the response patterns of the 62 electrodes, common to both EEG recording systems, separately for each participant. The EEG pre-processing pipeline of the training data was identical to that of our own dataset (see above). In constructing the training dataset, trials were sub-sampled to include an equal number of familiar and unfamiliar trials for each participant (on average 90 trials per participant). Next, all participants' data was concatenated and the LDA classifiers were trained at each of the 120 time points (À200 to 1000 msec in 10 msec steps) to classify familiar and unfamiliar trials.
In the classification procedure, data from each participant of our EEG recordings was tested separately. To further increase sensitivity, single ERPs, elicited by the 8 repetitions of the same images were averaged (Grootswagers et al., 2017), resulting in 10 to-be-classified signals per stimulus identity (200 signals in total). For each of these signals, at each timepoint, decoding performance, i.e., the ratio of 'familiar' classifications, made by the corresponding classifier, was recorded, and averaged for the given identity.
Finally, for each participant and time-point the decoding accuracy and the MFI were correlated using Spearman rank correlations. The same procedure was repeated with reaction times recorded during the face matching session. This led to a time-series of correlations that reflect the correspondence of the familiarity-specific neural signals and the behavioral measures of the degree of familiarity. Note that personally familiar faces and the own faces were automatically assumed to be more familiar than the faces of the celebrities, as such, to have an MFI above the maximum (100) attainable for the famous identities. We further assumed that the own face will be more familiar than personally familiar faces. Thus, these identities were assigned, arbitrarily, the MFI values of 105 for personally familiar identities and 110 for the own faces. Please note, that the exact values do not affect the results as rank correlations were calculated.
The Fisher-transformed Spearman rank-correlation values between classification performance and MFI, and classification performance and matching task reaction times, were then tested for statistical significance using two-tailed, one sample cluster permutation tests (against 0, i.e., no correlation) with 10,000 iterations [implemented in MNE-Python (Gramfort et al., 2014)]. As several previous studies suggest the important role of personal familiarization and the differential processing of famous and personally familiar persons (Campbell & Tanaka, 2021;Ramon & Gobbini, 2018;Wo zniak et al., 2018), this procedure was performed twice, once with and once without the signals obtained for the own face and the personally familiar identities in the test set.

Data availability
The database from Wiese et al. (2022) is already publicly available (as noted in section 3.1.5). We have uploaded all of the applied codes to OSF (https://osf.io/2czu5/). We have also uploaded the experimental stimuli, with the exception of personal photos of the participants and their friends. The conditions of our ethics approval do not permit public archiving of these images and of study data. The entire data and stimulus sets will be made available to interested researchers following completion of a data sharing agreement and approval by the local ethics committee. No part of the study procedures or analyses was pre-registered prior to the research being conducted.  2022) was merged into a single dataset (a). At each time point, a classifier was trained to differentiate between personally familiar and unfamiliar trials (b). These classifiers were used to predict familiarity in the dataset collected for the present report. The classifiers were used to classify each stimulus identity, at each time-point, as familiar or unfamiliar (c). This resulted in a time-series of the ratio of familiar/unfamiliar classifications (d), which was then rank-correlated with the participant-level familiarity metrics (MFI, matching RTs). Finally, the participant-level correlation time-series were Fisher-transformed and averaged across the sample (e).

Behavioral results
3.2.1.1. MFI. First, we made sure that the 20 identities (16 celebrities, three personally familiar faces and the own face; for details see Methods section) do reflect different degrees of familiarities. Fig. 4a shows the average MFI, collected after the EEG recording session. The results confirmed that the selected identities range from being maximally familiar (own face and personal familiarity faces) to being little or not at all familiar, with most of the celebrity faces being situated in between these values. This suggests that the stimuli are suitable for measuring the neural correlates of the degree of familiarity over a broad range.

FACE MATCHING. Previous studies have found that face
matching performance is better for familiar as compared to unfamiliar faces (Andrews et al., 2015;Burton et al., 2010;Clutterbuck & Johnston, 2002, 2004, 2005. The results of the face matching test, conducted after the EEG recording session, confirmed that the stimuli reflect a wide range of familiarities, required for our purposes. Fig. 4b  identities which were judged more familiar and had higher MFIs. Moreover, while discrimination performance was relatively high for unfamiliar faces as well, reaction times were very sensitive to the degree of familiarity. This is also shown by the strong and significant correlation of the reaction time values with the MFIs (Fig. 4d). Therefore, in the subsequent EEG analysis we used the reaction times of the face matching session to correlate with the electrophysiological data. Our stimulus set contained both personally familiar and famous faces. To determine whether the above-described temporal dynamics of familiarity processing is different for these two familiarity types we also correlated the neural data with the MFI by excluding trials where personally familiar or own face stimuli were presented (Fig. 5a orange line). While the onset was much later (400 msec) and the time-period was also shorter (until 670 msec) in this analysis as compared to the entire dataset, significant familiarity information was still present in the signal after the removal of personally familiar faces [cluster p ¼ .0002, peak at 550 msec, peak t(24) ¼ 5.29, peak Cohen's d ¼ 1.06]. This suggests that the earlier components of familiarity encoding, between 200 msec and 400 msec depend on familiarity type, a conclusion in line with our prior results (Ambrus et al., 2021). At the same time, the peak of the available information was similar in both cases, corresponding to the 400e600 msec time window where previous ERP (Wiese et al., 2019b) and MVPA (Ambrus et al., 2021) studies also reported differential encoding of familiar and unfamiliar faces.
To estimate further if the 400e600 msec time window reflects the degree of familiarity in a quantitative manner we plotted the correlations of familiarity decoding performance, averaged over this time window, with the MFI for each presented identity separately (Fig. 5b). The correlation was significant with (Rho ¼ .866, p < .00001), and without personally familiar faces (Rho ¼ .74, p ¼ .001). This emphasizes further the important role of this time-period in the representation of the degree of familiarity.

CORRELATION OF EEG-DATA BASED FAMILIARITY DECODING WITH
FACE MATCHING REACTION TIMES. To confirm the previous results, we performed the same analysis with the reaction times in the face matching task. Fig. 6a shows a significant correlation of the EEG decoding and the reaction times with an identical time window and peak as the correlations with the MFI [significant cluster from 200 msec to the end of the epoch, peaking at around 450 msec; cluster p < .0001, peak t(24) ¼ À13.01, peak Cohen's d ¼ À2.60].
To test the role of the 400e600 msec time window further in a quantitative manner we plotted the correlations of familiarity decoding performance of this time window with the reaction times for each presented identity separately (Fig. 6b). The correlations of decoding performance within the 400e600 msec time window with the reaction times was also significant with (Rho ¼ .785, p < .00001) as well without personally familiar faces (Rho ¼ .62, p ¼ .01).
Overall, both the MFI and reaction time correlations show the importance of the 400e600 msec post-stimulus time window in the representation of the degree of facial familiarity, both for personal and famous faces.

Discussion
In the current study, we applied MVPA to the EEG signal to investigate the neural dynamics of face familiarity processing.
Our results show that the degree of face familiarity can be recovered very well from the EEG response patterns between 400 and 600 msec. However, the decoding performance and its correlations with behavioral familiarity measures depend strongly on the type of familiarity: familiarity information is more reliable, occurs earlier and lasts longer when one includes personally familiar and viewers' own faces as well as celebrities.

4.1.
The degree of face familiarity is reflected in the 400e650 msec time-window A machine learning algorithm was first trained to discriminate personally familiar and unfamiliar faces from a previous EEG study  and was tested to establish whether it could identify the degree of familiarity in a set of highly variable famous faces from our EEG signal. We showed that the neural representations of the degree of familiarity emerge between 400 -650 msec post-stimulus onset. This timing is consistent with previous ERP studies that reported differences between familiar and unfamiliar faces in averaged waveforms after 400 msec for personally familiar (Wiese et al., 2019b) and well-known famous faces (Wiese et al., 2021). Previous MVPA studies could also identify familiarity representations within a similar time window: Dobs et al. (2019) found separate representations for famous and unfamiliar faces while Ambrus et al. (2021) showed that experimental familiarization by media exposure and by personal meetings leads to a robust familiarity representation around these times.
In addition to confirming these prior findings we have also shown that this relatively late time-period is sensitive to the degree of familiarity. Prior studies, measuring ERP (Caharel et al., 2002;Wiese et al., 2021) or MVPA correlates of the degree of familiarity (Ambrus et al., 2021;Karimi-Rouzbahani et al., 2021), have always compared familiarity levels across several different types of familiarities, such as the participants' own face, personally familiar and famous faces. Thus, these studies were not ideal to separate the effect of the degree and the quality of the familiarity from each other. To the best of our knowledge this is the first study seeking the correlates of the degree of familiarity in the electrophysiological signal within a given type of familiarity, famous faces. The only comparable univariate study, published during the preparation of the current manuscript by Popova and Wiese (2022), compared ERPs for the faces of more or less familiar friends. They found that the longer the friendship was, the larger the magnitude of the difference of their ERPs within the 400e600 msec time window. However, this study showed no correlation with the subjective familiarity ratings of the faces, making the interpretation of the results difficult in regard of the degree of familiarity. Thus, our study confirms the results of Popova and Wiese (2022) and extends it towards famous faces, but it also emphasizes the necessity of detailed measurement of the degree of familiarity, preferably including several different components.

4.2.
The location of the familiarity degree effect The origin of 'degree of familiarity' representations remains open. Such gradually emerging representations of face familiarity may either originate from the initial perceptual stages of face processing or from post-perceptual processes (such as context, emotions, semantic information, and person-related traits) that contribute to the superior recognition memory, elicited by familiar faces. The relatively late onset and peak of the representation makes it highly likely that it is anchored to the interface of visual perception and recognition memory. Indeed, Ramon et al. (2015) reported that (personally) familiar face recognition emerges categorically in medial temporal and anterior regions of the extended cortical face network while posterior face regions accumulated evidences linearly. A replication of the above study in nonhuman primates (Landi & Freiwald, 2017) located these familiarity specific categorical responses into the perirhinal cortex and to the temporal pole, areas which are situated ideally for encoding the degree of familiarity and which are proposed as parts of the extended face network recently (Kov acs, 2020). Recognition memory is typically considered as having two functionally and anatomically different parts: recollection and familiarity (Renoult et al., 2019;Rugg & Yonelinas, 2003;Yonelinas, 2002). The fact that the representation does not merely reflect familiarity in a binary fashion, but it correlates significantly with the degree of familiarity shows that it is related to the later process, which is a gradual and continuous measure. Thus, the current results, although only indirectly, also support the dissociation between familiarity and recollection, a conclusion in line with our previous MVPA results (Ambrus et al., 2021).

4.3.
The role of personally familiarity Interestingly, while the peak of familiarity-level representation was independent of the type of familiarity, its onset, length and magnitude were positively modulated by the inclusion of personally familiar faces in the data. One simple interpretation of this dependence is that our training stimulus set (the data of the Experiment 1 of Wiese et al., 2022) included two highly personally familiar and two unfamiliar faces. This training set was selected for the purposes of the current study for the following reasons. We wished to study the temporal dynamics of the representation, related to the degree of familiarity as independent of stimuli, paradigms, set-ups, experimental procedures, and participants as possible. Indeed, cross-classification MVPA methods are well-suited for this task (Kaplan et al., 2015). In our case not only the abovementioned factors, but also the quality of familiarity was different for the training and testing datasets. Thus, any conclusion, drawn from the decoding performance of the MVPA should reflect a general, shared information processing across all these factors (Dalski et al., 2022). Nonetheless, it is possible that an LDA trained on personally familiar versus unfamiliar faces picks up the idiosyncratic information in the signal that is related to personal familiarity better than that of other familiarity types. This could explain the differences of the representation with and without personally familiar faces. However, this also suggests that the type of familiarity has a strong effect on the representation, as the inclusion of personally familiar faces led to an early (200e400 msec) and to a very late (from 650 msec onwards) component of the representation that was not present without the personally familiar faces. This conclusion is in line with our previous media versus personal familiarity cross-experiment decoding results (Dalski et al., 2022) where decoding of familiarity was possible from 190 to 1010 msec. The results are also in line with the conclusion of a recent univariate ERP study (Wiese et al., 2021) which suggested that familiarity for faces known personally or via media-sources is not represented qualitatively differently, at least from 400 msec post-stimulus onset onwards. The authors also concluded that the early familiarity effects (corresponding to the N250 ERP component) are more variable and presumably influenced by factors other than familiarity. We argue that the more sensitive multivariate decoding analysis of the data refines this picture and suggests that the representation of personally familiar faces might already start at around 200 msec. Whether this early component is related to the high degree or to the type of familiarity is difficult to decipher, as personally familiar faces, as used in these experiments, are at the same time also the most familiar ones. Interestingly, the very recent experiments of Popova and Wiese (2022) show an effect of increasing personal familiarity across years for the N250 time window as well as for the later SFE ERP components. This would suggest that it is the degree rather than the type of familiarity which determines the early familiarity representation as well. An explicit test of this hypothesis still remains to be done, by testing the effect across a broader range of familiarities, for example by comparing the highly familiar faces of family-members and close friends with that of less familiar persons, such as the barista of your favorite caf e or the concierge of your office-building.

Cross-participant classification
One important implication of the current results is that they clearly show the superior sensitivity of cross-classification methods over univariate and within-participant multivariate methods. Dalski et al. (2022) showed that such crossexperiment, cross-participant, and cross-stimulus decoding analyses are useful in identifying the general signatures of familiarity processing. Here we applied cross-classifications by training and testing a machine learning algorithm on the data of two independent laboratories and we extend these findings towards the degree of familiarity as well.

Reaction times in face matching signal familiarity reliably
The correlation between the neural data and the degree of familiarity measured by MFI and by the reaction times of the face matching task revealed identical temporal dynamics. On the one hand, this validates the current results -as despite two entirely independent behavioral measures being used, the obtained correlations are essentially identical. On the other hand, this also emphasizes the importance of face matching tasks in estimating the degree of face familiarity. Face matching performance has been shown to be a sensitive and objective measure of familiarity in the past (Ambrus et al., 2017;Andrews et al., 2015;Burton et al., 2010;Clutterbuck & Johnston, 2002, 2004, 2005Kramer et al., 2018). Our results show in a quantitative manner, that the reaction times in a face matching task signal the degree of familiarity at least as well as the index which was composed from subjective familiarity ratings and the amount of explicitly recalled memories.

The concepts of familiarity and expectation
To what degree might expectation contribute to these results? One can argue that there are strong similarities between familiarity and expectation as both require previous experiences. In fact we only can expect the occurrence of a given object if we were previously familiarized with it. However, recent MEG studies found that familiarity and expectation affect the neural responses jointly, but independently of each other (Manahova et al., 2018). In the current experiment, expectation was kept constant across trials, therefore the temporal dynamics of familiarity degree is unlikely to depend on the differential expectation of the stimuli. Nonetheless, the study of the relationship of expectation and the degree of familiarity requires future specifically aimed studies.
In conclusion, our findings, obtained via the crossexperiment decoding analysis of the electrophysiological signal, provide new insights into how the brain represents faces with various degrees of familiarity. Our data shows that the degree of familiarity can be decoded reliably from the EEG at a relatively late time window, corresponding to the temporal dynamics of prior familiar identity encoding results. This result supports further the idea that this component is a general neural signature of face familiarity per se.