The Role of Multilingual Script Systems in Face Processing

Becoming multilingual has a broad impact on cognitive abilities, especially visual processing. An important theoretical issue is whether the acquisition of distinct script systems affects face processing in an identical way, or, if not, how this acquisition may exert differential impacts on face processing. By reviewing the existing literature, we propose that Asian participants with the logographic script system differ from Western counterparts with the alphabetic script system in viewing faces. The contribution of the chapter is to identify the possible role of types of script systems in face processing mechanisms and to put forward the research direction in the future with several new methodological efforts.


Introduction
As globalization progresses worldwide, more and more individuals have become bilinguals or even multilingual. Multilingual differ from monolinguals in at least two aspects. First, multilingual usually have a larger vocabulary size compared with monolingual because these multilingual need to use words from different languages to express the same concept. Second, multilingual may have to deal with differences in many linguistic aspects of different languages, such as word-to-sound mapping, phonemes and the number of letters/letters, and orthography/word forms [1]. Such differences affect cognitive abilities (visual working memory [2], attentional control [3]), and efficient communication [4,5] of multilingual. However, little attention was paid to the question of how acquiring different script systems, for example, with different phonological transparency of orthography and different visual configurations, impacts the visual perception of words, and even non-words (such as faces).
Before discussing the relationship of the acquisition of multiple script systems with face processing, we first reviewed the link of the script system to face processing in a single language. One important aspect of literacy acquisition is to use script systems to write and read in daily life. An interesting and fundamental issue underlying literacy acquisition is how our brain deals with faces. Why does literacy acquisition (in particular, what script system is learned to read) affect face processing? One dominant view, i.e., the neural recycling hypothesis [6], has been proposed. Since the script system is not fully formed until 5000 ~ 6000 years ago due to a recent cultural invention, unlike faces, it is possible that our brain does not evolve in time to develop a specific cortical territory dedicated to processing words, relative to faces. This invention of the script system inevitably invades the pre-existing brain cortex, originally acted as other functions (such as recognizing faces and other objects), and re-organizes these brain structures to adapt themselves to word processing. That is, the process of acquiring a new script system may share some neural resources with recognizing faces. This opinion is called the neural recycling hypothesis [6], which assumes that: first, the anatomy of connection constrained strongly by the evolutionary pressure determines our brain organization, which, in turn, guides our subsequent learning. Second, learning to read must find suitable neural substrates, which are a set of circuits close enough in their function and revealing enough plasticity, in order to recycle a large part of the circuits for this new function. Third, although these other-serving cortical territories are (partially) occupied by literacy, their prior organization is never completely erased. Thus, prior neural constraints have a powerful impact on the acquisition of cultural invention and individual brain organization. Based on this recycling process, it is reasoned that literacy acquisition most likely has much to do with face processing.
This chapter is organized as follows. We first briefly introduced empirical evidence with respect to the impact of literacy acquisition on linguistic and non-linguistic (especially face) processing. Then, experimental evidence on cross-linguistic and cross-cultural comparisons provided insights into the role of script systems in processing faces, which echoes controversies about two kinds of theoretical hypotheses, i.e., language specificity and cross-language universality. Moreover, multilingual differences in face processing could be accounted for by different possibilities, such as the perceptual expertise hypothesis and attention-reshaped-by-language hypothesis. Finally, further studies are encouraged to increase the weight of the script system in explaining face processing (or reduce the weight of social and cultural interpretation) and to distinguish the visual form of the script system and the role of speech in face processing.

Literacy effect on the processing of linguistic and non-linguistic materials
According to the neural recycling hypothesis, literacy acquisition reshapes the processing of linguistic and non-linguistic materials (e.g., faces). Without a doubt literacy acquisition impacts the linguistic materials owing to that the brain structures of faces and objects are engaged in the representation of words.

Literacy effect on the processing of linguistic materials
Literacy acquisition is a milestone for human civilization. It is well-documented that literacy acquisition modulates our ability to deal with linguistic information [7], such as word repetition, speech segmentation, and character identification. For example, using an auditory-verbal repetition paradigm, Petersson et al. revealed that literates performed better than their illiterate counterparts in the pseudo-word repetition task [8]. Also, Morais et al. showed that ex-illiterates, who attended classes of elementary instruction during adulthood, were superior to the illiterate in multiple speech segmentation tasks [9]. Besides, Duñabeitia et al. found that, in two perceptual matching tasks, literates showed great sensitivity to changes in the letter position (i.e., transposed-characters) and identity (i.e., replaced-characters) of a character, whereas illiterates were less sensitive to these changes [10]. These findings suggested that literacy acquisition has a profound influence on processing different linguistic aspects, and the supportive evidence mostly is obtained in the monolingual context.

The effect of literacy on the processing of non-linguistic face materials
Many studies have demonstrated that literacy acquisition modulates the ability to process non-linguistic objects [11], especially faces. We here review a tremendous amount of work on children, adults, and special populations.
One way of understanding the associations between literacy acquisition and face processing is to track the development of face processing ability with the size of literacy or to the relations between lexical and face processing in children. Evidence from the functional magnetic resonance imaging (fMRI), a high-spatial-resolution technique, has suggested that, during literacy acquisition in children, there may be coordination between word and face processing in the left fusiform gyrus (FFA), also termed visual word form area (VWFA) [12,13]. For instance, neural responses to faces in VWFA declined gradually with the increase in children's letter knowledge [14]. Consistently, event-related potentials (ERPs) studies have identified a stable electrophysiological hallmark, N170 response, of viewing words, which was elicited at electrodes over the left occipitotemporal areas (roughly corresponding to the VWFA). The N170 peaks about 150-200 ms after word onset, and may index the visual processing of words at an expert level. Using the color-matching task, Li et al. found that both the left-lateralization indexed by the N170 of words and the vocabulary was associated positively with the right-lateralization of faces in Chinese preschool children [15]. Similarly, employing the "half-field" paradigm, Dundas et al. showed that the emergence of face lateralization was related positively to reading competence with the control of age, reasoning scores, and face discrimination accuracy [16]. Furthermore, they also reported that the N170 evoked by faces in the right hemisphere was positively related to that by words in the left hemisphere in American children [17]. In a word, there is a tight link between visual face processing and word lateralization or between this processing and the size of literacy.
Also, empirical work from normal adults' study provided stronger evidence of the relationships between word and face processing. For example, using the adaptation paradigm, in which the first adaptor face (or word) was followed subsequently by a target face (or word) of either the same or different identity, Cao et al. found that the adaptor face led to the reduced N170 response to the target word, while the adaptor word did not result in the decreased N170 response to the target face [18]. Neural adaptation occurred because the neural response to the test stimulus was reduced when the stimulus was preceded by a physically identical or categorically identical adaptor stimulus. Therefore, the results of this study may indicate that the facial N170 function partially encompasses the N170 function of word processing, which is consistent with the neural recycling hypothesis. In another study, subjects were asked to view artificial objects (i.e., face-like Greebles) centrally presented with the concurrent lateral presentation of faces and then to judge which side each face was presented. Results showed that the N170 response to faces tremendously decreased after subjects were trained to recognize Greebles compared to before those Greeble novice [19]. Analogously, the bilateral N170 response to faces decreased in identifiable Chinese characters and faces as compared to unidentifiable conditions [20]. In Robinson et al.'s study with an attentional blink paradigm, they found that word (target 2) recognition performance was inferior at short inter-target lags when the word stimulus was preceded by faces compared to glasses and words condition. No effects were observed when words were followed by other objects. Furthermore, ERP results indicated that N170 responses to face (target 1) were associated with the reduction of N170 to words within the face-word condition in the left hemisphere, but not for other object-word conditions [21]. These findings indicated that face and word shared some overlapping neural resources, possibly associated with the same specialized processing module.
In contrast with these studies mentioned above, the special population, such as dyslexia, alexia, prosopagnosia, illiterates, and so on, can provide stronger evidence for the relationship between words and face processing. For example, dyslexic readers performed poorer on recognition of both word and face and even decreased level of hemispheric lateralization to words and faces compared with normal readers [22]. Face recognition was impaired severely with bilateral compared to unilateral temporo-occipital cortex lesion [23], and a left occipital impairment gave rise to both pure alexia and prosopagnosia [24]. Also, prosopagnosic patients showed mild but reliable words recognition deficits, and pure alexic patients showed face recognition deficits [25]. Another important avenue is to examine the differences in face processing between literates and illiterates. The ideal illiterate and literate groups differ only in whether the script system could be used to read and write by them, so they are the ideal group for researchers to understand the relationship between face and word processing. For example, Deheane et al. found that reduced responses to faces in VWFA were observed for literates compared to illiterate adults [26]. And ERP results showed that the literacy effects were observed not only in the letter strings but also in faces, revealing the impact of literacy on common early visual processing [27]. These findings have led to a proposal that face and word processing engage some overlapping neural substrates, and there are interactions between the development of visual representations for faces and words [28]. However, these studies did not weigh the potential contribution of the script system to face processing because they used the script system of one language in a particular social culture.

The differences in face processing between Western and eastern participants
Although the neural recycling hypothesis points out the possible relationships between word and face processing in VWFA, it does not postulate whether different script systems affect face processing in the same way, and if not, how the script system influences face processing. There are two possible theoretical hypotheses, the language specificity hypothesis, and cross-language universality hypothesis, to explain the impact of literacy on face processing.
The first hypothesis holds that neural computations are functionally independent underlying languages of different script systems. Evidence for the language specificity hypothesis comes from Siok et al.'s study [29], showing that the left medial frontal gyrus is crucial and unique to normal Chinese reading, and its dysfunction is only linked with reading difficulty in Chinese, but not other languages. Similarly, prior studies have consistently found declined activation in left temporoparietal regions, which is a biological signature of English reading difficulty in a homophone judgment task [30,31]. Moreover, Xu et al., using a passive reading task, found that distinct activity patterns in the middle occipital cortices, fusiform gyri, and lateral temporal, temporoparietal, and prefrontal cortices were observed between Chinese and English [32]. In contrast, the second hypothesis deems that VWFA consistently and equally responds to words of different script systems. For instance, Feng et al. measured the fMRI responses to words, faces, and houses among Chinese and French 10-year-old children, half of them with reading difficulties. The results did not reveal any effects of language on the peak locations and activations in the bilateral FFA [33].
These findings mainly focused on whether there are differences in neural substrates underlying different script systems. It remains unclear whether and how these different script systems affect face processing. However, behavioral, eye-tracking, and neural evidence of face processing from cross-cultural studies provided insights into this issue.

Behavioral evidence
Holistic face processing, a typical hallmark of perceptual expertise for faces, refers to that participants tend to deal with face parts as a whole, rather than as separate features [34,35]. Numerous studies have demonstrated that Asian adults outperformed Western adults in holistic face processing [36][37][38][39]. For instance, Miyamoto et al. found that the Japanese performed more holistic, rather than featural, strategies in comparison with Americans in choosing to match the prototype faces [37]. Furthermore, Rhodes et al. found that, with the face inversion paradigm in which there is an impaired recognition in inverted compared to inverted faces [40], Chinese subjects exhibited a larger face inversion effect than European counterparts. Some studies used the complete composite face paradigm to tap the holistic face processing, in which the top and bottom parts of two faces are constructed to form a new composite face. In this paradigm, two factors were manipulated. The first manipulation is whether the study face is aligned, meaning that the position of the bottom part of the face is shifted right or left from the top part (misaligned) or not (aligned). And the second is consistency: the consistent trials refer to which the top and bottom parts of the study face are the same as the test face or changed simultaneously; In inconsistent trials, the study face is different from the test face in either the top part or the bottom part. And observers are asked to attend to the target part (such as the top part of the face) and meanwhile to ignore other parts (such as the bottom part of the face). Results showed that the recognition accuracy was better in consistent than inconsistent conditions when faces were aligned, while the consistency effect became weak or disappeared when faces were misaligned [41]. Using the composite face paradigm, Michel et al. showed that Asians had stronger holistic processing (indexed by the composite face effect) as compared to Caucasians [42]. Employing the part-whole paradigm, Tanaka et al. asked Caucasian and Asian observers to recognize facial features of Caucasian and Asians in isolation or in the whole face, showing that Caucasians processed own-race faces holistically compared to Asian faces, while the pattern of holistic processing was observed for both Caucasian and Asian faces in Asians [43]. In summary, converging evidence from different experimental paradigms reaches a consistent agreement that Asians were superior to Westerners in the holistic face processing.

Eye-tracking evidence
Evidence from eye-tracking studies has suggested that participants employed distinct processing strategies to fetch visual information from faces in crosscultural studies. For instance, participants are instructed to learn, recognize, and categorize faces of Western Caucasians and East Asians according to race, and their eye movements were monitored. Results revealed that Western participants tended to fixate on a triangular region (eyes and mouth) of faces, not affected by facial races and tasks; While East Asian observers paid more attention to the central region (nose) of faces [44]. Moreover, Kelly et al. asked children aged 7-12 from the UK and China to complete an old/new face recognition task while simultaneously recording their eye movements. The patterns of fixations observed in children are consistent with those of adults from their respective cultural groups reported in previous studies [44,45], that is, children from the UK fixated more on the eyes and mouth regions whereas children from China fixated more centrally on the nose region. These findings distinguished different fixation patterns for western and eastern subjects during face recognition.

Neural evidence
Many studies have also examined differences in neural responses to faces in observers from different countries. For example, Wang et al. investigated whether distinct attended areas between two cultures tunes the time course of face processing towards configural and featural information respectively. In this experiment, participants were asked to judge the two concurrent faces identical, the two faces either different in the distance between the face features (configural processing), or in the face features (featural processing). Results showed that a configural processing bias is associated with P1 amplitude in their own-race faces and other-race faces and a featural processing bias is associated with P2 amplitude for own-race faces in Chinese participants. In contrast, both a featural processing bias for their own-race faces and a configural processing bias for other-race faces are correlated with P1 amplitude, and a configural processing bias for both own-and other-race faces is related to P2 amplitude in Western participants [46]. A recent study conducted by Ma et al. revealed that relative to German children in the second grade, the N170 response to face is remarkably higher in Chinese children [47]. During an fMRI experiment conducted by Goh et al. [48], East Asians and Westerners were asked to passively view Singaporean and American faces and the corresponding scrambled pictures. They found that more neural responses to faces in the bilateral FFA, especially in the left FFA, were found in Western participants, while more neural responses to faces in the right FFA were observed in East Asian participants. In a nutshell, there are stable and reliable neural mechanisms of different participants underlying face processing.
The existing studies showed that differences in processing faces can be attributed to distinctions among disparate social cultures. Indeed, prior work has been made with respect to scene perception [49,50], description [51], and categorization [52] in support of cultural differences. For example, Western participants paid attention to objects with more salience in an analytic fashion and based on categorization, while East Asians (e.g., Chinese, Japanese, and Korean) paid more attention to relationships and similarities among objects in a holistic fashion when they organized their environment [44]. However, most of these studies did not try to control the participants' second language experience. Therefore, the cultural differences in face processing between Western and Eastern subjects may be confounded by linguistic experiences as well.

Multilingual experience and face processing
A great deal of evidence has suggested that the multilingual vs. monolingual experience has a differential impact on the processing of linguistic and non-linguistic stimuli. For instance, the cerebral lateralization of the word [1] and face processing [53] has been reduced in bilinguals. Regarding how multilingualism affects linguistic and non-linguistic stimuli processing, three hypotheses were proposed to explain the effect of multilingualism on face processing. One hypothesis is the perceptual expertise hypothesis, that is, the amount of exposure to face modulated face processing [54,55]. For example, in Canada, there are many immigrants whose children have to learn not only English, but also their mother tongue, and even other languages. At the same time, they are also exposed to different faces, which may make them process faces more efficiently when viewing different faces. This chapter does not intend to spend much time in discussing the relationship between the amount of exposure to face and face processing given that much work with respect to this hypothesis has been done.
The second hypothesis is that attention is reshaped by visual features of the script. According to the second hypothesis, during the process of learning words, language shapes how its language user deploys the attentional resource to the visual processing of words. For example, Awadh et al. found that French and Spanish individuals possess lower visual attentional span than Arabic individuals [56]. This advantage concerning the visual attention span may have been transferred into the processing of faces. Face processing can be attentionally-driven, so it is possible to change the outcomes and mechanisms of face processing by changing individuals' attentional allocation [57,58]. This hypothesis is partly supported by two recent studies using different script systems, with individuals exposed to different systems behaving differently in the visual processing of face tasks. For example, in the Portuguese script system, Ventura et al. showed that illiterate participants processed faces and houses consistently more holistically compared to literate participants with the composite face paradigm [57]. However, in the Chinese script system, Cao et al. found that literates had a great sensitivity to the spatial configuration of upright, rather than inverted, faces, compared to illiterates in a second-order configuration task [58]. Since social culture was kept constant in the two studies, the inconsistent findings are difficult to be accounted for by the cultural differences, but instead, can be explained by differences in orthographical and visual features in the respective script systems (and possibly the experimental paradigms). On one hand, the face composite paradigm is reflected as the failure to selectively attend to and compare some parts of the face [41]; while the spatial configural distance paradigm typically emphasizes that the viewer attends to the spatial relationship between different parts of the face [59]. Distinct experimental paradigms may ease the level of face processing in different groups of language users. Since the majority of the previous studies demonstrated a generally consistent pattern that East Asian participants show stronger holistic processing compared to Western counterparts, regardless of experimental demands in a short-term task setting, it is more likely that the long-term exposure to the different visual form of a script system (e.g. alphabetical vs. logographic) can be the reason why the attentional allocation is reshaped. On another hand, in terms of visual characteristics, one Chinese character comprises strokes and sub-character components, which is packed into a square configuration with similar size. And Chinese characters are possessed in a highly nonlinear visual complex shape [60,61]. Portuguese words are comprised of some basic letters, and line-shaped. Extensive exposure to these differences in visual features between both script systems could give rise to distinct reading demands for Chinese in comparison with Portuguese words, which are further transferred into face processing. Additionally, some experts have proposed that the research on the impact of literacy acquisition on cognitive ability should always take differences among the distinct script systems into consideration [62].
Last but not the least, the impact of multilingualism on face processing can be traced to the effect of phonemic differences and their impact on the attentional allocation to a speaker's face (attention-reshaped-by-speech hypothesis). Robust and reliable evidence comes from the comparison between monolingual and bilingual infants. Infants in a bilingual environment fixated more at the mouth region of talking and non-talking faces compared to those in a monolingual environment [63]. Given the absence of the script exposure, these findings cannot be attributed to the differences in visual features between writing systems; In contrast, it is more possible that the language environment to which infants are exposed constrain their visual processing. Using the Cambridge Face Memory Tests, Burns found that bilingual Singaporean Chinese participants showed a decreased other race effect with the increase of reported cross-language proficiency [64]. This relationship was driven by Chinese, rather than English, listening ability. These findings suggested that multilingual exhibit different processing of faces compared with monolinguals is attributed to how words are realized in the face in communication.
Given these remarkable differences between the logographic and alphabetic scripts outlined above, extensive exposure to distinct script systems may result in differences in processing faces. Combined with the existing findings and theories, we proposed that Asian participants with logographic script system process face different from Western counterparts with alphabetic script systems. Future studies are encouraged to clarify these confounding explanations.

Future studies
Despite much evidence from the cross-cultural and cross-linguistic studies that indicate differences in face processing, more efforts should be devoted to effectively dissociate possibly separate contributions of social culture and script system to the explanation of differences in face processing and to clarify how multilingualism affects face processing. Moreover, the relationship between the script system and face processing provides an important window into the understanding of brain plasticity. To answer the issue, we proposed to test multilingual speakers to unveil the impact of a distinct script systems on face processing. Several possible research directions are put forward, such as 1) training illiterates to acquire distinct script systems respectively in an identical culture; 2) comparing monolingual Chinese and Korean subjects sharing the East-Asian culture; 3) comparing monolinguals and bilinguals (or multilingual) speakers; 4) using one artificial language to train participants to either learn the visual form or the speech in communication.

Training illiterates to learn distinct script systems respectively
One way to dissociate the contributions of social culture and script systems to holistic processing is to train the distinct types of script systems on illiterates from an identical social culture. Illiterates are a special population who share the social culture (including spoken language) with literates. One important distinction between literates and illiterates is that literates acquired the script system. Therefore, it helps to understand how a script system shapes the processing of faces in a particular social culture. Two previous studies, Ventura et al. [57] and Cao et al. [58], examined the relationships between the acquisition of a script system and face processing in the Western and the Eastern culture respectively. However, these studies cannot reveal the potential contributions of the distinct script systems to face processing because of the absence of a direct comparison. Based on the previous studies, we proposed one possible way of segregating the role of social culture from script system in face processing, in other words, we propose ways to heighten the weights of the script system in the explanation of face processing, thus merely representing the influence of different script systems on face processing after controlling social culture. More specifically, this can be achieved by teaching Western illiterates in Chinese and English respectively or teaching Chinese illiterates in Chinese and English respectively. Combined with a variety of experimental paradigms which tap distinct dimensions of holistic face processing, the relationships between script system and face processing could be systematically investigated.

Comparing face processing mechanisms between monolinguals and multilingual
The first suggestion is proposed against the monolingual context. With the development of globalization, more and more individuals have become bilingual or multilingual. It is possibly easier for bilinguals to acquire the script system of a second language than master the social culture. Therefore, one viable way of examining the relationships between distinct types of script systems and face processing is to recruit the monolinguals and bilinguals (or multilingual) and compare the differences between these groups. In this case, Chinese-English bilinguals (or multilingual) and Chinese monolinguals, whose mother tongue is Chinese, or English-Chinese bilinguals (or multilingual) and English monolinguals, whose mother tongue is English, are recruited to decrease the possible role of social culture in the explanation of face processing.

Comparing face processing mechanisms between Chinese and Korean
Here, we focused on the role of script or spoken form of one language in face processing. Previous studies have shown that too many differences in face processing existed between East Asian and Western cultures that prevent attribution of sole factor to the cultural difference in face processing. Therefore, one way is to investigate the role of two disparate script systems in face processing in East Asian culture. As we know, respecting the East Asian culture are people from China, Japan, and Korea, and so on. Notably, one possibility is to explore the relationship between script systems and face processing by recruiting subjects with different script systems in East Asia, which lies in an identical culture. A typical way to achieve this is to compare individuals from China and South Korea. One unique feature of this comparison is that the pronunciation of Korean resembles that of an alphabetic language, however, the visual characteristics are closer to Chinese [65]. Therefore, employing the identical experimental task, one could morph the Chinese and Korean faces. By doing so, one could eliminate the impact of the amount of exposure to face processing, and explain face processing by the weights of speech.

Training the participants to learn words via visual or speech forms in an artificial language
Compared with the natural language, an artificial language has some advantages in studying how the script system affects face processing. For example, the artificial language can allow researchers to train participants to learn only the visual features of characters or only the pronunciation of characters, which facilitates understanding of through what mechanisms the script system could affect the face processing.

Conclusion
Massive evidence revealed a tight link between the acquisition of script systems with face processing, and by reviewing prior studies, we proposed that distinct script systems impact face processing in a different way. To clarify how the script system affects face processing, we proposed the attention-reshaped-by-language hypothesis. Finally, further research directions were proposed.