Neural organization for recognition of grammatical and emotional facial expressions in deaf ASL signers and hearing nonsigners

Abstract

Recognition of emotional facial expressions is universal for all humans, but signed language users must also recognize certain non-affective facial expressions as linguistic markers. fMRI was used to investigate the neural systems underlying recognition of these functionally distinct expressions, comparing deaf ASL signers and hearing nonsigners. Within the superior temporal sulcus (STS), activation for emotional expressions was right lateralized for the hearing group and bilateral for the deaf group. In contrast, activation within STS for linguistic facial expressions was left lateralized only for signers and only when linguistic facial expressions co-occurred with verbs. Within the fusiform gyrus (FG), activation was left lateralized for ASL signers for both expression types, whereas activation was bilateral for both expression types for nonsigners. We propose that left lateralization in FG may be due to continuous analysis of local facial features during on-line sign language processing. The results indicate that function in part drives the lateralization of neural systems that process human facial expressions.

Introduction

Recognition of facial expressions of emotion is a crucial communication skill relevant for both human and non-human primates. Sensitivity to emotional facial expressions occurs very early in development, and the neural circuitry underlying facial affect recognition is partially independent of neural systems that underlie recognition of other information from faces, such as person identity or gender [13], [17], [34]. Humans have clearly evolved an ability to quickly recognize emotional and socially relevant facial expressions, and this ability appears to be processed by a distributed neural circuitry, generally lateralized to the right hemisphere [8], [24], [31]. We investigate the plasticity and functional organization of this neural circuitry by studying facial expressions that do not convey emotional or social-regulatory information, namely the linguistic facial expressions produced by users of American Sign Language (ASL).

A unique and modality specific aspect of the grammar of ASL and other signed languages is the use of the face as a linguistic marker. Distinct facial expressions serve to signal different lexical and syntactic structures, such as relative clauses, questions, conditionals, adverbials, and topics [4], [29]. Linguistic facial expressions differ from emotional expressions in their scope and timing and in the facial muscles that are used [28]. Linguistic facial expressions have a clear onset and offset, and are coordinated with specific parts of the signed sentence. These expressions are critical for interpreting the syntactic structure of many ASL sentences. For example, restrictive relative clauses are indicated by raised eyebrows, a slightly lifted upper lip, and a backward tilt of the head. When this combination of head and facial features occurs, the co-occurring lexical items are interpreted as constituting a relative clause [22]. Facial behaviors also constitute adverbials that appear in predicates and carry various specific meanings. For example, the facial expression glossed as MM (lips pressed together and protruded) indicates an action done effortlessly, whereas the facial expression TH (tongue protrudes slightly) means “carelessly” (see Fig. 1A). These two facial expressions accompanying the same verb (e.g., DRIVE) convey quite different meanings (“drive effortlessly” or “drive carelessly”).

Of course, deaf signers also use their face to convey emotional information. When perceiving visual linguistic input, ASL signers must be able to quickly identify and discriminate between different linguistic and affective facial expressions in order to process and interpret signed sentences. Thus, signers have a very different perceptual and cognitive experience with the human face compared to nonsigners. This experience appears to result in specific enhancements in face processing. Several studies have found that both hearing and deaf signers perform significantly better than nonsigners in distinguishing among similar faces (e.g., the Benton Faces Test), in identifying emotional facial expressions, and in discriminating local facial features [2], [7], [15], [16], [23]. It is possible that ASL signers exhibit a somewhat different neural representation for face perception due to their unique experience with human faces.

Recently, it has been argued that cognitively distinct aspects of face perception are mediated by distinct neural representations [17]. We hypothesize that the laterality of these representations can be influenced by the function of the expression conveyed by the face. Linguistic facial expressions are predicted to robustly engage left hemisphere structures only for deaf signers, whereas perception of emotional expressions is predicted to be lateralized to the right hemisphere for both signers and nonsigners.

An early hemifield study by Corina [10] found distinct visual field asymmetries for deaf ASL signers when recognizing linguistic and emotional facial expressions, compared to hearing nonsigners. The visual field effects were contingent upon the order of stimulus presentation. Both emotional and linguistic facial expressions produced significant left visual field (right hemisphere) asymmetries when emotional facial expressions were presented first. In contrast, when deaf signers viewed the linguistic expressions first, no significant visual field asymmetries were observed. Although suggestive, the results do not provide support for a dominant role of the left hemisphere in recognizing linguistic facial expressions. A right visual field (left hemisphere) advantage was not observed for linguistic facial expressions.

Nonetheless, data from lesion studies indicate that damage to the left hemisphere impairs signers' ability to produce ASL linguistic facial expressions [11], [21]. In contrast, damage to the right hemisphere impairs the ability to produce emotional facial expressions, but leaves intact the ability to produce linguistic facial expressions [11]. With respect to perception, a recent study by Atkinson et al. [3] examined the comprehension of non-manual markers of negation in British Sign Language (BSL) by signers with left- or right-hemisphere damage. Non-manual negation in BSL is marked by a linguistic facial expression and an accompanying headshake. Right-hemisphere-damaged signers were impaired in comprehending non-manual negation, in contrast to left-hemisphere-damaged signers who were unimpaired. However, a negative headshake is obligatory for grammatical negation in BSL, and recognition of a headshake is distinct from the recognition of the linguistic facial expressions because (1) a headshake can be used non-linguistically to signal a negative response and (2) a headshake can occur without signing, unlike grammatical facial expressions which are bound to the manual signs and do not occur in isolation. Thus, the neural organization for the recognition of linguistic facial expressions may differ from that for the recognition of headshakes marking negation. With the advent of functional neural imaging, we can now study the brain regions involved in the perception of linguistic and emotional facial expressions in intact deaf signers with much greater anatomical precision than is possible with lesion studies.

Neuroimaging results with hearing subjects indicate that the superior temporal sulcus (STS) is critically involved in processing changeable aspects of the face, such as eye gaze [19], mouth configuration [27], and facial expression [17]. Furthermore, attention to emotional facial expression can modulate activity within the right superior temporal sulcus [24]. We predict that attention to linguistic facial expressions will produce greater activity within the left STS for deaf signers than for hearing nonsigners.

In addition, recognition of facial expressions may modulate activity within the face-responsive areas within inferior temporal cortex. In particular, the fusiform gyrus (FG) has been identified as crucial to the perception of faces and as particularly critical to perceiving invariant properties of faces, such as gender or identity [17], [20]. Activation within the fusiform gyrus in response to faces may be bilateral but is often lateralized to the right hemisphere. We hypothesize that the linguistic content of ASL facial expressions for deaf signers will modulate activity within the fusiform gyrus, shifting activation to the left hemisphere. In contrast, we hypothesize that hearing nonsigners will treat the unfamiliar ASL linguistic expressions as conveying social or affective information, even though these expressions are unique and non-identical to canonical affective expressions [28]. Thus, activation in the fusiform gyrus is expected to be bilateral or more lateralized to the right hemisphere for hearing subjects with no knowledge of ASL.

In our study, subjects viewed static facial expressions performed by different models who produced either emotional expressions or linguistic expressions (adverbials indicating manner and/or aspect) with or without accompanying ASL verbs. Subjects made same/different judgments to two sequentially presented facial expressions, blocked by expression type. This target task alternated with a control task in which subjects made same/different judgments regarding gender (the models produced neutral expressions with or without verbs). Fig. 1 provides examples of the ‘face only’ condition, and Fig. 2 provides examples from the ‘face with verb’ condition. In this latter condition, models produced ASL verbs with either linguistic, neutral, or emotional facial expression. The ‘face only’ condition was included because most previous face processing studies presented isolated face stimuli. Although emotional facial expressions can be produced without an accompanying manual sign, ASL linguistic facial expressions are bound morphemes (like –ing in English) that must co-occur with a manually produced sign. Therefore, we included a second ‘face with verb’ condition in order to present the linguistic facial expressions in a more natural linguistic context.