Emotion recognition and expression in ASD

Numerous studies have shown that individuals with autism spectrum disorder (ASD) struggle with interpreting emotions conveyed through vocal cues (e.g., [3, 5, 6]). Additionally, evidence (e.g., [2, 4]) suggests difficulties in expressing social communicative intentions, making it challenging for NT listeners to infer the emotional intention. Autistic speakers often do not conform to the same conventions as NT speakers when using vocal cues such as pitch, loudness, and speech rate. For example, Grossman and colleagues [2] investigated how children and adolescents diagnosed with high-functioning autism (HFA) retold emotionally laden stories compared to an NT group. The results showed that two NT coders rated HFA participants as more vocally expressive than NTs, but also as sounding more awkward. Despite these differences, HFA output was more accurately identified than NT speakers’ output. Similarly, [4] found an increase in pitch range for autistic individuals when repeating emotional prosody. Furthermore, research by [7] demonstrated that autistic children differ from non-autistic children in pitch control abilities, with a faster response to vocal pitch changes associated with poorer social competence as rated by parents. These findings collectively suggest that ASD speakers use prosodic features differently from NT populations, and these differences in prosody production are linked to social difficulties in autism.

Emotional communication between neurotypical speakers and individuals with autism

These data prompt the question of whether the differences in vocal cue usage have an actual impact on the emotional communication between NT speakers and individuals with autism. We know from studies on NT speakers that emotional prosody adheres to common display rules, such as the tendency for happiness to be expressed with a high pitch compared to neutral speech [8]. When these norms are followed, others can easily recognize the intended emotion of the speaker. Additionally, brain responses indicate that deviations from listeners’ expectations are quickly detected [9]. Event-related brain potentials specifically demonstrate that listeners detect such violations within approximately 450 milliseconds of a change in vocal expression. Therefore, not conforming to a listener’s expectations may result in communication difficulties. However, despite substantial evidence indicating that abnormal prosodic production is characteristic of ASD (see [10] for a comprehensive review), there is still a scarcity of data on how atypical prosodic production affects the perception of NT listeners. One of the few studies that investigated this important issue examined emotional prosody production in autistic males and their NT counterparts. Autistic speakers, when compared to NT speakers, spoke at a slower and louder pace. They also exhibited a wider pitch range when expressing emotions but not when using a neutral voice. These examples were judged to sound "less natural" [11]. Similar to the findings of [2], who reported higher emotion recognition for ASD speech when assessed by two NT coders, [4] demonstrated that naive listeners were also better at categorizing vocal examples from autistic speakers compared to NT speakers. These data once again suggest that non-typical use of cues may not directly impact emotion identification itself but rather primarily lead to an increased perception of oddness in NT listeners, as overemphasizing cues violates common display rules.

However, not all studies have confirmed that emotional speech produced by individuals with autism is ultimately better recognized. [12] asked 10 listeners to evaluate materials from individuals with HFA and NTs and reported better identification of angry, happy, and neutral speech when produced by NTs, with a trend towards better recognition of sad speech from individuals with autism. These findings imply that successful categorization might depend on the specific target emotion and/or the possibility that overemphasizing certain cues is more advantageous for certain emotions. In contrast to most other studies, [12] also accounted for response biases in judges, which refers to their inclination to preferentially choose one response category over another. They specifically considered the frequency at which each emotional category was selected (e.g., raters attributed happiness and neutral more often than fear or sadness). When examining these results, the only remaining group difference indicated better recognition of sad utterances expressed by individuals with autism.

In summary, reviewing the literature leads to the following conclusions: 1.) Individuals with ASD exhibit distinct qualitative differences in their expression of emotions compared to NT individuals. 2.) These differences do not necessarily lead to inferior recognition of emotions but rather appear to depend on the specific emotion being expressed. 3.) Further research employing more rigorous methodologies (e.g., a larger number of raters and emotional categories, correction for response bias) is necessary to delve deeper into this issue.

Current study

Our aim was to make a contribution to this objective by combining the strengths of previous studies and addressing gaps in the literature. Firstly, we examined the ability of NT listeners to recognize the emotional intentions of both HFA individuals and NT speakers. We included both female and male speakers since previous research has predominantly focused on male participants. Although ASD is more commonly diagnosed in males than females [13], testing both groups is crucial due to gender-related differences in autistic characteristics across behavioral and neural domains [14, 15]. Notably, women are often more challenging to diagnose than men [16], which is linked to their stronger masking or camouflaging tendencies and compensatory behaviors. These masking and compensatory efforts help navigate social situations and may involve suppressing and controlling behaviors associated with autism that are deemed inappropriate in a given context or imitating NT behaviors [17]. These types of behaviors are particularly observed in cognitively high-functioning women, especially those whose autism is recognized later in life [18–20]. In the domain of emotional prosody, compensatory strategies might include autistic speakers attempting to closely imitate or replicate "typical" productions that adhere to emotional "display rules" [21]. Autistic speakers may demonstrate an inclination to exaggerate their emotional speech, similar to observations made in actors when they are instructed to portray various emotional voices [22]. This exaggeration facilitates the determination of the intended emotion by the listener.

Secondly, we adopted the approach of [11] by testing a larger number of NT listeners instead of relying on a small group of coders or trained judges. Thirdly, similar to the methodology of [12], we accounted for response biases in our judges. This is crucial in evaluating which emotions might be more challenging for judges to identify accurately. Finally, to gain a comprehensive understanding of how NT listeners perceive emotions expressed by autistic speakers, we delved into voice perception in greater detail, focusing on the perceived valence/affect, voice modulation, and voice control.

Stimulus development

Speakers.

All speakers were recruited in summer 2017 on a voluntary basis via email and facebook (snowball effect) calling for individuals previously professionally diagnosed with HFA. We aimed to recruit a similar number of speakers reported in past emotional prosody recognition tasks. While studies exploring differences in listener groups tend to present one or two speakers who express materials to listeners (e.g., [12, 23]), studies that aim to compare whether listeners respond differently to different speaker groups (e.g., age, gender, culture) often present three to four speakers in their studies (e.g., [8, 24]). Our call for speakers resulted in six HFA volunteers (no payment was offered to participants). We asked all speakers to complete an unlabeled copy of the Autism-Spectrum Quotient (AQ; [25]) after the recordings. The instrument was developed to allow for a brief assessment of autism traits. An AQ score of 35 or higher was obtained from all HFA speakers recruited (scores ranging from 32–50 indicate a strong likelihood of HFA or Asperger’s syndrome, see [25]). To match our HFA sample, we then aimed to recruit an additional six speakers with similar background characteristics (e.g., place of origin, age, gender) who had not received a professional diagnosis of HFA. All volunteers filled out an unlabelled version of the AQ and received a score of 21 or less (i.e. within range of a NT = population; [25]). Table 1 lists the characteristics that we aimed to control for. Paired samples t-test confirmed that groups only differed in AQ scores (p < .001) but no other variables (all ps≥0.15).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Speaker information.

https://doi.org/10.1371/journal.pone.0293233.t001

Study 1: Emotional prosody recognition study

Study 1 data analysis

The statistical analysis was run on SPSS 24 (IBM software). For the analysis, Wagner’s unbiased hit rate was used to calculate accurate responses to statistically control for biases toward selecting a specific ‘default’ emotion for exemplars [28]. It is a metric used to evaluate the accuracy or reliability of results in a study. The higher the hit rate, the more accurate and reliable the identifications are (1 = highest systematic selection; 0 = represents no systematic selection). Unbiased hit rates were arcsine transformed, a procdeure commonly applied to stabilize the variances and improve the normality of data that represents proportions or percentages. Arcsine-transformed unbiased hit rate data were analyzed using a General Linear Model treating Speaker Type (ASD or NT), Speaker Sex (female, male), and Emotion (angry, fearful, happy, neutral, sad, [pleasantly] surprised) as within-subjects variables.

Study 1 results

Unbiased hit rates were highest for angry vocalizations, followed by pleasantly surprised. Recognition for fearful, happy, neutral and sad exemplars varied depending on speaker type (see Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. This figure depicts emotion recognition scores (as arcsine transformed Hu scores) obtained from neurotypical (NT) participants when listening to emotional sentences expressed by female and male high-functioning autistic (ASD) and neurotypical (NT) speakers.

Error bars reflect standard error. * indicate significant differences at p < .05.

https://doi.org/10.1371/journal.pone.0293233.g001

The main effect of Speaker Sex, F(1, 52) = 262.297, p < .001, ηp2 = .835, Speaker Type, F(1, 52) = 21.468, p < .001, ηp2 = .292, and Emotion, F(5, 260) = 27.592, p < .001, ηp2 = .347, were significant. Planned post-hoc t-tests for the latter effect showed that all emotional comparisons yielded significant effects (all p’s ≤.02) apart from contrasts between fearful and neutral, fearful and sad, and neutral and sad stimuli.

The significant interactions between Speaker Type x Emotion, F(5, 260) = 16.016, p < .001, ηp2 = .235, and Speaker Sex x Emotion, F(5, 260) = 8.604, p < .001, ηp2 = .142, were informed by a significant three-way interaction, Speaker Type x Speaker Sex x Emotion, F(5, 260) = 4.943, ηp2 = .087, p < .001. Planned pairwise comparisons by Speaker Sex and Emotion revealed that male NT speakers were significantly better recognized than male ASD speakers (all p’s < .001). For female speakers, there was only a difference between speaker groups when listeners had to recognize fearful stimuli (p < .001) which were better recognized if spoken by NT speakers. These data showed differences in how well NT listeners recognized emotions expressed by NT and ASD speakers, highlighting speaker sex effects: male ASD speakers were less well recognized by NT listeners irrespective of which emotions ASDs expressed. In contrast, only fearful expressions were less well recognized when spoken by ASD female speakers when compared to recognition rates for materials spoken by female NT speakers.

Study 2: Valence, voice modulation, and control over voice cue use

In a second study, we explored how the exemplars from both speaker groups are perceived in terms of affection expressed, pitch modulation, and control over the use of voice cues. This is important to gather information on how listeners perceive the speakers’ speech, rather than just focusing on the ability to recognize emotions expressed.

Study 2 data analysis

Ratings were aggregated across all stimuli by participants for each dependent variable (DV). A Shapiro-Wilk test was performed for each DV (separately for materials spoken by ASD and NT speakers) and showed that the distribution of scores did not depart significantly from normality for any of the DVs (valence, voice modulation, voice control; all ps >.15). Based on this, means of ratings were analyzed with SPSS 24 (IBM software) using a General Linear Model treating Speaker Type (ASD or NT), Speaker Sex (female, male), and Emotion (angry, fearful, happy, neutral, sad, [pleasantly] surprised) as within-subjects variables. Mean ratings for each indicator separately can be found in Fig 2A–2C.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2.

a-c. These figures depict rating scores obtained from neurotypical (NT) participants when listening to a subset of emotional sentences expressed by female and male high-functioning autistic (ASD) and neurotypical speakers. Error bars reflect standard error.

https://doi.org/10.1371/journal.pone.0293233.g002

Study 2 results

Atypical emotional prosody in male ASD speakers?

It has long been documented that individuals with ASD exhibit distinct prosodic characteristics in their speech compared to NT individuals. Specifically, studies have reported differences in pitch usage, such as higher mean pitch and increased pitch variation, as well as difficulties in controlling loudness (e.g., speaking too loudly or too quietly) in ASD speech [4, 30, 32]. Additionally, ASD speakers tend to have longer utterance durations compared to NT speakers [33, 34]. However, previous studies have primarily focused on acoustic data from male speakers or included a small sample of female speakers within a larger male speaker group (e.g., [30]). Consistent with this body of evidence, the emotional prosody exemplars produced by male speakers here displayed a similar pattern (see S1–S12 Tables). Descriptive analyses suggest that male autistic speakers exhibited higher mean pitch and longer utterance durations compared to male NT speakers. In the case of female speakers, the autistic group showed less variability in loudness and decreased vocal effort (energy in high-frequency regions) when compared to NT speakers. While these descriptive acoustic findings should be interpreted with caution due to the small number of speakers in our study, they provide valuable insights into understanding why emotional prosody produced by female speakers was similarly well recognized across both speaker groups, while male ASD emotional speech exemplars presented perception difficulties (e.g., pitch variations causing confusion for NT listeners rather than reduced loudness variability).

Our results differ from those of [11], who examined male speakers and reported higher recognition rates for stimuli produced by individuals with ASD compared to NT speakers. They suggested that NT listeners found it easier to categorize vocal expressions as ASD speakers exaggerated certain vocal features. Here, we find no such effect, indicating that the acoustic variations observed between our male speaker groups, previously linked to greater expressiveness, did not lead to a more accurate identification of emotional attributes. Instead, the data suggest that deviating from the conventions of NT emotional vocal cues, such as speaking more slowly and with an increased mean pitch, made it more challenging for NT listeners to discern the intended emotion. The observation that happy and surprised expressions sounded more monotone when spoken by ASD speakers compared to NT speakers further supports the notion that emotions were not conveyed in line with NT expectations. Additionally, male ASD speakers were perceived as having less control over their voices. Interestingly, neutral expressions produced by autistic speakers were not less accurately recognized, suggesting that adhering to stereotypical norms may be less crucial in conveying neutrality.

In summary, our findings support the idea that male autistic speakers express emotions differently from their NT counterparts, leading to difficulties for NT listeners in accurately perceiving the intended emotional expression. Importantly, these patterns were not observed for female ASD speakers.

Typical emotional prosody in female ASD speakers?

Given the past prominent focus on male ASD speakers, it is important to examine the speech of female ASD speakers more closely. Evidence suggests that even girls who do not meet the full criteria for an ASD diagnosis may still exhibit difficulties in social communication contexts [35], suggesting that atypical communication is not solely a characteristic of male autistic speakers. Nevertheless, it remains to be determined how prosody usage differs between male and female ASD speakers and how it compares to NT speakers.

Descriptive results from our study (see S1 Table) indicate that female ASD speakers only differed from female NT speakers in their use of loudness and vocal effort. Thus, the two groups exhibit comparable use of acoustic characteristics for other key variables, resulting in similar recognition rates among NT listeners for sentences expressed by both groups. In contrast to male ASD speakers, female individuals with autism had their emotional intentions accurately recognized by NT listeners in all tested categories, except for the category of fear. This finding is further illuminated by findings from Study 2. Of particular interest is that NT female speakers were perceived as sounding more negative than female ASD speakers when expressing fear, potentially explaining why NT fear expressions were easier to recognize. Furthermore, in terms of voice modulation, listeners rated fearful stimuli expressed by autistic speakers (regardless of gender) as sounding more modulated. This raises the question of whether autistic speakers were attempting to overcompensate in expressing this specific emotion, an effect not observed for other emotions where NT speakers were rated as sounding more modulated.

Building on the idea of over-expressed output [11], we speculate that the excessive use of specific features can impede the recognition of emotions when the produced pattern deviates too much from prototypical displays or creates too much overlap with other emotional intentions, leading to ambiguity between categories. Thus, not fully adhering to prototypical or normative standards (e.g., using a wider pitch range or a louder voice) may hinder the recognition of their expressions at times. However, the effect of "misuse" or "exaggeration" of cues could depend on the relative "importance" of an acoustic cue in expressing the intended emotion. For example, if speakers were to vary their pitch in a non-standard way, it could affect the production of happiness (which requires pitch modulation) more than the production of anger (which could be expressed with either a wide or small pitch range). This hypothesis should be directly tested in future studies.

A final point regarding the specific effect observed for fear expressions is worth noting. In the literature on facial expression recognition, it has been repeatedly suggested that individuals with HFA process fearful expressions differently from NT individuals [36, 37], with associated differences in brain activation patterns. Based on the findings of our study, it can be speculated that these processing differences extend to the speech channel as well, suggesting modality (e.g. vocal, facial, gesture) non-specific effects.

Compensation effect?

To the best of our knowledge, this is the first study to consider gender effects when examining whether the emotional expressions of ASD and NT speakers are recognized differently by NT listeners. The finding that emotional expressions from female ASD speakers are well recognized suggests that productions generally align with the expectations of listeners. This could indicate that female autistic speakers do not exhibit atypical emotional prosody or that they make an effort to imitate NT behavior. There is a growing body of evidence suggesting that autistic individuals, particularly females, engage in compensatory behaviors and attempt to conceal their ASD symptoms in order to fit in with NT individuals [17, 38–40]. This camouflaging behavior is thought to contribute to the later diagnosis or even misdiagnosis of autistic females [41–45]. While the current data do not provide specific evidence of imitating NT expressions by female speakers, the acoustic analyses of our sample indicate that there are differences in speech expression (less variation in loudness and reduced vocal effort), regardless of the emotion. This is consistent with developmental studies, which have shown atypical prosodic expressions in ASD (e.g., [32]), although girls were often underrepresented in those samples. Here, ASD speakers were perceived as sounding more monotone than NT speakers, suggesting that the reduced loudness variation in autistic female speakers is detectable by NT listeners, even if it does not lead to difficulties in recognizing emotional intentions.

Yet, the current findings highlight that female individuals with ASD largely adhere to (NT) display rules for emotional behavior in the vocal domain (e.g., [8]), except when expressing fear. This could mean that female autistic speakers develop compensatory vocal strategies during social communication to a greater extent than their male counterparts, a finding observed in other emotional domains (e.g., forcing eye contact or displaying expected emotional facial expressions; [17]). This effect is noteworthy as speakers were asked to express emotions in a manner they deemed appropriate, though the instruction itself may not be enough to overcome a desire to express stimuli in line with what they believed the experimenter expected. This should be directly investigated by providing explicit instructions that compensation strategies are not anticipated. Similarly, future research could directly examine the influence of camouflaging strategies on prosody production by administering the Camouflaging Autistic Traits Questionnaire (CAT-Q; [46]) and correlating the results with prosodic patterns.

Severity of ASD

An alternative explanation for the observed sex effects may be that differnces in production stem from differences in the severity of ASD. Although the AQ scores obtained in this study were similar, and there is currently no research employing AQ to predict ASD severity, it would be valuable for future studies to explore alternative approaches to assess HFA populations and ascertain the role of ASD severity in these differences. Future studies should thus aim to even further increase number of female and male speakers. Here, we aimed to present several exemplars from speakers but future studies could increase speaker size by reducing exemplars needed from each speaker.

Clinical relevance

The current findings have relevance for autism assessment and diagnosis. It is essential to consider that female ASD speakers exhibit distinct patterns of prosody production compared to their male counterparts. Whether these patterns result from mimicking prosody from NT speakers as a compensatory strategy or simply displaying less divergent patterns, the resemblance of prosody production to that of NT speakers can potentially hinder accurate and timely diagnosis, particularly for individuals with HFA (see [47]). It is hoped that the present results will prompt future studies to systematically investigate these behaviors in larger sample sizes, while also controlling for other characteristics. For instance, although age was matched between ASD and NT speakers in this study, it is important for future research to carefully consider and control for potential age effects (though note prior evidence indicates successful recognition of emotional prosody from both young and middle-aged speakers (e.g., [24]).

Conclusion

This study is the first to document differences in the perception of vocal emotions by NT listeners when expressed by both male and female ASD speakers. It emphasizes the importance of studying emotional prosody not only in male ASD speakers but also in females. Future investigations should aim to move beyond simple comparisons of emotion recognition and instead delve into understanding the underlying mechanisms through which non-prototypical cue use and a lack of control over voice cues may contribute to difficulties in social interactions. Additionally, the potential impact of camouflaging strategies on emotional speech output should be further explored.