Abstract
This paper deals with the interaction of visual and acoustic cues of irony, observed in the speech of Russian professional actors. We selected ironic and non-ironic utterances from modern films and series taking into account narrow and broad context, lexical and semantic markers. Then we extracted the target utterances from the context eliminating any markers of irony. The participants of the perceptual experiments could rely only on the visual and acoustic (prosodic) cues. In the first experiment we suggested to the participants mute video files containing the target ironic and non-ironic utterances. The second experiment was conducted with the audio files only of the same utterances extracted from the films. In the third experiment video and audio were suggested simultaneously, as in the natural situation of film watching, but still without any context or lexical marker. Segment duration, pitch movement, gestures and mimics, as well as their synchrony in the well-recognized target utterances were analyzed. The results of the experiments demonstrated that the visual cues were more important for irony perception than the audio signal. Yet, some video stimuli that had low recognition of irony were better recognized in the experiment with audio. It led us to suppose that actors use in various proportion visual and acoustic cues to express irony in speech. The results of this research can have practical application in both speech recognition and speech generation used in artificial intelligence systems, as well as in the forensic phonetics and second language acquisition.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wagner, P., Malisz, S., Kopp, S.: Gesture and speech in interaction: an overview. Speech Commun. 57, 209–232 (2014)
Becker, R., et al.: Aktionsarten, speech and gesture. In: Proceedings of GESPIN2011: Gesture and Speech in Interaction, Bielefeld, Germany (2011)
Bergman, K., Aksu, V., Kopp, S.: The relation of speech and gestures: temporal synchrony follows semantic synchrony. In: Proceedings of GESPIN2011: Gesture and Speech in Interaction, Bielefeld, Germany (2011)
Aylett, R., Krenn, B., Pelachaud, C., Shimodaira, H. (eds.): IVA 2013. LNCS (LNAI), vol. 8108. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40415-3
Brugman, H., Wittenburg, P., Levinson, S.C., Kita, S.: Multimodal annotations in gesture and sign language studies, In: Third International Conference on Language Resources and Evaluation, pp. 176–182 (2002)
Kipp, M.: Multimodal annotation, querying and analysis in ANVIL. In: Multimedia information extraction, pp. 531–368. John Wiley and Sons Inc., Hoboken, NJ (2009)
De Ruiter, J.P., Bangerter, A., Dings, P.: Interplay between gesture and speech in the production of referring expressions: investigating the tradeoff hypothesis. Top. Cogn. Sci. 4(2), 232–248 (2012)
Barbulescu, A., Ronfar, R., Bailly, G.: Generative audio-visual prosodic model for virtual actors. In: EEE Engineering in Medicine and Biology Magazine: The Quarterly Magazine of the Engineering in Medicine & Biology Society, pp. 40–51 (2017)
Haverkate, H.: A speech act analysis of irony. J. Pragmat. 14, 77–109 (1990)
Skrelin, P., Kochetkova, U., Evdokimova, V., Novoselova, D.: Can we detect irony in speech using phonetic characteristics only? – looking for a methodology of analysis. In: Karpov, A., Potapova, R. (eds.) SPECOM 2020. LNCS (LNAI), vol. 12335, pp. 544–553. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60276-5_52
Kochetkova, U., Skrelin, P., Evdokimova, V., Novoselova, D.: Perception of irony in speech. In: Sherbakova, O. (ed.) Proceedings of the 4th International Conference on Neurobiology of Speech and Language, pp. 72–73. Skifia-Print, Saint Petersburg (2020)
Cutler, A.: On saying what you mean without meaning what you say. In: Proceedings from the 10th Regional Meeting of the Chicago Linguistic Society, pp. 117–123. CLS, Chicago (1974)
Niebuhr, O.: Rich reduction: Sound-segment residuals and the encoding of communicative functions along the hypo-hyper scale. In: 7th Tutorial and Research Workshop on Experimental Linguistics, pp. 11–24. St. Petersburg, Russia (2016)
Cheang, H., Pell, M.: Acoustic markers of sarcasm in Cantonese and English. J. Acoust. Soc. Am. 126(3), 1394–1405 (2009)
McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)
Loehr, D.: Temporal, structural, and pragmatic synchrony between intonation and gesture. In: Laboratory Phonology. Journal of the Association for Laboratory Phonology 3, 71–89 (2012)
Chui, K.: Temporal patterning of speech and iconic gestures in conversational discourse. J. Pragmat. 37, 871–887 (2005)
Grishina, E.A.: Russkaia Gestikulatsia s Lingvisticheskoi Tochki Zrenia [Russian Gesticulation from the Liguistic Point of View]. Iazyki Slavianskoi Kulturi, Moscow (2017). (in Russian)
Acknowledgments
The project “Acoustic correlates of irony with respect to basic types of pitch movement” was supported by the RFBR grant № 20-012-00552.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kochetkova, U., Evdokimova, V., Skrelin, P., German, R., Novoselova, D. (2022). Interplay of Visual and Acoustic Cues of Irony Perception: A Case Study of Actor’s Speech. In: Malykh, V., Filchenkov, A. (eds) Artificial Intelligence and Natural Language. AINL 2022. Communications in Computer and Information Science, vol 1731. Springer, Cham. https://doi.org/10.1007/978-3-031-23372-2_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-23372-2_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23371-5
Online ISBN: 978-3-031-23372-2
eBook Packages: Computer ScienceComputer Science (R0)