Through the looking glass: a social semiotic and linguistic perspective on the study of video chats

Abstract

This paper provides a theoretical framework for the study of the video chat, a spontaneous web-based synchronous text that allows forms of communication in which social semiotic resources come into play and produce a new terrain of investigation for researchers in the field of linguistics, multimodality, communication and media studies, visual ethnography, and digital literacy. In particular, the paper singles out some aspects for analysis, such as the alternation of speech and writing, new proxemic and kinetic patterns, gaze management, and impossibility of eye contact and discusses some examples from digital field work on multiparty video-based interactions. Speech and writing are technologically integrated, allowing participants to mode-switch, i.e., to alternate between spoken and written discourse. New arrangements of verbal and nonverbal resources attempt to simulate face-to-face conversations. However, the illusion of a face-to-face conversation dissolves as soon as video chat-specific resources are unpacked. Despite growing research into nonverbal behavior, video chat data challenge visual analysts and researchers for a number of reasons. A transcription model, developed for the purpose of analyzing these specific texts, will be sketched to give a brief account of significant data that need to be incorporated into multimodal transcription and annotation. Reflections and conclusions are drawn according to the contribution that intersemiotic studies can potentially provide for web text analysis, given the constant expansion of web-based texts and the challenge it brings as regards new notions of textuality.