Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus

2.1.1 Inter-pausal units (IPUs)

The conversational data are transcribed and annotated in ELAN (Wittenburg et al. 2006). Transcriptions are organized around IPUs; i.e. we separate annotations when a speaker pauses for more than 180 ms. This threshold reflects the human threshold for the detection of acoustic silences, which lies between 120 and 200 ms (Heldner 2011, Walker and Trimboli 1982) and it facilitates comparability with Levinson and Torreira (2015) and Roberts et al. (2015). The onsets and offsets of the IPUs were determined through visual and auditory inspection of waveforms during transcription in ELAN’s annotation mode. So far, we have been using ELAN mainly in annotation mode, after trying out segmentation mode additionally in the beginning phase of the corpus construction. We found that the annotation mode provides the most flexibility to create annotations, add their content, and align them with the displayed waveform of the audio signal. ELAN’s audio resolution goes down to 1 ms, and the video time resolution is 3,300 ms in our case (30 fps of splitscreen video). We do make use of the different playback options such as slow motion, frame by frame, loop mode, etc. – especially for gesture annotation (Section 2.1.4). Furthermore, automatic pause measurement between annotations (IPUs) can be done in ELAN with a few clicks.

2.1.2 Types of transcription and error checks

Two types of transcriptions are used: conversations are transcribed orthographically (by native speakers of English) and using Jeffersonian conventions (e.g. Jefferson 2004). This latter transcription is referred to as CA transcription and records not only verbal content but also interactionally relevant details of sequencing (e.g. overlap, latching), temporal aspects (pauses, acceleration/deceleration), phonological aspects (e.g. intensity, pitch, stretching, truncation, voice quality), and laughter and free-standing (silent) gestures (see below). Departing from CA conventions, we express emphasis using the !…! convention (as italicization is not supported in ELAN) and also annotate what we refer to as the ‘scale’, or ‘Tonleiter’, phenomenon, i.e. a continuous upward or downward glissando in pitch. The conventions used are listed in Figure 1.

Figure 1

Key to CA-style transcription conventions.

The amount of detail in the CA transcriptions is considerable and, as a result, presents a serious source of potential error. There are two specific problem areas: (i) illegal characters and (ii) missing legal (special) characters. While illegal characters are easy to detect and remove/replace, missing legal characters require more work. Two types can be distinguished:

1. Type A error – missing mirror delimiters, i.e. delimiters that are unambiguously either opening or closing, e.g. [word] for overlapped speech, (word) for candidate hearing, (1.1) or (.) for pause, ((word)) for comment, etc.

2. Type B error – missing twin delimiters, i.e. delimiters that can be used for both opening and closing annotations, e.g. °word° for quiet/hushed speech, !word! for emphasized speech, ≈word≈ for tremulous voice, etc.

To weed out Type A and B errors, automatic error checks were devised in R for each potential error type and each transcription convention, largely based on Regular Expression, to detect utterances with missing opening or closing character; once detected, the errors were manually corrected in the ELAN files.

2.1.3 Transcription of quotes

Quotation, also referred to as (free) direct speech (e.g. Labov 1972), constructed dialog (e.g. Tannen 1986), or re-enactement (e.g. Holt 2007), is a recurrent feature of conversational language. It is particularly frequent in conversational storytelling (e.g. Rühlemann 2013), where it often occurs at or around the story climax (e.g. Li 1986, Mayes 1990, Mathis and Yule 1994, Rühlemann 2013). In FreMIC, re-enactments are exhaustively transcribed as well as annotated on a separate tier to obtain exact temporal data for it.

At the time of writing, FreMIC contains 1,135 quotes. Consider extract (1): Speaker A’s utterance is transcribed in full, whereas the two quotes she uses are separated out on quote tiers (signified by the ‘-Q’ suffix to the ‘Speaker’ Id):

(1)

The attention paid to re-enactments suits a multimodal corpus well in that such re-enactments are known to attract heightened activation in the vocal and bodily channels by the speaker (e.g. Holt 2007, Blackwell et al. 2015; Stec et al. 2016; Soulaimani 2018) and a prime means by which speakers express stance and emotion (e.g. Stivers 2008, Rühlemann 2022).

2.1.4 Transcription of silent gestures

A first multimodal element in the ELAN transcription concerns what we call ‘silent gestures’, that is, gestures that are iconic, embedded in speech, but not co-speech and not depictions of non-depictive speech (cf. Hsu et al. 2021). Silent gestures are exhaustively recorded in both the orthographic and the CA transcriptions. While in the former, the description is merely cursory, the description in the latter is comprehensive, indicating the articulating organ(s) (m: hand, f: face, t: torso, etc.) and providing a concise description of the gesture. At present, FreMIC contains 300 such silent gestures (virtually always by the current speaker). To illustrate, consider Speaker B’s silent gesture in extract (2). She is telling about an incident where she has been followed by a stranger in the night in a small French city (Metz):

(2)

Speaker B verbally describes the streets she was followed through as ‘dark’ and ‘narrow’. In the context of her story, dark and narrow streets are threatening because eyesight is poor, and others can hardly see (and possibly help) you and you cannot easily escape an attacker. The silent gesture – a hand movement whose key component is a zigzagging of both hands shown in Figure 2 – adds another quality to the streets not referred to on the verbal plane, a quality that could be glossed as ‘labyrinthine’. The silent gesture thus underscores how dangerous the situation was.

Figure 2

Three stills from Speaker B’s silent zigzagging gesture; frame grabs extracted from participant A’s eye tracking video for illustration (i.e. not automatically retrievable from the corpus).

2.1.5 Transcription of gestures in storytelling interaction

One layer of transcription and annotation in ELAN that is at present not yet available at a large scale but currently being extended concerns gestures in storytelling interaction. So far only a subset of storytellings^[2] have been fully examined for gestures performed by the storyteller.

Gestures are manually annotated in ELAN on three tiers: gestures as such, gesture phases based on Kendon’s (2004) gesture phase model, and the gesture expressivity index, which captures gesture dynamics. Both the phase model and (a prior version of) the Gesture Expressivity Index are explained in detail in Rühlemann (2022). Here, a mere sketch is attempted.

The gesture phase model distinguishes five phases: preparation, i.e. the departure of the hand(s) from a rest, or home, position; the pre-stroke-hold; the stroke, i.e. the “phase of the excursion in which the movement dynamics of ‘effort’ and ‘shape’ are manifested with greatest clarity” (Kendon 2004, 112); the (optional) hold, i.e. the “phase in which the articulator is sustained in the position at which it arrived at the end of the stroke” (Kendon 2004, 112); and the recovery, i.e. the movement from the stroke (and optional hold) back to the home position. Stroke and hold together form the nucleus, which is that part of the gesture “that carries the expression or the meaning of the gesture” (Kendon 2004, 112). The only phase that must be present in a gesture annotation is the stroke phase; all other phases are optional (cf. the Coding Manual for Gestures and Gestures Phases in Appendix 2).

The Gesture Expressivity Index comprises altogether seven variables; parameters 1–4 are manually annotated on what we call the index tier; parameters 5–7 are extracted from the gesture and gesture phase tiers (cf. Coding Manual for Gesture Expressivity Index in Appendix 3, which also gives a rationale for the coding parameters used):

1. Gesture viewpoint (gesture viewpoint is the character’s viewpoint rather than the observer’s viewpoint)

2. Gesture silence (gesture is a silent gesture without co-occurring speech)

3. Gesture size (gesture radius is sizable)

4. Gesture force (gesture execution requires muscular effort)

5. Gesture hold (gesture contains a hold phase)

6. Gesture articulation (gesture is articulated in concert with other bodily activation)

7. Gesture duration (gesture nucleus is longer than story average)

Gestures are annotated in FreMIC with a specific research focus in mind, namely, the contribution of gestures (and gesture expressivity) by the storyteller to emotional resonance in the story recipients. As will be explained later (in Section 2.2.2.2), in a subset of the FreMIC recordings, the participants wore wrist watches measuring, inter alia, electrodermal activity (EDA), a psychophysiological proxy for emotion arousal. Given this research focus (and the inherent complexity of gesture annotation as well as the inevitable necessity of manual annotation), the aim in FreMIC is not to achieve full-scale annotation for all storytellings in the whole corpus. Rather, the annotation is restricted to those files where EDA data are available.

To illustrate, in a pilot study (Rühlemann 2022), the Gesture Expressivity Index was implemented for two triadic storytellings that were comparable on a number of counts but starkly different on others, the main selection criteria being the fact that both stories were stories from the sadness/distress spectrum of emotions and the fact that the storytellers came from the opposite ends of the gesticulation spectrum: the storyteller in story ‘Toilet woman’ using their hands a lot and the storyteller in story ‘Sad story’ using them scarcely (for a complete description of the selection criteria, see Rühlemann (2022, 3–4)). The metric used to estimate how gesture expressivity develops across a storytelling was the slope of a linear regression computed for the index values in the storytelling, with positive slopes indexing intensifying expressivity and negative slopes indexing weakening expressivity.^[3]

It was shown that the unequal use of gesture dynamics by the two storytellers was differentially correlated with the storytellings’ progression from background to climax: while the gestures by the storyteller in ‘Toilet woman’ gained in expressivity, the gestures used by the storyteller in ‘Sad story’ increasingly lost in expressivity (Figure 3).

Figure 3

Gesture Expressivity in ‘Toilet woman’ vs ‘Sad story’. Dashed line: mean gesture expressivity index. Solid pink line: regression (‘trend’) line computed from story beginning to climax onset. Red rectangles: duration of climax. X-axis: timestamps of gestures in the storytellings.

Gesture expressivity may impact on whether, or to what degree, the ultimate goal in storytelling – emotional resonance between storyteller and story recipient(s) (e.g. Labov 1972, Stivers 2008, Peräkylä 2015) – is achieved (see Section 2.2.2.1, and Rühlemann 2022 for more detail).

2.1.6 Annotation of Question–Answer (Q&A) sequences

FreMIC is comprehensively annotated for Q&A sequences. Q&A sequences are of particular interest to research on turn-taking. This is because questions are “among the most frequent high-level action type in all languages” (Levinson 2013, 112) and, unlike other turn types (such as, for example, assessments), an (information-seeking) question puts maximal pressure on the interlocutor(s) to produce an answer that provides the sought information (Stivers and Rossano 2010, 29). If the information is not provided, a negative sanction may be issued (Stivers 2013, 204). Questions thus count among the most strongly next-action and turn-transition mobilizing actions; in Stivers (2010), for example, 93% of all questions were indeed followed by turn transition.

Questions are a functionally highly diverse class of turns/social action. They can be used not only for information requests but also, inter alia, repair initiations, confirmations, and assessments (Stivers 2010), and not all question types exert the same kind or amount of response relevance. For example, most questions that become questions only by virtue of an appended question tag serve not to fill a knowledge gap on the part of the questioner but to elicit agreement with the questioner. Other question types are per se restricted in terms of who is selected as the answerer (e.g. a recipient in multi-party interaction asking a repair question is highly likely to select as answerer the current speaker whose talk contained the repairable).

As the focus of interest in the FreMIC annotation of Q&A sequences is on information-seeking questions with high response relevance, we target four types of questions: wh-questions, polar questions, declarative questions, and (multi-clausal) or-questions, as illustrated in (3):

(3)

Identifying such Q&A sequences is by no means a trivial task. There are questions that remain unanswered, questions that may involve a wh-pronoun but do not seek information but rather affirmation of the stance displayed in the question (as in rhetorical questions), questions that get responded to but not in a type-fitted manner by the selected next-speaker but by a third, non-selected party inserting some (intrusive) talk that does not provide the sought information, to name only a few types of unorthodox question – ‘response’ sequences. Another complicating factor is the occurrence of the question in turbulent turn-taking, for example due to multiple overlap, which makes identification of question and particularly answer difficult. Q&A sequences were therefore annotated by both authors and rigorously filtered. The sequences were also annotated for a large number of additional factors, including the following:

1. Q_by: Which participant asks the question?

2. Answ_by: Which participant gives the answer?

3. Q_dur: How long is the question (and, respectively, the answer) in total?

4. Last_G_to: Which participant does the questioner last gaze at?

5. Last_G_to_Answ: Is the last gaze to the answerer or not?

6. Last_G_dur: How long is that last gaze?

7. Preselected: Is the answerer preselected sequentially or epistemically?

While variables 1–6 are straightforward, variable 7, Preselected, requires a rigorous case-by-case analysis, often extending deep into the sequential contexts prior to the question. Recipients can be preselected sequentially or, for example, due to the ‘last-as-next’ bias observed by Sacks et al. (1974) or due to ‘tacit addressing’ (Lerner 2003), where, for example, an epistemic asymmetry holds between recipients that effectively limits eligible recipients to the single recipient who is ‘in the know’. Detecting such constraints on next-speaker selection is a key for analyses of particular next-speaker selection methods, such as gaze (cf., for example Lerner 2003): if the recipient that is last looked at during the current speaker’s turn and speaks next is, for example, the only one eligible to respond due to some epistemic advantage, it is impossible to determine which method effected the selection – it could be gaze or tacit addressing or both. We return to this issue in Section 4.2, where we report on a FreMIC-based study involving gaze and other selection meachnisms.

2.1.7 Descriptive statistics for transcription and annotation layers in FreMIC-so-far

To summarize, Table 2 presents the basic statistics of the transcriptions and annotations at the time of writing.

The data in Table 2 seem to suggest that FreMIC is a very small corpus; its word count, for example, is barely above 200,000 words. It should not be forgotten, however, that FreMIC is still under construction, i.e. new recordings will be added while not even half of all existing recordings have yet been transcribed and integrated into the corpus. Moreover, the CA transcriptions are of a granularity not found in any other corpus we are aware of. Finally, the transcriptions and manual annotations are just half the story; the other half are the automatically generated multimodal and psychophysiological observations, which already at this early stage count in the (hundreds of) thousands (Section 2.2).

2.1.8 From ELAN to a corpus architecture

Given our aim to construct a corpus, we cannot yet be satisfied with the large and detailed ELAN files elaborated so far. The files need to be exported and combined into a framework that is capable of holding the information contained in them in ways that are conducive to corpus linguistic methods.

While ELAN offers a large number of export options, we found only one of them to suit our needs, albeit in a form that requires substantial post-processing. We export transcriptions into R using ELAN’s “Traditional Transcript Text …” option and use Regular Expression to extract what are distinct variables into columns of a data frame. Distinct variables at this point comprise line number, speaker Ids, utterances (which need not be only verbal but can be of any modality, cf. Kendon 2004), timestamp, and file Id. Further, based on timestamps, we compute for each utterance its starttime, endtime, and duration, as shown in Figure 4.

Figure 4

Screenshot of FreMIC data frame with temporal data.

Considering that interaction plays out in time, these detailed temporal data are essential for most interactional-linguistic/multimodal analyses; see, for example Stivers et al. (2009) on turn transition times, within-turn speech rate (Roberts et al. 2015), timing of eye blinks used as addressee feedback (Hömke et al. 2017), gesture-speech dissociation (Ter Bekke et al. 2020), pauses, gaps, and overlap (Heldner and Edlund 2010), delay of dispreferreds (Kendrick and Torreira 2014), to name only a few.

2.1.9 PoS-tagging

A defining feature of many traditional corpora is that speech is part-of-speech tagged. That is, based on an automatic analysis that considers each word’s usage in its immediate co-text, the words are assigned their (most likely) part-of-speech tag, thus making possible, for example, distinctions between ‘like’ used as a verb, noun, preposition, etc. For FreMIC, we submitted our orthographic transcriptions (after the replacement of unclear speech, silent gestures, comments, and the like with placeholder ‘NA’) to the CLAWS web tagger (Garside and Smith 1997) and its c7 tag set (http://ucrel-api.lancaster.ac.uk/claws/free.html), which has an accuracy rate of 98.5% (Leech et al. 1994). The resulting PoS transcriptions were then added to the data frame in R. One of the advantages of PoS-tagging speech is that it allows the computation of turn size based on the number of grammatical words; i.e. even in contracted forms, the underlying grammatical words are recognized, tagged, and counted separately, for example the phrase ‘I’m gonna’ is tagged ‘I_PPIS1 ‘m_VBM gon_VVGK na_TO’, resulting in four words rather than two words. The word count for each Utterance is added as the new column ‘N_c7’ in the data frame (cf. Figure 5).

Figure 5

Screenshot of FreMIC data frame with PoS-tagging (in column ‘c7’) and count of grammatical words (in column ‘N_c7’).

2.2.1 Multimodal data – Gaze direction

All participants to the video recordings wear Ergoneers eyetrackers. Eye tracking traces a participant’s foveal vision, which is indicative of gaze direction (e.g. Auer passim), which is interactionally key, for example in turn-taking (e.g. Auer 2021) and re-enactments (e.g. Pfeiffer and Weiss 2022).

While the eye tracking technology has been around for quite some time, eye tracking has to date predominantly been used in qualitative studies examining sequences of interaction manually.

Eye tracking data come in two separate streams: as continuous measurements of the X/Y coordinates of a participant’s foveal vision, produced on average every 12 ms, and as what is called area-of-interest (AOI) data, produced whenever a gaze falls into that particular area. While the X/Y coordinate data are at present not (yet) accessible in the FreMIC data frame, the AOI data are comprehensively integrated into the corpus architecture.

The AOI technology relies not only on the participants’ eyetrackers and the foveal vision detected by them but also on two more components: on polygons the researchers inscribe into the eye tracking videos around the AOIs – in the case of FreMIC, on the co-participants’ faces – and on QR code markers applied to the participants’ clothing (as shown in Figure 6). Each polygon is assigned to one or more QR markers. The role of the QR markers is to prevent the AOI polygons from shifting when the eyetracked person shifts their head – as a result, the markers fixate the polygon on the face. Moreover, the markers form their own coordinate system so that any gaze into an AOI polygon is recorded in the marker’s coordinate system. The relation between QR code marker and AOI thus remains stable and independent of head movements of the eyetracked person.

Figure 6

Still from participant A’s eye tracking video (File F01) showing the interplay of QR code markers, AOI polygons, and foveal vision detection (shown in the crosshair).

In Figure 6, for example Speaker B’s and Speaker C’s faces were defined as AOIs by polygons delineating their heads (plus some surrounding space). Whenever the gaze of the eyetracked participant (Speaker A in this case) falls into the polygon for a predefined time threshold (as shown by the crosshair, see also Appendix), that gaze and its duration are recorded. It is important to acknowledge that the AOI technology is not bullet proof: it fails if the QR code is illegible because, for example subjects have removed their clothing with the QR code or the codes are covered by hair strains, scarfs, the participant’s own gesturing or extreme torso movements, etc.^[4]

AOI gazes are comprehensively integrated into the FreMIC corpus architecture: for each participant, there are two columns recording AOI data: one indicating the sequence of AOI targets looked at and another indicating the durations of these gazes. Consider Figure 7, which, for illustrative purposes, shows just the Speaker and Utterance columns as well as the four AOI columns from a dyadic interaction.

Figure 7

Screenshot of FreMIC data frame with AOI-related columns.

In Figure 7, for example Speaker B requests Speaker A to =tell me everything about your <quaranti:ne>=. The ‘*’ symbol in column ‘B_aoi’ indicates that during her utterance, she does not fixate A. Speaker A, by contrast, does look at B during the utterance: the string ‘*B*B*B’ in column ‘A_aoi’ contains six gaze data points: the ‘B’ values indicate fixed gaze on B, whereas the ‘*’ values indicate averted gaze. In column ‘B_aoi_dur’, we see that B’s averted gaze (‘*’) during the request had a duration of 2,716 ms (which is the duration of the whole utterance); in column ‘A_aoi_dur’, the durations for the six gaze values are given; they sum up to 2,716 ms, the duration of the utterance.

The AOI technology for the detection of areas of special interest is fully automatic. That is, notwithstanding its present shortcomings, the promise to gaze, and indeed multimodal, researchers is substantial: it opens up the possibility to study gaze behaviour and its interaction with other relevant multimodal semiotics at a large scale, thus facilitating serious quantification and statistical analysis.

2.2.2 Psychophysiological data

The psychophysiological data fall into two categories: pupillometric and EDA data.

2.2.2.2 EDA data

At the time of writing, participants in 11 video recordings (totalling c. 10 h) wear Empatica wrist watches. These devices record a large number of psychophysiological observations including, most notably, EDA^[5] (or GSR). EDA reflects changes in sweat gland activity and skin conductance. Sweating on and near the palms is involved in “emotion-evoked sweating” (Dawson et al. 2000, 202) rather than “physical activity or temperature” (Bailey 2017, 3). EDA is therefore a reliable indicator of emotional arousal (Peräkylä et al. 2015, Scherer 2005), that is, of intensifying excitation of the sympathetic nervous system associated with emotion (Dael et al. 2013, 644, Peräkylä et al. 2015, 302): emotional excitement leads to increases in EDA amplitude.

In the aforementioned pilot study (Rühlemann 2022), EDA responses have been examined in the context of two triadic storytellings with a focus on the storyteller’s deployment of multimodal resources, including re-enactments, prosody (pitch and intensity), gaze (gaze direction and gaze movement), and, as noted, gesture expressivity. It was found that the storytellers differ substantially in their multimodal investment in the storytelling, with the storyteller in the story called ‘Toilet woman’ using re-enactments, prosody, gaze, and gestures in more expressive ways than the storyteller in the story called ‘Sad story’. Figure 8 shows the EDA responses for storytellers and story recipients during the two storytellings. While the results are mixed to a degree (e.g. although the storytelling in ‘Toilet woman’ is more expressive on all counts, recipient C’s EDA amplitude shows no signs of arousal), EDA responses on the whole are stronger for ‘Toilet woman’ than for ‘Sad story’. This is congruent with the Multimodal Crescendo Hypothesis (Rühlemann 2022), according to which greater multimodal effort by the storyteller predicts greater emotional resonance in story recipients – a hypothesis to be tested against larger and more diverse data in future research.

Figure 8

Scatter plots of EDA responses by participants in storytelling ‘Toilet woman’ (left panel) and storytelling ‘Sad story’ (right panel).

2.2.3 Synchronizing ELAN transcription, multimodal, and psychophysiological data

At the point where all the different data streams are available, the corpus construction faces a major challenge: the transcriptions made in ELAN, the EDA data provided by the Empatica wrist watches, the eyetracking data on gaze direction, and pupil size provided by the Ergoneers eyetrackers are independent and heterogeneous data streams of different origin. Moreover, the ELAN data on the one hand, and the EDA and pupil observations on the other hand, play out on very different time scales: while IPU ‘observations’ transcribed in ELAN have a mean duration of 1,700 ms, EDA and pupil observations are produced every 16 ms! The challenge then is to synchronize the various data in a single data frame structure that will hold all the information in a way that benefits corpus research.

Although the data stream are heterogeneous in kind and play out on different time scales, they have one element in common: timestamps. Based on this information, ELAN, EDA, gaze direction, and pupil data can be joined and synchronized in large data frames in R (needless to say that due to the heterogeneity of data and time scales, that synchronization involves large amounts of coding).

A critical decision in the synchronization task is which data stream should be prioritized over the other; that is, which ‘variable’ is promoted to the reference variable to which the others are aligned. The decision in FreMIC is that the ELAN transcriptions (stored in column ‘Utterance’) serve as the reference category to which all multimodal data are synchronized.

As mentioned earlier, IPUs, which represent the bulk of ELAN transcriptions, are slow-paced observations, with typical rhythms between 1 and 2 s, whereas the rhythm of most multimodal observations is fast-paced, running up large numbers of observations during one and the same IPU. Moreover, there is the ‘binding problem’ noted by Holler and Levinson (2019) for multimodal data: i.e. the speech (or any other interactionally relevant behaviour) captured in the IPU and the multimodal data are often offset in time rather than neatly aligned.^[6] For example, we may address a question to a co-participant and select that participant as the next speaker by gazing at them but will keep our gaze on them during the ensuing turn transition and at least during the initial part of the answer. What this means is that multiple multimodal data points need to be funnelled and tailored into the IPU’s single time slot.

For illustration, consider the IPU like I mean she’s not like in the (.) <!risk! age group> recorded in column ‘Utterance’ in the FreMIC data frame and shown in Figure 10.

The utterance like I mean she’s not like in the (.) <!risk! age group> starts at 13,100 ms and ends at 16,898 ms, as shown by the columns ‘Startttime_ms’ and ‘Endttime_ms’ at the top of Figure 10. The AOI data for this time span show that the speaker, Speaker A, first gazes at Speaker C. That gaze starts long before the utterance starts, namely, at 11,365 ms, and it ends at 14,287 ms, which is within the time span of the utterance. The following non-AOI gaze, shown as ‘NA’ in the figure, the next gaze to Speaker B, and the non-AOI gaze thereafter fall within the utterance’s time slot. The last gaze, however, which is directed to Speaker C, starts within the utterance’s time span, at 15,421 ms, but outlasts it, ending at 20,054 ms. In other words, while the two non-AOI gazes and the one gaze to Speaker B can be seamlessly funnelled into the time span of the utterance, the two gazes to Speaker C cannot – they need to be tailored to fit the utterance’s time span. Note how the result of that funnel-and-tailor operation is given in new columns in the FreMIC data frame (shown at the bottom of Figure 9): in column ‘A_aoi’, we find the string ‘C*B*C’, which represents the speaker’s shift from AOI C, to non-AOI (signified by *), to AOI B, to non-AOI, and to AOI C. In column A_aoi_dur, we find the string “1187,151,132,851,1477”, which contains the exact durations of the speaker’s gazes during the time span of the utterance.

Figure 9

Data streams feeding into FreMIC – ELAN: ELAN transcription tool; EDA: electrodermal activity observations provided by Empatica wrist watches; ET: eyetracking data provided by Ergoneers eyetrackers; PP: pupillometric observations provided by Ergoneers eyetrackers.

The funnel-and-tailor operation shown in Figure 10 for one utterance and that speaker’s gazes are done for all utterances and all speakers in each recording, and it is implemented not only for AOI data but also for the psychophysiological data. The end result is a data frame that is far too large to be shown in a print medium, containing almost 50 columns. It is also a data format that, to some observers, may appear ‘untidy’, as long sequences of character or numeric data are converted to strings and squeezed into single cells. However, that untidiness, if anything, is the price we are willing to pay for the orderliness we gain with respect to the reference variable ‘Utterance’: we know exactly what happens multimodally and psychophysiologically during any one utterance.

Figure 10

Illustration of the ‘binding problem’ in synchronizing ELAN transcription data with multimodal (in this case, AOI) data.

Large strings of data require special treatment when it comes to analysing the stringed data. Modern packages such as the ‘tidyverse’ super package in R allow for quick and easy unpacking of data strings across multiple columns. For example, using the functions ‘pivot_longer’ and ‘separate_rows’ we can easily fan out the AOI data for all three speakers during the aforementioned utterance like I mean she’s not like in the (.) <!risk! age group> to transform the densely packed cells with strings of multiple data points, which are spread out across multiple columns to individual data points neatly ordered in few columns, as shown in Figure 11.

Figure 11

Result of transformation of the AOI data for all three speakers during the utterance “like I mean she’s not like in the (.) <!risk! age group>” from wide to long format.

Now that the data are unpacked so that each gaze observation has its own row, and each variable has its own column – for example, all the AOI gazes are assembled in column ‘aoi’, all the durations of these gazes are collected in column ‘aoi_dur’ – we can address a large number of analytic questions and the transformation from an ‘untidy’ spread-out format to a ‘tidy’ gathered format of the multimodal data opens up novel ways of visualization. In Section 4, these visualizations are discussed, and one ongoing research project is briefly described. In Section 3, we describe yet another corpus component, the FreMIC Turns component.

4.1.1 Speech-Gaze transcripts

A novel visualization concerns multimodal transcription, specifically representations of gaze in transcriptions of talk-in-interaction. Various conventions have been offered to integrate gaze movements into transcripts of talk-in-interaction (e.g. Goodwin 1984, Rossano 2012, Mondada 2016, Rühlemann et al. 2019, Auer 2021, Laner 2022). While all prior methods rely on manually annotating gaze in transcripts, which require much extra work by the researcher and often produce hard-to-read results, the method presented here is both visually intuitive and computationally automated; that is, with the R code for it in place, the researcher just selects the turn or sequence of turns for which a multimodal transcript is needed and runs the code. For illustration, let’s stick with the turn discussed earlier and visualize not only the speaker’s gazes during that turn but also everybody’s gazes during the sequence of which the turn is an element. The multimodal speech-gaze transcript of the sequence is shown in Figure 13.

Figure 13

Speech-gaze transcript of file F01 [Lines 2:9].

The turn discussed earlier represents an extension of a Q&A sequence, with A inquiring about C’s mother’s age: “how old’s your mom¿”, after which a long gap (0.855 s) ensues before Speaker C starts the answer somewhat hesitantly with ‘eh’ before she gives her age with sixty:::-one. Speaker A’s turn “like I mean she’s not like in the (.) <!risk! age group>” extends the sequence in that it makes the implications of C’s mother’s age explicit: at age 61, she’s not in the age group at (highest) risk to attract the Corona virus.

As regards gaze, the transcript shows that Speaker B, gazes to whom are highlighted in green, is not gazed at by either Speaker A or Speaker C during the Q–A sequence; only during Speaker A’s sequence-extending turn is she briefly gaze-addressed by Speaker A. The exclusion of Speaker B from the gazes by A and C during the Q-A sequence is arguably due to the fact that the question is addressed by A only to C: it is her mother’s age she wants to know. So Speaker A’s gaze during the question serves to select C, rather than B, to give the answer (on the issue of selection, see the work-in-progress report in Section 4.2). Interestingly, note that Speaker B retracts her gaze from A during the gap following the question and during C’s answer, as if to signal her not being involved in the completion of the Q–A sequence. Her gaze returns to A during A’s acknowledgment token “yeah … yeah yeah” and is also steady on her during the sequence-extending turn.

All of these (and possibly more) observations are facilitated by the multimodal Speech-Gaze transcript, which shows exactly when gazes start and end and who gazes at whom when. The Speech-Gaze transcript thus represents a viable and cost-effective alternative to traditional multimodal transcripts.

4.1.2 Gaze activity plots

Another novel visualization technique afforded by the unpacking of the multimodal data strings is the Gaze Activity plot. This plot type shows the participants’ gaze behaviour for long stretches of transcription. In Figure 14, for example, we see all AOI gazes for all three participants in the first 10 min of recording F01. Note that the extended Q–A sequence discussed in Section 4.1.1 and shown in the Speech-Gaze transcript in Figure 13 is highlighted in the black rectangle in the left upper corner of Figure 14.

Figure 14

Gaze activity plot F01 (minutes 0:10).

Such gaze activity plots give the researcher valuable insights into gaze patterns and can help single out stretches of particular interest. Notice, for example how in minutes 2–7 (indicated in the left margin) both Speakers B and C keep their gaze fixated on A with few and short interruptions only. The pattern of sustained gaze by Speakers B and C is particularly strong in minute 3. What happens in the interaction during that time?

The talk-in-interaction during most of 3 min of file F01 is shown in extract (6) from the Turns component:

(6)

Clearly, Speaker A is engaged in storytelling, specifically a trouble telling about her being mistaken as a man and consequently being disallowed from using a public bathroom for women. Speaker B’s and C’s produce empathic responses ranging from incredulous repetitions of the acts of denial by the toilet ladies (“she said ∼no you’re not?∼”) said in unison by both speakers, to sarcastic advice to the storyteller to next time “show them your boobs” as well as extended laughter. Clearly, given the engaging stories and the story recipients’ empathizing with them, it is not surprising that the recipient’s gaze should be largely fixated on the storyteller. This line of inquiry can be pursued further with Gaze Path plots, described in the next section.

4.1.3 Gaze path plots

As noted, eye tracking does not only produce AOI data (whether the gazes hit the AOIs or not) but also continuously records the X/Y coordinates of the foveal vision. This type of eye tracking data is quite unlike the AOI data: while the latter produces discrete data (either AOI or not AOI), X/Y coordinate gaze data are continuous, produced on average every 12 ms. While the AOI data then allow us to see when and how long a participant looked at particular areas, the X/Y data allow us to comprehensively track the gaze movements over time.

For example, while it is well known that non-current speakers gaze at the current speaker more than the other way around (e.g. Kendon 1967, Goodwin 1980), little seems to be known about how this asymmetry plays out in storytelling. To illustrate, consider the Gaze Path plots in Figure 15; they depict not only the storyteller’s but also the recipients’ gaze trajectories.

Figure 15

Gaze path plots for storyteller and recipients in the story ‘Sad story’ in F16; X/Y scales aligned for all panels per row; lower row: close-ups on narrow X/Y range; curved lines depict gaze movement from one X/Y coordinate datum to the next; curves are colour coded for temporal progression ranging from green for story-early, to red for story-late gaze points.

First consider the upper row in Figure 15; it shows all X/Y values recorded for all three participants. Note that for comparison, the X/Y coordinate scales of the three panels have been aligned. The result is striking: while the storyteller’s gaze fans out in all directions of their visual space, the recipients’ gazes can barely be seen as they are steadfastly directed to a very small section of their visual field. It almost appears as if they did not move their gaze at all.

The lower row in Figure 15 shows the gaze paths in close-up resolution. Here, then we see that recipients’ gazes are far from being arrested but, compared to the storyteller’s gaze movements that reach far beyond the plotting margins, they are restricted to a very small radius.

Which ‘target’ are the two recipients fixating their gaze on? We will hypothesize that, in storytelling, they will most likely look at the storyteller. But at this point and based merely on the X/Y data, this is merely a hypothesis. To test it, we can synchronize the X/Y data with the AOI data. This synchronization allows us to not only see all gaze observations, see how they travel across the participant’s field of vision and obtain a sense of how the gaze paths relate to the storytelling’s progression through time; it also allows us to discover which X/Y gazes are at the same time gazes to AOIs, that is, to co-participants.

As shown in Figure 16, all X/Y gazes by Recipient A are at the same time AOI gazes – that is, the recipient’s gaze is completely fixated on the storyteller. Recipient B, by contrast, does allow his gaze to wander off the storyteller’s face, particularly towards the end of the storytelling. During the main bulk of the telling, however, his gaze is fixated on the storyteller too.^[7]

Figure 16

Integrated X/Y gaze path and AOI plots of recipients’ gazes in ‘Sad story’; curved lines depict gaze movement from one X/Y coordinate datum to the next; curves are colour-coded for temporal progression (green for story-early gazes and red for story-late gazes); blue crosshairs depict AOI gazes (Note: to reduce the effect of over-plotting, the blue crosshairs are plotted with large colour transparency, which may make the detection of isolated crosshairs difficult.).

To judge from the example visualized in Figures 15 and 16 (and many other examples), it appears that the recipients’ gaze behaviour in storytellings is firmly synchronized and aligned: their gazes perform the same gesture – namely, focusing on the storyteller. Storytelling then may provide a means for the storyteller of galvanizing gaze and, hence, attention and a site for story recipients to align and synchronize their behaviour towards the storyteller. In storytelling as a shared activity, the ‘eye-to-eye ecological huddle’ (Goffman 1964, 95) is pulled together very closely.