Assessment of noticing of classroom disruptions: a multi-methods approach

1. Introduction

Teaching as an interactive and complex profession needs spontaneous and flexible decisions to handle the requirements of individual situations (Doyle, 1985). In the classroom, features that need attention are, for example, potential disturbances (Grub et al., 2020), student hand-raising (Kosel et al., 2023a), or uninterested or struggling students (Seidel et al., 2021). Only when teachers are able to place a conscious focus on the mentioned aspects, they can react adequately (e.g., Grub et al., in press). Teachers’ noticing—also often named as professional vision (PV)—as an important prerequisite for their professional behavior can be understood as a situation-specific skill to perceive and interpret the demands of the situation (e.g., Blömeke et al., 2015). In particular, the basic perceptual process of attending to relevant elements in the classroom and ignoring the irrelevant is a widely investigated construct in research on teacher professionalization (König et al., 2022), although there are different conceptualizations and definitions of the terms (Stahnke et al., 2016; König et al., 2022). Despite the various definitions, it has been shown that expert teachers with an elaborated knowledge structure are better able to attend to the specific relevant features of a situation than novices (Lachner et al., 2016; Stahnke et al., 2016; Wolff et al., 2016, 2021; Boshuizen et al., 2020; Grub et al., 2022a). Due to technical development in recent years, a direct assessment of the noticing process is possible with process-based measurements such as eye-tracking and they are increasingly used in research of teachers’ noticing (Grub et al., 2020, in press; Stahnke and Blömeke, 2021a).

A basis for eye-tracking research is the eye–mind assumption, which means that conclusions about cognitive processes can be drawn from fixations in a given moment (Just and Carpenter, 1980). This hypothesis is debatable, because it overlooks hidden attention or parafoveal perception, and it is unclear from eye-tracking data alone why a person fixates on a given stimulus (Posner, 1980; Anderson et al., 2004; Holmqvist et al., 2011; Jarodzka et al., 2017; Orquin and Holmqvist, 2018; Schindler and Lilienthal, 2019). This leads to the necessity of triangulation using additional measurement methods (Orquin and Holmqvist, 2018; Gegenfurtner et al., 2023; Grub et al., in press). For this purpose, in research on teachers’ noticing, verbalization methods such as think-aloud protocols are often used (Wolff et al., 2016; Stahnke and Blömeke, 2021b; Grub et al., 2022a). However, in most cases, the different data sources are not cross-linked but rather are analyzed separately from each other. A direct link between eye-tracking measurements and verbal data for assessing attentional processes has been implemented only in a few studies (Yamamoto and Imai-Matsumura, 2013; Wyss et al., 2020).

A look toward other research fields, where focused attention and quick reaction are based on the perception of visual stimuli in dynamic scenarios, is worth considering to identify further methods that assess attentional processes. For example, in traffic psychology, a widespread method is the keystroke task to assess attentional processes in hazard perception scenarios (e.g., Malone and Brünken, 2020). Such reaction-based tasks are already used in research on teachers’ noticing, but the generated timestamps are used to select material for the analyses (e.g., van den Bogert et al., 2014) or as cues for retrospective think-aloud protocols (RTA; Stahnke and Blömeke, 2021b), and not for analyzing attentional processes itself.

In the present study, we aimed to investigate where the participants had their particular attention during the observation process. For this, we use reaction-based measurement (keystroke) and stimulated RTA to measure the noticing process and validated the data with different parameters of gaze behavior as well as with expertise. More precisely, we aimed to investigate, on one hand, whether the participants who noticed critical incidents (CIs) in the scenarios differed in their attentional gaze behavior (fixation count, fixation duration, and revisits). On the other hand, we aimed to determine whether experts showed higher accuracy in noticing CIs with keystroke measurement and RTA when compared to novices.

1.1. Teachers’ noticing

It is indisputable that the perception of essential cues within a profession is a key component of expertise, especially for teachers (Blömeke et al., 2015; Stahnke et al., 2016). Nevertheless, different authors use different conceptualizations and names for the construct of the perceptual processes (Stahnke et al., 2016; König et al., 2022). The concept of Professional Vision was originally introduced as a holistic concept from a sociocultural perspective by Goodwin (1994). It describes a specialized way of seeing and understanding meaningful events in a specific professional context. The adaptation of the concept in the context of teaching entailed a shift to a cognitive psychological perspective and was first used by Sherin (2001) (see also Sherin and van Es, 2009; Seidel and Stürmer, 2014) with the intertwined subprocesses of selective attention (noticing) and knowledge-based reasoning (Weyers et al., 2023b). In later publications, Sherin and van Es subsumed the perception and reasoning process under the term noticing (van Es and Sherin, 2002; Sherin et al., 2011). Other authors added the decision-making process (Jacobs et al., 2010; Kaiser et al., 2015) or rather the concept of shaping (van Es and Sherin, 2021) to noticing; while some concepts only subsumed the perceptual process under this term (Star and Strickland, 2008; Seidel et al., 2021). The common feature of these analytic approaches (König et al., 2022) is the possibility of separate consideration of the subprocesses. An empirical example is the study from Seidel and Stürmer (2014), who referred on the term professional vision and the subprocesses of reasoning. The authors could show in their study that a three-factor model with the components description, interpretation, and prediction fits better than a one-factor model (see also the study of Weyers et al., 2023a). We follow the analytical approach and focus with the term noticing only on the attentional focus of teachers (see the narrower definition of Star and Strickland, 2008 and Seidel et al., 2021).

In all previously mentioned conceptualizations, expertise, and the accompanying professional knowledge, is a crucial basis for noticing (Blömeke et al., 2015; Stahnke et al., 2016; Grub et al., 2020, 2022a; Wolff et al., 2021). Even the perceptual focus on relevant cues presupposes knowledge of the relevance (or irrelevance) of signals for teaching and learning. The direction of attentional processes can be based on professional knowledge (top-down), but also on salient cues in the situation itself (bottum-up; see; for example; Wolff et al., 2021; Gegenfurtner et al., 2023). From expertise research, it is well known that experts with a professional knowledge base, that has well-structured and organized schemata, are better able to direct their attention to relevant cues than novices (Lachner et al., 2016; Stahnke et al., 2016; Wolff et al., 2016, 2021; Boshuizen et al., 2020; Grub et al., 2022a).

1.2. Measuring the noticing process

1.2.1. Eye-tracking

In the last nearly 15 years, however, process-based methodologies such as eye-tracking have been increasingly used in research on professional vision of teachers, and thus, have been able to provide further insight into the noticing process (Grub et al., 2020, in press). For example, eye-movement data reflect attention and shifts in attention (van Gog et al., 2009) and can be used to assess and analyze teachers’ allocation of attention in the classroom (Bucher and Schumacher, 2012; Beach and McConnel, 2018; Haataja et al., 2019; Grub et al., in press). According to the eye–mind assumption (Just and Carpenter, 1980), the cognitive processing of an object corresponds to the visual focus or rather the fixation on the object. A fixation is defined as “a period of time when the eye is relatively still” (oculomotor definition; Holmqvist et al., 2011, p. 377). Therefore, gaze-based indicators for attention, number of fixations, and fixation duration can be used (Holmqvist et al., 2011). Furthermore, analyzing revisits (areas of interests—AOIs—where people look back) allows conclusions to be made about the relevance of AOIs for participants (Wolff et al., 2016; Grub et al., 2020). Many studies in the context of professional vision in teaching lay their primary focus on contrasting novice and expert teachers (Wolff et al., 2017; Seidel et al., 2021; Shinoda et al., 2021; Stahnke and Blömeke, 2021a,b; Grub et al., 2022b), but not on the attentional focus itself. From these studies, it is known that expert teachers place a stronger focus on relevant aspects of the classroom (e.g., disruptions or hand raising-behavior). Yamamoto and Imai-Matsumura (2013) and Wyss et al. (2020) both directly focused on attentional processes, showing that participants who were aware of a CI fixated on it more often and with a longer duration (see Section 1.2.2 for further description of the studies).

Nevertheless, inferences about the cognitive processes or location of an individual’s attention made from eye-tracking measures alone can be imprecise and skewed because of parafoveal processes (Anderson et al., 2004) as well as hidden attention (Posner, 1980). Therefore, it is strongly recommended that gaze data should be complemented with other measurements to identify where attention is actually being directed (methodological triangulation; Denzin, 1989; Holmqvist et al., 2011; Orquin and Holmqvist, 2018; Grub et al., in press).

1.2.2. Post hoc verbalization and standardized tests

For assessment of the noticing process—that is, of what was seen—standardized instruments (for an overview see Weyers et al., 2023b), as well as RTA protocols (Wolff et al., 2015; Stahnke and Blömeke, 2021b; Grub et al., 2022a), are worth considering. Standardized test instruments are used in most cases after the actual stimulus materials, in the form of videos or pictures of teaching scenarios in combination with open- or closed-ended questions. Most of these tests differentiate between the dimensions of professional vision or noticing (attending/perception, reasoning, and decision-making; Weyers et al., 2023b), but do not represent the spontaneity of classroom situations (Stahnke and Blömeke, 2021a).

Likewise, the think-aloud method (Ericsson, 2018) in the context of professional vision uses videos or pictures as stimulus materials (Wolff et al., 2015; Stahnke and Blömeke, 2021b). The verbalizations of participants will give insight into their cognitive processes as they handle a task (e.g., the observation of a teaching situation). Although the concurrent collection of verbal data during the task is a more accurate and valid procedure (Ericsson, 2018), it is not suitable for complex tasks such as teaching, or for use while eye-tracking, because the think-aloud process can distort gaze behavior and the cognitive processes themself (van Gog et al., 2005; Prokop et al., 2020). Therefore, the collection of verbal data takes place after the observation of a scenario and is often supported with videos (with or without integrated eye-movements), which are used as cues to increase the validity of the verbal data and avoid oblivion or fabrication problems (van Gog et al., 2005; Prokop et al., 2020). With these stimulated RTA protocols, researchers have investigated attentional processes in teaching (with instructions such as “Could you tell me, what you have seen?”; Wyss et al., 2020; Grub et al., 2022a), as well as reasoning processes, whereby participants are prompted to report their thoughts about what they have seen and the importance of the stimuli (Wolff et al., 2015, 2016, 2017; Gegenfurtner et al., 2020a; McIntyre et al., 2021; Grub et al., 2022a).

Combining verbal methods with eye-tracking as measurement methods of the noticing process is a promising approach (Wolff et al., 2016; Beach and McConnel, 2018), and previous studies of teachers’ professional vision have used multiple data sources such as eye-tracking and (stimulated) RTA protocols (Wolff et al., 2016; Pouta et al., 2020; Stahnke and Blömeke, 2021b; Grub et al., 2022b). Nevertheless, in these studies, the data sources were analyzed separately from each other rather than being directly combined. In contrast, a direct combination of eye-tracking data and RTA was used by Muhonen et al. (2021, 2023) to investigate reasoning processes, as well as by Wyss et al. (2020) and Yamamoto and Imai-Matsumura (2013) to investigate attentional processes. Wyss et al. (2020) combined eye-tracking data and RTA for one short video clip to examine the association between the awareness of a CI and the gaze behavior of the participants (student teachers as novices and university teacher educators as experts). From the RTA protocols, they identified who mentioned the CIs and who did not and analyzed the gaze behavior of the two groups. First, they showed that expertise was connected to the identification of the CI (only experts were aware of them), and second, they showed that participants who were aware of the CI evidenced more and longer fixations on the incident. Nevertheless, the study had some drawbacks regarding generalizability: only six of the 56 participants were aware of the CI, and the authors used only one short video clip with one incident. Also using a comparable design with one short video, Yamamoto and Imai-Matsumura (2013) also found differences regarding fixation count (aware participants had more fixations) and fixation duration (a longer duration for aware participants). Since the cultural context of this latter study was situated in Japan, in East Asia, the transferability of the results to the Western context is potentially problematic.

1.2.3. Concurrent measures

In the field of traffic psychology, reaction-based measurement during the observation of traffic scenarios is a common method for investigating the skill of hazard perception (the ability to detect and evaluate road hazards quickly; Horswill and McKenna, 2004; Malone and Brünken, 2020). As the task is for the observers to press a key when they identify a relevant cue within the scenario, an inference to the corresponding cognitive attention process is possible (Chapman et al., 2002). This reaction-based method is a valid measurement in hazard perception research (Malone and Brünken, 2016; Moran et al., 2019): Experienced drivers react faster to CIs, and they identify more potential hazards correctly; this also applies to drivers with less accident liability (Malone and Brünken, 2020). Keystroke tasks have also been applied in research into the PV of teachers, but they are used as stimulus segments for RTA (e.g., Stahnke and Blömeke, 2021b) or for analyzing a selection of video segments in depth (van den Bogert et al., 2014), rather than for analyzing the focus of the teachers’ attention. Like eye-tracking, the attentional process can directly be covered with a keystroke during the task but it should be combined with the fixations from eye-tracking to identify the location of the visual attention (Crundall, 2016; Malone and Brünken, 2020).

1.3. Aims and hypotheses

In the present study, first, we aimed to investigate whether keystroke measurements and the RTA are appropriate ways of assessing the noticing process (awareness of a CI). The first step is methodological triangulation with the eye-tracking parameters fixation count, fixation duration, and revisits, which corresponds to the focus of attention and the relevance of the area of interest (AOI) for the participants. Second, we aimed to validate the measures using the expertise of our participants.

For the first research question, we wanted to identify whether participants who were aware of a relevant CI in the videos (assessed with keystroke as well as with RTA) showed differences in their gaze behavior. According to the eye–mind hypothesis, the focus of attention can be inferred from the location of fixations (Just and Carpenter, 1980; Holmqvist et al., 2011). Wyss et al. (2020) and Yamamoto and Imai-Matsumura (2013) both showed that participants who were aware of a CI had more fixations on the CI and a longer fixation duration. Wolff et al. (2016) indicated that participants had more revisits to AOIs of higher relevance, which referred to a conscious noticing of the AOI.

Hypothesis 1a: Participants who identify the CI correctly with a keystroke (they press a key and have a fixation on the CI within the relevant time period) will have more fixations, a longer fixation duration, and more revisits.

Hypothesis 1b: Participants who mention the CI in the stimulated RTA protocols will have more fixations, a longer fixation duration, and more revisits.

For the second research question, we wanted to examine whether expert teachers (experts) were more aware of the CI than prospective teachers (novices). Experts in the field of teaching are better able to direct their attention to relevant cues; that is, they have more fixations on these AOIs than novices (Lachner et al., 2016; Stahnke et al., 2016; Wolff et al., 2016, 2021; Boshuizen et al., 2020).

Hypothesis 2a. Experts will identify more CIs correctly with a keystroke (they press a key and have a fixation on the CI within the relevant time stamp) as novices.

Hypothesis 2b. Experts will identify more CIs in the stimulated RTA protocols than novices.

2. Materials and methods

2.1. Participants

A total of N = 71 student and experienced teachers participated in the study. Teachers (n = 37) were recruited via email, telephone, and newspaper from schools in South-Western Germany and had at least 5 years of teaching experience. Student teachers (n = 34) were recruited via email lists and flyers at Saarland University and had a maximum of 40 h of teaching experience during their regular practice. Data from 12 participants (six experts, six novices) were excluded from the analyses due to insufficient data quality (e.g., an outlier with more than three standard deviations for at least one of the relevant variables). In addition, only those participants for whom both gaze and verbal data were available were used in the analyses. Thus, 26 student teachers and 26 experienced teachers remained (N = 52); see Table 1 for the details of the sample.

2.2. Design and procedure

We re-analyzed data from an already published study (Grub et al., 2022b) for the purposes of the present study in support of the intensive use of scientifically generated empirical data (Machado, 2015; see Müller and Gold, 2023; Kosel et al., 2023a, for examples). Overall, the participants observed seven short videos of staged lessons in mathematics and informatics. For the re-analysis presented here, we focused on only two of the videos with, in total, five CIs regarding classroom management (see Section 2.3 for more details).

The study consisted of three sequential parts (see Figure 1). In Part 1, participants answered a knowledge test and demographic questions, which were presented online on Unipark. Part 2 of the experiment, an eye-tracking experiment in the laboratory, took place 10 days later, on average. Here, participants were quasi-randomly assigned to one of six video sequences, in which the videos were balanced regarding the order of presentation by Latin square to exclude sequence effects (the first of the seven videos was always the same and was provided as a tutorial video). The participants’ eye movements were recorded as they observed the videos. Meanwhile, the task was to identify CIs in each video via a keystroke. In Part 3, a stimulated RTA based on individual eye-tracking data was performed directly after watching all the videos, in which the eye movements presented as scan paths served as a cue for verbalization. During this stage, the noticing process was assessed by asking participants to explicitly specify the previously identified events in more detail. In total, the experiment lasted around two hours (Part 1: around 45 min, Part 2: around 20 min, Part 3: around 55 min).

2.3. Apparatus and videos

Eye movements were recorded under standardized environmental conditions using a stationary, binocular eye tracker (Tobii Pro Fusion, 120 Hz).¹ High-quality eye-tracking data were recorded for the participants (calibration accuracy: M = 0.55, SD = 0.18; calibration precision: M = 0.35, SD = 0.18).

The videos presented in the eye-tracking experiment were developed by “Toolbox Teacher Education” (“Toolbox Lehrerbildung”) from the Technical University of Munich (Lewalter et al., 2020). They were based on scripted lessons from the 10th and 11th grades in the advanced track at a German secondary school (“Gymnasium”), covered topics in mathematics and informatics, and had already been used in preceding studies (Grub et al., 2022a,b). Each video was presented once during eye-tracking. The videos, in general, were selected based on events related to classroom management, audiovisual quality, and authenticity of the situation by three independent raters. For the aim of the present study, we selected two videos for a deeper analysis. The criteria for the selection of the videos were that they should consist of at least two CIs to increase the complexity of the scenario (Wyss et al., 2020), and approximately half of the participants should be aware of the CIs to give enough power for the analyses. The selected videos showed two or three CIs of classroom disruptions (for details, see Table 2).

2.4. Dependent variables and data analysis

2.4.1. Gaze data

An AOI-based evaluation of the eye-tracking data (fixation count, fixation duration, and revisits)² was performed; that is, the parameters were aggregated for the predefined AOIs. For this purpose, polygonal dynamic AOIs were determined deductively, including those that corresponded to the CI, that is, individual students or groups of students. The parameters were calculated for every CI in both video vignettes; therefore, each parameter was present for each CI.

2.4.2. Verbal data (RTA)

Using three general questions, participants were instructed to think aloud while watching their scan path-cued videos again (“What did you see?” “Why did you notice it?” and “How is what you saw relevant to the lesson?”); however, only the answer to the first question was relevant for the present study. Participants could either verbalize a CI (1) or not mention it (0) in the RTA. Therefore, audio tracks from the RTA were transcribed verbatim. All the data were then read into MAXQDA 2022 and coded with respect to the descriptions of the CIs in the videos based on the master ratings (see Table 2). Coding was performed for each CI by two independent raters (the first author of this paper and an educational psychology student), with a satisfactory interrater reliability (Cohen’s kappa) between 0.73 and 0.88.

2.4.3. Concurrent measure (keystroke)

Participants were instructed to watch the videos carefully and received the following instruction regarding the keypress: “If you notice something relevant, press the keyboard.” Each CI was checked to identify whether a keystroke was recorded for it during the period in which the CI occurred. This keystroke defined whether the participant was aware of the CI and whether they had at least one fixation on that AOI. Therefore, each participant had either a keystroke score of 0 (not pressed) or 1 (pressed + fixation) for every CI.

3. Results

The analyses were calculated using SPSS IBM (version 29). An alpha level of 0.05 was used for the statistical tests. The analyses were conducted for each CI separately because the size and location of the AOIs could affect the values of the gaze parameters. Therefore, an aggregated analysis can lead to confounded results (Pappa et al., 2020; Holmqvist et al., 2022; Grub et al., in press).

3.1. Preliminary analyses and descriptive results

Because some previous studies have found effects of expertise on gaze behavior (see Grub et al., 2020, in press), we wanted to avoid possible confounding effects in our analyses. Therefore, we conducted MANOVAs for each CI with expertise as the between-subject variable and the eye-tracking parameters (fixation count, mean fixation duration, and revisits) as dependent variables. The means, standard deviations, and detailed results are displayed in Table 3. No statistically significant differences were found, so we removed expertise as a control variable in the subsequent analyses.

The descriptive results and the consistency regarding the accuracy of noticing the CIs using the two measurements (keystroke and RTA) are displayed above in Table 2. In Video A, around half of the participants reacted with a keystroke to the CI and mentioned it in the RTA. The consistency between the accuracy of the two measurements in Video A was moderate with Cohen’s ĸ = 0.46 for CI 1 and ĸ = 0.58 for CI 2. In Video B, we found larger differences between accuracy by keystroke and accuracy by RTA: around 33–75% of the participants reacted with a keystroke, but only 33 to 41% mentioned the CI in the RTA, which led to low consistency values (ĸ _{CI 3} = −0.11, ĸ _CI4 = 0.40, and ĸ _CI5 = 0.20).

3.2. Differences in gaze behavior regarding awareness

3.2.1. Concurrent measurement of noticing (keystroke)

A MANOVA was performed for each CI separately, with the concurrent measurement of noticing (keystroke yes vs. no) as the between-subject variable and the eye-tracking parameters (fixation count, mean fixation duration, and revisits) as dependent variables. The means, standard deviations, and detailed results can be found in Table 4. The omnibus test was significant for four CIs; therefore, the post hoc univariate ANOVAs were interpreted. Regarding fixation count, generally, participants who responded with the keystroke fixated on them more often. The difference was statistically significant for three CIs (CI 1, CI 2, and CI 4): F_{CI 1} (1,50) = 5.31, p = 0.01, η_p² = 0.096; F_{CI 2} (1,50) = 32.81, p <. 001, η_p² = 0.401; F_{CI 4} (1,50) = 11.82, p = 0.001, η_p² = 0.188. Regarding mean fixation duration, participants who noticed the CI fixated on it longer only in the case of CI 3: F_{CI 3} (1,50) = 4.60, p = 0.02, η_p² = 0.085. Regarding revisits, aware participants looked at the AOIs more often in CI 3 and CI 4: F_{CI 3} (1,50) = 4.62, p = 0.02, η_p² = 0.085; F_{CI 4} (1,50) = 4.61, p = 0.02, η_p² = 0.084.

3.2.2. Verbal measurement of noticing (RTA)

As before, a MANOVA was performed for each CI separately, with the retrospective measurement of noticing (CI mentioned in RTA yes vs. no) as the between-subject variable and the eye-tracking parameters (fixation count, mean fixation duration, and revisits) as dependent variables. The means, standard deviations, and detailed results can be found in Table 5.

The omnibus test was significant for three CIs. Therefore, the post hoc univariate ANOVAs were interpreted. Regarding fixation count, generally, participants who mentioned the CI in the RTA, fixated on them more often. The difference was statistically significant for three CIs (C1 2, CI 4, and CI 5): F_{CI 2} (1,48) = 32.34, p < 0.01, η_p² = 0.403; F_{CI 4} (1,49) = 22.01, p < 0.01, η_p² = 0.300; F_{CI 5} (1,49) = 5.05, p = 0.02, η_p² = 0.093. Regarding mean fixation duration, participants who noticed the CI fixated on it longer only in one case (CI 5): F _{CI 5} (1,49) = 5.10, p = 0.02, η_p² = 0.094. Regarding revisits, aware participants looked at the AOIs more often and statistically significant differences were found for two CIs (CI 4 and CI 5): F_{CI 4} (1,49) = 7.91, p = 0.003, η_p² = 0.139; F_{CI 5} (1,49) = 3.44; p = 0.02; η_p² = 0.066.

3.3. Differences in noticing regarding expertise

To validate the measurement keystroke and RTA for the assessment of noticing, we investigated whether experts responded to more CIs with a keystroke (Hypothesis 2a) and mentioned it more frequently in the RTA (Hypothesis 2b). A MANOVA was performed for each CI separately, with expertise (expert teachers vs. student teachers) as the between-subject variable and the two CI noticing variables (keystroke, RTA) as dependent variables. The means, standard deviations, and detailed results can be found in Table 6. No statistically significant differences were found for noticing the CIs, for keystroke or RTA. This means that experts and novices noticed the CIs equally.

4. Discussion

In the present study, we used a keystroke task and a stimulated RTA task to assess the noticing process of (novice) teachers and combine the two tasks with eye-tracking measurement (fixation count, fixation duration, and revisits as known parameters for awareness) as well as with expertise through methodological triangulation.

Regarding our first research question, we showed that aware participants who pressed the key (and had a fixation on the corresponding AOI) fixated on the AOI more often and had more revisits than non-aware participants. Comparable results could be found for the verbal task: participants who mentioned the CI in the RTA fixated on the AOI more and had also more revisits. Nevertheless, for both gaze parameters, this was statistically ensured only for three of the five CIs. A statistically significant difference in mean fixation duration was found for only one CI. Contrary to our hypothesis, the mean fixation duration for CI 5 was shorter for participants who mentioned the CI in the RTA. Therefore, Hypotheses 1a and 1b were only partially confirmed.

The findings that participants who were aware of the CI (reacted with a keystroke or mentioned it in the RTA) showed gaze behavior that corresponded to attention to a CI (more fixations, more revisits), is aligned with theoretical assumptions (Gegenfurtner et al., 2011; Holmqvist et al., 2011) and empirical evidence on the PV of teachers (Yamamoto and Imai-Matsumura, 2013; Wolff et al., 2016; Wyss et al., 2020). The partial lack of statistical findings can possibly be explained by the fine-grained analysis for each separate CI (Kaakinen, 2020). We decided against an aggregated analysis because the size and location of the AOIs can influence gaze parameters (Pappa et al., 2020; Holmqvist et al., 2022; Grub et al., in press) and therefore confound the results. Further research should take this into account.

Contrary to our hypotheses, there were no mean fixation duration differences between the aware and unaware participants. The studies of Wyss et al. (2020) and Yamamoto and Imai-Matsumura (2013), which showed longer fixation durations for the aware participants, had only one CI in their videos. In contrast, our videos had two or three CIs, so we cannot completely rule out an overlap. Especially in dynamic fields like teaching in a classroom, continuous monitoring is necessary to update what is going on. Therefore, participants may have used more scanning or monitoring gaze behavior, which is characterized by more and shorter fixations (Wolff et al., 2016; Grub et al., 2020; Huang et al., 2021), to detect everything that could be important. That participants show more monitoring gaze behavior, might be also an effect of the presentation mode: Minarikova et al. (2021), for example, in their study, compared the IN-mode (during class instruction with a mobile eye tracker) and the ON-mode (observation of the video of that instruction with a stationary eye tracker) and showed more monitoring gaze-behavior in the ON-mode. During the instruction (IN-mode), the teacher must monitor the classroom, but, for example, they must also interact with the students, whereas in the ON-mode, the task is mostly observation. To interpret the results of our study, the processes of monitoring the whole scene and focusing on single cues could have canceled each other out, what resulted in non-significant differences. This leads also to the question, whether fixation duration is an appropriate parameter for assessing the attentional focus because of its sensitivity for influencing factors like presentation mode or rather the task of the observation (see also 4.2).

Referring to our second research question, we found no differences between experts and novices in the accuracy of noticing, either in the keystroke or the RTA tasks. In many studies, experts have been better able to identify relevant cues of a dynamic scene (Lachner et al., 2016; Stahnke et al., 2016; Wolff et al., 2016, 2021; Boshuizen et al., 2020). Nevertheless, the expertise effect tends to occur in more complex conditions, when elaborated knowledge is needed. Stahnke and Blömeke (2021a), for example, showed expertise differences in perceiving potential disturbances only for partner work scenarios, and not for whole group scenes. Seidel et al. (2021) demonstrated an advantage in detecting the characteristics of students for expert teachers only for inconsistent student profiles; the authors also showed expertise differences only in the seatwork scene. These results suggest that the recognition of potentially disruptive situations may depend on the knowledge or expertise of the observer but also that the context matters. It may be easier to recognize relevant disruptive situations in whole group scenes—as were used here—than in other formats. This could be because salient visual impressions, such as movements, are easier to recognize via bottom-up perception processes, and the advantage of experts (namely, that they already have elaborate, flexible knowledge schemata that enable top-down perception) is not necessary for identifying disruptive situations. In addition, it may be easier for student teachers to recognize situations that they have probably actively experienced themselves as students (disruptive teaching situations) than to identify situations that represent, for example, cognitive activation, since this requires much more elaborated knowledge of educational science and didactics.

The heterogeneous results lead to questions concerning validity and the benefit of using keystroke and RTA to complement the measurement of noticing in the methodological triangulation of data. Despite only partially significant results regarding the eye-tracking parameters, we see the keystroke task as a promising addition to gaze-data in dynamic scenarios. By a keystroke, the researcher can infer that a participant has a conscious focus on a specific AOI at a specific moment, especially when the keystroke is combined with a fixation on that AOI (Crundall, 2016; Malone and Brünken, 2020). Nevertheless, we propose some important conditions for the validity of this inference: (1) The AOIs should be definable and not overlapping. If this is not possible, additional information should be gathered about the thoughts or visual focus of the participant (see also Section 4.2). (2) A given instruction about the task should be well considered due to its influence on the gaze behavior (Yarbus, 1967; DeAngelus and Pelz, 2009; Grub et al., 2022b; Martin et al., 2023) but also on reaction-based measurements such as the keystroke. In our task, the participants had the instruction, “If you notice something relevant, press the keyboard,” which ensured the focus of attention in any given moment of an AOI, but instructions to press the button when the participant identifies that a reaction is needed (Moran et al., 2019) would also be useful. Further research on this is necessary and important.

The verbal data gained through RTA can give important insight into the thought processes of the participants and are, therefore, a useful additional source of information about the noticing process (Yamamoto and Imai-Matsumura, 2013; Wyss et al., 2020). Although, we used the replay of participants’ gaze as a stimulus for the RTA to obtain higher validity data, a drawback of our design was the administration of the RTA after the observation of all the videos. We could not prevent oblivion or mixing up the videos by participants. Indeed, in the RTA protocols, we found, for example, several statements such as “Cannot remember why I pressed the button” or about the behavior of students visible in another video. In addition, an active reconstruction process by means of cues instead of reporting the thoughts during observation from memory cannot be avoided (cf. van Gog et al., 2005).

Additionally, in some RTA protocols, the verbal allocation to a specific student group was not clear enough to rate it as “mentioned” (for example, with statements such as “they are chattering,” where it was unclear which students were being referred to in the use of “they”). This is also a possible explanation for the lack of consistency between RTA and the keystroke in Video B. To prevent these effects, the timespan between observing videos and gaining verbal data should be as short as possible. Furthermore, instructions in the RTA should be very clear (for example, “Name or describe the students you focused on”; see Yamamoto and Imai-Matsumura, 2013). Alternatively, the relevant part of what is being verbalized could be clicked on again during the RTA, either via touchscreen monitor or mouse click (depending on the technical requirements). In follow-up studies, it may also be worth considering a modification of the classic RTA toward a somewhat more guided semi-structured interview with the investigator, so that they could make follow-up inquiries about ambiguous statements to reduce later ambiguities.

The finding that awareness based on RTA verbalizations was generally lower than awareness via keystroke (see Table 2) could be due to the method of recording, namely the verbalization itself. Novice teachers, for example, have difficulties applying their knowledge and reasoning about noticed events (Schäfer and Seidel, 2015). Thus, it could be that they had already noticed the events (which can be seen in the keystroke) but they interpreted and verbalized the situation differently compared to our master rating (for more information, see Grub et al., 2022a,b). The experts, on the other hand, might have had difficulties in verbalizing what they actually saw because their knowledge was often rather implicit and so-called tacit, and therefore, was not available for introspection (Sternberg and Horvath, 1999). Tacit knowledge is typically reflected in eye movements “but is not necessarily available for conscious thoughts” (Kaakinen, 2020, p. 172), and therefore is not verbalizable. These are conceivable reasons why the CIs were assigned awareness more often via keystroke than RTA. Further research is therefore necessary to derive, for example, causal relationships.

4.1. Strengths and limitations

Our research has some clear strengths, but also some weaknesses. We used a very well-balanced experimental design with a comparatively large sample size. What is positive in comparison to the studies of Wyss et al. (2020) and Yamamoto and Imai-Matsumura (2013) is that our data collection had a significantly higher sampling rate (120 vs. 60 Hz) and a more accurate calibration procedure (nine-point vs. five-point calibration). Furthermore, we used videos with multiple CIs (two or three CIs vs. one CI in the mentioned studies) and two videos instead of one, which makes the data more accurate and multifaceted and thus slightly increases the external validity.

Another strength is the multi-methods approach, which allowed us to combine the different measurement methods for noticing and examine the validity of the keystroke and verbal data with gaze parameters, a so-called methodological triangulation with three different sources of information. In addition, by systematically analyzing post hoc verbalizations (RTA) and concurrent measures (keystroke) in conjunction with eye-tracking data (fixation count, mean fixation duration, revisits), this article represents one of the first methodological attempts in the research area of teachers’ PV to combine and triangulate different data sources in order to mitigate the problems associated with the eye–mind assumption and eye-tracking research. Thus, this study lays a foundation for further systematic investigations based on it and represents a starting point for continuing studies.

Nevertheless, on top of the mentioned limitation regarding administrating the RTA protocols, the study has some drawbacks. Although we used standardized video samples, these were very short extracts of very standardized lessons in a frontal teaching setting with low complexity and very “usual” disturbances (Grub et al., 2020, 2022a,b; Grub, 2023). To recognize differences in competence, the situation or the focus to be observed should be more complex, because only then will the top-down-based perception typical for expertise become relevant, and thus, the differences between novice and expert teachers will become visible (Seidel et al., 2021; Stahnke and Blömeke, 2021a). If explicit investigations regarding differences in expertise are to be conducted, care should be taken to ensure that the material used is also suitable for this purpose, i.e., that a knowledge-based top-down perception is a prerequisite for successful PV.

4.2. Implications for further research

With the keystroke and RTA approaches, we were able to investigate two additional measurement methods for noticing, so providing more information about the location of attention than gaze parameters alone. Both approaches have their particular (dis-)advantages. A keystroke is an economic and non-invasive process-based method of assessing the spontaneous noticing process during an observation. Researchers should bear in mind, however, that with a keystroke alone, only information about the timepoint, but not about the location of the noticing process, can be inferred. For this, additional information from eye-tracking (fixation at the moment of the keystroke) or verbalizing is necessary. Another possibility for future research is a mouseover with a click to tag the attentional focus during observation; however, this can lead to distortion of the gaze behavior (Malone and Brünken, 2020). Therefore, the question remains open about what (creative) possibilities there are for matching eye-movement data relative to the two investigated variants, and whether there might be more suitable methods for triangulation.

The RTA is a less economical way of assessing what participants have seen during an observation and can lead to fabrication or oblivion despite a careful research design. Furthermore, it has higher requirements regarding verbalization skills for the participants. Nevertheless, RTA is an appropriate method to gain insight into the knowledge schemata as well as the characteristics of a situation as a basis for making inferences about professionals. In our study, we only rated a defined disturbing behavior if it was mentioned in the RTA. Additionally, it might be interesting to collect information on which concrete characteristics lead to the inference that a specific behavior is relevant. Seidel et al. (2021), for example, investigated whether (student) teachers can assess student profiles (struggling, uninterested, underestimating, etc.) accurately, asking participants which features lead to their decision. In our view, this is promising for gaining insight into the beliefs or intentions of (prospective) teachers. For example, the valuation of a behavior as disturbing depends on one’s normative beliefs and expectations, the rules of the situation (e.g., frontal instruction vs. seatwork; Stahnke and Blömeke, 2021a), or the didactical aims of a teaching situation (McIntyre et al., 2017; Muhonen et al., 2021, 2023). Beyond that, however, many other conceivable factors can influence the relevance of an event and a participant’s decisions and behavior; for example, what education the (student) teacher has enjoyed, generational effects, more conservative vs. liberal attitudes facing the teaching process (which could influence the estimation process), experience in dealing with disturbances, motivation, or (self-effective) handling of disturbances in one’s own teaching practice.

Finally, we want to emphasize a well elaborated selection of eye-tracking parameters carefully derived from a theoretical foundation and the hypotheses of the research (Holmqvist et al., 2011; Carter and Luke, 2020; Grub, 2023). For example, the parameter fixation duration could be influenced by some conditions as the aim of perception (e.g., observing or interacting) or the presentation mode of the stimuli (e.g., ON mode vs. IN mode). As a parameter for attention, it is maybe not sufficient for dynamic tasks, when monitoring or scanning is needed (as observation in the classroom or a traffic scene to detect incidents or hazards). For assessing monitoring gaze behavior, the Gini coefficient as a measure of (un-)equal distribution (Cortina et al., 2015) as well as the gaze relational index (GRI) as a combined measure was used (Gegenfurtner et al., 2020b; Kosel et al., 2023b). The latter is calculated as the relation of fixation count and fixation duration, a lower value indicate a more equally distribution of gaze over the scene. It is an open question, if other combinations of parameters are suitable for assessing the focus of attention or other relevant perceptional skills. For future research, we also see the application of a person-centered approach as promising to combine different eye-tracking parameters as well as other methodological measures within a person to identify different profiles and take the heterogeneity of persons or situations into consideration (Bergman and Andersson, 2012; Hickendorff et al., 2018).

5. Conclusion

With the present paper, we create a first step toward the systematic elaboration of a methodological approach for triangulating eye-tracking data from (prospective) teachers by providing new insight into triangulating eye-movement data with concurrent measurement as well as post hoc verbalization. We were able to find heterogeneous results regarding the validity of the two triangulation methods: when considering eye movements and measures of noticing (RTA, keystroke), it is clear that conscious awareness is associated with increased monitoring of video footage (in particular, more fixations and revisits), but not for all CIs. In accordance with these findings, verbal data and concurrent measurements seem to be useful, albeit limited, ways to link eye movements to awareness. Furthermore, we did not find differences in attention between expert and student teachers, either in terms of verbal data or their behavioral responses. All in all, this type of research needs further systematic investigation and purposeful manipulation to ensure a more suitable resource-saving method of data triangulation for eye-tracking data that can be used in the medium term to examine teachers’ PV. Only adequate methodological triangulation will allow valid conclusions to be drawn from eye-tracking data, and ultimately, they will be of great importance, especially in the long term, for applying study results to teacher education and training.

With regard to open science research, we would like to emphasize that our study is presented as transparently as possible: the study design was preregistered, our data are accessible for interested researchers, and regarding the eye-tracking procedure, as much information as possible is provided to make replication possible (e.g., information on calibration precision/accuracy, threshold filter, information on the generation of AOIs, etc. see Pappa et al., 2020; Holmqvist et al., 2022; Kosel et al., 2023b).

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Data Protection Officer, Saarland University. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

AB: Conceptualization, Data curation, Formal Analysis, Writing – original draft. RB: Writing – review & editing, Funding acquisition. DL: Resources, Writing – review & editing. A-SG: Writing – review & editing, Conceptualization, Data curation, Investigation, Methodology, Project administration, Resources.

Funding

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

References