How Signaling Principle Affects Learning: An Eye Tracking Study

Studies examining the effects of signaling principle on learning mostly used data sources such as interviews, achievement tests and think aloud protocols. There is, however, limited research on the use of quantitative measures to promote findings from interviews and think aloud procedures. This study aimed to investigate the effects of signaling principle on learning by supporting the results with the eye movement data. Participants included 34 Information and Communication Technology (ICT) pre-service teachers divided into experimental and control groups randomly. Participants were presented with two types of MS Power Point presentations with and without highlighted clues as multimedia material. Participants’ pre-test, post-test, and retention test scores were measured. Participants’ gaze behavior (fixation duration, fixation counts and time to first fixation) during the presentation of instructional material was also recorded. The results revealed that while the experimental group had higher scores than control group in the postand retention tests, there was no significant difference between the two groups. The analysis of eye movement data showed that signaling directed the attention of the students on relevant information and decreased students’ cognitive effort. Results revealed that both eye movement data and achievement scores were consistent with each other.


Introduction
The advances in ICTs and learning sciences enhanced the design of traditional learning materials (e.g., printed books) with varied forms of multimedia elements such as pictures, animations, videos, sounds, etc. However, to organize these elements in a multimedia learning material, enhance learning, and decrease cognitive load at the same time, researchers investigated several design patterns. Mayer's Cognitive Theory of Multimedia Learning (CTML) provided principles to design multimedia learning materials (Mayer, 2013). Research revealed positive effects of multimedia learning principles on learning (Mousavi, Low & Sweller, 1995;Owens & Sweller, 2008;Scheiter & Eitel, 2010). Most of these studies used traditional data collection tools (e.g., interviews, think aloud) to explain the cognitive processes associated with learning with multimedia. Recently, researchers used eye tracking as a method to gather data regarding quantitative measures such as fixation duration, fixation counts and time to first fixation to promote qualitative data obtained from the interviews and think aloud procedures. In many learning studies, interviews which were based on think-aloud procedures were widely used as for data collection tools to explain cognitive processes during learning (Ariasi & Mason, 2011;Soruç, 2015). According to Amadieu, Mariné and Laimay (2011), these studies can be supported by using eye tracking technique in order to provide objective eye movement data such fixation duration and fixation counts.

Multimedia Learning and Cognitive Load Theory (CLT)
With the enhancements in ICTs, the traditional teaching methods started to incorporate visual elements such as charts, videos, animations, photos and illustrations (Mayer, 2013). Multimedia learning can be asserted as learning deeply from words and pictures instead of words alone (Mayer, 2005b). Multimedia principle constructs the basis of multimedia learning (Mayer, 2005a). For example, studies revealed that learners studying text with visual elements showed better performances than the ones who studied text alone (Anglin, Vaez & Cunningham, 2004;Fletcher & Tobias, 2005;Vekiri, 2002). However, putting words and pictures together does not guarantee and enhance meaningful learning. According to Mayer (2005a), understanding how to use words and pictures together is crucial to provide ways to maximize learning effectiveness (Mayer & Moreno, 2008;Sorden, 2012). Mayer's (1997) CTML is an interdisciplinary explanation to support maximizing effectiveness of learning.
CTML emphasizes three kinds of processing: Extraneous, essential and generative processing. These processes are similar to what is proposed in Cognitive Load Theory (CLT) (Sweller, Ayres & Kalyuga, 2011). The explained types of processing imply three instructional design goals such as managing essential processing, fostering, generative processing and reducing extraneous processing (Mayer, 2013). For conducting the latter instructional goal, there are five multimedia principles including coherence, redundancy, contiguity, temporal contiguity, and signaling (Mayer, 2008). All these principles are important to reduce extraneous processing. However, the current study focused on investigating the signaling principle. Moreno (2007) explained signaling principle as learning better from a coherent summary highlighting relevant words and pictures than from a longer version. In addition, Mautone and Mayer (2001) defined signaling as presenting cues to learners in an effective way to process the instructional material (Mayer, 2013). Many studies showed that providing highlighted words or cues resulted in better performances by directing learners' attention to the important parts of the instructional material (Doolittle & Altstaedter, 2009;Jeung, Chandler & Sweller, 1997;Mautone & Mayer, 2001;Moreno, 2007). Signaling principle can help students to focus on important points in a multimedia material and decrease their cognitive load. Research that examined the effects of signaling principle on learning by reducing cognitive load mostly used measures such as interview, think-aloud or learning outcomes to provide information about cognitive processes (Techaraungrong, Suksakulchai, Kaewprapan & Murphy, 2017). With the advancements of eye tracking measures, recently, researchers suggested the use of direct measures such as eye movement data with fixation counts and fixation duration to explain cognitive processes related to learning and cognitive load (Henderson, Brockmole, Castelhano & Mack, 2007). In other words, eye movement data can support qualitative findings such as interview and think aloud by providing more direct measures.

Eye tracking research in multimedia design
Eye tracking is a technique that provides objective measures of learners' real-time cognitive processes (Lai et al., 2013;Paas, Tuovinen, Tabbers & Van Gerven, 2010;van Gog & Scheiter, 2010). There are many parameters such as fixation counts, fixation duration and time to first fixation to interpret the eye movement data. Fixation count indicates how many times the user fixates on the specific area while fixation duration measures the duration of each individual fixation on the specific area. For example, fixation duration can be indicator of difficulty or cognitive complexity of task (Henderson et al., 2007;Raney, Campbell & Bovee, 2014). Moreover, the study conducted by Hannus and Hyönä (1999) indicated that there is positive correlation between total fixation time of participants on relevant information and their performances. Besides, time to first fixation indicates how long it takes before participant fixates on relevant area for the first time. Time to first fixation parameter for relevant information can be used to measure the efficiency of visual search for it (Hyönä, 2010).
Research presented different interprations of fixation duration and fixation counts in multimedia design. For example, some studies revealed that participants investing less mental effort can have longer fixation duration during the task (Amadieu, van Gog, Paas, Tricot & Mariné, 2009;van Gog, Paas & Van Merriënboer, 2005). However, other studies showed that total fixation duration was longer in non-cueing group rather than cueing group because students spent more effort to understand the material (van Gog, Kester, Nievelstein, Giesbers & Paas, 2009). Fixation duration and fixation counts parameters interpreted differently in eye tracking studies conducted in multimedia learning principles, particularly with the investigation of the signaling principle. For example, some studies showed that total fixation duration on relevant information was significantly longer for cueing group than non-cueing group. (Amadieu et al., 2011;Boucheix, Lowe, Putri & Groff, 2013;Jamet, 2014;Ozcelik, Arslan-Ari & Cagiltay, 2010). Participants in cueing group focused on highlighted words more than non-cueing group. In other words, signalling principle directed the attention of participants to relevant words more (Moreno, 2007).
The aim of this study is to investigate effects of signaling principle by using test scores of participants and eye movement data such as fixation duration, fixation counts and time to first fixation. It is important to show similarities between eye movement data and test results because eye movement data can provide information about cognitive load of participants according to their performances on tests. However, there are different point of views about relationship between cognitive load and eye movement data namely fixation duration and fixation counts in the literature. The current study follows the idea, that is, lower cognitive load results in lower fixation duration and fixation counts (Hyönä, 2010). Time to first fixation parameter is another way to provide information about cognitive load by presenting time for catching relevant information (Hyönä, 2010). In addition, eye movement data can present quantitative data besides test scores. In previous learning studies, qualitative techniques such as interviews and think aloud providing information about cognitive processes of participants by depending on their discourses were preferred by different researchers (Ariasi & Mason, 2011;Soruç, 2015).

Hypothesis on the effect of signaling principle in multimedia design
In the current study, three main hypotheses were set: H1: Signaling by highlighting the key words of the subject in multimedia material results in higher test scores for experimental group than control group. H2: The time spent for catching relevant information is lower for experimental group than control group. H3: Experimental group' fixation duration and fixation counts are lower than control group during task, but higher than control group on relevant information.

Method
This experimental study included 34 pre-service ICT teachers who were assigned randomly to the experimental and control groups. The experimental group received MS Power Point presentation consisting of highlighted clues or words, while the control group received MS Power Point presentation with non-highlighted words. Independent variable of the study was studying with Power Point presentation, while dependent variable was the test results that included pre-test, post-test, retention test results, and eye tracking data such as fixation duration, fixation counts and time to first fixation.
Participants studied the subject, network typologies, using two kinds of MS Power Point presentations. After determining the target group, participants were asked to sign inform consent form to show their willingness for the current study. Three weeks before the study started, participants were given a pre-test related to the topic to determine whether or not they had prior knowledge about the subject. After studying the subject with a presentation based material consisting of highlighted and non-highlighted words, students were given a post-test. During the study, participants' eye movement data were recorded via embedded eye tracking device (Tobii 1750) and analyzed via Tobii Studio, the software of the related eye tracker device. After three weeks from the actual study, a retention test was applied to determine their maintained knowledge about the subject studied.

Participants
Participants were 34 pre-service teachers (male: 21; female: 13) who studied instructional technology in a large public university. The preservice teachers (16 freshmen, 7 sophomores, and 11 junior) were selected from the ones who did not take the "Computer Networks and Communications" course, to be sure that participants did not have any prior knowledge about the selected subject. In addition, pre-test was applied to be sure that participants had similar knowledge about the selected subject. These students were divided into the experimental and control groups randomly (group 1: 11 male, 6 female, group 2: 10 male, 7 female). The first group students studied the material consisting of text element with highlighted words, the other group studied the material consisting of text element with no highlighted words.

Task and material
The designed presentation took two forms. In the first one, the material was prepared by using the text including highlighted words (See right side of the Slides 1, 2, 3, 4, 5, and 6 in Figure 1). In the second presentation, the material was prepared by using the same text with no highlighted words (See left side of the Slides 1, 2, 3, 4, 5, and 6 in Figure 1). The content of the materials was adopted from the course "Computer Network and Communications". The content was examined by three subject matter experts who were the instructor and the teaching assistants of the course to ensure that the content presented in the material covered correct and enough information about the selected topic. Each presentation used in the study consisted of eight slides. The main subject was "topology of networks", and three of the topologies were selected to cover the multimedia material according to the recommendations of the experts. At the beginning part of the presentation, two slides were provided for the title and definition of the main subject. While covering this topic, three subtopics, namely star (See Slides 1 and 2), bus-line (See Slides 3 and 4) and ring topologies (See Slides 5 and 6) were provided with two slides for each. The first slides of subtopics explained the typology (See Slides 1, 3, and 5) and the second ones provided detailed information about the advantages and disadvantages of topology types (See Slides 2, 4, and 6). In order to collect data regarding participants' eye movement, Tobii 1750 Eye Tracker device was used. This device tracks participants' eye movements and provides data about where the participants look, and how long and how many times they look at which location using the reflector and the infrared detector camera. The data rate of Tobii 1750 Eye Tracker device for tracking is 50 Hertz. It provides screen recording and eye movement recording of users.

Data collection instruments
The data sources included pre-test, post-test, retention test and eye tracking data such as fixation duration, fixation counts and time to first fixation. These instruments were applied before, during and after the study. Figure 2 illustrates the procedure of the study. First, a test (used as a pre, post, and retention test) consisting of the same five multiple choice questions was prepared in parallel to the highlighted clues provided in the material. The instructor and the teaching assistants of the "Computer Networks and Communication" course checked these materials in terms of their appropriateness for the level of selected participants. After making revisions in the direction of suggestions, the test was piloted with three students who did not participate to the study to make necessary adjustments regarding the clarity of the questions. Then, the participants took the pre-rest three weeks before studying the multimedia material. Pre-test was distributed as hardcopy after asking students' voluntary participation to the study. In addition to the multiple-choice questions, some demographic information such as gender, grade level was also gathered.
Second, two different eye tracking tasks were prepared. In each task, there was an MS Power Point presentation (material) and the post-test. The material for the experimental group consisting highlighted words and material for the control group consisting non-highlighted words were used. In both tasks, the same post-test was applied to the participants. After studying the multimedia material, the post-test embedded into the studied material, was implemented. Participants' answers were drawn or recorded to MS Excel automatically. During the whole eye tracking tasks-studying material and answering post-test, participants' eye movement data (e.g., fixation duration, fixation counts and time to first fixation) were recorded (See Figure 3). The aim was to determine where they focused more with the calculation of fixation counts, how much time they spent with the measure of fixation duration, and when they caught the relevant information with the calculation of time to first fixation. During the eye tracking tasks, students had a control over the tasks without the limitation of a specific time frame. They used their own time, they passed slides and answered questions whenever they felt ready and progressed in their own pace.
Finally, participants completed the retention test three weeks after the eye tracking tasks to check how experimental and control groups retained their knowledge they learned while studying the material. The retention test was provided as hardcopy.

Data analysis
Students' pre-test, post-test, and retention tests answers were scored by two of the researchers. Students' test scores ranged from 0 to 5 points. An independent sample t-test was conducted to determine whether or not the experimental and control groups differed in terms of their pre-test test results. Groups' post-test and retention test results were analyzed through mixed ANOVA (two-way repeated measures ANOVA). Eye movement parameters such as fixation duration, fixation counts and time to first fixation were gathered via the eye tracking device. The eye tracking software automatically provided statistical data according to the selected parameters. Before calculating these parameters, area of interests (AOI) were determined for each highlighted word and screens separately. Then, students' fixation duration and fixation counts both during task and on relevant information, and time to first fixation on relevant information were drawn from the eye tracking software. Independent samples t-tests were conducted for each of these parameters to determine if there was any difference between groups.

Validity and reliability
Several measures were followed to enhance the content validity. First, instructional materials and the test contents were examined by three different subject matter experts to make sure that appropriate and correct information for the selected "Network Topologies" topic was included. Second, the instructional materials, pre-test and post-test were piloted with three pre-service ICT teachers to provide clear and easily understandable content and questions. Then, students' interpretations about unclear parts were revealed and revised. Third, students' pre-test and retention tests which were conducted as paper pencil test were scored by researchers. Finally, different data sources such as test results and eye movement data were used to triangulate and validate the data.

Results
Independent samples t-test and ANOVA results for pre-test, post-test, retention test, and eye movement data such as fixation duration and fixation counts on during task and on relevant information and time to first fixation for relevant information were combined and presented below to promote each other.

H1: Signaling by highlighting the key words of the subject will result in higher test scores for experimental group than control group.
To identify participants' prior levels, their pre-test results were compared. The independent test results showed that there was no significant difference between the experimental (signaling) group (M = 1.53, SD = .87) and control (non-signaling) group (M = 1.65, SD = 1.37). Both experimental and control groups had similar prior knowledge for the selected topic before the study.
A two-way repeated measures ANOVA was conducted to compare the effects of signaling principle on post-test and retention test scores of both experimental and control groups. The results indicated that the experimental group's post-test (M = 3.89, SD = .93) scores were higher than the control group's post-test (M = 3.65, SD = 1.46) scores. In addition, the results indicated that the experimental group's retention test (M = 2.88, SD = 1.22) scores were higher than the control group's retention test (M = 2.65, SD = 1.54) scores. Results showed no significant interaction between post-test and retention test scores of experimental and control groups, F(1,32) = 0, p > .05, partial ² = 0. There was a significant main effect for test types, F (1, 32) = 18.76, p < .05, partial ² = .37. The post-test scores (M = 3.76, SD = 1.21) were higher than retention test scores (M = 2.76, SD = 1.37). According to the results, experimental group had higher scores in both post-test and retention test than control group. In sum, there was no significant difference between groups' performances on post-test and retention test, but experimental group had higher mean scores for both tests than control group.

H2: The time spent for catching relevant information is lower for experimental group than control group.
To examine how signaled words directed students' attentions, time to first fixation of students on relevant information was analyzed. Time to first fixation referred to time to catch highlighted word. An independent samples t-test was conducted to determine whether signaling group and non-signaling group differed in terms of their time to first fixation on relevant information. The independent test results showed that there was no significant difference between the experimental group (M = 8.23, SD = 1.77) and control group (M = 8.97, SD = 1.99). Experimental group spent less time to catch highlighted words than control group. In other words, signaling by highlighting directed students' attention to relevant information.

H3: Experimental group' fixation duration and fixation counts are lower than control group during task, but higher than control group on relevant information.
To determine cognitive load during task, both experimental and control groups' fixation duration and fixation counts were used. An independent samples t-test was conducted to determine whether experimental group and control group differed in terms of their fixation duration (secs) during task. The independent test results showed that there was no significant difference between the experimental group (M = 79.92, SD = 19.02.) and control group (M = 87.68, SD = 29.03). Moreover, an independent samples t-test was conducted to determine whether experimental and control group differed in terms of their fixation counts during the task. The independent test results showed that there was no significant difference between the experimental group (M = 392.65, SD = 60.98) and control group (M = 420.29, SD = 25.90). In both independent samples ttests, total fixation duration and fixation counts of students in control group were higher than experimental group. Students spent more effort to conduct the task and their cognitive load was higher than the experimental group. In addition to cognitive load during task, cognitive effort on relevant information was also investigated by fixation duration and count on highlighted words. Students' cognitive effort on highlighted words needed to be lower according to signaling principle. An independent samples t-test was conducted to determine whether signaling group and non-signaling group differed in terms of their fixation duration (secs) on relevant information. The independent test results showed that there was no significant difference between the experimental group (M = 12.29, SD = 3.77) and control group (M = 12.10, SD = 3.82). Moreover, an independent samples t-test was conducted to determine whether signaling group and non-signaling group differed in terms of their fixation counts (secs) on relevant information. The independent test results showed that there was no significant difference between the experimental group (M = 59.18, SD = 12.46.) and control group (M = 58.30, SD = 15.44). Although there was no significant difference between groups, fixation duration and fixation counts of students in control group were higher than students in experimental group on relevant information. Students in control group spent more effort to cover the highlighted words. In sum, independent samples t-tests results showed no significant difference for eye movement data namely, fixation duration and fixation counts (both during task and on relevant information) and time to first fixation (on relevant information) of groups. Students in control group spent more effort to complete the task than students in experimental group in terms of their fixation duration and fixation counts. Although the difference was not significant, students in experimental group captured the relevant or signaled information more quickly than control group by considering their time to first fixation. Parallel to this, they spent less effort to cover highlighted (relevant) words by considering their fixation duration and fixation counts on relevant words.

Discussion
This study investigated the effects of signaling principle on learning of students by supporting the test results with the eye movement data of participants via eye tracking technique. There were three hypotheses: (1) signaling by highlighting the key words of the subject provides higher test scores for experimental group than control group, (2) the time spent for catching relevant information is lower for experimental group than control group, and (3) experimental group' fixation duration and fixation counts are lower than control group during task, but higher than control group on relevant information. The results were similar to the previous literature showing lower fixation duration and fixation counts in the experimental group during task (van Gog et al., 2009;Amadieu et al., 2009), but higher fixation duration and fixation counts on relevant information (Ozcelik, Arslan-Ari, & Cagiltay, 2010;Boucheix et al., 2013;Jamet, 2014).
However, similar to the results of Boucheix et al. (2013) and Ozcelik et al. (2010), there was no significant difference between groups in terms of these parameters. In parallel, there was no significant difference between the experimental and control groups' post-test and retention test scores but the experimental group' scores were higher than the control group (Jamet, 2014;Ozcelik et al., 2010). Higher fixation duration on relevant information may result in higher test scores or better performances. Thus, the results of the current study were consistent with the studies arguing that there was a relationship between total fixation duration on relevant information and performances of test scores (Amadieu et al., 2011(Amadieu et al., , 2009Boucheix et al., 2013;Hannus & Hyönä, 1999;Jamet, 2014;Ozcelik et al., 2010;van Gog et al., 2005). In addition to these, results showed that time to first fixation of experimental group were lower than control group. Signaling by highlighting directed students' attention to the relevant information. In other words, student in experimental group spent less time to catch relevant information in the text. This finding is in line with Ozcelik, Karakus, Kursun and Cagiltay's (2009) study on investigating the effect of color coding on multimedia materials by using eye movement data. This study indicated that first fixation time is lower for the color-coded material. As a result, eye movement data of the current study were parallel to achievement scores and all of the data support each other.
One of the reasons why there was no significant difference in student outcomes could be related with the target group and selected topic. Selected participants were familiar with design issues and multimedia learning principles because of their background. This could result in having low tendency to being affected by the selected principle, signaling. Although the selected topic was taken from one of the courses they did not take yet, there might be small possibility that they could carry out ideas about the questions. In other words, these students could have more tendency to answer questions intuitively because of their background (Sabella, Cochran, Marx, Franklin & Cummings, 2004).
In the literature, there were different points of views about interpreting fixation duration and fixation counts terms. One of them argued that signaling principle decreases the effort used in studying the multimedia material in parallel to fixation duration and fixation counts (Boucheix et al., 2013;Jamet, 2014). The other view supported that signalling principle directs the attention of participants to relevant words more and increases the fixation duration and fixation counts (Moreno, 2007). Although the findings of the current study were parallel with the latter one, it was hard to say which one is more acceptable or consistent. However, time to first fixation can be more acceptable parameter to be considered (Hyönä, 2010). In the current study, time to first fixation, the passing time for catching highlighted clue, was lower in experimental group than control group. It indicated that highlighting guided the experimental group to catch the important points faster than control group (Lai et al., 2013). It can be said that signalling directed students' attention to the highlighted words and helped to catch more quickly than the control group.

Conclusions and Future Directions
Current study is about investigating the effect of signaling principle on multimedia learning with the help of eye movement data. The results of this study supported the idea that eye movement data such as fixation counts, fixation duration and time to first fixation can help to figure out students' cognitive load. Eye movement measures can provide quantitative data about cognitive processes of the participants as a complementary procedure to the interview and think aloud methods. Findings from eye movement data can support the findings from achievement test scores, namely post-test and retention test. In other words, both results from achievement scores and eye movement data can be combined to interpret the results regarding the effects of signaling principle on learning. Therefore, this study concludes that eye tracking results provide supportive evidence for achievement scores of students to explain the significant contribution of signaling principle to multimedia learning.
Current study was limited with the students of ICTs and they were from different grade levels. Having participants from another field who were not familiar with design issues and multimedia learning principles could result in significant difference between groups. Furthermore, selecting target group from first grade level from the ICT field might also change the results since they are new in the field or the department. Moreover, the study was limited with the specified subject "Network Topologies". This might be a handicap for this study because students could have interest for the selected subject since it was offered in their department and field. Selecting different topic which was not relevant with their background and department could change the results of the study. Future studies can be conducted with different target groups or different topics which are not related to the participants' academic interests.