From analysis to training: Recycling interaction data into learning processes

Analysis of learner interactions with CALL systems has become an imperative aspect of CALL research, with recommendations to focus the research not solely on the design of the system but also, and most importantly, on the learner(s) and the learning task(s). Indeed, while collecting users’ data is an essential step to improve the engineering of a system under development, analysing interactions data may also be very pertinent to improving learning processes. Adopting this stand, this article discusses methods used to (1) capture behaviours of learners interacting with web-based tools during specific interventions, (2) measure systems’ effectiveness, and (3) recycle these data into future learning processes.


Introduction
As technology (and more particularly any Internet mediated tools used for learning or teaching) has become an integral part of most language learning environments, its successful integration in and outside of the classroom (either physical or virtual) requires learning environment and structure that have been carefully designed (e.g.Levy, 2002).Researchers in educational ergonomics have argued that a holistic approach to learning design will help us better comprehend what learners actually do when they are working with web-based technology (Raby, 2007;Bertin and Gravé, 2010).Within this educational ergonomics approach, recommendations have also been made to focus the research not solely on the design of the system but also, and most importantly, on the learner(s) and the learning task(s) (Chapelle, 2001;Colpaert, 2006;Felix, 2005).Consequently, while collecting users' data on interactions with computer-assisted language learning (CALL) tools have traditionally been used to improve systems under development, we argue here that analyzing interactions data may also be very pertinent to improving learning processes (Hamel, 2012).
In this article, we will discuss the methodology that we have used to recycle interaction data into learning processes.First, we will describe the type of data that we collected, then we will briefly present the tools that we used to capture the interactions, and last, we will explain how this empirical data may be recycled into learning environments in order to bring innovations to either pedagogy or learning, or both.
We base our work on the premise that computers and the Internet constitute artefacts that require cognitive and functional adaptation by their users (Nardi, 1996).This view derives from former studies, guided by an Activity Theory perspective, taking the artefacts as point of departure and focusing on how their design affects both the user and the task (Jonassen and Rohrer-Murphy, 1999).For instance, Norman (1991) claimed that a clear understanding of the role played by tools (i.e.cultural artefacts) was critical to the improvement of their design.He stated: Every artefact has both a system and a personal view, and they are often very different in appearance.From the system view, the artefact appears to expand some functional capacity of the task performer.From the personal view, the artefact has replaced the original task with a different task, one that may have radically different cognitive requirements and use radically different cognitive capacities than the original task.(p. 22) This concept is as essential for the development of CALL (Computer Assisted Language Learning) tools today because we need to account for the effects that new systems, and new e-learning tasks, may have on learners at both cog-nitive and functional levels.Along the same line of thought, Felix (2005) stated that research in CALL should focus on the "processes of learning rather than outcomes alone" (p.3).She added that recording what is happening during an intervention was as critical as studying the effectiveness of the tool itself (upon reaching a learning outcome).Indeed, while Internet-based learning is fast expanding, we need to fully understand the mediation that happens between a CALL tool, a learner and a task.
In a survey of research on computer tracking in CALL, Fisher (2007) also concluded that computer tracking constituted a sound ethnographic method to obtain feedback on learner behaviours while they interact with a CALL system.The main advantage of using computer tracking is that it provides objective data on the way in which students actually use software.These data may then help developers refocus their work on the user and avoid that too much focus be put on the on the "perceived technological potential" of the design (Hémard, 2003, p. 21).Hémard also believes that CALL design should be centered on developing very user-friendly systems that motivate users to "interact productively and voluntarily instead of simply acting and reacting" (23).In sum, in an ideal learning scenario, all actors involved in the learning processes should take part directly or indirectly in the CALL tool and task development (Colpaert, 2006;Hémard, 2003;Levy, 2002).
The educational ergonomic approach is a learner-centered method that can be used in the design and development of CALL resources.The core methodology seeks to analyze interactions between a learner, a task, and a tool, and inherently, it implies that there exists a symbiotic relationship between tools (also called 'instruments') and human beings ('users') that may be analyzed within a socio-cognitive framework.In other words, artefacts (namely CALL tools) take their meaning within a social and cultural practice (e.g.Rabardel, 1995;Raby, 2007).

Background
The goal of our research was to investigate how students who work with CALL programs interact with such systems, and propose criteria to enhance the quality of the learner-task-tool interaction.Our research method is derived from the Analysis-Design-Development-Implementation-Evaluation (ADDIE) model, an instructional system design (ISD) method that is particularly well suited to guide developers in the creation and evaluation of language software or other language related computer systems (Strickland, 2006;Colpaert, 2006).As described by Colpaert, one key benefit of the ADDIE model is that "each stage delivers output which serves as input for the next stage" (p.115).Through an

FIGURE 1
Research and development based on a cycle of interventions iteration of studies (Figure 1), results of interventions serve as a basis for the next cycle.As such, data are analyzed and recycled into the design process.Such method can also be assimilated to quasi-experimental studies.

Interventions
The interactions that we analyze are based on two specific web-based systems that have been developed at our respective institutions.Each system was originally designed to fill a particular pedagogical need and to help learners develop specific skills (Caws, 2009;Hamel and Caws, 2010;Hamel, 2012).

Tools
We make the distinction here between the systems (also referred herewith as CALL tools) used for learning and the tools, i.e. the instruments used to collect research data.
Tool A, FrancoToile (hosted at francotoile.uvic.ca/, is a digital library of videos and transcripts of French-speaking individuals from around the world.Tool B, Dire autrement (hosted at web5.uottawa.ca/direautrement) is a web-based, open-access lexical resource for the intermediate to advanced learners of French.It is an electronic dictionary enhanced with tutorials, i.e. a CALL dictionary.Dire autrement is also a CALL dictionary of a particular type, which was termed a 'reformulation' dictionary for its active nature, its orientation on learners' production skills, and in particular, on their ability to use synonyms, collocations and paraphrases (Milicevic and Hamel, 2007).
Instruments used to collect and analyse data on the interactions include the following: • For data collection: video-captures of learner-task-tool interactions, verbal protocols, pre/post reflective tasks; background and user satisfaction questionnaires; • For data analysis: action/behaviour-based markers enabling the annotation and compilation of data within computer video-clips, discourse-

FIGURE 3
Screen capture of the Dire autrement online dictionary based analysis of transcripts generated by learners' reflective tasks and verbal protocols, correlation-based analysis for data triangulation.
Both set of data include product and process oriented quantitative and qualitative parameters (Hamel, 2012).While understanding interactions on a functional level (to measure the effort made by a user to reach a learning goal i.e. the efficiency of the task process), it is also critical to measure the degree of accuracy with which a user completes a task i.e. its effectiveness and overall outcome (Hamel, 2012).

Participants
Previous pilot studies (Hamel and Caws, 2010), including evaluations and experimentation with the system helped us identify the target user group that was the most likely to benefit from the system.The participants are typically French as second language (FSL) students enrolled in university courses at the intermediate to advanced levels.Courses hosting the systems are geared towards the development of language skills, including linguistic abilities (in particular lexical accuracy in the case of Dire autrement) and cultural literacy.In order to better understand and analyze the interactions between users and tools, we collect semi-ethnographic data (through online questionnaires) informing us on participants' background experience with the language and with various technologies that are related to the tools with which they are going to interact.

The tasks
Before the intervention, the instructor/s and the researcher/s discuss the course content and objectives, as well as the role played by the various actors in the course in order to design tasks that will address one or several outcomes.During the intervention (typically done in a computer lab, and if possible within an authentic teaching and learning/classroom context) we analyse specific "work situations" (e.g.Raby 2007) to determine what learners actually do while they are immersed in a task that involves interaction with a specific instrument (i.e.we analyse the process).We also measure the achievement score (namely, learners' performance).Last, we collect their perceptions of the task and of their interactions after the task and analysed these data using a discourse analysis method.Skills solicited by the tasks belong mostly to the cognitive domain of learning (for both tools).In the case of FrancoToile some degree of affective skills is also required.Learners are asked to identify, explore, analyze, evaluate, paraphrase, interpret, discriminate, or manipulate language or culture related items.In addition, we try to assess users' level of functional literacy by observing the skills (or lack of) that seem to transpire through the interactions and the measures of efficiency (the task process) and effectiveness (the task outcome).Table 1 offers a sample of tasks with both systems.

The "recycling" metaphor
A full cycle of evaluation (i.e. through an intervention that is set within an authentic learning environment) allows us to: 1. know our learners; 2. engage our learners in cognitive and functional (psychomotor) tasks; 3. value our learners' perceptions and feedback.

FIGURE 4
Recycling data into learning Figure 4 illustrates the several stages of the research from collecting data to recycling data into new processes and/or product(s).

Knowing the learner
Data collected on learners' past experience with CALL and on their interactions with our systems are recycled into new or improved features within the systems, as well as new pedagogical interventions (such as new tasks, improved training, new types of interactions).As explained by Levy and Stockwell (2006), and shown in other studies on interactions between artefacts and learners, "user's past experiences influence present perceptions of these artefacts" (p.120).Indeed, the experience that we create through the interventions is influenced by former interactions with similar systems and it will also influence future interactions with new systems.For that matter, training should become an integral part of any interventions using CALL.Getting to know our learners in depth through pre-tasks surveys and/or observations will help us better train participants to use systems, as well as adjust our systems to better match participants' functional skills.

Engaging the learner
Once we analysed the interaction data and measured the effectiveness and efficiency produced by learners, we tried to distinguish between the various skills, online working strategies manifested by learners (Hamel, in press).Referring back to Bloom's taxonomy of Learning Domains (e.g.Anderson and Krathwohl, 2001), we identified gaps in soliciting critical skills.Ultimately, to increase learning at a deep level, learners need to go beyond the understanding or knowing stages and reach the valuing and creating stages of learning.With regard to the CALL tools that we are developing, it means that we need to ensure that learners can interact with the system and not simply react to it, that their design reaches acceptability (Hémard, 2003).Within a socio-cultural perspective of learning, the cognitive engagement with the various elements of an activity is significant, and, as explained by Lantolf and Pavlenko (2001), this engagement matters because "it is the activity and significance that shape the individuals' orientation to learn or not" (p.148).In the case of FrancoToile for instance, we could adapt the system so that users also become producers of knowledge.Any function that allows users to interact in a productive manner, such as facilitating the submission of annotation or interacting with other users to reflect on any aspect of the system, will engage users in meaningful learning.Ultimately, engaging the learner in a participatory manner may lead to a more reflective and critical method of interaction with CALL tools.

Valuing the learner
Post-tasks data are designed to collect students' feedback and perceptions of the learning tasks and systems.These data gives us an opportunity to note potential gaps in either the system and/or the learning tasks (either its goals/evaluation/setting/presentation).Following the reflective stance evoked earlier, our post-intervention tasks (by way of surveys, questions or focus group interviews) may develop students' "rhetorical literacy", as coined by Selber (2004), by engaging them in a "self-conscious and self-critical manner" (p.160) such as reflecting on the usability of interface design in a context specific to their course or program.Inherently, by valuing the learner we value the role that they play by interacting with artefacts and helping us improve learning activities.

Conclusions
Recycling research and interaction data directly into a redesign of learning processes appears to be a method that is both innovative and liberating in regard to pedagogy.It is also an effective method to better train participants to use systems, to interact critically with such systems, and to engage actively with the learning tasks.Evidently learners become participants of the activity in their own rights.
Analysis of learners' interactions with CALL tools has effects also with respect to the development of autonomy (Blin, 2004).Learners need to be more than self sufficient to become autonomous learners.They also need to learn to work at a meta level by developing (meta)cognitive and reflective skills and shy away from a tendency to rely on a behaviourist approach to learning.Empowering users with a meta discourse to discuss, analyse and critique computer systems is yet another way to contribute to critical literacy and help students understand the cultural, political, social and psychological role played by computers, or the Internet.
With regard to learning, the developing of new learning systems becomes a balancing act between addressing learners' needs and system's requirements.This balancing act is in line with mediated activities where other aspects of the learning situation (rules, community, division of labour) will impact the success or failure of the tasks within a set activity (e.g.Engeström, 1987).In addition, when designing a learning task to be performed with a CALL tool, both instructor and researcher (if the task is being empirically evaluated) should carefully craft the specifics of the course, learning outcomes and users' functional abilities.
In sum, design is a multi-faceted concept, and CALL design is very "context dependent" (e.g.Levy, 2002).Yet, combining pedagogy and CALL tool methodology to create a symbiotic relationship that enhances users' experience is often challenged by institutional or program related decisions.Consequently, future research in CALL should increase its impact and connection to learning design so as to better prepare learners to function competently in a knowledge-based society.

FIGURE 2
FIGURE 2Video page in FrancoToile showing subtitle and annotation It offers students a window into Francophone culture through the testimonials of individuals who currently live or who have lived in a Francophone environment.It also gives access to authentic, spontaneous oral discourse and annotated transcripts.Tool B, Dire autrement (hosted at web5.uottawa.ca/direautrement) is a web-based, open-access lexical resource for the intermediate to advanced learners of French.It is an electronic dictionary enhanced with tutorials, i.e. a CALL dictionary.Dire autrement is also a CALL dictionary of a particular type, which was termed a 'reformulation' dictionary for its active nature, its orientation on learners' production skills, and in particular, on their ability to use synonyms, collocations and paraphrases(Milicevic and Hamel, 2007).Instruments used to collect and analyse data on the interactions include the following:

TABLE 1
Sample of tasks in FrancoToile and Dire Autrement