Student profiling in a dispositional learning analytics application using formative assessment

Highlights

• Formative assessment data have high predictive power in generating learning feedback.

• Learning disposition data are most actionable: triggering educational interventions.

• Dispositional LA is instrumental in chaining dispositions, traces, performance.

• Student profiling based on traces allows characterization in terms of dispositions.

Abstract

How learning disposition data can help us translating learning feedback from a learning analytics application into actionable learning interventions, is the main focus of this empirical study. It extends previous work (Tempelaar, Rienties, & Giesbers, 2015), where the focus was on deriving timely prediction models in a data rich context, encompassing trace data from learning management systems, formative assessment data, e-tutorial trace data as well as learning dispositions. In this same educational context, the current study investigates how the application of cluster analysis based on e-tutorial trace data allows student profiling into different at-risk groups, and how these at-risk groups can be characterized with the help of learning disposition data. It is our conjecture that establishing a chain of antecedent-consequence relationships starting from learning disposition, through student activity in e-tutorials and formative assessment performance, to course performance, adds a crucial dimension to current learning analytics studies: that of profiling students with descriptors that easily lend themselves to the design of educational interventions.

Introduction

The challenge to design “an optimal sequence of data collection and economic response times …” that includes “the minimum requirements for making valid predictions and creating meaningful interventions” (Ifenthaler, 2015) as one of the challenges to the application of learning analytics (LA), is the main topic of this empirical contribution to dispositional learning analytics. Learning Analytics (LA) is defined as ”the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Buckingham Shum and Ferguson, 2012, Gasevic et al., 2016, Siemens, 2013). In the early stages of LA, many scholars focused on building predictive models based on data extracted from both institutional student information systems (SIS) and digital platforms that organize and facilitate learning, such as learning management systems and e-tutorials (LMS, taking them together). While these studies provide important markers on the potential of LA in education, the findings were rather limited to the descriptive functions of LA, which is mostly based on demographics, grades, and trace data. Given the rigidity of SIS and LMS data, educators may encounter difficulties in designing pedagogically informed interventions (Conde and Hernández-García, 2015, Tobarra et al., 2014, Xing et al., 2015).

To overcome this shortcoming, Buckingham Shum and Crick (2012) proposed a Dispositional Learning Analytics (DLA) infrastructure that combines learning data (i.e. those generated in learning activities through the LMS) with learner data (e.g., student dispositions, values, and attitudes measured through self-reported surveys). Learning dispositions represent individual difference characteristics that impact all learning processes and include affective, behavioral and cognitive facets (Rienties, Cross, & Zdrahal, 2017). Student's preferred learning approaches are examples of such dispositions of both cognitive and behavioral type; in research on their role in learning, they are often simply labeled as ‘self-report data’ (see e.g. Buckingham Shum and Ferguson, 2012, Gašević et al., 2017). Different from LA research, stakeholders of DLA applications are typically restricted to students and teacher/tutors, as these applications can be positioned at both the meso- and micro-level (Ifenthaler, 2015), rather than the mega- or macro-level. Our study is a follow-up of previous research by the authors on the application of LA in a ‘data-rich context’ (Tempelaar, Rienties, & Giesbers, 2015). The availability of formative assessment data constitutes a crucial aspect of that data richness, together with learning activity trace data of students practicing in e-tutorial systems in order to be optimally prepared for these formative assessments, and later summative assessments. That data of cognitive type was complemented by learning disposition data to cover all “affective, behavioral and cognitive facets of the ABC framework of student learning” (Rienties et al., 2017).

Our previous research indicated a sensitive balance between timing and predictive power of the several data sources in a rich data context. Most informative, but least timely, is typically formative assessment data. Given that formative assessment data is not available until several weeks into a course, trace data from e-tutorial systems are a good second-best. However, it is important to note that the use of e-tutorial trace data is ill-advised at the very start of the course when practicing activities of students have not yet settled into stable patterns. Therefore, learning disposition data are an informative data source next to the trace data in predicting student performance (Tempelaar et al., 2015).

This follow-up study focuses on this very early stage of generating learning feedback at the start of courses that is “personalised, dynamic and timely” (Ifenthaler, 2015). The requirement of learning feedback to be timely implies a crucial role for learning disposition data. The requirement of learning feedback to be actionable too has strong links with the availability of dispositions; learning interventions such as academic counselling are often based on the same social-cognitive frameworks as the instruments used to measure learning dispositions (such as improving one's learning style, or changing mal-adaptive into adaptive approaches to learning, in case of setbacks) (Tempelaar, Rienties, & Nguyen, 2017a).