The scalable implementation of predictive learning analytics at a distance learning university: Insights from a longitudinal case study

Highlights

• This study describes a large-scale, longitudinal PLA implementation.

• It presents lessons learnt from the 4-year enactment and evaluation of OUA.

• Faculty, champions, and evidence as facilitating scalability.

• Conceptions of teaching and digital literacy affecting teacher's adoption.

Abstract

A vast number of studies reported exciting innovations and practices in the field of Learning Analytics (LA). Whilst they provided substantial insights, most of these studies have been implemented in single-course or small-scale settings. There are only a few studies that are large-scale and institutional-wide adaptations of LA and have explored the stakeholders' perspectives (i.e., teachers, students, researchers, management) and involvement with LA. This study reports on one such large-scale and long-term implementation of Predictive Learning Analytics (PLA) spanning a period of 4 years at a distance learning university. OU Analyse (OUA) is the PLA system used in this study, providing predictive insights to teachers about students and their chance of passing a course. Over the last 4 years, OUA has been accessed by 1159 unique teachers and reached 23,180 students in 231 undergraduate online courses. The aim of this study is twofold: (a) to reflect on the macro-level of adoption by detailing usage, challenges, and factors facilitating adoption at an organisational level, and (b) to detail the micro-level of adoption, that is the teachers' perspectives about OUA. Amongst the factors shown to be critical to the scalable PLA implementation were: Faculty's engagement with OUA, teachers as “champions”, evidence generation and dissemination, digital literacy, and conceptions about teaching online.

Introduction

Across the globe Higher Education Institutions (HEIs) are exploring opportunities technology affords to provide a consistent and personalised service to students and other stakeholders (e.g., Herodotou et al., 2017; Rienties et al., 2016; Gasevic, Dawson, Rogers, & Gasevic, 2016; Gelan et al., 2018; Tait, 2018). In the last 9 years, Learning Analytics (LA) has been strongly ‘pushed’ forward by policy makers, managers, teachers, and researchers as a means to address student retention (e.g., Larrabee Sønderlund, Hughes, & Smith, 2019; Zacharis, 2015), to improve learning design (Colvin et al., 2015; Ferguson et al., 2016; Macfadyen & Dawson, 2010), and to provide real-time actionable feedback to teachers and students (Cheng, Liang, & Tsai, 2015; Jovanović, Gašević, Dawson, Pardo, & Mirriahi, 2017; Scheffel et al., 2017; Tempelaar, Niculescu, Rienties, Giesbers, & Gijselaers, 2012).

A range of mostly western HEIs have started to explore the use of LA dashboards that can display learner and learning behaviour to teachers and instructional designers, and provide just-in-time support to students (Herodotou et al., 2017; Bodily et al., 2018; Jivet, Scheffel, Specht, & Drachsler, 2018; Scheffel et al., 2017). Furthermore, several HEIs have developed Predictive Learning Analytics (PLA) approaches, or have adopted existing integrated predictive solutions embedded into Virtual Learning Environments (VLEs), to help identify students who may be ‘at risk’ of failing (Calvert, 2014; Wagner & Longanecker, 2016).

Although substantial progress has been made in terms of early adoption and uptake of LA in the form of experiments and single-course designs, several researchers have argued that most LA adaptations are mainly on a small, micro level (e.g., Dawson et al., 2018; Ferguson et al., 2016; Gasevic et al., 2016; Higher Education Commission, 2016; Viberg, Hatakka, Bälter, & Mavroudi, 2018). While some of the LA and technology conferences might give the impression that ‘everyone’ is using LA, in reality most institutions across the globe, and teachers in particular, have limited or no experience with LA (Ferguson et al., 2016; Ferguson & Clow, 2017; Viberg et al., 2018). There is only a handful of institutions that have adopted LA as a main organisational approach. One such example is the Open University, UK (OU) (e.g., Ferguson & Clow, 2017; Higher Education Commission, 2016; Hoel, Griffiths, & Chen, 2017; Raths, 2016). The OU is the first university to implement an institutional ethics policy in LA (Slade & Boroowa, 2014), has a university-wide implementation of PLA for its 170,000+ students (Herodotou et al., 2017; Calvert, 2014; Wolff, Zdrahal, Nikolov, & Pantucek, 2013), and has worked extensively with teachers to use (near) real-time data of students to inform their teaching and learning practice (Herodotou et al., 2019a, Herodotou et al., 2019b, Herodotou et al., 2019c; Rienties and Toetenel, 2016, Rienties et al., 2017).

Nonetheless, a recent study about the state of LA at the OU (Rienties et al., 2019) indicated that there is substantial room for improvement in how the organisation and its stakeholders use LA, in particular: a) Improved communication supported by LA, b) Personalisation to recognise unique distance learners' needs, c) Integrated design from inquiry to lifelong learning, and d) Development of a strong evidence base about what works and what does not. In this paper, we focus specifically on the second area: LA could be used to support ‘at risk’ students who may struggle with course content and assessments, by tailoring teaching and support staff resources. OU Analyse (OUA) is one approach used at the OU to tackle this issue. During the last 4 years, OUA has been implemented on a large scale in 231 undergraduate courses, engaging 1159 unique teachers, and reaching 23,180 students. A range of studies have shown that OUA is effective both in terms of identifying students at risk at an early stage (Wolff, Zdrahal, Herrmannova, Kuzilek, & Hlosta, 2014), and helping teachers to effectively support their students (Herodotou et al., 2017, Herodotou et al., 2019a, Herodotou et al., 2019b). Yet, in line with other research (e.g., Arbaugh, 2014; van Leeuwen, 2018), large differences in actual OUA usage by teachers were reported (Herodotou et al., 2017, Herodotou et al., 2019a, Herodotou et al., 2019b), with some teachers actively using OUA, while others using it only sporadically. Furthermore, there was substantial divergence in terms of uptake within particular Faculties and qualifications (Herodotou et al., 2019a, Herodotou et al., 2019b). In order to better understand the complex dynamics of OUA uptake on a large scale and inform strategies of scalable PLA adoption, this study will use two complementary perspectives. On a macro level, an interdisciplinary project team responsible for OUA implementation and evaluation will reflect on the main challenges and factors facilitating implementation and adoption by 1159 teachers in 231 courses over a period of 4 years. On a micro level, the experiences of eight teachers and how they make use of OUA in their daily practice will be explored. This study aims to address the following two Research Objectives (ROs):

RO1: To reflect, at a macro-level, on the use of PLA by detailing the degree of OUA usage, challenges, and aspects facilitating adoption, over a period of 4 years.

RO2: To reflect, at a micro-level, on the use of PLA by detailing the perspectives of teachers who make use of OUA.