The scalable implementation of predictive learning analytics at a distance learning university: Insights from a longitudinal case study
Introduction
Across the globe Higher Education Institutions (HEIs) are exploring opportunities technology affords to provide a consistent and personalised service to students and other stakeholders (e.g., Herodotou et al., 2017; Rienties et al., 2016; Gasevic, Dawson, Rogers, & Gasevic, 2016; Gelan et al., 2018; Tait, 2018). In the last 9 years, Learning Analytics (LA) has been strongly ‘pushed’ forward by policy makers, managers, teachers, and researchers as a means to address student retention (e.g., Larrabee Sønderlund, Hughes, & Smith, 2019; Zacharis, 2015), to improve learning design (Colvin et al., 2015; Ferguson et al., 2016; Macfadyen & Dawson, 2010), and to provide real-time actionable feedback to teachers and students (Cheng, Liang, & Tsai, 2015; Jovanović, Gašević, Dawson, Pardo, & Mirriahi, 2017; Scheffel et al., 2017; Tempelaar, Niculescu, Rienties, Giesbers, & Gijselaers, 2012).
A range of mostly western HEIs have started to explore the use of LA dashboards that can display learner and learning behaviour to teachers and instructional designers, and provide just-in-time support to students (Herodotou et al., 2017; Bodily et al., 2018; Jivet, Scheffel, Specht, & Drachsler, 2018; Scheffel et al., 2017). Furthermore, several HEIs have developed Predictive Learning Analytics (PLA) approaches, or have adopted existing integrated predictive solutions embedded into Virtual Learning Environments (VLEs), to help identify students who may be ‘at risk’ of failing (Calvert, 2014; Wagner & Longanecker, 2016).
Although substantial progress has been made in terms of early adoption and uptake of LA in the form of experiments and single-course designs, several researchers have argued that most LA adaptations are mainly on a small, micro level (e.g., Dawson et al., 2018; Ferguson et al., 2016; Gasevic et al., 2016; Higher Education Commission, 2016; Viberg, Hatakka, Bälter, & Mavroudi, 2018). While some of the LA and technology conferences might give the impression that ‘everyone’ is using LA, in reality most institutions across the globe, and teachers in particular, have limited or no experience with LA (Ferguson et al., 2016; Ferguson & Clow, 2017; Viberg et al., 2018). There is only a handful of institutions that have adopted LA as a main organisational approach. One such example is the Open University, UK (OU) (e.g., Ferguson & Clow, 2017; Higher Education Commission, 2016; Hoel, Griffiths, & Chen, 2017; Raths, 2016). The OU is the first university to implement an institutional ethics policy in LA (Slade & Boroowa, 2014), has a university-wide implementation of PLA for its 170,000+ students (Herodotou et al., 2017; Calvert, 2014; Wolff, Zdrahal, Nikolov, & Pantucek, 2013), and has worked extensively with teachers to use (near) real-time data of students to inform their teaching and learning practice (Herodotou et al., 2019a, Herodotou et al., 2019b, Herodotou et al., 2019c; Rienties and Toetenel, 2016, Rienties et al., 2017).
Nonetheless, a recent study about the state of LA at the OU (Rienties et al., 2019) indicated that there is substantial room for improvement in how the organisation and its stakeholders use LA, in particular: a) Improved communication supported by LA, b) Personalisation to recognise unique distance learners' needs, c) Integrated design from inquiry to lifelong learning, and d) Development of a strong evidence base about what works and what does not. In this paper, we focus specifically on the second area: LA could be used to support ‘at risk’ students who may struggle with course content and assessments, by tailoring teaching and support staff resources. OU Analyse (OUA) is one approach used at the OU to tackle this issue. During the last 4 years, OUA has been implemented on a large scale in 231 undergraduate courses, engaging 1159 unique teachers, and reaching 23,180 students. A range of studies have shown that OUA is effective both in terms of identifying students at risk at an early stage (Wolff, Zdrahal, Herrmannova, Kuzilek, & Hlosta, 2014), and helping teachers to effectively support their students (Herodotou et al., 2017, Herodotou et al., 2019a, Herodotou et al., 2019b). Yet, in line with other research (e.g., Arbaugh, 2014; van Leeuwen, 2018), large differences in actual OUA usage by teachers were reported (Herodotou et al., 2017, Herodotou et al., 2019a, Herodotou et al., 2019b), with some teachers actively using OUA, while others using it only sporadically. Furthermore, there was substantial divergence in terms of uptake within particular Faculties and qualifications (Herodotou et al., 2019a, Herodotou et al., 2019b). In order to better understand the complex dynamics of OUA uptake on a large scale and inform strategies of scalable PLA adoption, this study will use two complementary perspectives. On a macro level, an interdisciplinary project team responsible for OUA implementation and evaluation will reflect on the main challenges and factors facilitating implementation and adoption by 1159 teachers in 231 courses over a period of 4 years. On a micro level, the experiences of eight teachers and how they make use of OUA in their daily practice will be explored. This study aims to address the following two Research Objectives (ROs):
RO1: To reflect, at a macro-level, on the use of PLA by detailing the degree of OUA usage, challenges, and aspects facilitating adoption, over a period of 4 years.
RO2: To reflect, at a micro-level, on the use of PLA by detailing the perspectives of teachers who make use of OUA.
Section snippets
Predictive Learning Analytics (PLA) in higher education
As defined by the Higher Education Commission (2016, p. 53), PLA “can identify which students may not complete their degree on time or even hand in individual assignments, which is already being seen in the UK through the OU Analyse tool. Apart from the OU the Commission does not believe that any UK institution has made significant headway in this area”. Indeed, most LA studies to date have been focused on improving learning outcomes, while less than 6% of 252 reported studies used LA at a
Settings: OU Analyse (OUA)
OUA is a predictive system used to identify learners at risk of failing their studies (Fig. 1). OUA predicts on a weekly basis whether (or not) a given student will submit their next teacher-marked assignment. The OUA dashboard visualises predictive information about who is at risk of not submitting their next assignment for individual students, as well as VLE engagement, and assignment submission rates at the cohort level. It uses a traffic light system to pinpoint: (a) in red students at risk
RO1: To reflect, at a macro-level, on the use of PLA by detailing the degree of OUA usage, challenges, and aspects facilitating adoption, over a period of 4 years
In terms of RO1, the first step of this reflective account was to visualise the degree of OUA adoption over the last four academic years (2015/16–2018/19), in terms of (a) numbers of unique teachers, and (b) usage patterns of unique teachers over the course of a presentation. The number of teachers accessing OUA at least once increased considerably over the 4 years (see Fig. 2), with 52 teachers in 2015/16 and 1159 in 2018/19. Yet, the overall percentage of those accessing OUA out of those who
Discussion
This study described a large-scale implementation of a PLA tool called OUA at a distance learning university by reflecting on the macro-level of use, that is the challenges and factors facilitating implementation over a period of 4 years, as experienced by the research or project management team behind the project (RO1). Furthermore, this study analysed on a micro-level the diverse perspectives of eight teachers who used OUA in their courses (RO2). In relation to RO1, a set of factors were
Conclusions
In line with recommendations by two systematic literature reviews on LA (Ferguson & Clow, 2017; Viberg et al., 2018), an interdisciplinary project team of six authors reflected on the large-scale implementation of PLA over a period of 4 years. It detailed the complex and diverse perspectives of various stakeholders involved in the macro- and micro-levels of adoption. They showcased that an emergent bottom-up approach of PLA through a strong consultation process (Dawson et al., 2018) and support
Declaration of competing interest
None.
References (60)
The theory of planned behavior
Organizational Behavior and Human Decision Processes
(1991)- et al.
Factors influencing beliefs for adoption of a learning analytics tool: An empirical study
Computers & Education
(2013) - et al.
Examining the role of feedback messages in undergraduate students' writing performance during an online peer assessment activity
The Internet and Higher Education
(2015) The problem of teaching presence in transactional theories of distance education
Computers and Composition
(2016)- et al.
Learning analytics should not promote one size fits all: The effects of instructional conditions in predicating learning success
Internet and Higher Education
(2016) - et al.
Learning analytics to unveil learning strategies in a flipped classroom
The Internet and Higher Education
(2017) - et al.
Supporting teachers in guiding collaborating students: Effects of learning analytics in CSCL
Computers & Education
(2014) - et al.
Teacher regulation of cognitive activities during student collaboration: Effects of learning analytics
Computers & Education
(2015) - et al.
Mining LMS data to develop an “early warning system” for educators: A proof of concept
Computers & Education
(2010) - et al.
Predicting secondary school teachers’ acceptance and use of a digital learning environment: A cross-sectional study
Computers in Human Behavior
(2011)
The impact of learning design on student behaviour, satisfaction and performance_ a cross-institutional comparison across 151 modules
Computers in Human Behavior
A meta-analysis of e-learning technology acceptance: The role of user types and e-learning technology types
Computers in Human Behavior
How achievement emotions impact students’ decisions for online learning, and what precedes those emotions
Internet and Higher Education
The current landscape of learning analytics in higher education
Computers in Human Behavior
A multivariate approach to predicting student outcomes in web-enabled blended learning courses
The Internet and Higher Education
System, scholar, or students? Which most influences online MBA course effectiveness?
Journal of Computer Assisted Learning
Open learner models and learning analytics dashboards: a systematic review
Transforming qualitative information: Thematic analysis and code development
Developing a model and applications for probabilities of student success: A case study of predictive analytics
Open Learning: The Journal of Open, Distance and e-Learning
Braced for turbulence: Understanding and managing resistance to change in the higher education sector
Management
A practical model for the management of resistance to change: An analysis of political resistance in South Africa
International Journal of Public Administration
Student retention and learning analytics: A snapshot of Australian practices and a framework for advancement
Understanding accessibility as a process through the analysis of feedback from disabled students
User acceptance of computer technology: A comparison of two theoretical models
Management Science
Rethinking learning analytics adoption through complexity leadership theory
Examining instructor use of learning analytics
Research evidence of the use of learning analytics; implications for education policy
Where is the evidence? A call to action for learning analytics
Affordances and limitations of learning analytics for computer-assisted language learning: A case study of the VITAL project
Computer Assisted Language Learning
Implementing predictive learning analytics on a large scale: the teacher's perspective. Paper presented at the Proceedings of the Seventh International Learning
Analytics & Knowledge Conference. Vancouver, British Columbia, Canada
Cited by (65)
Stakeholders’ insights on learning analytics: Perspectives of students and staff
2022, Computers and EducationCitation Excerpt :There has been an increase in the development of student and staff-facing Learning Analytics Dashboards (LAD) in HE. However, analysis of LAD usage has evidenced low engagement, sporadic usage, and a lack of evidence of impact (Corrin & Barba, 2015; Dawson, Joksimovic, Poquet, & Siemens, 2019; Herodotou et al., 2020; Herodotou et al., 2017; Kollom et al., 2021). While data visualisations are reported to be intuitive regarding the information being presented, the difficulty arises in mapping that data to something actionable within a person's own learning or teaching context.
Factors and conditions that affect the goodness of machine learning models for predicting the success of learning
2022, Computers and Education: Artificial IntelligenceCitation Excerpt :Aldowah et al. systematize 402 studies on data mining and ML models in education (Aldowah et al., 2019), and Pedroza et al. systematize 63 studies on the topic (Pedroza et al., 2019). These reviews are not exhaustive, and even since their publication, a large body of literature on the topic has been published (Herodotou et al., 2020; Chasoy et al., 2020; Tamada et al., 2019; Etemadpour et al., 2019; Mduma et al., 2019; Hassan, 2020; Dinh et al., 2021; Barramuño et al., 2021; Bujang 2021, Ali et al., 2021; Csalódi & Abonyi, 2021; Roslan, 2021; Queiroga et al., 2020; Kabathova & Drlik, 2021; Jin, 2020; Dharmawan et al., 2018; Perez et al., 2018; Ahmed et al., 2019). The literature identifies and categorizes a number of factors affecting student success according to different aspects.
Reimagining post-COVID distance education support services with achievement motives and career personality
2024, International Review of EducationExploring and comparing pedagogical beliefs of university instructors in relation to their behavioural patterns regarding learning management system use
2024, Behaviour and Information TechnologyIntegrated model for asynchronous learning and predictive analytics for enhanced learner experience
2023, Emergent Practices of Learning Analytics in K-12 ClassroomsGathering data through learning analytics: Will this methodological adaptation for understanding student engagement be a panacea or a panopticon?
2023, Researching the New Normal Social World: Methodological Adaptations and Innovations Post-Pandemic