Improving Student Retention by Providing Targeted Support to University Students who do not Submit an Early Assessment Item. A Practice Report

Charles Sturt University is a regional university with six main campuses located throughout regional New South Wales, Australia. Fifty-six percent of domestic undergraduate students come from regional, rural, and remote (RRR) areas, 24% from low-socioeconomic (LSES) backgrounds, and 4% of students identify as First Nations. Data shows that RRR students are: “around 40% less likely to gain a higher-level tertiary education qualification and less than half as likely to gain a bachelor and above qualification by the time they are 35 years old, compared to individuals from metropolitan areas” (Department of Education, Skills & Employment [DESE], 2019). Even when RRR students overcome these barriers and enter tertiary education, they are less likely to complete their course (DESE, 2019). Established first year design principles highlight the need to provide early response systems for all students who appear to be disengaging using targeted communication regarding available support services (Kift, 2009; van der Meer et al., 2018). The significant number of students from non-traditional backgrounds at Charles Sturt highlights the need for innovative programs to support students in their transition to university study and improve progress and retention. It is essential that these support programs recognise the unique challenges that are faced by RRR and LSES students (Cardak et al., 2017; Nelson et al., 2011; van der Meer et al., 2018). The Charles Sturt University Retention Team has developed, tested, evaluated, and refined a retention model through 14 action-research cycles from 2017-2021. The project has expanded from a small pilot in one faculty to monitoring the engagement and submission of an early assessment item for over 70% of all commencing undergraduate students across the University. The Retention Model synergistically overlays curriculum design and student support and ensures academics embed best practice transition pedagogy and learning engagement activities into key first-year subjects. By monitoring the submission of early assessment items, the team can accurately identify and proactively contact students who are not engaged in their studies prior to their first census date. Every aspect of this program supports equity student groups that are over-represented at our regional university. This work has significantly improved commencing progress rates across the institution.


Introduction
Charles Sturt University is a regional university with six main campuses located throughout regional New South Wales, Australia. Fifty-six percent of domestic undergraduate students come from regional, rural, and remote (RRR) areas, 24% from low-socioeconomic (LSES) backgrounds, and 4% of students identify as First Nations. Data shows that RRR students are: "around 40% less likely to gain a higher-level tertiary education qualification and less than half as likely to gain a bachelor and above qualification by the time they are 35 years old, compared to individuals from metropolitan areas" (Department of Education, Skills & Employment [DESE], 2019). Even when RRR students overcome these barriers and enter tertiary education, they are less likely to complete their course (DESE, 2019). Established first year design principles highlight the need to provide early response systems for all students who appear to be disengaging using targeted communication regarding available support services (Kift, 2009;van der Meer et al., 2018). The significant number of students from non-traditional backgrounds at Charles Sturt highlights the need for innovative programs to support students in their transition to university study and improve progress and retention. It is essential that these support programs recognise the unique challenges that are faced by RRR and LSES students (Cardak et al., 2017;Nelson et al., 2011;van der Meer et al., 2018).
The Charles Sturt University Retention Team has developed, tested, evaluated, and refined a retention model through 14 action-research cycles from 2017-2021. The project has expanded from a small pilot in one faculty to monitoring the engagement and submission of an early assessment item for over 70% of all commencing undergraduate students across the University. The Retention Model synergistically overlays curriculum design and student support and ensures academics embed best practice transition pedagogy and learning engagement activities into key first-year subjects. By monitoring the submission of early assessment items, the team can accurately identify and proactively contact students who are not engaged in their studies prior to their first census date. Every aspect of this program supports equity student groups that are over-represented at our regional university. This work has significantly improved commencing progress rates across the institution.
The first census date for a commencing student represents a critical juncture in the academic journey; for both the institution and the student. Attrition rate is a measure of the proportion of students who withdraw from study after the first census date. Once a student passes this first census date and then withdraws from study, they are counted as attrited. There is a short period between the students accepting their offer and the first census date for universities to identify disengaged or under-prepared students, and to provide the relevant support and advice (Linden & Webster, 2019). The census date is required to be at least in the first quarter of the semester at Australian universities. For a normal semester of approximately 12-14 weeks, census is most commonly between weeks 4 and 6. There has been much anticipation over the last 10 years that the use of predictive learning analytics in higher education will transform the identification of disengaged students to allow for more timely interventions (UNESCO, 2012). Many studies have described complex algorithms and models that predict, with variable levels of accuracy, the likelihood of a student failing their subjects or leaving university (Foster & Siddle, 2020;Tempelaar et al., 2018;UNESCO, 2012). One constant throughout these studies is the difficulty in modelling the myriad of interactions between learners and learning design as well as the challenge, possibly insurmountable, in finding a one-size-fits-all model to apply at scale. It is also incredibly difficult to identify disengaged commencing students in the first few weeks of university as there is simply not enough learning data collected. Even in a relatively homogenous course, such as Medicine, only just over half of the students who would later fail were identified in week 6 of a 14 week semester (Dewar et al., 2021). Like most universities, Charles Sturt had developed an "at-risk" model using statistical modelling from a large dataset that included student demographic features. Unfortunately, this model had limited effectiveness in identifying students who required support in the critical pre-census period. One reason for this is in the early weeks of a commencing semester is there is little opportunity to include academic performance or history in the model so it relied heavily on demographic data.
One simple model with promising results is to monitor the submission of early assessment items and offer targeted support to those who do not submit (Cox & Naylor, 2018;Linden & Webster, 2019;Nelson et al., 2009). Students frequently use assessment to define what is important in the curriculum (Kift, 2009). Early assessment can help to engage students with the subject in the early weeks of the teaching semester and provides an opportunity for students to complete tasks that help transition from previous educational experiences into university. Early assessments can also help identify students who are not engaged and are thus at risk of failure or withdrawal (Cox & Naylor, 2018;Kift, 2009;Nelson et al., 2009), however to date, this has not been attempted at scale. This practice report will detail the results of the successful expansion of the Charles Sturt University Retention Team to monitor the submission of a pre-census, early assessment item of over 70% of commencing undergraduate students.

Developing the Retention Model
In 2017, the Retention Team built a collaborative and integrated process that synergistically overlaid the curriculum and student support in the critical pre-census period. An action-research framework was developed, to refine and expand the Retention Model with the aim to increase domestic undergraduate commencing progress rates and student retention. We hypothesised that genuine integration of the curriculum and support services in the critical pre-census window will improve student progress rates and success. The program has undergone 14 cycles of action research which correspond to the 14 teaching sessions where the team has been able to test, refine, and exponentially expand its reach. From humble beginnings as a pilot in the Faculty of Science, monitoring 20 subjects, the project now monitors approximately 200 subjects a year for student engagement precensus. By selecting these subjects the project monitors approximately 16,000 enrolments per year pre-census, across the three main teaching sessions (16,536 in 2019 and 15,711 in 2020). Most subjects selected are large, level 1 1 subjects that as part of the project also receive priority access to educational designers, as well as library and academic skills support.
In 2020, in response to COVID-19 and the need for a rapid pivot to online learning, the project expanded to monitoring assignment submission of all undergraduate subjects from weeks 6-14 of each of the three main sessions. Approximately 5,110 students from 587 subjects were contacted during this time. To achieve the scale of reach the team collaborated with divisional staff from across the university to build automated data pipelines. Accurate and up-to-date access to student enrolment status, student subject enrolments, learning management system (LMS) activity data, grade data from the LMS (grade centre), and the assignment submission software have allowed for automation of much of the work. Although the Retention Model itself was informed by years of practical teaching experience and the scholarly literature (Cox & Naylor, 2018;Nelson et al., 2009); since 2017 many hypotheses have been tested and refinements made in response to both quantitative and qualitative data. Ethics approval was received from the Charles Sturt Human Ethics Committee (HREC Protocol No H21170). Below are five examples that illustrate how this approach has tested and refined every aspect of the model.

Are learning analytics alone sufficient to identify disengaged students?
Non-submission of an early assessment item is an accurate trigger of student disengagement. Nuances in student behaviour make it incredibly difficult to identify disengaged students pre-census using learning analytics alone. Exploration of the available analytics has shown that no access for a 10-day period flags the fewest false positives which supports the findings of Foster & Siddle (2020). It does, however, miss a high proportion of disengaged students. When the time is decreased to six days of no access, a high proportion of engaged students are incorrectly contacted. The average progress rate for the subjects which included a pre-census assessment task (n=16) increased from 70% (in 2017) to 83% (in 2018; p<0.05), while those without an early assessment (n=9) had a non-significant increase from 69% to 73% (Figure 1). This has been tested in subsequent sessions with the same result; showing that it is difficult to accurately identify a high proportion of disengaged, commencing students in the pre-census period with learning analytics alone van der Ploeg et al., 2020).

Figure 1
Progress Rates in Subjects With and Without an Early Assessment Item * p<0.05

Are academics or support staff more effective in contacting disengaged students?
In 2017 submission of a pre-census 5% test was monitored in an online subject with 227 first-year students. The 58 students who did not submit the early assessment item were randomly divided into two groups to investigate who should be contacting disengaged students. All 58 students were phoned twice and then sent an email; 30 were contacted by the subject coordinator and 28 by the Outreach Team. Seventyseven percent of students who spoke with the Outreach Team modified their enrolment while only 22% of those who spoke to the subject coordinator took action. The subject coordinator reported difficulty in explaining the process of changing enrolment to students and felt they were much better suited to supporting students in their studies. Of the 36 students who remained in the subject, 35 did not submit any assessment items and one student submitted a single assessment item. Furthermore, 14 of the students did not access the LMS at any point during the session. This certainly confirms that the non-submission of an early assessment item is an accurate early trigger for disengagement.

Does SMS messaging improve contact rates with students?
In 2019 the Retention Team tested whether the addition of twoway SMS prior to the call increased the call success rate. In response, the call success rates lifted from 20% to 37%. This was further improved to ~ 55% successful dialogue when students were sent a phone number to call via SMS. Disengaged students are notoriously difficult to contact and the following sequence has been found to be the most successful: first phone call, twoway SMS, second phone call followed by an email if no contact has been made. The most likely time for students to answer the phone is from 3-5pm and call success rates are consistently very low in the morning.

How can the team most effectively on-board time poor academic staff?
The success of this program is reliant on the involvement of time-poor continuing and sessional academics. The team tested various communication and engagement strategies and found that clear timelines, drop-in zoom meetings, emails with embedded videos, infographics, streamlined forms, and customised reports at the subject level delivered the best results. The team offers mentoring and advice with the ability to direct academics to all areas of the University.

How can the team maximise outreach success?
The Outreach Team has consistently provided feedback to the Retention Team that contacting students because of missed early assessment deadlines rather than low LMS activity results in betterquality conversations. Students are more likely to 'open up' and enter a conversation that allows the Outreach Team to help the students re-engage. The Outreach Team members look forward to making calls about missed assessments for these reasons. Automated processes have been developed and the Outreach Team is provided with rich data via the centralised customer relationship management (CRM) to inform the phone call (Figure 2a). Following successful dialogue with a disengaged student, the Outreach Team records the conversation in the Retention Teams database. The word cloud in Figure 2b highlights the main reasons for non-submission of an assessment task; whilst Figure 2c shows the predominant course of action proposed by the student.

The Retention Model
After much refinement based on project data and the broader literature, the following Retention Model (Figure 3) accurately identifies disengaged commencing undergraduate students prior to their first census date. Approximately 6-12 weeks prior to the commencement of the teaching session (pre-session) the team sends out individual emails to subject coordinators. As the list of subjects monitored has become quite stable, approximately 75% of academics in a given session have previously participated in the project. Academics who are new to the project are provided with a more personalised, descriptive email with links to resources and an opportunity for a one-on-one meeting with a member from the Retention Team, the Library, Academic Skills, and/or an Educational Designer. Where possible an early assessment item, valued between 5-20%, is included in each subject. The optimal timing of this due date is from the end of week 2, so that most students are settled in their subjects, up until 3 days before census (week 4), ensuring there is time to make contact and for the students to make a decision about study before census.

Figure 3
The

Charles Sturt Retention Model
At the beginning of the first week of the session, academic staff are asked to communicate the subject expectations to students, such as the times and dates of any compulsory classes, workplace learning, or practical requirements of the subject. This information is available in the course brochure, the subject handbook, and the subject outline, however, it is important to make this explicit to commencing students.
Midway through week 2, students who have not accessed the LMS since the first day of the session are sent a single email through the CRM that lists the subjects they have not accessed. Many students are still finalising their enrolment at this time, so the email is very friendly and provides links to the LMS, student portal, and student support.
In Weeks 3 and 4 of session, disengaged students are identified either due to non-submission of an early assessment item or low LMS activity. The day after an early assessment item is due, a form is emailed to academics with a list of enrolled students and the names pre-ticked of those who have not submitted either via the electronic assignment submission tracking system (EASTS) or the LMS. The academic can untick students who have been in contact and they can tick the names of students they are concerned about. Customised comments can be added to individual students or the whole class. The data from the form, combined with past student performance from the Retention Team database, is then collated and uploaded to the CRM to inform the conversations by the Outreach Team. Just over 70% of all commencing students are monitored in this way. Inevitably, despite best efforts, some subjects will not have a pre-census assessment item and a standard 10 days of no LMS activity is used to identify disengaged students in these subjects. Low LMS activity monitors approximately 10% of commencing students. On the Monday post-census, the team identifies students who have not accessed the LMS since the first day of the session and have missed at least one assessment item. The students are sent an email with five days to respond before their enrolment is cancelled and a small number (~20%) of students speak to someone about their options, no student has thus far decided to remain enrolled.
The project has reduced the number of students who receive a zero-fail grade which has significantly improved subject progress rates, university-wide official commencing progress, and first-year attrition rates. In the Faculty of Science pilot in 2018, there was an increase in the average subject progress rate from 66% to 79% (Figure 4a). This impact has been sustained and scaled, and in 2020, a total of 83 subjects were flagged as having progress rates below 80% in the previous year. The support offered to these subjects by the Retention Team resulted in a significant increase in subject progress rates of 71.8% to 77.9% ( Figure  4b).

Figure 4
a. Significant Increase in the Average Subject Progress Rate in 2018 in the Faculty of Science, n= 25 Subjects b. In 2020, n=83 Subjects *p<0.0001

Concluding Discussion
There is no doubt that the work of the Retention Team has had a sustained and positive influence on student learning and the student experience. By correctly identifying and contacting disengaged students in the critical pre-census window, students are supported to succeed or assisted to alter their enrolment. Since the program commenced in 2017, over 10,100 students enrolled in 1,006 subject offerings have been identified as disengaged and offered targeted support. The expansion of the Retention Team has led to widespread benefits for students (by way of support), staff (who have been supported by professional development, priority access to learning design, and a reduction in the outreach work they would otherwise perform), and the institution (with improved commencing progress rates and increased retention). In response to the COVID-19 pandemic, in 2020 the program rapidly expanded to monitor assignment submission of all undergraduate subjects in the post census period. This was only possible because of the ability of the team to automate data pipelines and connect academics and professional staff in ways that had not previously been done.