Lockdown Learning: Changes in Online Foreign-Language Study Activity and Performance of Dutch Secondary School Students During the COVID-19 Pandemic

1 Introduction

The COVID-19 pandemic has led to school closures around the world. As schools shifted to distance learning in early 2020, teachers were forced to swiftly revise their teaching methods (Hall et al., 2020; Lorente et al., 2020; Mohan et al., 2020). The rapid transition has caused widespread concern about compromised learning and mental health in children (UNICEF, 2020). Indeed, initial reports have suggested learning decrements Engzell et al. (2020), Kuhfeld et al. (2020) and adverse mental health effects (Bignardi et al., 2020; Pearcey et al., 2020; Wright et al., 2020) in primary school students, compared to preceding years. In many cases, the shift to distance learning also reinforced existing inequalities: students from disadvantaged backgrounds were found to suffer greater learning setbacks than their more advantaged peers (Alvi and Gupta, 2020; Sevilla et al., 2020; Dietrich et al., 2020; Doyle, 2020; Engzell et al., 2020), and there were reports of disparities in access to resources and preparedness of instruction materials required for distance learning between educational levels and schools (Bol, 2020; Mohan et al., 2020; van de Werfhorst et al., 2020).

While many studies have reported on pandemic-related learning losses in primary education, relatively little is known about the impact of the pandemic on older students. Surveys of secondary school students have suggested that these students spent significantly less time on school work during lockdown (Anger et al., 2020; Grätz and Lipps, 2021), and had difficulty concentrating at home (de Haas et al., 2020). In higher education, preliminary results point to a similar drop in motivation and effort, but indicate that academic performance was unaffected or even somewhat improved (Gonzalez et al., 2020; Jacques et al., 2020; Meeter et al., 2020). These findings suggest that learning losses may vary with student age—perhaps as a function of students’ developing ability to engage in self-regulated learning (Paris and Newman, 1990).

One factor that likely contributed to variation in the extent to which students were affected by school closures is the availability of digital educational materials and online learning tools (Ferri et al., 2020; Hall et al., 2020; Klapproth et al., 2020). German secondary school students reported spending more time on school work if their school offered more digital learning materials (Anger et al., 2020). A study of French university students showed that, in a course redesigned around appropriate online tools, distance learning achieved similar learning outcomes to a course taught in person (Jacques et al., 2020). Offering online adaptive practice software as part of the curriculum could even benefit students, as a study of Dutch primary schools showed that including such software in mathematics education led to students performing better, not worse, than they would otherwise have (Meeter, 2021). Dutch educators surveyed before and during the lockdown reported seeing the benefits of digital learning tools, remarking on their efficiency and ability to offer a personalised learning experience in particular (van der Spoel et al., 2020). The development of suitable digital materials and learning tools may thus mitigate (some of) the impact of school closures on learning.

To explore this further, we studied the use of online learning tools in secondary education in the Netherlands during the spring 2020 school closures. Educational systems—and the effects of the pandemic on these systems—differ substantially between countries (e.g., Crawford et al., 2020; Loima, 2020; Toquero, 2020; Assunção Flores and Gago, 2020; Wu, 2021; Watermeyer et al., 2021). Robust national-level empirical studies are therefore essential in understanding the impact of COVID-19 in the unique context of each country. The Netherlands was relatively well-prepared for distance learning in terms of technological infrastructure (Engzell et al., 2020); the regular curriculum already included digital components, and, by-and-large, students had access to the necessary tools at home (de Haas et al., 2020; Bol, 2020). Nevertheless, the sudden lockdown meant that schools and teachers had little time to prepare for the switch to fully-online teaching. In many cases, scheduled lesson hours were reduced or shortened, and more emphasis was placed on digital components of the curriculum (Voogel, 2020). Dutch teachers reported having to quickly reevaluate and adapt their teaching methods, which was particularly challenging for those less familiar with educational technology (van der Spoel et al., 2020). Furthermore, a survey of students’ parents indicates that, in secondary education, school involvement differed between educational levels: compared to students in the pre-vocational track (vmbo), students in the higher pre-university track (vwo) were more likely to have digital classes, a structured lesson programme, and schoolwork checks (Bol, 2020).

The online learning tool that we studied enables self-regulated, autonomous rehearsal of foreign-language vocabulary through retrieval practice, a study method requiring learners to actively recall information (Roediger and Butler, 2011). The tool was accessible to a large number of students in different year groups (12–16 years old) and educational tracks (pre-vocational, general secondary, and pre-university), and was already widely used before the lockdown. This enabled us to compare usage and performance during the school closure period to measurements from earlier in the same school year and from the same period in the year before. As such, the data collected from this tool provides valuable insight into the effects of lockdown on Dutch secondary students’ foreign-language learning.

In this paper, we use the collected data to address two research questions:

1. Did usage of the retrieval practice tool change during the lockdown period?

2. Did retrieval practice performance change during the lockdown period?

The analyses that we report are limited to the domain of the online learning tool, which means that other components of the curriculum and learning environment are left out of consideration. Nevertheless, these results, when combined with other studies addressing the impact of lockdown on other facets of education, bring us closer to a complete picture of the effects of distance learning on education.

2 Method

We recorded the online retrieval practice activity of a large sample of secondary education students in the Netherlands during two consecutive school years (18/19 and 19/20; both school years lasting from 1 August to 31 July). The sample includes students from each of the three educational tracks in Dutch secondary education—pre-vocational (vmbo), general secondary (havo) and pre-university (vwo)—and from year groups 1 (age: 12; corresponds to grade 7) through 4 (age: 16; corresponds to grade 10).

Students in the sample used SlimStampen, an online, adaptive retrieval practice tool made available to them through the educational publisher Noordhoff Uitgevers as part of the foreign language learning curriculum for English and French. Retrieval practice is a well-established and effective study method in which learners are prompted to actively recall the learning material (Roediger and Butler, 2011; Rowland, 2014). The workings of the retrieval practice tool used here are described in detail in Sense et al. (2016); van der Velde et al. (2021); van Rijn et al. (2009). The tool enabled students to rehearse the course material through retrieval practice sessions, both upon their teacher’s instruction and of their own volition. Sessions consisted of a sequence of trials in which students rehearsed a set of foreign vocabulary items by answering retrieval prompts in various formats. For example, a student practising French vocabulary might see the Dutch prompt waarom (English: why) and be asked to retrieve and type its French translation pourquoi. Data recorded in each trial included the response accuracy and response time, as well as information about the question format, the studied item, and the corresponding textbook chapter.

Approval to analyse the anonymised activity data was granted by the Ethics Committee Psychology of the University of Groningen (study code: PSY-1920-S-0397).

3 Results

Over the course of two school years and across both courses, a total of 133,450 students completed 115,232,555 trials (18/19: 52,917,284 trials; 19/20: 62,315,271 trials). Supplementary Table 1 gives a breakdown of the data in the sample by course and school year.

We addressed the two research questions outlined in the Introduction by first looking at usage of the retrieval practice tool over time, and then investigating how study performance changed during lockdown.

All analyses were conducted in R (version 3.6.3; R Core Team, 2020). Regression models were fitted using the lme4 package version 1.1–21; Bates et al. (2015) and the lmerTest package (version 3.1–0; Kuznetsova et al., 2017). The analysis code is available at https://osf.io/t25fe/. Additional figures and tables are available in the supplement to this article. Wherever analyses are split by year group and educational track, we only show the French results in the main article. Results for English-language study are included in the supplement.

3.1 Study Activity

To identify changes in usage of the retrieval practice tool, we looked at the frequency of study trials over time, as well as the time of day at which students were most active.

3.1.1 Study Frequency

Figure 1 shows French-language study activity during both school years in terms of the total number of practice trials completed per week, stratified by year group and educational track. It also shows the percentage change in trial count during the whole lockdown period, marked in grey, relative to the same period in the previous school year. Across all strata, there was a notable increase in study activity during the lockdown period; in the most extreme cases, the number of completed trials grew almost nine-fold. Usage increased more strongly in higher year groups, where baseline usage was much lower. The increased usage persisted after schools started to reopen in June 2020. Similar patterns were found in English-language study activity (see Supplementary Figure 1).

FIGURE 1. Comparison of French-language study activity during a regular school year (18/19) and the school year affected by COVID-19 (19/20), stratified by year group (columns) and education track (rows). Activity is measured through the number of trials completed per week. The shaded area marks the school closure period in the 19/20 school year. The change in activity during this period is shown as a percentage in each plot.

3.1.2 Study Timing

The effects of distance learning during lockdown were also visible in the time of day at which students were actively using the retrieval practice tool. Figure 2A shows how study activity was distributed over the week during the school closure period, as well as during the same period in the preceding school year. The year-on-year change is depicted in Figure 2B, which shows that study activity during lockdown shifted towards the weekday mornings, with less activity being recorded on weekday afternoons and on Sundays.

FIGURE 2. (A) Distribution of study trials over the week during the school closure period (16 March–2 June) in both school years. The colour of each square shows how many trials were recorded in a learning session starting within a particular hour on a particular weekday, as a proportion of the entire week. (B) Change in distribution of study trials from the 18/19 school year to the 19/20 school year.

3.2 Study Performance

We assessed the effects of distance learning on study performance in two ways. Firstly, we tracked students’ progression through the study materials in aggregate to identify changes in study pace. Secondly, to identify more immediate behavioural effects, we looked at response accuracy and response time at the level of individual trials.

3.2.1 Progress Through Materials

Figure 3 visualises students’ progress through the textbook chapters of the French course over the school year. Supplementary Figure 2 in the online supplement shows the same for English. The time plots show the weekly share of trials that correspond to each chapter per year group and track in both school years. Gaps indicate weeks in which no trials were recorded. There was generally a smooth progression over time from one chapter to the next; a trend that appeared to continue during the lockdown.

FIGURE 3. Progress through the French materials over time: the share of study trials pertaining to each textbook chapter as a percentage of all trials completed in a week. The school closure period in the 19/20 school year and the corresponding period in the previous year are marked by a dashed border. The change in trial share during this period is shown, per chapter, below each set of plots, where the dark-grey and light-grey bands indicate the variability between the two school years in the period preceding the lockdown (1 SD and 2 SD, respectively).

To measure the effect of distance learning on progress, we performed pairwise comparisons of the chapter distribution (i.e., the proportion of trials associated with each chapter) during the lockdown period in the 19/20 school year and the same period in the preceding school year using a chi-square test of homogeneity. This test was done separately for each combination of year group and education level. In all cases, we found that there was a significant year-on-year change (all p < .001). However, the observed changes in trial share were often within the bounds of typical fluctuations between school years, as a comparison to changes outside the lockdown period indicated. We calculated the year-on-year change in trial share per chapter over the period preceding the lockdown, using a sliding window of equal duration to the lockdown period. The change plots in Figure 3 and Supplementary Figure 2 show the spread of these typical changes as two grey bands, extending to 1 SD and 2 SD, respectively. For a large part, the changes in trial share during the lockdown period, shown as points, fall within this range. This means that, while there were statistically significant changes in progress during the lockdown period, many of these changes were comparable to typical year-on-year fluctuations.

Whenever there were unusually large changes outside the typical range, these sometimes pointed to students being behind the previous year’s schedule (i.e., one chapter’s trial share was higher than the year before while that of a subsequent chapter was lower), and sometimes to students being ahead of schedule. The tendency to be behind or ahead in these cases appeared to differ somewhat between educational tracks: for both French and English, pre-university (vwo) students were ahead of schedule in five year groups and on schedule in one group; general secondary (havo) students were ahead in two year groups, on schedule in three, and behind in one; pre-vocational (vmbo) students were ahead of schedule in two year groups, on schedule in three, and behind schedule in two groups.

3.2.2 Trial-Level Performance

Trial-level performance was measured through response accuracy and response time. These variables were analysed separately for multiple choice questions, which only required the student to select the answer from a set of options, and open answer questions, which required the student to type the answer. The English-language study data contained virtually no open answer questions (see Supplementary Table 1), so only multiple choice performance is reported for English. We saw similar changes in trial-level performance across educational tracks and year groups; here we report performance at the population level.

Figure 4A shows mean response accuracy over the course of the school year. Accuracy on multiple choice questions was consistently high in both courses, a pattern that continued during the lockdown period in the 19/20 school year. Accuracy on open answer questions, however, did change over the course of the 19/20 school year, increasing by about six percentage points from the period before the lockdown (M = 77.48%, SD = 17.76%) to the lockdown period itself (M = 83.70%, SD = 15.17%) and remaining higher after schools had started reopening (M = 81.90%, SD = 18.16%). These patterns were confirmed by a binomial generalised linear mixed-effects model fitted to students’ daily accuracy scores (see Supplementary Table 3). Due to the size of the data set, the model found significant changes in accuracy over time for both question types, but there was only a meaningfully large effect size for open answer questions in the lockdown and post-lockdown periods of the 19/20 school year.

FIGURE 4. Comparison of study performance over time during a regular school year (18/19) and the school year affected by COVID-19 (19/20). (A) Mean response accuracy ( ± 1 SE). (B) Mean response time (± 1 SE) on correct answers.

There were similar trends in the response time measured on correct answers, as Figure 4B shows. While response times on multiple choice questions remained reasonably stable across both school years, response times on French open answer questions did change substantially, increasing by about 0.25 s from the pre-lockdown period (M = 2.44 s, SD = 4.58 s) to the lockdown (M = 2.71 s, SD = 3.58 s) and the period that followed (M = 2.82 s, SD = 8.73 s). These changes were confirmed by a generalised linear mixed-effects model (see Supplementary Table 4). Although we cannot be certain about the cause, the higher response time and accuracy may indicate that students experienced less time pressure or were more focused on the task at home than at school, typing out their responses more carefully.

4 Discussion

Our results show that the shift to distance learning during the COVID-19 lockdown of spring 2020 coincided with an increase in usage of an online retrieval practice tool by students in Dutch secondary education, with activity increasing on weekday mornings in particular. In general, we found little evidence of major study delays as a result of the lockdown; students in the highest educational track were likely to be on or even somewhat ahead of the previous year’s schedule, and students in lower tracks were also roughly on schedule. Students’ trial-to-trial learning performance was, if anything, better during the lockdown period than the year before. In particular, we saw higher accuracy and higher response times on open answer questions, perhaps indicating that students experienced less time pressure or were better able to focus at home.

The online learning data reported here offer a detailed look at day-to-day changes in behaviour of a large and diverse group of students. Of course, observing students’ activity in a single online learning tool provides only a limited view into their learning, and students who do not use the tool (anymore) fly completely under the radar. A more complete analysis of the effects of lockdown should also consider other components of the curriculum and students’ environment. As previous work has shown, there are likely to be differences among school subjects, students, and schools in how much learning fell behind during lockdown (Engzell et al., 2020; Lek et al., 2020; Maldonado and Witte, 2020; Meeter, 2021). Identifying the causes of such differences can be helpful in effectively combating the negative consequences of lockdown. Analyses like the current one contribute to this goal. In addition, it is possible that the initial sudden switch to distance learning caused certain learning behaviours to persist out of inertia. It would therefore be valuable to compare our current findings to data from the second lockdown in the Netherlands, during the winter and spring of the 20/21 school year, at which point schools and students would have had more time to adapt to distance learning.

Because of differences between countries in educational systems, access to education and technology, and responses to the COVID-19 pandemic, it is important to interpret these findings within their national context. Compared to other countries, the Netherlands was relatively well-prepared for distance learning, with good availability of the necessary technological infrastructure for digital education (Engzell et al., 2020; de Haas et al., 2020). Technological hurdles related to the accessibility of digital resources have been much more problematic in developing and newly industrialised countries (Alvi and Gupta, 2020; Owusu-Fordjour et al., 2020; Lorente et al., 2020). Nevertheless, even in the Netherlands access to appropriate digital educational materials and online learning tools was not universal, but was found to vary depending on socio-economic factors (Bol, 2020), teachers’ familiarity and experience with these resources (van der Spoel et al., 2020; Voogel, 2020), and, in secondary education, students’ educational track (Bol, 2020). While our findings are based on a large sample of Dutch secondary school students, they offer only limited insight when it comes to such issues of accessibility. They depend on students actively using a particular online retrieval practice tool, and no further information about the students, teachers, or schools was recorded. We did observe slight differences between educational tracks in pre-lockdown usage of the online retrieval practice tool, but usage increased fairly uniformly during the lockdown. We therefore did not see evidence in our sample of disparities in technological preparedness between educational tracks that Bol (2020) reported.

Where studies have reported decrements in learning during the school closures, these could be caused by both lower quantity of learning, and lower quality. Surveys have already shown that secondary school students, on average, spent much less time on school-related work during the lockdown than they would normally do (Anger et al., 2020; Grätz and Lipps, 2021). In contrast, we found an increase in usage of the retrieval practice tool. However, learning activity increased during what would normally be students’ regular school hours, suggesting that retrieval practice may have replaced other parts of the language-learning curriculum, rather than being added on top of it. The retrieval practice tool we studied addresses the learning of vocabulary, which is only one part of learning a foreign language. It may be that students focused on vocabulary learning, which may be relatively well-suited for distance learning, to the detriment of other aspects of language learning that could not be done as easily from home, in effect leading to a narrowing of the curriculum (Voogel, 2020). Our results do suggest that in the specific case of vocabulary learning, the quality of learning remained constant or improved slightly, as indicated by normal or above-normal progress through the chapters of the textbook and stable or increasing response accuracy.

This work contributes to a growing literature aiming to understand the positive and negative impact of distance learning on students and to identify the factors that may amplify positive effects or mitigate negative effects. The reported findings offer a retrospective view of the impact of lockdown in a specific educational context that, together with similar studies on other aspects of education, can help shape future educational policy (World Bank, 2020). Online learning tools such as the one discussed here have the potential to offer an efficient, personalised learning environment and give immediate and detailed insight into students’ learning. This can be very helpful for teachers and curriculum designers, particularly when regular classroom teaching becomes impossible.

Author’s Note

The online retrieval practice tool SlimStampen is licensed to Noordhoff Uitgevers by the University of Groningen.

Data Availability Statement

The datasets presented in this article are not readily available because of the terms of the data processing agreement between Noordhoff Uitgevers and the University of Groningen. Requests to access the datasets should be directed to HR, d.h.van.rijn@rug.nl.

Ethics Statement

The studies involving human participants were reviewed and approved by Ethics Committee Psychology of the University of Groningen. Written informed consent from the participants’ legal guardian/next of kin was not required to participate in this study in accordance with the national legislation and the institutional requirements.

Author Contribution

MV performed the data analysis and wrote the first draft of the manuscript. All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Conflict of Interest

Author RS was employed by Noordhoff Uitgevers.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2021.712987/full#supplementary-material

References