Evaluation of Educational Loss in Europe and Central Asia

The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.


Policy Research Working Paper 10542
To what extent has the learning progress of school-aged children slowed during the COVID-19 pandemic? A pre-registered analysis of the first international assessment to be published since the pandemic is conducted to estimate the impact of COVID-19 on student reading. The effect of closures on achievement is modeled by predicting the deviation of the most recent results from a linear trend in reading achievement using data from all rounds using data from 28 countries in Europe and Central Asia. Reading scores declined by an average of 20 percent of a standard deviation, equivalent to just less than a year of schooling. Losses are significantly larger for students in schools that faced relatively longer closures. While there are no significant differences by sex, it is shown that lower-achieving students experienced much larger losses. This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at hpatrinos@worldbank.org.

Introduction
There is a learning crisis (World Bank 2017). It existed before COVID-19 (Angrist et al. 2021(Angrist et al. , 2022 and it is global. It has been exacerbated since the school closures began in March 2020 (McKinsey 2022).
COVID-19 caused significant disruption to the global education system. Early in the pandemic, the World Bank (Azevedo et al. 2021) presented simulations of the potential effect of school closures on learning outcomes. It forecast significant declines in the global level of schooling and learning outcomes. School closures could result in a loss of 0.3-1.1 years of schooling and students from the current cohort could, on average, face a reduction of $366 to $1,776 in yearly earnings. Globally, a school shutdown of 5 months could generate learning losses that have a present value of $10 trillion.
Early reviews of the first wave of lockdowns and school closures suggested significant learning loss in a few countries (see, for example, Engzell et al 2021;Haelermans et al. 2022;Jack et al. 2022;Maldonado and De Witte 2022). More recent and thorough analyses of recorded learning losses documented since the beginning of the school closures in March 2020 find even more evidence of learning loss (Donnelly and Patrinos 2021). Robust studies suggest learning losses on average amounting to 0.17 of a standard deviation, or at least one-third of a year's worth of learning (Betthäuser et al. 2023;Hammerstein et al. 2021;Patrinos et al. 2023;Zierer 2021). Most studies observe increases in inequality where certain demographics of students experienced learning losses that were more significant than others. But there are also outliers, countries that managed to limit the amount of loss.
The Progress in International Reading Literacy Study (PIRLS) is the largest international test of 4 th -grade students' reading achievement. It is the largest international assessment of reading skills at the primary school level. The 2021 edition was implemented when most countries reopened schools after closures related to the COVID-19 pandemic. Comparing the results from the previous editions conducted in 2001, 2006, 2011, and 2016, one can estimate the impact of the school closures and online teaching in the pandemic on student reading skills. Evidence from individual countries suggests that in many cases students in primary schools suffered larger education losses due to distant teaching in the pandemic. Moreover, the results suggest that the decline is larger among the low-achieving students. Finally, some research shows that the pandemic affected student attitudes, motivation, and well-being.
Negative consequences of the pandemic for students should be addressed through targeted policies aiming at groups of students with the larger losses and addressing their varying needs regarding attitudes and well-being. Changes in student performance might be driven by other factors than the pandemic and school closures. However, the COVID-19-related school closures are the only factor that is common to most countries.
Due to the emergency nature of the COVID-19 pandemic, education systems around the world are facing extreme disruption. At its peak, UNESCO (2020) reported that nearly 1.6 billion learners in more than 190 countries, or 94 percent of the world's student population, were impacted by educational institution closures. Given the abruptness of the situation, teachers and administrations were unprepared for this transition and were forced to build emergency remote-learning systems almost immediately. In response to this disruption, education researchers are beginning to analyze the impact of these school closures on student learning progress or lack thereof. The term learning loss is commonly used in the literature to describe declines in student knowledge and skills (Pier et al. 2021). Historic data provides researchers with information regarding where student learning should be year over year and is often measured through regular testing. Learning loss occurs when educational progress does not occur at the same rate at which it has historically compared to previous years.
School closures have a large, persistent, and unequal effect on learning. Long-term, nationwide school closures result in learning losses (see, for example, Belot and Webbink 2010). In turn, learning losses can lead to decreased future employment potential and lower earnings (see, for example, Chetty et al. 2014;Currie and Thomas 2001;Psacharopoulos et al. 2021). Online education is an imperfect substitute for in-person learning, particularly for children from low-income families. Peer effects also change: schools allow children from different socioeconomic backgrounds to mix, and this effect is lost when schools are closed. Another factor is the response of parents, some of whom compensate for the changed environment through their own efforts, while others are unable to do so. All this leads to the finding that school closures have a large, persistent, and unequal effect on human capital accumulation (Agostinelli et al. 2022). The channels operating through schools, peers, and parents all contribute to growing educational inequality during the pandemic.
As a result of the first wave of lockdowns and school closures beginning in March 2020, on average, a student has lost about one-third to a half a year's worth of learning. The magnitude of our findings appears to validate scenarios projected by the World Bank (Azevedo et al. 2021). This has consequences in terms of further education as the lost learning may impact opportunities for higher levels of schooling. There are also long-term future earnings losses associated with lost human capital, with students potentially losing trillions of dollars in future incomes.

Data
The just-released PIRLS 2021 data include results for 65 countries and benchmarking participants. This is the first international picture of student achievement after the COVID-19 pandemic.
Our analysis uses data from 24 education systems of the European Union (22 countries and separate data for the Flemish and French parts of Belgium) and for an additional 7 countries when providing results for all ECA+ countries. We compare PIRLS results from assessment rounds in 2001, 2006, 2011, 2016 and 2021. Overall, our sample includes 582,112 students (459,039 in EU countries) participating in all rounds of PIRLS, representing data for nearly 30 million 4 th grade students across 20 years. The sample from 2021 -the post-COVID-19 dataincludes 157,167 students (120,895 in EU countries), representing the results of nearly 7 million 4 th graders in ECA+ countries (4 million 4 th graders in the European Union).
PIRLS data include the plausible values measuring reading achievement and information on student socio-demographic characteristics (e.g., age, grade, and gender). In addition, school principals surveyed by PIRLS provided details on how instruction was affected by COVID-19-related school closures. These data are available for 93% of the sample in ECA+ countries and 98% of the sample in EU countries. We also merge PIRLS data with information from UNESCO on the total number of weeks schools were closed fully or partially during the pandemic. These data are available for all EU countries, although they do not distinguish between primary and secondary levels. Thus, the UNESCO information might be misleading, as primary schools remained open in some countries while secondary schools were closed.

Empirical Strategy
We model the effect of school closures on student achievement by predicting the deviation of the most recent 2021 results from an estimated linear trend in reading achievement using data from all PIRLS rounds. We estimated each country's linear trend separately and included country-level fixed effects to control for unobserved time-invariant country characteristics. More formally, we estimate a linear regression model on pooled PIRLS data from different years, given by the following equation: where i denotes students, j schools, and k countries, with n equal to the number of countries. In this case, D equals 1 for 2021 data collected after the pandemic and zero for previous years, so captures the impact of the pandemic on student achievement. Country fixed effects are denoted by and country-specific time trends by , while is a vector of demographic variables (student age, grade, and gender).
In general, PIRLS surveys 4 th -grade students who are around ten years old at the assessment time. However, students might take the test earlier or later. Some are repeating, and some are advancing a grade. Also, some countries decided to test students in 3 rd , 5 th, or even 6 th grade. Moreover, while most countries sampled the same population over the years, some changed the target population. Finally, due to the pandemic, some countries postponed the assessment to the beginning of the fifth grade. To consider these differences, we took two approaches. In one, we additionally control for student age (10 dummies, one for each age decile, to allow for a non-linear age effect), student grade, and for the average age effect calculated as a mean age for each round and country. In the second approach, we re-estimated all regressions for countries from Wave 1 in PIRLS. These countries did not postpone the assessment due to the pandemic, and according to PIRLS organizers, their data are fully comparable with previous rounds without additional adjustments.
The pandemic could affect student results differently, not only through school closures. Disruptions could also affect health, economic and psychological situation, and general wellbeing. We distinguish these effects by comparing students in countries or schools closed for different periods. In this regard, we used information provided by principals on how their schools were affected by the pandemic. The baseline in the model is more than eight weeks of school closure, and we compare how they differed from a linear trend compared to schools that were closed for between 2 and 8 weeks and those affected for less than two weeks or not closed at all. We also used UNESCO-provided data on the number of weeks of school closures for each country. The estimated coefficient indicates how student results depart from the time trend depending on the school closure duration.
To investigate heterogeneity in the impact of school closures on achievement by gender and by achievement distribution, we introduced interaction with gender and ran the quantile regressions. To deal with many dummy variables and interactions, we use recent implementations of fast quantile regression algorithms (Chernozhukov et al. 2022).
PIRLS data are collected through a complex stratified survey with schools sampled as primary sampling units and classrooms with students sampled at the second stage. We used the jackknife replicate weights provided by the PIRLS database to estimate the sampling error. We use plausible values for reading achievement to adjust for the measurement error.
The paper also provides estimates for individual countries. We first compare achievement change between 2016 and 2021. Then we use a similar model as for the populations of students in EU or ECA+, but with individual interactions between country dummies and indicator for 2021 data. The general model is given by equation below: Which is similar to equation [1] but includes separate estimates of the time trend departure for each country. We estimate this model with and without controlling for age and grade. We also estimate a simpler model, assuming a common time trend for all countries, which is given by equation below: These simplified model allows obtaining country-specific estimates for countries, which has only data from two rounds of PIRLS or have missing or fully correlated data on age or grade. They can be also used to check if the country-specific results are robust to inclusion of changes in age or grade. If they vary across models, then the change over time in reading achievement should be interpreted with more caution as it could be driven by changes in the sampling or population definition in these countries. Table 1 provides the main results using data for students in all education systems in the EU, while Table 2 provides similar results for students in all ECA+ countries. Column 1 presents a raw estimate of the departure of achievement in 2021 from a country-specific time trend.

Learning loss estimates for the population of students in EU countries
Column 2 provides a similar estimate but adjusted for changes in age and grade. Columns (3) to (5) provide comparisons between schools that were more or less affected by the pandemic or countries with shorter or longer school closures.
According to the adjusted regression (column 2), on average, across EU countries, reading achievement is lower than predicted by a linear trend by -14.86 points. For ECA+ countries, a similar departure from the linear trend is estimated at -16.43 points. Typically, effects on educational achievement are expressed as percentages of achievement standard deviation. The average within-country standard deviation of reading achievement in 2021 is 74.8 for ECA+ countries, and 73.8 points considering only EU countries. Thus, the average learning loss for EU countries equals 20% of SD, while for all ECA+ countries it is 22%. The unadjusted estimate from column (1) is -9.11 points for EU and -15.27 for ECA+ (12% and 20%, respectively, of within-countries SD of reading achievement). However, this estimate can be biased by within-country changes in student age and grade.
Results in columns (3), (4), and (5) show that the decline in achievement is significantly larger for students in schools that faced relatively more extended closures. In schools that were affected for more than eight weeks (declared by principals), actual achievement was lower than expected by 18.8 points in EU countries. Table 1 and 2 shows how school achievement changed for shorter periods than those affected for more than eight weeks. In schools affected for 2 to 8 weeks, in EU countries achievement declined by around 4 points less, but the difference is not statistically significant. However, in schools that were not closed or affected by two weeks maximum, the decline is lower by around 12 points in EU countries, and by 6 points in ECA+ countries. When comparing the departure from linear trend for this group, the decline is statistically significant at the 0.05 level and equal to 7.2 points. Thus, students in EU countries in schools affected by the pandemic for a relatively short time, two weeks or less, also experienced a decline in reading achievement, although it is half of the loss of students in the most affected schools.
Taking all ECA+ countries, the decline in schools affected for more than 8 weeks is 19.2 points. In schools affected for between 2 and 8 weeks the decline is similar -the estimate is smaller by 1.4 points, and it is not statistically different. However, in schools that were not affected or affected for less than 2 weeks the decline is lower by around 6 points and the difference is statistically significant. Estimating the same regression with these schools as a baseline, shows that students in schools that were not affected by the pandemic or affected for less than 2 weeks also experienced a substantial, statistically significant decline in achievement by around 13.2 points. Another way of estimating the impact of school closures is to compare the departure from the learning achievement trend between countries with shorter and longer school closures, according to UNESCO country-level data. These results are presented in columns (4) and (5) using the number of weeks of full school closures and the sum of weeks of partial and full school closures. In both cases, for the EU, countries with no closures achieved 2021 results as predicted based on their previous achievement (with an insignificant overall decline by -0.48 points and -2.25 points, depending on the specification). However, the results decline by 1.14 points every week of full closures in EU countries. Thus, ten weeks of closure resulted in 11.9 points decline (or 16% of within-countries SD), while 25 weeks of closure resulted in 29 points decline or 39% of within-countries SD (the most prolonged full closures were in Poland and lasted for 26 weeks). When considering the total number of full or partial closures weeks, the score decline is 0.41 points per week, but in this case, the closures are longer, ranging from 12 weeks (France) to 49 weeks (Latvia). Taking all ECA+ countries, estimates are not significant for the weeks of full closures with the significant average decline of 15.7 points. For the total number of full and partial closures the results show a decline in achievement by 0.32 for every week with an overall decline by 7.9 points. Thus, for 25 weeks of full or partial closures, the estimated decline is 15.9 points, and 23.9 points for 50 weeks.

Estimates for individual countries
Under some assumptions, we can identify the learning loss for the EU student population as a whole. We take into account differences between countries that are stable over time using country fixed effects. We also control for time-variant effects of changes in the student sampling, namely for changes in individual student age and grade and country-average student age. We estimate the linear trend in student achievement, separately for each country.
Assuming that these are valid controls for time variant and time invariant country characteristics, we identify the average departure from the achievement trend for the population of students in EU or ECA+ countries participating in PIRLS 2021. This departure can be interpreted as the learning loss caused by the COVID-19 pandemic as the pandemic was the only factor affecting achievement results in all countries. To separate the overall negative impact of the pandemic from the learning disruption caused by school closures, we also provide comparisons between schools and countries affected by shorter or longer closures.
Providing similar estimates for individual countries is possible with the model presented in the section discussing our empirical strategy. However, in this case, it is not possible to interpret achievement change in 2021 as a measure of learning loss as for individual countries other factors could change between 2016 and 2021 affecting student scores. Anyhow, we provide these results to allow comparisons between countries that are adjusted for the overall time trends and changes in the sampling (age and grade). Table A1 in the annex compares results for ECA+ countries, using separate data for each country. The first three columns provide results for 2016, 2021, and the difference. These unadjusted results should not be interpreted as measures of the learning loss as they could reflect long-term trends in student achievement. Column 4 compares 2021 results to the country-specific time trend in reading achievement, using data from all PIRLS rounds available for this country. Column 5 provides similar results, but additionally adjusting for student age and grade. However, these adjustments are made using country-specific data only and not using information on age or grade effects from other countries' data. Table A2 provides model-based estimates of the learning loss for all ECA+ countries. The first column compares 2021 results to those from the previous rounds. The second column shows the estimate of linear trend departure, with the linear trend fixed at the same value for all ECA+ countries. These estimates are based on strong assumptions that there is no trend in the data or that the trend is the same for all countries. Column 3 reports similar estimates to those reported in Table A1 in column 4 but testing whether the change is statistically different from zero. Finally, column 5 provides results controlling for the country-specific time trend but also for age and grade effects estimated for all ECA+ countries. Table A3 provides calculations for EU countries that did not participate in PIRLS 2021. Learning loss is estimated for these countries considering the UNESCO data about the number of weeks with full closure and taking the average EU estimates from Table 1 for the overall decline and the decline associated with one week of closures.

Within-countries differences in the learning decline
Some research has shown that boys suffered larger learning losses during the pandemic. However, interactions of the COVID-19-related effects for boys show no difference in the decline of achievement by gender. In general, we found no evidence of gender differences in the impact of school closures on reading achievement of 4 th grade students in EU or ECA+ countries.
However, as expected, quantile regression results show that lower-achieving students experienced much larger losses. Considering principal-provided data, presented for the EU countries on Figure 1A, students at the 1 st achievement decile (lowest achieving students) in schools affected for more than eight weeks suffer a loss of nearly 25 points compared to less than 15 points for the top students (9 th decile). Similarly, for students in schools affected between 2 and 8 weeks, the lowest achieving students lost more than 20 points, while those at the top lost less than 10 points. Figure 1A also shows that in schools not affected or affected by less than two weeks, the lowest achieving students lost only 10 points, while those at the top lost around 5 points. Overall, for EU countries, these results suggest that longer school closures affected the low-achieving students more, extending inequalities.

Figure 1A: Quantile Regression Estimates of the Differences between Schools More or Less Affected by the Pandemic According to their Principals, EU Countries Only
Results for the ECA+ countries suggest that in non-EU countries the decline in achievement affected students across the ability spectrum. The top-performing students lost between 10 and 20 points, with larger losses for students in schools more affected by the pandemic. For the lowest achieving students, the losses are larger, similarly to the estimates based solely on EU countries. Students at the 1 st or 2 nd decile lost around 25 points or more, both in schools affected for between 2 and 8 weeks, and those affected for more than 8 weeks. That can be compared to the lowest-achieving students in schools affected for two weeks or less. They lost between 15 and 20 points. In general, in ECA+ countries, the learning loss is larger for students in schools more affected by the pandemic, and this effect tend to be larger among the lowestachieving students. In EU countries with around 10 weeks of full or partial closures, the best students have the achievement as expected -similar to the predictions before the pandemic. The weakest students in EU countries with around 10 weeks of closures lost around 10 points, which is considerable but also not a large loss. However, in EU countries with 50 weeks of full or partial closures, the best students lost around 20 points, while those at the 1 st decile lost nearly 30 points. Overall, according to the estimates based on UNESCO data, the lowest-achieving students in EU countries lost more, but the differences to the best students are similar for countries with shorter and longer school closures. Figure 2B show similar comparisons for all ECA+ countries. Overall, taking students in all ECA+ countries, the losses are larger. However, similarly to EU countries, the lowest achieving students lost more during the pandemic. The main difference is that in countries with the longest school closures, also students at the highest achievement levels tend to show large learning losses similar to those with lower achievement levels.

Figure 2B. Quantile Regression Estimates of the Differences between Countries Depending on the Total Number of Weeks of Full and Partial School Closures (UNESCO Data), All ECA+ Countries
How do these estimates compare to previous national estimates of learning loss? There are robust estimates of learning loss from 18 ECA countries published prior to PIRLS between 2020 and 2023 (Patrinos 2023). Learning losses range from 0 in Sweden (no school closures) to 0.37 SD in Turkey (40 weeks of closures, the longest in our sample). The average learning loss is 0.17 SDs for those studies; very close to the reading loss of 0.20 SDs recorded in our analysis of PIRLS over time.
Globally, using the same PIRLS data but for all countries -55 countries and regions, representing more than 16 million 4 th graders worldwide -the losses are estimated at 0.33 SDs, or just over a year's worth of learning (Jakubowski et al. 2023). Again, losses are larger for students in schools that faced relatively longer closures and for lower-achieving students. The economic losses may translate into a 0.68 percentage point reduction of GDP growth totaling a global loss of more than $80 trillion.

Summary and conclusions
We estimate the impact of COVID-19 on student reading on standardized tests over time for EU and ECA+ countries. We model the effect of closures on achievement by predicting the deviation of the most recent results from a linear trend in reading achievement using data from all rounds. Scores declined from 2016 to 2021 by an average of 20 percent of a standard deviation, just less than a year's worth of schooling. This validates the scenarios projected by the World Bank (Azevedo et al. 2021) early in the pandemic. Losses are significantly larger for students in schools that faced relatively longer closures. While there are no significant differences by sex, it is shown that lower-achieving students experienced much larger losses. In EU countries, losses were greater for weaker students and those experiencing longer school closures. This is in line with other studies (Goldhaber et al. 2022;Jack et al. 2022;Patrinos 2023). Losses are larger in all ECA+ countries, compared to EU countries only.
Our results are similar to national micro-studies conducted before the publication of PIRLS 2021 data (Patrinos 2023) and are in line with analyses of the same PIRLS data for the rest of the world (Jakubowski et al. 2023).
While the data go back to 2016, it is likely safe to say that most of the deviation from over time trends is due to the school closures induced by COVID-19. The analyses conducted correspond to the analysis plan set out in our pre-registered protocol (https://osf.io/nrbgz/).

Pre-registration
We prospectively registered a protocol of our analysis in the Center for Open Science (OSF) on 10 February 2023 (https://osf.io/jcp2b).   Note: Calculations using UNESCO data and estimates for the average loss in EU countries per week of school closures from Table 1, and quantile regression estimates.