Accounting for Repetition and Dropout in Contemporaneous Cross-Section Learning Profiles: Evidence from Rwanda

How much do children learn in a year of school? Longitudinal data that tracks children over time is scarce in developing countries, and so recent studies estimate learning profiles by comparing the ability of people with different amounts of schooling, at a single point in time. Such estimates of the effect of schooling on learning may be biased upwards by not controlling for repetition and dropout. In this paper I estimate contemporaneous cross-section learning profiles for Rwanda, using data from a nationally representative survey of 3,053 children aged six to eighteen. I show how adjusting this learning profile for the total number of years enrolled in school (accounting for repetition and dropout), using detailed schooling histories, reduces the average amount learnt per year by over 60 percent. The learning profile for Rwanda is not just too flat, but flatter than previous estimates suggest.


Introduction
How much do children learn in a year of school? In contrast to many standardised assessments that focus on a single age or grade, learning profiles allow us to trace the relationship between learning achieved and the amount of schooling attained. There are several types of learning profile. A true learning profile, which portrays the amount learned by an individual child or group of children across ages or grades would require panel data that follows the same children and their learning over time. Since panel data is scarce, more efforts have been made to estimate cross-sectional descriptive learning profiles, comparing the learning and schooling of a single cross-section of children or adults. Such cross-sectional descriptive learning profiles however suffer from bias due to ignoring repetition and dropout, which bias upwards the estimated effect of schooling on learning.
In this paper I estimate a cross-sectional descriptive learning profile for Rwanda. I depart from previous estimates for Rwanda that have used data from adults, that are informative primarily about the quality of schools (in terms of learning imparted per year) in the past when adults were in school. Instead, I use data from a household-based survey of children. I also make use of a detailed school history record, that allows me to assess directly the magnitude of bias due to repetition and dropout.
Learning profiles in Rwanda have been estimated previously using surveys of adults (Oye, Sandefur, & Pritchett, 2016;Kaffenberger & Pritchett, 2017). These studies look at the relationship between the highest level of schooling attained by adults, and their probability of being able to read a sentence. Both papers find that learning increases with schooling by more in Rwanda than in most other countries, but also point out that these learning profiles over-state learning gains to the extent that they omit data on dropout and repetition. If we know only the highest level of schooling attained and take no account of repetition, we over-estimate the learning per year of someone who has taken twice as long to reach their highest grade, because they repeated every year. If we take no account of dropout, then our estimates of the effect of schooling are biased upwards by unobserved factors that relate to both learning ability and dropout, such as family income or support. Rwanda has one of the highest rates of dropout across 51 developing countries analysed with relevant Demographic and Health Survey (DHS) data (Pritchett & Sandefur, 2017). This means that the sample of adult women whose highest level of schooling is grade 5 or 6 is more selective in Rwanda than in most other countries.
One approach to dealing with this issue is looking at officially reported average national rates of repetition and drop-out. However adjusting for average national repetition rates does little to change estimates of school quality based on retrospective descriptive learning profiles (Le Nestour & Sandefur, 2020). Instead, data is need on repetition and dropout of individual children.
In this paper I estimate a contemporaneous descriptive learning profile, using detailed schooling histories for each child to explicitly account for repetition and dropout. I use data of children in several grades taking comparable tests at the same point in time. I show that the size of the over-estimate from a learning profile that does not take this into consideration is substantial -over 60 percent.
The rest of the paper is organized as follows. Section 2 reviews the literature on learning profiles in more detail. Section 3 lays out my research questions. Section 4 describes the study context and presents some basic facts about the Rwandan education system. Section 5 describes the data used. Section 6 presents the methods, and Section 7 the results.
I then discuss the results in Section 8, and conclude in Section 9.

Literature Review
Following the typology laid out in Kaffenberger (2019), we can think of four types of learning profiles: causal estimates, a panel learning profile, an adult retrospective learning profile, and a contemporaneous cross-section learning profile.
First, causal estimates of the effect of schooling on learning are scarce due to the rarity of quasi-experimental variation in schooling. Where quasi-experimental studies do exist, they typically focus on a specific grade or level of schooling rather than allowing for estimates of the effect of multiple individual grades. For example Singh (2019) provides causal estimates of the effect of a single year of school on learning, exploiting age cut-offs in enrolment guidelines in a regression discontinuity design. Analysis of a lottery for scholarships in Ghana shows that a year of secondary school increases standardized test scores by 0.1 standard deviations (Duflo, Dupas, & Kremer, 2018).
Second, a panel learning profile is based on longitudinal data following the same students over time, on a vertically-linked assessment. This requires both tracking the same students over time, and an assessment with some identical question items (used for linking).
One example is Muralidharan & Zieleniak (2015), who follow a large sample of children as part of a large policy evaluation in Andhra Pradesh, India. Such analysis allows us to measure actual learning progress, and by comparing gains in learning we are able to control for a large amount of unobserved differences between children that might otherwise bias estimates.
Third, adult retrospective learning profiles compare the learning ability of adults who attained different levels of schooling in the past (see for example Beatty, Berkhout, Bima, Coen, Pradhan, & Suryadarma, 2018;Kaffenberger & Pritchett, 2017). This is useful in telling us something about school quality in countries where other learning assessment data does not exist. By using data on adults, retrospective learning profiles provide estimates of historical rather than current school quality (Le Nestour and Sandefur, 2020). These estimates of school quality also need stronger assumptions about conditional independence based on observed control variables. Fourth, contemporaneous cross-section learning profiles compare students across grades at a single point in time on a comparable assessment (for example Akmal & Pritchett, 2019;Asadullah & Chaudhury, 2015;Muralidharan, Singh, and Ganimian, 2019;Spaull & Kotze, 2015). Such profiles can provide policymakers with an up-to-date estimate of the quality of schooling. These observational cross-sectional estimates also require strong assumptions -for example that we are able to control for critical correlates of grade attainment and learning and that there are no unobserved shocks that may affect learning levels in some grades but not others. Other studies have used different approaches to dealing with bias in contemporaneous cross-section learning profiles. First, Le Nestour & Sandefur (2020) control for average repetition rates across countries, showing that this makes little difference to learning profiles. Another approach proposed by Sandefur et al. (2016) is to estimate Lee (2009) bounds on the bias by assuming the maximum possible bias through sorting of different children into different grades (a standard approach to dealing with missing data). This approach suggests that failing to account for dropout may significantly overstate true learning, but the bounds are wide.

Research Questions
In this paper I address three main research questions. First, how steep is the learning profile in Rwanda across grades (or how much do children learn in a year of school)?
Second, to what extent do high levels of repetition and dropout bias estimates of learning per year? Third, how do learning profiles vary for different disadvantaged groups -specifically girls, children from low socioeconomic status families, and children with disabilities.
Documenting where learning profiles are flat (there is little learning per year) can highlight issues of over-ambitious learning curricula (Pritchett & Beatty, 2015). Estimating accurate learning profiles for different disadvantaged groups allows us to show the evolution of learning gaps between groups -whether they grow with school or are present at the start of school.

Context
Rwanda has achieved a reputation for effective service delivery in the years since the genocide against the Tutsis, achieving a rapid increase in primary school enrolment (Akresh & De Walque, 2008;Guariso & Verpoorten, 2018), and better health outcomes than other low-income countries. This success has been due to a political economy in which legitimacy has been based on rapid socio-economic development (Chemouni, 2018), a highfunctioning bureaucracy, 1 and government focus on meeting international development goals (Abbott et al., 2017;Binagwaho et al., 2014).

Despite these successes, learning outcomes remain poor. The World Bank Human
Capital project puts learning in Rwanda on an international scale for the first time, by linking Early Grade Reading Assessment (EGRA) comprehension scores from Rwanda to international standardized assessments (Patrinos & Angrist, 2018). On this scale, Rwanda ranked 27th of 41 countries in sub-Saharan Africa for learning in 2018 (compared to 4th and 5th in Africa adult and child survival, respectively). The average child in Rwanda can expect to receive 6.6 years of schooling. Adjusted for the quality of learning, this is equivalent to just 3.8 years in the best performing country on the scale (Kraay, 2018).
A likely explanation for this difference in performance is that government has focused on achieving the Millennium Development Goals, which in education only included enrolment, but in health included outcomes (child and maternal survival). Both national and local government targets and performance incentives have focused on the most readily observable aspects of education quality, such as classroom construction rather than teaching and learning outcomes that are more difficult to measure (Williams, 2017). The government has also struggled to ensure adequate supply of basic inputs such as textbooks (Milligan et al., 2017).
Other causes of poor overall educational performance include over-ambitious curricula (Pritchett & Beatty, 2015;van de Kuilen et al., 2019), low teacher pay and support, and the switch of the language of instruction from French to English (which is not yet widely spoken) in 2018. Kinyarwanda remains the language of instruction in grades one to three, but this switches to English from grade four (Williams, 2017). Rwanda's official average repetition rate in primary school as reported to UNESCO was 13 percent in 2017, slightly above the average of 10 percent for low-income countries. in grade one . Neither of these studies is though able to provide a complete picture of repetition.
Survey-based learning assessments show a clear pattern of poor performance going back several years (Table A1) These learning assessments present much worse results than suggested by retrospective learning profiles based on adults. Oye et al. (2016) show that adult women with five or six years of education are more likely to be literate in Rwanda than in almost all other developing countries for which comparable DHS data exists. The estimated gain in the probability of being literate for each additional grade is higher in Rwanda than for most other countries. This presents a puzzle.
Children are expected to start grade one at the age of seven. Data on preschool provision is mixed. Friedlander et al. (2014) found that 71 per cent of students in their sample reported attending some kind of pre-school care provision, though it was unclear whether this was a formal center with trained caregivers or not. This is substantially higher than the level attending formal pre-primary classes as reported by the Ministry of Education.

Data
In this paper I use a nationally representative household survey of 8,122 school age children (between 6 and 18 years old), conducted between February and April 2017 (Laterite et al., 2017). The survey was conducted by a private survey firm (Laterite) for UNICEF and the Ministry of Education. Data was shared by the Ministry of Education. All children in selected households aged between six and eighteen were assessed. I restrict the sample to include only those tested with Early Grade Reading and Early Grade Mathematics Assessments, enrolled in grade one to six or having dropped out of school, and with data on student and household characteristics, leaving 3,053 children from 1,788 households. The literacy assessment measures Kinyarwanda (grade 1-3) or English (grade 4) reading comprehension skills, and the numeracy assessment tests basic mathematics skills.
In addition to asking both parents and children for the child's current grade and whether each child had ever repeated a year of school, enumerators also recorded a full schooling history for each child. This recorded for up to 12 previous years what grade the student was in that year. This schooling history was asked of children, and then checked by enumerators with their parents to ensure agreement. Based on this schooling history, I calculate the number of times that each student repeated a grade, and the total number of years that they were enrolled in any grade between one and six. I calculate a parent support index based on the extent to which students agreed with a series of seven statements about their parents. Each answer is scored on a five-point Likert scale. This index is standardized to mean zero with a standard deviation of one. The statements are; (1) Your siblings helped you more with your homework than your parents.
(2) Your parents always knew the solutions to your homework questions.
(3) Your parents think chores and supporting the household business/farm is more important than school.
(4) Your parents forced you to go to school even when you did not want to.
Your parents sometimes asked you to miss a day of school in order to support household business/farm.
Your parents were satisfied with your performance at school.

Assessments
Assessments are based on a sub-set of the Kinyarwanda EGRA and EGMA items used in the REB-USAID Literacy, Language and Learning Initiative (L3) Laterite et al., 2017). These items were initially developed by the Rwanda Education Board (REB) and EDC, based on international standards for measuring early grade literacy and mathematics, Rwandan national grade-level standards, and the Rwandan competency-based curriculum. A reliability analysis of the mathematics items showed a strong reliability for all four sub-tests (addition, subtraction, multiplication, and division) . Questions were set at four different grade levels covering grades one to four. Questions from each grade level were asked of pupils in that grade and the grade above, meaning that, for instance, grade 1 level questions were asked of both children in grade one and grade two. Grade four questions were asked of students in grade four, five, and six. This overlap in the same question items being asked of students in different grades allows analysis of descriptive learning profiles, showing the increase in learning per grade.
Reading assessments consisted of a simple comprehension task -being given an unlimited amount of time to read a 50-word paragraph, and then answering five questions about this paragraph without referencing the reading passage. The expert group that developed the EGRA assessment deemed that answering four out of five questions correctly was the minimum expectation to be at grade level ) (examples are included in Appendix 2). A limitation to these assessments is that some of the questions might be guessed without reference to the reading passage. This applies most to the grade four English assessment, but does not affect the validity of the comparison across grades.
In mathematics, grade one children were given nine minutes to answer up to thirty grade one level items. These are split evenly between number discrimination, addition, and subtraction. Grade two children were given either the same 30 grade one items, or 30 grade two items (in addition, subtraction, and multiplication). Grade three children were given either the 30 grade two items, or 40 grade three items (addition, subtraction, multiplication, and division). Grade four children were given either the 40 grade three items, or 40 grade four items. Grade five and six children all received the 40 grade four items. Children were handed a pen and paper and given three minutes to complete each block of 10 mathematics questions.
Data in the released dataset includes the number of items attempted and answered accurately in English and maths at each question grade level. For mathematics I take the average correct score across all items, and for reading the average correct score. I standardize each outcome to mean zero and standard deviation one.

Methods
I begin by presenting simple mean outcomes by grade. Next, I regress literacy and numeracy outcomes on three different formulations of student grade. First using actual current grade, second using highest grade achieved (thus including those currently out of school), and third the total number of years spent at the relevant grade levels (thus accounting for repetition). I estimate each of the three models with and without student and household control variables. I standardise outcomes to a mean of zero and standard deviation of one. As the data contains multiple children from some households, I cluster standard errors in all models at the household level to account for the non-independence of observations. Finally, I estimate heterogeneity in the gradient between learning and grade by different student characteristics.
I also use the learning function (OLS regression of learning on schooling and other covariates) to convert years of schooling into standard deviation effect sizes, following Evans and Yuan (2017) and Baird and Pane (2018).

Learning Profiles in Rwanda
I begin by addressing the first research question, plotting the learning profile in Rwanda across grades (Figure 1). Progress does appear to be made between grades. On the grade 1 level assessment, a child in grade 1 answers 14 percent of mathematics questions correctly, while a child in grade 2 answers 39 percent of the same questions correctly. In reading, comprehension level was set by the test designers at being able to answer four out of five questions correctly, something only achieved by grade 6 pupils (on a grade 4 level assessment).
45 percent of children were not able to answer any of these additions correctly in the allocated time. 20 percent did not even attempt to answer any of these questions. On average children were able to compute 28 percent of questions.

Adjusting Learning Profiles for Repetition and Dropout
The previous section showed low starting learning levels but a steady improvement across grades. However, this picture is biased by ignoring children that have dropped out and repeated multiple grades. In this section I focus on the 1,052 children in grades 4, 5, 6, (or out of school) who took the grade four level test. I begin by regressing reading comprehension and mathematics test scores on highest grade attained. This step expands the sample to include out of school children. In this specification I include controls for student and family characteristics, as the objective is to obtain an estimate of school quality. These control variables are student age, sex, family wealth, parental support, living in a rural location, having a disability, and the number of children's books at home. These controls do not substantially affect the coefficient on highest grade (see Table 3). I control for district level fixed effects in the core specification, but the results are robust to instead controlling for either school or province fixed effects.
Each grade is associated with an increase in reading comprehension of 0.39 standard deviations, and mathematics of 0.6 standard deviations (Table 3, column 1).
I next replace the highest grade achieved with the total number of years attended school in grade 4 or higher. The coefficient falls by over 60 percent.

Table 3
Determinants of Test Scores 0.061 0.155 0.058 0.153 0.004 0.093 Note: * p<0.1, ** p<0.05, *** p<0.01. The outcome variables for literacy and numeracy are grade four level test scores, standardized to mean zero and standard deviation of one. Controls include age, gender, disabilities, living in an urban area, parental support, family wealth, and number of books in the home. Standard errors are clustered at the household level. Figure 2 again focuses on the three grades for which we have common assessment itemsgrades 4, 5, and 6. Figure 2(a) shows the average relationship between learning and highest grade attained, as well as the wide overlap in learning ability within grades. Highest grade attended explains just 6 percent of the total variation in reading comprehension, and 15 percent of the total variation in mathematics. Figure 2(b) shows the gradient of test scores to total years spent in school in grade 4 or later.

Figure 2 Mathematics Learning Profile Adjusted for Repetition
Note: Figure 2 (a) shows a scatterplot of learning scores against current grade, with a line of best fit. Figure 2 (b) shows the same scatterplot against the total number of years that pupils have spent in school, including repeated years.
I then draw learning profiles for five different sub-groups of children to observe whether learning happens at different rates for different students (Figure 3). Though there are differences in the level of learning, there are little differences in the change in level over time by sex, wealth, urban residence, or having books at home. Disabled children learn less across time than other students, and those with high parental support learn more (see Table   A2 for statistical tests).

Figure 3 Learning Profiles for Sub-Groups
Note: These figures show average mathematics test scores (the share of questions answered correctly), by years in school, for six subgroups -sex, household wealth, urban or rural residence, disability status, level of parental support, and having books at home. Shaded areas represent 95 per cent confidence intervals.

Discussion
In this paper, we set out to understand the trajectory of learning in Rwandan primary schools, how naïve estimates are biased by repetition, and how trajectories vary by student characteristics. We find that, contrary to some prior cross-country studies (Oye et al 2016; Kaffenberger & Pritchett, 2017), learning per year in Rwanda is very low. We also document a large dispersion of ability within classrooms (Figure 2), making teaching to the full range of abilities incredibly difficult, and may be one reason itself for the poor quantity of learning in each teaching year.
The slope is now much flatter, with pupils achieving grade 4 proficiency only after six or seven years of enrolment. Achieving something closer to mastery (4 of 5 questions correct) takes the average pupil 10 years of being enrolled in school.
I do not find significant differences in the pace of learning for children with different background characteristics. This suggests that although there are inequalities and differences in learning between different sub-groups, these largely exist already at school entry, and do not substantially narrow or widen during school. This data is therefore consistent with the view that learning crises in low-income countries are general ones caused by dysfunctional overall systems, rather than driven primarily by inequalities between different groups or individuals.
How do we reconcile the large difference between current grade and total years of enrolment, with an official national repetition rate of only 13 percent? Rwanda has in fact seen a large reduction in repetition rates as reported by survey respondents in our samplefrom 55 percent in 2005 to 17 percent in 2017. This means that the current repetition rate in our data is close to the officially reported one. Many students in our sample repeated at higher rates in previous years ( Figure A2), and are still captured in this "contemporaneous" data.

Equivalent Years of Schooling Per Standard Deviation
Considering total time spent enrolled in school, we estimate that each additional year of enrolment is associated with a 0.16 standard deviation improvement in reading and a 0.25 improvement in mathematics (Table 3). These numbers are well within the range of estimates for other developing countries (Bolivia, Colombia, Ghana, Kenya, and Vietnam) (Evans & Yuan, 2019).
This estimate could then be applied to other studies that estimate the effect of interventions to improve learning in Rwanda, in order to provide non-researchers with a more intuitive interpretation of effect sizes than standard deviations. For example a recent impact evaluation of a teacher performance pay scheme in Rwanda reported impacts of 0.09 standard deviations per year on student learning (combining Kinyarwanda, English, Mathematics, Sciences, and Social Studies) (Leaver et al., 2019). If one year in school is associated with 0.16 standard deviations more learning, then a 0.09 standard deviation effect sizes is equivalent to over six months of actual current learning (0.09/0.16). Nsabimana & Mensah (2020) find that a year of school feeding improves maths test scores by 0.18 standard deviations -or 72 percent of business as usual learning. Borzekowski et al. (2019) report effects of watching a children's educational television show on early learning, with average effects of 0.08 standard deviations, or half a year of learning.

Limitations
This study has several limitations. First, and importantly, the approach relies upon children and parents accurately remembering the grade they were in throughout school, up to twelve years ago.
Second, we are unable to identify children who attend for only part of a year and then repeat the same grade the following year, and thus we count these children as having repeated a whole year.
Third, though our learning assessment is more detailed than some single-item assessments used in estimating learning profiles, our assessment is still a simple one, including just five reading comprehension items and 30-40 mathematics exercises per student. Students may learn many more things during a school year that are not captured by this assessment, though it is arguable that reading with comprehension and basic mathematical operations are essential foundational buildings blocks for much other learning.

Conclusion
In this paper I document the slow pace of learning in Rwanda, using nationally representative household survey data. Students, particularly in early grades, learn little and progress slowly. Using detailed data on student schooling trajectories I estimate the impact of repetition and dropout on estimated learning profiles. Measuring learning against the total number of years enrolled in school rather than just the current or highest attained grade substantially flattens the curve.
As the literacy and numeracy assessments used in this paper are based on Early Grade Assessments that have been used in a range of other countries, there may be scope for drawing comparisons with learning profiles in other countries. Rwanda is an outlier in having a particularly high rate of repetition, but other studies estimating learning profiles using observational data (adult retrospective and contemporaneous cross-section learning profiles) should assess the extent to which estimates might be biased upwards by omitted data on repetition. The direction of this bias is cause for even more pessimism about the effectiveness of school systems in low-income countries in imparting basic skills to all.
An important feature of the learning distribution for policy implications is the marked large bulge of students at the bottom of the distribution who are not learning anything. Significant improvements in average performance could be achieved by ensuring that all children are able to achieve basic foundational literacy and numeracy skills.
The wide variation in student ability within grades poses a serious challenge. The government of Rwanda (and other countries with similar challenges) might look to approaches designed exactly to help teachers deal with such variation in ability (A. Duflo et al., 2020).

Figure A 2 Trend in repetition rate
This figure shows the average repetiion rate by year, calculated as the share of all students in grades 1 to 6 who were repeating their current grade. 189 Note: * p<0.1, ** p<0.05, *** p<0.01. High SES is defined as being in the top quintile of the wealth distribution, and high parent support as being in the top quintile of the distribution of a parent support index. Both values are compared to the bottom quintile of the respective indices. Standard errors are clustered at the household level. The sample in all cases is children in grades 4, 5, or 6 or out of school who sat the grade 4 level test. Outcomes are standardized test scores.