Can Preconditions for Results-Based Financing Be Established in Fragile States?

mechanisms such as conditional cash transfer (CCT) programs, school grants, and teacher and school incentives are increasingly being used around the world to improve education results and outcomes and move away from the procurement of “more of the same” inputs.1 However, for RBF to work, certain preconditions must be met, the most important of which is the need to build the technical capacity of countries to define and implement RBF mechanisms and to develop monitoring systems that measure the results. One of the most important preconditions of RBF is a functioning education management information system (EMIS) so that results can be accurately monitored and reported.2 Without an EMIS to precisely measure both learning outcomes and intermediate indicators of educational quality, it is impossible for education ministries to effectively incentivize improvements.

Results-based financing (RBF) mechanisms such as conditional cash transfer (CCT) programs, school grants, and teacher and school incentives are increasingly being used around the world to improve education results and outcomes and move away from the procurement of "more of the same" inputs. 1 However, for RBF to work, certain preconditions must be met, the most important of which is the need to build the technical capacity of countries to define and implement RBF mechanisms and to develop monitoring systems that measure the results. One of the most important preconditions of RBF is a functioning education management information system (EMIS) so that results can be accurately monitored and reported. 2 Without an EMIS to precisely measure both learning outcomes and intermediate indicators of educational quality, it is impossible for education ministries to effectively incentivize improvements.
A crucial question is whether these types of preconditions can be established in countries affected by REACH funded the development of a quality assurance system (QAS) to help the Government of Haiti establish the necessary preconditions for the adoption of RBF.
fragility, conflict, and violence (FCV) as a first step towards implementing RBF programs in these countries. Using RBF may be an attractive option to consider in some FCV countries given that donor and local exhaustion may have set in after decades of investments in the education sector that have not produced the expected results. The flexibility of RBF approaches also means that they might be the financing modality that is best suited to the challenging and ever-changing environment in FCV countries, allowing all stakeholders to keep an eye on desired results while adjusting the actions being taken to achieve those results when necessary. This question is very relevant given the fragility of institutions and regulatory frameworks in FCV countries as well as the weaknesses in their monitoring and governance systems. Despite these gains in enrollment, student achievement remains very weak. The average Haitian child enters first grade nearly two years late due to a combination of household financial constraints and restrictive school policies that require children to receive at least one year of pre-primary education in order to prepare for first grade. Once in school, 13 percent of first graders repeat the year, and only about half will reach the sixth grade. Student learning is also very weak. For example, a 2016 nationally representative early grade reading assessment found that 42 percent of students near the end of their second-grade year could not read a single word of Haitian Creole. 7 Furthermore, a 2015 fourth grade math assessment by the MENFP in public and private schools found that student scores were only slightly better than random guessing. This shows that the learning crisis has also affected the Haitian education system. 8 The idea behind the QAS was to help the MENFP to tackle some of the toughest challenges in the sector and to develop a system that would make it possible to link financing to results in the future. The QAS would be designed to cover both public and private primary schools. The approach also included complementary activities to strengthen the technical capacity of MENFP staff to develop the QAS-in particular, to establish standards for learning conditions and for student learning, as well as diagnostic assessments linked to those standards.
Although the MENFP was already putting substantial emphasis on improving the quality of education, different actors had different definitions of quality. It was critical to establish a consensus on a definition of educational quality before specific learning standards could be developed. Since multiple previous attempts to develop accreditation systems for non-public schools had failed, the work under the REACH grant focused on developing a definition of quality that would apply to both public and non-public schools.
The final QAS design addressed several issues, starting with the

WHY WAS THE INTERVENTION CHOSEN?
definition of learning standards. Standards of learning were defined to prioritize the most important elements of the curriculum and ensure that any learning assessment that is developed is aligned with the curriculum. The QAS also established the necessary tools to measure the school learning conditions-outputs and intermediate outcomes-that are needed to improve learning. By establishing systems to collect data on learning inputs and outcomes, the QAS would be setting the stage for strengthening the monitoring and evaluation of interventions such as the tuition waiver programs and to increase the accountability of both private and public schools. The establishment of the QAS would also provide donors and policymakers with the data that they needed to measure progress and results and to target interventions.
To increase the capacity of the MENFP, the World Bank team in Haiti worked with the ministry to identify and carry out specific tasks that would lead to the development of the QAS. The process of building this capacity was intended to emphasize the MENFP's ownership of each activity and to increase the likelihood that it would sustain the QAS over time. It also brought together diverse stakeholders and incentivized them to coordinate their work. The QAS was also intended to empower local actors to get more involved in the decisions that impact learning in schools, to prioritize resource use, to increase government accountability, and to meet the local population's most urgent needs.
By developing a QAS for all primary schools in the country, the REACH Trust Fund hoped to improve governance, enhance data systems to measure results, and establish the preconditions necessary to put in place an RBF mechanism to improve the quality of education in Haiti.

HOW DID THE INTERVENTION WORK?
The   Table 1.
An important goal of the QAS was to build Haiti's capacity to measure student learning in primary education. To achieve this, the REACH grant also supported workshops attended by international and national experts, teachers, and MENFD staff to develop learning standards aligned with the curriculum and funded an accompanying assessment of third grade reading in Haitian Creole.
These learning standards were added to the QAS to provide clear milestones for the development of reading skills in the third grade. These standards can also be used to develop assessments to strengthen the capacity of teachers to assess their own students, to measure learning in a transparent and technically sound way, and to adjust their pedagogical improvement plans. These standards were defined across five levels of learning as seen in Table 2. School principals focus their practices towards student learning, thus requiring that they monitor results and carry out a strategic and planned vision.

D2 School environment
The school, the administration, and teachers follow procedures and practices that are clear in terms of the organization, preparation, implementation and evaluation of teaching, considering the needs of all students by taking concrete actions so that students achieve their learning objectives and reach their potential.

D3 Pedagogical Management and Support
The school has policies, procedures, and strategies in place that favor an environment that is respectful, organized, safe, and appropriate for student learning and the school community. Additionally, the school supports and promotes the spiritual, ethical, moral, emotional, and physical development of its students.

Management of Human and physical resources
The school has policies, procedures, strategies, and practices in place for the management of its staff and educational resources.

D5 Community Participation
The school has presents policies, procedures, strategies, and practices in place for creating and developing the participation of stakeholders within the educational community.

Levels of learning Description
Advanced Pupils can make simple inferences: they are able to draw conclusions that are obvious, to find the meaning or subject of a theme or main idea of a text, to make inferences that an event is the cause of another one, to make simple inferences regarding the main characters, their attributes, feelings, or motivations, and are able to relate two pieces of information contained in two adjacent sentences.
High Pupils can collect explicit information in a text: they are able to identify a word or idea already mentioned in a text and are able to retrieve information in a word or in a sentence.

Average
Pupils can read words and short sentences: they can recognize the image corresponding to the sentence or word that they read by themselves, identify the words read aloud by the teacher, and identify words written in uppercase or lowercase.

Low
Pupils can recognize letters and syllables: they are able to recognize images corresponding to short oral messages (a sentence or a word), to recognize syllables, to identify letters written in upper and lower case, to identify the first letter of a word and the initial sound of a word, as well as written characters among other symbols (such as numbers or signs).

Below Average
Pupils at this level do not demonstrate the skills measured by this test in Creole. These pupils have difficulties with the knowledge and skills at the low level.
School performance in Haiti varies significantly across the five dimensions of educational quality.
To assess progress towards the achievement of educational quality standards for each school, a set of questionnaires were developed, pre-piloted, and piloted in 2016 and 2017, one for school principals, one for teachers, and one for school characteristics. The data collected by these questionnaires was aggregated into a school's profile consisting of the school's scores on indicators that provide an overview of its level of development in each dimension. The average school performance ranged from 28 percent in the "Management of Human and Physical Resources" dimension to 62 percent in the "School Environment" dimension ( Figure 1).
These school profiles, in combination with measurements of learning, will be valuable inputs into the school improvement plans (SIPs) that each school is required to develop with the participation of their local community. There is potential for policymakers to use RBF to encourage schools to achieve the goals set out in their plans as measured by the indicators in the QAS. Some of these improvements might actually be inputs, such as constructing proper facilities for learning (as in Dimension 2. School Environment). While school facilities are often strictly a final outcome, the lack of proper classrooms and latrines can prevent students from being able to effectively attend school and learn.
The assessment of third grade student learning in Haitian Creole was piloted in 20 schools in Port-au-Prince in 2017. Almost 70 percent of students in the pilot had an "Average" level of reading skill or lower (Figure 2). Only 2 percent of students in the pilot were at the "Advanced" level, which is defined as having the skills expected by the official third grade curriculum, including being able to recognize letters and syllables, to read words and short sentences, to collect explicit information in a text, and to make simple inferences. In other words, if third grade students were reading at the expected third grade level, most students would have exhibited the "Advanced" level of reading in the assessment.
WHAT WERE THE RESULTS?  Given the prevalence of private schools in Haiti and the long history of weak public sector management, the QAS is also acting as a guiding framework through which the government can both improve the quality of public service delivery (by prioritizing its spending according to clearly identified needs) and regulate the private sector (by making public financing conditional on schools meeting high quality standards).
The continued interest and engagement that the World Bank team has experienced from the MENFP and other sector stakeholders demonstrates that there was clearly a substantial latent demand for a dialogue on results.
The development of the QAS is helping to legitimize the MENFP's efforts to regulate the education sector by creating clear, specific, and realistic standards that will apply to both public and private schools.
At the same time, moving towards RBF is a process that will take time and multiple iterations. Creating a sustained consensus about quality standards despite shifting priorities and political pressures will require ongoing technical assistance, dialogue, and knowledge sharing. Moreover, the QAS is a work in progress, as the realities of implementation mean that the tools and approaches must be regularly adjusted and refined to make them work on the ground.
Based on the experience of developing the QAS in Haiti, it seems that successfully implementing RBF in a low-income, fragile country requires at least three elements: (a) laying the groundwork for defining and measuring educational quality, as the World Bank team in Haiti has done over the past two years, and remaining persistent despite the inevitable setbacks and slowdowns that come from instability, shifting priorities, and emergencies on the ground; (b) recognizing that "results" can often include what we often dismiss as "inputs," which can actually matter a great deal for a school's learning outcomes (for example, having decent, functional latrines); and (c) recognizing that many low-income countries may already have policies that define fairly lofty ideals for standards but that having a dialogue on RBF can actually be more useful than introducing aspirational outcomes for bringing policy closer to reality.
The QAS has been able to shift the focus of policy and technical dialogues in Haiti away from inputs and towards results and helped the Ministry understand how to improve the quality of education.

CONCLUSION
The set of activities funded by REACH successfully created a quality assurance system (QAS) in Haiti that collects information on five dimensions of learning conditions in schools, provides schools with clear standards to meet, and gives them information about their progress towards reaching these standards. Now that this information is being measured by the QAS, policymakers can decide to offer monetary incentives to schools, teachers, or other relevant stakeholders to make the necessary efforts to improve these learning conditions and outcomes. REACH also supported the development of learning standards for reading in Haitian Creole. The QAS data collection instruments have been piloted and revised and have become important tools for measuring educational quality in Haiti.
These activities have had several benefits: (a) they have increased the Haitian government's capacity to measure educational quality; (b) they have advanced the technical design and piloting of accreditation processes and improved the governance and regulation of the school system; (c) they have made it easier to target programs to address schools' needs based on the best available data; (d) they have empowered local actors (such as school leaders, principals, teachers, and parent associations) to become more involved in school decisions; and (e) they have established the foundation for introducing an RBF mechanism to incentivize school improvement. By establishing the QAS in Haiti as a reliable mechanism for measuring learning and intermediate indicators of educational quality, REACH has shown that the preconditions for RBF can be established in FCV countries.