HAITI : Can Preconditions for RBF Be Established in Fragile States ? RESULTS-BASED FINANCING AROUND THE WORLD

Results-based financing (RBF) mechanisms such as conditional cash transfer (CCT) programs, school grants, and teacher and school incentives are increasingly being used around the world to improve education results and outcomes and move away from the procurement of “more of the same” inputs.1 However, for RBF to work, certain preconditions must be met, the most important of which is the need to build the technical capacity of countries to define and implement RBF mechanisms and to develop monitoring systems that measure the results. One of the most important preconditions of RBF is a functioning education management information system (EMIS) so that results can be accurately monitored and reported.2 Without an EMIS to precisely measure both learning outcomes and intermediate indicators of educational quality, it is impossible for education ministries to effectively incentivize improvements.


RESULTS-BASED FINANCING AROUND THE WORLD SEPTEMBER 2018
Results-based financing (RBF) mechanisms such as conditional cash transfer (CCT) programs, school grants, and teacher and school incentives are increasingly being used around the world to improve education results and outcomes and move away from the procurement of "more of the same" inputs. 1 However, for RBF to work, certain preconditions must be met, the most important of which is the need to build the technical capacity of countries to define and implement RBF mechanisms and to develop monitoring systems that measure the results.One of the most important preconditions of RBF is a functioning education management information system (EMIS) so that results can be accurately monitored and reported. 2Without an EMIS to precisely measure both learning outcomes and intermediate indicators of educational quality, it is impossible for education ministries to effectively incentivize improvements.
A crucial question is whether these types of preconditions can be established in countries affected by fragility, conflict, and violence (FCV) as a first step towards implementing RBF programs in these countries.Using RBF may be an attractive option to consider in some FCV countries given that donor and local exhaustion may have set in after decades of investments in the education sector that have not produced the expected results.The flexibility of RBF approaches also means that they might be the financing modality that is best suited to the challenging and ever-changing environment in FCV countries, allowing all stakeholders to keep an eye on desired results while adjusting the actions being taken to achieve those results when necessary.This question is very relevant given the fragility of institu-tions and regulatory frameworks in FCV countries as well as the weaknesses in their monitoring and governance systems.
The Results in Education for All Children (REACH) Trust Fund at the World Bank provided funding to the Government of Haiti with the goal of establishing the preconditions for the adoption of RBF in the Haitian National Ministry of Education and Professional Training (Ministère de l'Education National et de la Formation Professionnelle, MENFP).To this end, the grant funded the development of a quality assurance system (QAS) based on specific standards for the most important dimensions of educational quality in the country.The idea was to include clear indicators for each quality dimension that would make it possible to measure education results on the ground.The grant also funded a series of complementary activities aimed at strengthening the technical capacity of MENFP staff to define and measure quality.By developing a QAS for all primary schools in the country, the grant aimed to improve governance, enhance the data systems needed to measure results, and establish the preconditions necessary to introduce an RBF mechanism in the education sector in Haiti.

Context
Although there have been considerable increases in enrollment in recent years, the education sector in Haiti still faces many challenges. 3The net primary enrollment rate rose from about 60 percent in the early 2000s to between 70 and 80 percent in 2012. 4Because there are not enough public schools to meet this increased demand, private providers have stepped in to fill the gap and now operate over 80 percent of primary schools. 5owever, there is little oversight of these providers nor are they held accountable for the quality of their teaching or learning.The public sector plays a similarly limited role in both the provision and regulation of all other levels of education as well.
The Government of Haiti has been able to raise enrollment rates by making increasing access to primary education one of its top funding priorities.The government has been financing tuition waivers for students attending private schools through the donor-funded Education for All (Éducation Pour Tous, EPT) Project and its Tuition Waiver Program since 2007 and through the government-funded Universal, Free, and Obligatory Education Program (Programme de Scolarisation Universelle, Gratuite, et Obligatoire, PSUGO) since 2011.At their peak between 2011 and 2014, these two programs financed the school fees of over 60 percent of all primary students in Haiti, with PSUGO accounting for about 20 percent of domestic public spending on education. 6However, these gains in access are at risk as these programs have not yet been established as a permanent part of Haiti' s education system.Despite these gains in enrollment, student achievement remains very weak.The average Haitian child enters first grade nearly two years late due to a combination of household financial constraints and restrictive school policies that require children to receive at least one year of pre-primary education in order to prepare for first grade.Once in school, 13 percent of first graders repeat the year, and only about half will reach the sixth grade.Student learning is also very weak.For example, a 2016 nationally representative early grade reading assessment found that 42 percent of students near the end of their second-grade year could not read a single word of Haitian Creole. 7Furthermore, a 2015 fourth grade math assessment by the MENFP in public and private schools found that student scores were only slightly better than random guessing.This shows that the learning crisis has also affected the Haitian education system. 8y Was the Activity Chosen?
The idea behind the QAS was to help the MENFP to tackle some of the toughest challenges in the sector and to develop a system that would make it possible to link financing to results in the future.The QAS would be designed to cover both public and private primary schools.The approach also included complementary activities to strengthen the technical capacity of MENFP staff to develop the QAS -in particular, to establish standards for learning conditions and for student learning, as well as diagnostic assessments linked to those standards.
Although the MENFP was already putting substantial emphasis on improving the quality of education, different actors had different definitions of quality.It was critical to establish a consensus on a definition of educational quality before specific learning standards could be developed.Since multiple previous attempts to develop accreditation systems for non-public schools had failed, the work under the REACH grant focused on developing a definition of quality that would apply to both public and non-public schools.
The final QAS design addressed several issues, starting with the definition of learning standards.Standards of learning were defined to prioritize the most important elements of the curriculum and ensure that any learning assessment that is developed is aligned with the curriculum.The QAS also established the necessary tools to measure the school learning conditions -outputs and intermediate outcomes -that are needed to improve learning.By establishing systems to collect data on learning inputs and outcomes, the QAS would be setting the stage for strengthening the monitoring and evaluation of interventions such as the tuition waiver programs and to increase the accountability of both private and public schools.The establishment of the QAS would also provide donors and policymakers with the data that they needed to measure progress and results and to target interventions.
To increase the capacity of the MENFP, the World Bank team in Haiti worked with the ministry to identify and carry out specific tasks that would lead to the development of the QAS.The process of building this capacity was intended to emphasize the MENFP' s ownership of each activity and to increase the likelihood that it would sustain the QAS over time.It also brought together diverse stakeholders and incentivized them to coordinate their work.The QAS was also intended to empower local actors to get more involved in the decisions that impact learning in schools, to prioritize resource use, to increase government accountability, and to meet the local population' s most urgent needs.
By developing a QAS for all primary schools in the country, the REACH Trust Fund hoped to improve governance, enhance data systems to measure results, and establish the preconditions necessary to put in place an RBF mechanism to improve the quality of education in Haiti.

What is the Activity?
The QAS was developed by the MENFP through a series of workshops, meetings, and continuous technical assistance from the World Bank.The developers took into account relevant international experiences that could be appropriately adapted to the Haitian context and built on technical work done by the MENFP and other partners.So far, what has been accomplished includes: (i) the development of an officially endorsed framework for educational quality with a holistic perspective that includes conditions for learning in schools and student learning outcomes; (ii) the creation of validated measurement instruments for assessing the quality of learning; and (iii) a nationally representative pilot test of these instruments that will yield baseline measurements for the system.The QAS is now the guiding framework for providing support to both public and private primary schools under the IDA-financed Providing an Education of Quality in Haiti (PEQH) Project and is likely to be used by other donors to channel financing to schools as well.
An important aspect of the REACH grant was to strengthen the capacity of the MENFP to collect and analyze education data.The REACH funds have been used to reinforce the statistical and information functions executed by the ministry' s Direc-torate of Planning and External Coordination (Direction de la Planification et de la Coordination Externe, DPCE).In particular, REACH is supporting a coordinated effort among donors and MENFP directorates to build and sustain a functional EMIS.Using frameworks developed by the World Bank' s Systems Approach for Better Education Results (SABER) initiative, REACH funds were used to carry out a diagnostic to identify strengths and weaknesses in data collection and information management in the Haitian education sector and to support the MENFP' s design for an EMIS.A grant from the Trust Fund for Statistical Capacity Building, as well as IDA funds under the PEQH Project, will be used to implement this design.
REACH funds have also been used to strengthen the capacity of MENFP staff to define standards for learning conditions and to develop assessments to measure conditions in schools against these standards.These standards define quality education in five dimensions: (i) the school principal' s leadership; (ii) the school environment; (iii) pedagogical management and support; (iv) the management of human and physical resources; and (v) community participation.The standards for each of these dimensions are described in Table 1.
An important goal of the QAS was to build Haiti' s capacity to measure student learning in primary education.To achieve this, the REACH grant also supported workshops attended by international and national experts, teachers, and MENFD staff to develop learning standards aligned with the curriculum and funded an accompanying assessment of third grade reading in Haitian Creole.
These learning standards were added to the QAS to provide clear milestones for the development of reading skills in the third grade.These standards can also be used to develop assessments to strengthen the capacity of teachers to assess their own students, to measure learning in a transparent and technically sound way, and to adjust their pedagogical improvement plans.These standards were defined across five levels of learning as seen in Table 2.

Advanced
Pupils can make simple inferences: they are able to draw conclusions that are obvious, to find the meaning or subject of a theme or main idea of a text, to make inferences that an event is the cause of another one, to make simple inferences regarding the main characters, their attributes, feelings, or motivations, and are able to relate two pieces of information contained in two adjacent sentences.

High
Pupils can collect explicit information in a text: they are able to identify a word or idea already mentioned in a text and are able to retrieve information in a word or in a sentence.

Average
Pupils can read words and short sentences: they can recognize the image corresponding to the sentence or word that they read by themselves, identify the words read aloud by the teacher, and identify words written in uppercase or lowercase.

Low
Pupils can recognize letters and syllables: they are able to recognize images corresponding to short oral messages (a sentence or a word), to recognize syllables, to identify letters written in upper and lower case, to identify the first letter of a word and the initial sound of a word, as well as written characters among other symbols (such as numbers or signs).

Below Average
Pupils at this level do not demonstrate the skills measured by this test in Creole.These pupils have difficulties with the knowledge and skills at the low level.

Dimension
Dimension Description

D1 School Principal Leadership
School principals focus their practices towards student learning, thus requiring that they monitor results and carry out a strategic and planned vision.

D2 School Environment
The school, the administration, and teachers follow procedures and practices that are clear in terms of the organization, preparation, implementation and evaluation of teaching, considering the needs of all students by taking concrete actions so that students achieve their learning objectives and reach their potential.

D3 Pedagogical Management and Support
The school has policies, procedures, and strategies in place that favor an environment that is respectful, organized, safe, and appropriate for student learning and the school community.Additionally, the school supports and promotes the spiritual, ethical, moral, emotional, and physical development of its students.

Management of Human and Physical Resources
The school has policies, procedures, strategies, and practices in place for the management of its staff and educational resources.

D5 Community Participation
The school has presents policies, procedures, strategies, and practices in place for creating and developing the participation of stakeholders within the educational community.

What are the Results?
School performance in Haiti varies significantly across the five dimensions of educational quality.
To assess progress towards the achievement of educational quality standards for each school, a set of questionnaires were developed, pre-piloted, and piloted in 2016 and 2017, one for school principals, one for teachers, and one for school characteristics.
The data collected by these questionnaires was aggregated into a school' s profile consisting of the school' s scores on indicators that provide an overview of its level of development in each dimension.The average school performance ranged from 28 percent in the "Management of Human and Physical Resources" dimension to 62 percent in the "School Environment" dimension.
These school profiles, in combination with measurements of learning, will be valuable inputs into the school improvement plans (SIPs) that each school is required to develop with the participation of their local community.There is potential for policymakers to use RBF to encourage schools to achieve the goals set out in their plans as measured by the indicators in the QAS.Some of these improvements might actually be inputs, such as constructing proper facilities for learning (as in Dimension 2. School Environment).While school facilities are often strictly a final outcome, the lack of proper classrooms and latrines can prevent students from being able to effectively attend school and learn.
The assessment of third grade student learning in Haitian Creole was piloted in 20 schools in Port-au-Prince in 2017.
Almost 70 percent of students in the pilot had an "Average" level of reading skill or lower.Only 2 percent of students in the pilot were at the "Advanced" level, which is defined as having the skills expected by the official third grade curriculum, including being able to recognize letters and syllables, to read words and short sentences, to collect explicit information in a text, and to make simple inferences.In other words, if third grade students were reading at the expected third grade level, most students would have exhibited the "Advanced" level of reading in the assessment.It has also shifted the focus of the policy and technical dialogues in Haiti away from inputs and towards results and helped the MENFP to envision how to improve the quality of education to the benefit of Haiti' s children.While development organizations tend to focus on student learning as the ultimate and most important outcome, it is very difficult to "move the needle" and substantially boost learning in an enduring way without understanding and addressing the complex and interrelated aspects of what makes a good school.The QAS assesses not only student learning outcomes but also the intermediate indicators that make up the learning conditions within each school, including the quality of the infrastructure, teaching, school leadership, and the overall interpersonal climate.
Given the prevalence of private schools in Haiti and the long history of weak public sector management, the QAS is also acting as a guiding framework through which the government can both improve the quality of public service delivery (by prioritizing its spending according to clearly identified needs) and regulate the private sector (by making public financing conditional on schools meeting high quality standards).
The continued interest and engagement that the World Bank team has experienced from the MENFP and other sector stakeholders demonstrates that there was clearly a substantial latent demand for a dialogue on results.The development of the QAS is helping to legitimize the MENFP' s efforts to regulate the education sector by creating clear, specific, and realistic standards that will apply to both public and private schools.
At the same time, moving towards RBF is a process that will take time and multiple iterations.Creating a sustained consensus about quality standards despite shifting priorities and political pressures will require ongoing technical assistance, dialogue, and knowledge sharing.Moreover, the QAS is a work in progress, as the realities of implementation mean that the tools and approaches must be regularly adjusted and refined to make them work on the ground.
Based on the experience of developing the QAS in Haiti, it seems that successfully implementing RBF in a low-income, fragile country requires at least three elements: (i) laying the groundwork for defining and measuring educational quality, as the World Bank team in Haiti has done over the past two years, and remaining persistent despite the inevitable setbacks and slowdowns that come from instability, shifting priorities, and emergencies on the ground; (ii) recognizing that "results" can often include what we often dismiss as "inputs," which can actually matter a great deal for a school' s learning outcomes (for example, having decent, functional latrines); and (iii) recognizing that many low-income countries may already have policies that define fairly lofty ideals for standards but that having a dialogue on RBF can actually be more useful than introducing aspirational outcomes for bringing policy closer to reality.

Conclusion
The set of activities funded by REACH successfully created a quality assurance system (QAS) in Haiti that collects information on five dimensions of learning conditions in schools, provides schools with clear standards to meet, and gives them information about their progress towards reaching these standards.Now that this information is being measured by the QAS, policymakers can decide to offer monetary incentives to schools, teachers, or other relevant stakeholders to make the necessary efforts to improve these learning conditions and outcomes.REACH also supported the development of learning standards for reading in Haitian Creole.The QAS data collection instruments have been piloted and revised and have become important tools for measuring educational quality in Haiti.

Figure 1 :Figure 2 :
Figure 1: Average School Scores in the Pre-pilot Figure 2: Average School Scores in the Pre-pilot Photo credits: Cover: "Haitian Boys Creative Commons" by Feed my Starving Children is licensed under CC by 2.0 Pg 2: "Haitian Children Creative Commons" by The Global Orphan Project is licensed under CC by 2.0 Pg 6: "Haitian School Girls Creative Commons" by Feed my Starving Children is licensed under CC by 2.0 These activities have had several benefits: (i) they have increased the Haitian government' s capacity to measure educational quality; (ii) they have advanced the technical design and piloting of accreditation processes and improved the governance and regulation of the school system; (iii) they have made it easier to target programs to address schools' needs based on the best available data; (iv) they have empowered local actors (such as school leaders, principals, teachers, and parent associations) to become more involved in school decisions; and (v) they have established the foundation for introducing an RBF mechanism to incentivize school improvement.By establishing the QAS in Haiti as a reliable mechanism for measuring learning and intermediate indicators of educational quality, REACH has shown that the preconditions for RBF can be established in FCV countries.The Results in Education for All Children (REACH) TrustFund supports disseminates research on the impact of results-based financing on education outcomes.The goal to collect and build and governments and development organizations design and implement the most appropriate results-based financing mechanisms for improved learning outcomes.For more information about who we are and what we do, go to: http://www.worldbank.org/reach.