Testing Willingness to Pay Elicitation Mechanisms in the Field: Evidence from Uganda

Researchers frequently use variants of the Becker-DeGroot-Marschak (BDM) mechanism to elicit willingness to pay (WTP). These variants involve numerous incentive-irrelevant design choices, some of which carry advantages for implementation but may deteriorate participant comprehension or trust in the mechanism, which are well-known problems with the BDM. We highlight three such features and test them in the field in rural Uganda, a relevant population for many recent applications. Comprehension is very high, and 86 percent of participants bid optimally for an induced-value voucher, with little variation across treatments. This gives confidence for similar applications, and suggests the comprehension-expediency trade-off is mild.


Policy Research Working Paper 8904
How does adaptive implementation work in practice? Drawing on extensive interviews and observations, this paper contrasts the ways in which an adaptive component of a major health care project was implemented in three program and three matched comparison states in Nigeria. The paper examines the bases on which claims and counterclaims about the effectiveness of these approaches were made by different actors, concluding that resolution requires any such claims to be grounded in a fit-for-purpose theory of change and evaluation strategy. The principles of adaptive development may be gaining broad acceptance, but a complex array of skills, expectations, political support, empirical measures, and administrative structures needs to be deftly integrated if demonstrably positive operational results are to be obtained, especially when undertaken within institutional systems, administrative logics, and political imperatives that are predisposed to serve rather different purposes. This paper is a product of the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at k8bridges@gmail.com and mwoolcock@worldbank.org.

The challenge to operationalize "doing development differently"
Pointing out that something is not working is hard. Explaining why something is not working is harder. Developing and operationalizing alternatives that are sound, supportable, implementable and (ultimately, demonstrably) 'better' is typically harder still.
For many years now, the traditional, orthodox approach to development -one focused on transferring 'best practice' solutions and with limited assessment of endogenous constraints or supports -has been critiqued as a donor-driven endeavor, too often concerned with providing uniform solutions rather than crafting context-specific ('best fit') responses to locally nominated and prioritized problems. Traditional models, grounded as they are on the assumption that complex development challenges can be solved if sufficient money and expert technical advice are provided, work reasonably well for the types of problems for which they were designed, but have borne little fruit as their reach has expanded into more complex domains, such as public sector reform, which has frequently generated a plethora of institutions that look deceptively like those of their OECD counterparts but have little of the underlying functionality (Andrews 2013). The fact that a variety of countries have achieved a measure of inclusive economic development by way of contextualized institutional solutions, rather than copying the forms of others, confirms that a great deal of development consists of complex challenges for which we do not have ready-made answers. 2 What we do know is that transferring formal, institutional solutions from OECD countries to developing counterparts seldom succeeds in generating functional success.
As Booth (2018) lays out, institutions comprise both formal and informal elements, such that if we only ever transfer the more visible, formal elements we end up with structures, laws and organizations that do not have the supportive base necessary for them to be functional. 3 Moreover, institutions are shaped and impacted by idiosyncratic patterns of politics that cannot be replicated elsewhere, 4 as well as by the way power is distributed among various individuals and groups in society. 5 This unavoidable uncertainty within complex systems makes the wholesale transference of prefabricated solutions into countries like Nigeria not only unrealistic but potentially harmful, if in so doing it weakens domestic problem-solving procedures or de-legitimizes the very idea of reform.
The observations of complexity and isomorphism clearly apply in the Nigerian context. Analysis of even successful cases of functional change in Nigeria show that "reforms in Nigeria are reversible, rarely considered complete or predictable in their path" and that "[r]esults never follow best practice in terms of technical content and often represent small steps toward better development outcomes" (World Bank 2015, 5). Political economy analysis (PEA) confirms the salience of politics and leadership in the Nigerian context, highlighting the importance of state leadership and of ensuring that the federal level actors allow sufficient autonomy for the states to act (Porter and Watts 2017). In other words, like every other country, Nigeria has a slew of existing rules, often "below the waterline", that will condition the success or failure of any reform attempt. A development approach that ignores these rules can expect little traction.
Aside from presenting and analyzing the failures of the old orthodoxy, practitioners have had some success in building analytical consensus around the core principles of its proposed replacement (or complement). Terms such as 'Doing Development Differently', 'Thinking and Working Politically', 'Problem Driven Iterative Adaptation', 'Development Entrepreneurship', 'Agile Development' and the like all describe approaches that present a conscious shift from the traditional model of best practice transfer, and which are similar enough to be grouped under the umbrella term "adaptive development". 6 Emerging as a potential new orthodoxy, adaptive development emphasizes the importance of clearly identifying and understanding the nature of the problem being addressed (in particular its political economy factors) and taking small, incremental steps and adjustments towards a long-term goal. Adaptive programming is predicated on the notion that 'solutions' to complex development problems can only emerge through implementation itself; they are essentially impossible to identify at the outset of a program (Kauffman 2016).
The relevance of adaptive principles is established in recent analytical work conducted by the World Bank in Nigeria, particularly in a recent Governance Approach note, which commits the World Bank Nigeria Country Team to "move away from best practice to best fit" and to "focus on functionthat is, what works and why-rather than templates and generalized approaches copied from other operations and countries" (World Bank 2017). The righthand column of Table 1 highlights explicit commitments that the Country Management Unit of the World Bank office in Nigeria has made regarding implementation of its projects. As can be seen, these commitments correlate strongly with at least five of the six principles that are recognized as being at the heart of adaptive programming (Teskey 2017).
Though the rationale for a shift is well-established, and although alternative models have emerged, the task of implementing these models, and demonstrating their utility, is still in its very nascent stages. For the average frontline staff member, big questions remain: How does one effectively operationalize adaptive development in the midst of systems, mindsets, skillsets and incentives that have largely been built for its solution-driven predecessor? How does one do development differently in real world development contexts? And do these new approaches do what they claim they can do? Do they actually fix problems that matter?
The study from which this paper is drawn was undertaken to explore such questions. This paper is specifically written for operational staff in large development agencies, who accept the rationale for a more adaptive approach and wish to better understand how these can be actualized in country contexts. As such, details pertaining to the empirical findings -for those who wish to consult them -are largely consigned to annexes, with the main text providing an extended summary of the key findings and a detailed discussion of their implications for policy and practice. The research itself constitutes an analysis of the experience of a 'Doing Development Differently' initiative in Nigeria, which involved the piloting of adaptive assistance in eight Nigerian states, as part of World Bank support to the Saving One Million Lives (SOML) Program, itself Nigeria's first attempt to implement a project via a Program for Results (PfR) 7 framework. The Terms of Reference (ToRs) for the Technical Assistance (TA) firm hired to implement the adaptive approach state that "the most successful 6 In certain quarters 'adaptive development' itself is a noun rather than an adjective (in the fields of child psychology, for example, as well as international development). As exemplified in the vibrant online discussion group #adaptdev (see https://groups.google.com/forum/#!forum/adaptdev), however, the term has become, if only by default, an encompassing expression for the non-mainstream approaches to implementation used by those working within mainstream development agencies. 7 The original goal of SOML was to save the lives of one million mothers and children by 2015. The program has since been extended for five years as part of the Federal Government of Nigeria's (hereafter FGON) National Strategic Health Development Plan (NHSDP). The stated objective is to dramatically improve the coverage of six interventions (discussed below) that currently suffer from poor access and utilization. The World Bank has committed USD 500 million equivalent to support this initiative, to be delivered via the Program for Results instrument over a five-year period. approaches to improving development outcomes and improvements in service delivery in developing countries combine three key ingredients: working in problem-driven and politically informed ways; being adaptive and entrepreneurial and supporting change that reflects local realities; and is locally led." With a view towards drawing out emergent lessons for the implementation of flexible, adaptive technical assistance, this paper aims to do three things: (i) document the nature of the adaptive approach undertaken (specifically in relation to those "three key ingredients"); (ii) draw some conclusions about the impact of that approach; and (iii) present some guidance for individuals and groups that wish to operationalize adaptive programming in their own contexts.

Methodology
The analysis on which the conclusions presented here rest took place in three phases, between May 2018 and May 2019: Phase one of the fieldwork entailed conducting exploratory research in three of the TAreceiving states: Kogi, Abia and Anambra. It sought to document the nature of the tasks actually performed by the TA firm -Adaptive Development Group (hereafter ADG) 8 -and to make a preliminary identification of the contextual processes and organizational dynamics driving observed impacts. Interviews, focus group discussions, and participant-observations were undertaken with various stakeholders in the three states, and were complemented with an assessment of reports from the TA firm, SOML project documentation, and other literature relevant to the PfR and to the TA component.
Building on phase one, the aim of the second phase was to further determine impact and relevance of the provided TA, specifically by comparison of findings from TA states vis-à-vis otherwise 'comparable' states that did not receive TA from ADG. The strategy of this phase was to visit three states that did not receive TA -Enugu, Akwa Ibom and Rivers -but that had similar political, social, financial and geographical contexts to the three TA states already visited. These decisions, and choices regarding where specifically to visit in each state, were made on the basis of extensive consultations with Nigeria-based colleagues and government counterparts. Within each state, the team met with various SOML stakeholders to ask about the ways in which they have understood the SOML PfR, designed workplans and implemented interventions.
In each state and during the two fieldwork phases the team sought to meet with the following stakeholders: In addition, there were interviews undertaken with World Bank staff, ADG staff and stakeholders from the Federal Ministry of Health in Abuja. The final phase of the research entailed analyzing fieldwork data and integrating it with material collected from the two field visits, preparing papers for different audiences, incorporating feedback on them and securing the necessary clearances for their release. 1) Context is everything "We will continue to base operations on a solid understanding of technical challenges, but we will also strive to understand local contexts and feasible reform processes as opposed to filling technical gaps or deficiencies in our clients' current approaches." 2) There is no singular blueprint for development -rather flexible, responsive, adaptive programming "There will be a presumption in favor of flexibility and adaptation in designing and implementing Bank interventions." "…we will be looking to scale up our support to local pockets of efficacy where the likelihood of sustainable reforms is greatest." 3) Best fit not good practice "The Nigerian context calls for flexibility over the lifetime of projects with regard to technical approach, counterparts, phasing, communications, geographical scope and scale, and so on. Accordingly, we will encourage flexible project design and encourage and reward project restructuring based on solid supervision and implementation facilitation." 4) Enabling, not doing "While recognizing the importance of leadership for reform, we will expand the range of stakeholders included in the design and implementation of our engagements, with a view to creating broader support and sustainability for reforms." "Ownership of approaches will be tested throughout the delivery cycle and, where it does not exist, exit strategies will be employed." 5) Real-time learning "Information-baselines, evidence, and feedback-will be a central feature of our work as we test and tailor approaches." "Our approach will be based on problem solving and unpacking assumptions behind hypotheses." "We will supplement our focus on rules, legislation, manuals, and formal institutional designs with a focus on implementing these rules and institutional designs, etc." 6) Long-term commitments with staff continuity n/a The paper proceeds as follows. Following this introduction, section 2 lays out further details regarding the contexts within which 'adaptive' implementation work was undertaken within a national health project in Nigeria. Sections 3 and 4 provide an extended summary of the key findings, providing the empirical and analytical foundations for section 5, which explores some specific implications of these findings for policy, practice and theory. Section 6 concludes. (Detailed tabular presentations of the empirical results in each of the six states visited are presented in the annexes.)

Contexts: Doing 'adaptive' work in a national health project
The SOML PfR in Nigeria has as its primary goal helping states achieve specific improvements in six pre-defined health interventions (the 'six pillars'): This definition of success formed the boundaries in which ADG, the TA firm, was expected to then undertake a process of locally-driven problem-solving within the states. ADG was required to work with states to identify, analyze and 'fix' the wide range of problems that cumulatively contribute to low coverage in the above areas, and therefore low health outcomes among the Nigerian population. Since low coverage in the six different indicators is unsurprisingly caused by a wide range of factors, often very specific to the peculiar context and personalities present in the state, ADG was inevitably engaging with a wide range of very diverse problems and causes. 9 In documenting the approach undertaken, we find that the TA provided by ADG exhibited aspects of all three of the "key ingredients" stipulated in the firm's Terms of Reference: "[1] working in problem-driven and politically informed ways; [2] being adaptive and entrepreneurial; and [3] supporting change that reflects local realities and is locally led".
The problem construction and deconstruction process facilitated by ADG appears to have been locally-anchored and sincerely appreciated by state officials. This is despite the fact that, in our view, the scope of problems ADG was tasked with fixing were large and diverse to the point of being unmanageable. Officers from the three State SOML Program Management Units (PMUs) that ADG were assisting narrate how the firm constantly pressed them to justify their proposed interventions on the basis of how it would affect the six outcomes, thus inculcating a far greater attention to data than usual. While it is apparent that the process ADG facilitated for identifying the causes of low coverage in the indicators was more thorough and nuanced than the traditional solution-focused approach, the identified causes and problems themselves were not actively captured on paper or routinely revisited in the states. Nevertheless, with the help of their data capture platform, PIGEON, officers from the PMU and within local government were being alerted on a week-to-week basis of low reporting levels or unexpected results, and immediately following up on them directly with local government counterparts and with health facilities (often with advice from ADG as well 10 ). In this way, problem-solving was made an active feature of ongoing implementation efforts. 9 A small snapshot of the sort of problems that have been identified as requiring "fixing" is reflected in the following list: People use the bed nets we distribute for fishing so that they can catch fish to eat and sell; Health workers have not been paid for the past six months so they do not come to the facilities for work; The roof of the health facility has fallen in so no-one wants to go there; Basic equipment and drugs are unavailable in the health facilities, so people do not see the value in going there; Women do not use contraceptives or use them and do not report it; Religious leaders are opposed to the use of contraceptives by their members; Some health facilities are inaccessible in the rainy months; Health workers charge people for services that should be free, thus dissuading them from using them; Some people in the community do not trust immunizations, believing there is a plot by another identity group to infect the local people with a virus; Traditional birth attendants are more accessible, particularly for rural communities than skilled birth attendants in the health facilities and they offer more services (they will cook you pepper soup); There is no transport available for taking pregnant women to secondary health facilities; Health workers do not come to work because of security issues and doctors having been kidnapped in the past; It is culturally problematic for health workers to come into a person's home, particularly if they are of a different gender, thus limiting the ability to monitor or advise on mosquito bednet utilization; People get upset if we test them for HIV but then we do not have drugs available for treatment so some health workers avoid testing. 10 Like the DHIS data, which are collected at the facility rather than household level, the PIGEON data are known to have a poor correlation with state level household survey data; as such, their use by ADG (as providing an ADG placed a particularly strong emphasis on the identification and management of political and non-technical drivers of the problem. The consultancy was co-designed by Bank colleagues from the outset not to be driven by technical best practice but rather to provide what some staff felt had been missing in previous TA, namely politically savvy local knowledge. 11 Interviewees were unanimous in identifying politics and personalities as the main determinant of smooth workplan implementation in the Nigerian states. In both the TA and non-TA states we assessed, there was strong evidence of powerful individuals either positively or negatively affecting the smooth functioning of SOML (see Annexes 8.1 and 8.2=3). Accordingly, during state planning workshops in early 2016, ADG drew participants' attention to power dynamics in their states, helping participants to identify "key influencers" in their states -such as the first lady, traditional rulers, religious leaders, private sector actors, civil society, trade unions etc. -and the importance of managing them. ADG encouraged the state PMUs (SOML's Programme Management Units) to consider the "electoral value" within all their initiatives, prompting them to opportunistically access the political will of certain key players. ADG reportedly pushed the PMUs to be better at highlighting good results, using data visualizations and capturing stories of success in a bid to maintain buy-in.
Beyond simply applying a political lens and operating in a politically savvy manner, ADG also saw their role as one of advocating for the PMU and managing political interferences directly in an effort to create (and protect) space for implementers. In Kogi and Anambra, ADG made active, sustained attempts to manage problematic interference and was largely successful. In Abia, similar attempts were made, but the power of an interfering individual was so significant that success was only realized after the program had already been seriously undermined (see Abia snapshot, Annex 8.1).
We needed to uncouple the political from the technical. Civil servants in Nigeria are so cowed, so scared to push back on the interference. But when those interfering elements see that we are engaged as well, they think twice…" -Interview, liaison officer, ADG This "political hustling" goes beyond the typical facilitative role that would be expected of a firm supporting adaptive programming and makes the Nigeria work particularly unique.
Regarding the second ingredient -"being adaptive and entrepreneurial" -there is some evidence that an emphasis on problem diagnosis, planning and PfR sensitization left limited time for cycles of learning and adaptation. Though the SOML and ADG's assistance began in January 2016, it was early 2017 before any actual implementation of solutions began in any of the states. Partly this is due to a delay in SOML funds reaching the states 12 ; ADG had anticipated funds reaching states by March 2016 and implementing workplans from then, on the back of momentum gained and political authorization obtained during the preceding months. But due to the delay they instead entered what they describe as a "holding pattern…. all dressed up with nowhere to go" (interview, ADG). 13 Once accurate snapshot of health outcomes) has been questioned, particularly by Bank staff. Federal level staff also questioned why there was a need for what they see as "a parallel system to DHIS". But at the state level, the impression given was that PIGEON functioned as a rudimentary problem warning system that became invaluable in establishing a culture of close monitoring, performance management and follow up. 11 Bank staff weighted the evaluation of the proposals with a particular view towards political and technical competence and then insisted on face to face interviews in order to test these skills. Reportedly, the weighting given to ADG was largely down to the support they expressed for a performance pay arrangement (which Bank staff took as a "sign of their commitment to a paradigm shift") and their evident local and political knowledge. 12 Funds were made available for draw down in the states much later than initially anticipated (for a variety of reasons, including Bank delays as well as bureaucratic and political obstacles within the states), which had the effect of delaying anticipated implementation by over a year in some cases. 13 For the next year ADG continued engagement, mainly by focusing on the setting up of a data capture platform -known as PIGEON -geared towards ensuring weekly data collection once implementation began. They describe their task during this period as "preaching, handholding and maintaining a presence". This was manly done via phone and email since at this stage they had no liaisons based in the states. implementation finally got going, in early to mid-2017, there was evidence of adaptation and of innovation (see Annex 8.3 for comparison of innovation across the visited states). The workplans are clearly understood to be flexible documents by the TA states and this fact appears to have left room for some "muddling" and iteration in the states during implementation. But the basis on which microadjustments were encouraged by ADG was not always apparent. In the absence of well-documented theories of change, it does not seem that adaptation involved methodical reassessment of initial assumptions or was always carried out in a way that required clear justifications. State implementers also reflected that they felt the need for better data analysis skills to help them adapt in accordance with the evidence. 14 Regarding the third ingredient -"supporting change that reflects local realities and is locally led" -ADG conveyed that they felt it was within the implementer's capability to identify challenges and to come up with the most appropriate solutions to address these. While ADG did recommend the possible application of certain solutions based on their own knowledge of what had worked well in other contexts, they appear not to have seen their role primarily as experts offering solutions. 15 Rather, they envisaged their role as being to create space for a latent creativity and innovation to emerge. Accordingly, even though they gave training to assist with problem identification and asked questions that helped PMUs to refine their analysis, the onus was placed on program officers responsible for each of the interventions 16 to be the ones to actually identify the critical drivers relating to their theme. Decisions about strategy and prioritization seem to have been made by certain individuals (typically program officers responsible for the different thematic areas) largely on the basis of their intuitive ('gut') sense of what was achievable. (As one of ADG's liaison officers put it, "The states actually know what to do and how to do it. But they need space and confidence…they need the leeway to do what they do. You have to clear the way. Our role was to create space.") It is our impression that after the MICS 2016 survey results came out, highlighting the negative performance of the eight TA states against the six outcome indicators thus far (in our section on impact, below, we question whether this was an appropriate means of measuring the success of the TA), ADG began to operate somewhat more out of a pressure to achieve results than to ensure a locally-driven process. Having begun with an emphasis on very state-specific, tailor-made solutions, it appears that towards the later part of their contract, ADG began to push for more standardized solutions, most likely reflecting their increasing desperation to improve results (and thus obtain a performance payment as well as a potential extension of their contract with the federal government). Apparently, Bank staff also gave explicit guidance that ADG should 'do more' to support expansion of two particular activities: Maternal and Neonatal Child Health (MNCH) 17 weeks and Quality Implementation Supervision Support (QISS). 18 ADG thereafter placed various "liaison officers" in the 14 Though public sector staff in the TA states did refine interventions based on data analysis, they admitted finding it difficult to drill down into what is driving failure of certain interventions and expressed concern that they were sometimes misinterpreting data. 15 One of ADG's liaison officer notes that in two states there were proposals to do costly launches of the SOML program, which would have used about N 10 million of project funds. Rather than saying they should not do it, ADG's approach was to ask the states: How far will this move any of your indicators and are there other places it could be put that would move them more? The response by the states was to revise the workplans and to incorporate the launch into an actual rolling out of activities. 16 In most states there is one program officer for each of the SOML thematic areas e.g. a malaria officer, nutrition officer, immunizations officer etc. 17 States are expected to conduct week-long campaigns twice a year that are targeted to cover a range of immunizations, nutrition objectives and other interventions for mothers and children. 18 QISS refers to regularized visits conducted by state and local government staff to monitor the range of equipment, infrastructure and services in operation at health facilities across the state. Teams of government staff visit each facility with a checklist and submit their findings back to local government representatives and the SOML PMU.
states towards the end of the contract, with explicit ToRs to strengthen implementation of these specific solutions, rather than to continue the locally-driven problem-solving process more generally.
Part adaptive facilitator, part political hustler, the firm also wore a more traditional technical assistance hat at times, devoting considerable effort to sensitization, training and advisory tasks typical of more solution-driven approaches. A large part of the firm's work in the state was devoted to sensitization on the mechanisms and objectives of the SOML PfR (which ADG insists took up far more of their energy than originally anticipated), setting up and training on data capture solutions (specifically PIGEON) and strengthening the roll-out of some pre-selected health interventions (as encouraged by the Bank), such as the QISS and MNCH weeks.
In short, the firm's approach might be best described as a hybrid, or merged TA approach, combining elements of old and new, with an accompanying tension, messiness and innovation all part of the package.

Findings 1: How context conditions the approach
The hybridity of the firm's approach should not surprise us, nor should the fact that the approach exhibited weaknesses. Sometimes the obvious requires stating: adaptive approaches do not take place in perfectly insulated environments but rather in complex contexts comprising (by definition) competing incentives, varied skills, divergent mindsets and inherently imperfect mechanisms.
Firstly, the quality of the adaptive process is necessarily conditioned by the firm facilitating that process. Wild et al (2017, 25) note that "whether or not programmes are genuinely doing development differently depends on who is implementing them" and that "[s]ervice providers or implementing partners need teams and staffing with the right capacities, skills and experience to be able to manage programmes in new ways". The scope of problems ADG was tasked with fixing was large and diverse to the point of being unmanageable; indeed, to some degree we consider it a reflection of their inexperience that they took on a performance-based contract with these payment terms. 19 The firm also struggled to support effective monitoring of outcomes, with Bank staff expressing serious concerns about ADG's ability to collect credible data and carry out quality data analysis. In the absence of the firm carrying out a household survey (as had been recommended in the ToRs), it was very difficult for Bank staff to monitor their progress. Our analysis of ADG's reports also indicates that their reporting skills were weak -language was often convoluted and opaque, and there was little reflection on problems fixed, adaptations made, or assumptions revised. 20 The reports tend to follow a more traditional structure, focusing on inputs delivered. A failure to methodically capture collectively identified problems and their causes appears to have led to limited reassessment of assumptions and reduced the quality of ongoing learning. Adaptation was certainly encouraged and incidents of it during implementation are evident, but it only got going latterly, shortly before the TA itself ended altogether. Some Bank staff and federal level actors questioned whether ADG had sufficient technical understanding and expertise to carry out their assistance role. According to the federal level actors (with whom ADG had an unfortunately antagonistic relationship), ADG implemented technical solutions in the states that were ineffective and not based on a good understanding of what interventions would have the best results. Partly, this critique reflects a more traditional best-practice mindset, unaccustomed to this notion of finding locally-driven, hybrid solutions. 21 Our own analysis confirms some instances where the firm could have been stronger at offering a variety of technical solutions as options for implementers to consider -options that were in keeping with the existing conditions and problems as identified by local stakeholders. That said, the very novelty of adaptive implementation -in Nigeria and elsewhere -means that, by definition, every firm that takes on this challenge for the first time is bound to encounter serious difficulties. (This does not absolve ADG of reasonable criticism, but it also means that their genuine efforts in these challenging circumstances should be fully appreciated.) Aside from the capabilities of the firm conditioning the quality of the approach, there are elements of the World Bank's organizational environment -its skillsets, incentives, instruments and resources -that either support or undermine a more flexible, problem-driven model. For example, although state level officials testify to the fact that the PfR instrument made possible a far more innovative, locally-empowered and responsive way of working, it is also true that the instrument itself is still extremely novel in Nigeria and required significant socialization. Field work confirms that various agents in the states have faced significant challenges, delays and unintended effects as they have tried to understand and navigate this new performance-based system that they have hitherto never encountered. The initial state advances were themselves seriously delayed and there continue to be challenges passing PfR payments through the state system: at the end of a year of implementation, an average of only 33% of funds had been disbursed in total. Drivers of this slow disbursement include bureaucratic obstacles (such as imposition of and subsequent required training for the accounting system, REMITA), infighting between the state and primary health care providers (the SPHCDAs) and rank opportunism and interference, as exemplified by the Abia case (Annex 8.1). In this context, ADG's TA was required to be not only an agent of socialization into the brave new world of adaptive implementation but also into PfR. Delays in fund disbursement contributed to a misalignment of TA and state implementation, 22 a somewhat unbalanced emphasis on planning, and what was perceived by various actors as an abrupt (and unfortunate) exit of ADG's support. World Bank staff and federal level officials have also been learning and adjusting to the PfR instrument as they go, and they readily refer to aspects that, given a second chance, could be designed differently. For example, although the SMART survey, on whose results performance payments were to be made, was expected to take place every year, in practice it was used to form a 2015 baseline but then did not take place in 2016 or 2017 and has only recently been undertaken in 2018. 23 The use of an alternative survey tool in 2016 raised significant outcry and has affected perceptions of overall fairness. 24 The fact that this 2016 survey, against which performance was measured and payments made, took place in August 2016 -i.e., before the states had begun implementation -also generated controversy. Stakeholders disagree as to whether it was reasonable to expect that states could have initiated some activities even without these funds, but all agree that the level of delay was unanticipated and that funds were expected to have been in the states much earlier.
Regarding Bank monitoring of the TA, the SOML supervision budget covered two weeks of staff time supervising the TA, though no in-country staff had a direct monitoring responsibility. 25 Initially, the firm was encouraged to carry out a mini household survey to check progress mid-way through the assignment, but this did not take place. 26 The Bank TTL for the TA (from the Governance practice and based out of country), who acted as the contact point for ADG, evaluated their deliverables (sharing them with health sector colleagues) and managed their payments, confirms that monitoring was made additionally difficult by the fact that adaptive assistance, by its nature, is tough to report on or assess. In the absence of a reliable means of viewing progress on indicators or a reporting system that captured evidence of problems solved or capabilities enhanced, Bank staff began to question what was actually happening on the ground -but had no means of verification other than the word of the firm itself and their fairly convoluted, opaque reports. 27 Missions to the states (and especially rural areas) are logistically difficult 28 and the TA firm's communication style has also been challenging. 29 As a result of growing concerns, Bank staff began to push the firm to implement solutions, specifically some solutions that it felt were technically appropriate. These solutions were dutifully implemented across the range of contexts in which ADG was operating, and as a result (as noted above) the TA began to resemble a more orthodox approach. Although there was certainly an incentive for states to try and track how their chosen interventions impacted the six indicators, the focus on those state-level aggregate measures meant that examples of local level or discrete problems being addressed tended to be ignored, or at most viewed as a means towards an end and not as an end in and of themselves. In a determined and arguably well-intended effort to "get out of the kitchen", Bank and federal staff may have inadvertently neglected the chance to observe and learn from successful, local level performance improvements.
The political currency of adaptive development is such that authorization and engagement for such an approach is often more passive than active, and in this particular case the loss of key Bank staff during implementation of the adaptive TA had a negative effect on internal authorization. While the commitments outlined in the Governance Approach note (World Bank 2017) are certainly 'adaptive', it is our view that they reflect the commitment of a core governance team, rather than the Country Office more broadly. And although passive authorization may be sufficient for programs that already achieve wide organizational buy-in, that passivity is often insufficient in large aid facilities, where "the context, the inheritance, strongly favours doing development pretty much as it was done previously" (Teskey and Tyrell, 2017). Even where staff are in principle remarkably open to adaptive programming, they will likely struggle to resist the incentives aligned against it -e.g. pressure to preprogram, to demonstrate linear results and to deliver best practice solutions. In the absence of felt they had expended particular effort on this indicator were particularly frustrated with the results. The outcry was such that it was eventually agreed to advance all states 25% of the possible performance payment, regardless of actual results. 25 Attempts were made to address this -by inviting locally-based staff to join supervision trips -but they were unsuccessful. 26 The main reason seems to be the cost associated with such a survey. 27 The firm for its part, appears to have been reporting to the best of its ability but genuinely struggled with how to report on the vast scope of problems that they were engaged in tackling with a wide range of stakeholders. 28 Testament to this point, the logistics required for the state visits conducted during this review took many weeks to organize, required lengthy visa processes, security clearances, armed police escorts and, needless to say, significant cost. 29 The key ADG point person has a loquacious, somewhat confounding communication style, which, given that he has been the key point of contact, meant factual details were sometimes hard to come by. significant 'unlearning', upskilling and consensus-building having taken place, it is likely that authorization will fluctuate and there will be a continual (perhaps inexorable) reversion to a more traditional approach. In our case study, this manifested as progressively less active engagement in the TA component by non-core staff and increased pressure from Bank staff towards the firm for implementation of specific targeted solutions.
In addition, the Nigerian states themselves also present highly varied political contexts, capabilities, personalities and donor presence, all of which have the potential to significantly affect the impact of the SOML and of the TA component. (Annex 8.2 summarizes these factors while Annex 8.3 contains a brief side-by-side comparison of the six states, giving some indication of how these contextual divergences have significantly altered the way SOML has operated in each of the states.) We argue that even if the selection criteria for the eight TA states are relatively clear, 30 there remain many differences (observable and unobservable) between the states, a number of which are likely to contribute to divergence in health outcomes and which necessitate a tailoring of the TA approach.

Findings 2: In the midst of messiness, impact nonetheless
Despite hybridity and the various challenges highlighted above, there are strong indications that the TA was indeed successful in solving discrete problems and in stimulating positive shifts in organizational behaviors and strategies. We find there is evidence that the TA states have been somewhat successful in fixing various performance-related problems, albeit at a more discrete level than the six aggregate outcome level indicators can possibly capture. Although this current assessment of the six states is not sufficiently wide or deep to claim with certainty that the adaptive TA was 'successful', it is certainly the case that stakeholders in the two TA states where implementation is well underway (Kogi and Anambra 31 ) were able to talk compellingly about a range of specific problems they had identified and context-specific ways they had discovered of resolving them. By contrast, stakeholders in the non-TA states that have made some progress with implementation (Akwa Ibom and Rivers) refer mainly to standardized interventions like QISS and MNCH weeks when asked about interventions that have had impact. 32 (See Annex 8.3 for side-by-side comparisons.) Not only do the states that received TA exhibit more evidence of functional problems being fixed than their non-TA counterparts, but they also exhibit more evidence of enhanced problemsolving capabilities and behaviors generally. TA states demonstrated greater appreciation and pursuit of coordination across stakeholders, were undertaking more collective problem solving, were more cognizant of the necessity of managing higher level "authorizing" relationships, and demonstrate a greater understanding and appreciation of the non-technical (or informal) drivers of the problems they are trying to fix. Along with a more nuanced understanding of the problem, TA states appear to 30 Eight states were selected to be recipients of the TA from ADG. According to Bank and Government staff, the aim was to select states that were already quite capable at health implementation, and that might thus act as "pace setters" for the others by providing a positive demonstration effect showcasing the value of adaptivestyle TA while also motivating other states to improve their own performance (interviews, World Bank, Federal officials). Implementation 'capability' of the states appears to have been judged solely on the basis of how well states were performing on the six SOML indicators prior to the program's commencement. Sixteen top performers were identified on this basis, covering four different regional zones of Nigeria. (The divergence in performance between Northern and Southern states is so marked that all 16 of the sample group come from the center or the South.) Two states were then randomly selected from each zone to be part of the TA complement, making eight states in total: Abia, Anambra, Delta, Edo, Imo, Kogi,Osun,and Oyo. 31 Recall that the other TA state -Abia -experienced a derailment in its program due to excessive political interference (see Annexes 8.2 and 8.3). 32 The only contextualized, successful solution that we identified in the non-TA states was in Akwa Ibom, where they were piloting "husband schools". have selected solutions that show a greater appreciation of the context in which they operate, have favored more innovative solutions, and seemed more confident about experimenting and adapting during implementation than their non-TA counterparts. TA states have made greater attempts to improve collection and utilization of what data are available and are actively using them to ensure that interventions link to SOML objectives. TA states also demonstrate a noticeably higher level of understanding regarding the nature and mechanisms of the PfR instrument.
In short, our analysis of a hybrid adaptive programming approach in Nigeria finds that such approaches can make important gains in complex areas and on problems that have hitherto appeared intractable. This is despite the fact that such approaches will tend to exhibit all the tension, messiness and hybridity that one would expect of attempts to introduce a novel approach into a development sector such as health (where changing behavior, especially intimate behavior, is widely recognized as deeply challenging). On the other hand, we recognize that these gains are not irreversible, and in keeping with the views of ADG and various recipients of the TA, we believe that the TA would have benefitted from a longer running time -too much was expected from too few too soon. We note that the Program Support Units (PSUs), which have since been hired to support SOML in the states, have been given ToRs that are far more orthodox (i.e., solution-driven) in their approach, and are highly unlikely to enhance the behavioral shifts that have been made thus far.

Implications and applications: Some guidance for the intrepid
For the development staff who are disenchanted with the old orthodoxy, eager to operationalize adaptive principles and ready for the adventurous (and at times messy) ride, we conclude by offering some guidance and reflections that we hope will be helpful and useable. This guidance is organized according to four core challenge areas associated with implementation of adaptive programming: (1) Identifying the problem to be fixed; (2) Monitoring and evaluation; (3) Selection and management of the firm; and (4) Maintenance of internal authorization and support. It fleshes out some practical lessons in relation to each of them. This guidance is by no means exhaustive; rather, we have focused on aspects that we think are currently under-addressed in the existing literature on the subject.

Identifying the problem
Start by facilitating the selection of a good problem. Note the word facilitating. Often donors create projects to fix problems that they care about, rather than ones that matter to the actual implementers or beneficiaries. Often the problems that are selected are actually "the absence of a specific solution" (e.g., there is no functioning Anti-Corruption Bureau; staff salaries need to be rationalized; compliance with reporting is low) rather than something that constitutes a specific issue of service delivery. In the Nigerian case, the fact that the metrics of success (and by inference, the problem) were already predetermined and framed at a very high, aggregate level, (i.e., the six state-level health outcome indicators) made the scope of challenges that the firm had to deconstruct virtually unmanageable, and possibly not prioritized by the implementers they were working with.
It does not work when outsiders analyze the problem on behalf of those who will act to solve it. It works when those in the insider PDIA teams construct and deconstruct the problem (whether they do this 'right' or 'wrong'). The insiders must own the process, and the outsiders must 'give the work back' to the rightful owners.
-Matt Andrews 33 To be clear: we are not saying that the selection of the six indicators was 'wrong' -these are eminently sensible goals that the elected government has determined it wants to achieve -but rather that the measure of ADG's success should have been far narrower and more locally proscribed, focused on the extent to which it helped implementers solve problems that the implementers themselves (in partnership with recipients) have discerned. At the aggregate level, movement in the six health indicators, whether positive or negative, is a function of a vast assemblage of interlocking factors, only a small fraction of which could ever be clearly attributable to ADG.
For proponents of adaptive development, beginning an intervention with collective identification of a clear, locally relevant problem is considered critical to building support for the program and for ensuring that whatever solutions are developed are relevant, legitimate and practical. In an effort to assist that problem construction process, Andrews et al. (2017) provide three pointers for what constitutes a 'good problem' as well as some tools for how to get there: To ensure that whatever problems identified do matter sufficiently to Nigerians, teams can try to ensure that in constructing a problem statement, the statement should be able to answer five questions: 'What is the problem?', 'Why does it matter?', 'To whom does it matter?', 'Who needs to care more?' and 'How do we get them to give it more attention?' In answering these questions teams can arrive at a problem statement that is clearly focused on important service delivery failures and also enables easy construction of an aspirational goal of problem solved.

2.
A good problem can be broken down into easily addressed causal elements.
Tools for deconstruction such as the "5 Why Technique" and "fishbone diagram" can be used to identify the various causal elements of a problem, including those that are typically "below the waterline" (such as the more cognitive or normative drivers that form behavior). This deconstruction of our problem will likely show that there is not any one cover-all solution but instead a multitude of small solutions that will need tailoring to each causal dimension. It will also raise questions such as "Where do I start in trying to solve the problem? What do I do? How do I ensure that all causal strands are addressed?".

A good problem allows real, sequenced, strategic responses
Each identified cause and sub-cause should be treated as an essentially separate-albeit connected-point of engagement, each offering a different opportunity for change ("space for change" or "readiness"). Teams can ask therefore-for each sub-causal strand-what the authorizing environment looks like and where authority for intervention will come from, whose acceptance is needed to move ahead, and what kinds of abilities are needed to make real progress. This will enable teams to determine which areas will warrant activities that grow the change space, whilst others will allow more aggressive reform or policy adjustment because the change space is already perceived as sufficient.
Defining and deconstructing the problem requires savvy facilitation. The pointers and tools mentioned above are undoubtedly helpful and practical, but it bears repeating that one should never underestimate the difficulty that is often to be found in constructing and deconstructing problems. In the absence of savvy facilitation, actors struggle to decide which level of a problem to focus on, how far to break it down into its causes, sub-causes and so on as well as which aspects are 'contributors' or fresh separate 'problems' in and of themselves. 34 Most development professionals are trained and conditioned to think in terms of providing solutions. Often, when one asked staff of MDAs the question "What is it you want to fix?", the answer was framed in terms of the absence of a specific solution, rather than in terms of specific service delivery issues that matter to communities. Government counterparts have been taught the language of 'best practice' after years of learning that speaking in these terms is what keeps donor funds flowing. It is often difficult to sufficiently 'remove hierarchy in 34 Within the adaptive community perhaps more can be done to refine and expand existing toolsets, but there is also clearly a need to work on identifying and building the particular skill-sets that are required of facilitators who can assist these kinds of construction and deconstruction processes and establishing networks of such individuals who can assist in-country teams. Some of these skillsets may be teachable, but some may be innate.
the room' such that stakeholders can be honest about where they see breakdowns in the implementation process or where there is tension about the diagnosis itself. In such contexts, it requires savvy facilitation to help actors peel back the necessary layers until they identify the elemental (and actionable) issues at stake.
Do not get 'stuck' in the problem or planning phase. Identifying and deconstructing the problem is a necessary foundational phase. Plans of action are also important. But in the Nigeria case there is a sense that work floundered somewhat in this stage, that 'planning' continued for too long and that action was overly tied to incoming funds rather than existing resources. 35 The result was frustration, loss of momentum, and implementers being distracted from the daily work of doing. Teams need to move quickly from analysis and plans into action and learning if momentum is to be maintained. An adaptive approach is predicated on the notion that you cannot plan everything out in advance in any case; no team can achieve perfect understanding before starting the work. Given the flexible, adaptive nature of this approach, it is useful to see planning as a continuous exercise rather than a one-off event, manifested in a single document. The BSC PDIA team, for example, initiates planning in what they call a 'Launchpad Event', that takes all of a day and involves their facilitators working with a number of internal teams to tease apart existing problems. That process generates a one-page action agenda that is intentionally short and simple and includes the following information.
-a description of the problem, and why the problem matters -the causes of the problem -what the problem might look like solved (and especially what this kind of result would look like in the time period within which they are working; usually 6 months) -what the 'indicative' results might involve at the 4-month and 2-month marks (working backwards, the facilitators ask 'Where would you need to be to get to the 6-month 'problem solved' result?) -fully specified next steps (where the teams identify what they will do in the next two weeks and what they plan to do in the two weeks after that) -what is assumed in terms of authorization of the next steps, acceptance of these next steps, and abilities to do the next steps (they want teams to specify their assumptions, so they can track and learn where they are right and wrong and adapt accordingly in future steps). 36 World Bank teams can learn from these approaches and in doing so shift partners' and implementers' emphasis from lengthy planning to iterative, action-oriented learning.

Monitoring and evaluating an adaptive approach
Adaptive approaches, given their novelty, need to make monitoring front and center of implementation. The novelty of adaptive programming and the extent to which it is still poorly incentivized in major development institutions ensures that passive or distant management of an adaptive approach is likely to result in derailment, especially in contexts where a complex mix of disparate motivations and capabilities prevails. Aside from being committed to adaptive principles themselves, task managers responsible for adaptive programs should anticipate the need for both minor and major course corrections through the lifetime of the program and will need to be on hand to guide and navigate these. This role is unlikely to be fulfilled by the occasional 'supervision mission'. As one experienced practitioner recently put it, "designing, contracting and implementing programs 35 There may be a sense in which the provision of external funds to undertake adaptive work is inherently problematic, since it complicates incentives and drive, adding an additional 'carrot' into existing motivations. If what is desired is that stakeholders implement solutions based on existing capability and a drive to fix the problem, then the provision of funds can risk diluting ownership and momentum. This is evidently the case when those funds are delayed, though it is hard to know whether it would still have had a negative impact on ownership if they had been timely. This is an area for continued research. 36 https://buildingstatecapability.com/2017/03/27/active-and-adaptive-planning-versus-set-plans-in-pdia/ to work in this way is time consuming and challenging -often involving swimming against the tide of conventional practice and expectations" (ODI 2016, 9). 37 It is unrealistic to imagine that such work can be managed out of country or with negligible supervision budgets. In the Nigeria case, Bank staff found that management of this adaptive component required far more of their time than had been budgeted for and required smarter monitoring tools than those they had at their disposal.
The success or failure of adaptive programming cannot be measured with tools designed to measure traditional approaches. Ladner (2015, 3) points out that "standard performance monitoring tools are not suitable for highly flexible, entrepreneurial programs as they assume that how a program will be implemented follows its original design". A traditional method of monitoring will typically track a program's progress in achieving predetermined milestones and indicators of success against an assumed path to change. In a project where the causal connections are well-established and where we 'know what works', this approach makes sense. But for a project where we expect to learn much about what does and does not work during implementation, the attempt to formalize milestones and process indicators at the start can risk derailing the iterative approach needed for success (Preskill et al 2014, USAID 2014, Pritchett et al. 2013. As Ladner (2015, 3) puts it, "Investing significant resources up front to establish baselines, identify indicators, and set up data collection protocols could inadvertently 'lock in' a theory of change and prevent program teams from exploring other options or seizing on opportunities when they emerge."

Success measures should focus on the targeted problem, and not an aggregate indicator far removed.
Proponents of adaptive programming insist that the adaptive approach is a better way of solving problems and building implementation capability than the typical solution-driven, best practice model. This claim should be put to the test. It is legitimate and necessary to examine whether an adaptive approach actually fixed the performance problems it targeted and whether the capabilities of individuals, teams and organizations were improved such that they are better equipped to resolve problems as they emerge in the future. Of course, the critical factor here is that the problem being measured must correlate to the one that the adaptive approach was actually working on, and not one that is many miles down the line of one's Theory of Change assumptions. In the Nigeria case, the firm was being judged on the attainment of state-wide aggregate health outcomes. In reality, it was working on issues as discrete as ambulances arriving late or doctors not turning up to work. We may assume that the fixing of these discrete, local level issues will progressively combine to shift statewide health outcome indicators, but that is an assumption that requires testing. As noted above, a vast assemblage of factors must come together to shift a state-wide indicator, and even if they ultimately do, the trajectory of change in such complex interventions is rarely linear (Woolcock 2013) and the specific contribution of the 'adaptive' component will be almost impossible to empirically isolate. In the meantime, solving local, discrete but functional problems matters -in its own right and because failing to do so creates a blockage point, or 'binding constraint,' on the larger system's overall functionality. What should be measured is the extent to which such constraints are actually being 'unblocked' and 'unbound'.
Measuring whether improved problem-solving is taking place will require more continuous and more 'relational' ways of monitoring. Drawing on insights from the technology sector -where principles of 'agile' or 'adaptive' technology development are well-established - Wild et al (2017, 25) note that an adaptive program by its nature will most likely require "more continuous and close involvement between both sides from the start (rather than the classic model where a funder hands over what is contracted, and only interacts if there is a problem)." As such, more 'relational contracts' become relevant. Teskey and Tyrel (2017) highlight the utility of donors participating in regularized formal and informal Review and Reflection (R&R) points, during which the contractor (and, we would insist, the implementers too) demonstrates how politics, interests, incentives and institutions were 37 Staff involved in the SOML PfR have likewise acknowledged how much more of their work and input the TA management required beyond initial anticipation and budget.
systematically considered in problem selection and design, and justifies why certain choices were made to stop, drop, halt or expand any activity or budget during implementation. The Harvard BSC team highlights light-touch methods, such as their PDIA "check ins", which include a series of probing questions to assist teams in capturing learning and maximizing adaptation. 38 Their more detailed "SearchFrame" is offered as a tool that "fosters a structured iterative process that is both well suited to addressing complex problems and meeting the structural needs of formal project processes". 39 Drawing from the lessons of tech start-ups, others recommend tools such as Strategy Testing 40 (Ladner 2015) as a means of evaluating process and method with a view to improved problem-solving and iteration. Wild et al 2017 give an example of an adaptive logframe, drawn from DfID experiences, that sets out a set of clear objectives at the outcome level, and focuses monitoring of outputs on the quality of the agreed-upon rapid-cycle learning process.
Document the process. Whichever method of measuring problem solving is chosen, we recommend that documentation of that process be undertaken. In the Nigeria case, the absence of documented problem construction and deconstruction -or of problems solved, or solutions tried and abandoned -has left implementers with very little ability to re-evaluate their assumptions or adjust their implementation in a methodical manner that foregrounds learning. In Nigeria, the absence of such information also left the World Bank with very little confidence as to what was actually taking place on the ground. We do not recommend that this documenting take the form of long donor reports, for which implementers have to spend inordinate hours preparing. Instead it should be action-oriented, capturing ongoing discussions and reflections. It can be the regularized updating of a project's fishbone diagrams (showing problem targeted, and causes identified), as well as the answers to key questions asked at regular R&R check-ins: What was done? What was learned? What is next? What are your concerns? In this way it covers the basic facts of action as well as initiating a process of reflection. Note that expert facilitation will be necessary to help ensure that participants do ask deep questions about their lessons. 41 Bank teams may wish to consider arms-length monitoring. The average Bank TTL is not given the time or scope to invest considerable energy in R&R with various project partners. Neither do they necessarily have the skills and ability to facilitate honest learning and reflection sessions with their partners. Until that changes (and it will certainly not change overnight), the type of relational monitoring we are talking about may be best handled by a firm that has the skillsets and experience necessary to facilitate thoughtful R&Rs, to document lessons and feed them back in synthesized form to donors. If the adaptive endeavor is small scale, then the outsourcing may be to a single consultant, practiced in the skills required; if the adaptive endeavor spans a country portfolio, then hiring a firm for the monitoring and quality assurance task would seem appropriate. Either way, monitoring needs to be budgeted as a core element, and not as an afterthought.

Selecting and managing the firm
Select service providers with the right combination of skillsets for the work and the context. We think the World Bank team did well to select a firm to carry out the TA that had experience navigating Nigeria's political realities. Clearly politics and personalities are key drivers of challenges in the Nigerian health sector, and an ability to think and work politically will therefore be critical to effective implementation. On the other hand, the firm displayed some serious weaknesses in its ability to provide a range of technical solutions that could be relevant to the identified context. They also appear to have had few tools at their disposal to assist with problem construction, deconstruction and adaptation and seemed to struggle with documentation and reporting in a manner that was light touch and responsive. A balance needs to be struck between technical expertise, political networks and skills and experience in rolling out and reporting on adaptive approaches. Where the skills do not already exist, they need to be identified and addressed (whether by the Bank, the firm or a intermediary manager). Teskey and Tyrel (2017) include some helpful suggestions of the sort of organizational capabilities that might be assessed for and that can help the donor gain confidence of the contractor's ability to facilitate a truly adaptive, iterative methodology: -Technical and analytical tools for problem construction and deconstruction -Systems for developing multiple theories of change and action -Systems of embedded learning and monitoring that ensure they are capable of testing their theories of change -A management structure that delegates high levels of discretion, allowing those on the ground to make micro-adjustments -High numbers of staff with social connections and insider knowledge -A budget management system which allows flexibility and transfers between streams based on learning Consider longer-term 'partnerships' with service providers. Teskey and Tyrel (2017) insist that the levels of trust required to allow learning from failure as well as the space necessary for adaptive work requires the relationship between donor and contractor to be one of 'partnership' rather than principal-agent. We would tend to agree with this, while adding that such relationships will necessarily take time and active cultivation. It is our view that the skillsets required for effective implementation of adaptive programming (e.g., an ability to facilitate joint problem construction, to support consensus-building among diverse stakeholders, to provide a range of solutions with understanding of preconditions that drive their success, to provoke learning and analysis, etc.) seldom exist readymade in most of the contexts that we wish to use them. ADG had elements of these skillsets but also had various deficiencies (not surprisingly, given that both PfR and 'adaptive implementation' had not previously been attempted at this scale in Nigeria). But if donor organizations seek to work only with those who excel in all these aspects, they will likely have no partners to work with. We therefore recommend seeing the development of adaptive service providers as a medium-term but eminently worthy investment objective. In future relationships of this type, the Bank may wish to build in an initial set of monitoring and/or performance milestones that are associated with the service provider's program and operational systems for designing and delivering adaptive TA (see bulleted list above).

Put in place obvious incentives for being frank about and learning from failure.
If there exists a commitment to adaptation, testing and experimentation, then there must of necessity exist an expectation and appreciation of the fact that failures will occur. Failures are critical to experimentation and learning and are a necessary step on the path to crafting fit-for-purpose solutions. Relationships between donors and contractors seldom foster honest discussion about failure, however, since donors are typically understood to reward only success stories and punish failures. Although ADG was told to foster experimental solutions, their rewards -e.g., potential continuation of their contract by the Nigerian government and their performance payments -were contingent not on learning from failure but on aggregate indicators being moved. It would have been difficult for them to honestly share the aspects of their approach that were not working with their World Bank counterparts, even though this was rhetorically encouraged. This links back to our discussion above, about how to best evaluate success.

Performance-based payments for service providers of adaptive assistance may be appropriate in certain contexts but chosen measures of success must be tied to the problems being targeted.
The utilization of a performance payment for ADG in this case was problematic, since the measure of performance (improvements in state level indicators of health coverage) was too far removed from the actual performance problems that ADG's assistance was directly targeting. That disconnect resulted in a clash between attempts to attain performance payments and attempts to promote a locally-driven program. But it is conceivable that a performance payment could have been appropriate if what had actually been measured was discrete problems fixed or some other evidence of active, ongoing learning and an increase in problem-solving abilities (refer to previous section on monitoring). Teams need to be thoughtfully experimental in this regard. Performance-based payments will require savvy design if they are to avoid creating perverse incentives and ensure that focus remains on actually fixing problems rather than gaming the system.

Maintaining internal authorization
Do not underestimate the potential costs of piloting adaptive approaches. The perceived 'failure' (at least by some) of the adaptive component within the SOML PfR has likely had costs for future adaptive approaches in the Nigerian context. The costs are both relational (e.g., bridges burned with a bold service provider who has found the costs of adaptive programming not worth the benefits) and reputational (e.g., federal actors have judged the approach a 'failure' and appear less inclined to consider adaptive approaches going forward; some Bank actors appear to reflect similar sentiments). 42 Regardless of whether we believe these costs could have been better mitigated (or that the perceptions of failure are misguided), it is important that actors who wish to implement adaptive approaches do recognize that they operate in a contested space and that the attempt to shift the status quo may on occasion engender some resistance (even hostility). Implementation therefore needs to be as carefully considered, thoughtfully implemented and methodically evaluated as possible. Effective messaging and communication is key, particularly in highlighting stories of success when they occur and acclimatizing colleagues to the inherent messiness of a more adaptive approach. Proponents may not have many chances to 'get it right' and can risk burning significant political capital with rushed or poorly resourced and ill-communicated efforts. More broadly, it is also important to be able to discern the difference between 'failure' that stems from honest efforts to explore possible solutions to vexing problems (i.e., short-term failure which is the 'price one pays' to find the longerterm solution) versus failure that is a result of professional incompetence versus apparent failure that comes from trying to resolve a problem that, it turns out, is just too large or too entrenched. Few in any branch of development implementation and evaluation have adequately engaged with these key issues (Rao et al 2017 is a notable exception).
Prioritize and sustain active authorization and support'; passive is insufficient. The prevalence of powerful, systemic incentives for best practice, solution-driven programming in large donor organizations (Bridges and Woolcock 2017;Bain 2016) is such that even if an adaptive approach is initially agreed by World Bank managers and co-TTLs, these individuals will inevitably face pressures to revert to a traditional approach as the program continues. The ability of the program to resist such pressure will be primarily down to the presence of powerful, active authorizers capable of making the case for continued implementation. In such cases, passive buy-in by critical implementing and managing staff of adaptive programs does not appear to be sufficient to withstand existing institutional recalcitrance. Adaptive teams need to not only identify active authorizers but must work hard to maintain and build that authorization throughout implementation, seeking alternative support when staff turnover leaves them bereft of a critical backer.
Recognize the role of 'intrapreneurs'; cultivate space for them and ensure long-term staffing arrangements. In their review of DfID's adaptive attempts, Wild et al (2017) note that adaptive approaches tend to "cluster" around specific contexts and individuals. Denney and McLaren (2016, 31) refer to the need for internal 'champions' within the donor organization who can advocate for the adaptive approach, as well as close working relationships between the donor and implementing team; flexible and potentially longer-term funding and staffing arrangements; and a recognition that trajectories of change and program results cannot always be predicted. Faustino and Booth (2015, 27) reflect on the critical role of "development intrapreneurs" who "maneuvered through the bureaucratic procedures and approvals from senior management, finance and contracts officers and desk officers… to create program modalities that supported iterative, politically informed programming." Other commentators have highlighted the importance of long-term staffing arrangements for adaptive programs (Teskey and Tyrell 2017;ODI 2016), observing the extent to which such programming benefits from the kind of relational management and embedded knowledge that only comes with longer term placement. This review likewise finds that the role of individual "intrapreneurs" or "mavericks" (Bain 2016) is critical in making adaptive approaches possible in big institutions, and in ensuring they are well-implemented. Relatedly, the long-term staffing of specific individuals over the course of such a program's implementation cycle is recommended for program momentum and broader support. The novelty and counter-organizational-culture nature of adaptive programming is such that for the near future, specific individuals are going to be particularly critical to the success of programs. In practical terms, this means that organizations wishing to undertake such an approach need to think carefully about the human resources they have available to devote to the endeavor, and even more so if they are not adopting an arms-length arrangement (see earlier recommendation) to manage the TA. They would do well to prioritize the placement of national, incountry individuals on the adaptive team, since these will typically reside in-country for far longer than their international counterparts. And they need to ensure that these teams have sufficient reputational clout (or at least some powerful supporters/allies to call upon) within their organization so that they can defend their approach in the face of potential internal resistance.
More generally, if organizations are committed to adaptive methods, they need to carefully think about the longer term 'unlearning' and upskilling that it may require of their organizational staff.
Much of the discussion here revolves around the outsourcing of adaptive facilitation to partners and firms, who then provide support to World Bank clients in a manner that seeks to be problem-driven, locally led, adaptive and iterative. But given that a great number of Bank staff, particularly those based in country, have direct responsibility for supporting government(s) in the resolution of complex problems, the elephant in the room, so to speak, is "why are we so focused on outsourcing adaptive facilitation rather than building such capabilities within our existing teams, among Bank staff who are meant to be supporting the strengthening of Government problem-solving capabilities on an almost daily basis?" This would be no small feat of course, particularly for a cadre of professionals whose identity is that of a detached 'expert', and who have been trained in the delivery of solutions with confidence (and arguably hubris). A move from a solution-providing to problem-driven orientation requires that 'experts' learn to be 'facilitators', adept in prompting the deconstruction of existing challenges in a manner that harnesses local knowledge and fosters creativity. It requires the abandonment of "we know what works" and an embrace of experimentation, 'muddling through' and 'good enough' solutions. The unlearning, upskilling and consensus-building required may not even be attainable by everyone; various commentators have reflected on the difficulty of teaching development professionals to "think and work politically", suggesting that a good deal of the skill required may be innate (Marquette 2014, Rocha Menocal 2014, Carothers and Gramont 2013. Rocha Menocal likens it to learning a new language, in the sense that "natural ability or predisposition matter, but so does the learning process itself". 43 We do not have conclusive advice on this broader challenge, but we cannot help but be excited by the possibility of a country office that sees adaptive programming not simply as a case of subcontracting other entities to run facilitative TA, but also as an upskilling of whole Bank teams to be more humble, more problem-driven and experimental about the manner in which they support government problem-solving.
6 Conclusion: Pragmatism, hope, reasoned persistence This work takes time, it takes perseverance and it requires trust, and the task of attacking some of the most challenging areas in government is frustrating but absolutely worth it with each breakthrough. -Poobalan 2017 In conclusion then, our message is both pragmatic and hopeful. Though we might like to imagine adaptive approaches being implemented by perfectly equipped teams, funded by unequivocally committed donors and implemented in wholly conducive contexts, the reality, at least in the near future (but most likely any future), is going to be far more hybrid, messy and imperfect. What makes this review hopeful is that despite all that imperfection -be it funding delays, mixed motivations, inflexible instruments, political interference, passive authorization or destabilizing staff lossespositive impact still appears to be possible. Clearly there is much that remains to be learned about these approaches and a great deal more effort must be put in to demonstrating their utility, but it certainly seems realistic that with each attempt at adaptive programming, Bank staff can improve on their ability to manage such processes for increasingly greater impact.

Snapshot of Abia state: The primacy of politics and its impact on performance
Of the three TA-receiving states we visited, Abia experienced the most detrimental political interference. Although Anambra and Kogi both experienced interference, they appear to have created some space for successful operation within that. For Abia, space has been highly constrained. Their case provides an excellent example of how political interference can derail program performance.
Although the initial $1.5 million for SOML implementation was released by the Bank in June 2016, it was only accessible to the PMU from June 2017. The PMU team and counterparts undertook implementation between June and August 2017 on the basis of workplans devised with ADG. Then, in September, it became apparent that the then Permanent Secretary of Health had unilaterally changed three signatories to Remita (Remita is the account approval software). There are six signatories in total, though only three required for approval), apparently replacing them with persons thought to be more amenable to "doing his bidding" (interview, anonymous). In order to effect this change, the PS in question is said to have forged the Governor's signature. Finding themselves no longer able to access the account, members of the PMU raised the issue with staff of the Auditor General, who in turn raised it with the Governor. This initiated a feud with the Governor and the PS.
The issue is complicated by the fact that the sitting Governor, who would ordinarily be far more powerful than the PS, has been recently contesting his election in court (which resulted in him being removed for a short period and then brought back) and is thus preoccupied with an existing power struggle. Moreover, the PS in question is said to have considerable political clout and commands a large number of potential votes. In addition, the power of the health commissioner in Abia has been reduced by the fact that, owing to circumstances we do not understand, the Governor suspended all commissioners for 5 months. The result has been that the PS is far more powerful in relation to both the Governor and Commissioner than would normally be the case.
ADG made use of its political capital and reputational clout to engage directly with these various players in an effort to prevent the program going off track and to return power to the PMU. They describe the PS as "frankly criminal" and characterized the change of signatories as him "wanting his cronies to sign the cheque". Meetings with him were unable to effect change. ADG was surprised that although the Governor was made aware of the PS forging the Governor's signature, he did not fire him. After engaging directly with the Governor, ADG learned that this was due to the fact of the Governor considering it too politically sensitive to remove him by firing. Instead, the Governor told ADG they would wait and that he would gradually remove him from the program. In ADG's words "He asked us to lie down and be quiet. We were to trust him that eventually the changes would be made".
True to the Governor's promise, the PS was eventually transferred, though this was done during a mass transfer of PSs in general, presumably to reduce the chance of political fallout. A number of Directors that had been appointed by the unruly PS remain in position. The PS transfer happened in November 2017. Unfortunately, as one of his parting moves, and apparently as revenge against the PMU members who had alerted AGF to the signatories issue, the PS wrote a letter requiring that the then Program Manager be removed from the PMU. Staff are currently waiting on the AGF to write to the new PS, requiring him to update the account verifier and signatories. The removed PM is hopeful that such an update might lead to his position being returned, though our own assessment is that this is unlikely.

What has been the effect of all this on implementation?
In Abia we met with both old and new PMU members (it seems that in addition to removing the PM formally, a number of other PMU members have been informally side-lined). It was very apparent that implementation was only recently begun by the new team and in contrast to Kogi and Anambra, they have little sense of any impact yet. When asked about whether they'd improved on any indicators, they indicated that it is too early to tell. Moreover, we saw less sign of the team making decisions based on evidence or a detailed understanding of the problem. Members of the old PMU claim that tasks have deviated significantly from the previous workplan (for example the QISS component was removed on the basis that it was too expensive, though it has since been reinstated) and that the emphasis has been on large procurements that they believe will have little bearing on the indicators. We were told that four vehicles have been procured at inflated costs for example, two of which are being used by individuals and not for the program at all. We also noted that some procurements the team discussed with us are actually related to the advancement of general health objectives (for example procuring generators) and do not seem to have been made with a clear attention to their impact on the six indicators.
We found that the current PMU does have technically capable people in it, some of whom benefitted from the ADG TA. We also found that their discussion of future interventions showed some appetite for innovation and some attention to outcomes, though this was not so pronounced as their Kogi and Anambra counterparts. Current interventions, such as the installment of a data monitoring chart in all the health facilities, show some promise and an appetite for innovation, but in general, we noted that most of their interventions are still in planning phase, with very little actually delivered. Given that the budget is supposedly over 80% spent, this suggests that a large part of expenditure may have been diverted to inputs that were not directly relevant to SOML.
Unlike the other states, Abia was unable to present us with any current DHIS data on their performance against the six indicators. We also noted that the new PS is fairly unaware of SOML and, unlike his Abia and Anambra counterparts, did not seem to know the various indicators.

Existing levels of state funding
The 36 Nigerian states have very different levels of existing revenue available to them and therefore different levels of additional resource to draw on in order to supplement SOML funds. It is apparent that whilst the delay in SOML funds arriving in some states meant those states were unable to begin any substantial interventions, there are other, more financially viable states who were able to continue complementary activities even in the absence of fresh resources. A state like Rivers which has a higher Internally Generated Revenue (IGR) and receives a greater federal allocation than any other state, has drawn on a large bank of resources to supplement SOML efforts, building health infrastructure, improving health worker remuneration and heavily investing in equipment at health facilities. The simple fact of state indebtedness is likely to have a major impact on health interventions since it directly impacts health worker salaries and therefore motivation.

Presence and capability of the private sector
As indicated, the ADG states were selected from a group of 16 states that began with a higher baseline on the six coverage indicators; apparently this was considered a rough proxy for existing levels of "capability". These states were judged to be higher in existing capability and lower in existing donor activity. But this is a crude measure of capability. It has since been pointed out that higher-than-average baselines probably reflected the capability of the private more than the public sector in many of these states (interview, WB). In practice, there is likely to be wide divergence between capability of the public sector across the states and there is little hard evidence to support the assertion that the pool from which ADG states were selected have a higher capability to start with. And assuming that the private sector is a major driver of improved coverage, then the extent to which the private sector is supported, enabled and capable (a fact which is largely beyond the scope of SOML) becomes a major factor in the ability of states to improve the six indicators.

Levels and quality of existing donor support
In addition to having varied levels of financial capacity, the 36 states have highly varied levels of implementing partner (IP) support. The six states covered as part of this review are reportedly considered to all be "under-resourced in TA" (Interview, WB), 44 but in practice, there are still significant divergences between them. The details of which IPs are operating in the state, on which thematic areas and with how much support clearly varies significantly. For example, we noted that the indicator in which Enugu currently has the best rates of reporting and performance (according to the very imperfect data available) is in Penta 3 coverage, despite the fact that none of their SOML funds have been utilized towards Penta 3 thus far. Instead, that aspect continues to be almost completely financed and driven by an IP. Other states that do not have the presence of such an IP are clearly at a major disadvantage. Bank staff likewise confirmed that one state advanced in its PMTCT indicator because "[an INGO] ran the whole program". Referring to TA and non-TA states is thus something of a misnomer. A state may have had no ADG presence but still have been able to draw on considerable donor resources and TA.

Status of PHCUOR implementation
'Primary Health Care Under One Roof' refers to a policy of the FGON to shift all PHC-related financial resources, employees, programs and facilities from Local Government to the SPHCDA. The SPHCDA/B in each state is required to integrate all Primary Health Care (PHC) activities hitherto run by the Local Governments, on the principle of "three ones" -one plan, one management and one monitoring and evaluation system -as a means of improving primary health care implementation. 45 This shift has happened to varying degrees in the different states and in some states, has been a major source of blockage, internal conflict and funding delays 46 (interview, WB). State officials insist that a major reason for resistance and delay with the PHCUOR has been that it entails local government players losing out on the personal gains they were previously able to make when PHC funds flowed through their structures.
The transfer of PHC staff from Local Government to the SPHCDA has particularly important implications for the ability of the state to manage staff performance and remuneration. This transfer has been completed in only one of the states that we visited, and it was immediately noticeable that this state has been able to harmonize salaries and clear arrears to the great advantage of the health sector generally.

Political/ security context
The political economies of the states vary enormously and are a major factor in successful implementation. As our comparison of the three TA states -Abia, Anambra and Kogidemonstrates, politics appears to be a major contributor to performance of the SOML program in each state. In Abia, despite similar levels of assistance, orientation and resources, the program has been severely hampered by the opportunistic interference of powerful individuals (see Bridges and Woolcock 2018, Annex D). By contrast, a non-TA state like Rivers has a highly supportive political context for SOML, with a Governor who is committed to funding major improvements in the health sector from the state budget. Stakeholders confirm that north-western states, which are said to exhibit far less negative Levels of insecurity and conflict also vary greatly between the states and are likely to have major impact on SOML performance. Our six selected states are generally less conflictridden than some of their Northern counterparts but still have considerable divergences between them. It should be noted that the effect of conflict or insecurity on the indicators can go both ways. On one hand we find that health facilities in Rivers have historically been affected by kidnappings of health workers for example, which will clearly have a negative effect on performance; in other states, the presence of Internally displaced peoples camps for example is said to have enabled easier targeting of interventions.

Survey timing
The timing of surveys should theoretically benefit those whose interventions coincided fortuitously and disadvantage those whose did not. Bank staff indicated that state officials should have been aware of the dates when surveys were to take place in their states. States were apparently notified of a three-month window during which surveys would take place. Some staff pointed out that savvy teams would seek to "game the system" by timing their MNCH weeks for example, so that they took place shortly before the survey was undertaken, for maximum effect. 47 Our interviews with states however indicated that they had little to no awareness of when the surveys were taking place and that they certainly did not time their activities to take fullest advantage of this. It is not clear whether this is because they were not given the information about dates or whether they were given it too late to shift plans. Either way, it is apparent that indicator performance varies significantly in relation to the timing of certain interventions (see graphs in Anambra snapshot for example; Bridges and Woolcock 2018, Annex D), and if any states were lucky or savvy enough to time their interventions with the survey, this may have had significant effect.

The state of the states
As part of this review, visits were made to six states, covering three regional zones, three of which received the TA from ADG and three of which did not. In each of the states we made a rough assessment of the political context, progress in implementation and fund utilization, problem identification, range of solutions arrived at and success in improving outcomes. Our findings are briefly summarized in Table 3. Kogi has had a somewhat supportive political environment. Governor was up for re-election and has been eager to demonstrate gains in health service delivery since this was a key campaign promise. This has not translated into regularized payment of health worker salaries however and salary arrears were running at 5 months during our visit. Commissioner highly supportive and well-versed in SOML objectives. There were some issues that the team describe as "internal competitive rivalry" rather than political interference, with certain figures wanting to chair committees or be part of the PMU -ADG helped manage these. Proposed interventions, such as the installment of a data monitoring chart in all the health facilities, show some promise and an appetite for innovation though whether such solutions will achieve implementation remains to be seen.

Sustained implementation since
In the absence of implementation, no evidence of specific problems being fixed.

Enugu
Very delayed implementation and low expenditure. Momentum appears low in the team. Little to no evidence of thorough problem analysis or of ability to link inputs compellingly with outcomes. Somewhat unimaginative solutions, few of which have been implemented. Major issues of data completeness.
Overly reliant on implementing partners.

Elements
of interference. Stakeholders in Enugu tend to insist that the political context is highly supportive. But deeper questioning reveals evidence of interference that has affected performance. For example, the SOML account for Enugu was suspended for some months due to an irregular transfer of SOML funds (15 million naira) into the state account that was initiated by the Governor. It is also apparent that additional layers of approval have been added to the standard approval process in Enugu such that the commissioner and Governor have more input in the workplans than ordinarily required, which leads to additional delays in the process. It is unclear what the drivers of these interferences are, but it is clear that they have slowed down implementation of the program substantially.

Akwa Ibom
Powerful commissioner creating a very supportive context for improvements in the sector. Implementation well underway but expenditure seems mainly to be contributing to sector-wide objectives rather than specific SOML objectives. Issues of weak problem diagnosis, data analysis and an overreliance on implementing partners. Low levels of understanding regarding the PfR instrument and its implications for procurement, contracting etc.
Somewhat supportive political context but bureaucratic obstacles. The Governor in Enugu is reportedly committed to improving health service delivery. He has appointed a Commissioner for health who was former Chief Medical Officer of a major private company -Exxon Mobil -and has been given space and resources to rejuvenate the health sector. The Commissioner appears supportive and committed to SOML objectives. The fact that PHCs still fall under the remit of Local Government (PHCUOR having not been completed), but the commissioner has proven powerful/ respected enough to enforce a level of discipline (he regularly visits facilities unannounced, has fired doctors, initiated prosecutions for fraud and reward programs, etc.) and has succeeded in completely clearing a backlog of salary arrears. So, despite PHCUOR not yet being fully implemented, motivation and performance of health workers are said to have improved significantly.
Above average expenditure with an emphasis on equipment and infrastructure. 60% of SOML funds utilized thus far. A significant amount of expenditure is being allocated to inputs that are not directly related to SOML goals e.g. Essential drugs, generators, refrigerators, HIV test kits, Infrastructure. A number of standard SOML interventions have not been able to run, apparently due to "lack of funds". MNCH weeks for example did not run in 2016 and have not yet run in 2018. Highly reliant on implementing partners e.g. UNICEF for supplying vitamin A. We noted pronounced confusion about the nature of the PfR, with members of the PMU very unclear about procurement requirements.
Evidence of some innovative solutions, but many solutions that do not seem clearly linked to the problem or goal. One innovative solution is the piloting of "husband schools": in an effort to target men, who have significant decision-making power in the household, short term 'schools' are set up in a community that men are invited to attend, for instruction on immunizations, contraceptives, skilled birth attendance etc. But it seems that high levels of expenditure on infrastructural needs and large equipment purchases (neither of which link specifically to SOML goals) have s q u e e z e d o u t o t h e r interventions.
The team consistently reflected on a lack of funds and a reliance on implementing partners. And other solutions, such as training of TBAs and procurement of nets do not appear to be being done with deep thought regarding the foundational problem and its drivers.
The one example that the PMU could give of an intervention having impact was the 'husband schools'. Apparently DHIS data demonstrate improvements in CPR and in SBA in the areas where the schools have been piloted. As with other states, they appear generally unaware of the impact of other inputs. Data are incomplete and not considered reliable.
Akwa Ibom had a sum total of 24% reduction from its baseline according to the 2016 MICS survey. We are not privy to the breakdown.

Rivers
Implementation has been slow to start but now moving. Standard interventions for the most part, with some isolated examples of innovation and little evidence of impact as of yet. Strong indications that any improvements on SOML indicators thus far have been down to a particularly conducive political environment.
Highly supportive political context. Health worker unions are powerful in Rivers and prior to the initiation of SOML there was a health worker strike that lasted one year. The current Governor -nicknamed "Mr Projects" -came to power on the back of a campaign to deliver major improvements in the health sector and address health worker remuneration.
Considerable amounts have since been spent from the state budget raising and harmonizing salaries, building and upgrading health facilities (e.g. 14 secondary health facilities completely renovated), and harmonizing administrative structures. The Commissioner for Health refers to the Governor as "The friend of the health sector" and narrates how health receives the second highest expenditure in the state (after works). Rivers is the only state in our sample where PHCUOR has been implemented and the responsibility for PHCs now lies with the State Primary health care facility rather than Local Governments. Salaries are reportedly all paid on time.
Implementation slow to start but now moving fairly smoothly. Despite money having arrived earlier in Rivers than the other states (around January 2017), there was a major delay in accessing funds due to incorrect designation of the account as a naira rather than dollar account. Implementation eventually began in June 2017. By August 2017 an MNCH week had been run, with a positive impact on the data. Another round has been run in January 2018. Evidence of trainings, procurement of drugs, nets and equipment ongoing. Some standard interventions -like QISS -still have not been initiated. TCG not meeting regularly due to limited funds. And orientations on SOML still in process despite 2 years into program. Complaints that procurement and approval processes lead to long delays in interventions. Little evidence of adapting workplans.
Most solutions follow typical model but some evidence of innovation. The majority of solutions seem focused on retrainings, procurement of drugs and equipment, immunization campaigns etc. But also some interesting attempts to shift behavior in rural communities e.g. one intervention involves the training of community members who have used SBAs or CPR as "interpersonal change agents" on the basis that individuals are more likely to listen to their peers than an outsider. Also experiments with results-based payments e.g. the retired health workers receive incentives if they succeed in bringing 20 women for ANC.
The one area where the team is confident that they have had an impact so far is with MNCH weeks. With the other interventions there does not appear to be any certainty/ evidence as yet of a positive impact.
Rivers had a 21.5% improvement on its baseline according to the 2016 MICS survey. We are not privy to the breakdown. Given that no implementation had begun by the time these results came in, they seem to be testament to the impact of the broader context, specifically the Governor's efforts to address health worker performance and improve health infrastructure.