http://orcid.org/0000-0002-2279-2921
Figures
Abstract
Good governance (GG) is an important concept that has evolved as a set of normative principles for low- and middle-income countries (LMICs) to strengthen the functional capacity of their public bodies, and as a conditional prerequisite to receive donor funding. Although much is written on good governance, very little is known on how to implement it. This paper documents the process of developing a strategy to implement a GG model for Health Technology Management (HTM) in the public health sector, based on lessons learned from twenty years of experience in policy development and implementation in Benin. The model comprises six phases: (i) preparatory analysis, assessing the effects of previous policies and characterizing the HTM system; (ii) stakeholder identification and problem analysis, making explicit the perceptions of problems by a diverse range of actors, and assessing their ability to solve these problems; (iii) shared analysis and visioning, delineating the root causes of problems and hypothesizing solutions; (iv) development of policy instruments for pilot testing, based on quick-win solutions to understand the system’s responses to change; (v) policy development and validation, translating the consensus solutions identified by stakeholders into a policy; and (vi) policy implementation and evaluation, implementing the policy through a cycle of planning, action, observation and reflection. The policy development process can be characterized as bottom-up, with a central focus on the participation of diverse stakeholders groups. Interactive and analytical tools of action research were used to integrate knowledge amongst actor groups, identify consensus solutions and develop the policy in a way that satisfies criteria of GG. This model could be useful for other LMICs where resources are constrained and the majority of healthcare technologies are imported.
Citation: Houngbo PT, Coleman HLS, Zweekhorst M, De Cock Buning T, Medenou D, Bunders JFG (2017) A Model for Good Governance of Healthcare Technology Management in the Public Sector: Learning from Evidence-Informed Policy Development and Implementation in Benin. PLoS ONE 12(1): e0168842. https://doi.org/10.1371/journal.pone.0168842
Editor: Kim-Kwang Raymond Choo, University of Texas at San Antonio, UNITED STATES
Received: July 27, 2016; Accepted: November 27, 2016; Published: January 5, 2017
Copyright: © 2017 Houngbo et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are contained within the document and, where cited, in the following articles: Houngbo PT, Kishna M, Zweekhorst M, Medenou D, Bunders J. The Effect of Benin’s First Public Procurement Code and its Amendment on Healthcare Equipment Acquisition Prices. J Public Procure. Forthcoming 2017. Houngbo PT, De Cock Buning T, Bunders J, Coleman HLS, Medenou D, Dakpanon L, et al. Ineffective Healthcare Technology Management In Benin’s Public Health Sector: The Perceptions Of Key Actors And Their Ability To Address The Main Problems. Int J Heal Policy Manag. Forthcoming 2017. Houngbo PT, Zweekhorst MBM, Bunders JFG, Coleman HLS, Medenou D, Dakpanon L, et al. The Root Causes of Ineffective and Ineffecient Healthcare Technology Management in Benin’s Public Health Sector. Heal Policy Technol. Forthcoming 2017.
Funding: This work was supported by the Netherlands Organization for International Cooperation in Higher Education (NUFFIC) [grant numbers NFP-PHD.07/218, CF3914/2007]. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
1. Introduction
Problems with governance in low- and middle-income countries (LMICs) can lead to ineffective and conditional donor aid, the waste of scarce resources and societal disillusionment with public institutions. Moreover, it leads to a lack of available resources for, and accessibility to, health care, which affects the poor and marginalized most [1]. GG in the health sector, especially in healthcare technology management (HTM), looks like “moving a mountain” for many LMICs where a great number of contextual challenges exist. Since about 95% of the healthcare technology used in these countries is imported [2], numerable layers of bureaucracy are present, each being prone to corruption and administrative malfeasance. Furthermore, imported technology is often not tailored to the specific medical and technological demands of the receiving nation, leading to further problems of outreach and public aid [3–5]. Ineffective HTM has been reported in many poor countries in sub-Saharan Africa [4, 6–8]. The aim of this study is to analyze these problems and intervene with environment-specific solutions, through the concept of “good governance” (GG). This research attempts to explore how weak governance in the public health sector can be transformed into strong governance, and what that process looks like.
The GG concept has evolved, since it was first discussed at the World Bank’s Annual Conference on Development Economics in 1991 [9], into a set of principles and conditional prerequisites for LMICs to strengthen the functional capacity of their public bodies, and receive donor funding [10–12]. Although ample attention has been paid to the concepts constituting GG [13–16] as well as general GG recommendations, suggestions and advocacies, little has been published on the benefits and pitfalls of implementing GG practices. Few evaluation studies exist on practices to transform a poorly performing governance system to one that exemplifies the tenets of GG [12, 17–19].
In this article we present a good practice—the HTM policy process in Benin. We describe in detail the step-wise rationale to bridge the implementation gap between general GG principles and contextualized practices, over the past two decades. We will discuss the strength of our approach in bounded rationality [20], as a generalizable model on “how to implement GG principles” in LMICs. Previous research of Houngbo et al. [21] investigated problems in HTM in Benin showing that the sub-sector lacks functional capacity and appears vulnerable to corruption. Additionally, ineffective public procurement (PP) of healthcare technologies has been criticized by the Ministry of Health (MoH) as incapable of converting public money into available and accessible healthcare technologies for improved health outcomes [22–27]. Since 1995, many seminars, workshops and surveys have been organized with diverse stakeholder groups, but they have yet to be translated into a successful policy, such as the first national maintenance policy for healthcare infrastructure, equipment and vehicles of 2002 that was halted due to a lack of political will and budgeted action plan. Piecemeal attempts by the MoH, and isolated, short-term actions of donors have tried to improve HTM capacities, but have not been sufficiently coordinated to deal with problems. HTM in Benin has typically favored top-down policy development and implementation whilst neglecting local experiential knowledge, thus overlooking characteristics of GG [21].
Since 2006, participative action research has been conducted to create environment-specific solutions to the problems in HTM, with a policy development process that satisfies criteria of GG. Principles of participatory action research and GG have been combined to gather insights into the challenges that health facilities in Benin face. Together, these have guided research actions and been enriched by contextual knowledge from the user-level.
The aims of our study were to develop detailed and validated insights into HTM problems at the local, meso- and macro-level, to help design—alongside policy makers, hospital managers, maintenance technicians, donors and users of healthcare technologies—policy actions to progress towards GG, and to reduce the weaknesses in HTM over the period 2006–2015. The guiding research questions were: (i) what would a bottom-up and inclusive implementation strategy to realize GG in HTM look like? (ii) What are the key lessons learned from the application of the developed strategy in Benin?
In this article, we present the principles of the developed GG Implementation (GGI) model, followed by its phases, general aims, the research activities that can be utilized during each phase, and a description of how the policy development process proceeded in Benin.
2. Principles of the GGI model
Numerous studies have documented the ineffectiveness of policy reforms in Africa. Actors at lower levels in the health sector, for example health workers, are often those responsible for the implementation of policy. As such, these actors have the ability to limit the implementation of decisions made at higher levels [28, 29]; a study in South Africa showed corrupt actors actively undermined governmental attempts to reform PP [30]. Therefore, although international development organizations often provide functional or task-based descriptions of GG and health system stewardship principles, this does little to describe the actors involved in health systems, their roles and responsibilities and their willingness to fulfill those responsibilities [31]. Principal-agent theory posits that the motivations of agents and principals are often not aligned, and thus low-level actors have the discretion to neglect policy decisions made by those higher up. By strengthening the relationship between policy makers and the agents charged with implementation, the extent of goal divergence could be diminished. This notion has been increasingly discussed in the literature as the perceptions of low-level actors and lay people are being incorporated into policies and development projects more frequently [28, 32–34]. It also provides a mechanism through which many principles of GG can be effected. Accountability, transparency, citizen voice, participation and inclusivity are all objectives that, in part, can be improved by an increased diversity of actors taking part in policy making decisions. The success of this idea as a means to fulfilling GG criteria and for effective policy making has been documented in a large-scale review of countries whose health systems have greatly improved [35].
The Interactive Learning and Action (ILA) approach is a participatory action (or transdisciplinary) research approach. It stimulates knowledge co-creation between societal actors and experts, starting from the premise that relevant stakeholders have different institutional/societal backgrounds and hence, their perceptions on the problem at hand, as well as its solutions, are likely to differ. To realize effective knowledge co-creation, it is crucial to make different stakeholder’s perceptions explicit, and enhance mutual understanding and learning. This process takes place through interactive and analytical tools such as interviews, focus groups, dialogue meetings and argumentation trees. Key principles of the ILA approach include involvement of (end-)users, facilitation of knowledge integration, enhancing coalition building, identifying shared visions, enhancement of trust relations, and an emergent design [36]. These principles and their operationalization have been tested in Zimbabwe, South Africa and Bangladesh [37, 38], used in the context of patient participation for setting health research agendas [39], to develop a constructive technology assessment [40], and in interactive policy making for biotechnologies [41]. The adaptability of this approach stems from its deliberative nature and its space to let the design of research activities emerge from the findings.
Principles of GG and the ILA approach (Table 1) were combined and re-formulated as implementation strategies that are prescriptive in nature but can be enacted via the phased process of the ILA approach (see next section). The implementation strategies are described ex post, as many new considerations for policy development emerged as a result of the context in Benin, however they were implicit in the design of research activities. As principles of GG have already served to reduce strategic task uncertainty, the implementation strategies attempt to provide more detailed direction for good practices for GG, based on reducing the functional task uncertainty of health system stewards.
The five main strategies of the GG implementation (GGI) model are (Table 2): (i) develop evidence-informed and action-oriented knowledge for sustainable development, (ii) employ a strategy that is built on a shared future vision and anticipates the risks and benefits of possible interventions, (iii) engage and facilitate inclusion of the entire collective of relevant stakeholders, (iv) prevent corruption, and (v) work for independent justice. Alongside the strategy to prevent corruption, new sub-strategies are introduced that are relevant to all phases of policy development and the context; informed by experiences with attitudes in the civil service, government and public. These are:
- “Depoliticizing” public decision-making is based on the aim of appointing the rightly qualified and deserving public servant to the correct position, independently of his affiliation to a political party or social or ethnic group.
- “Patriotism” can be interpreted as civil servants exhibiting a high-level of concern for the well-being of the country, and respecting, defending and maintaining public goods and services. This philosophy was fundamental to successful HTM reform in Costa Rica [3] through the actions of a dedicated HTM workforce.
- “Taking swift, punitive action against corruption” stipulates public servants (policy makers, hospital managers, equipment users, technicians and engineers) being held responsible for their administrative misconduct independently of their social or political status. Corrupt attitudes have endured in Benin due to the existence of political impunity. Similar actions need to be taken against corrupt actors outside the public administration, including healthcare technology suppliers.
- “Civism” can be understood as “good citizenship and discipline”. It means the “priority given by a citizen to its nation or country’s interests before his own interests” [53].
- “Strengthen capacity in institutions” refers to the adequate training of human resources in HTM. By developing the skills of both high-level public officials and hospital-based health professionals, implementing change within the HTM system would be a much more responsive process.
3. Phases of the GGI model
The GGI model is roughly structured along six phases that employ different research and managerial activities in each (Fig 1). This is based on the ILA approach that is organized into five phases: (i) initiation and preparation, (ii) in-depth study of needs and visions, (iii) integration, (iv) priority setting and planning, and (v) project formulation and implementation [36]. In the GGI model, phase 2 of the ILA approach is divided in two, giving rise to six phases. Below, each phase of the GGI model is briefly described and illustrated by its application in HTM policy development in Benin.
3.1 Phase 1: Preparatory analysis
The aims of this phase are to learn from the effects of previous policies, assess the background context of HTM and assess the needs for a new policy. Relevant activities include desk research, key informant interviews and workshops to gain an up-to-date understanding of the situation. In Benin, this comprised desk research via three main activities: (i) a preliminary analysis of weaknesses in Benin’s HTM sub-sector [21], (ii) an analysis of Benin’s first HTM policy [24], and (iii) a quantitative analysis of Benin’s first PP codes’ effects on the acquisition prices of healthcare technologies [54]. Each activity is further elaborated, below.
The preliminary analysis comprised two surveys carried out in the six southern departments in Benin during 2006–2007 to:
- Identify the weaknesses in the HTM system. Data were collected through observational visits, interviews and questionnaires in 11 health centers/hospitals.
- Determine the extent of disparity between what technologies were documented in each health facility and what was actually available, to facilitate the procurement of essential medical devices for poorly equipped health facilities, and to identify weaknesses in Benin’s HTM cycle. Data were collected through observational visits, document analysis, interviews and questionnaires in 310 health centers/hospitals.
The study was part of a development project commissioned by Coopération Bénin-Union Européene, entitled Projet 8ème FED d’Appui au Secteur Santé (08 ACP-BEN 027). Semi-structured interviews were conducted with policy makers from the MoH and Ministry of Finance, hospital managers, users of medical equipment, maintenance technicians, some local suppliers, and some international organization representatives, i.e. bilateral or multilateral health development partners. Informed consent was sought in advance of interviews and included in questionnaire forms. The findings highlighted many weaknesses in the HTM system, and particularly drew attention to the high prices paid by government for healthcare technologies. Weaknesses included insufficient human resources to manage equipment and monitor supplier prices, unavailability of spare parts, lack of an annual maintenance budget, and unequal distribution of devices among health care facilities. This suggested PP processes had been hindered by a lack of policy and management tools, such as an up-to-date list of essential medical devices. The preliminary analysis also included a quantitative study of PP that had been specifically maligned by the MoH. The prices of ten medical devices were compared using PP acquisition prices against reference prices quoted to private health providers. The study found that the MoH paid over the odds for medical devices compared to other purchasing bodies, which informed the decision to conduct the in-depth quantitative analysis of the effects of Benin’s PP policies, seen in the third research activity.
The second research activity comprised an analytical review of the content of the “National policy of maintenance of infrastructures, equipment and vehicles” [24] and an assessment of the contextual factors that affected its development and implementation, through discussions with key policy makers in the MoH. These discussions highlighted that a lack of political will and a change in the leadership at the Department of Infrastructure, Equipment and Maintenance (DIEM) responsible for HTM had destabilized efforts for the policy’s implementation. The content analysis showed the policy failed to consider many key components specified in HTM frameworks, such as the Temple-Bird Healthcare Technology Package System (TBHTPS) [4], which includes strategic management and planning, allocation of financial, material and human resources, personnel and training, technology assessment, procurement, distribution, continued operation and use, and maintenance and repair (Fig 2). A failure to consult technical experts and lower-level managers during the policy development process was suggested to underpin the policies ignorance of key components of HTM.
The third research activity analyzed the effects of Benin’s first public procurement code (BPPC) on the public procurement acquisition price (PPAP) of healthcare technologies, via retrospective desk research. The data comprised the PPAPs of devices from 249 PP contracts and the private sector acquisition prices of the same devices, collected from 576 trading sheets from 1995 to 2010. Longitudinal data were organized into interrupted and controlled interrupted time series, taking the implementation of the BPPC and its amendment as intervention points. Segmented linear regression analyses showed that both the first and amended code thoroughly failed to deliver cheaper healthcare technology prices. For the simplest types of equipment (e.g. beakers, splits, X-ray films), a 93% price increase was observed, compared to if the procurement code had never been implemented. To understand this, we looked at what had occurred during the development process. The first BPPC was preceded by a World Bank (WB) assessment in 1999 that stated that the code was plagued by many weaknesses, including: (i) a weak execution body and a lack of institutional capacity at the national level; (ii) a lack of effective policy and management tools; and (iii) a lack of accountability and a highly politicized process of awarding contracts. Due to the delay that would be incurred by revising the code, the government implemented the first BPPC anyway and resolved to review it regularly and amend it as needed. It was also considered that prices may have been increasing due to technological innovations, or conversely, that weaknesses in the PP system made it susceptible to corruption. To disentangle the two, public and private sector acquisition prices were compared for the same devices and the interrupted time series data were controlled for seasonal fluctuations in healthcare technology prices. This showed that the public sector paid significantly more for healthcare technologies than their private counterparts, and that PPAPs were greater than could be accounted for by seasonal fluctuations, suggesting that suppliers were inflating their margins during international tenders. Despite the efforts made to improve the system via the BPPC’s amendment from 2002–2004, PP frameworks still failed to meet international procedural standards and the required guarantees of transparency and integrity [55].
3.2 Phase 2: Stakeholder identification and problem analysis
The objective of the second phase is to identify the stakeholders and actor groups to be invited for the policy development process, gain an in-depth understanding of their perceptions of problems in HTM and how their power positions relate to solving these problems. Various data collection methods, both qualitative and quantitative, can be utilized during this phase. Qualitative methods, such as interviews and focus groups (FGs), can help to gain an understanding of the perceptions of state and non-state actors, whereas quantitative methods can be used to provide an empirical analysis of the problem. Research activities in Benin centered on two surveys: (i) questionnaires and interviews with stakeholders to produce an exhaustive list of HTM problems, and (ii) site visits to observe first-hand the causes of ineffective and inefficient use, management and maintenance of healthcare technologies. The results of these research activities have been explored in greater detail elsewhere [56].
As a result of the weaknesses identified in Benin during Phase 1, interviewing all stakeholders relevant to HTM to elucidate an exhaustive list of its problems was a priority. Understanding that expertise and knowledge of different components of HTM are held by different actors across the system, every actor group involved in a capacity with health technologies (apart from patients) was recruited for the first survey: (i) policy makers, planners, and administrators at the MoH; (ii) hospital managers and directors; (iii) end users of medical equipment; (iv) biomedical, clinical, or healthcare technology engineers and technicians; (v) local and foreign suppliers, and; (vi) international organization representatives, i.e. bilateral or multilateral health development partners. Equally, as the weaknesses described in Phase 1 were found in many different areas of HTM, the ability of stakeholders to contribute to solving these problems may be different. Inclusion criteria pertained to the willingness of stakeholders to participate, as well as the aim of establishing a diversity of equally represented perceptions, based on our understanding of historically high- and low-power actors in HTM. Of 5302 key actors in HTM in Benin identified, a theoretical sample size of 244 was calculated based on a sample size calculation with 10% precision rate and 95% confidence level for a finite population [57]. In total, 500 questionnaires were distributed, 377 were returned (75.4%) and 372 were used in the analysis (74.4%). Additionally, 259 semi-structured interviews were conducted in 2011 with informed consent. Questions assessed the aforementioned components of the TBHTPS framework, tailored to the specific context of Benin. Participants were also asked to rank other actor groups, based on their degree of political or administrative power that pertained to solving HTM problems. During the second survey, a calculated sample of 117 healthcare facilities were visited (out of 787 in Benin) in 2012 regarding the placing of equipment and the causes of ineffective use, management and maintenance of equipment. During these visits, notes were taken based on the adapted TBHTPS framework.
The results showed that 85% of actors believed that HTM was failing in all components of the TBHTPS framework. The severity of the problems was rated differently by different stakeholder groups. Biomedical, clinical and HTM engineers and technicians rated the severity of the problems the highest. This was followed by decreasingly severe perceptions from users of equipment, managers and hospital directors, international organization representatives, local and foreign suppliers and, the policy makers, planners and administrators at the MoH, respectively. Questionnaires and interviews also identified the relative problem-solving power of actor groups. Policy makers, planners and administrators at the MoH were collectively regarded as the actor group most empowered to solve HTM problems (acknowledged by 39% of the questionnaire respondents and interviewees). This was followed by international organization representatives (16%), hospital managers and directors (10%), local and foreign suppliers (9%), users of equipment (9%) and finally, biomedical, clinical and healthcare technology engineers and technicians (5%).
3.3 Phase 3: Shared analysis and visioning
The goal of the third phase is to identify the root causes of problems in HTM and begin devising possible solutions for policy instrument development. During this phase, interactive learning activities can be organized to stimulate mutual learning between different stakeholders, develop a shared definition of problems and causes, and begin devising possible solutions. This includes interviews, FGs, heterogeneous dialogue meetings and design workshops. In Benin, the research activities comprised: (i) identification of the causes of problems identified in the previous phase, (ii) root cause analysis (RCA), and (iii) organization of problems and solutions into problem trees for prospective policy instrument development. The complete results of the root cause analysis are available in a separate article [58].
During the first research activity, several FGs took place with different stakeholders separately. Subsequently these inputs were combined into a larger discussion that included the users of health technologies, clinical, biomedical and health technology engineers and technicians, hospital managers and directors, and policy makers from the MoH. The FGs yielded a range of causes underlying HTM problems, some of which were more commonly identified by one actor group than another. During FGs, participants were also asked to conceive feasible solutions to the problems in HTM. Integrating knowledge between different groups built a consensus on problems and solutions, developed a shared vision for an improved HTM system, and allowed actors to envision future trajectories of the solutions discussed. This pertains to considering the backlash that might be encountered by alternative policy routes.
The RCA was performed during a session amongst experts, using the organized set of problems and causes formulated during the previous research activity. Main causes identified were the unwillingness and self-interested attitudes of policy makers to engage in HTM problems, and the high degree of politicization influencing public sector decision-making.
For the third research activity, problems and causes discussed during FGs and the RCA were organized into problem trees in an expert meeting. The logic guiding these meetings was to identify ‘low hanging fruit’ in the problem trees, i.e. solutions that could be implemented immediately to mitigate pertinent problems in HTM. The solutions prioritized included the development and implementation of: (i) policy and management tools to guide distribution of healthcare technologies; (ii) reference price lists and an essential equipment list for procuring equipment; (iii) generic specifications of equipment and architectural and technical requirement documents for installation and use; (iv) policy and management tools to guide financial resource allocation on the life-cycle costs (LCC) of equipment; (v) a healthcare equipment and maintenance directorate; (vi) policy and management tools for obsolete equipment; and (vii) a new HTM policy with a budgeted action plan.
3.4 Phase 4: Development of policy instruments for pilot testing
The aim of this phase is to develop and trial policy instruments based on the solutions identified in the RCA, in an attempt to understand the system’s responses to change and incorporate the lessons learned. This involves integrating the knowledge of previous phases to begin priority setting and planning. Again, interactive learning activities such as FGs, dialogue meetings and workshops, can be organized to negotiate the content of agendas, as well as questionnaires to assess support for the different solutions identified. The research activities conducted in Benin were: (i) development of the pilot policy instruments, (ii) pre-validation of pilot policy instruments, and (iii) final validation of pilot policy instruments.
The first research activity involved recruiting the services of experts to draft policy instruments for three new Zone Hospitals, funded by the African Development Bank and Islamic Development Bank. The policy instruments were developed from solutions devised during the previous phase that satisfied criteria of feasibility and acceptability and could be implemented immediately. An overview of these is presented in Table 3.
For the second research activity, FGs were conducted to validate the first draft of policy instruments. Apart from local and foreign equipment suppliers, every stakeholder group was consulted. After a second validation procedure with the same stakeholder groups and experts, the policy instruments were implemented and used to guide the procurement of equipment for, and construction of, three Zone Hospitals in Benin (Djidja, Cove, and Djougou). Although the long-term effects of the policy instruments cannot yet be determined, contracts for the basket of medical equipment required by one hospital were reduced by 18–25% [59]. Evaluating the successes and failures of the policy instruments’ implementation is ongoing but will inform implementation decisions for the prospective HTM policy.
3.5 Phase 5: Policy development and validation
The policy development and validation phase aims to draft the policy based on a combination of feasible and acceptable solutions. This includes the development of a strategic five-year action plan to guide implementation and which, alongside the contents of the policy, should be validated. Maintaining engagement with stakeholders is key to foster support for the policy and its implementation. Knowledge co-production, through FGs, workshops and expert meetings, can be used to draft the policy, with particular attention to its implementation. The research activities in Benin were (i) to draft the policy and action plan, and (ii) validate the policy.
The policy was drafted from solutions mapped in problem trees during phase 3. These solutions were organized into feasibility tables to determine their order of implementation. Fig 3 presents one of three problem trees, which shows the relationships between problems, causes and possible solutions for three key components of HTM: operation and safety, maintenance and repair, and commissioning and disposal. Nine of the fifteen intermediate causes shown in Fig 3 were selected based on their candidacy for feasible resolution: (i) acquisition of unsafe equipment, (ii) limited training opportunities for equipment users, maintenance technicians and procurement officers, (iii) lack of consumables and spare parts for maintenance and repair activities, (iv) lack of user technical manuals for equipment operation and maintenance, (v) high proportion of obsolete equipment in healthcare facility wards, (vi) lacking public-private partnerships for maintenance, (vii) lack of mechanisms to sell obsolete equipment to private healthcare facilities, (viii) lack of planning and budgeting for recurrent LCC of equipment, and (ix) lack of professional recognition and incentives for healthcare technology and maintenance technicians. The main cause of these problems was high-level corruption in HTM processes. Specifically, procuring low quality and expensive equipment was linked to opportunities for financial gain, and a culture of ‘replace over repair’ had emerged from parsimonious attempts to conserve the costs of paying a technician’s income.
Solutions for the above intermediate causes were parsed through criteria of feasibility (availability of competencies and funds) and acceptability (anticipated conflicts of interest between actors) during expert meetings. The feasibility table (Table 4) shows solutions from Fig 3, their typology (health policy-, money-, technical capacity-based) and the length of time for which the solution can provide adequate system improvement (short-, middle- or long-term). Eight appropriate solutions were devised in order of their priority for implementation: (i) development and implementation of policy and management tools for the oversight of equipment and materiovigilance; (ii) training grants and scholarships for equipment users, maintenance technicians and procurement officers; (iii) organization of public auctions to sell obsolete equipment (due to lacking spare parts and repair capacities); (iv) establishment and improvement of public-private partnerships for maintenance; (v) development and implementation of policy and management tools to guide financial resource allocation for LCC of equipment; (vi) creation of a healthcare equipment and maintenance directorate within the MoH, alongside a specific maintenance program; (vii) negotiation of maintenance contracts with suppliers for procured equipment, and; (viii) policy and management tools for obsolete equipment. This process was used for each of the problem trees (constructed from the RCA) when drafting the policy. The policy is unique as the first in Benin to incorporate the use of a conceptual framework and to account policy interventions against a budgeted action plan. The latter is particularly important for LMICs where some interventions depend on the technical or financial assistance of development partners.
The drafting process was followed by a series of validation workshops during which the prioritized actions were debated, with particular attention to their implementation, and to create opportunities for stakeholders to take responsibility for implementation actions. The space for heterogeneous dialogue during the workshop facilitated the process of conflict resolution, specifically the redistribution of some responsibilities and funding were heavily debated.
3.6 Phase 6: Policy implementation and evaluation
The aims of the final phase are to successfully implement the prioritized actions in the policy and to organize the capacities and activities for monitoring and evaluation of the policy’s implementation. The phase should proceed through reflexive learning cycles of planning, action, observation and reflection during implementation. Local-level experiments and incremental changes to the policy environment offer opportunities to monitor and evaluate the policy’s trajectory. The managerial activities conducted in Benin comprised: (i) mobilization of public and donor financial resources for implementation of the prioritized actions in the policy; (ii) development of the remaining policy instruments for each component of the HTM cycle; (iii) dissemination of the policy to local departments for coordinated implementation; (iv) creation of a permanent technical committee to define performance indicators and supervise all monitoring and evaluation activities; and (v) supervised training of key actors in the use of the policy instruments. Advocacies and lobbying at the MoH, as well as to donors, is necessary to ensure consistent funding for implementing prioritized action in the policy. Although some actions have already been taken with regards to mobilizing resources, the majority of implementation activities in Benin are forthcoming.
4. Discussion
In this section, the lessons learned from the results of each phase are described, the future barriers of implementation of the new policy are analyzed, and the strengths and limitations of using the model are presented. Finally, the applicability of the model in other African countries is discussed.
4.1 Lessons learned in each phase of the model
From the results of phase 1 (preparatory phase), the key lessons relate to the insights gained into the weaknesses of HTM system, particularly finding that the procurement market for health technologies is closed and vulnerable to corruption. Furthermore, the failure to implement the first HTM policy was related to top-down approaches employed. This informed the subsequent development process, particularly with the aim of building the policy from the bottom-up, enhancing transparency and participation, and utilizing an action plan to guide its implementation. Through analyzing the failures of previous HTM and PP policies, it was clear that power positions in HTM are complex, and that a lack of political will had kept some previous policies from being implemented.
The main lesson learned from phase 2 (stakeholder identification and problem analysis) was that different actors held different perceptions of the problems, e.g. HTM problems prioritized by clinical engineers would not necessarily align with those of MoH policy makers. Furthermore, it was found that the ability of each key actor to solve HTM problems (the degree of political or administrative power they possess) was inverse to their perception of the severity of the problems. This means that actors with low power (such as, engineers and technicians) in the system are highly aware of the problems in HTM but have little capacity to resolve them, whereas individuals in powerful positions have little awareness of the problems but possess the authority to implement solutions. Previous policies directed from the top-down had been controlled by a small number of policy makers, and the opinions of stakeholders with clear perceptions of the problem had never been connected to those with the power to change the system. As a consequence, it was necessary to acknowledge and identify, at each level, the relevant key actors who are important as enablers towards more effective and efficient HTM processes, as their support increases the chances of success. It also holds important implications for subsequent phases with regards to managing conflicting opinions, and ensuring that a small number of views from more powerful participants do not dominate FGs.
From the findings of phase 3 (shared analysis and visioning), we can learn that solutions differ in feasibility. The conditions that must be aligned to realize the aims of certain policy interventions require different combinations of resources, time and contingent factors. For example, during this phase it was possible to identify quick win policy instruments that did not require other contingent factors to be present to be implemented, whereas some policy interventions can be better planned for implementation in the future. Another key finding relates to the large number of politicized decisions that influence HTM processes. In some instances, technical decisions had been compromised due to political influence, e.g. politicized health technology distribution has had an effect on the relative accessibility of health technologies across the country. We recognized that the design of the new policy development process should be structured to limit the opportunities for corruption. As such, development processes should be transparent, and the decisions set forth in policy documents should be evidenced by technical knowledge produced during interactive research activities.
There were three lessons learnt from phase 4 (development of policy instruments for pilot testing):
- Strong resistance: One of the key challenges was satisfying (or in some cases, overcoming) the combination of official and unofficial interests of different actors during policy instrument development. Their implementation, especially the architectural and technical requirements for equipment document, faced strong resistance as it acted against the interests of infrastructure contractors and equipment suppliers. Civil engineers strongly disapproved as more efficient construction of infrastructure projects would decrease the amount of future reparative works they could solicit. Additionally, the generic technical specifications of equipment worked by ignoring considerations of brand and stimulating international open tender competition, yet was lobbied by many suppliers to have their equipment preferred over others’ for specific types of interventions.
- Bottom-up integration of technical knowledge: Policy decisions can be enriched by using experiential knowledge as it offers, for top-level decision makers who may not have the technical knowledge to solve a problem, a thoroughly discussed solution which is supported by actors charged with implementation. Training high-level government personnel in this approach, alongside committed stakeholders and transparency to diminish the space for closed-shop corruption, can bridge the gap between political decisions and technical knowledge.
- Saving scarce resources: The 25% reduction in procurement costs, eliminated by the equipment reference price list, is a hopeful indication for the future effects of the policy instruments, and supports the idea that previous PP decisions had not been fully executed with the best interests of Beninese society in mind.
The lessons from phase 5 (policy development and validation) illustrate that changes to the formal structure of the HTM system are necessary to reduce the space for poor practices. The joint development of policy instruments to direct the processes of health technology procurement, distribution, resource allocation and maintenance will begin to change the values and practices of actors. Particularly, the establishment and formalization of an independent medical device directorate will promote the importance of HTM and provide it with the central authority and resources to help it manage HTM processes. By anchoring this control in law, it has already been observed to improve economic efficiency.
4.2 Future barriers of implementing the policy
Predictable barriers are those which have some scope for management, such as, stakeholder resistance or the lack of financial, technical, and political support. Financial support requests can be improved if, during the submission of budgeted action plans to donors, policy implementation activities are aligned to those donors who promote a similar remit of activities. To improve technical support, transparency and accountability in government can help rebuild trust between high-level MoH officials and lower-level HTM professionals, and improve the knowledge flow from the bottom-up. The implementation of the policy looks to decrease the levels of corruption and unethical practices, however follow-up by a permanent technical committee should be in place to monitor and evaluate the policy’s implementation. To maintain political support, it is important that the technical permanent committee in charge of follow-up preserves a productive correspondence with political authorities.
Unpredictable barriers that could hinder policy implementation processes are radical shifts in political support and similarly unpredictable events (loss of the policy’s key promoters, political instability, and war). A radical lack of political support could, for example, arise if conflicting opinions between a newly appointed Minister of Health and technical HTM director cannot be resolved. This could cause deliberate action by the government to undermine policy implementation. In this context, motivated or unmotivated replacements of officials in the MoH would have a similarly detrimental effect.
4.3 Strengths and limitations of using the model
The policy development process was guided by five general principles: (i) develop evidence-informed and action-oriented knowledge for sustainable development, (ii) employ a strategy that is built on a shared future vision and looks to anticipate the risks and benefits of possible interventions, (iii) inclusivity (complete stakeholder involvement), (iv) preventing corruption, and (v) independent justice; and five context-specific sub-principles: (i) depoliticize public decision-making, (ii) promote patriotism, (iii) punitive actions against corruption (iv) enhance civism, and (v) strengthen capacity in institutions. During the implementation of policy instruments in phase 4, the importance of the new principles (patriotism, civism, decreasing politicized decision-making) to the Beninese context was evidenced. Changes made to the PP process, by the new policy instruments, required a different professional culture and practice of procurement staff, as procurement decisions were now directed by stringent guidelines. This was recognized during a goods contracts reception, as staff agreed that greater shrewdness was needed to acquire the correct goods from suppliers under the new guidelines. This could, in part, underpin the 18–25% reduction observed in PPAPs for constructing the new zone hospitals, believed to be due to less payoffs. Many authors have also stressed the concept of patriotism as a key element of GG for public administrations [60, 61], and the philosophy is well illustrated by a quote from Joe Biden: “Fighting corruption is not just good governance. It's self-defense. It's patriotism” [62]. At a national scale, corruption has been linked to poorer health system performance indicators including the maternal mortality ratio [63]. One example, in Costa Rica, described successful HTM reform due to a product of clear oversight, provided by health authorities during the project period, and a dedicated HTM workforce [3]. This edifying example of public officials taking ownership of actions to improve the system is needed in Benin. The new principles are important as they make explicit what hurdles other African countries can expect to encounter. The access to, and maintained engagement of, key stakeholders during the policy development process was also a crucial factor, which is stressed in principles (ii) and (iii). Initially this proved difficult, particularly with healthcare technology suppliers with a vested interest for maintaining the status quo. Recognizing the importance of each stakeholder’s knowledge and encouraging their contribution aided the process of building trust. This has been stressed, under the concept of ‘sensitivity’, in other participatory projects [64]. Although efforts to realize the effective implementation of these new GG principles is highly recommended, they are nevertheless subject to system and contextual limitations in their achievability.
4.4 Applicability of the model in the other African countries
The development of the policy in an evidence-informed fashion required quite some time to complete, as well as compliance and continuing support for the research team’s work. The replication of the model in other African countries would be enhanced when tailored to the specific demands of its context. Furthermore, implementing GG practices will always face resistance as the self-interest of the stakeholders are usually at stake. Chances of successful implementation will be increased when values of the society and state are built upon democracy, solidarity and altruism. The translation to other countries also depends on how contextual barriers can be successfully managed, as the six phases are not equally implementable, e.g. the fourth phase requires long periods of experimentation for pilot projects. It is important to state that, during policy development, the project benefitted from the unique position held by the first author, both as a high-level civil servant of the Beninese government, and as an academic researcher, which allowed the introduction of new approaches to build capacity in government (e.g. by incorporating experiential and technical knowledge in policy development, and using monitoring and evaluation frameworks after policy implementation). Indeed, there are many constraints to using research in public health decision making, including the concentration of power, the extent of health policy centralization and democratization, institutional processes and predominant values, the influence of donors and pressure of wider policy strategies [65], and the division in disciplines between researchers and decision-makers [66]; the latter of which this policy development process has sought to diminish. As a result of this process, the need for stronger capacities in governmental monitoring and evaluation has been observed in Benin, as well as greater capacities for conducting policy-oriented research, that partners with society to attain collective health goals, in low-income countries [67]. We hope this will be taken on board by the West African Health Organization (WAHO), which has a wide policy coordination and harmonization program for its member states. Tools to ensure accountability of public officials are required for implementation and success in other contexts. The case presented here contributes to the increasing body of literature that emphasizes the importance of accountability in public decision-making, particularly for those individuals entrusted with prominent public functions and with access to state accounts and funds [68]. Establishing a positive feedback loop of information between state and society is important for continued progress.
Acknowledgments
Icons used in Fig 1 made by SimpleIcon, Freepik, Vectors Market, Hanan, Pedro Nieto Villamandos, Elegant Themes, Google from www.flaticon.com
Author Contributions
- Conceptualization: PTH DM JFGB.
- Data curation: PTH.
- Formal analysis: PTH HLSC MZ TDCB JFGB.
- Funding acquisition: PTH MZ JFGB.
- Investigation: PTH.
- Methodology: PTH MZ TDCB JFGB.
- Project administration: PTH.
- Resources: PTH DM.
- Supervision: DM JFGB.
- Validation: PTH MZ TDCB DM JFGB.
- Visualization: PTH HLSC.
- Writing – original draft: PTH HLSC.
- Writing – review & editing: PTH HLSC MZ TDCB DM JFGB.
References
- 1. Lewis M. Governance and Corruption in Public Health Care Systems. Cent Glob Dev Work Pap [Internet]. 2006; (78):57. Available from: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=984046
- 2.
World Health Organization. Essential Health Technologies Strategy 2004–2007: Taking Basic Health Solutions to Countries: a Needs-driven Programme Increasingly Based on Country-prepared Proposals [Internet]. World Health Organization; 2004. https://books.google.nl/books?id=qZJwHQAACAAJ
- 3.
Remmelzwaal BL. Technological learning and capacity building in the service sector of developing countries: the case of medical equipment management. University of Sussex; 1996.
- 4.
Temple-Bird C. Managing the import and use of healthcare technology in Sub-Saharan Africa. Open University; 2005.
- 5. Perry L, Malkin R. Effectiveness of medical equipment donations to improve health systems: how much medical equipment is broken in the developing world? Med Biol Eng Comput [Internet]. 2011;49(7):719–22. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21597999
- 6.
Halbwachs H. The Importance of Maintenance & Repair in Health Facilities of Developing Economies. In: Summit Conference of the African Federation for Technology in Health Care. Harare, Zimbabwe; 1998.
- 7.
Kachieng’a MO. Health technology assessment in sub-Saharan Africa: A cross-national study of Kenya and South Africa. University of Cape Town; 1999.
- 8.
World Health Organization. Baseline country survey on medical devices. 2010;308. Available from: www.who.int/iris/handle/10665/95785-17k
- 9.
Summers LH, Shah S. Proceedings of the World Bank Annual Conference on Development Economics. Vol. 39. 1992. 1–317 p.
- 10. Doornbos M. “Good Governance”: The Rise and Decline of a Policy Metaphor? J Dev Stud. 2001;37(6):93–108.
- 11. Santiso C. Good Governance and Aid Effectiveness: The World Bank and Conditionality. Georg Public Policy Rev. 2001;7(1):1–22.
- 12. Grindle M. Good enough governance: Poverty reduction and reform in developing countries. Governance. 2004;17(4):525–48.
- 13.
UNDP. Governance for Sustainable Human Development—Human Development Report 1997. 1997;159.
- 14.
World Bank. Governance: The World Bank’s Experience. 1994. 88 p.
- 15.
DFID. Making government work for poor people. Building state capability. 2001;39.
- 16.
World Health Organization. Everybody’s business: strengthening health systems to improve health outcomes: WHO’s framework for action. Production [Internet]. 2007;1–56. http://www.who.int/healthsystems/strategy/everybodys_business.pdf
- 17.
Brinkerhoff DW, Bossert TJ. Health Governance: Concepts, Experience, and Programming Options. Usaid [Internet]. 2008;(February):1–32. http://www.healthsystems2020.org/content/resource/detail/1914/
- 18.
Baez-Camargo C, Jacobs E. A Framework to Assess Governance of Health Systems in Low Income Countries Basel Institute on Governance. 2011;(11).
- 19. Barbazza E, Tello JE. A review of health governance: Definitions, dimensions and tools to govern. Health Policy (New York) [Internet]. Elsevier Ireland Ltd; 2014;116(1):1–11. Available from: http://dx.doi.org/10.1016/j.healthpol.2014.01.007
- 20.
Buse K, Mays N, Walt G. Making health policy. McGraw-Hill Education (UK); 2012.
- 21.
Houngbo PT, van der Wilt GJ, Medenou D, Dakpanon L, Bunders J, Ruitenberg J. Policy and Management of Medical Devices for the Public Health Care Sector in Benin. In: Public Health—Methodology, Environmental and Systems Issues [Internet]. InTech; 2012 [cited 2016 Jul 8]. http://www.intechopen.com/books/public-health-methodology-environmental-and-systems-issues/policy-and-management-of-medical-devices-for-the-public-health-care-sector-in-benin
- 22.
Ministère de la Santé Publique. Etude sur l’élaboration d’un système décentralisé de maintenance hospitalière (30 mars au 15 avril 1995). Cotonou, Benin; 1995.
- 23.
Ministère de la Santé Publique. Politiques et Stratégies Nationales de Maintenance Hospitalière en République du Bénin (2001–2005). Cotonou, Benin; 2000.
- 24.
Ministère de la Santé Publique. Politique de maintenance des infrastructures, des équipements médico-techniques et du parc automobile en République du Bénin. Cotonou, Benin; 2002.
- 25.
Ministère de la Santé Publique. Rapport de la mission d’expertise thématique en gestion et maintenance des équipements et infrastructures de la santé. Cotonou, Benin; 2005.
- 26.
Ministère de la Santé. Recueil d’informations de la Cellule de Passation des Marchés. Cotonou, Benin; 2006.
- 27.
Union Européenne. Assistance Technique auprès du Ministère de la Santé du Bénin en vue d’une étude préliminaire sur l’élaboration d’une politique en matière d’acquisition, de gestion et de maintenance des équipements médico-techniques au Bénin, Rapport final présenté par D. 2010.
- 28. Lehmann U, Gilson L. Action learning for health system governance: The reward and challenge of co-production. Health Policy Plan. 2015;30(8):957–63. pmid:25161271
- 29. Erasmus E, Gilson L. How to start thinking about investigating power in the organizational settings of policy implementation. Health Policy Plan. 2008;23(5):361–8. pmid:18664526
- 30. Ambe IM, Badenhorst-Weiss J a. Procurement challenges in the South African public sector. J Transp Supply Chain Manag. 2012;6(1):242–61.
- 31. Brinkerhoff DW, Bossert TJ. Health governance: Principal-agent linkages and health system strengthening. Health Policy Plan. 2013;29(6):685–93. pmid:23411121
- 32. Sheikh K, Ranson MK, Gilson L. Explorations on people centredness in health systems. Health Policy Plan [Internet]. 2014;29(suppl 2):ii1–ii5. Available from: http://www.heapol.oxfordjournals.org/cgi/doi/10.1093/heapol/czu082
- 33. Abimbola S, Negin J, Jan S, Martiniuk A. Towards people-centred health systems: a multi-level framework for analysing primary health care governance in low- and middle-income countries. Health Policy Plan [Internet]. 2014;29 Suppl 2:ii29–39. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=4202919&tool=pmcentrez&rendertype=abstract
- 34. Scott V, Schaay N, Olckers P, Nqana N, Lehmann U, Gilson L. Exploring the nature of governance at the level of implementation for health system strengthening: the DIALHS experience. Heal Policy Plan [Internet]. 2014;29(suppl 2):ii59–ii70. Available from: http://heapol.oxfordjournals.org/content/29/suppl_2/ii59.abstract%5Cnhttp://heapol.oxfordjournals.org/content/29/suppl_2/ii59.full.pdf
- 35. Balabanova D, Mills A, Conteh L, Akkazieva B, Banteyerga H, Dash U, et al. Good health at low cost 25 years on: Lessons for the future of health systems strengthening. Lancet. 2013;381(9883):2118–33. pmid:23574803
- 36.
Bunders JFG, Broerse JEW, Keil FP, C. Scholz RW, Zweekhorst BM. How can transdisciplinary research contribute to knowledge democracy? In: in’t Veld RJ, editor. Knowledge Democracy: Consequences for Science, Politics, and Media. 1st ed. Springer-Verlag Berlin Heidelberg; 2010. p. 125–52.
- 37.
Zweekhorst MBM. Institutionalising an interactive approach to technological innovation: The case of the Grameen Krishi Foundation. Vrije Universiteit Amsterdam; 2004.
- 38.
Broerse JEW. Towards a new development strategy: How to include small-scale farmers in the technological innovation process. Delft University of Technology; 1998.
- 39.
Caron-Flinterman JF. A new voice in science: Patient participation in decision-making on biomedical research. Vrije Universiteit Amsterdam; 2005.
- 40.
Roelofsen A. Exploring the future of ecogenomics: Constructive technology assessment and emerging technologies. Vrije Universiteit Amsterdam; 2011.
- 41. Broerse JEW, de Cock Buning T, Roelofsen a, Bunders JFG. Evaluating Interactive Policy Making on Biotechnology: The Case of the Dutch Ministry of Health, Welfare and Sport. Bull Sci Technol Soc. 2009;29(6):447–63.
- 42.
Jahn T, Keil F. Transdisziplinärer Forschungsprozess. In: Becker E, Jahn T, editors. Soziale Ökologie Grundzüge einer Wissenschaft von den gesellschaftlichen Naturverhältnissen. Campus-Verlag; 2006. p. 319–29.
- 43.
Kaufmann D. Rethinking Governance: Empirical Lessons Challenge Orthodoxy. SSRN Electron J [Internet]. 2003;47p. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=386904%5Cnhttp://www.ssrn.com/abstract=386904
- 44. Scholz RW, Lang DJ, Wiek A, Walter A, Stauffacher M. Transdisciplinary case studies as a means of sustainability learning. Int J Sustain High Educ. 2006;7(3):226–51.
- 45.
Hyden G, Court J, Mease K. Making sense of governance: the need for involving local stakeholders. Gov An Int J Policy Adm [Internet]. 2003;1–11. http://www.odi.org.uk/resources/details.asp?id=3135&title=making-sense-governance-need-involving-local-stakeholders
- 46. Siddiqi S, Masud TI, Nishtar S, Peters DH, Sabri B, Bile KM, et al. Framework for assessing governance of the health system in developing countries: Gateway to good governance. Health Policy (New York). 2009;90(1):13–25.
- 47.
Kickbusch I, Gleicher D. Smart governance for health and well-being: the evidence. Smart Gov Heal well-being … [Internet]. 2014;172. http://www.euro.who.int/__data/assets/pdf_file/0005/257513/Smart-governance-for-health-and-well-being-the-evidence.pdf#page=151
- 48.
Lewis M, Pettersson G. Governance in Health Care Delivery Raising Performance. Policy Research Working Papers. 2009. Report No.: 6074.
- 49.
Savedoff WD. Governance in the Health Sector: A Strategy for Measuring Determinants and Performance [Internet]. Vol. 1, Social Insight. Portland, ME, USA; 2009. http://www-wds.worldbank.org/servlet/main?menuPK=64187510&pagePK=64193027&piPK=64187937&theSitePK=523679&entityID=000158349_20110509125737
- 50.
International Monetary Fund. Good Governance, the IMF’s Role. Washington, DC, USA; 1997.
- 51. Dobriansky P. Principles of good governance. Econ Perspect. 2003;8:6–8.
- 52.
UNDP. Human Development Report 2007. New York, USA; 2007.
- 53.
Larousse Dictionnaire.
- 54.
Houngbo PT, Kishna M, Zweekhorst M, Medenou D, Bunders J. The Effect of Benin’s First Public Procurement Code and its Amendment on Healthcare Equipment Acquisition Prices. J Public Procure. Forthcoming 2017.
- 55.
African Development Bank. Republic of Benin: Country Governance Profile Report. 2005.
- 56.
Houngbo PT, De Cock Buning T, Bunders J, Coleman HLS, Medenou D, Dakpanon L, et al. Ineffective Healthcare Technology Management In Benin’s Public Health Sector: The Perceptions Of Key Actors And Their Ability To Address The Main Problems. Int J Heal Policy Manag. Forthcoming 2017.
- 57.
Kothari CR. Research methodology: Methods and techniques. New Age International; 2004.
- 58.
Houngbo PT, Zweekhorst MBM, Bunders JFG, Coleman HLS, Medenou D, Dakpanon L, et al. The Root Causes of Ineffective and Ineffecient Healthcare Technology Management in Benin’s Public Health Sector. Heal Policy Technol. Forthcoming 2017.
- 59.
Ministère de la Santé. Marchés relatifs à la fourniture, l’installation et la mise en service d’équipements hospitaliers au profit des Hôpitaux de Zone de Djidja, Djougou et Covè: Projet PADS Santé III-BAD, Ministère de la Santé du Bénin, Financement BAD/FAD et BID, 2012 et 201. Cotonou, Benin; 2013.
- 60.
Tan K. Justice without Borders: Cosmopolitanism, nationalism, and patriotism. In: Shapiro I, editor. Contemporary Political Theory. 1st ed. Cambridge, UK: Cambridge University Press; 2004.
- 61.
Odika U Jr.. Patriotism Has No Party: Defining Democracy Within the Context of Ensuring Peace. Xlibris; 2013.
- 62.
Biden J. Remarks by Vice President Joe Biden to the Global Entrepreneurship Summit. In: Global Entrepreneurship Summit [Internet]. Marrakech, Morocco: The White House; 2014. p. Para. 25. https://www.whitehouse.gov/the-press-office/2014/11/20/remarks-vice-president-joe-biden-global-entrepreneurship-summit
- 63. Pinzón-Flórez CE, Fernández-Niño JA, Ruiz-Rodríguez M, Idrovo ÁJ, López AAA. Determinants of performance of health systems concerning maternal and child health: A global approach. PLoS One. 2015;10(3):1–27.
- 64. Sarkki S, Heikkinen HI, Karjalainen TP. Sensitivity in transdisciplinary projects: A case of reindeer management in Finland. Land use policy [Internet]. Elsevier Ltd; 2013;34:183–92. Available from: http://dx.doi.org/10.1016/j.landusepol.2013.03.004
- 65. Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy A systematic review. PLoS One [Internet]. 2013;8(10):e77404. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3813708&tool=pmcentrez&rendertype=abstract
- 66. Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The use of research evidence in public health decision making processes: Systematic review. PLoS One. 2011;6(7).
- 67. Adam T, Ahmad S, Bigdeli M, Ghaffar A, Røttingen J-A. Trends in Health Policy and Systems Research over the Past Decade: Still Too Little Capacity in Low-Income Countries. PLoS One [Internet]. Public Library of Science; 2011 Nov 22;6(11):e27263. Available from: http://dx.doi.org/10.1371%2Fjournal.pone.0027263
- 68. Choo K-KR. Challenges in dealing with politically exposed persons. Trends & Issues. 2011;(419):1–4.