Increasing access to health workers in underserved areas: a conceptual framework for measuring results

Many countries have developed strategies to attract and retain qualified health workers in underserved areas, but there is only scarce and weak evidence on their successes or failures. It is difficult to compare lessons and measure results from the few evaluations that are available. Evaluation faces several challenges, including the heterogeneity of the terminology, the complexity of the interventions, the difficulty of assessing the influence of contextual factors, the lack of baseline information, and the need for multi-method and multi-disciplinary approaches for monitoring and evaluation. Moreover, the social, political and economic context in which interventions are designed and implemented is rarely considered in monitoring and evaluating interventions for human resources for health. This paper proposes a conceptual framework that offers a model for monitoring and evaluation of retention interventions taking into account such challenges. The conceptual framework is based on a systems approach and aims to guide the thinking in evaluating an intervention to increase access to health workers in underserved areas, from its design phase through to its results. It also aims to guide the monitoring of interventions through the routine collection of a set of indicators, applicable to the specific context. It suggests that a comprehensive approach needs to be used for the design, implementation, monitoring, evaluation and review of the interventions. The framework is not intended to be prescriptive and can be applied flexibly to each country context. It promotes the use of a common understanding on how attraction and retention interventions work, using a systems


Introduction
Health workers' willingness to practise in underserved areas, such as rural, remote or poor areas, is a recognized challenge in achieving equitable access to health services.Many countries have developed strategies to attract and retain qualified health workers in these areas.5][6] However, this has yet to be applied to the evaluation of human resources for health interventions and, specifically, to those seeking to increase access to health workers in underserved areas.There is an urgent need to achieve an agreement and strengthen the evidence that would underpin sound policy recommendations in this area.
Seeking to address this gap, this paper proposes a conceptual framework to guide managers, policy-makers and evaluators in the assessment of interventions to increase access to health workers in underserved areas.The framework aims to support all stages of policy development.It suggests a logical sequence that can be followed when deciding on any intervention that ad-dresses attraction and retention issues and assists in formulating key questions to be answered when designing, monitoring and evaluating such interventions.It includes examples of indicators to inform the process, which can be adapted to a specific context.A comprehensive consideration of indicators to monitor the health workforce in general is offered elsewhere. 7

Challenges in evaluation
Policy-and decision-makers need to know whether interventions work or not, why they work and in which context.Therefore it is important to have information about the effects of the interventions, but also about the factors that made the intervention succeed or fail (the questions dealing with "when, why, how, and in what circumstances such interventions work well or fail to work"). 8valuating interventions to improve human resources for health is complex for different reasons.First, relating these interventions to health status is very difficult due to broad socioeconomic, cultural, political and health systems factors that influence health.For instance, although improved health outcomes, such as reduction in maternal mortality, are directly correlated with increased availability of health workers, it is difficult to attribute the improvement directly to a certain health workforce intervention. 9,10Illustrative country case studies showed that Afghanistan and Ethiopia have implemented Une traduction en français de ce résumé figure à la fin de l'article.Al final del artículo se facilita una traducción al español.

‫املقالة.‬ ‫لهذه‬ ‫الكامل‬ ‫النص‬ ‫نهاية‬ ‫يف‬ ‫الخالصة‬ ‫لهذه‬ ‫العربية‬ ‫الرتجمة‬
Abstract Many countries have developed strategies to attract and retain qualified health workers in underserved areas, but there is only scarce and weak evidence on their successes or failures.It is difficult to compare lessons and measure results from the few evaluations that are available.Evaluation faces several challenges, including the heterogeneity of the terminology, the complexity of the interventions, the difficulty of assessing the influence of contextual factors, the lack of baseline information, and the need for multimethod and multi-disciplinary approaches for monitoring and evaluation.Moreover, the social, political and economic context in which interventions are designed and implemented is rarely considered in monitoring and evaluating interventions for human resources for health.This paper proposes a conceptual framework that offers a model for monitoring and evaluation of retention interventions taking into account such challenges.The conceptual framework is based on a systems approach and aims to guide the thinking in evaluating an intervention to increase access to health workers in underserved areas, from its design phase through to its results.It also aims to guide the monitoring of interventions through the routine collection of a set of indicators, applicable to the specific context.It suggests that a comprehensive approach needs to be used for the design, implementation, monitoring, evaluation and review of the interventions.The framework is not intended to be prescriptive and can be applied flexibly to each country context.It promotes the use of a common understanding on how attraction and retention interventions work, using a systems perspective.
Increasing access to health workers in underserved areas: a conceptual framework for measuring results

Luis Huicho et al.
Measuring efforts to increase access to health workers Special theme -Health workforce retention in remote and rural areas comprehensive health sector strategies with multiple co-existing interventions, like the recruitment of community health workers, which may have promoted access to life-saving services, thus improving health status. 11nother problem is that many evaluations lack a baseline against which to assess the results, particularly in countries with a major health worker deficit, 3,10,12 as well as a specific intervention logic that clarifies the expectations of the intervention designers (S Kane, B Gerretsen, R Scherpbier, M Dal Poz, MA Dieleman, unpublished data, 2009). 13Moreover, the social, political and economic context in which interventions are designed and implemented is rarely considered in monitoring and evaluation of human resource interventions. 14,15hus, the main challenges which evaluators face are related to the multidimensional nature of interventions, and the difficulty of assessing the influence of contextual factors.To be able to assess such interventions in complex systems, there is a need for multimethod and multidisciplinary monitoring and evaluation approaches, inclusive of all relevant stakeholders. 16Another challenge is generating lessons across regions and countries to inform the global health agenda and promote a meaningful dialogue at both national and international level.There is clear need for a framework that offers a model for monitoring and evaluation of health worker retention interventions taking into account such challenges.

Conceptual framework
The proposed framework is based on a systems approach and differentiates between "inputs", "outputs", "outcomes" and "impact", with regards to the results of interventions to attract and retain health workers in underserved areas.It builds on the proposed common framework for monitoring performance and evaluating progress in the scale-up for better health, 17 and proposes indicators to measure progress in implementing various strategies, allowing users to determine what works or not, and to explore contextual factors influencing their success or failure.The framework has two aims: (i) to guide the thinking in evaluating an intervention to increase access to health workers in underserved areas, from its inception/ design phase through to its results, by suggesting key questions about the relevance, efficacy, efficiency, effectiveness and sus-tainability of the intervention; and (ii) to guide the monitoring of interventions, through a focus on a routine collection of a set of indicators, applicable to the specific context.We used the following definitions for evaluation and monitoring: 4 We define "evaluation" as the systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results.The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability.
We define "monitoring" as a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds.
Later in this paper, we examine the application of the framework to the stages of policy development: design, implementation, monitoring and evaluation (Fig. 1).The sequence in design and implementation phases does not necessarily imply a linear policy process, as monitoring and evaluation may help identify a need for new interventions.They are, in effect, part of an ongoing policy cycle.

Design
The evaluation of the design stage should identify the extent to which the chosen intervention was relevant and adequate to the need, the expectations of the population and the health system context, and the reasons for selecting a particular option. 18By relevance, we mean the extent to which the objectives and the elements of the intervention are consistent with beneficiaries' requirements, country needs, global priorities and partners' and donors' policies. 4The relevance issue can and should also be addressed retrospectively; and then the question is whether the objectives of an intervention or its design were and are still appropriate given changed circumstances. 4uestions that an evaluator should ask are: did the intervention respond to a documented need, was it appropriate for the given context and how was it selected?Is the choice of the intervention based on evidence or at least on robust arguments?

Context:
Social determinants, political situation, stakeholders power and interests, economic issues (fiscal space, fiscal decentralization), individual level factors (marital status, gender) IMR, infant mortality rate; MMR, maternal mortality rate.

Luis Huicho et al.
Measuring efforts to increase access to health workers Special theme -Health workforce retention in remote and rural areas The assessment of the situation analysis should check if the problem of limited access to health workers has been clearly described.Is it due to an insufficient number of students willing to work in underserved areas, or an insufficient production of health workers, or were there other reasons?The evaluation needs to assess if a health labour market analysis has been conducted, to identify whether the lack of access to health workers in underserved areas was a matter of insufficient demand, insufficient supply, or insufficient remuneration and incentives, and unattractive working conditions. 19he situation analysis should also identify whether the elements in place to support the implementation of the intervention were assessed and if a feasibility analysis preceded the choice of the intervention.Strategies to increase access to health workers in underserved areas require participation of stakeholders from different sectors.Therefore, questions to be asked include: were mechanisms for multi-stakeholder engagement in place and used, were regulatory frameworks analysed before the intervention, were the intervention costs calculated and considered affordable (availability of funds), was the organizational capacity to support its implementation analysed?At this stage, the Human Resources for Health Action Framework is a recognized tool that can be used to guide the situation analysis (Fig. 2). 21It comprises health system dimensions such as policy, finance, education, partnership, leadership and human resources management systems.
At the end of this phase, a strategy can be proposed to address the perceived need of lack of access to health workers in underserved areas and of maldistribution of health workers.Such a strategy can be in one or more of the following thematic areas: education, regulatory, financial, and/or management and support, described elsewhere. 3

Implementation
When the implementation of an intervention is evaluated, questions might include: did the activities take place according to plan, did any changes occur in implementation capacity or in the context that influenced the implementation of the intervention components?Did any changes occur regarding the involvement and commitment of stakeholders?Which ones and why?Was any action undertaken to adapt the intervention to the changes?What was done and why?Or why was no action taken?Monitoring the progress in implementation can be informed by the systematic collection and analysis of a set of key indicators, which are further discussed in this paper.

Results
Intervention results can be evaluated at different levels (Fig. 1): outputs, interme-diate outcomes, outcomes and impact.Questions to be asked include: did the intervention achieve its intended results at the level of outputs, outcomes, or impact?Is the intervention sustainable?What were the contextual factors (political, economic, social, or organizational) that have influenced the results?
The level of "outputs" is the direct effect, attributable to the intervention, such as increased attractiveness of the areas/or the profession, improved recruitment and deployment, and increased retention of health workers in those areas.
The level of intermediate "outcomes" is that of improved health workforce performance, the dimensions of which have been described in The world health report 2006 as "availability, competence, responsiveness and productivity". 10Some attraction and retention interventions can have a direct effect on these dimensions too.For example, changes in rural curricula can improve competencies to address rural health problems and performance-based financial incentives can improve productivity.However, often improved health workforce performance cannot be attributed to retention interventions alone, as there are many other factors that contribute, such as education strategies or health sector management reforms.
The level of "outcomes" is that of increased health systems performance, measured by increased coverage and utilization of services, and improved responsiveness.Improved health workforce performance could contribute to improved health system performance but a host of other factors can contribute as well, such as the availability of drugs, infrastructure and equipment. 22inally, the level of "impact" is understood as the effect on health outcomes and an improved health status, and  Special theme -Health workforce retention in remote and rural areas other unexpected effects on underserved communities (e.g.attraction/retention of other professionals or services).This cannot be directly attributed to retention interventions, as the simple availability of health workers cannot directly improve health outcomes in the absence of effective components such as drug supply, functioning facilities and good clinical practices. 9But a well performing health workforce is a significant component of a strong health system that, together with other social determinants, can lead to improved health status. 22valuations of any attraction and retention intervention need to consider the contextual factors that may explain why and how such interventions have achieved the intended outputs, outcomes and impact, i.e. the social, economic and political context.This includes a stakeholder analysis including their respective power bases and interests, as well as an analysis of the capacity of the health sector to implement.Evaluations should use both qualitative and quantitative methods to measure outputs and impacts, but also to analyse formal and informal policy processes accompanying the design and implementation of interventions.Ideally, to find out if an attraction and retention intervention works or not, a cluster randomized controlled trial with an adequate control group would be needed, but this may not be feasible or ethical in the "real life" policy scenario.Moreover, causal chains are often complex in public health interventions, making results of randomized controlled trials subject to effect modification in different populations. 20Alternative observational methods such as before-after and plausibility designs can also be considered, taking care to document and control the contextual factors appropriately. 20,23Studies with adequacy and plausibility designs can improve both the internal and external validity of randomized controlled trials, and plausibility studies are often the only feasible option for getting valid evidence of impact when attempting the evaluation of large-scale interventions. 23

Monitoring
The framework depicted in Fig. 1 proposes a tool to guide the monitoring of interventions to increase access to health workers in underserved areas.At each stage of the implementation process, indicators are proposed to assess whether the implementation is on track.
Indicators to monitoring at the design stage should be those that will allow for a detailed description of the stocks and flows of health workers across sectors, age and sex categories, geographical areas and professions.Detailed description of these indicators, as well as data sources for health workforce monitoring, are described elsewhere. 7hen monitoring the implementation of interventions, systematic data collection is required on the planned activities, by constantly reviewing the production capacity, as well as the organization and management capacity.Monitoring implementation also requires tracking changes in context and stakeholders, so as to be able to adapt the interventions when required.
Regarding the monitoring of results of the intervention, at the level of "outputs" it is important to monitor whether the dimensions of "attractiveness", "engagement/ recruitment", "deployment" and "retention" are achieved.Indicators to monitor progress on these dimensions can include, for example, changes in the intentions of students or health workers to relocate, reside in or leave underserved areas (attractiveness).The available funded positions in underserved areas can be an indicator of effective engagement, whereas the number of health workers recruited and changes in vacancy rates can be indicators of effective deployment and recruitment.Finally, the effects of an intervention on retention can be measured by duration in post, turnover, absenteeism, stability index or even survival rates. 24,25t the level of "outcomes", it is proposed to monitor the progress on improved health workforce performance and health systems performance, through dimensions and indicators proposed by The world health report 2006 (Table 1). 10As said before, assessing the longer-term impact of human resource interventions on health outcomes should be done within the broader context of factors influencing health status.
The effectiveness of attraction and retention interventions can only be measured at the level of outputs and, to a certain extent, at the level of outcomes.In addition, implementing human resource retention interventions can take many years before any results can be seen, for example in the case of creating new schools in rural areas or rolling out financial incentive schemes.The conceptual framework presented here needs to be extensively piloted and will be refined through further inputs from different sources.There are ongoing plans for applying it in several countries that are in the process of designing and implementing attraction and retention strategies in underserved areas.Efforts will also be made to make sure the evaluation tool is user-friendly.For the sake of argument, a hypothetical example of its potential use is presented in Box 1.

The way forward
This framework proposes a common approach to facilitate the evaluation and monitoring of interventions to increase access to health workers in underserved areas.It suggests that a comprehensive approach needs to be used for the design, implementation, monitoring, evaluation and review of such interventions.The framework is not intended to be prescriptive and can be applied flexibly to each country context.It is hoped that it will promote the use of a common understanding/logic on how attraction and retention interventions work, using a systems perspective.It starts with a common set of indicators, which enable comparison between various cases, and facilitates reviews of published studies.As such it can be used to monitor and evaluate interventions, either using a method-based or a theory-based evaluation approach with a specific set of indicators.■

‫ملخص‬ ‫النتائج‬ ‫لقياس‬ ‫مفاهيمي‬ ‫عمل‬ ‫إطار‬ ‫الخدمات:‬ ‫من‬ ‫املحرومة‬ ‫املناطق‬ ‫يف‬ ‫الصحيني‬ ‫العاملني‬ ‫إىل‬ ‫الوصول‬ ‫سبل‬ ‫زيادة‬ Box 1. A hypothetical use of the conceptual framework
The framework can be applied to a hypothetical scenario of country X.The evaluation will explore: (i) Does the intervention respond to the problem identified and has the intervention achieved the intended results?(ii) What improvements in intervention and policy design are needed, or are alternative strategies required?(iii) What indicators should be used to measure results?
Step 1: situation analysis For example, in country X, a situation analysis has identified a lack of health workers in rural areas.A study of intentions of young graduates to practise identified the unwillingness of health workers to go to those areas due to a lack of opportunities for professional development and low salaries.Thus, the main bottlenecks seem to be in initially attracting staff to rural areas.
Step 2: choice of interventions The intervention that has been implemented sought to "create more schools in rural areas" (education category) and to implement a "rural allowance scheme" (financial category) for some categories of health workers, and improve support and supervision (management).The intended result of this multi-level intervention was to increase attractiveness of rural practice for health professions and improve access to professional training to students with rural backgrounds, as well as an increased duration of stay of the categories of health workers covered by the scheme.
Step 3: measuring results using a selection of country-appropriate indicators The indicators chosen for monitoring progress and measuring the direct effects ("outputs") of the intervention would be: "intentions to go to rural areas after graduation", "number of applicants from rural background to the newly developed schools compared to urban", "number of graduates choosing to work in rural areas", "duration of stay in post of the health workers covered by the scheme".The indicators to measure the indirect effects ("outcomes") of the intervention would be: "changes in staff ratio within 3 years", "increase in outpatient visits" before and after, "changes in staff satisfaction", "user satisfaction" perceptions of community representatives of service responsiveness to need.The outputs and outcomes can be related statistically to take-up of training programmes, to the percentage of staff covered by financial incentive schemes and supported by an experienced manager, to establish correlation and formulate plausible explanations for the effect of the different types of incentives on the way the intervention has performed.Eventually, the final impact of the intervention can be estimated by improved "health outcomes", bearing in mind that these changes cannot entirely be attributed to the intervention itself.
Step 4: monitoring Analysis of longitudinal data from indicators to examine how the situation has changed, especially seeking to identify unintended consequences of implementing the intervention.For example, attracting staff in rural areas may have resulted in their recruitment by nongovernmental organizations and private facilities rather than working in the public sector.Resumen Ampliar el acceso a los trabajadores sanitarios en las zonas subatendidas: marco conceptual para medir los resultados Muchos países han desarrollado estrategias para atraer y conservar a personal sanitario cualificado en zonas insuficientemente atendidas, pero la evidencia sobre el éxito o fracaso de esas iniciativas es escasa y poco robusta.Es difícil comparar las enseñanzas extraídas y cuantificar los resultados de las escasas evaluaciones disponibles.Las medidas de evaluación tropiezan con varios problemas, en particular la heterogeneidad de la terminología, la complejidad de las intervenciones, las dificultades para evaluar la influencia de factores contextuales, la falta de información basal, y la necesidad de aplicar enfoques multimetódicos y multidisciplinarios en la vigilancia y la evaluación.Por otra parte, en la vigilancia y evaluación de las intervenciones relacionadas con los recursos humanos para la salud rara vez se tiene en cuenta el contexto social, político y económico en el que se diseñan y aplican esas intervenciones.En este artículo se propone un marco conceptual que brinda un modelo de vigilancia y evaluación de las intervenciones de promoción de la permanencia que sí tiene en cuenta esos factores.Dicho marco está basado en un enfoque sistémico y pretende servir de orientación a la hora de evaluar una intervención de ampliación del acceso al personal sanitario en las zonas subatendidas, desde su fase de diseño hasta sus resultados.También tiene por objeto orientar la vigilancia de las intervenciones mediante la recopilación sistemática de un conjunto de indicadores, aplicables al contexto específico.Se sugiere que es necesario aplicar un enfoque omnicomprensivo al diseño, ejecución, vigilancia, evaluación y examen de las intervenciones.El marco no pretende ser prescriptivo y se puede aplicar con flexibilidad en el contexto de cada país.Promueve el uso de una perspectiva común sobre el funcionamiento de las intervenciones de fomento de la captación y la permanencia, aplicando un enfoque sistémico.

Fig. 1 .
Fig. 1.The conceptual framework for measuring efforts to increase access to health workers in underserved areas

Fig. 2 .
Fig. 2. The adapted Human Resources for Health Action Framework 20
conceptual framework for measuring efforts to increase access to health workers in underserved areas

Table 1 .
11mensions and indicators to assess health workforce performance11