Making products available among community health workers: Evidence for improving community health supply chains from Ethiopia, Malawi, and Rwanda

Background A UNICEF review of the challenges to scaling up integrated community case management (iCCM) found that drug shortages were a common bottleneck. In many settings, little thought has gone into the design of supply chains to the community level and limited evidence exists for how to address these unique challenges. SC4CCM’s purpose was to conduct intervention research to identify proven, simple, affordable solutions that address the unique supply chain challenges faced by CHWs and to demonstrate that supply chain constraints at the community level can be overcome. Methods SC4CCM selected three countries to implement supply chain innovations and developed a theory of change (TOC) framework for the learning phase, which identified the main drivers of product availability and was used for baseline assessments, design, implementation and evaluation of interventions in Ethiopia, Malawi, and Rwanda. Interventions were developed in each country and tested over 12–24 months. Mixed–method follow up assessments were conducted in each country in 2012–2013. The Supply Chain for Community Case Management (SC4CCM) Project then simplified the TOC into a Community Health Supply Chain (CHSC) framework to enable cross country analysis Results The findings from interventions in the three countries suggest that the greatest supply chain benefits are realized when all three CHSC framework elements (data flow, product flow, and effective people) are in place and working together. The synergistic effect of these three elements on supply chain performance was most effectively demonstrated by results from the Enhanced Management and Quality Collaborative interventions in Malawi and Rwanda, respectively, which were characterized by lower mean stockout rates and higher in stock rates on day of visit, when compared to other interventions. Conclusions Many conditions are necessary to ensure continuous product availability at the community level, however a supply chain works best when three key elements (product flow, data flow, and effective people) are deliberately included as an integral part of the system design. Although these elements may be designed differently in different settings, streamlining and synchronizing them while ensuring inclusion of all components for each element improves supply chain performance and promotes product availability at the community level.

Background A UNICEF review of the challenges to scaling up integrated community case management (iCCM) found that drug shortages were a common bottleneck. In many settings, little thought has gone into the design of supply chains to the community level and limited evidence exists for how to address these unique challenges. SC4CCM' s purpose was to conduct intervention research to identify proven, simple, affordable solutions that address the unique supply chain challenges faced by CHWs and to demonstrate that supply chain constraints at the community level can be overcome.
Methods SC4CCM selected three countries to implement supply chain innovations and developed a theory of change (TOC) framework for the learning phase, which identified the main drivers of product availability and was used for baseline assessments, design, implementation and evaluation of interventions in Ethiopia, Malawi, and Rwanda. Interventions were developed in each country and tested over 12-24 months. Mixed-method follow up assessments were conducted in each country in 2012-2013. The Supply Chain for Community Case Management (SC4CCM) Project then simplified the TOC into a Community Health Supply Chain (CHSC) framework to enable cross country analysis

Results
The findings from interventions in the three countries suggest that the greatest supply chain benefits are realized when all three CHSC framework elements (product flow, data flow, and effective people) are in place and working together. The synergistic effect of these three elements on supply chain performance was most effectively demonstrated by results from the Enhanced Management and Quality Collaborative interventions in Malawi and Rwanda, respectively, which were characterized by lower mean stockout rates and higher in stock rates on day of visit, when compared to other interventions.
Conclusions Many conditions are necessary to ensure continuous product availability at the community level, however a supply chain works best when three key elements (product flow, data flow, and effective people) are deliberately included as an integral part of the system design. Although these elements may be designed differently in different settings, streamlining and synchronizing them while ensuring inclusion of all components for each element improves supply chain performance and promotes product availability at the community level.
A UNICEF review of the challenges to scaling up integrated community case management (iCCM) conducted in six countries found that drug shortages were one of the most frequently reported bottlenecks and were evident during the implementation and scale-up stages of iCCM [1]. Low or no product availability has even been shown to lead to delays in implementation of iCCM. One of the main conclusions of the March 2014 iCCM Evidence Review Symposium was the need to reduce stock outs in order to increase uptake of iCCM services [2].
Public health supply chains, of which the community is part, generally face chronic challenges in the areas of human resource capacity and skills, general management/ management of processes, communication between levels, budget planning, physical infrastructure and capabilities and resources (including storage and distribution capacity), availability and use of data for management decisions, commitment and motivation, and accountability [3]. Each of these elements plays a role in disrupting the availability of essential medicines throughout the supply chain.
These problems are magnified at the community level, as a result of the unique challenges faced by community health workers (CHWs). CHW programs vary widely, but CHWs generally work in remote, rural locations characterized by difficult geographies. Transit to resupply points can be long and difficult and CHWs typically have limited transportation options, given the terrain; often they are forced to use non-motorized forms of transportation such as bikes, donkeys, camels, mules, boats, and even foot [4][5][6]. Public transport is uncommon and costly. Often, CHWs are not highly literate-which can cause challenges around recording, reporting, and submitting data-and often have no dedicated facility to work from. Medicines are often stored in drug boxes along with paperwork, and storage space is limited, potentially compromising the quality and security of product storage. CHWs in many countries are unpaid [7], increasing the need for motivation of these workers, especially with regard to supply chain (SC) tasks, which are often seen as tedious, time consuming, and burdensome. Finally, given that CHWs are at the end, or "last mile" of the supply chain, they have no platform for advocacy, so when shortages of essential medicines occur in the system, CHWs tend to miss out on supplies.
In many settings, little thought has gone into the design of supply chains to the community level-community health supply chains have not been deliberately designed to address the unique circumstances of CHWs [8]. Furthermore, limited evidence exists for how to address these unique challenges with a view to improving community health supply chain performance and product availability. The Supply Chain for Community Case Management (SC4CCM) Project' s purpose was to conduct intervention research to identify proven, simple, affordable solutions that address the unique supply chain challenges faced by CHWs and to demonstrate that supply chain constraints at the community level can be overcome. SC4CCM' s mandate was limited to strengthening the community level of the supply chain and did not include funding for commodity procurement. The project' s intent was to gather evidence on "game changing" interventions for ensuring product availability among CHWs, with the goal of helping countries achieve Millennium Development Goal (MDG) 4: Reduce Child Mortality. This paper presents evidence from community health supply chain innovations implemented in the three project countries that confirm product flow, data flow and effective people as elements that need to be deliberately incorporated into design and which need to work together to effectively improve supply chain performance and the availability of life-saving medicines among CHWs.  relationship between the TOC preconditions and the CHSC Framework is presented in Figure 2. The CHSC Framework enabled cross-country analysis as well as a demonstration of the interdependency of the three basic elements in enhancing supply chain effectiveness-vital information going into the scale-up phase.
Product flow describes how CHWs are resupplied-for example, using a demand-based system or a fixed-quantity supply-and requires clear procedures and processes for inventory management, distribution, and storage.
Data flow ensures that logistics (consumption and stock level) data are available and usable for supply chain decision making, management, monitoring, and problem solving. Data flow solutions incorporate mechanisms to capture logistics data at the lowest levels of the system and transmit it in a disaggregated form so that it can be useful for management and decision making. Data flow and product flow are interconnected as the correct data must be collected and visible to the right people to inform product flow.
Effective people refers to the workforce involved in making sure product flow and data flow happen; effective people ensure continuous use and improvement of SC skills and practices at the lower levels of the system, build district leadership and ownership for tackling community health SC problems rather than waiting for solutions from higherlevel managers, and recognize CHW achievements.
Effective people encompasses: • Management processes and skills, including clear standard operating procedures (SOPs), roles and responsibilities, and provision of SC training/skill and knowledge building, • Teamwork, using a formal structure across multiple levels and/or tools to facilitate group problem solving toward common objectives, • Motivation and recognition of CHWs for SC accomplishments. CHWs take on their responsibilities because they want to serve their community, but often they will need to be motivated to take on SC tasks.
Effective national-level coordination and routine quantification are fundamental keystones for continuous product availability by ensuring funding for and the timely procurement and distribution of medicines. Well-functioning systems include routine mechanisms for quantification, regular updates to forecasts and supply plans, and close coordination between Ministry of Health (MOH) programs, donors, quantification teams, and procurement units to ensure continuous product availability and to maximize the use of limited resources. In particular, routine quantification and coordination are necessary for ensuring continuous availability of products at the resupply points for CHWs, which in turn is a prerequisite for ensuring that CHWs have products. While we present evidence that product flow, data flow, and effective people will help programs maximize community health supply chain performance, quantification and coordination ensure that there are products available to flow through the supply chain so that performance can be improved.
Our presentation of results in this paper will focus mainly on the evidence related to the CHSC Framework, since the project did not have a mandate to fully participate in all aspects of quantification and procurement, thus limiting our available evidence.
Intervention design using theory of change framework and data. The baseline surveys showed poor combined availability of products required to provide iCCM services among CHWs on the day of visit (DOV) in all countries [8]. Table 2 shows country findings at baseline. Countryspecific results from the baseline surveys were organized according to the original TOC to analyze bottlenecks in the supply chain affecting product availability at the community level. After assembling preliminary results, the project presented findings to in-country stakeholders and repre-

Product flow
Demand-based resupply but using non-standardized forms and data not consistently used for resupply: • 56% of HC staff determined resupply quantities using a standard formula, though 10% used the same quantity as last month, 5% used knowledge from past experience, 5% used another method, and 23% did not know.
Transportation is a constraint for CHWs in collecting products: • 18% of CHWs identified a transport related challenge as their number one challenge with collecting and receiving supplies The problems included "it was too long to reach the resupply point," "there was no transport available," "the transport was always broken" and "difficulties carrying supplies." Unstructured approach with no defined rules or process to drive resupply: • 62% of HCs resupplied based on (non-standard) documentation; 19% of HCs used a variety of ("other") methods; 8% of HCs provided the same as last month; 7% and 4% of HCs "didn't know" or used a formula, respectively Transitioning to a demand-based system, Integrated Pharmaceutical Logistics System (IPLS), but using fixed-quantity supply (kits): • More than 50% of CHWs reported submitting requests when stock runs low or when they stock out • 66% of CHWs report getting their health products from the HC, 44% report getting from the district health office

Data flow
Despite the existence for SC procedures, visibility of CHW logistics data was poor at higher levels: • 43% of CHWs reported to HCs using a standard form • 55% of HC staff across ten districts (n = 73) reported HSA supply chain data up to district level, and 14% reported this data disaggregated from HC data Misaligned reporting system, where data flow did not support decision making: • 97% of CHWs received products from HCs, but only 54% of CHWs submitted logistics data to HCs Due to lack of training and kit system, CHWs were not using the manual IPLS reporting system for iCCM products: • CHWs mentioned 6-7 different reports that they submitted regularly with no single report having more than 30% of HEWs using them. ACT -artemisin-based combination therapy, CHW -community health worker, COC -combined oral contraceptive, DMPA -Depo Provera, DOVday of visit, FP -family planning, HC -health center, HC -health center, HSA -health surveillance assistant, IPLS -Integrated Pharmaceutical Logistics System, LMIS -logistics management information system, ORS = oral rehydration solution, RUTF -ready-to-use therapeutic food, SC -supply chain sentatives of all levels of the supply chain in a series of participatory data validation workshops, which served to both validate results and obtain inputs for designing intervention packages. In all countries, data supported the possibility of testing more than one approach to improving outcomes, but also indicated that intervention packages required a two-phase approach to lay a strong foundation in supply chain knowledge, skills, and procedures before implementing "value-added" innovations. Intervention packages are shown in Table 3.The testing period ranged from 12 to 24 months and was characterized by regular monitoring to guide intervention support and adjust the intervention design toward achieving the respective objectives.
1. Malawi. At baseline, Malawi was in the process of implementing a demand-based resupply system; however, data was not visible at all levels of the system with only 55% of health center staff across ten districts reporting CHW supply chain data up to district level, and only 14% reported this data disaggregated from health center data; therefore CHW-specific supply chain data was not available for management decision making or performance monitoring at higher levels of the system. Reporting rates were low and few data were available to district managers for identifying and resolving stock outs or other management issues. Given that 89% of the CHWs surveyed at the 2010 baseline had mobile phones, and network coverage was high, the project developed a simple SMS and web-accessible reporting and resupply system, cStock. cStock was intended to improve the resupply process by enhancing communication between CHWs and their resupply points, to facilitate visibility of real-time CHW logistics data at district and central levels, and to enable supply chain managers to respond immediately to performance or product availability issues. The design of cStock mirrors processes ACT -artemisin-based combination therapy, CHW -community health worker, EM -enhanced management, DMPA -depo provera/depot medroxyprogesterone acetate, DPAT -district product availability teams, EPT -efficient product transport, FP -family planning, IcSCI -incentives for community supply chain improvement, mHealth -mobile health, ORS -oral rehydration solution, QIT -Quality Improvement Team, QC -Quality Collaboratives, RSP -standar resupply procedures, RUTF -ready-to-use therapeutic food, SC -suplly chain for the demand-based resupply system while improving data visibility through better data flow for operations and management, and improving product flow using a streamlined resupply process. cStock was combined with two different approaches (Enhanced Management [EM] and Efficient Product Transport [EPT]), that were tested side by side in three districts each. EM addresses all three framework elements by combining cStock, which addresses product flow and data flow, with the establishment of District Product Availability Teams (DPATs), which aimed to improve the effectiveness of the people by promoting team performance practices through the use of data to inform decisions and improve supply chain performance. DPATs comprise district management, health center staff, and CHWs who have a shared vision and collective commitment to ensuring continuous availability of products through use of data for continuous improvement and recognition of good CHW performance.
The EPT intervention only aimed to address two of the three framework elements, namely product flow and data flow, and did not address the element of effective people. Transport was a big challenge identified at baseline; although the MOH provides all CHWs with bicycles, breakdowns were frequent, reducing CHWs' ability to collect supplies regularly. In addition to cStock, EPT introduced two approaches to improve product flow. First, a continuous-review inventory control system that allowed CHWs to make more frequent trips to collect smaller amounts of supplies during their scheduled visits to health facilities and reducing the need for them to make special trips for product pickup. Second, EPT trained CHWs in regular, preventive bicycle maintenance to reduce breakdowns and repairs needed to keep the bicycles functioning.
Malawi conducted an annual quantification that included the iCCM program. Quantification for iCCM is complicated, however, by the fact that all of the products used by CHWs in Malawi are also used at higher levels of the health system or by other programs, requiring the input of good quality data from all programs and levels to develop a robust iCCM program forecast and supply plan. Further, donor support to CHWs in Malawi often targeted individual districts. Thus distribution data and data on products used by level and program were not always available. These circumstances made coordination and monitoring overall stock levels, other than the community level through cStock, difficult.

Rwanda.
Baseline results demonstrated that CHWs were not resupplied according to any rules. The foundational intervention, therefore, was to establish a demand-based resupply system, called Standard Resupply Procedures (RSPs) for CHWs. The processes targeted the Cell Coordinator (CC) as the primary actor to collect and aggregate data from CHWs in their cell and resupply them with prod-ucts, to increase efficiency at the health center level. If scaled nationally, RSPs would result in monthly reports for approximately 2150 CCs rather than 30 000 individual CHW reports. The intervention required the use of three basic tools: CHW stock cards to capture consumption data and stock data, a simple tool to calculate resupply quantities ("the magic calculator"), and a resupply worksheet (RSW) that CCs use to aggregate data for all CHWs in their cell each month.
The RSPs were implemented in six test districts and ensured sufficient and appropriate data flow for operations and SOPs as part of the first step toward developing effective people. The hypothesis was that designing supply chain processes and imparting skills were necessary first steps but not sufficient alone to significantly improve product availability. Thus, once RSPs were implemented, and the foundation for product flow and data flow was established, two different strategies (Quality Collaboratives [QCs] and Incentives for Community Supply Chain Improvement [IcSCI]) were tested side by side; both aimed at making CHW, health center and district staff engaged in supply chain tasks more effective and improving product availability.
The QC approach, previously used successfully to solve bottlenecks in clinical work (9], involves establishing and training Quality Improvement Teams (QITs) at health centers to find solutions for operationalizing the new resupply procedures at the CHW level. The aim of the QCs, or networks of QITs, is to close the gap between desired and actual performance by using data to target and address problems, and then developing, testing/implementing, and spreading changes quickly across many teams and/or organizations. QITs brought CCs, health center, and district staff together as a team to look at data on SC performance collected by CCs on supervision checklists; problems were identified and prioritized; action plans were developed and progress tracked. The project worked closely with the MOH to ensure the QITs functioned as expected during the testing period.
In contrast, IcSCI aims to strengthen the commodity supply chain by adding supply chain related-indicators to Rwanda' s existing community performance-based financing scheme for CHWs, which targets improvements in delivery of health services at the village level. IcSCI provides an incentive package that specifically rewards CHWs for improved performance of supply chain tasks linked to nine supply chain indicators by providing monetary incentives to CHWs through their community cooperatives based on quarterly performance scores. In essence, both QCs and IcSCI targeted all three elements of the framework, namely product flow, data flow and effective people, although QCs addressed effective people more comprehensively than IcSCI by including a formal teamwork component, which was more informal and indirect in the incentives approach.
In Rwanda, the community health desk within the MOH plays a strong role in coordination, ensuring that quantification in collaboration with different programs (Malaria, MCH, etc) takes place annually and supply plans are monitored regularly, as well as providing funding for product procurement for iCCM and working with the national procurement unit (Medical Procurement and Production Division) to coordinate procurement of products, many of which are used exclusively at the community level.

Ethiopia.
Baseline results showed that CHWs and health centers were ineffective in managing health products because they lacked SC knowledge and skills. The Federal Ministry of Health (FMOH) and donors were supplying iCCM products to CHWs using a fixed-quantity supply (FQS) method: kits. The national logistics system in Ethiopia is transitioning to a new demand-based supply chain system (the Integrated Pharmaceutical Logistics System, or IPLS); hence, SOPs existed for all levels, although they had not been fully implemented at any level of the system. The IPLS outlines how data and products should flow between the levels of the supply chain, so the project' s priority was to test a way of rapidly and affordably building foundational CHW supply chain knowledge and skills around the SOPs for IPLS, as a first step toward addressing the effective people element, with the expectation that the approach could be scaled up to all 30 000 plus CHWs, who could then move away from fixed quantity supply to a demandbased system. Three different approaches were taken to implement the training using existing activities at health centers as opportunities to impart SC knowledge and skills. Two approaches used a group training method during monthly meetings at the health center, and one approach used one-on-one training or on the-job-training (implemented by another project and called the comparison group) at the time HEWs came to collect products. In the two groups that used the group training method, one group received follow up support (intensive group) and the other received no additional support (non-intensive group). Key supply chain skills for CHWs were distilled into five one-hour "Ready Lessons" that could be administered in any order and/or repeatedly; these lessons were combined with supply chain problem solving to address bottlenecks and identify gaps in skills that needed to be addressed. While recognition/motivation is a key element of "effective people," this element was deliberately excluded from the design since "Ready Lessons" were intended to be administered during pre-existing meetings that already had HEW recognition on the agenda. Because of Ethiopia' s vast geography and large numbers of CHWs, the 2010-2013 period was spent implementing and testing the foundational in-tervention, with the "added-value" intervention planned for 2013-2015 (not included in this paper). Supply chain knowledge and skills are critical prerequisites for operationalizing IPLS; however, training is necessary but not sufficient for developing "effective people" or for significantly improving product availability. Therefore, significant improvements in product availability and other key supply chain indicators were not expected from this intervention.
Ethiopia conducted regular quantification for iCCM but faced additional challenges with coordination because of the kit system, as six months of supply for each site had to be available in fixed quantities for kitting centrally before distribution. Given the different funding and procurement cycles of the government and various donors, the required level of coordination was difficult to achieve. Additionally, as the quantities in the kits were based on initial estimates of need that were not revised in light of actual consumption patterns, CHWs ran out of some items rapidly while others lasted much longer than anticipated.

METHOdS
Using the TOC as the guiding evaluation framework, the project conducted baseline and follow up assessments in select areas of the three countries in 2010 and 2012-2013, respectively, using complementary quantitative and qualitative methods. The quantitative survey tool, called the Logistics Indicator Assessment Tool (LIAT), was adapted from tools originally developed by the USAID | DELIVER PROJ-ECT, including questionnaires, inventory assessment forms, storage assessment forms, and key informant interview guides [9,10]. The survey was tailored to each level of the supply chain, from central medical stores down to the community level, to capture processes, behaviors, and product availability along each step in the chain, and to measure indicators of intervention implementation. Tools were field-tested and adapted for each country setting [11]. Permissions for the assessments were obtained from all relevant MOH partners and institutional review board (IRB) approval was obtained in Malawi and Rwanda, where it was required.
Survey samples were not intended to be nationally representative, but rather chosen to first diagnose major iCCM supply chain strengths and weaknesses in a cross-section of districts served by key iCCM partners, and then to follow the supply chain from the central level to the community level. Purposeful selection at the district level was done based on existence of a functioning iCCM program, geographic variation, and balance of iCCM partner support. Probability proportional to size sampling was used to randomly select health facilities and CHWs at the lower levels of the supply chain. In all countries, CHWs were the unit of analysis. Table 4 shows the full details on survey dates, sample sizes, and levels of the supply chain visited by country surveys.
Quantitative data for both surveys and all three countries were collected by local evaluation partners, all selected through competitive processes. Enumerators were trained to interview CHWs and other staff managing supplies of medicines, conduct product inventories, and rate storage conditions. Data collectors used Nokia e71 and e63 smartphones loaded with DataDyne' s Magpi application, which allowed for streamlined data entry and immediate review of data after uploading records to a web-based system. Qualitative methods were also employed for deeper under-standing of user experiences, but this paper focuses primarily on quantitative results.
In Malawi and Rwanda, supplemental data sources were also considered, including routine data collected through cStock and the IcSCI indicators database respectively. In Malawi, routine logistics monitoring data submitted by CHWs using cStock were utilized to study inventory trends over time between the EM and EPT groups. The web-based cStock dashboard provided reports showing monthly stock reporting rates, average time taken to restock the drugs (lead time), product availability, and stock outs for these time periods for the six intervention districts (three for the  Overall LIAT sample: Districts FU (BL) 10 (10)  EM group and three for the EPT group). In Rwanda, the project, over the intervention period maintained quarterly performance scores for the nine supply chain indicators, submitted by health centers, for the IcSCI group (three districts) in the IcSCI indicators database.

Study Groups and DiD
After the formative assessments in each country in 2010, the project formed three groups from original baseline evaluation areas by matching geographical and demographic characteristics, and other external dimensions including iCCM partner coverage, prevalence of diarrhea, malaria, and cough, as well as baseline CHW iCCM product availability, to create comparable groups. Two of the three groups were randomly assigned a unique intervention, while the third group was assigned as the comparison group. For Malawi and Rwanda, this division of areas was designed to facilitate a difference in difference (DiD) analysis to calculate the effect of the interventions on a key supply chain indicator (CHW product availability) by comparing the average change over time in this indicator for the intervention group to the average change over time for the comparison group. Table 4 provides further details on and division of areas into intervention and comparison groups. In Malawi and Rwanda, a DiD regression analysis was conducted using iCCM product availability as the outcome variable. Because of the extraordinary challenges related to parallel supply chains in Malawi, the DiD results were inconclusive and further analyses were conducted to determine the effect of the interventions on supply chain performance and product availability. In Rwanda, the DiD analysis attempted to control for factors that may have affected the product availability over time indicator, including formal training of CHWs on the management of medicines and health products, training of CHWs in pneumonia, malaria, or diarrhea, and CHWs reporting transport obstacles in getting to their resupply point ( Table 7). Limitations to this model include the real possibility that outside factors, applied unequally between groups over the three year period between baseline and follow up surveys, also caused changes, reducing the ability to attribute changes to the interventions alone.

Country analyses
For both baseline and follow up surveys, frequencies and cross-tabulations were carried out using SPSS 18 and STA-TA version 11. Analyses were conducted using pathways identified in each country-specific TOC [12][13][14]. Indicators associated with the precondition pathways were laid out to determine progress along the pathway of change, both to validate the TOC as well as identify where obstacles may have prevented achievement of outcomes.
For Malawi, cStock data were retrieved for the 18-month period from January 2012 to June 2013 and average values of key supply chain indicators were calculated. Paired Student' s t-tests were conducted to compare the trends between the EM and EPT groups. For data from the Rwanda IcSCI indicators database, Pearson chi squared tests were run to determine whether there was a significant difference in the performance of CHWs on select indicators by district over the four quarters of the testing period.

Cross-country analysis
Following completion of follow up surveys in all countries, the project partnered with Accenture Development Partners (ADP) to develop a practical framework to facilitate cross-country analysis and synthesis of intervention findings using a broader lens. SC4CCM and ADP refined the project and country TOCs into the CHSC framework to capture important and consistent results from each country evaluation. Results related to each intervention package were categorized by product flow, data flow, and effective people and interpreted with a view to determining the effectiveness of each package.
Effectiveness was defined as achieving the intended or desired outcome of the intervention. The ultimate goal of any supply chain is improved product availability-to ensure that the service delivery point, in this case the CHW, has usable and quality medicines available to serve clients when needed. However, product availability is influenced by numerous factors, as shown by the TOC. Although supply chain performance is a critical factor, the greatest prerequisite is having products flowing through the national supply chain; supply chain performance is irrelevant if there are no products. The expected outcome for interventions in Malawi and Rwanda consisted of improvements in community supply chain performance, which we hypothesized, would lead to improvements in product availability. Improved supply reliability, defined as reductions in stockout rates, was used as an alternate outcome measure in Malawi due to limitations associated with attributing improvements in product availability to project efforts. Indicators for supply chain performance varied in each country, given the data available. In Ethiopia, the expected outcome was an improvement in supply chain competencies, leading to improvements in key supply chain practices.

RESULTS
Follow up results for Malawi and Rwanda are presented first according to the CHSC Framework elements of product flow, data flow, and effective people, and then the results of intervention packages as a whole are compared to understand how the combination of the different elements affected the supply chain performance indicators and product availability. In Ethiopia, results are presented based on the aspect of the effective people element only. Data source is the LIAT survey unless otherwise noted, but results draw from focus group discussions (FGDs), cStock dashboard reports and the IcSCI indicators database.

Malawi
All aspects of the EM intervention were fully implemented, while only the cStock component of EPT was implemented -the continuous review inventory management system was not implemented, with users finding it burdensome, and neither was regular practice of preventive bicycle maintenance by trained CHWs. Since EPT was not designed to address the effective people element, and its product flow design was unchanged, the project used the comparison of key supply chain performance indicators between EM and EPT groups to show the added value of the effective people (DPAT) component to product and data flow (cStock) in the EM group. Table 5 summarizes key follow up results for Malawi.
Product Flow. Follow up evaluation results show that inventory management was streamlined and standardized product flow in both EM and EPT groups, largely due to cStock, compared to the non-intervention (NI) group, where less consistency was shown in forms used to request and resupply. Ninety-eight percent of CHWs in the EM group and 91% in the EPT group reported using cStock for requesting health products, compared to NI, where 48% of CHWs reported using Form 1A, 34% use a request form, 9% use LMIS-01G, and 14% use another form (multiple responses allowed). Additionally, 92% of Drug Store in-Charges in the EM group and 91% in the EPT group reported using cStock to determine quantities to resupply CHWs. The NI group reported using various resupply mechanisms, with 48% of Drug Store in-Charges using Form 1A, 17% using LMIS 01G, 10% reporting that they "issue standard amount," 10% "give as much as I have available," and 24% reporting "other."

Data flow.
Results also showed improvements in the reporting and visibility of community-level stock logistics data. At baseline, the average CHW reporting rate using a standard form across the ten districts was 43%. Data visibility at higher levels also improved, with logistics data from all cStock-reporting CHWs (94% in EM and 79% in EPT) accessible at district-level through the dashboard. At baseline, while 55% of health center staff reported CHW supply chain data up to district level, only 14% reported sending disaggregated CHW logistics reports to a higher level, resulting in very limited visibility of community data for district level decision makers.
Effective people. Follow up findings show a high frequency of DPAT meetings in the EM group and evidence that data was used to monitor and improve supply chain performance and recognize CHW achievements. Eighty-four percent of CHW Supervisors reported that district-level DPAT meetings were held and 96% of CHW Supervisors reported conducting a DPAT meeting at the health center level. Health center staff monitored the performance of the community supply chain using cStock data, with the majority using either reports pulled from cStock (56%) or resupply worksheets (40%) where cStock transactions are recorded. FGDs highlight how DPATs meetings were used to improve key supply chain performance indicators and product availability; as one CHW explained, "We talk about our reporting rate and how best to improve it, the products." Another CHW Supervisor shared that they, "...discuss the over-stocking or under-stocking and we discuss how we can share the drugs." FGDs highlighted the benefits of the DPATs in improving communication and team work, as described by one CHW, "We also discuss and encourage teamwork among the medical assistant and us to work together, because when we send stock on hand, we depend on them to respond all the time, and that has enhanced our communication and team work." CHWs also described how DPATs motivated them; one CHW shared "sometimes when we are in our meeting the medical assistant compliments one of the CHWs and when he does so, we are motivated as well to perform better so that we can be complimented."

Comparing intervention packages.
Comparing the results of EM and EPT in Table 5 for three key supply chain performance indicators (lead times, reporting rates, and completeness of reporting) demonstrates how the EM group outperformed the EPT group suggesting the difference was likely due to the DPAT -effective people component of EM. These indicators were calculated using cStock data for the period of January 2012 to June 2013 and Ttests yielded significant differences (P < 0.001) for all three indicators.
Product availability for four tracer iCCM products on the day of visit more than doubled, increasing from 27% of all CHWs with products at baseline to 64% in EM, 59% in EPT, and 63% in NI at follow up. However, due to the presence of a number of parallel supply chains bypassing the government supply chain to deliver directly to health facilities or CHWs over the testing period, it was not possible to isolate the impact of our interventions from that of the donor-supported drug distribution for this indicator. Therefore, given that product availability data could not be used to evaluate program effectiveness, the project measured supply reliability by comparing stock out rates in cStock over the period of the intervention between EM and EPT groups. EM stock out rates were consistently lower  (below 10%) than those for EPT for all six products over the period January 2012-June 2013. Stock out results are presented in Figure 3 as a measure of mean percent CHW stock out rates by product. These differences were statistically significant at the P < 0.001 level for all products. Results suggest higher levels of supply reliability in the EM intervention than the EPT intervention.

Rwanda
Results show that all aspects of the RSPs, QCs, and IcSCI interventions in Rwanda were well implemented. The design and implementation of RSPs was meant to set the foundation for good product flow and data flow processes for the community level and rationalize the movement of people, data, and medicines. However, because the comparison group had no equivalent role to Cell Coordinator, it is not possible to evaluate improved product and data flow due to RSPs in the intervention groups vs the NI group. Table 6 provides a summary of key follow up survey results for Rwanda.
Product flow. The RSP intervention was meant to shift responsibility for product collection away from CHWs to the CCs, while streamlining resupply by reducing inefficiencies in travel and congestion at health center pharmacies. Results presented in Table 6 suggest improvements in product flow, with 100% of CCs in QC and 91% in IcSCI reporting that they picked up products for all CHWs in their cell after every monthly meeting and 95% of CHWs in QC and 93% in IcSCI reporting that they received products from CCs. Looking across groups at follow up, significantly more CHWs from both intervention groups (93% for QC and 93% for IcSCI) reported regularly receiving medicines and health products to treat sick children, compared with the non-intervention group (85%; P < 0.05). Data from the IcSCI indicators database showed that the percentage of CCs who collected needed products for their cell after the HC meeting was 96% during the last quarter of the testing period.
Qualitative findings also supported an improvement in product flow.     BL -base-line, CHW -community health worker, DiD -Difference in Differences, DOV -day of visit, FU -follow up, IcSCI -Incentives for Community Supply Chain Improvement, QC -Quality Collaboratives *Controls: CHW has formal training on how to manage medicines and health products, CHW has training in pneumonia, malaria, or diarrhea, CHW has obstacles to transport †DiD is calculated as DID = (Intervention Follow up% -Intervention Baseline%) -(Comparison Follow up% -Comparison Baseline%). Results displayed represent two steps in the analysis of the data: the significance, denoted by the stars, represents the results from the multivariate logistic regression on the time-group interaction variable, which is the key independent variable of a DiD regression. Since the interaction coefficient is non-intuitive, we have instead depicted the difference over time between the intervention and non-intervention groups using the predicted probabilities resulting from the regression. Essentially, this is the net percentage point change in the intervention region once the comparison group change is subtracted. ‡P < 0.001.

challenges. This allowed us to learn from those who had faced a similar challenge in the past and how they solved it."
In the IcSCI group, three of nine incentives indicators showed strong evidence of significant SC improvements between Q1 and Q2, and continued high performance across the group over the remaining three quarters of the test period, as would be expected for an effective incentive scheme. Results from the IcSCI database show the proportion of CHWs with stock cards for iCCM products where physical inventory matches stock card balance for all on the DOV, increased from 86% in Q1 to 96% in Q4 (n ranges 3157-3201 CHWs visited in the three intervention dis-tricts each quarter, over four quarters). We found an improvement of 7.03% between Q1 and Q2 for this indicator across the three districts, P < 0.001, 95% CI [5.5-8.57%]. The proportion of CCs who presented complete RSWs without any calculation errors during monthly health center meetings, in the past quarter, rose from 77% in Q1 to 98% in Q4 (n same as above). Results from the three districts show a 13.43% difference between Q1-2, P < 0.001, 95% CI [8.8-18.2%]. The proportion of CHWs who have at least one treatment for a five-year-old child in stock, for each iCCM product on the DOV, rose from 79% in Q1 to 92% in Q4 (n same as above). Results from the three districts  show a 14.06% difference for this indicator between Q1-2, P < 0.001, 95% CI [12.9-15.2%].
Comparing Intervention Packages. In comparing the results of QCs and IcSCIs, it is possible to consider the additional value of the teamwork component of the effective people element. While QCs performed slightly better than IcSCIs in key supply chain performance indicators (eg, reporting completeness, stock card accuracy, and sixmonth stock out rates) there were no significant differences, and both performed better than the NI group. However, there were differences in the overall impact on product availability.
The follow up survey found significantly greater availability among CHWs of all five iCCM products in stock on the DOV in the QC group (63%) compared to NI group (38%; P < 0.001), and non-significantly greater availability in the IcSCI group (45%) compared to NI group (38%). A significant decline was detected since baseline for this measure in the NI group (from 58% to 38%; P < 0.01).
Further analysis of in-stock data using DiD analysis showed a highly significant improvement (P < 0.001) in the QCs group compared with the NI group for the key composite indicator of all five iCCM products in stock on DOV. The DiD detected significant improvements in availability for all products individually (P values range from <0.05 to <0.01 for ORS, zinc, and ACT 1 × 6), with the exception of amoxicillin and ACT 2 × 6. In the IcSCI group, a significant result was detected only for one product, ACT 1 × 6 (P < 0.05), but no results for other individual products or the composite indicator.

Ethiopia
The supply chain Ready Lessons and Problem Solving approach was not implemented exactly as designed; however it still proved to be a rapid, affordable and effective way to build a foundation in supply chain knowledge and skills for CHWs. As previously mentioned, the project' s activities only targeted the effective people element, and only laid the foundation for two components of this element -management and teamwork. As this was primarily a training intervention, the aim was not to measure product availability but to determine whether the training led to competency, setting a foundation for improved practices supporting product flow and data flow. Therefore the follow up survey focused on whether health center staff could train CHWs opportunistically and affordably and in a way that built skills. Survey results are largely limited to coverage and competency of the CHWs six months after the Ready Lessons were introduced. However other supply chain indicators are provided in Table 8 but the results are mixed and not always a reflection of the intervention. Further interventions are being tested to determine what is required to fully achieve all three elements and improve product availability significantly (results are expected in late 2014). Table 8 summarizes follow up survey results for Ethiopia.
Product flow and data flow. Product flow and data flow indicators did improve across all groups; however, the intensive group, where the effective people component was implemented to a greater extent through follow up support, showed better results in terms of CHWs' knowledge: more CHWs knew they should receive products monthly and that they should submit a HPMRR form each month to the higher level. Data are not available on if this knowledge was translated into practice. The process of receiving products from the health center appeared more standardized across groups with more CHWs receiving products from health center (as per IPLS) however this was not necessarily due to the intervention as there was a policy change at national level directing CHWs to collect salaries and products from the nearest health center.
Effective people. The follow up survey results show that six months after training health center staff, the number of CHWs trained in supply chain had increased five-fold and CHW competency and knowledge had improved. In the intensive group (I), where health center staff received support from the project and higher levels to organize trainings and conduct follow up [15], 84% of CHWs surveyed had been trained in supply chain, compared with 62% in non-intensive (NI), and 17% in the comparison group (C). All increased from the baseline (11%). CHW knowledge improved, with 87% of CHWs in the intensive group, 59% in non-intensive group, and 14% in comparison group knowing to submit reports to health centers, compared to 5% at baseline. CHW competency varied by task, being higher for a simpler task of completing a bin card correctly (65% intensive, 59% non-intensive, 62% comparison), and lower for the most complicated skill of completing the HPMRR form (36%, 29%, and 25% respectively) [16]. The latter modest performance scores were, in fact, a dramatic improvement over baseline (0%) and would likely improve further over time with practice and targeted supportive supervision. Ready Lessons had improved the skills of CHWs laying the groundwork for improving management processes.
Sixty eight percent of CHWs in the intensive group reported participating in a Problem Solving session during monthly meetings compared to 26% in the non-intensive group, the team building component of this intervention (problem solving was not part of the intervention package for the comparison group). At the health center level, 85% of HC respondents in the intensive group and 53% in nonintensive group reported conducting Problem Solving sessions with CHWs. However, despite slow rollout of the Problem Solving sessions, when CHWs were asked about their usefulness, CHWs from all regions stressed that the problem solving was very important for strengthening the IPLS at the CHW level.
Comparing Intervention Packages. As seen in Table 8, improving CHW knowledge of the reporting and resupply processes was not sufficient to have an impact on product availability at community level. The results show that in the comparison group where the least number of CHWs were trained, the product availability was higher. More CHWs in this group reported receiving products from an NGO and it is therefore likely due to the presence of kits which were being distributed ad hoc and not in response to need. When considering the other results and comparing the different approaches to making people effective we see that the intensive group which received more follow up and had a larger number of Problem Solving sessions did better for many of the indicators, suggesting that achieving the effective people element requires more than just training staff.

dISCUSSION
The findings from all interventions in the three countries suggest that the greatest supply chain benefits are realized when all three elements-product flow, data flow, and effective people-are in place and working together. This is most clearly demonstrated by the benefits of synergy on the supply chain as demonstrated by the EM and QC results; these interventions brought together product flow, data flow, and effective people to achieve the greatest improvements in supply chain to the community level. The Rwanda IcSCI intervention, that also brings all three elements together but does not directly address the multilevel team work component of effective people, showed less detectable improvements in SC performance and product availability compared to the QC. This suggests that the three elements work best together when all components related to each element are part of the intervention design.
Results from the EPT group in Malawi where data flow and effective people were only partially addressed and Ethiopia where effective people was only partially addressed, further suggest that when only one or two elements are present, only minor or incremental benefits are observed, and effectiveness of the supply chain -as measured by improvements in supply chain performance, supply reliability or product availability -are not affected. Quantification and national product availability proved to play an important role in determining product availability at the community level -pointing to the importance of implementing community health supply chain improvements within the context of the overall supply chain.
Product flow, data flow and effective people: interconnected elements Product flow and data flow, though two distinct elements of the CHSC Framework, need to be deliberately linked, aligned and synchronized during intervention design to ensure that the right data are collected and made available to the right person (eg, the person resupplying a CHW needs access to consumption and stock on hand data to determine resupply quantities) who then uses it to make informed decisions on resupply/product flow. The interrelationship between these two elements are further strengthened and sustained when levels of the system supplying the data see the associated benefits, such as products flowing to them based on demand. In addition, data and product flow must also align with management practices and workflows, as part of the effective people element, to realize maximum benefits in the SC, as seen in the case of Rwanda QCs and Malawi EM where streamlined data flow combined with a structured mechanism for reviewing and using data made people more effective and involved in managing product flow thus resulting in improvements in supply reliability and supply chain performance.
The "gold standard" is an EM-like solution that combines a demand-based system with a real time reporting system, such as a mHealth system, that allows inventory data to be available at all levels of the system simultaneously to enable rapid decision making and response as well as activities such as performance monitoring, management and quantification. Although the RSPs in Rwanda included a demand-based system that resulted in effective data flow and product flow between the community level and resupply point, the manual nature of the data flow system prevented CHW logistics data from being immediately available at levels beyond the CHW resupply point to enable effective community supply chain performance monitoring and management by district and central level managers.

Effective people
The effective people element of supply chain interventions -despite its potential for reinforcing product and data flow and improving community health supply chain practices -is the element most often left out, in part because of the required time investment and challenges to monitoring and measurement. Additionally, because CHWs generally are at the last mile they can be isolated from the main health system. Program design to support effective people can reduce perceptions of isolation by making supply chain performance and supply reliability a joint goal amongst facility-based staff and CHWs. However, this requires ongoing commitment at multiple levels to ensure that CHWs, at the end of the supply chain, receive routine support and feedback from managers who are close to them in the chain. The qualitative data for the EM and QC interventions best demonstrate how and why the effective people component is so important in enhancing results; DPATs and QITs strengthened linkages across multiple levels of the health system, enhanced communication and understanding of tasks, and established common goals and a collective responsibility for achieving results, while motivating CHWs who performed well.
In Malawi, the difference between the EM and EPT groups on key supply chain performance indicators such as complete reporting, reduced lead times, and stock out rates, can be attributed to the DPAT component in the EM intervention, underlining the importance of the effective people component. In Rwanda, the RSP intervention rationalized data and product flow and the QC and IcSCI groups addressed management and motivation each in slightly different ways. The IcSCI intervention included the management and motivation components of effective people which accelerated the immediate uptake and utilization of the data and product flow process (RSPs) and contributed to behavior change of CHWs towards improved performance of supply chain tasks. However the difference in findings between QC and IcSCI demonstrates the added benefit of the formalized team component in the QC group, where greater improvements in supply chain performance and product availability could suggest that this is related to the effects of multi-level teams working together to identify and solve problems related to community supply manage-ment, rather than the single-level nature of cooperatives as teams. The significant improvements detected by the DiD for the QCs intervention also establish evidence of this as a successful method for improving iCCM product availability at the CHW level.
In Ethiopia, an important finding was that using health center staff to train CHWs in basic supply chain knowledge and skills by incorporating lessons into existing activities can significantly improve training coverage in a short period of time and is affordable since it doesn't require extra travel or allowances. However, although the Ready Lessons/Problem Solving intervention yielded a reasonable improvement in supply chain competency levels, it wasn't sufficient to affect supply chain performance or improve product availability. In Ethiopia, the intervention mainly addressed management, while structured problem solving increased contact between CHWs and HCs, but did not establish a sense of a team with common goals or involve district levels to help with more complex problems, as was seen in Malawi and Rwanda, two differences that might explain limited achievements. Essentially, the Ethiopia experience demonstrates that within the effective people component, all three components must work together -management, teamwork, and motivation -so that optimal results are achieved.

Quantification and national coordination
As the foundation to the product flow -data flow -effective people cycle, quantification and national coordination are important central-level activities that support community-level product availability. If community-level needs are not carefully considered and estimated in national quantifications and procurements, CHWs will likely suffer the most from shortages and expiries since they are at the end of the supply chain, regardless of how well the lower level supply chain functions. Therefore, to see optimal product availability at the community level and realize the benefits of designing and implementing interventions using the CHSC framework, sufficient quantities of products need to be available at higher levels. This requires the flow of data from the community level as well as coordination with data from higher levels where the same products are used and careful oversight of stock levels, considering total system demand for each product. Some of these challenges are overcome by using unique products as was seen in Rwanda, as it is easier to quantify for iCCM as a standalone program and ensure that products are not used up before they arrive at the community level.
Coordination between MOH programs, donors, and procurement units is also important for ensuring that available resources are used efficiently and that products used by multiple levels/programs are sufficient for all intended uses. The level of coordination required is often difficult to achieve, especially in countries where procurement for community-level products spans multiple programs and donors. In Rwanda, for instance, follow up results for product availability were aided by strong coordination at the MOH level and the use of unique products at the community level which made it easier to estimate CHWs' needs and ensure that products were available at resupply points for community use. On the other hand, Malawi experienced an economic crisis and currency devaluation during the test period, which impacted the distribution and availability of supplies. Given the sudden lack of funds available for functions that typically fall under the government' s purview, partners stepped in to support parallel supply chains and procure and distribute products outside the system that the interventions were meant to strengthen. Simultaneously, government budgets for procurement of essential medicines were dramatically reduced and uncertain, making central-level coordination and planning very difficult. Intervention results in Malawi made clear that the political and economic environment as well as the national product availability environment and distribution mechanisms play important roles in determining product availability at the community level -pointing to the importance of implementing community health supply chain improvements within the context of the overall supply chain, where quantification and national coordination take place regularly and effectively, and the overall supply chain is characterized by strong organization/leadership. Aligning management, motivation, and teamwork with coordination and routine quantification creates a system with improved availability of data and products when and where needed.

CONCLUSIONS
In addition to an enabling political and economic environment, there are many factors necessary for ensuring continuous product availability at the community level. Assuming the presence of sufficient capacity and funding to procure products on a regular basis, an in-country distribution system works best when three key elements (product flow, data flow, and effective people) are included in system design. The way these are implemented may look different in each setting, as they were designed with the local context and longer term scale and sustainability in mind [17]. However in all countries the common finding was that intervention designs need to ensure that data flow and product flow processes are streamlined, aligned, and reinforced by an effective and supportive workforce that is organized into multi-level teams with common objectives and structures for supervision, that use data to improve supply chain performance and communicate regularly to promote product availability at the community level.