Leveraging existing program data for routine efficiency measurement in Zambia

Rationale: As donor contributions for HIV/AIDS stagnate globally, national governments must seek ways to improve use of existing resources through interventions to drive efficiency at the facility level. But program managers lack routine information on unit expenditures at points of care, and higher-level planners are unable to assess resource use in the health system. Thus, managers cannot measure current levels of technical efficiency, and are unable to evaluate effectiveness of interventions to increase technical efficiency. Phased Implementation of REMS: FHI 360 developed the Routine Efficiency Monitoring System (REMS)-a relational database leveraging existing budget, expenditure and output data to produce quarterly site-level estimates of unit expenditure per service. Along with the Government of the Republic of Zambia (GRZ) and implementation partner Avencion, we configured REMS to measure technical efficiency of Ministry of Health resources used to deliver HIV/AIDS services in 326 facilities in 17 high-priority districts in Copperbelt and Central Provinces. REMS allocation algorithms were developed through facility assessments, and key informant interviews with MoH staff. Existing IFMIS and DHIS-2 data streams provide recurring flows of expenditure and output data needed to estimate service-specific unit expenditures. Trained users access REMS output through user-friendly dashboards delivered through a web-based application. REMS as a Solution: District health managers use REMS to identify “outlier” facilities to test performance improvement interventions. Provincial and national planners are using REMS to seek savings and ensure that resources are directed to geographic and programmatic areas with highest need. REMS can support reimbursement for social health insurance and provide time-series data on facility-level costs for modeling. Conclusions and Next Steps: REMS gives managers and planners substantially-improved data on how programs transform resources into services. The GRZ is seeking funding to expand REMS nationally, covering all major disease areas. Improved technical efficiency supports the goal of a sustainable HIV/AIDS response.

report report report report

Amendments from Version 1
The version 1 was originally classified as a "Research Article" by the Gates Open Research Editorial Office due to the presence of methodology and software source code. However, on a reassessment of the article, and taking the comments of reviewer 1 into consideration, we have changed the article type to an "Open Letter". Open Letters discuss policies relevant to a broad research community, present guidelines or announce new initiatives. This revised article describes the development and implementation of a new initiative, thereby fitting the criteria for an Open Letter.

Rationale
Global demand for HIV/AIDS services is increasing faster than available resources, resulting in a substantial funding gap. In 2016, a total of US$19.1 billion was spent by all contributors (international and domestic) to the global HIV/AIDS response in low-and middle-income countries (LMICs) 1 . The UNAIDS "Fast Track" strategy has accelerated the timeline for ending the public health threat from HIV/ AIDS, but intensification of effort comes at a high cost: by the year 2020, the global HIV response is projected to require US$26.2 billion in annual resources, a 37% increase over current spending levels 2 . This gap could grow even larger, due to increases in treatment costs (more people on treatment and more expensive treatment regimens), demand for emergent interventions such as medical male circumcision and preexposure prophylaxis and flattening trends in international and domestic HIV/AIDS financing 3 .
Closing the HIV/AIDS funding gap will require a multifaceted approach, including substantial new resources from international and domestic sources, as well as stronger efforts by national health authorities to increase efficiency of resource use in their respective countries 4 . But international funding commitments leveled out after the 2009 global financial crisis, and in 2016 worldwide financial pledges declined by 7% over the previous year 1 . More optimistically, recent analyses of HIV/AIDS service delivery programs suggest substantial scope for cost savings through improvements in technical efficiency. Zeng et al. 5 noted that existing literature on unit costs of HIV/AIDS interventions shows sizeable variation across programs and countries, much of which is unexplained and likely related to poor governance and weak human resource capacity. The authors conducted an econometric analysis of HIV/AIDS program efficiency in 68 countries and concluded that a typical country program could double its output if it used inputs more efficiently. Remme et al. 6 analyzed several different policy levers for increasing domestic support for HIV/AIDS programs in 14 sub-Saharan countries and found potential efficiency gains equivalent to 29 percent of current spending. Di Giorgio et al. 7 examined ART provision in Zambia, Kenya and Uganda and determined that if health facilities boosted their efficiency levels to an 80 percent level, utilization could increase by 33% -62%.
To realize these potential efficiency gains, managers at all levels of LMIC health systems need information on current technical efficiency in facilities under their supervision. Much of the existing evidence on technical efficiency at the facility level has come from large multi-country cost studies 8-10 or "one-off" studies conducted in single countries [11][12][13] . These studies, typically cross-sectional, have generated useful information on differences in costs across sampled services and facilities within LMIC health systems, across countries and regions, and trends in service costs over time. In 2017 the Global Health Cost Consortium (GHCC) created a "Unit Cost Repository for TB and HIV Prevention, Treatment, and Care Interventions," (see GHCC Study Repository) which seeks to consolidate unit cost information from HIV/AIDS and TB-related interventions in a searchable database. But few cost studies are nationallyrepresentative, which diminishes their utility in guiding facilities not sampled toward higher technical efficiency. Also, release of study results often lags data collection by one or two years, making it likely that reported findings correspond to a program that has changed substantially in the interim. Finally, these studies themselves can be costly and intrusive to service delivery.
In upper-income countries (UICs), health system managers increasingly are using data-driven cost accounting systems to track and improve performance across networks of similar service outlets. Cost accounting can be differentiated from financial accounting primarily by the intended audience. Cost accounting information is used by decision makers inside the organization to improve cost control and efficiency, while financial accounting information is geared toward external groups such as shareholders, lenders and regulators. Cost accounting systems encompass multiple interoperable databases, and enable managers to monitor patterns of resource use, identify cost drivers for cost control, measure total costs of care at various levels of the health system, integrate information on financial inputs and clinical outputs and outcomes, and improve data utilization through use of executive dashboards (see Becker's Hospital CFO report on advanced cost accounting). But such systems are expensive to install and maintain, and have not yet been adopted within the LMIC context.
Managers of health systems in LMICs clearly need different data and tools to drive efficiency and effectiveness within facilities under their supervision. The purpose of this Open Letter is to describe an effort -led by the Government of the Republic of Zambia (GRZ) with technical support from FHI 360 and Avencion Ltd. -to develop an automated Routine Efficiency Measurement System (REMS) that uses existing budget, expenditure, and health output data to produce near-real-time estimates of expenditure per unit of service at the individual health facility. REMS incorporates two main innovations: (1) a computational mechanism called a "resource allocation matrix" (RAM) that enables total expenditures at the facilitylevel to be translated via a set of allocation weights to the program's main "service lines" and the resources used to support each service line; and (2) a relational database accessed through a web-based portal that combines expenditure data from the existing financial management information system with output data from the DHIS2 system to generate quarterly estimates of expenditure per unit of service at the point of care. Output from REMS can be used by health system managers to monitor expenditure patterns over time, guide resource allocation decisions, and identify best practices for lowering unit costs. The REMS build in Zambia currently covers 326 facilities spread across 17 districts in Central and Copperbelt provinces. Welfare manages the health system and resources at the District and Community level. We also sought to create a culture where government officials would be empowered to provide active and engaged leadership in project planning and eventual implementation. Besides reflecting our belief in user-centered design, past experience has shown many projects fail when a solution is imposed from outside and then transferred to local operation. This empowerment was accomplished through a concerted effort to engage ministerial senior leadership and, in parallel, by convening regular consultative meetings with integrated teams of government technical experts. Our approach increased investment of time by government senior leadership and subject matter experts. This integrated group of government officials evolved into a REMS core team that championed and guided the project by providing continuous feedback on direction. The core team members were identified through an iterative process of regularly engaging with key leadership within the relevant government ministries to discuss REMS functional requirements, overall implementation, validation and sustainability of the REMS system. REMS core team members were chosen based on their level of commitment, overall interest, and level of influence within the government hierarchy.

Phased Implementation of REMS
At the constitution of the MoH REMs core team, a REMS implementation workshop was held where the core group established team principles, values, and capacity developmental needs which were the basis for the implementation strategy and work plan activities. To establish these principals, values, and capacity developmental needs, the workshop agenda incorporated a team building activity. The results from the core team exercise are shown in Table 1.

Dissemination of REMS
The REMS project was fully integrated into the sector wide approach (SWAp) mechanism, which is the health sector coordination platform encompassing various layers of technical

Technical description of Zambia REMs build
A key feature and advantage of REMS is that it leverages existing financial and output data, reducing the amount of "new" data needed to generate unit expenditure estimates. Governments routinely produce information on expenditures and outputs, but bringing these two data streams together electronically to produce recurring unit expenditure estimates at the facility level is an innovation. The Zambian MoH uses the Integrated Financial Management Information System (IFMIS) 14 to track financial expenditures, and the District Health Information System (DHIS 2) to report on service outputs at the facility level. The Zambia REMS relational database creates an electronic linkage between IFMIS and DHIS2 data in two steps: first by stepping down quarterly IFMIS expenditures (which are often reported at the national, provincial, or district level) to the facility level and allocating expenditures at the facility level to specific services, and second by dividing these allocated expenditures by the number of units of output for that service for the same calendar quarter. Figure 2 provides a schematic of the logic flow used in the REMS database. In the upper left, the existing IFMIS data system is represented by a green cylinder. Additional data sources required by REMS are the allocation weights shown as the green cylinder in the upper right of Figure 2 and the Facility Resource Allocation Matrices (RAMS) shown in the lower right. These data exist as look-up tables within the REMS database, and were assembled as follows: Allocation weights to isolate HIVrelated expenditures and allocate HIV-related expenditures from National level IFMIS accounts to the Provincial or District level were derived from discussions with key MoH informants at national, provincial, and district level. Existing GRZ MoH planning and budgeting documents were then used to develop allocation weights to assign expenditures down to the point of care. These allocation weights were applied to the expenditures reported by IFMIS account number. IFMIS account numbers comprise a unique combination of head/department/unit/ programme (and in some cases)/activity numbers. For each IFMIS account controlled at each level, informants were asked to identify what portion of the expenditures in that account were being used to support HIV/AIDS services, and of those expenditures, what portion was flowing down to the lower level(s) vs. remaining at the current account level. Portions of expense accounts that were flagged as HIV/AIDS-related but not flowing down further were classified as "above-facility" expenses.
Once expenditures were tracked to the point of care (facility) the facility RAMS (shown in lower right of Figure 2) were used to distribute HIV-related expenditures incurred at a service delivery point (facility) to specific HIV services delivered in that location (facility or community) and also to classify into expense categories (i.e., labor, drugs, other supplies, equipment, etc.). The facility-specific RAMs were derived from detailed facility assessments in which trained data collectors interviewed clinic staff to determine inputs used to provide specific HIV services within each facility. Data collectors obtained inventories of equipment and supplies used to support service delivery as well as records of HIV-related drugs dispensed in the prior month. This is similar to micro-costing where an ingredients approach is used to build up the cost estimates. Standard unit costs were applied to each resource identified and these data were summarized in an annual prototypical operational budget format for each facility. Each line item in the operational budget was then allocated across the service(s) for which it was used, considering times and locations when services are available, as well as the relative volume of services when resources such as staff or equipment are shared. This budget was then used to compute the percentage of total annual resources used for specific combinations of service and resource type. These percentages were then stored as a RAM weight. Each facility-specific RAM comprises a matrix of weights corresponding to unique combinations of resources and the services they support within that facility. For example, the RAM may tell us that 5% of total annual HIVrelated resources are used for personnel providing HTC services. REMS uses this information to assign 5% of each HIV-related Kwacha reaching the facility to personnel for HTC services.
By dividing the allocated expenditures for a specific service at a facility by the volume of service provided during the same time period as reported in the DHIS database (not shown in Figure 2) we were able to estimate the unit cost of service provision at the facility by expense category (shown as blue parallelogram in lower right of Figure 2). This unit cost estimate serves as a performance metric that can be compared across facilities and monitored over time to identify promising service delivery practices for replication in other facilities.
The Zambia REMS system is programmed in Microsoft SQL version 13.0 and the annotated source code is available through Zenodo 15 . REMS uses C# for the user interface in which users select parameters of interest (period, organization, geographic level, service(s) of interest, type of resource(s), facility(s) and total or unit cost. The results of these selections are displayed graphically using HighChart to generate visualizations and REMS allows the user to save the most relevant performance metrics to an individual dashboard (see animation of REMS dashboard from REMS site).
In the screen shot below (Figure 1), user-generated results show the unit cost of Elimination of Mother to Child Transmission (EMTCT) services for 6 facilities for the period January-March 2016. In the left panel, a ten-fold variation is apparent in the unit costs of an EMTCT visit (range: <125 Kwacha (~$13) in Kalwelwe Health Centre to 1,337 Kwacha in Kohima Camp Hospital (~$134)). In the right panel, the volume of services (patient visits for EMTCT) is shown for the same time period and facilities. While we would expect that facilities with larger service volumes would have lower unit costs (as they can spread fixed costs across more visits) some noticeable exceptions exist: Kawama Urban Health Centre reports high volume and high unit costs, while both Kalwelwe Health Centre and Kasanda urban Health Centre exhibit low volumes and low unit costs. REMS users at the District Medical Office are trained to analyze comparative unit expenditures across a range of facilities in their district, and also to track trends in unit expenditure over time; thus, both exceptions would serve as prompts for further investigation by the District Health Management team. These results can be viewed as a sliced bar or stacked column that would show the type of resource contributing to the results (i.e., staffing, supplies, equipment, etc.). This detail allows for a comparison of how productive the fixed resources are being utilized as these are what largely drive any potential efficiency gains.
Users at a Provincial Management Office (PMO) have broader permissions to monitor performance and resource efficiency in select priority districts or facilities. At the provincial and national level, users can examine whether overall expenditures are being directed to services or geographic areas where needs are greatest, and can monitor trends in service-level versus above-facility expenditure. By leveraging existing data systems and flows, REMS capitalizes on prior investments in health management information systems and increases the value of the data for decision making.

REMS as a Solution
In the current environment of scarce resources and growing demand for health services, there exists an urgent need to ensure more efficient use of existing resources. An ongoing problem in LMIC health systems is lack of data to measure and monitor economic performance at the service delivery level. Information on technical efficiency from cost studies is not sufficient for day-to-day program management that seeks to drive efficiency and effectiveness in front-line health facilities. As a business intelligence platform, REMS delivers routine information on unit expenditure by type of service, type of expense, time period and funding source, covering all facilities within a health system. We envision several use cases for LMIC managers and decision makers using REMS information, including the following: • Performance improvement -REMS is enabling Zambian officials at district and provincial levels to carry out routine monitoring of technical efficiency. For the first time, differences in high and low-performing facilities and districts are visible, allowing managers to develop and monitor initiatives that translate into improved efficiency outcomes. Focusing on "outliers" -i.e., facilities or districts with high or low expenditure per unit of output relative to a standard -managers can investigate potential reasons for the differences. For example, differences in program output may be driving unit expenditure variation in similar facilities. If so, what are the underlying reasons for differences in output and how might these be addressed? Another driver of unit  expenditure could be staffing levels that are not aligned with patterns of demand in a facility, opening the possibility of reassigning excess staff to busier facilities. Prior to REMS these differences were obscured, making it impossible to identify efficiency differences, or to measure the impact of changes designed to improve efficiency.
• Resource allocation at provincial and national levels -Planners in higher levels of the health system can use REMS expenditure roll-ups by district and province to analyze resource flows, ensuring that funding and services are directed to geographic areas where disease burden is greatest.
• Assessment of HIV/AIDS financing and sustainability -National-level health officials now have information on total expenditure per unit of service, and when the system is fully operational, how the expenditure burden is distributed across different funding entities such as the Global Fund and PEPFAR. They also will be able to document facility and above-facility-level expenditure, and examine trends in the GRZ share of the HIV/AIDS financing burden.
• Support implementation of social health insurance -the GRZ is assessing feasibility of a Social Health Insurance Scheme but lacks a source of routine information on expenditure per unit of service. REMs generates timeseries data on unit expenditures that will enable calculation of benchmark unit expenditures that can be used to set reimbursement rates. REMs data also may help planners decide on the design of the system (i.e., claims versus capitation), and could assist in measuring the administrative burden associated with claims.

Recent experience with REMS users in Copperbelt and
Central provinces confirm that REMS data are opening doors to new insights into the economics of service delivery. Users are working together to explore some of the "whys" behind indicators in the REMs dashboard. For example, at User Conferences in both provinces in late 2017, mixed groups comprising MoH planners and accountants from District and Provincial levels were tasked with building REMS charts based on specific scenarios. Facilitator-led discussions assisted users to analyze differences in unit expenditure across service delivery points and over time, focusing on potential causes of variation displayed in the charts. User familiarity with health facilities enabled informed discussions about such factors as differential staffing patterns, procurement of vehicles and drugs, and variability in catchment population leading to disparate levels of output. In certain cases, users could cite no obvious reasons for differences in unit expenditures, highlighting a need for further investigation at the facilities themselves 16 .
In addition to these direct applications of REMS data, we foresee additional indirect benefits that contribute to health system strengthening in various ways. First, existence of routine comparative data on efficiency will engage local staff in efficiency improvement, incentivizing them toward taking a more active role in interpreting and acting on their own data. REMS data also enable supervisory staff to prioritize their support visits to facilities where the need is greatest, and to evaluate whether their support is having the desired effect over time. Second, any sudden or unexpected changes in REMS unit expenditures will trigger scrutiny, and the quality of data streams feeding REMS will be considered as a possible cause of any changes. As an example, while DHIS 2 data are collected at facility level, they are subsequently rolled-up to district and provincial levels for ordinary reporting purposes. If DHIS 2 data are keyed incorrectly or are missing at the facility level, the roll-up may obscure these errors. But REMS uses facility-level DHIS2 data as denominators for unit expenditure estimates, and so data entry errors (or omissions) will be immediately obvious in the district-level REMS dashboard screen that compares unit expenditures across facilities. Similarly, any data quality issues in the IFMIS also should be reflected in the REMs dashboard. Thus, increasing reliance on REMS data will focus greater attention on the quality of data that feed the system. Third, access to detailed budget information at the facility level is uncommon in Zambia and most other LMICs. Detailed facility budgets produced by the REMS facility assessment tool will provide local managers with additional information to improve program implementation and increase transparency.
In the current Zambia REMS build, health planners and managers are using the system to monitor efficiency of HIV/AIDS service delivery. REMS was designed purposely to be adaptable for use in other areas such as malaria, child nutrition, and immunization, as well as other global development sectors outside of health, such as education, nutrition and economic livelihoods. REMs requires a set of conditions to be in place before the system can be implemented. Data requirements include budgets at the service delivery level and above; an electronic financial management system like IFMIS; and use of DHIS-2 or a similar electronic application for tabulating system outputs. These conditions already are in place in many LMICS and across multiple development sectors, creating multiple opportunities to bring routine efficiency measurement and monitoring to other sectors and countries.
In summary, the introduction of REMS in two provinces of Zambia addresses the problem of insufficient information on economic performance of healthcare delivery at the site level. REMS produces near-real-time data on expenditure per unit of service, emulating business intelligence typically produced by enterprise resource planning (ERP) software. In Copperbelt and Central provinces, managers now can compare current unit expenditures across facilities and districts, highlighting those sites that are producing services at high or low relative levels of efficiency. We think of REMS as analogous to a thermometer. It can be used to document a fever, but it does not tell the user how to treat the fever. The user is presented with REMS output to help target investigations of facility performance, identify outliers, or perhaps identify data quality issues within the IFMIS or DHIS2 data. If corrective action is taken, REMS (like a thermometer) can be used to assess whether the patient is responding to the treatment and if not lead to further investigations. Our goal is to put actionable information into the hands of persons who manage health systems. This information currently does not exist. Asking health systems to become more efficient users of scarce resources, while providing no means of monitoring resource use is doomed to failure. All too often we see a standard resource complement assigned to a facility based solely on its designation in the health system hierarchy and not tied to the service volumes for which it is expected to deliver.
For the first time in a LMIC, routine data are available to support economic changes to enable the health system to serve more HIV/AIDS clients with the same resources, reducing the fiscal gap currently facing many countries struggling to control HIV/AIDS epidemics. REMS also can be used to monitor the impact on efficiency of innovations such as task-shifting, which a recent literature review of HIV/AIDS and TB programs suggests can lead to efficiency improvements and cost savings 17 .
In addition to the facility-level performance improvements mentioned above, REMS also can be used to estimate abovefacility expenditures, which, if managed effectively, may create opportunities to expand service availability within the current resource envelope.

Next steps
While our award was funded to 1) develop a prototype, 2) secure participation and commitment from the Government of Zambia, and 3) implement the system in two provinces, we are seeking funding to take the system to the next level (expand geographically and programmatically), introduce additional feedback loops within the program logic, bring in additional data systems such as HMIS as it matures, and create a user-interface to generate routine performance reports and query the facility assessment data. In addition, we will work with the MoH to document changes in resource deployment resulting from REMS insights, and measure the impact of these changes on unit cost of services. This validation of REMS, demonstrating how use of the system leads to program improvement, will be essential to the further development of REMS and potential adoption in other countries. We are not claiming to have produced a perfect performance monitoring system but rather to have laid the foundation stones for a more complete and robust system. In an era where countries are being asked to take on a larger share of the financing of the health sector, tools to enable effective management of health systems are necessary.

Data availability
The source databases used in the development of REMS include DHIS2, IFMIS, and results of the facility assessments conducted for this project (see Table 2). DHIS2 is a system being used to capture statistical data on health activities throughout the country. IFMIS provides budget and expenditure data for government ministries. The facility assessments give details of inputs, service delivery patterns and resource use for each facility. In the scale-up phase of REMS, we expect that facility assessments will become routine annual exercises to document inputs and resource use at MoH facilities, and to form the basis for activity budgets. Below is a summary  The sub-Saharan Africa demographic 'bulge' matched with flat or marginal increases in HIV/AIDS funding will likely hamper efforts to achieve the UNAIDS 2030 Fast Track targets. As Dr. Anthony Fauci stated on many occasions, we have the programmatic tools at our disposal, but perennially face an 'implementation gap' that prevents us from adequately and efficiently deploying our collective resources and achieving the scale and coverage necessary to ending the epidemic. As the authors assert, measuring efficiency is increasingly paramount.
The authors are also correct that decision-making is often hamstrung by a dearth of quality, fit-for-purpose resource data, both for those available in the future and those already deployed. Country governments are routinely asked for data-driven concept notes, investment cases, and funding proposals that include resource projections and plans for stronger accountability, efficiency gains, and measured increases in domestic resource contribution. However, systems to institutionalize access to these data -including validation/quality assurance -remain elusive.
Further, capturing the full spend towards achieving a particular program target at facility level is not easy. It requires, not only sophisticated methods and digital systems, but also extensive political and technical negotiations with those that are closest to the point of service and cost centers. The authors did a great job of outlining the necessity of country ownership and the steps taken for stakeholder consultation/feedback. These both go a long way to demystify the math and garner critical buy in of the results for use in decision-making.
In general, the REMS platform and approach may help to rectify a glaring monitoring gap and provide easy to use tools for better program management, but there are some notable caveats: For unit costs to be useful, all sources of funding must be included: As illustrated in Figure 2, it appears that the only financial data used in REMS comes from the Government's financial management system -IFMIS. It would be useful for the authors to clarify if REMS includes only domestic resources or if it is inclusive of the other major HIV funding streams namely PEPFAR and Global Fund. Given that external sources often make up the bulk of HIV program resources, it's critical that these are included in any estimates of unit cost. If these data are not currently captured, how would they be in the future using REMS? Partners have 2. 3.

4.
program resources, it's critical that these are included in any estimates of unit cost. If these data are not currently captured, how would they be in the future using REMS? Partners have traditionally been hesitant to release the full extent of the financial data they gather, limiting country efforts toward a unified financial monitoring system for health and HIV. Has this been a challenge in the Zambian context and how would it be handled in other use cases?
Transparency and easy adjustment to allocation coefficients must be the norm: Though estimating unit inputs for HIV care and treatment at each facility sounds rigorous, this approach may yield varied results based on the quality of the enumerators and the understanding of health workers at the time of data collection. There may not be a better method at current, but it would be interesting if the authors could comment on the degree to which the RAMS coefficients vary and if clear patterns emerge -i.e. different intensity based on size of HIV patient cohort, size of facility, external technical assistance, etc. Further, is there a process in place for how the RAMS are accessed, updated, and validated routinely? We hope HIV services evolve as new and more efficient means of HIV treatment are incorporated, so the coefficients assigned at facilities will have to keep up to remain accurate and relevant.
The results must be validated in context: Given the REMS platform represents a new approach to allocating expenditures and generating unit costs, it is important the results are triangulated with other methods and data sources and the disparities are highlighted for further investigation. Have unit cost results thus far been compared with related sources -e.g. the PEPFAR unit expenditures for similar indicators and/or more traditional cost studies? Are they similar or dramatically different? Though variation is expected for a number of reasons, a deeper triangulation will help to understand how best to interpret the results and how they should be applied in planning and management.

Progress to a global good:
It's clear the authors are keen to expand the REMS system and deploy it to additional sites, countries, health programs, and sectors. A major challenge for health programs currently is the emergence of siloed data streams and digital systems, resulting in fragmentation and incongruence of data that limits application and value. Many countries have developed eHealth national strategies and articulated a vision for an enterprise system architecture that is efficient, streamlined, and government owned. Given the inherent value of a system like REMS, matched with its broad applicability, perhaps the authors would consider positioning the platform as a global digital good -an open-source solution that is designed to plug and play nicely with other country level systems (e.g. DHIS2). Preparing this product for an open source deployment would require harmonization with global standards for interoperability and substantial, ongoing technical support to operationalize the platform at country level.
Finally, the authors provide good example use cases of how the routine outputs of REMS could be used, but it would be helpful to outline concrete examples of how the data have been used in Zambian health facilities to adjust resource allocations, improve service delivery, or trigger deeper investigations.
In sum, the REMS platform provides a promising solution to fill a critical data gap and could be widely deployed if the financial data captured are comprehensive (including non-government resources), the deployed if the financial data captured are comprehensive (including non-government resources), the allocation methods are transparent and easily adjusted/updated, the results are validated/triangulated, and the platform evolves to a public good that easily integrates with other systems in an enterprise fashion.

Where applicable, are recommendations and next steps explained clearly for others to follow? Yes
No competing interests were disclosed.

Competing Interests:
Reviewer Expertise: Health economics, public health policy, data systems, data use I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. This nicely written Open Letter describes the creation and illustrative output from the Routine Efficiency Monitoring System (REMS). The salient idea of REMS is linking data on inputs from the country's financial system maintained by its Ministry of Finance with its output system (numbers of health services delivered) at the facility level by quarter. This linkage and the associated dashboard allows managers to determine and visualize unit costs by service, facility or district, and by quarter. The authors note that the system has been piloted in Zambia's Central and Copperbelt Provinces. The strength of this system is best summarized in following excerpt: "We think of REMS as analogous to a thermometer. It can be used to document a fever, but it does not tell the user how to treat the fever. The user is presented with REMS output to help target investigations of facility performance, identify outliers, or perhaps identify data quality issues within the IFMIS or DHIS2 data. If corrective action is taken, REMS (like a thermometer) can be used to assess whether the patient 1,2,3 1 2 3 data. If corrective action is taken, REMS (like a thermometer) can be used to assess whether the patient is responding to the treatment and if not lead to further investigations." The authors describe a participatory system across government agencies and partners intended to build a sustainable system. It would informative to read about how long the system has been in operation and how sustainable the model has proved.
As donors and governments decide on the value of maintaining and possibly extending such a system, several pieces of information would be useful. Following their fever analogy, it would be helpful to understand the incidence of "outliers" that they uncovered. How often were the outliers related to omissions or mistakes in financial data or service utilization data? When data issues were identified, were they one-off or systematic? Where the outliers were related to facility performance, the authors noted that smaller facilities would be expected to have higher costs, as they had less volume against which to distribute the fixed costs. It would be helpful to develop volume-adjusted standardized costs that recognize these fixed costs. Then actual unit costs could be compared against these standards for a more meaningful comparison.
Facilities with high costs could be candidates for reallocating personnel or supplies; ones with low costs might illustrate exceptional good practices, or reflect understaffed facilities trying to cope with high demand.
The authors did not describe the cost in developing REMS. While they noted that they were seeking funding for possible extensions, they did not describe their projections. Policy makers would be interested both in extending the system to additional provinces in Zambia as well as to another country with parallels in data systems. Both of these financial items would be informative.
Policy makers are often familiar with the 80-20 rule, where 80 percent of the value could be obtained with 20 percent of effort. Applying that rule in this case, policy makers may wish to consider two simplified versions.
One simplification would relate to the number of services reported. If the "Resource Allocation Matrix" uses a set weight for each type of service (which is a plausible process in view of the available data), then a facility with a cost of three times the mean for one service will have three times the mean for every other service. In that case, the unit costs could simply be reported for one benchmark service, such as a "standard facility visit." Each type of service would have a weight relative to a standard visit.
A similar process determined the cost per outpatient visit and cost per bed day from a hospital visit (Shepard , 2000 ). A district manager could examine the cost per standard visit across the health et al. centers in his/her district in a given quarter, as well as the evolution of these unit costs over time in a single health center. A similar process could examine costs per bed day across hospitals.
The second simplification relates to the level of the delivery system. Instead of analyzing data at the facility level, it may be informative to examine allocations at the district level. This aggregation would cut the number of units by an order of magnitude. Supplies and personnel are often allocated to districts, so the quality of data and ability to act may be far greater.

Does the article adequately reference differing views and opinions? Partly
Are all factual statements correct, and are statements and arguments made adequately supported by citations? Partly

Is the Open Letter written in accessible language? Yes
Where applicable, are recommendations and next steps explained clearly for others to follow? Partly No competing interests were disclosed.

Competing Interests:
Reviewer Expertise: Health economics and health care policy I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. The authors have responded well to the concerns expressed in the initial review and have reclassified the submission as an "open letter" rather than a "research article". In light of these changes, I am pleased to change my approval status to "approved". Thank you for having invited us to review.
No competing interests were disclosed.

Competing Interests:
We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Overall, the paper is well-written and highlights a very important issue (the need for better data collecting and aggregation) that is worth addressing. The paper is a description of the development and implementation by FHI 360 of a version of a health management information system they call REMS, or the Routine Efficiency Measurement System. The system takes the financial data from the Integrated Financial Management System and service delivery data from the DHIS2 system and uses those data to generate unit-specific efficiency estimates. The authors describe the system and its successful implementation in partnership with the MoH of Zambia. The paper addresses the process that was followed with the Government of Zambia to shape the REMS and ensure that it is responsive to the GoZ's needs. It also describes the implementation of the system, the characteristics of the system, and some preliminary evidence of the system's utility to its users.

Concerns:
The paper shoehorns a description of the implementation of a health management information system into a background/methods/results/conclusion format as if this were a research study addressing a research question. This is not a research paper, but rather a well written journalistic recounting of a project designed to improve the collection and analysis of efficiency in Zambia. It doesn't have a discernible research question; it doesn't apply discernible research methods to answering a research question.
The paper claims that: As a business intelligence platform, REMS delivers routine information on unit expenditure by type of service, type of expense, time period and funding source, covering all facilities within a health system.
Implied in the claim is that the REMS system is delivering accurate, useful, actionable data about performance of clinics within the system. It estimates unit expenditure by type of service using Facility Resource Allocation Matrices (RAMS) to allocate facility-level costs among the different activities in facilities. The authors mention that the "allocation weights are based on input from key MoH informants" and that the Facility Resource Allocation Matrices (RAM) "weights were derived from a detailed facility assessment" with interviews but that is the extent of the description about how exactly the matrix determines unit cost. However there is no reported validation of this approach. The reader has no reason to have confidence that the estimates generated by the system are sufficiently accurate so as to be actionable. There is only the implicit suggestion that because the Ministry has started to use the system to assess efficiency therefore the data generated must be useful. While I am a big believer in the power of data that reveals heterogeneity in performance to improve performance across the system, it too much to expect blind trust in a new method to estimate efficiency. How much of the estimated heterogeneity (variation in efficiency among clinics) is real and how much is an artifact introduced by the estimation tool?
The paper implicitly makes the claim that efficiency data are very important for improving system performance (with which I agree, as a general point) and that the REMS system, in providing such data, is thus transforming health system efficiency. While this story is plausible, no evidence is presented that the interest expressed by the MoH in the REMS data are actually leading to transformational changes in health system performance. For example, the paper offers graphs of unit costs of certain types of care across different healthcare facilities showing highly unit costs of certain types of care across different healthcare facilities showing highly heterogeneous costs per unit of service delivered. However, there is no indication of whether the large differences in cost of EMTCT are just a reflection of differences in maternal HIV prevalenceor whether there are clear differences in the efficiency of the delivery of the services provided. Thus, it is not clear from the paper that the estimation approach used will meaningfully result in better allocation of resources and improve efficiency.
The paper makes several statements that argue that the process followed by FHI 360 was critically important to the success of the implementation. One such example: All of these statements are plausible, but there is no evidence provided to support the assertions. Would the implementation of a similar system in another country require the same degree of complexity in the implementation process? Is there evidence that the implementation would not have been successful if it had just been implemented by the MoH and subsequently revealed to the other Ministries? etc. As such, they appear to be statements of opinion by the authors that the approach they took was the correct/efficient/ethical approach that should be replicated if another country were to consider implementing such a system. It perhaps goes without saying that it is especially important to qualify such opinions when the authors are those who implemented the approach being assessed. A scientific article would normally make such statements only after having tested and validated the assertions using some scientific method and using observers who are as unbiased as possible.
Thus, the paper in its current form is not a scientific publication but rather a well-written, well-organized report of the development and implementation of a health data system. It should not be published in the form of a scientific publication but rather as a report. Even as a report it should acknowledge (if true) the lack of rigorous validation of its estimation method and call for such validation. Such validation would not only give the Government of Zambia greater confidence in the estimates the system produces but provide much more useful information to other countries considering implementation of a similar system.
In addition, the report should recognize that comparisons of unit costs without any assessment of the sources of the unit cost differences can provide seriously misleading information to decision-makers. If they are going to share examples of revealed heterogeneity being usefully addressed, it would be very helpful to show that the system can identify the source of the cost differences and that those have led to changes that have improved system performance. I often refer to justifiable and unjustifiable heterogeneity (which depends upon the perspective of the observer). As an example, within Zambia there is more than an order of magnitude of difference in HIV prevalence from one locality to another. It stands to reason that the unit cost of HIV services (especially prevention services) would therefore also (justifiably) vary greatly from one locality to another. Revealing (true) heterogeneity among localities can therefore represent differences in efficiency of service delivery (that needs to be addressed with improved management of service delivery) or differences in return to investment in prevention (which might be addressed by reallocating prevention services to localities where the returns are greater). At a minimum the report should temper the enthusiasm of the authors regarding the utility of the raw system output -or it should reveal that the system output is already far more nuanced than the current example suggests.
I would like to acknowledge the assistance of Madhav Nekkar in preparing this review. I accept full responsibility for its content.

Is the work clearly and accurately presented and does it cite the current literature? Partly
Is the study design appropriate and is the work technically sound? No

If applicable, is the statistical analysis and its interpretation appropriate? No
Are all the source data underlying the results available to ensure full reproducibility? Partly

Are the conclusions drawn adequately supported by the results? Partly
No competing interests were disclosed.

Competing Interests:
I have read this submission. I believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

John Bratt
Overall, the paper is well-written and highlights a very important issue (the need for better data collecting and aggregation) that is worth addressing. The paper is a description of the development and implementation by FHI 360 of a version of a health management information system they call REMS, or the Routine Efficiency Measurement System. The system takes the financial data from the Integrated Financial Management System and service delivery data from the DHIS2 system and uses those data to generate unit-specific efficiency estimates. The authors describe the system and its successful implementation in partnership with the MoH of Zambia. The paper addresses the process that was followed with the Government of Zambia to shape the REMS and ensure that it is responsive to the GoZ's needs. It also describes the implementation of the system, the characteristics of the system, and some preliminary evidence of the system's utility to its users.

Concerns:
The paper shoehorns a description of the implementation of a health management information system into a background/methods/results/conclusion format as if this were a research study addressing a research question. This is not a research paper, but rather a well written journalistic recounting of a project designed to improve the collection and analysis of efficiency in Zambia. It doesn't have a discernible research question; it doesn't apply discernible research methods to answering a research question.

Response from the Gates Open Research Editorial Office: The article by Homan et al was originally classified as a Research article by the Gates Open Research Editorial Office due to the presence of methodology and software source code. However, on a re-assessment of the article, and taking the comments of reviewer 1 into consideration, we would suggest that this article should be reclassified as an Open Letter. Open Letters discuss policies relevant to a broad research community, present guidelines or announce new initiatives. The article by Homan et al describes the development and implementation of a new initiative, thereby fitting the criteria for an Open Letter.
The paper claims that:As a business intelligence platform, REMS delivers routine information on unit expenditure by type of service, type of expense, time period and funding Implied in the claim is that the REMS source, covering all facilities within a health system. system is delivering accurate, useful, actionable data about performance of clinics within the system. It estimates unit expenditure by type of service using Facility Resource Allocation Matrices (RAMS) to allocate facility-level costs among the different activities in facilities. The authors mention that the "allocation weights are based on input from key MoH informants" and that the Facility Resource Allocation Matrices (RAM) "weights were derived from a detailed facility assessment" with interviews but that is the extent of the description about how exactly the matrix determines unit cost. However there is no reported validation of this approach. The reader has no reason to have confidence that the estimates generated by the system are sufficiently accurate so as to be actionable. There is only the implicit suggestion that because the Ministry has started to use the system to assess efficiency therefore the data generated must be useful. While I am a big believer in the power of data that reveals heterogeneity in performance to improve performance across the system, it too much to expect blind trust in a new method to estimate efficiency. How much of the estimated heterogeneity (variation in efficiency among clinics) is real and how much is an artifact introduced by the estimation tool?

Response from the Authors: There is a lack of clarity here that will be addressed in the revised manuscript. The allocation weights based upon key MoH informants are the weights used to: 1) isolate HIV-related expenditures within the IFMIS accounts and 2) allocate HIV-related expenditures from National level IFMIS accounts to the Provincial or District Level. Key MoH informants also provided information to isolate HIV -related expenditures in IFMIS accounts pertaining to Provincial or District Level, but then existing GRZ MoH planning and budgeting data were used to further allocate these accounts down to the point of care. Once expenditures are tracked to the point of care, the facility-specific RAM is used to allocate those expenses to specific services and resource categories. The facility-specific RAM is derived from facility assessments where facility staff identified the resources used to support each HIV service within the facility. This is a similar process used in micro-costing, where an ingredients approach is used to build up the cost estimates. The facility-specific RAM is constructed in two steps: first, by creating an annual operating budget for HIV services at the facility, based upon actual resources identified during the assessment and the expected volume of services; and second, by using this budget to compute the percentage of total annual resources used for specific combinations of service and resource type. For example, the RAM may tell us that 5% of total annual resources are used for personnel providing HTC services. REMS uses this information to assign 5% of each Kwacha reaching the facility (in each calendar quarter) to "personnel" for HTC services. We recognize that this approach creates some rigidity in the cost estimates. The next version of REMS will allow for making RAM weights dynamic by linking to the output data, so that if there is an upsurge in testing, the 5% allocated to personnel for HTC would be adjusted upward (along with other resources within HTC) and personnel for HTC would be adjusted upward (along with other resources within HTC) and shares allocated to other services would be decreased.
The paper implicitly makes the claim that efficiency data are very important for improving system performance (with which I agree, as a general point) and that the REMS system, in providing such data, is thus transforming health system efficiency. While this story is plausible, no evidence is presented that the interest expressed by the MoH in the REMS data are actually leading to transformational changes in health system performance. For example, the paper offers graphs of unit costs of certain types of care across different healthcare facilities showing highly heterogeneous costs per unit of service delivered. However, there is no indication of whether the large differences in cost of EMTCT are just a reflection of differences in maternal HIV prevalence -or whether there are clear differences in the efficiency of the delivery of the services provided. Thus, it is not clear from the paper that the estimation approach used will meaningfully result in better allocation of resources and improve efficiency.

Response from the Authors: While not shown in the graphs, there is an option to created stacked columns or sliced bars that would break down unit expenditure by the type of resource contributing to the results (i.e., staffing, supplies, equipment, etc.). This allows for a comparison of how productively the fixed resources are being utilized, as these are what largely drive efficiency gains. We think of REMS as a thermometer. It can be used to document a fever, but it does not tell the user how to treat the fever. The user is presented with this information to help target investigations of facility performance, identify outliers, perhaps identify data quality issues within the DHIS2 data, etc. If corrective action is taken, REMS, like a thermometer, can be used to assess whether the patient is responding to the treatment, and if not, to catalyze additional investigations. The goal of REMS is to put actionable information in the hands of persons who are asked to manage the health system, information that currently doesn't exist. Asking health systems to become more efficient users of scarce resources without providing tools to monitor resource use is unlikely to result in more efficient health systems. All too often we see a standard resource package assigned to a facility based solely on its designation in the health system hierarchy, and not tied to the service volumes for which it is expected to deliver.
The paper makes several statements that argue that the process followed by FHI 360 was critically important to the success of the implementation. One such example:Ensuring ownership by the Ministry of Health (MoH), Ministry of Finance (MoF) and Ministry of Community Development and Social Welfare (MCDSW) was critical to the project's All of these statements are plausible, but immediate success and long-run sustainability. there is no evidence provided to support the assertions. Would the implementation of a similar system in another country require the same degree of complexity in the implementation process? Is there evidence that the implementation would not have been successful if it had just been implemented by the MoH and subsequently revealed to the other Ministries? etc. As such, they appear to be statements of opinion by the authors that the approach they took was the correct/efficient/ethical approach that should be replicated if another country were to consider implementing such a system. It perhaps goes without saying that it is especially important to qualify such opinions when the authors are those who implemented the approach being assessed. A scientific article would normally make such statements only after having tested and validated the assertions using some scientific method and using observers who are as unbiased as possible.

Response from the Authors:
We acknowledge the concern about evidence to support these statements and have edited the text to make clear that these are statements of opinion. In this specific example, access to the financial data is dependent upon the MoF, National and Provincial Level systems are run by the MoH and District Level systems are managed by MCDSW; with such inter-Ministerial complexity, it was critical that all three parties were on-board, engaged and in agreement about how REMS was to be built and accessed. The deployment of a user-centered design process was partly in response to previous experiences where a solution was imposed from outside with little (if any) uptake or interest in supporting the system. We have expanded our discussion of this issue in the revised manuscript.
Thus, the paper in its current form is not a scientific publication but rather a well-written, well-organized report of the development and implementation of a health data system. It should not be published in the form of a scientific publication but rather as a report. Even as a report it should acknowledge (if true) the lack of rigorous validation of its estimation method and call for such validation. Such validation would not only give the Government of Zambia greater confidence in the estimates the system produces but provide much more useful information to other countries considering implementation of a similar system.

Response from the Authors: We agree. Our award was funded to 1) develop a prototype, 2) show that the GRZ was willing to participate in the design and development of REMS in Zambia, and 3) implement the system in two Provinces. As such what we have now is essentially a functioning proof of concept for which we are actively seeking further investments to take the system to the next level (expand geographically and into additional health program areas) as well as introduce internal feedback loops within the program logic and create a user-interface to generate performance reports and query the facility assessment data. Our intent in publishing this Open Letter is to raise awareness of the work to date in hopes of securing funding for further validation and development of the system.
In addition, the report should recognize that comparisons of unit costs without any assessment of the sources of the unit cost differences can provide seriously misleading information to decision-makers. If they are going to share examples of revealed heterogeneity being usefully addressed, it would be very helpful to show that the system can identify the source of the cost differences and that those have led to changes that have improved system performance. I often refer to justifiable and unjustifiable heterogeneity (which depends upon the perspective of the observer). As an example, within Zambia there is more than an order of magnitude of difference in HIV prevalence from one locality to another. It stands to reason that the unit cost of HIV services (especially prevention services) would therefore also (justifiably) vary greatly from one locality to another. Revealing (true) heterogeneity among localities can therefore represent differences in efficiency of service delivery (that needs to be addressed with improved management of service delivery) or differences in return to investment in prevention (which might be addressed by reallocating prevention services to localities where the returns are greater). At a minimum the report should temper the enthusiasm of the authors regarding the utility of the raw system outputor it should reveal that the system output is already far more nuanced than the current example suggests.
Response from the Authors: As mentioned above, the current system is configured to