Reporting gaps in immunization costing studies: Recommendations for improving the practice

Highlights • Poor practices and reporting oversights limit the understanding and use of immunization cost data.• Our review identified reporting problems on the vaccines costed, types of costs, and data analysis.• Our checklist offers a standard of practice for reporting on immunization costing studies.• Reporting that adheres to this checklist will increase the interpretability and use of evidence.


Introduction
A number of global trends signal the need for higher quality, more easily accessible evidence about the costs of delivering immunization services, also referred to as immunization 'delivery costs' or 'operational costs' [1]. Although global immunization coverage remains high, there are still nearly 20 million children under the age of one who do not receive basic vaccines [2]. Most of these 20 million unreached children are in low-and middle-income countries (LMICs) that receive some type of donor support for immunization and health services. As countries transition away from donor support, increased domestic financing for vaccine delivery will be key to achieving and maintaining high coverage rates needed to ensure the population-level benefits of immunization. This is particularly relevant for the 73 countries receiving support from Gavi, the Vaccine Alliance, half of whom have transitioned, are currently transitioning, or are close to entering the final, accelerated phase of transition [3]. Immunization programs are already chronically underfunded [4,5], and the Learning Network for Countries in Transition (LNCT) predicts that vaccine costs and their associated delivery costs are likely to increase for countries during and after their Gavi transition [6]. At the same time, resource mobilization strategies rooted in updated financial projections and sound fiscal space assessments are required when advocating for funding from the Ministry of Finance and other key decision makers [7].
Until recently, evidence about immunization delivery costs was not easily accessible for use by country immunization managers and policymakers. Cost data were fragmented across organizational websites and peer-reviewed journals with paywalls, or simply not published in the public domain. Released in 2018, the Immunization Delivery Cost Catalogue (IDCC) consolidates the evidence across LMICs on the unit costs of vaccine delivery using a variety of delivery strategies, available at immunizationeconomics.org/ican-idcc.
While cost evidence is now more accessible, the quality of reporting remains an issue. Incomplete reporting is a longstanding shortcoming noted across the field of economic evaluation, and lack of publishing guidance is a recognized problem for global health interventions [8,9]. Improved reporting has been noted specifically as an area requiring further work for immunization economics [10]. Both methodological differences across studies and inadequate reporting make it hard to know if differences are due to actual variation in costs [11]. From a global perspective, comparing results across costing studies conducted in different countries and contexts is important to identify cost trends and https://doi.org/10.1016/j.jvacx.2020.100069 2590-1362/Ó 2020 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). inform high-level policy, such as grant-making from Gavi, as well as to provide a benchmark for policymakers when domestic data is lacking. Standardized reporting (and methods) would help increase the ''quality, interpretability and transferability" of cost study findings [12].
Currently, there is no recommended guidance for reporting on immunization costing results to improve the heterogeneity of the quality of articles and reports. Existing immunization-specific methods guidance-either standalone or linked to a costing tooldo not go as far as providing detailed reporting specifications. While extensive guidance exists on reporting on economic evaluations, the costing aspects of this guidance are not sufficiently detailed and often refer to secondary cost data as opposed to primary data which are frequently used in immunization and other health services costing studies in LMICs and carry unique reporting requirements [13,14]. The Global Health Cost Consortium's (GHCC) Reference Case, released in 2018, provides guidance for reporting costing study findings related to HIV and TB programs [8]. The 17-point principles and methods checklist is organized around study design and scope, service and resource use measurement, valuation and pricing, and analyzing and presenting results. While a useful tool with many relevant items for costing studies which fall outside the domain of HIV and TB programs, the reference case does not cover all relevant reporting needs for immunization program cost results. Notably, specifics about the vaccine(s) costed and guidance on common terminology used in the immunization costing community are understandably missing from the GHCC Reference Case. This paper outlines the methods for assessing the quality of articles and grey literature included in the IDCC, categorizes and quantifies the main reporting deficiencies found, and provides a reporting checklist specific to immunization costing for researchers and practitioners who are writing up primary costing study findings from LMICs, whether it be for published or grey literature.

Methods
We conducted a systematic review on immunization costs with a search that included 17,439 peer-reviewed articles and grey literature reports from January 2005 to March 2019, from which we ultimately extracted data from 68 articles and reports (the sample). Systematic review methods, summary findings, and the IDCC repository are discussed elsewhere [15,1].
During the review process, we conducted a quality assessment of the included articles and reports, which identified gaps and poor practices in reporting according to reporting and scope of the evidence. For reporting, we built on existing quality assessment systems and checklists to evaluate each article or report in three areas: methodological rigor and reporting standards (10 items), possible uncertainty of results (3 items), and risk of bias and limitations (2 items) [16][17][18][19]13,20,8]. This review on reporting included 64 of the 68 articles and reports in the sample; four did not undergo quality assessment: one was a dataset and personal communication with the EPIC Immunization Costing team, while for three others the review team participated in the writing of the grey literature reports, introducing a conflict of interest. Each of the 15 quality assessment items was given an individual score of 1 (lowest), 2, or 3 (highest); for some items there was also a ''not applicable" option. Scores for all items were summed and averaged, excluding any ''not applicable" answers, to produce a final score for each resource on the 1-3 scale.
For scope, we counted the unit cost dataset (IDCC, n = 68) to identify the number of articles and reports meeting different reporting criteria (e.g., number including paid human resources as a cost category, number identifying the types of costs included in each unit cost estimate, etc.). Based on these counts, we described the evidence and identified gaps.
We also attempted to identify commonalities between, for example, study publication date or study country, and reporting problems. We did not observe any such clustering of reporting problems so we have excluded this from our results.
We describe the results of both reviews by categorizing and quantifying the reporting problems in four areas: (1) Vaccines costed and their delivery; (2) Types of costs included; (3) Data analysis; and (4) Other study design and methodology issues.

Quality scoring
The quality review of the 64 articles and reports in the sample gave an overall mean score of 2.3/3.0 (Table 1). There was little difference in mean score between the three quality assessment areas, with each area scoring an average of 2.3/3.0. An overwhelming majority of the sample did not report sufficient methodological detail on the data analysis strategy, including use of statistical methods and confidence intervals (1.2/3.0). Sensitivity testing also scored poorly (1.3/3.0), despite sensitivity analysis being a widely recommended methodological best practice for economic evaluations, where costing is one component [18]. Contextual factors and the purpose of the study were most consistently reported (3.0/3.0).

Reporting gap: Vaccines costed and their delivery
We identified a number of issues related to reporting on the vaccines costed. In some cases, authors noted they costed the full schedule of vaccines, but failed to note which vaccines were included in the schedule. In other cases, they identified the vaccine simply as pneumococcal vaccine (PCV) or human papillomavirus vaccine (HPV), without providing any information on the type (e.g., PCV10, bivalent HPV), manufacturer, presentation (i.e., number of doses per vial versus pre-filled syringe) and required storage space and packaging of the vaccine. These details are important because vaccines represent the largest share (approximately 50%) of total immunization program cost, but the cost can vary substantially depending on these vaccine characteristics [15,21]. Vaccine delivery details, target population, and other related contextual information are also important, but frequently not reported. For example, 43% of the sample omitted information on the number of doses delivered within the period of study, information which gives you a sense of the cost-volume relationship and is needed for calculating some unit costs. Thirty percent of the sample did not state the types of facilities where immunization was provided and analyzed for the costing study (e.g., public facilities, private, NGO). Finally, 25% of the articles and reports excluded information on the target population. For some vaccines, such as Bacille Calmette-Guerin (BCG) vaccine given to newborns, one can deduce the target population, but for others it is not as clear (e.g., oral cholera vaccine (OCV) delivered through a campaign may target children and adults of different ages).

Reporting gap: Types of costs included
We found inconsistent practices in reporting on types of immunization costs. For 12% of the sample, it was not stated whether costs were valued economically, financially or fiscally. We also found variations in the types of immunization program cost categories used and their naming. For example, the terms ''administration", ''program management", ''program administration", ''overhead" and ''planning" were seemingly used interchangeably by different authors to describe costs related to managing the Expanded Programme on Immunization (EPI), though the individual cost items included in these categories were not specified and may have differed. Several cost categories were frequently not reported, but were not explicitly noted as being excluded either (i.e., per diem and travel allowances; program management; disease surveillance and activities related to adverse events following immunization; buildings, utilities, other overheads and/or shared costs; and waste management). Some articles and reports stated costs by activity, such as outreach service delivery or surveillance, while others reported line items such as paid human resources and transport. Still others combined activities and line items: reporting on vaccine transport (line item), staff (line item), training (activity), and implementation (activity).

Reporting gap: Data analysis methods
Reporting on data analysis methods was found to be a major gap. Important analytic techniques, such as how data were aggregated across facilities or extrapolated beyond the sample to arrive at a national cost estimate, were not reported in 19% of the sample. Recent research has shown wide variation in the results according to methods used in multi-site healthcare costing studies in LMICs [22], making this a critical item for analysis and reporting.
Eighteen percent of articles/reports in the sample did not report the currency year, and 47% did not report methods for converting the original currency to the reported currency (e.g., converting from 2016 Ugandan shillings to 2019 U.S. dollars). Reporting on methods for allocating shared costs and annualizing capital items was also frequently missing (in 53% and 35% of the sample, respectively). Other authors did not report the total cost (32% of the sample), number of doses delivered and/ or children immunized (44%), making it difficult to understand how unit costs were estimated. Finally, although a widely accepted standard best practice [18], 65% of the sample did not report any sensitivity analysis on delivery cost variables for which there is uncertainty.

Reporting gap: Other study design and methodology issues
The methods descriptions in 66% of the sample were not sufficient to allow another researcher to replicate the study. Fifteen percent of sampled articles and reports did not name or describe their data sources.
The sampling frame and sample size are essential reporting items, as size of the sample and how it was drawn help determine the representativeness and generalizability of the study. However, 18% of the sample did not state the number of sampled facilities, and 28% either did not describe the sampling strategy or described it in an insufficient manner.

Immunization costing study reporting: A checklist of key items to include
The identified reporting deficiencies make it difficult for the reader to assess the quality of the findings and therefore limit their use. Additionally, these omissions and reporting differences may or may not drive differences in results. Specific and structured guidance on how to write up a costing study is a critical need in the field of immunization economics to facilitate more comprehensive reporting on methods by authors. This will allow readers to evaluate studies on their robustness and scientific rigor, and better compare results of different studies.
We utilized the GHCC Reference Case checklist (summarized in Table 2) as a starting point for the development of an immunization costing reporting checklist. We deleted items which were irrelevant (e.g. specification of longitudinal versus cross-sectional data and patient sampling) and incorporated additional items (e.g. information about the vaccine(s) costed and their delivery), and also provided specific text recommendations for responses relevant for immunization costing in the ''Reporting Options" column (Table 3).
Where possible, all items should be reported in the main body of the report or publication. Where space does not allow, the full checklist can be included in an annex or supplemental file. Key parts of the checklist are described below.

Study design and scope
Study design and scope includes the study purpose, target population, perspective, type of cost, vaccines costed and time horizon. For immunization costing studies, this means clearly reporting the purpose of the study and any context for the study that will improve use and understanding, for instance, whether the costing exercise was done in the context of an economic evaluation, financial planning activity, policy input, or programmatic evaluation. The relevance of the results for health practice and/or policy decisions should be noted, along with any relevant parameters of prac-   [23] for 278,291 total doses delivered. Because not all vaccines require the same number of doses (e.g., Rotavirus 2-and 3-dose vaccines), and schedules vary by country, the number of doses per vaccine and their timing should be clearly specified. When multiple vaccines or a full schedule are costed, all vaccines included in the costing should be noted. The total number of doses delivered at the study sites (or totals used for extrapolation beyond the sample) should also be reported, as well as any coverage levels assumed. Target population should be named where it is not obvious. Authors should note the delivery strategy employed, whether delivery was in the context of routine immunization or a supplemental immunization activity (SIA), such as a campaign, and whether delivery occurred via the public, private and/or NGO sectors.
Authors should state whether a government, health care provider, or societal perspective was taken. This should be determined based on by whom the costs were incurred [24], but because of some confusion around the concept in general, as costs incurred can be interpreted as disbursing agents or the ultimate sources of funds (e.g. funds disbursed by the Ministry of Health but provided by a donor organization) [8], the perspective should also be defined. Reporting should also note whether the costing study was a retrospective costing of a past or existing immunization intervention or program, or an estimate of future program costs related to introduction of a new vaccine or change in delivery strategy. Possible sources of bias, limitations in the design, analysis, results and conclusion Free text; include data challenges and analytical techniques used that may bias results and conclusions as well as aspects of the cost estimates that would limit generalizability of results to other contexts * Depending on the Checklist Item, the Reporting Options column may include example response options, recommended reporting sub-items or further prompts to guide reporting.

Box 1 Definitions of Economic, Financial, and Fiscal Costs.
Economic costs: Financial outlays plus opportunity costs of health worker time and any donated items such as vaccines. Depreciation of capital items using a discount rate. Financial costs: Financial outlays, usually with straight-line depreciation of capital items. Fiscal costs: Financial outlays, usually without depreciation of capital items. Source: [1], adapted from [27].
Authors must clearly define whether the costs are economic, financial or fiscal (Box 1). Incremental costs should be clearly identified, and authors should indicate if costs are introduction/start-up costs (generally for a new vaccine introduction), recurrent/ongoing costs (for routine implementation), or a combination of both. If reporting introduction/start-up costs, clearly define the start and end points (with dates) for the start-up/introduction period, the included cost categories for each period, and lifetimes and discount rate for annualizing introduction costs. The reporting should indicate at which level(s) of the health system costs were incurred (e.g., facility, district, province/region, national, outside the country setting).
This checklist item includes the definition of cost categories, inputs and their quantities, data sources, sampling strategy and time of data collection. Many immunization costing studies have used the categorization in Box 2; it may be clearer to report the costs of line items (or resource inputs, i.e., the type of good or service that is purchased or used) separately from the costs of activities (i.e., the immunization program function for which the good or service is used). The line items (resource inputs) included in each activity should also be reported. The cost categories should be clearly listed and defined (i.e. which cost items make up each category). Exclusion of any cost categories should be justified.
Authors should report the sampling frame and method, as well as the sample size (facilities, districts, provinces/regions). Number of facilities may be omitted in favor of number of communities or inhabitants, such as for campaigns. Authors should also describe how respondents were selected (i.e., sampled, convenience) for interviews, if this technique was used for data collection. The timing of data collection should also be reported.

Valuation and pricing
This checklist item comprises data sources, annualization/depreciation of capital items, discount rate, inflation rate, exchange rate and opportunity cost. Authors should detail methods so that the study could be replicated by others. This means specifying the measurement approach used for each input (frequently either top-down via budget or expenditure report extraction or bottomup using the ingredients approach) and the data sources for both quantities and prices.
Box 3 Example of Reporting on Methods for Allocating Shared Costs.
''Labor time for RI [routine immunization] was assessed through a cascade of time-related questions. . .Other shared inputs, such as transport and vehicles, were allocated to RI on the basis of the share of distance traveled by vehicles. Building space was allocated based on the share of facility area used for RI activities including storage of vaccines." [26] Reporting should include any methods used to allocate shared resources both to immunization (such as building space and multi-use vehicles), and/or within immunization to specific activities (such as supervision or training) using allocation or tracing factors. An example is shown in Box 3. Authors should also specify methods for valuing volunteer time or donated items, if applicable. Further, they should describe the depreciation approach used for capital items, the useful lifetimes and any discount rates used.

Analyzing and presenting results
Included in this checklist category are all issues related to analysis and results, including variations by subgroups, uncertainty, bias, limitations and conflicts of interest. In terms of analysis, the use of analytical techniques, such as methods used to fill missing data, aggregate, generalize costs beyond the sample, as well as statistical methods to generate confidence intervals, sub-group analyses and sensitivity analyses (if appropriate given the data and policy question), should be reported. Any data quality issues should be clearly noted, along with the extent of such issues and any analytic methods employed to address them.
In terms of results, specificity is critical in reporting the findings of immunization costing studies, given the range in types of costs (economic/financial/fiscal). It is highly recommended that researchers report some combination of cost per dose, cost per capita, cost per full immunization of a vaccine, cost per fully immunized child (FIC), and cost per person in the target population. Full immunization of a vaccine refers to a completed series (all required doses of a specific vaccine (e.g., two doses of oral cholera vaccine (OCV)), while FIC refers to the provision of required vaccines to a target population by a clear point in time (e.g., infants who received all doses of a vaccine in the schedule before reaching one year of age). The definition of FIC utilized in the analysis should be clearly reported. Authors should make it clear how total or unit costs were calculated; any denominators (e.g., number of doses delivered, FICs, population, and/or persons in the target population) used to estimate unit costs should be noted. State the type of cost, year and currency for all total and unit costs.
Recommended best practice is to report unit costs with and without the vaccine costs included, as the vaccine price may differ by setting or change over time. Additionally, we recommend reporting unit costs with and without paid human resources, as human resources are generally financed from a separate budget than the immunization program. Further, it is best practice to report the percentage share of each cost category of the total and unit costs.
If a sensitivity analysis was done, results should be presented as part of the findings, and if not done, the rationale specified. Finally, any limitations, bias or conflicts of interest should be noted.

Conclusion
This paper provides much needed immunization-specific guidance for reporting on immunization costing studies, building on the seminal reference case developed for global health. Our recommendations are rooted in the review of 68 articles and reports undertaken as part of the IDCC systematic review, which shed light on common shortcomings in the reporting of immunization costing studies in LMICs. As part of the systematic review, we contacted authors to obtain and clarify missing or unclear key information, but the typical reader should not have to make this effort to understand basic study details.
Our immunization-specific checklist aims to increase the comprehensiveness and consistency in reporting of immunization costing studies, by providing a list of essential items to be included, along with sample text for reporting. Improved reporting may facilitate better understanding and correct use of cost findings for resource mobilization and allocation, planning and budgeting, and policy decisions, though likely other factors also contribute to cost evidence not being used in these processes [25]. Additionally, greater consistency and detail in reporting the methodology of costing studies can facilitate cross-study comparison of findings, offering insights into global trends and potentially providing policymakers with reasonable cost benchmarks when national data are lacking.
Ongoing work led by the World Health Organization aims to identify and agree on gaps in current immunization costing resources (i.e., methodological guidance, tools) and priorities for harmonization of terminology, definitions and methods, which will further equip researchers and practitioners with the guidance they need for excellence in reporting.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.