Adherence to the iDSI reference case among published cost-per-DALY averted studies

Background The iDSI reference case, originally published in 2014, aims to improve the quality and comparability of cost-effectiveness analyses (CEA). This study assesses whether the development of the guideline is associated with an improvement in methodological and reporting practices for CEAs using disability-adjusted life-years (DALYs). Methods We analyzed the Tufts Medical Center Global Health CEA Registry to identify cost-per-DALY averted studies published from 2011 to 2017. Among each of 11 principles in the iDSI reference case, we translated all methodological specifications and reporting standards into a series of binary questions (satisfied or not satisfied) and awarded articles one point for each item satisfied. We then calculated methodological and reporting adherence scores separately as a percentage of total possible points, measured as normalized adherence score (0% = no adherence; 100% = full adherence). Using the year 2014 as the dissemination period, we conducted a pre-post analysis. We also conducted sensitivity analyses using: 1) optional criteria in scoring, 2) alternate dissemination period (2014–2015), and 3) alternative comparator classification. Results Articles averaged 60% adherence to methodological specifications and 74% adherence to reporting standards. While methodological adherence scores did not significantly improve (59% pre-2014 vs. 60% post-2014, p = 0.53), reporting adherence scores increased slightly over time (72% pre-2014 vs. 75% post-2014, p<0.01). Overall, reporting adherence scores exceeded methodological adherence scores (74% vs. 60%, p<0.001). Articles seldom addressed budget impact (9% reporting, 10% methodological) or equity (7% reporting, 7% methodological). Conclusions The iDSI reference case has substantial potential to serve as a useful resource for researchers and policy-makers in global health settings, but greater effort to promote adherence and awareness is needed to achieve its potential.


Methods
We analyzed the Tufts Medical Center Global Health CEA Registry to identify cost-per-DALY averted studies published from 2011 to 2017. Among each of 11 principles in the iDSI reference case, we translated all methodological specifications and reporting standards into a series of binary questions (satisfied or not satisfied) and awarded articles one point for each item satisfied. We then calculated methodological and reporting adherence scores separately as a percentage of total possible points, measured as normalized adherence score (0% = no adherence; 100% = full adherence). Using the year 2014 as the dissemination period, we conducted a pre-post analysis. We also conducted sensitivity analyses using: 1) optional criteria in scoring, 2) alternate dissemination period (2014)(2015), and 3) alternative comparator classification.

Conclusions
The iDSI reference case has substantial potential to serve as a useful resource for researchers and policy-makers in global health settings, but greater effort to promote adherence and awareness is needed to achieve its potential.

Background
Policy makers and program managers, particularly those in low-and middle-income countries (LMIC), often face prioritization decisions with limited resources [1]. Economic evaluation, such as cost-effectiveness analyses (CEA), can provide insight into the comparative value of various health interventions and therefore help inform priority setting [2]. Since the original Panel on Cost-Effectiveness in Health and Medicine proposed the use of a reference case as a benchmark of quality and methodological rigor [3,4], various guidelines for conducting economic evaluation have been proposed [5,6]. The Consolidated Health Economic Evaluation Reporting Standards (CHEERS) Checklist, a widely cited reporting guideline, is used to ensure study results are reported with clarity and accuracy, yet does not provide methodological guidelines for how analyses should be conducted [7]. Many countries, particularly high-income ones, have also developed their own reference cases to inform decisionmaking in their health care systems [8][9][10][11].
In contrast, most low-and middle-income countries (LMICs) have not developed such guidelines, possibly due to their limited capacity to do so [12]. In fact, only 12 LMICs currently have economic evaluation guidelines specific to their country [13]. Although the general principles of guidelines for high-income countries can still be applied to LMICs, variations in both approaches and methods used limit their usefulness. For example, most high-income country guidelines suggest health outcomes be measured using quality-adjusted life-years (QALYs). The estimation of QALYs requires a preference weight for different health states, called healthrelated quality of life, on which LMICs often have limited data.
To address the need for a reference case that could broadly apply to different contexts, particularly in LMICs, the Bill and Melinda Gates Foundation (BMGF) supported the development of the Gates Reference Case to ensure high quality and transparent CEA in global health priority setting [14]. One of the key recommendations is to support the use of disabilityadjusted life years (DALY), as disability weights are more readily available and more easily transferable across different countries [15]. The first version was published in 2014 as the Gates Reference Case and, later in 2016, was renamed the International Decision Support Initiative (iDSI) Reference Case to convey the breadth of its intended applicability [14,16].
The iDSI reference case fills a major gap in global health economics, as it is the only resource of economic evaluation best practices for many policymakers in LMICs looking for guidance on resource prioritization. However, no study has assessed whether the development of the guideline is associated with an improvement in research practice for CEAs employing DALYs. This paper aims to quantify the methodological and reporting quality of cost-per-DALY averted studies over time, as measured by adherence to best practices enumerated by the iDSI reference case.

Data
The iDSI reference case. The iDSI reference case includes 11 principles: transparency, comparator, evidence, measures of health outcome, costs, time horizon/discount rate, perspective, heterogeneity, uncertainty, budget impact, and equity considerations. Each principle has a number of corresponding methodological specifications and reporting standards. By using this tiered structure, the reference case aims to serve as a framework that provides best practice guidance while allowing for flexibility depending on context [16], and thus is the most appropriate economic evaluation guideline for LMICs without their own national guidelines.
Global health CEA registry. We analyzed data from the Tufts Medical Center Global Health CEA Registry, a continually updated database of English-language economic evaluations in the form of cost-per-DALYs averted [17]. Among 620 cost-per-DALY averted studies in the database, we selected all articles published three years before and after the initial release of the iDSI reference case (2011-2017) to examine the impact of its publication on the literature (N = 398). We focused particularly on economic evaluations using the DALY metric because it is recommended as a main outcome metric by the iDSI reference case and it is used more often as a health outcome measure in LMICs than equivalent metrics such as the QALY [16,18].
To ensure a comprehensive assessment of adherence to the reference case, two independent readers (JE and AP) extracted additional information from each study in our sample using REDCap, an online data collection platform [19], including data on: currency reported; subgroup analyses conducted; limitations reported; structural sensitivity analyses conducted; budget impact conducted; justification of alternative methodology; and comparator setting.

Adherence score
We first translated all 30 methodological specifications and 38 reporting standards (across 11 principles) listed in the reference case into questions with discrete binary outcomes (standard satisfied or standard not satisfied) ( Table 1). We then designated reference case elements as "required" or "optional" based on our interpretation of the language in the report (Table A in S1 File). We deemed 19 methodological specifications and 21 reporting standards "required".
Our base-case analysis examined adherence scores consisted only of "required" elements. We evaluated each published cost-per-DALY averted study's adherence to methodological specifications (0-19 items) and reporting standards (0-21 items). We then separately calculated reporting and methods raw scores as a percentage of total possible points, measured as normalized adherence score (0% = no adherence, i.e., no requirements adhered to; 100% = full adherence, all requirements adhered to).

Analysis
Descriptive analysis. We examined the association between adherence score and certain study characteristics, including whether the study cited the reference case, the study funder characteristics, and journal attributes. We categorized study funders into the following groups (not mutually exclusive): academic, government, healthcare organization, industry, intergovernmental organization, BMGF, non-BMGF, and other. We also stratified selected articles into clinical versus non-clinical journals using SCImago Journal Rank's subject categorization (medicine vs. health policy, public health, non-health) [20]. Finally, we recorded 2016 journal impact factor quartiles and categorized studies as high impact (first quartile), medium impact (second quartile), or low-impact (third and fourth quartiles) [20]. Comparator is standard of care?
Comparator and its availability are clearly stated, and outcomes reported in incremental cost effectiveness ratio. Statistical analysis. To examine whether the iDSI guideline has, since its release in 2014, improved the methodological and reporting practices of cost-per-DALY averted studies, we calculated mean adherence scores by year from 2011 to 2017. We conducted a pre-post analysis of improvement in methodological and reporting adherence scores. As the reference case was first released in January of 2014 [21], we considered that year to be the reference case's dissemination period, and hence did not include articles published during that year in our prepost analysis. We also compared the overall methodological and reporting adherence scores, stratified by the 11 principles.
Sensitivity analysis. We conducted three sensitivity analyses. First, we included the "optional" criteria in the calculation of adherence scores for a random 10% subset of the articles to explore the impact of including optional items in the adherence score. Second, given that efforts to increase awareness of new guidelines may take longer than one year, and subsequent development and publication of adherent CEAs can span more time, we conducted a sensitivity analysis to explore alternate dissemination period lengths. Primarily, we expanded the dissemination period from 2014 to 2014-2015 to examine the influence of a longer dissemination period on adherence. Third, we used an alternative classification to determine adherence to the comparator principle's corresponding methodological specification. In our base case analysis, we designated an article adherent to the iDSI's comparator methodological specification only if the article explicitly reported their comparator as the "standard of care". In this sensitivity analysis, we classified an article as adherent so long as it specified a comparator other than "do nothing" or some other non-action. To be consistent with the iDSI reference case principle that the standard of care must include at least "minimal supportive care" [22], we designated "do-nothing" interventions as non-adherent.

Descriptive statistics
Among 398 cost-per-DALY averted studies published from 2011-2017, 215 (54%) focused on LMICs and 263 (68%) targeted communicable diseases, such as diarrhea, HIV/AIDs, tuberculosis, and malaria (Table 2). Articles averaged 60% adherence to the reference case's methodological specifications and 74% adherence to reporting standards. Table 3 summarized iDSI Reference case normalized adherence scores by year, sponsor, and journal aspects (The raw scores are available from Table B in S1 File). No article achieved full adherence to either the methodological specifications or the reporting standards.
Of the 213 articles published after 2014 (i.e. 2015-2017), only 9 (4%) cited the iDSI reference case. For articles that did so, adherence to reporting standards averaged 79%, five percentage points higher than mean adherence for the full sample, while adherence to methodological specifications did not differ from adherence for the full sample. Funding source (BMGF vs. non-BMGF) was not significantly associated with a change in adherence scores for either methodological (mean score of 60% vs. 60%) or reporting (mean score of 75% vs. 74%).
Studies published in clinical journals had marginally higher adherence (60% methodological adherence, 74% reporting adherence) than studies in non-clinical journals (57% methodological adherence, 73% reporting adherence). On average, methodological adherence scores for articles published in high-impact journals exceeded the corresponding scores for studies published in low-impact journals (61% vs. 50%); for reporting adherence, the corresponding difference was 74% vs. 71%.

Sensitivity analyses
Inclusion of optional criteria in our adherence score calculation decreased mean methodological adherence by 14 percentage points (60% to 46%) and mean reporting adherence by 22 percentage points (from 74% to 52%). When we increased the dissemination period to 2014-2015 (base case: 2014), we found no change in our results. Using an alternate comparator principle classification (base case: comparator must be standard of care; alternative: comparator can be any intervention other than "do-nothing") also had little impact.

Discussion
Since its introduction in 2014, adherence to the iDSI reference case among published cost-per-DALY averted studies has improved for reporting standards, but not for methodological specifications. Adherence to the reference case's reporting standards exceeds adherence to its methodological specifications, perhaps reflecting the relative ease of revising the way information is presented and greater effort needed to conform to analytic requirements. Moreover, other reporting guidelines, such as CHEERS [7] or country-specific recommendations, may have independently promoted more rigorous reporting, with the unintended effect of boosting adherence to the iDSI reference case.
However, methodological and reporting adherence scores varied substantially across reference case principles, demonstrating ways in which articles are falling short of guidelines. For example, articles almost always report their comparator clearly (as recommended by reporting standards), but do not necessarily specify whether the comparator is considered standard of care (as recommended by methodological specifications). Similarly, all articles reported findings from sensitivity analyses, but did not always conduct comprehensive structural, probabilistic, and deterministic sensitivity analyses.
In some cases, methodological adherence exceeded reporting adherence. For example, articles often included implementation costs (as recommended by methodological specifications), but did not as frequently report these costs in both US dollars and local currency (as recommended by reporting standards). Because the methodological specifications and reporting standards address distinct issues, future guidelines should continue to include recommendations for both types.
It is important to consider what level of adherence should be seen as satisfactory. Although articles in our sample were more adherent to reporting guidelines, they adhered to just over half of methodological specifications. Adherence scores were notably lower for particular principles-heterogeneity, budget impact, and equity-indicating overall neglect of these issues in cost-per-DALY averted studies. The adherence scores are perhaps best thought of as a baseline against which to measure improvement, and as a call to action to promote higher quality and comparability.
The lack of adherence to the iDSI reference case might reflect the competing influences of other guidelines, as authors may prioritize adherence to local guidelines that are more relevant to their context [3,11]. For example, the South African pharmacoeconomic guidelines recommend a base case 5% discount rate, which differs from the 3% value recommended by the reference case [23]. Although the iDSI reference case supports the use of alternative discount rates where appropriate to the decision problem and constituency, published CEAs that adhere to the local guidelines may be scored as non-adherent to the methodological specifications in this analysis.
Another possible explanation for relatively low adherence for certain items is that authors may not be aware of the guidelines. We found that only 4% of the identified studies published after 2014 directly cited the iDSI reference case. The BMGF and iDSI have focused educational campaigns on national payers and health technology assessment (HTA) agencies in LMICs, rather than on researchers, who are primary authors of published studies [22]. Future studies should examine whether the reference case has influenced country-specific guidelines, such as Thailand's HTA assessment guideline [9].

Limitations
The primary limitation of our study is that the post-evaluation period (2015-2017) may not have been sufficiently long to detect the impact of the reference case. Though it was initially released in early 2014, as noted, the iDSI reference case was not officially published in an academic journal until 2016 [16]. However, dissemination efforts began in 2013 at a BMGFhosted workshop for multi-sectoral stakeholders, which was later considered "a major part of the Gates reference case development" [14]. Although more time may be needed for the field to adopt these guidelines, as new CEAs can take years to conduct and publish, we believe our results on adherence to the iDSI reference case can serve as a baseline estimate. Adherence should be re-analyzed in the future as the field continues to grow.
Furthermore, our use of dichotomous (i.e., "yes/no") questions to score adherence may be inconsistent with the more nuanced goals of the iDSI reference case. Because the reference case is designed to be applicable in a range of different country-specific contexts, it must balance the goals of study comparability and quality against the goal of local applicability [6,22,24]. To address this limitation, we omitted "optional" standards from our adherence calculation for the base case. That is, we assumed that the "optional" elements represent conditional requirements intended by the reference case authors to allow for local adaptability. Our sensitivity analysis that included all elements in our calculation of the adherence score (i.e. both the "required" and "optional" elements) yielded lower adherence scores.
Assessing adherence to the comparator principle posed a particular challenge because this assessment requires subjective judgment on whether the specified comparator constitutes the "standard of care." Although our sensitivity analysis of altering the definition of appropriate comparator had little impact on our findings, a "do nothing" intervention, which is deemed inappropriate by the iDSI's comparator methodological specification, can be regarded as "standard of care" for some conditions in some settings, such as a population screening program for tuberculosis [25,26].
Also, our findings cannot be generalized to the rest of the economic evaluation literature as the Tufts Medical Center Global Health CEA Registry catalogs only published cost-per-DALY averted studies. For example, our analysis excluded gray literature (i.e., material not disseminated in regularly published, indexed journals). Gray literature may be more prevalent in some countries, especially those without local guidelines.
Finally, our approach for scoring articles inherently involves reviewer judgment to determine author intent and to resolve ambiguities (e.g., determining whether the comparator is "clearly" stated). We attempted to mitigate this problem by having two reviewers read each article and, in cases of no consensus, we appealed to a third reviewer.

Policy implications
As posited by Nugent and Briggs, future research on the subject should ask, "what specific help does the iDSI reference case offer the analyst, who, while attempting to conform to the principles, nevertheless has to choose and implement the methods?" [27] It is possible that the methodological guidelines impose an excessive burden on researchers, raising "issues about the resources and data requirements to meet the principles" [22].
Future qualitative research can focus on researcher consideration of best practice guidelines in study design and reporting, and on how to increase guideline acceptance among authors. Studies could also further evaluate the methods and reporting adherence for articles that strongly adhere to the iDSI reference case, as these analyses may serve as useful examples for other CEA authors. Analysis of the impact of the reference case on perceived quality and usefulness of economic evaluations by decision makers would be useful.
Moving from guideline development to implementation is a vital step towards improving the quality of economic evidence in global health. Future efforts could include additional educational workshops for researchers, students, and policymakers. Policymakers and major funders of economic evaluations, such as the BMGF, could require that researchers adhere to reference case recommendations in grant applications. Journals and reviewers should also impose high-quality standards for economic evaluations. Moving forward, journals may require reviewers to fill out a rubric similar to the instrument in our study that measures the adherence of economic evaluations to the iDSI reference case guidelines.

Conclusion
Since its initial launch in 2014, our study indicates that the development of the iDSI reference case is associated with improving reporting standards for economic evaluation focused on global health, but no improvement in methodological practice. Although the reference case has substantial potential to serve as a resource for researchers and policy makers in global health and economics, more effort to promote adherence and awareness may be needed.
Supporting information S1 File. Supplementary materials.