Systematic Review of Health Economic Evaluations of Diagnostic Tests in Brazil: How accurate are the results?

The aim of this study is to identify and characterize the health economic evaluations (HEEs) of diagnostic tests conducted in Brazil, in terms of their adherence to international guidelines for reporting economic studies and specific questions in test accuracy reports. We systematically searched multiple databases, selecting partial and full HEEs of diagnostic tests, published between 1980 and 2013. Two independent reviewers screened articles for relevance and extracted the data. We performed a qualitative narrative synthesis. Forty-three articles were reviewed. The most frequently studied diagnostic tests were laboratory tests (37.2%) and imaging tests (32.6%). Most were non-invasive tests (51.2%) and were performed in the adult population (48.8%). The intended purposes of the technologies evaluated were mostly diagnostic (69.8%), but diagnosis and treatment and screening, diagnosis, and treatment accounted for 25.6% and 4.7%, respectively. Of the reviewed studies, 12.5% described the methods used to estimate the quantities of resources, 33.3% reported the discount rate applied, and 29.2% listed the type of sensitivity analysis performed. Among the 12 cost-effectiveness analyses, only two studies (17%) referred to the application of formal methods to check the quality of the accuracy studies that provided support for the economic model. The existing Brazilian literature on the HEEs of diagnostic tests exhibited reasonably good performance. However, the following points still require improvement: 1) the methods used to estimate resource quantities and unit costs, 2) the discount rate, 3) descriptions of sensitivity analysis methods, 4) reporting of conflicts of interest, 5) evaluations of the quality of the accuracy studies considered in the cost-effectiveness models, and 6) the incorporation of accuracy measures into sensitivity analyses.


' INTRODUCTION
Diagnosing is a crucial process that is the basis of clinical practice and individual care. In the field of collective health, diagnosis also serves as the basis for decisions relating to numerous strategies, such as the implementation of screening, the control of epidemic outbreaks or the organization of secondary prevention actions (1).
Scientific research in the field of diagnosis involves estimating the accuracy and reliability of measurements. The costs of diagnostic tests should also be estimated, especially when new technologies that impose higher costs on health systems are involved.
Coverage and reimbursement for diagnostic tests face much higher entry barriers. The lack of evidence about health and economic outcomes has been cited as a common reason for unfavourable coverage decisions (2). Health technology assessment agencies worldwide are increasingly using information from health economic evaluations (HEEs), and they have intensified their scrutiny of diagnostic tests (3). In Brazil, HEEs constitute a relatively new area of scientific research (4).
In 2011, the National Committee for Health Technology Incorporation of Sistema Único de Saúde (Brazilian Public Health System [SUS]), CONITEC, began requiring HEEs to either help inform policy recommendations for the adoption of new technologies or review the policy recommendations made by SUS (5). From 2012 to 2015, CONITEC approved the incorporation of 15 new diagnostic procedures, including tuberculosis diagnosis and positron emission tomography (PET)/computed tomography (CT) for the diagnosis and staging of colorectal and lung cancer and the evaluation of treatment response in Hodgkin's and non-Hodgkin's lymphoma. Diagnostic procedures represented 17% (15 of 88) of the new technologies incorporated during this period. Some Brazilian HEEs have analysed the efficiency of diagnostic technologies, similar to those performed in other countries (6)(7)(8)(9); however, the characteristics of the body of HEEs related to diagnostic tests in Brazil remain unknown.
This systematic review focuses on the production of HEEs in the field of diagnosis and examines how economic models are implemented in the diagnostic area, thereby identifying methodological gaps that can be addressed in future studies. It is of particular interest to examine if and how test accuracy has been incorporated into economic evaluation models.
Therefore, the objective of this study was to identify and characterize the HEEs of diagnostic tests conducted in Brazil in terms of their adherence to international guidelines for reporting economic studies and specific questions in test accuracy reports over an extended period.

' MATERIALS AND METHODS
This systematic review forms part of a larger research project that systematically reviewed all HEEs related to Brazil that were published between 1980 and 2013 (10). This study is in accordance with the guidelines for the systematic review of HEEs published by the UK National Health Service (NHS) Centre for Reviews and Dissemination (11).

Study identification
We searched multiple databases, including Medline (via PubMed), Excerpta Medica, the Latin American and Caribbean Health Sciences Literature database, the Scientific Electronic Library Online, the database of the Centre for Reviews and Dissemination, the NHS Economic Evaluation Database, the NHS Health Technology Assessment database, Bireme and the Health Economics database of the Brazilian Virtual Library of Health. We searched the following citation indexes: Scopus, Web of Science, and the Brazilian Network for the Evaluation of Health Technologies. We also performed manual searches based on the reference lists of included articles and all issues of the Brazilian Journal of Health Economics (BJHE), which was a non-indexed journal in the databases mentioned above in 2013. The search strategy combined subject headings (Medical Subject Headings [MeSH] and Embase subject headings [EMTREE]) and free text terms ("Health Economics" OR "Economics, Hospital" OR "Economics, Medical" OR "Economics, Nursing" OR "Economics, Pharmaceutical" OR "Economics" OR "costs and cost analysis" OR "Cost" OR "Cost savings" OR "Cost of illness" OR "Analyses, Cost-Benefit" OR "Analysis, Cost-Benefit" OR "Cost-Benefit Analyses" OR "Cost Benefit Analysis" OR "Analyses, Cost Benefit" OR "Analysis, Cost Benefit" OR "Cost Benefit Analyses" OR "Cost Effectiveness "OR" Effectiveness, Cost "OR" cost effectiveness analysis "OR" cost-Benefit Date" OR "cost Benefit Date" OR "Date, Cost-Benefit" OR "cost Benefit" OR "Benefits and Costs" OR "Costs and Benefits") for the "economic/cost" concept with subject headings (MeSH and EMTREE) and free text terms ("Brazil" OR "Brazilian" OR "Brazi*") for the "Brazil" concept. All searches were limited to 1980-2013. The Medline full electronic search strategy is described in Appendix 1.

Study selection
Articles were included if they were partial or full economic evaluations, according to the classification devised by Drummond et al. (12), if they addressed a diagnostic test, if they were conducted in a Brazilian setting, and if at least one of the authors was affiliated with an institution in Brazil.
Studies were considered partial HEEs if they examined only costs (cost description), described the costs of a particular disease to society (cost of illness), described the costs and outcomes of a single service or programme (costoutcome description), described the financial consequences of technology adoption (budget impact analysis [BIA]) or compared only the costs of two or more interventions (cost analysis). Studies were considered full HEEs if they compared the costs and consequences of two or more health care intervention alternatives, including cost-consequences analysis (CCA), cost-minimization analysis (CMA), costeffectiveness analysis (CEA), cost-utility analysis (CUA) and cost-benefit analysis (CBA).

Data extraction
Two reviewers (TCD and RL), working independently, selected studies and extracted data using a template developed specifically for this study. The data extracted from each study included the following: year and journal of publication; type of economic evaluation; category of diagnostic test (laboratory tests, imaging, rapid tests or microorganism culture); purpose of the technology assessed (diagnosis, screening, or treatment); the type of affiliation of the first author; the geographical location of the first author; conflicts of interest, as defined by Valachis et al. (13); estimates of costeffectiveness; and the conclusion of the study (favourable, unfavourable, or neutral). For full HEEs, we also recorded whether the studies contained information on diagnostic test accuracy to answer the following questions: a) Was the accuracy of each test defined and incorporated into the model? b) Were the test accuracy parameters subjected to any sensitivity analysis? c) Was the quality of the evidence from the primary research used to estimate the model parameters of the accuracy test subjected to a quality assessment? Disagreements regarding the extracted data were resolved by consensus or through consultation with a third reviewer (PCS).
For studies in which the cost year was not specified, we assumed that the cost year was the same as the year of publication, a strategy that has been adopted in previous reviews of the literature (14). The monetary values of the results were compared to the year's Brazilian Produto Interno Bruto (gross domestic product [PIB]) per capita, which is available from the Brazilian Institute for Geography and Statistics (15).

Quality assessment
The methodology for the quality reporting of individual studies was assessed using some items of the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) (16).

Data synthesis
A qualitative narrative synthesis was conducted, and the study characteristics are summarized in the figures and summary tables.
Data analysis was performed with descriptive statistics, such as absolute frequencies (raw counts) for each category of the discrete variable and relative frequencies (proportions or percentages of the total number of observations).

' RESULTS
We identified 535 HEEs related to Brazil that were published in 1980-2013 ( Figure 1). Of those 535 studies, 43 (8%) addressed diagnostic tests . The number of studies and the proportion of full HEEs relative to the proportion of partial HEEs increased during the period under study ( Figure 2). Of the 43 HEEs related to diagnostic tests, the majority (74.4%, 32/43) were published in the last ten years (i.e., between 2005 and 2013), and 19 (44.2%) were full HEEs. Table 1 shows an overview of the main characteristics of the diagnostic tests addressed in the 43 selected studies. The technologies were applied to multiple clinical areas, including oncology, cardiology, endocrinology and infectious diseases. The most commonly evaluated tests were laboratory tests (37.2%, 16/43) and imaging tests (32.6%, 14/43). Most were non-invasive tests (51.2%, 22/43) and were utilized in the adult population (46.5%, 20/44). In terms of the intended purposes of the technologies evaluated, most (69.8%, 30/43) were diagnostic; diagnostic and treatment accounted for 25.6% (11/43), and screening, diagnostic and treatment accounted for 4.7% (2/43).
The authors declared conflicts of interest in 16 studies. However, when we evaluated the studies using the criteria proposed by Valachis et al. (13), we identified conflicts of interest in two additional studies. These conflicts of interest could include receiving remuneration in payment or in kind (e.g., stocks or shares) from the manufacturer as a result of any of the following: research support or employment contracts (salary, equipment, supply, reimbursement for participation in symposia and other expenses) or consulting services. In the article by David-Neto et al. (22), we identified manufacturer financing, which was not disclosed as a conflict of interest in the original article. Additionally, in the article by Bacchi et al. (43), we noted that some of the authors were employed by consulting firms, and this was not disclosed as a potential conflict of interest in the original article. Table 2 presents the main methodological characteristics of the 24 full HEEs, and Table 3 lists their compliance with recommendations for good reporting.
Ten (41.7%) of the 24 full HEEs reported their main outcome to be measurements related directly to the diagnosis strategy, for example, ''case diagnosed''. The others evaluated consequent diagnostic outcomes, such as case averted, surgery averted, complication averted, life years saved, and disability-adjusted life years (DALYs) avoided (Table 3). In one of the HEEs (26), the incremental cost-effectiveness ratio (ICER) reported was above the threshold of three times the per capita gross domestic product (GDP) per DALY gained, and the conclusion of that study regarding the technology was unfavourable (Table 3). Three studies (39,54,59) presented as their conclusion the measurement of the mean costeffectiveness ratio (CER), and two studies (34,52) only mentioned that the strategy was dominant without presenting ICER calculations.
In relation to the measurements of diagnostic effectiveness, among the 12 CEAs, nine studies (75%) listed the measurements of accuracy (sensitivity and specificity) as epidemiological parameters of the model, and in five of them (55%), the variations in the accuracy measurements were incorporated into the sensitivity analysis. The only CUA, which used DALY as an outcome, applied the estimates of sensitivity and specificity as epidemiological parameters in the decision model and the variations in these measures in the sensitivity analysis. Two studies (17%) referred to the application of methods defined in the literature to check the quality of the accuracy studies that underpinned the economic model (44,53).

' DISCUSSION
To the best of the authors' knowledge, this is the first systematic review of Brazilian HEEs in the area of diagnostic technologies. These HEEs were applied to diverse fields, including oncology, cardiology, infectious diseases and endocrine and metabolic diseases. The studies presented results that are relevant for decisions taken in clinical and public health situations in terms of diagnosis (69.8%), and some analyses were relatively broad, integrating screening and/or treatment strategies with diagnostic methods.
Regarding the quality of reporting, the Brazilian HEEs devoted to diagnostic testing performed reasonably well. Almost all studies satisfactorily stated the research question (100%), target population (95.8%, 23/24), competing alternatives (100%), source of effectiveness estimates (95.8%), source of cost estimates (87.5%, 21/24), and ICER (83.3%, 15/18). Strong compliance with HEE guidelines was also reported in a recent study that systematically reviewed all CUAs addressing diagnostic laboratory testing in the Tufts Medical Center Cost-Effectiveness Analysis Registry (3). Nevertheless, some issues still warrant further consideration. First, only 55.8% (24/43) of the studies declared the funding source, and second, only 37.2% (16/43) of the studies declared the potential for any conflict of interest. Missing details on these two important topics may affect the credibility and transparency of the results produced by HEE studies.
Three points were identified as requiring special attention and recommendations for improvement: 1) methods for estimating resource quantities and unit costs, 2) sensitivity analysis, and 3) discount rates. A previous review on the methodological issues of HEEs reached a similar conclusion in the 1990s, observing that many studies failed to report important issues, such as the study's perspective and sensitivity analyses (60).
Brazil published its official health technology assessment guidelines in 2009. Additionally, in 2011, the use of HEEs became compulsory for the incorporation of new technologies in the Brazilian public healthcare system. However, a clearly stated ''reference case'' that would increase the consistency of HEE methodology remains lacking. Despite the substantial number of HEEs produced (61), the local capacity to develop and use high-quality HEEs still needs to increase. The lack of technical expertise and the shortage of trained health economists were also reported in other middle-income countries (MICs) (62).
Challenges facing those conducting and using economic evaluations in many MICs include the scarcity, quality and accessibility of data; insufficient health economic research capacity; and limited funding. To conduct high-quality HEEs, far more investment in the training of researchers is needed, particularly for Master's degree programmes and topics such as study design, data collection and analysis. On-the-job training and mentoring of researchers are also very important and should be fully funded (63).   Quantiferon s -TB Gold-in-Tube, GDP (Gross domestic product) per capita of Brazil, in the price year cost reported by the study. When the study did not report the price year cost, it was assumed to be the year of the publication of the study. * Conventional staging includes physical examination, laboratory tests (lactate dehydrogenase [LDH], alkaline phosphatase, liver enzymes, bilirubin, renal function, calcium) and CT (chest, abdomen and pelvis).
** Algorithm A: based on a specific level of s/co ratio to determine the need for reflex supplemental testing. The s/co ratio chosen for cut-off (cut-off ratio) corresponded to the ratio that had the highest X95% concordance with positive results in the immunoblot (IB) test. Algorithm B: reflex testing by the supplemental nucleic acid amplification test (NAT) was required for samples that were positive or inconclusive in the screening test. A sample was considered as a true positive when both enzyme-linked immunosorbent assay (ELISA) and PCR were positive. When PCR was negative, IB was conducted.
*** Algorithm C: supplemental serologic reflex testing such as IB to confirm screening-test-positive results.
**** Strategy A: Repeat cytology every 6 months. Return to routine screening (every 3 years) only after two consecutive negative cytology results. In the case of a second abnormal smear, patients were referred to colposcopy. ***** Strategy B: colposcopy. Strategy C: HPV testing. In the case of a positive result, they were referred to colposcopy; otherwise, they had to repeat cytology. Strategy D: If the women were 30 years old or older, they were referred to colposcopy (as in Strategy B); otherwise, they had to repeat cytology (as in Strategy A). Strategy E: If the women were 30 years old or older, they were referred to HPV testing (as in Strategy C); otherwise, they had to repeat cytology (as in Strategy A).
****** Strategy E1: chest X-ray followed by smear microscopy for patients with chest-X-ray suggestive of tuberculosis; ******* Strategy E2: smear microscopy followed by chest X-ray for patients with one smear-positive or two smear-negative results; Strategy E3: microscopy and chest X-ray at the same visit.

=
The summary measure is a CER.
Diagnostic validity is the definitive stage of technology evaluations in terms of the probability of true positives (sensitivity) and true negatives (specificity) relative to a goldstandard technique. Indeed, measurements of validity are the principal data that determine the effectiveness of economic models (64).
Most of the CEAs (75%) reviewed clearly expressed the measurements of sensitivity and specificity used in the economic model, even when the final outcomes of the economic analysis were not accuracy as such (i.e., in terms of accurate or appropriate diagnoses) but were, instead, measurements arising from accuracy, such as deaths prevented.
Published accuracy studies should be rigorously evaluated for their quality before they can be sued to support the development of economic models. Two published sets of guidelines underpin this quality analysis: Quadas 2 and Stard (65,66); the former is especially directed towards evaluations in systematic reviews, whereas the latter is directed towards reporting in editorial terms. In this review, only two of the 12 HEEs (17%) cited the quality evaluation of the accuracy studies from which the parameters of sensitivity and specificity were extracted. This is an important omission in these publications and should be part of the reports because these are basic and essential measurements in costeffectiveness results and studies that discuss the costs and consequences of diagnostic technologies. Indeed, accuracy measurements are the main parameters that direct the analytical model, and therefore, methodological weakness in the estimation of measurements will directly impact the conclusions of the economic evaluation (9). Furthermore, evaluations of the quality of the studies underpinning economic models should be explicit and debated in a transparent manner (67). These parameters must be subject to sensitivity analysis because slight variations can strongly affect the economic conclusions (44).
A review of HEEs of the diagnosis of chronic renal disease suggested several topics that should be addressed when conducting economic studies on diagnosis: specification and clear definition of the measurements of the sensitivity and specificity of the tests and consideration of the consequences of including incorrect results in the economic model (6).
In the research protocol ''economic decision models for genomic testing strategies'', Peters et al. (2015) commented that it is rare for economic studies to provide suitable information about the methods used to evaluate how the evidence was identified (7). A similar situation was observed in this review, since only two of the 12 articles on costeffectiveness that were reviewed actually described the evaluation process used to determine the quality of the primary studies (44,53).
Otero et al. reviewed HEEs on diagnoses based on images and determined that the accuracy measurements were explained in most of the studies evaluated; however, in 23% of the studies, these measurements were not submitted to a sensitivity analysis. In the Brazilian review, five out of 12 cost-effectiveness studies reviewed and the only CUA varied in the accuracy measurements used in the sensitivity analysis. It should be stressed that this finding should be 100% (8).
This review demonstrates that Brazilian HEEs related to diagnostic tests are reasonably adherent to key recommended general methods for HEEs. However, quality evaluations of the accuracy studies from which the parameters of sensitivity and specificity are extracted is far from meeting current international standards. If these studies are to serve as an important source of information for local decision makers, important issues need to be addressed. First, they should improve the reporting of the methods used to estimate resource quantities and unit costs, sensitivity analysis, discount rate, and conflicts of interest. Second, the evaluation process used to determine the quality of the accuracy studies should be explicit and transparent, and the sensitivity analysis of sensitivity and specificity parameters should be clearly described. Finally, the accuracy measurements are the main parameters that impact decision analysis models' results and the conclusions of HEEs. Thus, they must be both valid and reliable to support decisions about the incorporation of new diagnostic technologies in Brazil.

Key Points for Decision Makers
The Brazilian HEEs in the area of diagnostic technologies require improvement in the quality of reporting of 1) methods used to estimate resource quantities and unit costs, 2) discount rates, 3) descriptions of sensitivity analysis methods, and 4) the reporting of conflicts of interest. Decision-making regarding the inclusion of diagnostic technologies should take into account the quality of the