Systematic Review of Health Economic Evaluation Studies Developed in Brazil from 1980 to 2013

Background Brazil has sought to use economic evaluation to support healthcare decision-making processes. While a number of health economic evaluations (HEEs) have been conducted, no study has systematically reviewed the quality of Brazilian HEE. The objective of this systematic review was to provide an overview regarding the state of HEE research and to evaluate the number, characteristics, and quality of reporting of published HEE studies conducted in a Brazilian setting. Methods We systematically searched electronic databases (MEDLINE, EMBASE, Latin American, and Caribbean Literature on Health Sciences Database, Scientific Electronic Library Online, NHS Economic Evaluation Database, health technology assessment Database, Bireme, and Biblioteca Virtual em Saúde Economia da Saúde); citation indexes (SCOPUS, Web of Science), and Sistema de Informação da Rede Brasileira de Avaliação de Tecnologia em Saúde. Partial and full HEEs published between 1980 and 2013 that referred to a Brazilian setting were considered for inclusion. Results In total, 535 studies were included in the review, 36.8% of these were considered to be full HEE. The category of healthcare technologies more frequently assessed were procedures (34.8%) and drugs (28.8%) which main objective was treatment (72.1%). Forty-four percent of the studies reported their funding source and 36% reported a conflict of interest. Overall, the full HEE quality of reporting was satisfactory. But some items were generally poorly reported and significant improvement is required: (1) methods used to estimate healthcare resource use quantities and unit costs, (2) methods used to estimate utility values, (3) sources of funding, and (4) conflicts of interest. Conclusion A steady number of HEE have been published in Brazil since 1980. To improve their contribution to inform national healthcare policy efforts need to be made to enhance the quality of reporting of HEEs and promote improvements in the way HEEs are designed, implemented (i.e., using sound methods for HEEs) and reported.

In recent decades, a large amount of local HEE studies have been published. There is a strong evidence of the upward stream of blossoming in HEE publications and its acceleration (8). The evolution of scientific literature in health economics published in Brazil between 1986 and 2007 has been evaluated and reported in the published literature (9)(10)(11). Recently, Brazil appeared among the top 15 countries in HE research, accounting for 1.7% of identified records, and has been identified as the South American country that published the largest number of HEE studies (8,12,13).
While a number of HEE have been conducted in Brazil, no study has systematically reviewed the quality of Brazilian HEE. The objective of this systematic review was to assess the state of the HEE research capacity development in Brazil and the ability to conduct good quality HEE. Specifically, this review evaluated the number, characteristics, and quality of reporting of published economic studies in a Brazilian setting.

mateRiaLS aND metHODS
This study followed the guidelines for systematic review of HEE studies published by the Centre for Reviews and Dissemination (CRD) and the preferred reporting items for systematic reviews and meta-analyses statement (29,30). The protocol is available from the authors on request.

Systematic Search and identification of Relevant Studies
A broad and exhaustive strategy search was formulated in order to identify all relevant studies published between January 1980 and December 2013. We systematically searched the following electronic data bases: MEDLINE (via PubMed), EMBASE, Latin American, and Caribbean Literature on Health Sciences Database, Scientific Electronic Library Online, NHS Economic Evaluation Database, HTA Database (CRD), Bireme, and Biblioteca Virtual em Saúde Economia da Saúde; citation indexes: SCOPUS, Web of Science, and the Sistema de informação da Rede Brasileira de Avaliação Tecnologia e Saúde (SISREBRATS). We also performed manual searches from the reference lists of included articles, and all issues of the Brazilian Journal of Health Economics (BJHE), a non-indexed journal in the previously mentioned databases in 2013.
The search strategy was reviewed by a librarian specialist and combined subject headings (MeSH and EMTREE) and free text terms ("Health Economics" OR "Economics, Hospital" OR "Economics, Medical" OR "Economics, Nursing" OR "Economics, Pharmaceutical" OR "Economics" OR "costs and cost analysis" OR "Cost" OR "Cost savings" OR "Cost of illness" OR "Analyses, Cost-Benefit" OR "Analysis, Cost-Benefit" OR "Cost-Benefit Analyses" OR "Cost Benefit Analysis" OR "Analyses, Cost Benefit" OR "Analysis, Cost Benefit" OR "Cost Benefit Analyses" OR "cost Effectiveness" OR "Effectiveness, Cost" OR "cost effectiveness analysis" OR "cost-Benefit Date" OR "cost Benefit Date" OR "Date, Cost-Benefit" OR "cost Benefit" OR "Benefits and Costs" OR "Costs and Benefits") for "economic/cost" concept with subject headings (MeSH and EMTREE) and free text terms ("Brazil" OR "Brazilian" OR "Brazi*") for "Brazil" concept. Keywords were matched to database specific indexing terms, taking into account the change in the indexing or classification of economic studies in different databases.

eligibility: Selection criteria
Articles were included if they were partial or full HEE according to internationally recognized criteria (31,32), referred to the Brazilian setting, and at least one of the authors was Brazilian and affiliated to a Brazilian institution. Multicenter studies, where Brazil was one of the participating countries, as well as studies conducted on Brazil by foreign authors, were excluded.
Studies were considered partial HEE if they examined only costs (cost description), described costs of a particular disease to society (cost of illness), described costs and outcomes of a single service or program (cost-outcome description), described financial consequences of technology adoption [budget impact analysis (BIA)] or compared only costs of two or more interventions (cost analysis). Studies were considered full HEE if they compared costs and consequences of two or more healthcare interventions alternatives, including cost-consequences analysis (CCA), cost-minimization analysis (CMA), cost-effectiveness analysis (CEA), cost-utility analysis (CUA), and cost-benefit analysis (CBA).
Abstracts, editorials, letters, posters and congress communications, methodological, discussion and review articles, and economic evaluation of other than health technologies (for example, environment) were excluded.
The titles and abstracts of identified citations were screened for relevance independently by two reviewers (Tassia Cristina Decimoni and Roseli Leandro). Disagreements were resolved through discussion or through consultation with a third reviewer (Patrícia Coelho de Soárez). Full texts of selected and those for which inclusion was in doubt were retrieved and independently screened by both reviewers.

Data extraction
Two reviewers (Tassia Cristina DecimoniTCD and Roseli Leandro) independently extracted data from each of the included studies on year and journal of publication, economic evaluation type, category of technology assessed (drugs, vaccines, equipment, clinical, surgical and diagnostic procedures, public health and health promotion programs), objective of the technology assessed (treatment, prevention, screening, and diagnosis), health problem studied (International statistical classification of diseases and related health problems, 10th revision, ICD-10) (33), first author affiliation (academy, government, research institutes, health organization, consulting, pharmaceuticals or equipment industry, international body), region of the first author, source of funding (research funding agencies, government, consulting, pharmaceuticals, or equipment industry), and authors' conflict of interest. Conflict of interest was defined according to Valachis et al. (34) who argued that an author may need to declare having a conflict of interest in if she/he has received remuneration in payment or in kind (e.g., stocks or shares) from the manufacturer as a result of any of the following: research support or employment contract (salary, equipment, supply, reimbursement for participation in symposia, and other expenses), or consulting services.
In addition to the above, the standardized extraction form used also contained 17 questions from a systematic review of quality assessment tools (35), and the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) instrument (36). These questions were intended to assess the quality of reporting of full HEE. Quality was defined as the extent to which a study complied and reported items included in the quality assessment tool and CHEERS checklist mentioned above (35,36). The quality of the sources of evidence used in the studies was assessed with the hierarchy proposed by Coyle and Lee (37,38). Where data sources ranked 1 are considered to be the most appropriate source (highest quality), and those assigned a rank of 6 are considered the least appropriate (lowest quality) (39) ( Table 1). Disagreements on the extracted data were resolved through discussion or through consultation with a third reviewer (Patrícia Coelho de Soárez).

Data Summary
Data were summarized using qualitative narrative synthesis. The study characteristics are summarized in figures and summary tables.

Data analysis
Data analysis was performed with descriptive statistics such as absolute frequencies (raw counts) for each category of the discrete variable, relative frequencies (proportions or percentages of the total number of observations), along with analytic statistics that included Pearson correlations to investigate the relationship between the quality of reporting of the full HEE and the publication time period (1980-2005, 2006-

Literature Search
In total 11,841 records were identified from database searches, and 105 further articles were identified through hand-searching in BJHE, SISREBRATS, and other sources. Figure 1 depicts a flow diagram with full details of searches output, and reasons for inclusion/exclusion. We identified 9,304 non-duplicate citations, of which 721 were recognized as potentially relevant and full papers were retrieved. Out of the 721 studies 186 of them were excluded, reasons for exclusion included: thesis (50 studies), not taBLe 1 | Hierarchies of data sources for health economic evaluation studies modified from Coyle and Lee (37)(38)(39).

Rank
Data components clinical effect sizes, adverse events, and complications 1+ Meta-analysis of RCTs with direct comparison between comparator therapies measuring final outcomes 1 Single RCT with direct comparison between comparator therapies measuring final outcomes 2+ Meta-analysis or RCTs with direct comparison between comparator therapies measuring the surrogate outcomes Meta-analysis or placebo-controlled RCTs with similar trial populations, measuring final outcomes for each individual therapy 2 Single RCT with direct comparison between comparator therapies measuring the surrogate outcomes Single placebo-controlled RCTs with similar trial populations, measuring final outcomes for each individual therapy 3+ Meta-analysis or placebo-controlled RCTs with similar trial populations, measuring the surrogate outcomes 3 Single placebo-controlled RCTs with similar trial populations, measuring the surrogate outcomes for each individual therapy 4 Case control or cohort studies 5 Nom-analytic studies, for example, case reports, case series 6 Expert opinion Resource use 1 Prospective data collection or analysis of reliable data for specific study 2 Recently published results of prospective data collection or recent analysis of reliable administrative data-same jurisdiction 3 Unsourced data from previous economic evaluation-same jurisdiction 4 Recently published results of prospective data collection or recent analysis of reliable administrative data-different jurisdiction 5 Unsourced data from previous economic evaluation-different jurisdiction 6 Expert opinion costs 1 Cost calculations based on reliable databases or data sources conducted for specific study-same jurisdiction 2 Recently published cost calculations based on reliable databases or data sources-same jurisdiction 3 Unsourced data from previous economic evaluation-same jurisdiction 4 Recently published cost calculations based on reliable databases or data sources-different jurisdiction 5 Unsourced data from previous economic evaluation-different jurisdiction 6 Expert opinion Direct utility assessment for the specific study from a sample either: (a) of the general population (b) with knowledge of the disease(s) of interest (c) of patients with the disease(s) of interest Indirect utility assessment from specific study from patient sample with the disease(s) of interest, using a tool validated for the patient population 2 Indirect utility assessment from a patient sample with the disease(s) of interest, using a tool not validated for the patient population 3 Direct utility assessment from previous study from a sample either: (a) of the general population (b) with knowledge of the disease(s) of interest (c) of patients with the disease(s) of interest Indirect utility assessment from previous study from patient sample with the disease(s) of interest, using a tool validated for the patient population 4 Unsourced utility data from previous study-method of elicitation unknown 5 Patient preference values obtained from a visual analog scale 6 Delphi panels, expert opinion RCT, randomized control trial.  (18), and other (11 studies), see Figure 1 for detailed description. Scientific papers derived from excluded thesis and reports were included.

Study characteristics
The publication of HEE studies in Brazil started in the 1980s. Since then, there has been an upward trend with a slight increase at the end of the 1990s, and a sharp increase in 2007 with the publication of 356 (67%, 356/535) articles (Figure 2). A total of 535 studies were identified as suitable for inclusion in this review and their characteristics are described in Table 2.  2; Figure 3). Out of the 535 included studies, nearly half (n = 248, 46.3%) did not report the type of HEE study performed. Among these, 228 were partial and 20 were full HEE.
Regarding the geographical distribution of the first authors, southeast region stands out as a major producer of HEE (73.6%), followed by south (12.5%), and northeast (8.2%). The proportion of publications by regions remained constant during the study period. São Paulo and Rio de Janeiro were the Brazilian states more productive, 51.6 and 14.6%, respectively.
Two hundred and thirty-four studies (44%, 234/535) reported the funding source, among these, 12 (34) to have a potential conflict of interest due to authors being industry or consultancy firm employees. Similarly, applying Valachis criteria to all 535 included studies, 84% (449/535) would be considered to not have a conflict of interest and 16% (86/535) could be considered to have a conflict of interest. Identified reasons for potential conflict of interest were: 49% (42/86) were developed by consultancy firms and industry; 45% (39/86) had at least one author contractually employed by the industry or funded by it; and 6% (5/86) were related to consultancy work.
Compliance with international recommendations for good reporting in the 197 studies identified as full HEE are presented in Figure 5. Most studies complied with the following items clearly stated: the research question (100%), competing alternatives (99%), primary outcome measure (95%), source of effectiveness estimates (94%), type of model (92%), and economic study design (90%).

Studies by Reporting Quality and Quality of the Sources of evidence
Thirty-two percent (63/197) of the studies did not state a perspective for their analysis. Only 37% (73/197) described the methods for the estimation of total volume of healthcare resources used in The reporting quality association with the publication period was statistically significant (P < 0.001). The reporting quality increased progressively during the study period. Most recent periods showed better reporting quality. Studies published between 2010 and 2013 showed better reporting quality compared with those published between 2005 and 2009 and those published between 1980 and 2004. No association was found between the reporting quality and the study source of funding. Moreover, positive association was observed between the reporting quality and the variable conflict of interest. Studies with conflict of interest are associated with a better reporting quality (P < 0.001).
The quality of the sources of evidence used in the studies performed in the Brazilian setting was analyzed in depth using the hierarchy proposed by Coyle and Lee (37)(38)(39). Figure 6 presents a graphical representation of results from this analysis. Our findings suggest that poorer quality of information was available for estimating utilities values than costs, resource use, and clinical effect size. No study directly (e.g., via a health state preference evaluation exercise) or indirectly (e.g., used utility   values from an alternative patient sample but with the disease of interest) performed utility assessment. Twenty-two percent of the studies estimated utilities parameters from a previous study, 37% had data source or method of elicitation unknown, 12% was based on expert opinion, and 29% was not possible to evaluate. In contrast to utility value estimates data on costs, resource use and clinical effect size were mostly estimated using high-ranked evidence. For instance, more than 50% of the studies used cost information from sources where quality was ranked as 1+, 1, 2+, 2. Costs calculations were mainly based on reliable administrative database or data sources conducted for the specific study, and on recently published cost calculations based on reliable databases.
Similarly, in 58% of the studies clinical effect sizes were estimated from meta-analysis or RCTs.

DiScUSSiON
To the best of our knowledge, this the first systematic review describing the number, characteristics, and quality of reporting of HEE studies in Brazil. The absolute number of all HEE published in Brazil between 1980 and 2013 (n = 535) is substantial. The volume of full HEE (n = 197) directly relevant to Brazilian settings is considerably higher than the number of HEE identified in other countries   (14,15,17,19,21,(24)(25)(26)(27)(28)40). Conversely, Brazil still produces a lower number of HEE compared with high-income countries such as United Kingdom (n = 675), Canada (n = 300), German (n = 283), and Australia (n = 245), where economic evaluation is well established and formally used for regulating reimbursement policies for the adoption of new technologies (16,20,41). BRICS (Brazil, Russia, India, and China) share in global wealth grew tremendously over past decade. In parallel to that, healthcare spending, health economic productivity, and research funding for health economics shifted toward low-and middleincome countries, especially in top emerging BRICS. These circumstances created drivers for further development of health economics in these countries (8,(42)(43)(44).
In recent years in Brazil, as in other LAC countries, there has been increased interest in incorporating HEEs as a formal tool to inform decision-making processes (12). This systematic review reveals a steady growing trend in the number of HEE studies being published in the last 6 years (2008-2013). Interestingly, this phenomenon is also observed in Colombia (26).
The steady growth in the economic evaluation literature relevant at national level in Brazil-the focus of this analysisreached a peak in the year 2007. The reason for this may be multifold; firstly, from as early as 1980 Brazilian researchers, funders, and public health system users have successfully promoted the conduct and use of HEE studies. Secondly, internationally there was been an increased interest in HEE studies demonstrated by a continuous growth of published articles and books, as well as the creation of several new specialist journals in this field (45). A third crucially important factor is some recent initiatives of the Brazilian Ministry of Health such as CITEC and CONITEC establishments. The development of this formal structure for regulating reimbursement policies for the adoption of new technologies, provides a strong incentive to promote the implementation and publication of a growing number of HEE studies.
Findings from systematics reviews suggest that the increase in the number of HEE studies published internationally may be related to requirements to use information from these studies to inform reimbursement decisions (41).
As reported in earlier systematic reviews of HEE studies in Latin America (17,26,27), our findings indicated that CEA was the most prevalent (39.1%, 77/197) study type for full HEE conducted in Brazilian settings between 1980 and 2013. This might be explained by the relative simplicity of the CEA approach compared with CUA that requires developing robust methodology to value health state preferences.
Despite this, the current review found considerable growth in CUA for the Brazilian setting from 2005 onward (P = 0.028), in line with a growth in CUA observed in the international literature (41). The absolute number of CUA in Brazil (n = 40), however, is still small. This may, in part, be explained by the fact that CUA is more labor and resource intensive than CEA. In addition to this, the 2009 Brazilian HTA guideline gave equal weight to CEA and CUA (46). While CONITEC came into force in 2011 it did not update its methods guidelines to recommend the use of CUA. This is in contrast with current recommendations in a number of countries worldwide (e.g., Australia, Canada, Ireland, New Zealand, Scotland, Sweden, and United Kingdom) (47). CUA was indicated as the preferred type of study, only in the update of the Brazilian HEE guideline published in 2014 (48).
National Incorporation of Technologies in SUS has a formal requirement for BIA alongside a full HEE. In spite of this, the number of published BIA is still limited (n = 15), and most of the so-called BIA submitted to CONITEC are cost studies with 2-5year annual costs for a specific cohort instead of a real estimate of the financial impact of a new intervention for the Brazilian healthcare system. These findings are consistent with a review of BIA studies (49).
Like other HEE in Latin America, Africa, and South/West Asia most studies focused on infectious disease (14,21,23,36). The majority of these studies focused on vaccines, driven largely by increased investment on CEA and related activities by major global health players such as Bill & Melinda Gates Foundation, GAVI alliance, and the World Health Organization (48).
We found that 75.5% (297/535) of the studies were published in medical and public heath journals. In reviews of Spanish, Iran, German, and South African, HEE studies covering earlier years of publication, the majority of studies (77, 77, 79, and 88%, respectively) were also published in medical journals (13,21,24,49). This high numbers may be related to lower publication standards in medical and public health journals when compared with specialized health economic journals.
Even though researchers prefer to publish their research in international journals, with higher impact factors, and a wider audience. Most of the studies (72.5%) were published in Brazilian journals. Among the 12% published in health economics specialized journals, 8% were published in a national health economic journal, which was not even indexed until 2013.
This may be related to publication requirements as editors of Brazilian journals have less stringent requirements to make use of international methodological guidelines as part of their peer reviewing processes. Consequently, articles with an inferior quality of reporting and equivocal methodological quality are published. Our data also suggest a need for improvements in the peer review process, especially among journals with limited experience publishing economic evaluations (50).
The majority (75%, 148/197) of identified full HEE studies were published between 2008 and 2013, and the reporting quality increased progressively during the study period (P < 0.001). Although overall quality of reporting was considered satisfactory, the review highlighted a number of issues associated with the reporting and methodological quality of the included HEE studies.
Two issues on quality of reporting deserve further attention. Firstly, reporting of funding source, 56%, 110 of the 197 full HEE studies identified here did not state their source of funding. This is in line with findings from reviews on the state of HEE studies conducted in South Africa (45%, 49/108) (24), Nigeria (55%, 24/44) (21), and Zimbabwe (62%, 16/26) (19). Secondly, and directly related to reporting source of funding is declaring potential for any conflicts of interest. Only 36% of the articles reported conflict of interest. As highlighted by Valachis et al. (34), conflicts of interest may be directly related to sources of funding. This is a phenomenon that has been researched by several authors (51)(52)(53), if studies are funded by the healthcare industry this could have a direct impact on the conclusion drawn. Industrysponsored HEE are believed to be more likely to report incremental cost-effectiveness ratios that favor products manufactured by the sponsor. Missing details on these two crucial pieces of information described above may impact on the credibility and transparency of results from HEE studies.
Two final points are identified as requiring further consideration and recommendations for improvement, these are (1) methods for the estimation of resources quantities and unit costs and (2) methods for the estimation of utilities parameters. While the majority of studies provided a source for resource utilization and costs, they omitted details on the identification and quantification of categories of resources and estimation of unit costs. This is in contrast to HEE guidelines (54)(55)(56), which indicate that all the relevant quantities of resources should be measured in a correct and transparent manner, and reported separately from the prices (unit costs) of those resources. This lack of detailed reporting on the quantification of healthcare resources and methods used for their valuation limits the ability to replicate costing processes in future studies.
The final critical issue identified for discussion was the poor quality of information used for estimating utilities parameters. Authors' often reported their main sources of information for the measurement and valuation of preference-based outcomes, these sources of evidence, however, were studies ranked as low quality. Many studies reported having used international data from previously published studies or expert opinion. This is in contrast with international guidelines on HEE which state that utility values obtained from other countries are, in general, not transferable because of cultural differences (57). Recently, in 2011, two of the most widely used generic preference-based utility instruments-EQ-5D and SF-6D-were cross-cultural adapted and validated, in addition societal preferences weights were estimated for the Brazilian population in 2013 (58-61). We expect that current efforts to estimate Brazilian utility weight "tariffs" will increase consistency in quality-adjusted life year calculations in future HEE studies.
One limitation of our study was that we critiqued the reporting, and not necessarily the actual manner that authors conducted their studies. However, this review was useful to assess the practice of HEE in the Brazilian setting.
cONcLUSiON This review identified that an increasing number of HEE studies are being conducted and published in Brazil. Their reporting quality has increased progressively during the study period.
Overall, the quality of these HEE studies is satisfactory, but we identified key areas where significant improvements could be made such as: reporting of funding source, conflict of interest, methods for the estimation of resources quantities and unit costs, methods and source of evidence to estimate utility parameters utilities parameters. Our findings can contribute to improve the way HEE studies are designed, implemented, and reported in Brazil.

cONSeNt FOR PUBLicatiON
There are no any individual person's data.

etHicS StatemeNt
There are no human participants involved.

aUtHOR cONtRiBUtiONS
All authors drafted the systematic review protocol. TD, RL, and PS conducted the search, selection of records, and data extraction. Quality appraisal was conducted by TD, LR, and PS. All authors have read and approved the final manuscript.