The Role of Systematic Reviews as Part of Decision-Making Process : Descriptive Study of the Brazilian Experience

Systematic reviews (SRs) form the first step in health technology assessment and coverage decision processes [1]. They are used to achieve better understanding of a topic or problem; to define priorities for assessments and new research; to support decisions relating to reimbursement for medications; to form the basis for economic assessment models; and to reduce the barriers against translation of knowledge [1-10].


Introduction
The rapid growth of health technologies and the range of therapeutic options have given rise to a need for decisions on healthcare service coverage.For effectiveness and safety to be attained in relation to therapeutic alternatives that are chosen, systematic searches and critical analysis of the literature are fundamental.Systematic reviews (SRs) form the first step in health technology assessment and coverage decision processes [1].They are used to achieve better understanding of a topic or problem; to define priorities for assessments and new research; to support decisions relating to reimbursement for medications; to form the basis for economic assessment models; and to reduce the barriers against translation of knowledge [1][2][3][4][5][6][7][8][9][10].
Few studies have described practical experiences relating to using SRs, especially in developing countries.Such studies are important for obtaining evidence about what SRs are used for, through seeking ways of measuring these results.
Given that in implementing the field of health technology assessment (HTA) in the MoH, the partnership with the Brazilian Cochrane Center was the first step towards producing SRs on topics that were prioritized by the national administrator, it is important to describe how these SRs have been used as a source of evidence in the process of assessing and incorporating health technologies at national level within the Brazilian National Health System (SUS).
The aim of this paper was to describe how SRs from the Cochrane were used in composing HTA reports in the MoH's and as information for decision-making by the permanent consultative committees of the MoH between 2004 and 2011.

Materials and Methods
A descriptive analytical study was conducted using retrospective data covering the period 2004 to 2011, before the National Committee for Health Technology incorporation (CONITEC -Lay 12401/2011).
The markers observed in order to measure the use of SRs are described in the following.Marker A refers to the presence of a bibliographic citation Open Access 2 of a Cochrane SR published in the Cochrane Library/Plus or produced by the Brazilian Cochrane Center, which was included in the reference list of the HTA report.Marker B refers to registration of a SR abstract in the information system of the Brazilian HTA Network (REBRATS).Marker C refers to SRs sent as support material to the permanent consultative committees of the MoH.Marker D refers to SRs used as a source of information for decision-making by the permanent consultative committees of the MoH.The MoH's permanent consultative committees taken into consideration in this study were the Technology Incorporation Committee (CITEC -after 2011 became CONITEC) and the Working Group for Clinical Protocols and Therapeutic Guidelines (PCDT).Over the period evaluated, the HTA Unit of the Department of Science and Technology (DECIT) was in charge of these committees and was responsible for producing HTA reports to provide support for the decision-making process.Use of SRs in these permanent committees' work was ascertained through analyzing the records of DECIT and CITEC (Chart 1), and through the first author's experience as a full member of the committees.
To correlate the markers observed in this study with the concept of use of research (in these case SRs) the conceptual model of Hanney et al was used [2].According to these authors, the stock or reserve of knowledge can interact with the political, professional, industrial and society environments.
Starting from this concept, these authors proposed several stages for feeding the stock of knowledge and supplying information for decisionmaking processes.The stages and interfaces of how the products and research results can be made use of by healthcare policymakers are as follows: i) identification of research questions; ii) selection and specification of the research project; iii) inputs for the research; iv) research development process; v) production of primary products from the research; vi) dissemination of these results; vii) backing for policy formulation; viii) adoption by professionals and the public; and ix) return of the final results to the store of knowledge.
According to these authors, SRs are used in policy formulation and in their adoption by professionals or political engagement.SRs are products from research into support for policies, and also, have the function of contextualizing this knowledge within the social environment.
For the purposes of this study, the stages of use of the conceptual model of Hanney [2] were taken to be the stage of dissemination of the results from the research and the phase characterized as backing for policy formulation.
Marker B, i.e. registration of SRs in the information system of the Brazilian HTA network (REBRATS) [13] (www.saude.gov.br/rebrats), was allocated as being in the stage of dissemination of the results from SRs.
The markers A, C and D were situated as indicators of results from SRs that provided backing for policy formulation.
For the present study, HTA reports were taken to be technical notes from rapid reviews (TNRRs) and scientific technical reports (STRs).TNRRs are synthesis documents that are produced by the internal team at DECIT within 20 days, and they place value on SRs, technology assessments from international agencies and randomized controlled trials (RCTs).STRs are also synthesis documents, produced within four months either by the DECIT team or ordered from institutions within REBRATS.These place value on all types of quality study, contain structures searches and a table showing a critical evaluation on the evidence.
The markers observed were identified through reading and analyzing official documents and records from the MoH, as listed in Chart 1 (Chart 1: Description of the data sources as additional files) The eligible analysis was systematic reviews ordered from the Brazilian Cochrane Center and all the SRs (Cochrane Library/Plus and Brazilian Cochrane Center) cited in the HTA reports.
The method for documenting the use of SRs followed the protocol described in Chart 2. Presence of one or more markers was taken to be a record of use for each marker defined.(Chart 2: Correlation between the markers and information sources of the retrospective study).Open Access

3
The document analysis process also included the first author's experience as a full member of the MoH's Technology Incorporation Committee between 2006 and 2011 and as the coordinator of HTA unit.The authors had free access to all the documents and official records necessary for conducting this study.

Results
Between 2006 and 2011, 77 SRs were ordered from the Brazilian Cochrane Center by the MoH, and 65 were concluded.There are different forms of use of SRs within the MoH's scope of the MoH according to the markers and definitions used.
In relation to marker A, i.e. bibliographic citations of SRs included in the composition of HTA reports (Table 1), a total of 83 reports with citations of all SRs were observed in 262 reports, thus corresponding to 31% (83/262) of the reports produced between 2006 and 2011.No records covering the period before 2006 were found for analysis.SRs obtained via the Cochrane Library/Plus were cited in 75 HTA reports between 2006 and 2011.SRs ordered from the Brazilian Cochrane Center (BCC) were cited in eight reports over the same period.
In relation to marker C, 27 Brazilian SRs (BSR) reached the definition of material for support to the consultative committees (Scorer C in the Graph 1).Eight were also used as information for decision-making in the technology incorporation committee (Scorer D) and seven were cited as references in HTA reports (Scorer A).These three markers were present in the following topics: gastric band implantation surgery for treating severe obesity; efalizumab for treating psoriasis; imatinib for treating chronic myeloid leukemia; and coated stents versus conventional stents for treating arterial insufficiency.
As shown in Graph 1, 38 Brazilian SRs (BSR) were available in full texts from the information system of REBRATS (SISREBRATS -scorer B).Since 2008, abstracts or SRs have been included in the database of the REBRATS.These BSRs can be accessed by individuals responsible for decision-making in the MoH and by the general public.
Most of the SRs were directed towards evaluating medications.Thirty-two of the SRs assessed related to medications, followed by nineteen relating to devices and equipment.All of the SRs evaluated efficacy and safety.(See additional file -Chart 3 Brazilian systematic reviews produced by the Cochrane Center on request from Ministry of Health, according to the categories selected, from 2005 to 2011).
The eight SRs were used in decision-making by technology incorporation committees are listed in Chart 4. The concordance analysis was conducted, comparing the recommendations in the SR and the decisions made by the committee.
From Chart 4, it can be seen that the majority of the decisions had a high degree of concordance with the conclusions in the SR, except in relation to the urinary sphincter, for which there was discordance, and in relation to the treatments recommended for pulmonary arterial hypertension, for which the two medications under analysis were indicated.It should be borne in mind that in addition to the SR, the committee had an arsenal of information relating to the relevance of the technology for the healthcare services, in the light of the costs and reports from the sectors responsible for care policies.
The seven BSRs that were cited in eight HTA reports according to the type of report, title, content and conclusion of the report.The data showed that all the SRs cited were ordered by the MoH.The conclusions of five HTA reports followed the results of the BSR.Among these, the case of macular degeneration in elderly individuals can be highlighted.Two reviews were cited because the content of the introduction was used, referring to the characteristics of the disease -psoriasis and rheumatoid arthritis (Chart 5).Open Access

Summary of findings
The findings from the MoH's experience reveal that SRs were used, but the small number of citations may have been due to many situations.For example, the topic requested might not have been a subject covered by the SRs; the SRs have not concluded; or the SRs results are inconclusive.
All the topics were prioritized by the MoH, but the decision-making moment within the MoH's permanent consultative committees could not always be reached.For example there was a lack between the time of the SR conclusion and the time of the meeting of decision-making by the committee.

Validity of results
The limitations of the present study comprised difficulties organized the records of the HTA reports and SRs produced, along with documents used in incorporation decisions made by the consultative committees over the period studied.Since this was a descriptive study using retrospective data, there was the possibility of memory bias and measurement bias, since the registers were not planned for this purpose.

Implications of the findings
Since 2004, the MoH has had a partnership with the REBRATS institutions to support decisions made by the MoH's Technology Incorporation Committee (CITEC), and for MoH to draw up HTA reports.The perspective adopted was that SRs supplied information of greater consistency for decision-making relating to the healthcare system, since they increased the magnitude and precision of the results relating to the outcomes assessed.
For seven years, the Brazilian MoH has been funding the production of SRs on topics prioritized by healthcare administrators for decisionmaking, through directly ordering them or through public calls with the National Council for Scientific and Technological Development [13].The partnership the REBRATS institutions included provision of master's degree courses as professional training and courses on evidence-based medicine for technical specialists at the MoH [14,15].However, contextual factors involve the degree of use, as the permanence of barriers as lack of prioritization process from the decision-makers and informal research body to compile HTA reports.
In this paper, we have suggested that markers should be used to prospectively monitor the use of SRs funded by the MoH.Analysis on the impact and use of SRs or HTA reports can be fundamental for avoiding duplication and expenditure on the decision-making process.Empirical studies on the use of research and on decisions informed by the results from research started to be published internationally in 1999.From a conceptual point of view, a variety of models for analyzing the use of research in policy formulation have been reported, with the conclusion that the process is diffuse and difficult to measure [16,17].

Forschen Sci
The majority of studies on models and levels of use of research in formulating policies have been conducted in developed countries, covering all types of research [18,19].
The impact of the ten years of existence of the National Coordinating Centre for Health Technology Assessment (NCCHTA) was evaluated by a research team [20], through applying interviews and analyzing documents.It was concluded that HTAs produced for the NICE had greater impact, in terms of application to decision-making in the British system, but that few studies had estimated returns in monetary values, in terms of costbenefit analysis or estimates of cost reduction.The review studies cost 40,000 pounds on average and clinical trials started at one pounds.A total of 133 projects were analyzed.impact indicators applied were publication in peer-reviewed journals, qualification of personnel, funding for new projects, presentation of results to different audiences (both academic and in practice).Qualitative analysis indicated that the professional body at national level and the agencies were the biggest beneficiaries.Overall, the times taken to present results, the quality of the results and the closeness of links with decision-makers were impact factors.
The impact of 21 HTA reports produced by a Canadian committee [21] was evaluated through estimates of cost reductions applied to the results and recommendations.Document analysis, interviews, questionnaires and databases were used.The results demonstrated that only three reports influenced policies and that cost minimization studies promoted savings of between US$ 16 and 27 million per year.It was concluded that precise estimates of the impact are rarely possible, but that systematic analysis of documentation on the effects was viable.
Through empirical studies some authors demonstrated the factors that influenced the use of healthcare research and highlighted the following as the most frequent factors: i) interactions between researchers and policy formulators through creation of formal advisory networks and committees or informal relationships; ii) research that brought together beliefs, values and interests or policy objectives; and iii) social pressure [21,22].
The factors that influence the use of research evidence by public policymakers are funding, human resources, time, evaluators' skills and access to databases [23].
The implication of this study for practice is that SRs are advantageous, since they cost 30,000 reais, which is around 10% of the cost of a review paid for in England and 1% of the cost of an RCT.In this case, the cost of a decision should also be taken into account, given that there is a series of stages in producing an assessment that will inform the decision-making process [24].
Meta-assessments on the assessment process itself have been conducted by consultancy companies such as Charles River Associates [25].In analyzing 15 HTA organizations as case studies representing countries that included France, United Kingdom, Germany, Canada, Australia, South Korea and Brazil, they questioned the quality, duplication and costs incurred in the assessment processes used in the regulatory processes of health technology incorporation.
The mixed method study demonstrated a total of 1,502 new and updated reviews were produced by the 20 National Institute for Health Research (NIHR) -funded Cochrane Review Groups between 2007 and 2011.These Cochrane reviews have an impact on clinical guidance development and may have influenced the conduct of primary research.Limited evidence was found to demonstrate that Cochrane reviews had had a direct effect on clinical practice, but they may have an indirect impact on health care through their role in clinical guidance [26].
Measuring the economic impact of the results from SRs used in healthcare decision-making will need investigation to aim promote efficiency within the healthcare system [27].

Conclusion
The present study demonstrated that systematic reviews were used in HTA reports and to support some decision-making process.These practices can be increase due to the new law in Brazil about the incorporation procedures for public health systems.Some barriers were found, as the topic requested might not had been a subject covered by the SRs; the results of SRs was delivered after decision-making, or the SRs results are inconclusive.
However, the notion of using evidence for decision-making is legitimated through the Brazilian federal constitution.Brazilian legislation has created rules based on scientific evidence for incorporation decision making process (Law 12401/2011).S0065 lection of technologies for assessment followed three basic criteria: those under development or at the pre-registration phase; those already incorporated in the healthcare system but with a need for economic evaluation or assessment of their effectiveness for new indications; and those already registered but not yet incorporated.
The element that generated use of SRs for supporting the MoH's decisions was the prioritization process, which took into consideration the three criteria described above and the demands from the National Committee for Health Technology Incorporation (CONITEC, reformulated through Law 12401/2011).
Use of evidence is fundamentally favored by communication mechanisms directed towards different target audiences.Decentralization of use is another aspect highlighted.For this, the interactions with state and municipal administrators need to be strengthened from the perspective of managing knowledge and identifying factors for stimulating greater use of SRs.
The implications for research cover periodic updating of SRs and the need to estimate impacts through expanding the theoretical approach from the point of view of translation of knowledge.There is also a need to measure the economic impact of systematic reviews for promoting the efficiency of the healthcare system, which will be the subject of a future study.
n H U B f o r S c i e n t i f i c R e s e a r c h Citation: Elias FTS, Atallah AN (2016) The Role of Systematic Reviews as Part of Decision-Making Process: Descriptive Study of the Brazilian Experience.J Epidemiol Public Health Rev 1(5): doi http://dx.doi.org/10.16966/2471-8211.127 Marker A: Bibliographic citation of all SRs (Brazilian and general) included in the composition of HTA reports, as described in the methods section a. Survey of the list of SRs ordered, in the management reports (F1) and register of projects selected in announcements (F2).b.Survey of the list of HTA reports (F3).c.Complete reading of the HTA reports (F3), with identification of citations of Cochrane reviews.d.Analysis on the titles of the Cochrane reviews cited and stratification according to whether the review was obtained from the Cochrane Library/ Plus or from the Brazilian Cochrane Center (BCC).Marker B: Systematic reviews registered in the information system of the Brazilian HTA Network (REBRATS).a.Search for and reading of abstracts of BCC reviews registered in the REBRATS information system (F10).Marker C: SRs sent as support material to the permanent consultative committees of the Ministry of Health.b. Reading of the minutes of CITEC meetings (F7).c.Search for the name of the technology and studies compiled by DECIT in the spreadsheet (F7) and CITEC's conclusions registered in the spreadsheet (F6).d.Checking of the spreadsheet (F4) and second checking of the spreadsheet (F6).e. Analysis on the records of CITEC's decision-making (F8) f.Search for studies in DECIT's files (F3) g.Search for CITEC's decisions (F11) a. Reading of official memoranda and e-mails (F9) to investigate sending of reviews to aid the working group for clinical protocols and therapeutic guidelines.Marker D: Brazilian SRs used as information for decision-making by the permanent consultative committees of the ministry of Health a. Search for the name of the technology and studies compiled by DECIT in the spreadsheet (F7) and CITEC's conclusions registered in the spreadsheet (F6).b.Analysis on the records of CITEC's decision-making (F8) c.Search for studies in DECIT's files (F3) d.Search for CITEC's decisions (F11) b.The first author's experience as a full member of CITEC during the study period.

Chart 2 :
Correlation between the markers and information sources of the retrospective study.Forschen SciO p e n H U B f o r S c i e n t i f i c R e s e a r c h Citation: Elias FTS, Atallah AN (2016) The Role of Systematic Reviews as Part of Decision-Making Process: Descriptive Study of the Brazilian Experience.J Epidemiol Public Health Rev 1(5): doi http://dx.

O
p e n H U B f o r S c i e n t i f i c R e s e a r c h Citation: Elias FTS, Atallah AN (2016) The Role of Systematic Reviews as Part of Decision-Making Process: Descriptive Study of the Brazilian Experience.J Epidemiol Public Health Rev 1(5): doi http://dx.doi.org/10.16966/2471-8211.127

O 7 Comparison with the literature
p e n H U B f o r S c i e n t i f i c R e s e a r c h Citation: Elias FTS, Atallah AN (2016) The Role of Systematic Reviews as Part of Decision-Making Process: Descriptive Study of the Brazilian Experience.J Epidemiol Public Health Rev 1(5): doi http://dx.doi.org/10.16966/2471-8211.127OpenAccess

Sl No. Concluded systematic reviews produced by the Brazilian Cochrane Center Year of conclusion Marker
doi.org/10.16966/2471-8211.127

STRs and TNRRs) Total number of reports with citations Reports with citations of SRs in the Cochrane Library/Plus Reports with citations of SRs from the
Concluded systematic reviews produced by the Brazilian Cochrane Center on request from DECIT, Ministry of Health, according to the categories selected, from 2005 to 2011.(As additional files) Chart 4: Systematic reviews from the Brazilian Cochrane Center that were used as decision-making information (marker D), from 2005 to 2011 Graph 1: Situation of systematic reviews from the Brazilian Cochrane Center funded Ministry of Health, between 2004 and 2011, according to the marker (scorer) selected.Source: Compiled by the authors.BSR: Systematic Review from the Brazilian Cochrane Center; SISREBRATS: Information System of the Brazilian Health Technology Assessment Network; HTA: Health Technology Assessment.

Table 1 :
Citations of all Cochrane reviews in HTA reports produced by the Brazilian Ministry of Health, from 2006 to 2011.Citation: Elias FTS, Atallah AN (2016) The Role of Systematic Reviews as Part of Decision-Making Process: Descriptive Study of the Brazilian Experience.J Epidemiol Public Health Rev 1(5): doi http://dx.doi.org/10.16966/2471-8211.127Systematic reviews (SRs) from the Brazilian Cochrane Center cited in HTA reports from DECIT (marker A), between 2005 and 2011 Source: Compiled by the authors.Legend: TNRR: Technical Note From Rapid Review; STR: Scientific Technical Report; CITEC: Technology Incorporation Committee of the Ministry of Health; SBHCI: Brazilian Society for Hemodynamics and Intervention Cardiology; PAO: Public Attorneys' Office; FGLO: Federal General Legal Office.
HTA: Health Technology Assessment; STR: Scientific Technical Report; TNRR: Technical Note from Rapid Review; SR: Systematic Review