Foundation of Evidence-Based Decision Making for Health Care Managers, Part 1: Systematic Review

Evidence-based decision making has become a benchmark of best practice. Sources of evidence are systematic reviews and meta-analyses. To support their decision making, health care managers and supervisors need to be able to critically read and interpret systematic reviews and meta-analyses. They also need to be able to determine the applicability of the evidence to their problems and settings. This 2-part series of articles aims to equip health care managers and supervisors with these skills. This article, part 1, explains the types of systematic reviews, defines key terms, and outlines the process of systematic reviews and meta-analyses. Part 2 focuses on the additional procedures associated with meta-analyses, describes the potential shortcomings of both systematic reviews and meta-analyses, and finally, provides a way to appraise the applicability of their results.

F OR THE PAST few decades, experts in many fields have stated that decisions should be based on scientific evidence. [1][2][3][4][5][6][7][8][9] Health administration is 1 of the fields calling for evidencebased decision making. 10,11 Systematic reviews are the source of the evidence that supports evidence-based decision making. Therefore, understanding systematic reviews has become an essential skill for health care managers and supervisors. This article provides an introduction to the technique by describing the types of systematic reviews and by overviewing the process of conducting them. Moreover, throughout the article, key terms are defined. The goal is to begin to equip health care managers and supervisors to critically read and interpret

INTRODUCTION
Systematic reviews critically appraise all relevant research. US policymakers have been applying the results of systematic reviews since the late 1990s. 12 Recent Congressional acts, emphasizing comparative effectiveness research, have further established systematic reviews as the key to informed decision making and policy making. For example, the Medicare Prescription Drug, Improvement, and Modernization Act of 2003, the American Recovery and Reinvestment Act of 2009, and the Affordable Care Act of 2010 all contained provisions related to comparative effectiveness research. Therefore, systematic reviews have become the benchmark of best practice.

SUPPORT FOR EVIDENCE-BASED DECISION MAKING
Professionals are busy and have limited time to monitor, read, assess, and digest new information and research. In addition, integrating and synthesizing evidence is often difficult for decision makers. Reasons for this difficulty include the following: Overwhelming volumes of data exist. Data are scattered through many individual studies. Individual studies may be methodologically flawed, producing flawed data. Individual studies may be unique to a time and setting (not generalizable). Data may be misinterpreted or misrepresented. Conclusion validity may be lacking. Conclusions may conflict across individual studies. By integrating and synthesizing information, systematic reviews assist decision makers in determining which results are reliable. Moreover, the aggregated result of multiple studies provides decision makers with a more precise estimate of an intervention's potential effects than a single study does. Decision makers, therefore, can have confidence in the information upon which they base practice and policy.

OVERVIEW
A systematic review is a comprehensive review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant published and unpublished research studies; to extract and analyze data from the studies that are included in the review; and to present integrated and synthesized information. 13,14 Avoidance of bias is essential to systematic reviews. Moreover, as a scientific procedure, systematic reviews have a specialized terminology (Table 1). This scientific procedure results in an assessment of the strength of the body of evidence pertinent to a question or problem. Thus, systematic reviews make information available to decision makers in a usable format.

COMPARISON WITH TRADITIONAL NARRATIVE LITERATURE REVIEWS
A traditional narrative literature review is a critical appraisal, conducted by summarizing, classifying, and comparing the published knowledge available about a topic from sources such as research and theoretical articles, annual reviews, books, and dissertations. Traditional narrative literature reviews are being replaced by systematic reviews because they are susceptible to bias. Traditional narrative literature reviews do have purposes in today's health sector, and a need for them still exists. First, they can trace the history of policies, principles, theories, and other concepts. 23 Second, they can support practitioners' continuing education by keeping them abreast of current trends. Finally, though, traditional literature reviews have less of a role in a field's body of knowledge because they are less able to support evidence-based decision making than systematic reviews are.
Systematic reviews differ from traditional narrative literature reviews. Systematic reviews must meet strict standards to avoid bias and to assess the quality of the articles. 24 Therefore, systematic reviews establish whether scientific findings are consistent and, therefore, can be generalized across populations, settings, and variations in interventions or exposures. Consequently, systematic reviews are becoming prevalent. Table 2 provides a comparison of traditional narrative literature reviews and systematic reviews.

TYPES OF SYSTEMATIC REVIEWS
No standardized term and definition have yet been established for systematic reviews. 24 Authors inconsistently and loosely use several terms, such as systematic review, research synthesis, structured review, meta-analysis, and overview. Occasionally, a traditional narrative review is wrongly labeled a systematic review when it does not meet the strict standards for conducting systematic reviews. 24 Four types of systematic reviews exist based on purpose and on analytical techniques: Scoping systematic reviews are exploratory. They are also known as scoping reviews, scoping studies, scoping literature reviews, and scoping exercises. They provide a frame of reference for a topic by outlining the range, extent, and breadth of relevant evidence, literature, and other information resources that are available. 25  Expression of a study's outcome showing the magnitude of 1 factor on another, such as the intervention and the outcome Overall outcome of a meta-analysis achieved by standardizing the individual effect sizes across studies, thus eliminating units and thereby allowing the combination of the studies' effect sizes Examples include weighted mean differences, standardized mean differences, odds ratios (ORs), relative risks, risk differences (RDs), Glass $, Cohen d, Hedge g, and the Pearson r correlation coefficient. Eligibility criteria Explicit standards, established before a systematic review, for inclusion or for exclusion of original studies in the review Forward tracking Process of using citations of the originally identified articles to find additional subsequent articles that cite the originally identified article 16 Global filter Screening mechanism that uses data about documents to limit results of the search, such as time period or the representation of 1 research study to 1 article per year 20 Grey (gray) literature Body of publications produced by government, academia, or business and industry, in print or electronic forms or both, which is not published in easily accessible journals and may not appear in databases or through Web searches Hand-searching Process of scanning the contents of journals, conference proceedings, and abstracts, page by page, to identify relevant publications Heterogeneity Variations among original studies in terms of study designs, populations, interventions, settings, date, and other variables Variations among effect estimates exceeding random chance Measure that quantifies the extent of heterogeneity by describing the percentage of the variability in effect estimates due to heterogeneity rather than chance. A value greater than 50% may be considered to represent substantial heterogeneity. 18 Individual patient data meta-analysis Specialized type of meta-analysis that pools individual patient data (rather than summary data) from multiple original studies and thereby allows analyses of data from subgroups that may not have been possible in the original studies because the subgroups were too small Meta-analysis Quantitative specialized type of systematic review that introduces statistical techniques to combine the summary-level information from of primary studies (sometimes also known as quantitative systematic review or overview) Methodological diversity Variability in study design, conduct, and quality, such as blinding and concealment of allocation 17

PICOS
Model to frame research questions, participants (patients, population), intervention (exposure), comparison, outcome, study design (alternately PICOC, substituting context for study design) Power Likelihood of detecting, within a sample, that an effect exists within the population; probability that a null hypothesis will be rejected when it is truly false (and the (continues) Evidence-Based Decision Making for Health Care Managers 99 of evidence. 26 By mapping a topic, scoping reviews assist researchers in refining their research questions and in selecting appropriate research designs and methods. Scoping reviews support the determination of whether additional research, such as an experimental study or a full systematic review, is warranted. Rapid reviews are simplified versions of a systematic review. Rapid reviews are also known as ultra-rapid review, rapid literature reviews, rapid health technology Overall, cumulative quality of the information from studies included in a systematic review taking into account their individual quality (risk of bias) but emphasizing the quality of their cumulative evidence in terms of precision, directness, and applicability Subgroup analysis Analysis of subsets (subgroups) of studies in a meta-analysis to estimate the effect of the intervention on the subgroups of interest and to make formal statistical comparisons across the subgroups Summary statistics (measures) Effect measure for each outcome, such as likelihood ratios, risk ratios, ORs, RDs, sensitivity and specificity, difference in means, and hazard ratios Systematic review Comprehensive review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant published and unpublished research studies; to extract and analyze data from the studies that are included in the review; and to present integrated and synthesized information (sometimes also known as qualitative systematic review or narrative systematic review) Thematic analysis Identification of recurrent and important themes, trends, patterns, and other issues across evidence and a review's sample of articles 20 Thesauri-based expansion Procedure to document the search and its comprehensiveness by recording and organizing, for each database, all search (key) terms, their synonyms, and their similar words in a table 20  Traditional narrative  literature review Critical appraisal, conducted by summarizing, classifying, and comparing the published knowledge available about a topic from sources such as research and theoretical articles, annual reviews, books, and dissertations Weighted mean difference Method used to combine measures on continuous scales where the mean, standard deviation, and sample size in each group are known; the weight given to the difference in means from each study is determined by the precision of its estimate of effect; this weight equals the influence that the individual study has on the overall estimate of effect size of the meta-analysis (also called mean difference). 21 Abbreviations: ORs, odds ratios; RDs, risk differences. 100 THE HEALTH CARE MANAGER/APRIL-JUNE 2014 assessments, rapid response, rapid evidence assessments, rapid assessments, realist reviews, evidence summaries, and accelerated systematic reviews. Unlike the tightly focused systematic reviews, rapid reviews generally conceptualize questions and succinctly summarize broad scopes of scientific evidence. 27 Rapid reviews are conducted when practitioners, decision makers, or policymakers need information quickly, typically within 1 to 6 months. 28 In rapid reviews, researchers and analysts shorten the length of time needed to conduct the review by simplifying the methods of systematic reviews. However, despite the simplified methods, the overall conclusions in rapid reviews are similar to the conclusions of systematic reviews. 28 Systematic reviews are syntheses of evidence on a topic. Systematic reviews are characterized by their comprehensiveness and transparency. Additional characteristics of systematic reviews include the following: ) Laser focus on a well-defined question for which appropriate study designs can be identified. ) Provision of answers to questions from a relatively narrow range of homogeneous studies whose quality has been assessed. Systematic reviews can take from 6 months to 24 months to complete. 27,28 Systematic reviews are sometimes called qualitative systematic reviews or narrative systematic reviews because researchers use qualitative techniques to analyze their findings. It should be emphasized, however, that qualitative (narrative) systematic reviews are not the same as traditional, narrative literature reviews, although their name is very similar. Meta-analyses are a specialized type of systematic review that introduces statistical techniques to combine summary-level information from at least 2 studies. In 1976, Gene Glass coined the term meta-analysis. 29 Meta-analysis is ''the statistical analysis of a large collection of results from individual studies for the purpose of integrating findings.'' 30(p3) Sometimes, a meta-analysis is called an overview or a quantitative systematic review. To conduct a meta-analysis, researchers must have a highly focused research question. Meta-analyses estimate the overall, combined effect of several studies' outcomes (effects) for an intervention or exposure. Specifically, this estimate of effect (also called treatment effect) is the observed relationship between an intervention (or exposure) and an outcome expressed as, for example, an odds ratio, a relative risk, a risk difference, a weighted mean difference, a standardized mean difference, Pearson r correlation coefficient, Glass $, Cohen d, or Hedge g. 17,21 Thus, meta-analysis is an integrative analysis of all available evidence. Studies that are included in a meta-analysis have common underlying characteristics. 29 The studies do not need to have identical  31 Advantages are derived from the quantitative procedures of meta-analyses. First, in metaanalyses, findings from many studies are combined, some of which are contradictory. These studies examine the same question; however, individually, they are either too small or give conflicting results. Second, researchers weight the studies before combining them. Studies that provide more information, such as having greater precision, are assigned greater weight than less precise studies are. 32 Thus, well-conducted studies have greater influence on the meta-analysis' results than poorly conducted studies do. Third, power and precision are increased. 33 Fourth, a special type of meta-analysis, individual patient data meta-analysis, pools individual patient data (rather than summary data) from multiple original studies. 33 Individual patient data meta-analysis allows analyses of data from subgroups that may not have been possible in the original studies because the subgroups were too small. Finally, cumulative meta-analyses add new data to the pool as new studies are published. 34 Cumulative meta-analyses show trends. It should be noted that the Cochrane Collaboration, a leading organization in evidencebased health care and systematic reviews, does not categorize a meta-analysis as a type of review. 14 The Cochrane Collaboration states that a meta-analysis is only the statistical techniques that combine multiple studies' results into a single estimate of effect. The Collaboration elaborates stating that the final estimate of effect may not always be the result of a systematic review of the literature.

PROCESS
A systematic review is a methodical approach to the body of published and unpublished evidence that reduces the possibility of bias. The method must be so clear, detailed, and exact that another person can replicate it. The process is characterized by the following: Exhaustiveness of the search for evidence. Transparency of the author's process of selecting evidence. The method of the search for evidence (literature and other data) is made explicit, including sources of evidence, search terms, and criteria to identify the evidence.
Evaluation of the quality of the evidence against consistent, specified methodological standards. Transparent explanation of how evidence, data, and information from studies, other published works, and other sources were included in or excluded from the review. Narrative systematic reviews and metaanalyses follow the same general process, with extra procedures being added for the meta-analysis. The steps of conducting narrative systematic reviews are presented in this first article of the 2-part series. The extra procedures required for meta-analyses are presented in the second article of the 2-part series. It should be noted that the following steps incorporate the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses. 35 Narrative systematic review Step 1 Provide an introduction that includes both the context of the systematic review and an explicit statement of its question, aim, or objective. The context justifies the review in terms of current knowledge on the topic and existing gaps in that knowledge. The introduction concludes with the question, aim, or objective that has been established a priori (before the review begins). For scoping or rapid reviews, the question, aim, or objective may be broad. For systematic reviews and meta-analyses, the research question, aim, or objective must be narrowly focused and stated with reference to participants (patients, populations), interventions (exposures), comparisons, outcomes, and study design (PICOS). Finally, and most importantly,

102
THE HEALTH CARE MANAGER/APRIL-JUNE 2014 the research question, aim, or objective must be stated in a way that is answerable.
Step 2 Delineate the method of the review. The method explicitly addresses eligibility, selection of information resources, the specifics of the search's strategy, data extraction, details of the analytic techniques, and the assessment of potential biases.
Predetermine eligibility criteria of both inclusion criteria and exclusion criteria for the selection of articles.  41 and others. [42][43][44] ) Contacts are made with authors and researchers in the topic, requesting whether they have unsubmitted manuscripts or suggestions for additional articles or studies. ) Google Scholar and other Internet search engines, such as Google and Bing, are queried. Present the search strategy with the search terms; the limits or filters; the search's start and end dates for each database; the details of 1 database's full electronic search strategy, such that it could be replicated; and the method of selecting the articles for inclusion in and exclusion from the review. ) To ensure the comprehensiveness of the search, the list of search terms (also known as key terms in some reference databases) must be as complete as possible. Two procedures that contribute to the documentation of the search and its comprehensiveness are the thesauribased expansion and the report-based expansion. 20 ) Also reported are any limitations in addition to the exclusion criteria, such as language and date restrictions. Two filtering strategies that can assist researchers are the global filter and the semantic filter. 20 ) The start and end dates for the search of each database are provided. These dates allow readers to assess the currency of the review and to take into account the time lag in publication. 45 ) The full electronic search strategy of 1 major reference database is reported (Figure 1). ) The process for selecting studies for inclusion is described. The process includes how the articles were screened, such as title and abstract; who conducted the screening, such as authors or research assistants; independence of the review; interrater reliability; and how disagreements were resolved, such as consensus or the principal investigator's decision. Describe the process of extracting the data.

Data extraction includes selecting and
Evidence-Based Decision Making for Health Care Managers piloting extraction forms, compiling lists of all the variables and outcomes with operational definitions, extracting the data, and finally, describing any procedures to ensure the quality of the data. ) Forms can be researcher designed or based on a published template. Any piloting of forms is noted. ) Researchers list and operationally define all the variables and outcomes, such as PICOS, that they originally sought. They also list and justify any variables or outcomes that they later added to the list. ) Researchers develop a protocol to code and extract the data that they will present, critique, and summarize in the review. The coded data usually fall into 4 basic categories: (a) methodological and substantive features (process and content), (b) study quality, (c) intervention descriptors, and (d) outcome measures. 47 ) Researchers extract the data. ) Any procedures taken to prevent bias or errors in data extraction are explained. For example, procedures could be that 1, 2, or more researchers extracted data independently and that interextractor reliability was checked. Another procedure noted is the way that the researchers resolved their disagreements, such as consensus obtained through discussion or consulting a third researcher. ) Any contacts or attempted contacts with the articles' authors to obtain or clarify data are also described. ) Researchers describe any procedures that they took to identify multiplicative articles on the same study because, often, studies are published multiple times. This precaution must be taken to avoid double (triple, etc) counting. Provide details of the planned analytical techniques. Researchers state the summary statistic (measure) that they intend to use for each outcome, such as likelihood ratios, risk ratios, odds ratios, risk differences, sensitivity and specificity, difference in means, and hazard ratios. 45,48 Researchers also explain any analytical techniques to examine data from subgroups, to assess heterogeneity, and to determine whether a meta-analysis is feasible. To perform the statistical techniques of meta-analysis, the outcomes of a set of studies need to be reasonably homogeneous (similar). Therefore, meta-analysts conduct the Cochran Q test and the I 2 index to check for statistical heterogeneity (homogeneity). 17 Potential causes of statistical heterogeneity are clinical and methodological diversity, bias, and mere chance. 17 Should the researchers conclude, after interpreting the Cochran Q test and the I 2 index, that the review's set of studies is not reasonably homogeneous and, therefore, that a meta-analysis is inadvisable, the researchers have analytic methods for the resulting qualitative systematic review. These methods deal with qualitative evidence and with the combination of quantitative and qualitative evidence. 19 This qualitative synthesis includes thematic analysis and content analysis. 19 Describe the strategies to assess the quality of the individual studies included in the review and to appraise the overall strength of the body of evidence (''strength of evidence'') across the studies. It should be emphasized that the terminology for quality varies across appraisal systems and throughout the literature on systematic reviews and meta-analyses. 49 Experts at the Agency for Healthcare Research and Quality prefer the term assessment of risk of bias or risk of bias when referring to the appraisal of the quality of the individual studies in the systematic review. 49 Specifically, ''assessing the risk of bias of a study can be thought of as assessing the risk that the study results reflect bias in study design or execution in addition to the true effect of the intervention or exposure under study.'' 49(p3) This term is preferred because flaws in the design of a study or its implementation, such as a sample unrepresentative of the target population, result in bias. Importantly, bias undermines the overall strength of the evidence.
) An overall rating of risk of bias (quality) is made, such as low, medium, high, or unclear risk of bias. 49 ) Instruments to assess the risk of bias in individual studies include the Cochrane Collaboration's tool 50 and checklists, such as the Quality Assessment of Diagnostic Accuracy Studies 51 and the Standards for Reporting of Diagnostic Accuracy. 52 ) Studies are weighted by their assessed quality (risk of bias). Higher weights are given to studies with greater sample sizes and narrower confidence intervals (certainty is linked to perceived quality). 53 Weighting can also be based on subjective ratings, such as low, medium, high, or unclear risk of bias. The appraisal of the strength of the body of evidence (''strength of evidence'') takes into account the risk of bias in the individual studies but focuses on the overall, cumulative quality of the information across all the studies (''interstudy bias''). To assess the strength of the evidence, researchers appraise the studies' evidence overall in terms of its precision, directness, and applicability. 49 The researchers specify how they will assess phenomena that may affect the strength of the cumulative evidence, such as publication bias and selective reporting of data and outcomes. ) The strength of the evidence is graded for each of the major outcomes of interest, such as PICOS. In addition, the strength of evidence is graded for comparisons of special interest to clinicians, decision makers, and policymakers. ) Three commonly used systems to grade the strength of the evidence are the US Preventive Services Task  Step 3 Present the results of the review. The results section documents the results of the search strategy, the characteristics of each of the studies included in the review, the analyses and syntheses, and the assessments of the risk of bias and the strength of the evidence.
Present the results of the search at each stage of the multistage filtering process. A flow diagram (Figures 2 and 3) shows the number of studies that the search initially identified (by information source), the number of studies screened, the number of studies assessed for eligibility, and the number of studies included in the review. At each stage, the reasons for the exclusion   The summary table includes pertinent information such as the citation, subjects, study size, setting, duration, intervention or exposure, major outcomes, and other key information. The summary table standardizes the results to clearly show missing or unclear information. 45 Present the results of the analyses and syntheses. Summary data, such as means, standard deviations, and odd ratios, from each article are provided in narrative and in graphics. Subgroup analysis on particular populations may be performed and reported. Qualitative synthesis is performed and patterns are reported. The reasons precluding meta-analysis are explained. Finally, the synthesized major findings of the review are presented. Report the outcome of the assessment of the risk of bias for each retained study. Potential biases may be design or methodological flaws related to the selection of subjects, the blinding of subjects and researchers, attrition and other sources of incomplete data, and researchers' selective reporting of outcomes. Finally, an assessment of the aggregate result of the individual studies' biases on the strength of the evidence for the review's major findings is stated.
Step 4 Provide a discussion that summarizes the evidence, addresses limitations, and presents a conclusion.
Present a summary of the evidence, including the evidence's strength, for each of the main outcomes. The outcomes should be framed in terms of their relevance for various target audiences, such as policy makers, decision makers, providers, payers, and other stakeholders. For example, many target audiences desire information on the outcomes' impact on individuals' and populations' disease and health; on quality of life; on costs across the health care delivery system; and on beneficial, harmful, and unintended consequences. 57 Moreover, the paucity or absence of data for some outcomes may be relevant to policymakers and future researchers and, therefore, should be mentioned. 45 Finally, the generalizability of the major outcomes across populations or subgroups, settings, and geographic areas is also important. Discuss the review's limitations in terms of the limitations both of the studies and of the process of the review itself. Examples of the studies' limitations include design flaws (risk of bias), incomplete reporting of data, substantial missing data, biased reporting of outcomes, and implausible conclusions. Limitations of the process itself include the inability to obtain published articles or unpublished manuscripts, to assess the credibility of the grey literature, or to translate non-English articles. Conclude the systematic review with a general interpretation, being careful to restrict the interpretations to those fully supported by the evidence. Neutrally and objectively, the general interpretation puts the results of the systematic review into the context of the current knowledge on the topic, offers implications for practice, and provides recommendations for future research. 45 Authors should state when they are unable to interpret results because the evidence is inadequate, insufficient, or absent. These 4 steps are the basis of the scientific rigor of systematic reviews and why their results can be used as evidence. The second article in this 2-part series will present the additional steps for meta-analyses.

CONCLUSION
Systematic reviews underpin evidence-based decision making. Systematic reviews critically appraise all relevant research. This first article of a 2-part series explained how systematic reviews differ from traditional, narrative literature reviews. It defined key terms and differentiated the 4 types of systematic reviews. Most importantly, health care managers and supervisors were introduced to the process of conducting systematic reviews and meta-analyses. Thus, this article provided information essential to health care manager's and supervisor's understanding of systematic reviews.