http://orcid.org/0000-0002-7469-1399
Figures
Abstract
Reporting bias in the literature occurs when there is selective revealing or suppression of results, influenced by the direction of findings. We assessed the risk of reporting bias in the epidemiological literature on health-related behavior (tobacco, alcohol, diet, physical activity, and sedentary behavior) and cardiovascular disease mortality and all-cause mortality and provided a comparative assessment of reporting bias between health-related behavior and statin (in primary prevention) meta-analyses. We searched Medline, Embase, Cochrane Methodology Register Database, and Web of Science for systematic reviews synthesizing the associations of health-related behavior and statins with cardiovascular disease mortality and all-cause mortality published between 2010 and 2016. Risk of bias in systematic reviews was assessed using the ROBIS tool. Reporting bias in the literature was evaluated via small-study effect and excess significance tests. We included 49 systematic reviews in our study. The majority of these reviews exhibited a high overall risk of bias, with a higher extent in health-related behavior reviews, relative to statins. We reperformed 111 meta-analyses conducted across these reviews, of which 65% had statistically significant results (P < 0.05). Around 22% of health-related behavior meta-analyses showed small-study effect, as compared to none of statin meta-analyses. Physical activity and the smoking research areas had more than 40% of meta-analyses with small-study effect. We found evidence of excess significance in 26% of health-related behavior meta-analyses, as compared to none of statin meta-analyses. Half of the meta-analyses from physical activity, 26% from diet, 18% from sedentary behavior, 14% for smoking, and 12% from alcohol showed evidence of excess significance bias. These biases may be distorting the body of evidence available by providing inaccurate estimates of preventive effects on cardiovascular and all-cause mortality.
Author summary
In the scientific literature, reporting bias occurs when communication and publication of results are influenced by the direction of findings. Reporting bias can distort scientific evidence and may misguide subsequent clinical and public health efforts. Our study provided an assessment of the degree of reporting bias in the literature on health-related behavior (smoking, alcohol, diet, physical activity, and sedentary behavior) and statins and their association with cardiovascular disease and mortality. We analyzed recently published systematic reviews. Most of the systematic reviews (90%) had a high risk of bias related to study eligibility criteria, identification and selection of studies, data collection and study appraisal, and synthesis and findings. We found evidence of reporting bias in about one-fifth of health-related behavior meta-analyses but none of the statin-related meta-analyses. Readers should be aware of the extent of reporting bias in these research areas when interpreting meta-analytical results.
Citation: Rezende LFMd, Rey-López JP, Sá THd, Chartres N, Fabbri A, Powell L, et al. (2018) Reporting bias in the literature on the associations of health-related behaviors and statins with cardiovascular disease and all-cause mortality. PLoS Biol 16(6): e2005761. https://doi.org/10.1371/journal.pbio.2005761
Academic Editor: Malcolm Macleod, University of Edinburgh, United Kingdom of Great Britain and Northern Ireland
Received: February 20, 2018; Accepted: June 7, 2018; Published: June 18, 2018
Copyright: © 2018 Rezende et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data files are available in the S1 Data file and in the Open Science Framework webpage at osf.io/wpb69.
Funding: Sao Paulo Research Foundation (FAPESP) http://www.fapesp.br/en/ (grant number #2014/25614-4 and #2016/21390-0). Leandro Fórnias Machado de Rezende receives doctoral scholarship from Sao Paulo Research Foundation (FAPESP). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. University of Sydney (grant number U2334). Juan Pablo Rey-López is supported by a Postdoctoral fellowship from the University of Sydney (Australia). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. National Health and Medical Research Council (Australia) (grant number). Emmanuel Stamatakis is supported by the National Health and Medical Research Council (Australia) through a Senior Research Fellowship and partly supported by a University of Sydney SOAR Fellowship. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: AMSTAR, Assessing the Methodological Quality of Systematic Reviews; CHD, coronary heart disease; DHA, docosahexaenoic acid; E, number of expected primary studies with P < 0.05 results; EPA, eicosapentaenoic acid; IHD, ischemic heart disease; MI, myocardial infarction; MOOSE, Meta-analysis of Observational Studies in Epidemiology; MUFA, monounsaturated fatty acid; O, number of observed primary studies with P < 0.05 results; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; PUFA, polyunsaturated fatty acid; SFA, saturated fatty acid
Introduction
The literature on the association between behavioral risk factors (e.g., smoking, alcohol, physical inactivity, and unhealthy diet) and cardiovascular diseases—the single largest cause of death globally [1]—has grown exponentially in the last decades [2–39]. Observational epidemiological studies are the dominant design assessing these associations, since clinical trials cannot always be ethically or logistically conducted [40]. Systematic review methods are used to synthesize and evaluate this growing body of evidence. It is important to evaluate the methodological risks of bias in systematic reviews [41], as well as the impact that reporting bias can have on the findings of reviews [42, 43].
Reporting bias is one of the most common biases identified in the literature. It includes selective publication of studies or outcomes of studies [44, 45] based on factors other than the study quality, such as nominally statistically significant results (P < 0.05) [46, 47] or authors’ “pedigree” [44, 45, 48]. These practices threaten the completeness and validity of scientific evidence [46] by distorting the estimates of causal effects of interventions or exposures on diseases [49]. The extent of reporting bias could differ between bodies of evidence consisting of randomized trials, such as drug studies, compared to observational studies, such as studies of health behavior. Different levels of reporting bias in the literature on health behavior may lead to inaccurate estimates of preventive effects on cardiovascular and all-cause mortality and therefore offer incorrect guidance for policymaking.
To gain a better understanding of the potential reporting bias in the literature on health-related behavior and cardiovascular disease mortality and all-cause mortality, we examined reporting and other risks of bias in a sample of systematic reviews published between 2010 to 2016. Our analysis also provided a comparative assessment of the reporting bias between health-related behavior and statins used in primary prevention.
Results
Of the 5,511 records identified while searching the databases, we selected 49 systematic reviews. All research areas (tobacco, alcohol, diet, physical activity, sedentary behavior, and statins) presented fewer than 20 eligible systematic reviews; therefore, we included all the systematic reviews within each area that met our inclusion criteria (Fig 1). Lists of excluded reviews and reasons for exclusions are described in S1 and S2 Tables. Studies were excluded most frequently for not including one of the exposures (28%) or outcomes (29%) of interest and for utilizing clinical samples (16%).
Most of the included systematic reviews (n = 35, 71.4%) analyzed only one outcome eligible for our study (cardiovascular disease mortality and all-cause mortality), whereas 9 (18.4%), 4 (8.2%), and 1 (2.0%) analyzed two, three, and four outcomes, respectively. All-cause mortality (69%), cardiovascular disease mortality (29%), and stroke mortality (14%) were the most frequent outcomes investigated (Table 1).
Risk of bias in systematic reviews—ROBIS results
The majority of the systematic reviews exhibited a high overall risk of bias (n = 44, 90%) (Fig 2). Among the four ROBIS domains, domain 1 (study eligibility criteria) presented the best scores, with 32 (65%) out of 49 reviews showing a low risk of bias. In domain 2 (identification and selection of studies), 2 (4%) reviews were scored as unclear, 40 (82%) showed a high risk, and 7 (14%) a low risk of bias. Whereas, in domain 3 (data collection and study appraisal), 7 (14%) reviews were scored as unclear, 28 (57%) scored with high risk, and 14 (29%) with low risk of bias. Finally, in domain 4 (synthesis and findings), 2 (4%) review was scored as unclear, 30 (61%) with high risk, and 17 (35%) with low risk of bias (Fig 2 and S3 Table).
Underlying data can be found in S1 Data.
Comparing risk of bias in the reviews across research areas, sedentary behavior performed worst in domain 1 (study eligibility criteria; 70% of reviews were regarded as having high risk of bias). All research areas performed poorly in domain 2 (identification and selection of studies), with high risk of bias ranging from 70% in smoking reviews to 90% in both sedentary behavior and statin reviews. Alcohol (70%) and diet (60%) reviews presented high risk of bias in domain 3 (data collection and study appraisal). Sedentary behavior (90%), smoking (70%), and diet (70%) reviews presented high risk of bias in domain 4 (synthesis and findings). Overall, statin reviews presented the best scores in the ROBIS assessment compared to other research areas. Among statin reviews, a low risk of bias was identified in 60% in domain 1, 10% in domain 2, 50% in domain 3, and 60% in domain 4 (Table 1 and Fig 3).
Risk of bias in systematic reviews of the associations of physical activity (A), smoking (B), sedentary behavior (C), diet (D), alcohol (E), and statins (F) with cardiovascular disease mortality and all-cause mortality. Underlying data can be found in S1 Data.
Risk of reporting bias in the body of evidence
We identified 111 meta-analyses (exposure–outcomes associations) that were performed across the 49 included reviews. On average, each meta-analysis synthesized results from 9 primary studies (ranging from 2 to 81), including 331,688 participants (ranging from 595 to 3,674,042) and 19,012 deaths (ranging from 33 to 320,252) (Table 2 and S4 Table). Of the 111 meta-analyses, 72 (65%) showed a nominally statistically significant result at P < 0.05.
Nominally statistically significant results (P < 0.05) were found in 92% of the meta-analyses from sedentary behavior and 100% of the meta-analyses from physical activity and smoking. Alcohol and statin reviews had 38% and 45% of meta-analyses with P < 0.05 results, respectively (Table 2 and S4 Table).
Small-study effect.
Indication of small-study effect was found in 17 (18%) of the 96 meta-analyses. Physical activity and the smoking research areas had more than 40% of meta-analyses with small-study effect. Sedentary behavior had less than 10% of meta-analyses with small-study effect (Table 2 and S4 Table). Overall, 17 (22%) out of 79 meta-analyses of health-related behavior presented small-study effect, as compared to zero-statin meta-analyses.
Excess significance.
More than half (56%; 53/95) of the meta-analyses displayed a greater number of observed primary studies with P < 0.05 results (O) than the number number of expected primary studies with P < 0.05 results (E). Of those, 19 meta-analyses (20% of the total meta-analyses and 36% of the meta-analyses with O > E) showed evidence of excess significance bias (P < 0.10). Half of the meta-analyses from physical activity, 26% from diet, 18% from sedentary behavior, 14% from smoking, and 12% from alcohol showed evidence of excess significance (Table 2 and S4 Table). Overall, 24% of the meta-analyses of health-related behavior showed excess significance, as compared to zero-statin meta-analyses.
Sensitivity analyses
We conducted a sensitivity analysis by restricting the sample in each research area to meta-analyses with ≥10 primary studies. In this subsample (n = 29), 86% of the meta-analyses showed statistically significant results at P < 0.05, as compared to 65% in the entire sample of meta-analyses. These results varied by research area, ranging from 60% in statin meta-analyses to 100% in physical activity, sedentary behavior, and smoking meta-analyses (Table 3).
Small-study effect was present in 31% of the meta-analyses. The proportions of meta-analyses in the sensitivity analysis with small-study effect were 80% for physical activity, 50% for sedentary behavior, 29% for alcohol, and 33% for smoking. Diet and statin meta-analyses had no evidence of small-study effect. Around 38% of the health-related behavior meta-analyses with ≥10 primary studies presented small-study effect, as compared to zero in statin meta-analyses (Table 3).
Excess significance was identified in 27% of the meta-analyses with ≥10 primary studies: 100% of the meta-analyses for sedentary behavior, 50% for diet, 40% for physical activity, 20% for alcohol, and 17% for smoking. Around 33% of the health-related meta-analyses with ≥10 primary studies showed evidence of excess significance, as compared to zero in statin meta-analyses (Table 3).
Overall, after excluding small individual studies (with <200 deaths) from meta-analyses, results from small-study effect and excess significance tests did not change (S5 Table).
Discussion
This study aimed to assess the extent of reporting bias among recent meta-analyses that examined the associations of health behavior and statins with cardiovascular and all-cause mortality. We found evidence of reporting bias across all health-related behavior areas. The degree of reporting bias varied by the method used to assess it. Reporting bias was present in 20% (according to excess significance test) or 18% (according to small-study effect test) of all meta-analyses included (health behavior and statins). Evidence of reporting bias was found in between a quarter and one-fifth of health-related behavior meta-analyses (22% small-study effect and 24% excess significance) but in none of the statin meta-analyses (0%).
In lifestyle epidemiology, the interpretation of evidence for researchers and policymakers is challenging for several reasons [61]. As observational studies are the dominant designs in this area, spurious associations can arise due to confounding or several sources of bias. The impact of such biases on statistical findings and interpretation of findings has been poorly reported and discussed [62]. Therefore, meta-analytical synthesis of the evidence in lifestyle health behavior epidemiology may provide precise but spurious results [63].
Reporting bias is a major threat to the validity of the relevant body of evidence. Our results suggest that around 20% of the meta-analyses on health-related behavior and cardiovascular disease mortality and all-cause mortality may be susceptible to reporting biases. The existence of reporting bias in the literature has several explanations. Failure to submit manuscripts of analyses that did not produce statistically significant results (“the file-drawer problem” [46]) and the low likelihood of publication of small studies (regardless of statistical significance) [44] are two possible reasons. The selective reporting of certain analyses with statistically significant results is another likely source of reporting bias [44, 46, 47]. Each of the research areas we examined is likely to be linked to variable levels of reporting bias due to the different economics, dynamics, and conflicts of interest in each discipline [64, 65]. Interpreting the literature as a whole is challenging, considering the numerous biases that may affect the reliability and integrity of the scientific enterprise [66, 67].
To obtain a complete picture of the evidence (i.e., without reporting bias), it is important to know the results from all conducted studies on a given research question [68]. In our study, results from meta-analyses of health-related behavior and cardiovascular disease mortality and all-cause mortality were more likely to be affected by reporting bias compared to statin meta-analyses (22% and 24% versus 0%, respectively). The literature of health-related behavior is almost exclusively composed by observational studies, whereas statins are most often studied using randomized controlled trials. Reporting bias may be less frequent among trials than observational studies because several efforts to increase transparency and reproducibility of results have been adopted over the history of randomized controlled trials [69]. These include the mandatory registration of all clinical trials in humans and disclosure of all results [70]. As of more recently, data sharing statements of clinical trials are also required [71]. Observational epidemiologic studies should embrace these reproducible research practices to reduce reporting bias in the literature [68–70, 72]. These practices could involve key elements of the scientific process, including (a) methods (e.g., rigorous training in statistics), (b) reporting and dissemination (e.g., disclosure of conflicts of interest), (c) reproducibility (e.g., open data), (d) evaluation (e.g., pre- and postpublication peer review), and (e) incentives (e.g., funding replication studies) [72]. Improving methodological training involves aspects of both research design and statistical analyses—for example, correct interpretation of P values [73], acknowledging the importance of statistical power, and improving the accuracy of effect sizes [72]. Protecting against cognitive biases is another major issue that has been overlooked [72]. Protecting against conflict of interests, especially financially related, is an imperative to achieve reproducible science. In addition to disclosure of potential conflicts of interest, promoting preregistration of study procedures and analytical plan may prevent reporting bias favoring positive results [72]. Funding replication of studies and encouraging openness in science and reproducibility practices by making datasets, scripts, and software publicly available may increase transparency and credibility of scientific claims [72]. For instance, food industry–sponsored studies are more likely to report conclusions favorable to the sponsors [74] but frequently lack transparency on acknowledgment of the funding source [75]. Further examples of reproducibility practices have been described and discussed by Munafò and colleagues [72].
To our knowledge, our analysis is the first comparative assessment of reporting bias across different fields of health-related behavior and statins. Our findings were based on well-established statistical tests developed to detect different aspects of reporting bias, as well as a complementary assessment of the risk of bias of systematic reviews using the ROBIS tool. We selected the ROBIS tool as it has greater specification to assess risk of bias compared to other tools. For instance, the “Assessing the Methodological Quality of Systematic Reviews” (AMSTAR) that has been used to evaluate the methodological quality of systematic reviews has constructs that are more related to quality of reporting than risk of bias [76, 77]. Risk of bias is linked to methodological quality of systematic reviews but provides further evaluation on how methodological limitations were considered to form conclusions. In this sense, the ROBIS tool is increasingly being used to assess risk of bias not only in systematic reviews [41, 76, 78] but also in guideline committees that evaluate evidence level (e.g., Australian government, National Health and Medical Research Council). Our ROBIS tool results showed that most of the systematic reviews had high risk of bias. Similar findings have been observed in previous studies appraising risk of bias in other research areas using the ROBIS tool [76, 78]. For instance, 18 (58%) out of 31 systematic reviews evaluating the effectiveness of intra-articular hyaluronic acid injection in treating knee osteoarthritis had high (n = 16) or unclear (n = 2) risk of bias [78]. Another survey assessing systematic reviews about psoriasis found that most reviews (86%) were classified as high risk of bias [76]. It is noteworthy that high risk of bias was found even for systematic reviews exhibiting high methodological quality as assessed through AMSTAR [76].
Our ROBIS assessment indicated that identification and selection of studies (i.e., appropriate range of databases, terms and filters used, and efforts to minimize errors in selection of studies) are major concerns. These biases in the review process could explain, at least in part, reporting bias results obtained from small-study effect and excess significance tests. The synthesis and findings domain also revealed potential risk of bias due to insufficient inclusion of studies and appropriate synthesis of estimates. This domain also reflects between-study variation, robustness of findings (e.g., sensitivity analyses), and biases in synthesis findings (i.e., if evaluated by systematic reviews).
We used small-study effect and excess significance tests to appraise reporting bias in the literature, which are the most commonly recommended and used methods [79]. However, results from these tests might also reflect methodological and clinical heterogeneity, or even chance [42]. In fact, most meta-analyses contained moderate to high heterogeneity (based on I2 statistic; S4 Table). Results from an Egger test (small-study effect) can give spurious false positive results due to correlation between log of effect size and its variance, especially in the presence of heterogeneity between studies in a meta-analysis. An alternative better-performing test has been proposed by Peters to identify reporting bias in meta-analyses, but it requires data from a 2 × 2 table [80]. Such data were rarely reported in individual studies in the meta-analyses of observational studies. As also noted by Tsilidis and colleagues [81], meta-analyses commonly use maximally adjusted relative risks rather than unadjusted relative risks calculated from 2 × 2 tables. For such data, the use of the Egger test is appropriate.
The egger test and excess significance test have low power to detect reporting bias and do not give indication about what the sources of bias are. Therefore, we performed sensitivity analyses, retaining only meta-analyses with ≥10 primary studies. In this subsample of meta-analyses, evidence of reporting bias was higher than the entire sample (small-study effect: 31% versus 18%; excess significance: 27% versus 20%). Differences between primary results and sensitivity analyses are likely related to low power of reporting bias tests, which could lead to false negative results in the former group of meta-analyses. Therefore, our estimates of reporting bias in the meta-analyses are possibly conservative. The ranking of research areas according to levels of reporting bias was also different between the main analysis and the sensitivity analysis (i.e., meta-analyses with ≥10 primary studies). For instance, meta-analyses of sedentary behavior appeared most sensitive to this restriction, as the estimated proportion of reporting bias increased when calculated with either the small-study effects (from 9% to 50%) or excess significance tests (from 18% to 100%). A possible explanation could be the small fraction of meta-analyses with ≥10 primary studies (2 out of 12) in this relatively new research field [82].
It is important to acknowledge that certain methodological decisions we made may have introduced bias in the sample of reviews selected or may compromise the generalizability of our findings. We excluded systematic reviews on alcohol published in Chinese language (n = 2), which potentially have high risk of bias [83]. In addition, we restricted our analyses to systematic reviews published in this decade only (2010–2016), which explains the small number of included meta-analyses in some research areas. This may have limited comparisons of the extent of reporting bias between research areas investigated. Our results may not provide a complete historical assessment of reporting bias in these areas. Nevertheless, our results reflect reporting bias in the literature of recent and relevant public health topics and from a time period when reporting standards have been improving due to, e.g., the widespread use of various manuscript reporting checklists [84]. Recent systematic reviews contain a higher number of primary studies than older systematic reviews and synthesize evidence of emerging fields that have flourished only recently (i.e., sedentary behavior).
In conclusion, we found evidence of reporting bias in approximately one-fifth of recent meta-analyses of observational studies of health-related behavior (physical activity, sedentary behavior, smoking, alcohol consumption, diet) and cardiovascular and all-cause mortality. Such a level of reporting bias may, to some extent at least, distort conclusions arising from this body of evidence. Contrarily, we found no evidence of reporting bias in meta-analyses of randomized controlled trials of statins.
Materials and methods
Identification and selection of relevant systematic reviews
We searched Medline (through PubMed), Embase (i.e., excluding Medline), Cochrane Methodology Register Database, and Web of Science for systematic reviews published between 2010 and 2016. We restricted our search to recent systematic reviews for several reasons. These systematic reviews belong to a “birth cohort” of systematic reviews published after the launch of the Meta-analysis of Observational Studies in Epidemiology (MOOSE) [85] and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [86] guidelines and are expected to have lower risk of bias. As we were interested in comparing levels of bias across different research areas, this restriction may have reduced confounding due to date of publication. We restricted the search, as well as the successive phases of our study, to systematic reviews aiming to investigate the associations of health-related behavior (tobacco, alcohol, diet [fat, fruits and vegetables, salt, and sugar], physical activity, and sedentary behavior) and statins with cardiovascular disease mortality (overall cardiovascular mortality and cause-specific deaths from cardiovascular disease) and all-cause mortality. We accepted any definition for the exposures and the outcomes as defined in the original systematic reviews. The keywords used in the search are described in S1 Text, and files exported from databases during search strategy with all studies screened and selected are available at https://osf.io/wpb69/.
Systematic reviews were screened and selected (by two reviewers, and disagreements solved by a third reviewer) based on the following eligibility criteria: (i) sought to investigate an exposure–outcome association in a nonclinical population; (ii) systematically searched for primary studies and performed a meta-analysis (i.e., weighted summary effect size) using results from primary studies; (iii) selected only observational studies (cohort and case-control studies) if a health-related behavior review and only randomized controlled trials if a statin review; (v) reported data from each primary study included in the meta-analysis (S1 Text).
We decided a priori that a random sample of up to 20 systematic reviews per research area (tobacco, alcohol, diet, physical activity, sedentary behavior, and statins) would be included to compare levels of reporting bias in the relevant literature. If our search retrieved fewer than 20 meta-analyses in a given research area, we included them all. A similar study-selection strategy was recently used in a study evaluating publication bias in meta-analyses of individual studies [87]. These methods were decided a priori as described in the analysis plan available at https://osf.io/wpb69/ (not published prior to the identification and selection of systematic reviews).
Risk of bias in systematic reviews
Reporting bias could be related to overall risk of bias in a review. Therefore, four reviewers (JPRL, NC, AF, LP), working in pairs, independently assessed the risk of bias in the included systematic reviews using the ROBIS tool [41]. ROBIS comprises three phases: (1) assess relevance; (2) identify concerns with review process; (3) judge risk of bias in the review. To assess relevance, we extracted the target question from each review using the PICOS acronym (participants, interventions, comparisons, outcomes) or equivalents for etiological questions (participants, exposure, comparisons, outcomes). In phase 2, we assessed the risk of bias in four domains related to the review process: (1) study eligibility criteria; (2) identification and selection of studies; (3) data collection and study appraisal; and (4) synthesis and findings. Questions included in each of the four domains are available in S3 Table. Questions were answered as “Yes,” “Probably Yes,” “Probably No,” “No,” and “No Information,” with “Yes” indicating low risk of bias. In phase 3, we summarized the concerns identified in each domain during phase 2 and risk of bias in the review as low, high, or unclear. Further details on the ROBIS tool are described elsewhere [41].
Risk of reporting bias in the body of evidence
For each meta-analysis performed in the selected systematic reviews, we assessed the extent of reporting bias in the included literature via small-study effect [42] and excess significance tests [88]. To perform these tests, we extracted necessary data (e.g., effect size, confidence intervals, sample size, and number of events [deaths]) for each primary study included in the main meta-analysis performed in the systematic reviews. We also used these data to reperform the meta-analyses (i.e., using random effect models, which was used in the majority of the original meta-analyses). We did this to describe the number of meta-analyses with nominally statistically significant results at P < 0.05 (S1 Text).
Small-study effect test (also known as regression asymmetry test, proposed by Egger and colleagues) evaluates whether smaller studies tend to overestimate the effect size estimates compared to larger studies. For this matter, the test evaluates whether the association between effect size (e.g., relative risk, odds ratio) and precision (standard error) is greater than might be expected by chance. We considered a P value < 0.10 as a statistical significance threshold for small-study effect bias (i.e., suggesting evidence of reporting bias), as initially proposed by Egger and colleagues [42, 89] and consistently used in the literature [42, 66, 81, 87, 90, 91].
Excess significance test evaluates whether the O differs from the E. The E in each meta-analysis was obtained from the sum of power estimates of each primary study. The power estimate of each primary study depends on the plausible causal effect of each research area (e.g., smoking and cardiovascular mortality), which was assumed to be the effect of the most precise primary study (smaller standard error) in each meta-analysis [88]. We considered P < 0.10 (one-side P < 0.05 for O > E) as a statistical significance threshold for excess significance bias [43, 88]. The excess significance is reported as a proportion of studies, with the higher proportion indicating more excess significance (O > E) and thus more evidence of reporting bias.
Due to the low power of these bias tests, we performed a sensitivity analysis excluding meta-analyses with fewer than 10 studies to analyze the impact in the results. We also performed a sensitivity analysis excluding small individual studies (fewer than 200 deaths) within meta-analyses to evaluate whether results reflect reporting bias among small studies only. We performed all statistical analyses using Stata version 15.0 (College Station, TX).
Supporting information
S1 Table. List of excluded studies and reasons for exclusion, by research area.
https://doi.org/10.1371/journal.pbio.2005761.s003
(DOC)
S2 Table. Summary of reasons for excluding studies during full-text screening, by research area.
https://doi.org/10.1371/journal.pbio.2005761.s004
(DOC)
S3 Table. Risk of bias in systematic reviews of physical activity, sedentary behavior, diet, alcohol, smoking, and statins, using ROBIS tool.
https://doi.org/10.1371/journal.pbio.2005761.s005
(DOC)
S4 Table. Meta-analyses of cardiovascular disease and all-cause mortality, by research area.
https://doi.org/10.1371/journal.pbio.2005761.s006
(DOC)
S5 Table. Sensitivity analysis excluding individual studies with fewer than 200 deaths.
Relative and absolute frequencies of meta-analyses with nominally statistically significant results, small-study effects, and excess significance, by research area.
https://doi.org/10.1371/journal.pbio.2005761.s007
(DOC)
References
- 1.
World Health Organization. The top 10 causes of death 2017 [cited 2017 November 27]. http://www.who.int/mediacentre/factsheets/fs310/en/.
- 2. Kelly P, Kahlmeier S, Gotschi T, Orsini N, Richards J, Roberts N, et al. Systematic review and meta-analysis of reduction in all-cause mortality from walking and cycling and shape of dose response relationship. Int J Behav Nutr Phys Act. 2014;11:132. pmid:25344355
- 3. Samitz G, Egger M, Zwahlen M. Domains of physical activity and all-cause mortality: systematic review and dose-response meta-analysis of cohort studies. Int J Epidemiol. 2011;40(5):1382–400. pmid:22039197
- 4. Woodcock J, Franco OH, Orsini N, Roberts I. Non-vigorous physical activity and all-cause mortality: systematic review and meta-analysis of cohort studies. Int J Epidemiol. 2011;40(1):121–38. pmid:20630992
- 5. Hupin D, Roche F, Gremeaux V, Chatard JC, Oriol M, Gaspoz JM, et al. Even a low-dose of moderate-to-vigorous physical activity reduces mortality by 22% in adults aged >/ = 60 years: a systematic review and meta-analysis. Br J Sports Med. 2015;49(19):1262–7. pmid:26238869
- 6. Biswas A, Oh PI, Faulkner GE, Bajaj RR, Silver MA, Mitchell MS, et al. Sedentary time and its association with risk for disease incidence, mortality, and hospitalization in adults: a systematic review and meta-analysis. Ann Intern Med. 2015;162(2):123–32. pmid:25599350
- 7. Chau JY, Grunseit AC, Chey T, Stamatakis E, Brown WJ, Matthews CE, et al. Daily sitting time and all-cause mortality: a meta-analysis. PLoS ONE. 2013;8(11):e80000. pmid:24236168
- 8. Grontved A, Hu FB. Television viewing and risk of type 2 diabetes, cardiovascular disease, and all-cause mortality: a meta-analysis. JAMA. 2011;305(23):2448–55. pmid:21673296
- 9. Wilmot EG, Edwardson CL, Achana FA, Davies MJ, Gorely T, Gray LJ, et al. Sedentary time in adults and the association with diabetes, cardiovascular disease and death: systematic review and meta-analysis. Diabetologia. 2012;55(11):2895–905. pmid:22890825
- 10. Ford ES, Caspersen CJ. Sedentary behaviour and cardiovascular disease: a review of prospective studies. Int J Epidemiol. 2012;41(5):1338–53. pmid:22634869
- 11. Pandey A, Salahuddin U, Garg S, Ayers C, Kulinski J, Anand V, et al. Continuous Dose-Response Association Between Sedentary Time and Risk for Cardiovascular Disease: A Meta-analysis. JAMA cardiology. 2016;1(5):575–83. pmid:27434872
- 12. Sun JW, Zhao LG, Yang Y, Ma X, Wang YY, Xiang YB. Association Between Television Viewing Time and All-Cause Mortality: A Meta-Analysis of Cohort Studies. American journal of epidemiology. 2015;182(11):908–16. pmid:26568572
- 13. Costanzo S, Di Castelnuovo A, Donati MB, Iacoviello L, de Gaetano G. Wine, beer or spirit drinking in relation to fatal and non-fatal cardiovascular events: a meta-analysis. Eur J Epidemiol. 2011;26(11):833–50. pmid:22076059
- 14. Jayasekara H, English DR, Room R, MacInnis RJ. Alcohol consumption over time and risk of death: a systematic review and meta-analysis. Am J Epidemiol. 2014;179(9):1049–59. pmid:24670372
- 15. Roerecke M, Rehm J. Ischemic heart disease mortality and morbidity rates in former drinkers: a meta-analysis. Am J Epidemiol. 2011;173(3):245–58. pmid:21156750
- 16. Roerecke M, Rehm J. Alcohol consumption, drinking patterns, and ischemic heart disease: a narrative review of meta-analyses and a systematic review and meta-analysis of the impact of heavy drinking occasions on risk for moderate drinkers. BMC Med. 2014;12:182. pmid:25567363
- 17. Ronksley PE, Brien SE, Turner BJ, Mukamal KJ, Ghali WA. Association of alcohol consumption with selected cardiovascular disease outcomes: a systematic review and meta-analysis. BMJ. 2011;342:d671. pmid:21343207
- 18. Park JE, Choi TY, Ryu Y, Cho SI. The relationship between mild alcohol consumption and mortality in Koreans: a systematic review and meta-analysis. BMC Public Health. 2015;15:918. pmid:26385795
- 19. Stockwell T, Zhao J, Panwar S, Roemer A, Naimi T, Chikritzhs T. Do "Moderate" Drinkers Have Reduced Mortality Risk? A Systematic Review and Meta-Analysis of Alcohol Consumption and All-Cause Mortality. J Stud alcohol and drugs. 2016;77(2):185–98.
- 20. Zheng YL, Lian F, Shi Q, Zhang C, Chen YW, Zhou YH, et al. Alcohol intake and associated risk of major cardiovascular outcomes in women compared with men: a systematic review and meta-analysis of prospective observational studies. BMC Public Health. 2015;15:773. pmid:26264040
- 21. Roerecke M, Rehm J. Irregular heavy drinking occasions and risk of ischemic heart disease: a systematic review and meta-analysis. Am J Epidemiol. 2010;171(6):633–44. pmid:20142394
- 22. Roerecke M, Rehm J. Chronic heavy drinking and ischaemic heart disease: a systematic review and meta-analysis. Open Heart. 2014;1(1):e000135. pmid:25332827
- 23. Gellert C, Schottker B, Brenner H. Smoking and all-cause mortality in older people: systematic review and meta-analysis. Arch Intern Med. 2012;172(11):837–44. pmid:22688992
- 24. Lv X, Sun J, Bi Y, Xu M, Lu J, Zhao L, et al. Risk of all-cause mortality and cardiovascular disease associated with secondhand smoke exposure: a systematic review and meta-analysis. Int J Cardiol. 2015;199:106–15. pmid:26188829
- 25. Sinha DN, Suliankatchi RA, Gupta PC, Thamarangsi T, Agarwal N, Parascandola M, et al. Global burden of all-cause and cause-specific mortality due to smokeless tobacco use: systematic review and meta-analysis. Tob Control. 2018;27(1):35–42. pmid:27903956
- 26. Farvid MS, Ding M, Pan A, Sun Q, Chiuve SE, Steffen LM, et al. Dietary linoleic acid and risk of coronary heart disease: a systematic review and meta-analysis of prospective cohort studies. Circulation. 2014;130(18):1568–78. pmid:25161045
- 27. Graudal N, Jurgens G, Baslund B, Alderman MH. Compared with usual sodium intake, low- and excessive-sodium diets are associated with increased mortality: a meta-analysis. Am J Hypertens. 2014;27(9):1129–37. pmid:24651634
- 28. Hu D, Huang J, Wang Y, Zhang D, Qu Y. Fruits and vegetables consumption and risk of stroke: a meta-analysis of prospective cohort studies. Stroke. 2014;45(6):1613–9. pmid:24811336
- 29. Li XY, Cai XL, Bian PD, Hu LR. High salt intake and stroke: meta-analysis of the epidemiologic evidence. CNS Neurosci Ther. 2012;18(8):691–701. pmid:22742770
- 30. Musa-Veloso K, Binns MA, Kocenas A, Chung C, Rice H, Oppedal-Olsen H, et al. Impact of low v. moderate intakes of long-chain n-3 fatty acids on risk of coronary heart disease. Br J Nutr. 2011;106(8):1129–41. pmid:21736820
- 31. Pan A, Chen M, Chowdhury R, Wu JH, Sun Q, Campos H, et al. alpha-Linolenic acid and risk of cardiovascular disease: a systematic review and meta-analysis. Am J Clin Nutr. 2012;96(6):1262–73. pmid:23076616
- 32. Poggio R, Gutierrez L, Matta MG, Elorriaga N, Irazola V, Rubinstein A. Daily sodium consumption and CVD mortality in the general population: systematic review and meta-analysis of prospective studies. Public Health Nutr. 2015;18(4):695–704. pmid:24848764
- 33. Schwingshackl L, Hoffmann G. Monounsaturated fatty acids, olive oil and health status: a systematic review and meta-analysis of cohort studies. Lipids Health Dis. 2014;13:154. pmid:25274026
- 34. Wang X, Ouyang Y, Liu J, Zhu M, Zhao G, Bao W, et al. Fruit and vegetable consumption and mortality from all causes, cardiovascular disease, and cancer: systematic review and dose-response meta-analysis of prospective cohort studies. BMJ. 2014;349:g4490. Epub 2014/07/31. pmid:25073782
- 35. Chen GC, Yang J, Eggersdorfer M, Zhang W, Qin LQ. N-3 long-chain polyunsaturated fatty acids and risk of all-cause mortality among general populations: a meta-analysis. Scientific reports. 2016;6:28165. pmid:27306836
- 36. Cheng P, Huang W, Bai S, Wu Y, Yu J, Zhu X, et al. BMI Affects the Relationship between Long Chain N-3 Polyunsaturated Fatty Acid Intake and Stroke Risk: a Meta-Analysis. Sci Rep. 2015;5:14161. pmid:26369699
- 37. Cheng P, Wang J, Shao W, Liu M, Zhang H. Can dietary saturated fat be beneficial in prevention of stroke risk? A meta-analysis. Neurol Sci. 2016;37(7):1089–98. pmid:26979840
- 38. de Souza RJ, Mente A, Maroleanu A, Cozma AI, Ha V, Kishibe T, et al. Intake of saturated and trans unsaturated fatty acids and risk of all cause mortality, cardiovascular disease, and type 2 diabetes: systematic review and meta-analysis of observational studies. BMJ. 2015;351:h3978. pmid:26268692
- 39. Narain A, Kwok CS, Mamas MA. Soft drinks and sweetened beverages and the risk of cardiovascular disease and mortality: a systematic review and meta-analysis. Int J Clin Pract. 2016;70(10):791–805. pmid:27456347
- 40. Grisso JA. Making comparisons. Lancet. 1993;342(8864):157–60. pmid:8101260
- 41. Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34. pmid:26092286
- 42. Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002. pmid:21784880
- 43. Ioannidis JP, Trikalinos TA. An exploratory test for an excess of significant findings. Clinical Trials. 2007;4:245–53. pmid:17715249
- 44. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990;263(10):1385–9. pmid:2406472
- 45. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457–65. pmid:15161896
- 46. Kennedy D. The old file-drawer problem. Science (New York, NY). 2004;305(5683):451. pmid:15273363
- 47. Newcombe RG. Towards a reduction in publication bias. BMJ. 1987;295(6599):656–9. pmid:3117278
- 48. Colonna G, Ragone R. Referee bias. Nature. 1994;367(6459):108. pmid:8114900
- 49. Turner EH, Knoepflmacher D, Shapley L. Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database. PLoS Med. 2012;9(3):e1001189. pmid:22448149
- 50. Bukkapatnam RN, Gabler NB, Lewis WR. Statins for primary prevention of cardiovascular mortality in women: a systematic review and meta-analysis. Prev Cardiol. 2010;13(2):84–90. pmid:20377811
- 51. Kizer JR, Madias C, Wilner B, Vaughan CJ, Mushlin AI, Trushin P, et al. Relation of different measures of low-density lipoprotein cholesterol to risk of coronary artery disease and death in a meta-regression analysis of large-scale trials of statin therapy. Am J Cardiol. 2010;105(9):1289–96. pmid:20403481
- 52. Kostis WJ, Cheng JQ, Dobrzynski JM, Cabrera J, Kostis JB. Meta-analysis of statin effects in women versus men. J Am Coll Cardiol. 2012;59(6):572–82. pmid:22300691
- 53. Lv HL, Jin DM, Liu M, Liu YM, Wang JF, Geng DF. Long-term efficacy and safety of statin treatment beyond six years: a meta-analysis of randomized controlled trials with extended follow-up. Pharmacol Res. 2014;81:64–73. pmid:24602799
- 54. Ray KK, Seshasai SR, Erqou S, Sever P, Jukema JW, Ford I, et al. Statins and all-cause mortality in high-risk primary prevention: a meta-analysis of 11 randomized controlled trials involving 65,229 participants. Arch Intern Med. 2010;170(12):1024–31. pmid:20585067
- 55. Savarese G, Gotto AM Jr., Paolillo S, D’Amore C, Losco T, Musella F, et al. Benefits of statins in elderly subjects without established cardiovascular disease: a meta-analysis. J Am Coll Cardiol. 2013;62(22):2090–9. pmid:23954343
- 56. Taylor F, Ward K, Moore TH, Burke M, Davey Smith G, Casas JP, et al. Statins for the primary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2011;(1):CD004816. pmid:21249663
- 57. Tonelli M, Lloyd A, Clement F, Conly J, Husereau D, Hemmelgarn B, et al. Efficacy of statins for primary prevention in people at low cardiovascular risk: a meta-analysis. CMAJ. 2011;183(16):E1189–202. pmid:21989464
- 58. Chou R, Dana T, Blazina I, Daeges M, Jeanne TL. Statins for Prevention of Cardiovascular Disease in Adults: Evidence Report and Systematic Review for the US Preventive Services Task Force. JAMA. 2016;316(19):2008–24. pmid:27838722
- 59. Preiss D, Campbell RT, Murray HM, Ford I, Packard CJ, Sattar N, et al. The effect of statin therapy on heart failure events: a collaborative meta-analysis of unpublished data from major randomized trials. Eur Heart J. 2015;36(24):1536–46. pmid:25802390
- 60. Teng M, Lin L, Zhao YJ, Khoo AL, Davis BR, Yong QW, et al. Statins for Primary Prevention of Cardiovascular Disease in Elderly Patients: Systematic Review and Meta-Analysis. Drugs Aging. 2015;32(8):649–61. pmid:26245770
- 61. Siontis GC, Ioannidis JP. Risk factors and interventions with statistically significant tiny effects. Int J Epidemiol. 2011;40(5):1292–307. pmid:21737403
- 62. Hemkens LG, Ewald H, Naudet F, Ladanie A, Shaw JG, Sajeev G, et al. Interpretation of epidemiological studies very often lacked adequate consideration of confounding. Journal of clinical epidemiology. 2017. pmid:28943377
- 63. Egger M, Schneider M, Davey Smith G. Spurious precision? Meta-analysis of observational studies. BMJ. 1998;316(7125):140–4. pmid:9462324
- 64. Stamatakis E, Weiler R, Ioannidis JP. Undue industry influences that distort healthcare research, strategy, expenditure and practice: a review. Eur J Clin Invest. 2013;43(5):469–75. pmid:23521369
- 65. Moodie R, Stuckler D, Monteiro C, Sheron N, Neal B, Thamarangsi T, et al. Profits and pandemics: prevention of harmful effects of tobacco, alcohol, and ultra-processed food and drink industries. Lancet. 2013;381(9867):670–9. pmid:23410611
- 66. Fanelli D, Costas R, Ioannidis JP. Meta-assessment of bias in science. Proc Natl Acad Sci U S A. 2017;114(14):3714–9. pmid:28320937
- 67. Fanelli D. Redefine misconduct as distorted reporting. Nature. 2013;494(7436):149. pmid:23407504
- 68. Ioannidis JP. The importance of potential studies that have not existed and registration of observational data sets. JAMA. 2012;308(6):575–6. pmid:22871867
- 69. Bothwell LE, Greene JA, Podolsky SH, Jones DS. Assessing the Gold Standard—Lessons from the History of RCTs. N Engl J Med. 2016;374(22):2175–81. pmid:27248626
- 70. Zarin DA, Tse T, Sheehan J. The Proposed Rule for U.S. Clinical Trial Registration and Results Submission. N Engl J Med. 2014;372(2):174–80. pmid:25539444
- 71. Taichman DB, Sahni P, Pinborg A, Peiperl L, Laine C, James A, et al. Data Sharing Statements for Clinical Trials—A Requirement of the International Committee of Medical Journal Editors. N Engl J Med. 2017;376(23):2277–9. pmid:28581902
- 72. Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, et al. A manifesto for reproducible science. Nature Human Behaviour. 2017;1:0021.
- 73. Ioannidis JA. The proposal to lower p value thresholds to .005. JAMA. 2018;319(14):1429–30. pmid:29566133
- 74. Chartres N, Fabbri A, Bero LA. Association of industry sponsorship with outcomes of nutrition studies: A systematic review and meta-analysis. JAMA Internal Medicine. 2016;176(12):1769–77. pmid:27802480
- 75. Serodio PM, McKee M, Stuckler D. Coca-Cola—a model of transparency in research partnerships? A network analysis of Coca-Cola’s research funding (2008–2016). Public Health Nutr. 2018:1–14. pmid:29560842
- 76. Gomez-Garcia F, Ruano J, Gay-Mimbrera J, Aguilar-Luque M, Sanz-Cabanillas JL, Alcalde-Mellado P, et al. Most systematic reviews of high methodological quality on psoriasis interventions are classified as high risk of bias using ROBIS tool. J Clin Epidemiol. 2017;92:79–88. pmid:28893571
- 77. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC medical research methodology. 2007;7:10. pmid:17302989
- 78. Xing D, Wang B, Zhang W, Yang Z, Hou Y, Chen Y, et al. Intra-articular hyaluronic acid injection in treating knee osteoarthritis: assessing risk of bias in systematic reviews with ROBIS tool. Int J Rheum Dis. 2017;20(11):1658–73. pmid:29044993
- 79.
Higgins JPT, G S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]: The Cochrane Collaboration; 2011.
- 80. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006;295(6):676–80. pmid:16467236
- 81. Tsilidis KK, Kasimis JC, Lopez DS, Ntzani EE, Ioannidis JP. Type 2 diabetes and cancer: umbrella review of meta-analyses of observational studies. BMJ. 2015;350:g7607. pmid:25555821
- 82. Owen N, Sparling PB, Healy GN, Dunstan DW, Matthews CE. Sedentary behavior: emerging evidence for a new health risk. Mayo Clin Proc. 2010;85(12):1138–41. pmid:21123641
- 83. Ioannidis JP. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. The Milbank quarterly. 2016;94(3):485–514. pmid:27620683
- 84. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296. pmid:17941714
- 85. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283(15):2008–12. pmid:10789670
- 86. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. pmid:19622551
- 87. Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ. 2011;344:d7762. pmid:22214758
- 88. Ioannidis JPA. Clarifications on the application and interpretation of the test for excess significance and its extensions. J Math Psychol. 2013;57(5):184–7. http://dx.doi.org/10.1016/j.jmp.2013.03.002.
- 89. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629–34. pmid:9310563
- 90. Poole R, Kennedy OJ, Roderick P, Fallowfield JA, Hayes PC, Parkes J. Coffee consumption and health: umbrella review of meta-analyses of multiple health outcomes. BMJ. 2017;359:j5024. pmid:29167102
- 91. Sterne JA, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol. 2000;53(11):1119–29. pmid:11106885