Article Text
PDF
Abstract
Objectives Epidemiological research on the association between diesel exhaust exposure and lung cancer risk has some methodological challenges that give rise to different conclusions and intense debates. This raises the question about the role of selective citation and of citation bias in particular. Our aim was to investigate the occurrence and prevalence of selective citation in this field.
Design Citation analysis.
Setting Web of Science Core Collection.
Participants We identified 96 publications in this network, with 4317 potential citations. For each publication, we extracted characteristics such as study conclusion and funding source. Some of these characteristics are related to the study content: study design, sample size, method of diesel exposure assessment, type of diesel technology under investigation, and whether smoking had been adjusted for.
Primary and secondary outcome measures Whether a citation occurs or not, measured and analysed according to the preregistered protocol. Exploratively we analysed the association between funding source and study conclusion.
Results Methodological content of a study was clearly related to citation, studies using more sophisticated methods were more likely to be cited. There was some evidence for citation bias: supportive publications had a higher chance of being cited than non-supportive ones, but after adjustment for study quality, this effect decreased substantially (adjusted OR 1.3, 95% CI 1.0 to 1.7). Explorative analyses indicated that three quarters of non-profit funded publications had a supportive study conclusion against only one quarter of the industry-funded publications.
Conclusions There is evidence for selective citation within this field, but the evidence for citation bias was weak. It seems that factors related to the methodology had more impact on citation than the conclusion of a study. Interestingly, publications that were funded by industry were more skeptical about a causal relationship between diesel exhaust and lung cancer compared to non-profit-funded publications.
- selective citation
- citation bias
- diesel exhaust
- lung cancer
- funding source
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This study is the first citation analysis on the diesel exhaust exposure–lung cancer hypothesis.
Special attention was given to the assessment and the analysis of the conclusion, factors related to the methodological content and funding source of studies on this hypothesis.
Not all quality aspects could be taken into account.
We did not score whether a citation was supportive or critical of the cited publication.
Background
Seven years ago, diesel exhaust was classified as a carcinogenic agent by the International Agency for Research on Cancer.1 2 It is estimated that, in Great Britain alone, about 2% of all lung cancer (LC) cases, and 13% of those related to occupation, are attributable to diesel exhaust exposure (DEE).3 Nevertheless, there is still debate about the causal relationship between DEE and LC. Many methodological weaknesses have been pointed out in the epidemiological literature, both by scientists working for federal agencies and by those funded by car manufacturers.4–6 At the same time, car manufacturers have been accused of misleading practices and cheating.7 Whether the different stakeholders take their opponents’ view into account could be revealed by a citation analysis. Ideally, no relevant evidence should be ignored, but this is not always the case.8 The current study does not concern the validity of the DEE–LC hypothesis per se but focuses on the citation patterns within the scientific literature on this hypothesis.
Studying the impact of DEE on LC indeed comes with methodological challenges. Perhaps the most notable methodological challenge is the measurement of exposure to diesel exhaust.9 10 It has often been measured not only by crude indicators like job title or self-assessment but also by more sophisticated methods such as a job-exposure matrix based on actual measurements.10 Other methodological challenges include the adjustment for potential confounders such as smoking,11 the healthy worker effect,12 and the change in composition of diesel exhaust over time, due to improved technology and regulations.13
The number of publications on the DEE—LC hypothesis is large. It is therefore not feasible for authors to cite every relevant publication in the network and some selection needs to take place. If this selection is based on the results or the conclusion of a study, we speak of citation bias.8 14 Citation bias has been studied in many disciplines.15 These studies typically show evidence for citation bias, with supportive publications being cited about two times as often as non-supportive ones. The consequences of citation bias can be similar to those of publication bias and reporting bias: disregard of counterevidence leading to unfounded consensus16 or polarisation,17 ill-advised research programmes and research waste,16 18 distorted information in the media19 and misplaced decisions about patients.20
Factors other than study conclusion may also have an impact on citation, as was recently shown by Onodera and Yoshikane.21 Measures for journal status (impact factor), author status (number of citations, country of affiliation) and collaboration (number of authors, number of affiliations) were often found to be related to citation counts. The same was consistently found for the number of references of the cited publication. Furthermore, the reporting22 and source23 24 of funding were shown to be related to citation, but the impact of the publication’s title25–27 and the author’s affiliation28 and gender29–31 is less clear. On the other hand, sample size and study design—both markers of study quality, and as such legitimate reasons to base a citation on—often seem unrelated to citation.24 28 32–34 In our previous citation networks, we also found associations with self-citation and the specificity of a publication.25 26
In this study, we aimed to (a) assess the occurrence of citation bias in the scientific literature on the DEE—LC hypothesis and (b) test for other forms of selective citation by assessing the impact of the other factors described above. We used the claim-specific methodology developed by Greenberg,16 but with a modification of the statistical analysis that allowed us to test the impact of multiple factors, to adjust for some quality indicators and to take into account the variation in publication time. Exploratively we also checked the association between funding source and the conclusion of a study.
Method
Prior to performing our study, the study protocol including data-analysis plan was stored at an online repository.35 Deviations from that protocol are described in the online supplementary text S1. In brief, we applied a search strategy on DEE and LC to the Web of Science Core Collection, selected relevant publications (see online supplementary text S2 for details) and extracted data for each selected publication. Publication selection was performed independently by GMHS and BD. Reasons to exclude publications were related to i.e. the language, study design and scientific discipline. All non-English publications were excluded, together with editorials and letters to the editor. With regards to scientific discipline, the network consisted only of epidemiological (human) studies. Consequently, all laboratory studies and animal studies and other studies that focused on the toxicological mechanism of diesel exhaust and LC were excluded from the network. The central inclusion criterion was that each publication needed to make a statement on the association between diesel exposure and LC in humans.
Supplemental material
After the publication selection, the data extraction was performed independently by MU and BD. Disagreements were resolved in consensus meetings. Next, we built a data set with potential citation paths and used specialised software to determine which potential citations had occurred. Finally, we analysed which characteristics were related to the likelihood to be cited.
For clarification: a publication in our network can both cite and be cited by other publications in the network, leading to a multitude of citation paths. Not all citation paths are possible as one can only cite publications that were published before. In our study, a citation is considered possible if the cited publication is published before the citing publication is submitted (see also online supplementary text S3). If such potential citation occurred, we call it an actual citation.
Data extraction
A range of characteristics were extracted or derived from each included publication. These characteristics are described below and were all tested as determinant of citation in the statistical analysis.
Study conclusion
Study conclusion had the following outcomes: (1) supportive (positive association between DEE and LC), (2) non-supportive (no positive association), (3) unclear (no conclusion could be drawn based on the included evidence), (4) mixed (supportive in some subgroups but not in other subgroups) or (5) no conclusion on DEE–LC was stated (see online supplementary text S3 for more details).
Other content-related characteristics
The following variables were in this category: publication type and study design, sample size, type of DEE, type of exposure assessment and smoking adjustment. We believe these variables are related to the methodological quality of a publication, as they were often highlighted in the DEE–LC literature as limitations of empirical studies on this topic.
Publication type was classified into empirical publications and reviews. Empirical publications were further classified into the study designs case–control and cohort study. Reviews were further classified into narrative reviews and systematic reviews.
Sample size concerned the number of participants in an empirical publication. Reviews had no sample size. Sample size was classified into three equal categories based on tertiles. In this way, we aimed to analyse the chance of being cited for small, medium and large studies.
Smoking adjustment was scored as yes or no. Smoking is a strong potential confounder when it comes to LC and studies with this adjustment are expected to produce more valid results. Consequently, a study that did adjust for smoking (regardless of how this was operationalised) is considered of higher methodological quality compared with a study that did not apply this adjustment in their analyses.
Type of exposure assessment. Many of the early studies assessed exposure by means of job proxy. Later on more sophisticated methods were employed (see online supplementary text S3 for more details).
Type of DEE changed over time due to improved diesel motor and fuel technology, leading to less exhaust containing fewer harmful particles. Roughly, three types of diesel exhaust have been differentiated in the literature: traditional (up to 1988), transitional and new technology diesel (2006 and after).36 37 To study the present-day carcinogenicity of diesel exhausts, it makes more sense to focus on newer (and potentially less carcinogenic) types of diesel exhaust rather than the old ones.13 These new insights into types of diesel exhaust can be seen as a justified reason to cite newer publications, instead of older publications, which might be based on outdated theories.
Content-unrelated publication characteristics
The other publication characteristics are conclusiveness of the title, funding source, number of authors, number of affiliations, number of references and specificity. Title conclusiveness was coded as yes if in the title a conclusion was stated that included the direction of the relationship (eg, ‘increased risk of LC among different types of professional drivers in Denmark’), otherwise as no (eg, ‘diesel exhaust and LC mortality in potash mining’). Funding source was coded as exclusively non-profit (eg, government or university), exclusively for profit, both or not reported (online supplementary text S3). The specificity of the publications was varied. Some publications only deal with the DEE–LC hypothesis (very specific), others are broader (online supplementary text S3).
Journal characteristics
Journal characteristics are publisher and journal impact factor. Journal impact factor, in the publication year of the cited publication, was retrieved from the Journal Citation Reports (JCR) database.38 Journal publisher was also retrieved from JCR.
Author characteristics
Author characteristics are gender of the corresponding author (online supplementary text S3), continent of the corresponding author and affiliation of the corresponding author. Affiliation was classified as government, university, industry or other. Academic hospitals were scored as ‘university’ and non-academic hospitals as ‘other’.
Citation characteristics
There were some variables that depend on the cited publication as well as the citing publication: self-citation and authority. A self-citation was defined as a citation between two publications that have at least one author in common. Authority was a measure for the authority of the authors within the network (online supplementary text S3). In case of multiple authors, we used the authority value of the author with the highest authority.
Statistical analysis
The data set consisted of all potential citation paths between cited and citing publications. Our binary dependent variable was citation within the network (or, more precisely, whether a potential citation had occurred or not; see also online supplementary text S4). All analyses were preregistered in the study protocol unless mentioned otherwise.
Determinants of selective citation
The characteristics of the cited publication, which are described above, served as determinants of citation in our analyses. Since each publication could refer to multiple other publications, the potential citation paths were related. Therefore, we used a multilevel approach in which the potential citations were nested under the citing publication. Specifically, we performed a univariate random-effects logistic regression for each determinant of citation.
Additional analyses of selective citation were performed on subsets of the network. First, to investigate specifically which empirica evidence is picked up by the rest of the field, we repeated the main analysis, but without the citations to reviews. We performed a univariate and multivariate analysis for each determinant. In this multivariate analysis, we adjusted for study design and for content-related variables that are distinctive for empirical publications, namely, log-transformed sample size, type of DEE, type of exposure assessment and smoking adjustment. The sample size was logtransformed to adjust for the fact that sample size as a continuous variable was very skewly distributed. Second, to investigate which empirical evidence is picked up by reviews in particular, we repeated this analysis on a subset of cited empirical publications and citing reviews. Finally, we checked the robustness of our findings by running a sensitivity analysis on the complete network, including citations to reviews, but without the most cited publications. The most cited publications could have a disproportionate impact on the ORs because of their high citation counts and lead to wrong conclusions about the citation patterns in the network as a whole.
Funding source and study conclusion
This association was exploratively analysed by fixed-effects logistic regression, with source of funding as determinant and study conclusion as outcome variable. To check whether an association could be explained by differences in study quality, we repeated this analysis while adjusting for the quality indicators described previously. Similarly, an explorative analysis was performed on the association of study conclusion with the corresponding author’s type of affiliation.
Software
We used the built-in algorithm of CitNetExplorer V.1.0.0 to extract the actual citations between publications.39 We used R V.3.4.3 to create a data set with all potential citation paths, based on the data extraction sheet and actual citations, and also to calculate the authority and self-citation scores for each potential citation path. We used Stata V.13.1 to analyse the results.
Results
A total number of 96 publications were identified that fit our criteria, published between 1988 and 2017 (figure 1, online supplementary text S5). Figure 2 depicts a visualisation of the network. This network had a total of 4317 potential and 678 actual citation paths (16%) between these publications. The number of citations ranged from 0 (10 publications) to 34, with a median of 5 citations per publication. On average, the number of citations to a publication reached its peak around 5 years after publication and then attenuates again (figure 3). A ranking of the most cited publications and authors are found in online supplementary table S1.
Flowchart of the article selection process.
Network visualisation of the 96 publications on the association between diesel exposure and lung cancer, published between 1988 and 2017. Each node is a publication, each line represents a performed citation. The 1998 meta-analysis of Bhatia et al.53 (indicated with the square) is the most cited publication, with a total of 34 received citations within the network.
Likelihood of citation over time.
The main characteristics of the publications are depicted in table 1 (for more details see online supplementary table S2). About three quarters of all publications in the network were empirical (34 case–control and 40 cohort studies), and about one quarter were reviews (17 narrative and five systematic reviews). The study conclusion for 51 of the publications was supportive regarding the hypothesis that DEE is related to LC, 9 publications concluded that the hypothesis must be rejected (no association) and 25 publications concluded that the association is unclear. Because both the publications with no association and those with unclear association did not support the hypothesis, and because they had similar citation rates, we analysed them as one group (non-supportive publications). The rest had stated a mixed conclusion (2 publications) or no conclusion on the DEE–LC hypothesis (9 publications).
Main characteristics of all 96 included publications on the DEE–LC hypothesis
Determinants of selective citation
The results of the univariate regression analyses are presented in table 2 (first column). Case–control studies and cohort studies had a similar likelihood of citation, whereas narrative reviews were less likely and systematic reviews were much more likely to be cited. Other quality-related determinants of citation were sample size, type of diesel exposure, type of exposure assessment and smoking adjustment, with higher quality publications indeed showing a higher likelihood of citation. There was some citation bias in this field: publications with a supportive study conclusion were more likely to be cited than non-supportive publications (OR 1.7, 95% CI 1.4 to 2.1). The magnitude of citation bias was quite stable over time (online supplementary figure S1). Other determinants that increased the likelihood of citation were high specificity, high journal impact factor, reporting of funding (but not source of funding), large number of authors, affiliations and references, authority and the continent of the corresponding author. With regards to the latter, being affiliated with an organisation in North America led to the highest chance of citation, whereas being affiliated in Asia and Australia resulted in the lowest chance of being cited.
Supplemental material
ORs (95% CIs) for the likelihood of being cited
We repeated the above analyses on the subset of citations to empirical publications only and tested which of their characteristics were related to the citation. The results of the crude analyses (table 2, middle column) are very similar to the analyses on the complete network that includes the cited reviews. Additionally, we repeated the analyses on the subset of cited empirical publications, while adjusting for the content-related variables described above. The results are shown in table 2 (right column). There were some small differences with the crude analyses. The impact of study conclusion decreased after adjustment for content-related variables, just like the impact of specificity, authority and number of authors and affiliations. The impact of sample size increased, and female authors had a higher likelihood of citation compared with male authors for empirical publications of similar quality.
Of particular interest is the question which empirical evidence is covered by reviews. To answer this question, the above analysis was repeated on a subset of citations between cited empirical publications and citing reviews. Overall, the results are very similar to the previous analysis, although it seems as if review authors have an even higher tendency to cite publications in high-impact factor journals and from authors with high authority (online supplementary table S3).
The results of the sensitivity analyses without the four most cited publications are shown in online supplementary table S4. The impact of study conclusion disappeared, and the increased likelihood of citation of systematic reviews disappeared. The other results are largely the same.
Funding source and study conclusion
Exploratively we checked the relationship between source of funding and study conclusion. There were 46 publications with exclusively non-profit funding, nine with exclusively for-profit funding and five with both sources of funding. The conclusions of publications with exclusively for-profit funding and with both sources of funding were very similar (table 3), so we combined them in the analysis (for-profit publications). The majority of the for-profit publications had received funding from the petroleum industry and car manufacturers (11 out of 14). Publications with exclusively non-profit funding were much more likely to reach a supportive conclusion than publications with for-profit funding (OR 8.2, CI 1.9 to 36). Adjustment for quality could not explain this association (online supplementary table S5 and S6).
Association between source of funding and study conclusion. Number of publications (and percentage of publications) per category
Affiliation of the corresponding author had a similar relationship with study conclusion. Publications from academic institutions were more likely to reach a supportive conclusion than those from government (OR 2.8, CI 0.8 to 9.5) and from industry. There were 11 industry-affiliated publications but their OR could not be calculated because none of them had reached a supportive conclusion.
Discussion
Our research aim was to evaluate the impact of study conclusion and other factors on the likelihood of being cited in the scientific literature on the DEE–LC hypothesis. There is some evidence for citation bias, with supportive publications being more likely to be cited than non-supportive publications. The magnitude of citation bias was consistent over the years, and thus independent of the number of publications or developmental stage of the research field. On the other hand, the magnitude of citation bias became smaller after adjustment for content-related variables, and it disappeared completely when we further excluded the four most cited publications from the analysis.
Additional to study outcome, factors related to the methodological content of the publication, such as study design and sample size, impacted the chance of being cited. Also using a more recent type of diesel exhaust and the use of a job-exposure matrix resulted in a higher chance of citation compared with older and more simple ways of measuring DEE. These factors are considered justified determinants of citation, since they all relate to the methodological quality of the study. In addition, we found that specificity, journal impact factor, number of authors and affiliations, the authors’ region, authority and self-citation all have a substantial impact on the likelihood of citation. Funding source was unrelated to citation, but we did find it to be related to study conclusion.
Publicly funded publications were found to be more supportive of the DEE–LC hypothesis than publications funded by industry. We are not sure why this is the case, but it seems to be in line with some previous research40–43—but not all44—suggesting that industry-funded research more often reaches outcomes that are favourable for industry. It has been suggested that the difference between publicly and industry-funded research mainly lies in the interpretation of the results rather than the results themselves.45 This could also be the case in our network, as we did not look at the results per se but at the conclusion drawn by the authors. If the authors had concluded that there was a relationship between DEE and LC, we scored the publication as supportive. However, if they stated the same but additionally emphasised that it remained uncertain whether this relationship was causal, we scored it as unclear. It is possible that industry-funded authors were more inclined to emphasise the lack of evidence for causality than publicly funded authors, for instance, by stressing inconsistent dose–response relationships or incomplete control for confounders. They may have been right in doing so, but it could also be part of a strategy, as was recently suggested in a New York Times article stating that ‘Research sponsored by industry all has the same fundamental aim (…) to weaken or discredit regulation—to say the evidence is not that clear’.46
Another possibility is that industry-funded researchers digged into the data until they found counterevidence for the DEE–LC hypothesis, exploiting the numerous researcher degrees of freedom when it comes to exposure measurement, latency times, subgroup analyses and so on.47 Similarly, it is possible that publicly funded authors had their own, intellectual conflict of interest48 that made them predisposed to confirm the hypothesis.49 50 They too might have ‘tortured the data until they confessed’47 or failed to notice inconsistencies and counterevidence. If this is the case, then the critical stance and argumentation of the industry-funded research could just as well have helped to improve the quality of the research field. In order to further investigate these possible explanations, one would need to check whether the original analysis plans had been followed by the researchers with public versus industrial funding. Unfortunately this was not possible, because no research protocol was mentioned in any of the publications.
An important assumption in this citation analysis is the assumption that each publication in the network can reasonably be expected to cite all other publications in the network. However, some publications might be more related to each other than others, for example by focussing on a specific population of workers. In that case, the identified determinants of citation might be slightly different in specific subnetworks within this network. In the statistical analysis, we did not adjust for these potential subnetworks, since they are often overlapping and therefore difficult to define. Additionally, only looking at a part of the network would lead to reduced statistical power.
Our study has several limitations. The research on DEE–LC is complex and has many challenges regarding measurement of exposure and impact of confounders. Also, the actual composition of diesel exhaust emissions varied over time and region. Some approaches to study DEE–LC were of better quality than others. We took the most important methodological factors into account in our analyses, measured their variation and adjusted for their impact on citation. Even though we did so to the best of our ability, we could not capture the complete variation in quality. For instance, we assessed whether adjustment for smoking was taken into account, but not the quality of the smoking adjustment, nor did we look at other potential confounders such as asbestos that specifically train personnel or mine workers may have been exposed to. Had we used more sophisticated means for quality assessment, then the impact of methodological content on the chance to be cited might have been different. Likewise, the impact of specificity and authority might have been different after adjustment for more quality indicators. Similarly, our results might have been different had we used an established assessment tool for risk of bias, rather than relying on a selection of quality indicators mentioned in the literature.
Another limitation is our use of ORs to assess the likelihood of citation. The OR may overestimate the true relative risk in studies where the outcome is common (ie, occurs in more than 5% of all cases,51). In our network, citation is quite common (16%) and thus our ORs overestimate the likelihood of citation. Additionally, a flaw in conducting our study is the lack of calculated kappa coefficients for all variables, to assess the inter-rater reliability. Data extraction was done by two researchers, in an iterative process with regular consensus meetings. Although consensus was reached in all situations, we did not report the data in such a way that a kappa coefficient for each variable could be calculated. We did have this information for the variables study outcome and funding source, which had a kappa coefficient of respectively 0.72 and 0.87. Based on these numbers and our experience in the consensus meetings, the inter-rater reliability was acceptable. Finally, we did not take into account whether the citations were supportive, neutral or critical towards the content of the cited publication, as we had decided to focus on the quantitative analysis. Reason for this can be found in the way citation is being used as a metric in the current research system. The number of citations is considered an important metric in granting promotions or research funding, regardless of the content of these citations. Additionally, one could argue that all citations generate attention for a paper, either correct or not, and in that sense each citation has the same value. Previous research has suggested that the majority of citations is supportive or neutral, and that critical citations are rather uncommon,52 but we have not checked if this is also the case in this network.
To conclude, the citation behaviour within the research on the DEE–LC hypothesis seems to be quite healthy. Methodological content had substantial impact on the likelihood of citation, and there was only weak evidence for citation bias.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors BD was involved in the research design and data analysis plan, prepared the research protocol, developed and performed the search strategy, wrote the R scripts for data transformation and calculation, performed article selection and data extraction, conducted the analyses and wrote the manuscript. MU was involved in the research design and data analysis plan, performed the data extraction, and read, commented on and approved the final manuscript. GMHS obtained funding, was involved in the research design and data analysis plan, performed article selection, and read, commented on and approved the final manuscript. MPZ obtained funding, was involved in the research design and data analysis plan, and read, commented on and approved the final manuscript. LMB was involved in the research design and data analysis plan, and read, commented on and approved the final manuscript.
Funding This work was supported by the Long-Range Research Initiative (LRI) from the European Chemical Industry Council (CEFIC), grant number LRI-Q3-UM. LRI had no role in study design, data collection and analysis, preparation of the manuscript or decision to publish.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available in a public, open access repository. The protocol, including the data-analysis plan, and the data set of this study are available upon reasonable request in the DataVerse repository,
https://hdl.handle.net/10411/G8MPEI, or by sending an email to b.duyx@maastrichtuniversity.nl.