visibility , self-citation , and reference limits : influences on Impact Factor and author performance review

Bibliometrics, including the Journal Impact Factor (JIF), assist decision-making in publishing and academic reviews. Some ‘cutting edge’ papers are not published in ‘high ranking’ journals because they were either too controversial, ‘blocked’ during the review process, rejected due to the random factor introduced by numerous competing good submissions, or editor bias. Such papers, when published in other journals, may still become well known and accepted. Examples of this may be found within conference proceedings, journals with moderate JIF and Citations per Publication (CPP), and PhD dissertations. Journals such as Science and Nature may be overrated. A paper published there may or may not be better than one published in journals with a more restricted distribution and coverage. Excellent papers occur in both types of journals. Low acceptance rates (10%) introduce a random component into the review process along with potential editor bias during preliminary review. The overflow moves to more standard journals. CPP is often used to assess a researcher’s impact on his/her field and as an impact indicator for a specific article. The belief that bibliometrics are manipulated by an author to increase the CPP through self-citation is probably perceptual. Self-citation is necessary to inform the reader about the author’s prior work and provide background information. Low self-citation rates can lead a reviewer to believe the author’s background is inadequate, while high rates might indicate that he/she is ignoring the work of colleagues. A balance is recommended. Limits on references can act similarly, limiting background information for the reader and denying justifiable citations to other authors conducting important relevant research. Bibliometrics can serve certain purposes well, but are not perfect. Low bibliometric indicators do not necessarily reflect poorly on a researcher or the relevant journal.


INTRODUCTION
Bibliometrics have been used by scientists, institutions, and publishers as a basis for decision-making in academia for decades (Luukkonen 1990, Sellen 1993, Browman & Stergiou 2008), and their accuracy and usefulness has been widely debated (Garfield 1994, 1996, Young 2006, Browman & Stergiou op. cit, Butler 2008, Harnad 2008, Todd & Ladle 2008).Citation frequency is commonly used to assess journals.Indicators include the Journal Impact Factor (JIF; Garfield 2006, Campbell 2008).In addition, the hindex (Hirsch 2005) and g-index (Egghe 2006) are frequently used for assessing researchers/authors.Number of citations per paper (CPP) can be used in either application.These metrics can be quite valuable, but may require a better understanding of some subtle forces of influence and their effects on the metric.This will help to reduce the possibility of misinterpretation (Phelan 1999, Browman & Stergiou 2008).When referring to impact factors, I will restrict my comments to the JIF.ABSTRACT: Bibliometrics, including the Journal Impact Factor (JIF), assist decision-making in publishing and academic reviews.Some 'cutting edge' papers are not published in 'high ranking' journals because they were either too controversial, 'blocked' during the review process, rejected due to the random factor introduced by numerous competing good submissions, or editor bias.Such papers, when published in other journals, may still become well known and accepted.Examples of this may be found within conference proceedings, journals with moderate JIF and Citations per Publication (CPP), and PhD dissertations.Journals such as Science and Nature may be overrated.A paper published there may or may not be better than one published in journals with a more restricted distribution and coverage.Excellent papers occur in both types of journals.Low acceptance rates (10%) introduce a random component into the review process along with potential editor bias during preliminary review.The overflow moves to more standard journals.CPP is often used to assess a researcher's impact on his/her field and as an impact indicator for a specific article.The belief that bibliometrics are manipulated by an author to increase the CPP through self-citation is probably perceptual.Self-citation is necessary to inform the reader about the author's prior work and provide background information.Low self-citation rates can lead a reviewer to believe the author's background is inadequate, while high rates might indicate that he/she is ignoring the work of colleagues.A balance is recommended.Limits on references can act similarly, limiting background information for the reader and denying justifiable citations to other authors conducting important relevant research.Bibliometrics can serve certain purposes well, but are not perfect.Low bibliometric indicators do not necessarily reflect poorly on a researcher or the relevant journal.

OPEN PEN ACCESS CCESS
Here, I will show how journal visibility alone, particularly with respect to 'super' vs. standard journals, can distort CPP and perception of the quality of an article, and how random effects and editor bias can affect the probability of acceptance in the super-journals.I will also discuss how self-citations, necessary to the publishing process, can be mis-perceived by reviewers of papers, proposals, and performance.Finally, the side effects and disadvantages for both the author and the reader of limiting the number of references given in a paper will also be considered.

CITATIONS AND THE JOURNAL IMPACT FACTOR
The JIF (Garfield 2006) has been used as an indicator of how well-read a journal is and how wellaccepted the papers published in it are.Citation frequency alone, however, can be misleading.It is important to take the visibility of the journal into account, because visibility of the article can have as much of an influence on the metric as its quality (Peritz 1995, Opthof 1997, Marusic & Marusic 1999, Porta et al. 2003).Papers that are more widely distributed, both in print and in electronic form, may become better known.In the current electronic age, online distribution and open access to articles has greatly increased the visibility and accessibility of publications, authors, and journals (Ugolini & Casilli 2003, Taylor et al. 2008), with some exceptions (e.g.Chinese scientific journals; Marusic & Marusic 1999, Ren & Rousseau 2002).The internet has had the effect of leveling the playing field in publishing.Many journals are getting attention not previously received (Harzing & van der Wal 2008).
Scientists often cite material to which they have been readily exposed, often from high impact journals.These journals have greater exposure, people more readily see the articles in them, the articles are cited more often, and thus the journal impact factor increases (Hecht et al. 1998).This is a positive feedback loop, maintained at a high financial cost to the publisher, generated by substantial marketing, promotion, and distribution activities.

'SUPER-JOURNALS' VS. STANDARD JOURNALS
The best example of high impact journals are the 'super journals' (e.g.Science, Nature).Because of the skewed visibility and attention they receive, their impact factors may be overrated (Campbell 2008).It is perceived that publications in these journals represent higher quality research, and thus a paper published there may be of a higher quality and therefore yield a higher CPP than one published in a specialty journal with a more restricted coverage and distribution.A close examination of such super-journals vs. journals such as Marine Biology, Marine Ecology Progress Series, or Journal of Experimental Biology and Ecology will reveal that 'cutting edge' and more standard papers appear in both.
Many excellent articles are submitted to the superjournals.These journals have a very high rejection rate (≥90%, Bloom 2000).Most manuscripts do not pass a preliminary screening process by an editorial board member -a person who has a general working knowledge of the field covered by a given paper, but may or may not have in-depth knowledge of the specific topic.The journal's necessity of a high rejection rate and its potentially relatively shallow preliminary scan introduces a substantial random component into the review process.When the editor is presented with a large number of good choices but a small number of slots for acceptance, a large component of luck and potential bias can enter into the equation for (or against) the submission.In the words of Browman & Stergiou (2008, p. 2), 'Once competition reaches such rarefied levels, decisions come down to qualitative judgments; that is, to people being people.No metric will change that.'The overflow of rejected excellent papers, however, will move downhill to the more standard journals, and there they will be published.The result is a high variance in citation rates within both types of journals.Indeed, Seglen (1994) (also see Whitehouse 2001) found no correlation between citation frequency of an article and JIF, and Garfield (2006) has pointed out that some articles published in journals with the highest JIFs are either lightly cited or not cited at all (cf.Browman & Stergiou 2008).
Should CPPs derived from papers published in super-journals vs. standard scientific journals be taken at face value?One might expect that a paper published in a journal such as Science would, on average, receive more citations than if the same paper was published in, for example, Marine Biology -purely on the basis of visibility.The quality of the science presented would be the same whether published in either place, but one would expect a difference in CPP.This opens up the question as to whether a weighting factor designed to compensate for visibility should be used for publications in these respective sets of journals.I believe that this might be a fruitful area for future research.
for considerations such as salary increments, promotion, and tenure (Bordons et al. 2002).These metrics are often misleading, and Garfield (1996) has recommended that, although JIF can be important in evaluating a journal, it should not be used in faculty evaluation.Browman & Stergiou (2008, p. 2) state that 'it is not always […] the best articles that get published in the top journals.'Interestingly, one may not necessarily receive the highest citation frequency for the article (or articles) that one personally believes is of greatest importance.Perhaps what we individually feel are our most important contributions to science are not necessarily such.
Some believe that CPP can be manipulated by the researcher by increasing self-citation and, as a result, increasing the apparent acceptance of their work (Garcia-Ferrer & Poncela 2002, Pichappan & Sarasvady 2002).One could theoretically cite oneself at every opportunity (West & Stenius 2005).Adjustments have recently been proposed for the h-index to help correct it for self-citations (Schreiber 2007a,b).There are a number of ways in which self-citation can automatically increase.For example, working in large collaborative groups can potentially influence the self-citation rate, as there are simply more authors to cite the same paper.Van Raan (1998), however, found that correcting for self-citations where international collaborations are concerned does not play a significant role in amplifying the CPP.Fowler & Aksnes (2007) found that author self-citation does influence citation rates of one's work later; however, Aksnes (2003) found that it plays only a minor role in affecting this increase.Fassoulaki et al. (2002) found that omitting self-citations from 36 journals did not change their JIF.
Self-citation is necessary for the author to orient the reader with respect to their prior work, providing critical background for the current work being reported, particularly if the paper is one in a series.Bonzi (1998) and Bonzi & Snyder (1991) found no difference in motivation for citing oneself vs. others.Gami et al. (2004) and Kovacic & Misak (2004) found that selfcitation frequency was not associated with the quality of a publication.It is likely that the abuse in this area is perceived and not real.Reviewers of grant proposals or submitted manuscripts often interpret lack of self-citation as a sign that the author has little or no background on the subject in question.On the other hand, they perceive over self-citation as a sign of the author's narcissism and of ignoring the research of their co-workers.A viable approach probably lies in moderate citation of an author's own papers, only to the extent required to provide adequate background information.
The impact factor of journals in which a scientist has published is also regularly considered in performance reviews.Campbell (2008) notes that this is inappropriate.He found that several researchers reported their highest citation rates as coming from publications in obscure journals -primarily due to novel techniques reported therein.Some valuable, cutting-edge papers are not published in 'high ranking' journals.Some classic, vanguard papers have been published in conference proceedings (e.g.Antonius 1981a,b), journals with a moderate JIF (e.g.Lang 1971Lang , 1973; Bulletin of Marine Science, JIF = 1.093, 2006 ranking 42nd among 79 in the Marine and Freshwater Biology category), and even PhD dissertations (Harrigan 1972).Google Scholar is one example of a search engine that has opened up access to books, conference papers, non-US journals, etc. (Harzing & van der Wal 2008).
Publication of key papers in journals with lower impact factors might also occur because the papers are too controversial to be included in a higher ranking journal, despite the fact that editors of such journals are continually under pressure to find 'hot papers' (Garfield 1996).On the other hand, a submitted paper may be 'blocked' by reviewers or an editor with competing interests who may feel threatened in some way by the new results (Anonymous 2007, Sammarco 2008, author's pers. obs.).Good, solid scientific work however dies hard and, through time, will eventually surface somewhere in the published literature and has a calculable chance of becoming well-known and wellaccepted by the scientific community.It simply takes longer through alternate channels.

IMPACT OF AN IMPOSED LIMIT FOR LITERATURE CITED
Some journals are now limiting the number of references to be cited within an article, understandably due to the high number of papers being published there and resultant increases in the cost of production (e.g.Marine Ecology Progress Series, generally 1 page of citations for 5 pages of text).Unfortunately, this can affect CPP for the authors referenced in the paper and the JIF for the journals being referred to.It can also affect the impact factor of the journal imposing the limit by potentially limiting the number of journal selfcitations within the article.Further, the restricted author will most likely deny herself/himself justifiable self-citations.
Limiting number of citations would appear to be a negative step for a journal.It can limit an author in providing a full background on his/her research topic.In addition, as occurs frequently in reviewing, it opens the author up to criticism regarding 'insufficient literature review', which then becomes un-addressable because of the very restrictions that precipitated the comment.It also increases the probability of not citing papers written by the reviewer, which may result in a negative review.To counter this, the author can increase the diversity of authors cited by keeping the number of citations-per-author low.This will help to ameliorate the situation, but it will still handicap the author's ability to provide a solid foundation for his/her report.Permitting the necessary number of references for an article is clearly preferable.Garfield (1996) has pointed out that authors should be encouraged to cite all relevant literature in their paper, and that editors should avoid any artificial limitation on number of references, except in cases where such may be selfserving.

CLOSING COMMENT
Bibliometrics can serve as valuable indicators in a number of applications (Saha et al. 2003, Browman & Stergiou 2008) such as the evaluation of research (Bornmann et al. 2008) or in performance evaluations (Butler 2008, Harnad 2008).A decrease in CPP for an individual or the impact factors of journals in which that person has published, however, do not necessarily cast a shadow on that person or his/her work.Such changes may be driven by a number of subtle variable forces acting independently of research or publication quality.The same principles apply to the impact factor of the journal itself (Amin & Mabe 2003).For example, Ogden & Batley (2008) reported that the JIF of the Annals of Occupational Hygiene rose by 68% in one year -from 2002 to 2003.They also found that journals with a JIF of ~1.5 exhibit a variance of 10 to 20% annually, making the reporting of JIFs to 3 decimal places meaningless.Another finding was that the JIF is a poor indicator of a paper's citation performance or the author's success; a CPP specific to a paper by the author is much better.
Bibliometrics need to be taken with a grain of salt.In concurrence with Campbell (2008), there is no better way to evaluate research or a researcher than to sit down and read his/her paper.The journal of publication is relevant, but of secondary importance.The number of self-citations in a CPP analysis is generally irrelevant, unless excessive.
KEY WORDS: Bibliometrics • Citations • CPP • Self-citation • Impact factor • Performance review • Super-journals • Visibility • Reference limits Resale or republication not permitted without written consent of the publisher