The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level

https://doi.org/10.1016/j.joi.2007.02.001Get rights and content

Abstract

The relationship of the h-index with other bibliometric indicators at the micro level is analysed for Spanish CSIC scientists in Natural Resources, using publications downloaded from the Web of Science (1994–2004). Different activity and impact indicators were obtained to describe the research performance of scientists in different dimensions, being the h-index located through factor analysis in a quantitative dimension highly correlated with the absolute number of publications and citations. The need to include the remaining dimensions in the analysis of research performance of scientists and the risks of relying only on the h-index are stressed. The hypothesis that the achievement of some highly visible but intermediate-productive authors might be underestimated when compared with other scientists by means of the h-index is tested.

Introduction

Bibliometric studies at the micro level are increasingly requested by science managers and policy makers to support research assessment decisions. Different indicators are frequently developed at this level of analysis, generally based on both the production of scientists as well as the impact of their documents, such as the number of citations, number of citations per document or the number of highly cited papers. The combined use of several indicators that give information on different aspects of scientific output is generally recommended (i.e., Van Leeuwen, Visser, Moed, Nederhof, & Van Raan, 2003). However, the h-index was introduced in 2005 (Hirsch, 2005), comprising in a single indicator a measure of quantity and impact of the scientific output of a researcher. According to Hirsch, “a scientist has index h if h of his or her Np papers have at least h citations each and the other (Np-h) papers have ≤h citations each”.

The scientific community has shown a huge interest for this indicator, as shown by the high number of publications on the topic (Ball, 2005, Cho, 2005, Dume, 2005a, Dume, 2005b, Glanzel, 2006, Monastersky, 2005, Nazaroff, 2005, Nature, 2005, Popov, 2005). The main advantage of h-index is that it combines a measure of quantity and impact in a single indicator. It has been calculated in different fields such as physics (Hirsch, 2005), biomedicine (Bornmann & Daniel, 2005), information science (Cronin & Meho, 2006), and business (Saad, 2006). It can be useful for journal assessment (Braun, Glanzel, & Schubert, 2006; Rousseau, 2006b), for comparative description of scientific topics (Banks, 2006) and also for awarding scientific prizes (Glanzel & Persson, 2005).

Among the advantages of h-index the following have been pointed out by Hirsch (2005):

  • It combines a measure of quantity (publications) and impact (citations).

  • It allows us to characterize the scientific output of a researcher with objectivity, and therefore may play an important role when making decisions about promotions, fund allocation and awarding prizes.

  • It performs better than other single-number criteria commonly used to evaluate the scientific output of a researcher (impact factor, total number of documents, total number of citations, citation per paper rate and number of highly cited papers).

  • The h-index can be easily obtained by anyone with access to the Thomson ISI Web of Science and in addition it is easy to understand.

However, several limitations of the h-index have also been remarked:

  • There are inter-field differences in typical h values due to differences among fields in productivity and citation practices (Hirsch, 2005), so the h-index should not be used to compare scientists from different disciplines.

  • The h-index depends on the duration of each scientist's career because the pool of publications and citations increases over time (Hirsch, 2005; Kelly & Jennions, 2006). In order to compare scientists at different stages of their career, Hirsch (2005) presented the “m parameter”, which is the result of dividing h by the scientific age of a scientist (number of years since the author's first publication).

  • Highly cited papers are important for the determination of the h-index, but once they are selected to belong to the top h papers, it is unimportant the number of citations they receive. This is a disadvantage of the h-index which Egghe has tried to overcome through a new index, called g-index (Egghe, 2006b).

  • Since the h-index is easy to obtain, we run the risk of indiscriminate use, such as relying only on it for the assessment of scientists. Research performance is a complex multifaceted endeavour that cannot be assessed adequately by means of a single indicator (Martin, 1996).

  • The use of the h-index could provoke changes in the publishing behaviour of scientists, such an artificial increase in the number of self-citations distributed among the documents on the edge of the h-index (Van Raan, 2006).

  • There are also technical limitations, such as the difficulty to obtain the complete output of scientists with very common names, or whether self-citations should be removed or not. Self-citations can increase a scientist's h, but their effect on h is much smaller than on the total citation count since only self-citations with a number of citations just >h are relevant (Hirsch, 2005).

To overcome the limitations of the h-index different modifications have been suggested in the literature (Batista et al., 2005, Batista et al., 2006; Bollen, Rodriguez, & van de Sompel, 2006; Egghe, 2006a, Egghe, 2006b; Imperial & Rodríguez-Navarro, 2005; Rousseau, 2006a).

We think it is essential to continue analysing this indicator carefully, in order to establish clearly its drawbacks and limitations with the same critical and strict approach that the more traditional indicators received. It is especially relevant to determine in which cases this index could be biased, since it could have serious consequences on the assessment of individual scientists.

Section snippets

Objectives

The objective of this paper is to analyse the relationship of the h-index with other bibliometric indicators at the micro level in order to identify some of its advantages and limitations. Differences between h-index and several traditional indicators in their ability to assess research performance of scientists are given special attention. Our hypothesis is that h-index is heavily influenced by the absolute number of documents and citations and that it fails to identify those researchers who

Methodology

Scientific publications of scientists at the Spanish Research Council (CSIC) in the area of Natural Resources published during 1994–2004 were obtained from the Web of Science (WoS). Natural Resources is one of the eight scientific areas at CSIC, comprising mainly research on earth sciences, geology, marine sciences, biology, environmental sciences, and zoology.

A total of 348 permanent scientists were working at CSIC in the area of Natural Resources in 2004. A search strategy was developed

Results

The production of Natural Resources scientists amounted to 6,093 documents (all types of documents considered) in the Web of Science during 1994–2004. Productivity ranged from 1 to 162 documents, while the number of citations ranged from 0 to 2201 and the number of citations per document from 0 to 40.96. The h-index ranged between 1 and 29 (see Table 1).

Conclusions

Both quantity and impact of publications are taken into account when calculating the h-index, but the number of publications plays a very important role, since it is the maximum h-index an author can obtain. The h-index tends to underestimate the achievement of scientists with a “selective publication strategy”, that is, those who do not publish a high number of documents but who achieve a very important international impact. A good correlation is found between the h-index and other

Acknowledgements

This research was supported by the project SEJ2004-08052-C02-02 financed by the Spanish Plan Nacional de I+D+I, and a predoctoral I3P grant conceded by the Spanish Research Council (CSIC).

References (36)

  • E. Jiménez Contreras et al.

    The evolution of research activity in Spain: The impact of the National Commission for the Evaluation Research Activity (CNEAI)

    Research Policy

    (2003)
  • C.D. Kelly et al.

    The h-index and career assessment by numbers

    Trends in Ecology and Evolution

    (2006)
  • P. Ball

    Index aims for fair ranking of scientists

    Nature

    (2005)
  • M.G. Banks

    An extension of the Hirsch index: Indexing scientific topics and compounds

    Scientometrics

    (2006)
  • Batista, P. D., Campiteli, M. G., Kinouchi, O. & Martinez, A. S. (2005). Universal behaviour of a research productivity...
  • P.D. Batista et al.

    It is possible to compare researchers with different scientific interests?

    Scientometrics

    (2006)
  • Bollen, J., Rodriguez, M. & van de Sompel, H. (2006). Journal Status. Arxiv:physics....
  • M. Bordons et al.

    Bibliometric analysis of publication of Spanish pharmacologists in the SCI (1984–1989). 2. Contribution to subfields other than pharmacology and pharmacy (ISI)

    Scientometrics

    (1992)
  • L. Bornmann et al.

    Does the h-index for ranking of scientists really work?

    Scientometrics

    (2005)
  • T. Braun et al.

    A Hirsch-type index for journals

    Scientometrics

    (2006)
  • L. Butler

    Modifying publication practices in response to funding formulas

    Research Evaluation

    (2003)
  • Cho, A. (2005). Your career in a number. Science Now. http://sciencenow.sciencemag.org/cgi/content/full/2005/812/1...
  • S. Cole et al.

    Scientific output and recognition: A study in the operation of the reward system in science

    American Sociological Review

    (1967)
  • R. Costas et al.

    Bibliometric indicators at the micro-level: Some results in the area of natural resources at the Spanish CSIC

    Research Evaluation

    (2005)
  • B. Cronin et al.

    Using the h-index to Rank influential Information Scientists

    Journal of the American Society for Information Science and Technology

    (2006)
  • Dume, B. (2005a). Number theory. Physicsweb. http://physicsweb.org/articles/news/9/8/9/1 Accessed...
  • B. Dume

    How high is your h-index?

    Physics World

    (2005)
  • L. Egghe

    How to improve the h-index

    The Scientist

    (2006)
  • Cited by (375)

    View all citing articles on Scopus
    View full text