Abstract
An evaluation exercise was performed involving 313 papers of research staff (66 persons) of the Deutsche Rheuma-Forschungszentrum (DRFZ) published in 2004–2008. The records and citations to them were retrieved from the Web of Science (Thomson Reuters) in March 2010. The authors compared productivity and citedness of “group leaders” vs. “regular scientists”, of “male scientists” vs. “female scientists” using citation-based indexes. It was found that “group leaders” are more prolific and cited more often than “regular scientists”, the same is true considering “male” vs. “female scientists”. The greatest contrast is observed between “female leaders” and “female regular scientists”. The above mentioned differences are significant in indexes related to the number of papers, while values of indexes characterizing the quality of papers (average citation rate per paper and similar indexes) are not substantially different among the groups compared. The mean value of percentile rank index for all the 313 papers is 58.5, which is significantly higher than the global mean value of about 50. This fact is evidence of a higher citation status, on average, of the publications from the DRFZ.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bar-Ilan, J. (2008). Informetrics at the beginning of the 21st century—a review. Journal of Informetrics, 2(1), 1–52.
Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572.
Ioannidis, J. P., Patsopoulos, N. A., Kavvoura, F. K. et al. (2007). International ranking systems for universities and institutions: a critical appraisal. BMC Medicine, 5(30).
Pudovkin, A. I., & Garfield, E. (2009). Percentile rank and author superiority indexes for evaluating individual journal articles and the author’s overall citation performance. Collnet Journal of Scientometrics and Information Management, 3(2), 3–10.
Sanz-Casado, E., Iribarren-Maestro, I., Garcia-Zorita, C. et al. (2009). Are productivity, impact and visibility indicators appropriate for measuring the quality of research conducted in universities? In B. Larsen and J. Leta (Eds.), Proceedings of ISSI 2009—12th International Conference of the International Society for Scientometrics and Informetrics (Vol. 1. pp. 286–290).
Tijssen, R. J. W., van Leeuwen, T. N., & van Wijk, E. (2009). Benchmarking university-industry research cooperation worldwide: Performance measurements and indicators based on co-authorship data for the world’s largest universities. Research Evaluation, 18(1), 13–24.
Wallin, J. A. (2005). Bibliometric methods: Pitfalls and possibilities. Basic and Clinical Pharmacology and Toxicology, 97(5), 261–275.
Acknowledgments
Part of this work by one of the authors (Kretschmer, H.) was supported by the 7th framework program by the European Commission, SIS-2010-1.3.3.1. Project full title: “Academic Careers Understood through Measurement and Norms”, Project acronym: ACUMEN.
Author information
Authors and Affiliations
Corresponding author
Appendix: Calculation of the PRI and the ASI
Appendix: Calculation of the PRI and the ASI
The procedure for obtaining the PRI and ASI is described in Pudovkin and Garfield (2009). It is a two step process. It requires that we first obtain the PRI for each of the individual papers an author has published, and then calculate the ASI, which is based on PRI values for all the author’s papers. The PRI for each paper is based on the citation rank of the paper among the papers published in the same journal in the same year. In other words, the comparison is made among the related papers of the target one published within the same specialty journal or any topical group of papers one may aggregate by various methods, provided the papers are of the same age as the other papers under comparison. Thus, PRI also may be applied to papers published in multi-authored books, proceedings volumes, or other topical collections of papers.
To retrieve the necessary data an author search is conducted within WOS to find all the papers of the specified author covered by WOS. For each paper one makes a journal search for a specific year and retrieves all the papers published by that journal in the same year. Then one clicks on the “citation report” button in WOS. This option sorts the papers by citation frequency and calculates the average citation rate. To calculate the PRI one needs the citation rank of the paper and the number of papers in the year set of the journal. Both are provided by the “citation report” option.
where N is the number of papers in the year set of the journal, R is the descending citation rank of the paper (among the papers of the journal published in the year of the target paper). In case of ties (several papers having the same citation frequency), each of the tied values is assigned the average of the ranks for the tied set. Thus, if a target paper is the most cited paper in a journal in a year, its PRI = 100.
Consider the paper in the 4th line of Table 7. It contains information on a paper published in 2006 in the journal Arthritis Research and Therapy. By March of 2010 it has been cited 70 times. The overall number of papers published in this journal in 2006 is 228. This paper is the most cited among the 228 papers of this journal (published in 2006). Thus, its citation rank is 1. Entering these values into the formula (1) we obtain PRI = (228 − 1 + 1)/228*100 = 100. Another example: a paper in the journal Arthritis and Rheumatism, 2005 (the 2nd line). It has been cited (by March 2010) 133 times, being the top 14th by “raw” citation number. Entering the data into the formula we obtain PRI = (437 – 14 + 1)/437*100 = 97.02, which is rounded up to 97.
The average PRI value over all the papers of a journal (within a year set) would be about 50.5. It would depend on the number of papers in the journal: for larger journals with the number of papers >100 it will be closer to 50, while for smaller journals the average will be higher. Exact value of average PRI = (50 + 1/N*100/2), where N is the number of papers in the year set of the journal. For instance, for a journal with 50 papers the average PRI will be 51.0, for a journal with 20 papers it will be 52.5. For our data set the expected average PRI is 50.2 as the median number of papers in the journals considered is 275. This property of the average PRI allows to directly see (without any external benchmarks or thresholds) whether an author (or a group of authors) performs better than his peers on average.
ASI is the number of papers of an author, which have PRI_X at (or higher) the percentile threshold of X. Thus, the ASI99 is the number of papers, which PRI is equal or higher than 99. Similarly, ASI95 is the number of papers with PRI ≥95.
Rights and permissions
About this article
Cite this article
Pudovkin, A., Kretschmer, H., Stegmann, J. et al. Research evaluation. Part I: productivity and citedness of a German medical research institution. Scientometrics 93, 3–16 (2012). https://doi.org/10.1007/s11192-012-0659-z
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-012-0659-z