PrestigeRank: A new evaluation method for papers and journals

doi:10.1016/j.joi.2010.03.011

Journal of Informetrics

Volume 5, Issue 1, January 2011, Pages 1-13

https://doi.org/10.1016/j.joi.2010.03.011 Get rights and content

Abstract

This paper studies how missing data in the PageRank algorithm influences the result of papers ranking and proposes PrestigeRank algorithm on that basis. We make use of PrestigeRank to give the ranking of all papers in physics in the Chinese Scientific and Technology Papers and Citation Database (CSTPCD) published between 2004 and 2006. We compared PrestigeRank result with PageRank and citation ranking. We found PrestigeRank is significantly correlated with PageRank and citation counts. We also used paper citation networks to rank journals, and compared the result with that of journal citation networks. We proposed PR_sum, PR_ave, and compared both of them with citation counts and impact factor. It indicates PR_sum, PR_ave can reflect journal's authority favorably. We also discuss the advantages and disadvantages, application scope and application prospects of PrestigeRank in the evaluation of papers and journals.

Introduction

Citation analysis is one of the most widely used bibliometric tools for ranking papers and journals. Garfield proposed that a citation count could be used to measure the impact of publications (Garfield, 1955) and the impact factor as a tool in journal evaluation (Garfield, 1972). Because these measures are easily comprehensible and quickly obtainable, they are being used more and more widely. However, the citation counts and impact factor have intrinsic limitations (Buela-Casal, 2004, Maslov and Redner, 2008). They assume that all citations are equal, no matter whether they are from an important paper or a poor-quality paper, which is clearly unreasonable.

Many researchers refer to the Search Engine Algorithms to obtain a solution to the importance differentiation of citations. Bollen, Rodriguez, and Van de Sompel (2006) undertook a journal ranking study for journal citation data from ISI using the PageRank algorithm. Using PageRank, the SCImago Research Group defined the SCImago Journal Rank (SJR) based on the SCOPUS Database (http://www.scimagojr.com). The Journal Citation Report promulgated by ISI in 2008 used a new journal evaluation index, Eigenfactor, the calculation of which is based on the PageRank algorithm, but eliminates self-citations in journals (http://www.eigenfactor.org). Having proposed a new journal ranking algorithm based on PageRank and the HITS algorithm, Su et al., 2009a, Su et al., 2009b, Su et al., 2009c made use thereof to do an empirical study of Chinese science and technology journals. Several studies (Chen et al., 2007, Li and Zhai, 2007, Luo et al., 2007, Ma et al., 2008, Walker et al., 2007) applied PageRank algorithm to the publication citation network for measuring the importance of scientific papers. The common factor in all these studies is the comprehensive consideration of the quantity and quality of citations to calculate the scores of journals or papers. They differentiate the importance of the citations, which is undoubtedly more reasonable than only considering the citation counts. Nevertheless, there are at least two new questions worth considering. First, with regard to journal evaluation, is the algorithm based on a journal level citation graph able to cover the differences in citations of the paper level graph? If this is possible, we should conduct journal evaluation using a network of paper citations. Because the quality of a journal is decided by the quality of all the papers contained therein, if the evaluation of the papers can be addressed, the problem of journal evaluation will be solved. Second, in contrast to a web page link graph, there is a wide range of types of citations in papers in a publication citation graph, including traditional journal articles, as well as conference papers, books, standards, patents and network information. There is no single database that contains all these types of documents and the issue of missing documents therefore has an effect on the metric. None of the above studies on designing networks of paper citations considered how to deal with missing parts of the literature.

To solve the two issues highlighted above, we propose PrestigeRank, a new algorithm for a publication citation graph based on PageRank. We also aim to find a solution suitable for cases where there are missing papers in the database citing network.

Section snippets

Data

The Chinese Scientific and Technical Papers and Citations Database (CSTPCD) is a scientific publications system developed by the Institute of Scientific and Technical Information of China. CSTPCD is based on representative domestic scientific and technical journals. It contains more than 1700 Chinese scientific and technological journals published in English and Chinese (2008) with the source journals covering mathematics, information and systems science, physics, mechanics, chemistry,

The results of citation counts, PageRank and PrestigeRank rank

As can be seen from Table 5, there is a positive correlation in the PrestigeRank, PageRank and citation counts. The Spearman correlation coefficient between the PrestigeRank and citation count is greater than between PageRank and citation counts, which suggests a correlated relationship between PrestigeRank and citation counts is higher than between PageRank and citation counts.

As mentioned earlier, the PageRank value of one paper is in inverse proportion to the references counts of other

Discussion and conclusion

In this work we discuss how papers that are cited by papers in the collection but are not themselves included in the collection in the PageRank algorithm influence the ranking of paper and propose PrestigeRank, which use a “virtual node” to represent those references not included in the collection and receives all citations that come from papers in the collection. We make use of PrestigeRank to give the ranking of all physics-related papers from 2004 to 2006 in CSTPCD. Furthermore, we compared

Acknowledgments

This work is supported by the Ministry of Science and Technology in China under contract 2006BAH03B05, National Natural Science Foundation of China (70973118) and the Foundation of the Institute of Scientific and Technical Information of China (YY-200902). The authors thank three anonymous reviews for valuable comments and suggestions which helped shape and improve this paper. We also thank Xiong Ping for her assistance in polishing this article.

References (21)

S. Brin et al.
The anatomy of a large-scale hypertextual web search engine
Computer Networks and ISDN Systems
(1998)
P. Chen et al.
Finding scientific gems with Google's PageRank algorithm
Journal of Informetrics
(2007)
N. Ma et al.
Bringing PageRank to the citation analysis
Information Processing and Management
(2008)
J. Bollen et al.
Journal status
Scientometrics
(2006)
G. Buela-Casal
Assessing the quality of articles and scientific journals: Proposal for weighted impact factor
Psychology in Spain
(2004)
E. Garfield
Citation indexes for science: A new dimension in documentation through association of ideas
Science
(1955)
E. Garfield
Citation analysis as a tool in journal evaluation
Essays of An Information Scientist
(1972)
J.M. Kleinberg
Authoritative sources in a hyperlinked environment
A. Langville et al.
Google's PageRank and Beyond: The science of search engine rankings
(2006)
C. Li et al.
Exploration of PageRank-based citation analysis method
Information Studies: Theory & Application
(2007)

There are more references available in the full text version of this article.

Cited by (39)

Citation counts prediction of statistical publications based on multi-layer academic networks via neural network model[Formula presented]
2024, Expert Systems with Applications
Citation counts is a crucial factor in evaluating the quality of research papers. Therefore, it is vital to accurately predict citation counts and explore the mechanisms underlying citations. In this study, we focus on predicting the citation counts in the field of statistics. We collect 55,024 academic papers published in 43 statistics journals between 2001 and 2018. Furthermore, we collect and clean a high-quality dataset and then construct multi-layer networks from different perspectives, including journal networks, author citation networks, co-citation networks, co-authorship networks, and keyword co-occurrence networks. Additionally, we extract 77 factors for citation counts prediction, including 22 traditional and 55 network-related factors. To address the issues of zero-inflated and over-dispersed citation counts, a neural network model is designed to achieve high prediction accuracy. Furthermore, we adopt a leave-one-feature-out approach to investigate the importance of these factors. The proposed neural network model achieves an MAE value of 7.352, which outperforms other machine learning models in the comparison. Thus, this study provides a useful guide for researchers to predict citation counts and can be easily extended to other research fields.
A novel method to identify influential nodes in complex networks based on gravity centrality
2022, Information Sciences
Citation Excerpt :
The complex network is an intuitive and efficient way to describe the connection between people or objects. An increasing number of researchers in the past decades applied this model to solve problems in various fields, such as finding social network spreaders [1–2], analyzing drugs and proteins [3–4], ranking professional sports players [5], spreading mechanisms of financial risks [6–7] and predicting outstanding scientists or journals [8–10]. Among them, finding influential nodes in complex networks is an enduring research hotspot, as it is a common problem in different fields.
Identifying influential nodes in complex networks is a significant issue in analyzing the spreading dynamics in networks. Many existing methods focus only on local or global information of nodes but neglect the interaction between nodes. Gravity centrality is a recently raised centrality method that combines a node’s local and global information to properly describe the interaction between nodes. However, node degree is the local information metric in gravity centrality, which omits the connection situation of its neighboring nodes. To overcome the drawbacks of node degree parameters used in gravity centrality, we introduced Laplacian centrality to optimize the initial gravity centrality and put up Laplacian gravity centrality. Regarding real networks from different fields, our method has the highest Kendall’s correlation coefficient to the result of node infection ability simulated by the susceptible-infected-recovered model than other methods in 7 out of 10 real networks. Furthermore, the proposed method's time complexity can be as low as linear time complexity in sparse networks. Results show that Laplacian gravity centrality is an effective method to identify influential nodes, especially in networks with smaller average node degrees and longer average path lengths.
Statistics in everyone's backyard: An impact study via citation network analysis
2022, Patterns
Citation Excerpt :
The method is fully automated and can be applied to any external topic of choice, providing a different application for PPR and related techniques. In contrast, existing applications of PPR31–33 and various modified versions of the algorithm34–36 to citation networks are focused on analyzing the internal network of a field and ranking the papers in terms of their internal impact. ( 3) Under a commonly used network model, we provide the first theoretical justification for the combined use of aPPR and conductance to identify a target community, demonstrating that our approach is principled and generalizable.
Statistical methodologies are indispensable in data-driven scientific discoveries. In this paper, we make the first effort to understand the impact of recent statistical innovations on other scientific fields. By collecting comprehensive bibliometric data from the Web of Science database for selected statistical journals, we investigate the citation trends and compositions of citing fields over time, and we find increasing citation diversity. Furthermore, in a new setting, we apply a local clustering technique involving personalized PageRank with graph conductance for size selection to find the most relevant statistical innovation for a given external topic in other fields. Through a number of case studies, we show that the results from our citation data analysis align well with our knowledge and intuition about these external topics. Overall, we have found that the statistical theory and methods recently invented by the statistics community have made increasing impact on other scientific fields.
A new citation concept: Triangular citation in the literature
2021, Journal of Informetrics
Citation Excerpt :
On the basis of citation and cooperation, Li, Liu, and Zhang (2016) constructed an evaluation framework for the academic influence of scientific papers based on the three dimensions of knowledge input, knowledge output, and knowledge flow. Based on the PageRank algorithm, Su et al. (2011) proposed a quality evaluation algorithm for a single paper by adding factors such as journal impact factor, paper publication time, publishing organization, author authority, and subject differences. In summary, the theories and methods of examining citations between literature have been widely used in the frontier exploration of science, the evolutionary analysis of knowledge, academic evaluation, and other fields.
This paper introduces the concept of triangular citation that combines direct citation, co-citation and bibliographic coupling, and defines three kinds of literature used in triangular citation, namely, original Literature A, intermediary Literature B and following Literature C. Taking citation data from the CNKI database as the experimental sample, it was found that there are triangular citations in the actual citation network, with a high coverage rate. Additionally, the authors analyzed the triangular citation sample data from the different perspectives of reference time, literature type, impact factor, interdisciplinary citation, and the author’s self-citation, thereby finding the external characteristics of literature and internal trends of citation in the triangular citation. Additionally, it was concluded that the literature between A and C has an “indirect citation” mechanism. The citation characteristics of the two, i.e., A and C, are different from the direct citation relationship characteristics of A and B and B and C. The triangular citation proposed in this paper provides a new research perspective for bibliometrics, and it can be used to explore hidden literature relationships, citation mechanisms and citation motivations.
Measurement and analysis of Chinese journal discriminative capacity
2020, Journal of Informetrics
This article has been retracted: please see Elsevier Policy on Article Withdrawal (http://www.elsevier.com/locate/withdrawalpolicy).
This article has been retracted upon request of the authors. After publication of this article, the authors contacted the Editor-in-Chief to alert her of errors in the data used in the study. The bibliographic data contains several journal mismatches which lead to the deviation of the data results, as well as the errors in the figures and tables in the paper. After careful consideration, the authors have requested to retract and the Editor-in-Chief has agreed.
The authors offer their apologies to readers, Editors and reviewers of this paper.
Forward search path count as an alternative indirect citation impact indicator
2019, Journal of Informetrics
Main path analysis is a popular method for extracting the backbone of scientific evolution from a (paper) citation network. The first and core step of main path analysis, called search path counting, is to weight citation arcs by the number of scientific influence paths from old to new papers. Search path counting shows high potential in scientific impact evaluation due to its semantic similarity to the meaning of scientific impact indicator, i.e. how many papers are influenced to what extent. In addition, the algorithmic idea of search path counting also resembles many known indirect citation impact indicators. Inspired by the above observations, this paper presents the FSPC (Forward Search Path Count) framework as an alternative scientific impact indicator based on indirect citations. Two critical assumptions are made to ensure the effectiveness of FSPC. First, knowledge decay is introduced to weight scientific influence paths in decreasing order of length. Second, path capping is introduced to mimic human literature search and citing behavior. By experiments on two well-studied datasets against two carefully created gold standard sets of papers, we have demonstrated that FSPC is able to achieve surprisingly good performance in not only recognizing high-impact papers but also identifying undercited papers.

View all citing articles on Scopus

View full text

PrestigeRank: A new evaluation method for papers and journals

Abstract

Introduction

Section snippets

Data

The results of citation counts, PageRank and PrestigeRank rank

Discussion and conclusion

Acknowledgments

Computer Networks and ISDN Systems

Journal of Informetrics

Information Processing and Management

Journal status

Scientometrics

Assessing the quality of articles and scientific journals: Proposal for weighted impact factor

Psychology in Spain

Citation indexes for science: A new dimension in documentation through association of ideas

Science

Citation analysis as a tool in journal evaluation

Essays of An Information Scientist

Authoritative sources in a hyperlinked environment

Google's PageRank and Beyond: The science of search engine rankings

Exploration of PageRank-based citation analysis method

Information Studies: Theory & Application