Skill ranking of researchers via hypergraph

Problem definition

We define the skill ranking problem as follow: given a complex network H = (V, E, w), where V denotes the vertices, including researchers, fields and skills in our model, E denotes different types of edges, and w denotes weight of the edges. Skills indicate the ability a researcher got when he/she took part in some real works. Skill ranking problem is to evaluate a researcher’s specific skill in a specific field by ranking methods.

It is unconvincing to consider only the pairwise relationship between the researcher and the skill in skill ranking problem. It would be more convincing to take account of three factors. These three factors can be integrated into three kinds of relationships, including relationships between researchers and skills, researchers and fields, fields and skills. In this paper, we use the hypergraph to represent a high order relationship, which is a generation of simple network. A hypergraph can be used to represent the relationships and interactions of two or more nodes. For example, in the citation network, a paper and its reference papers can compose a hyperedge. In music recommendation problem (Theodoridis, Kotropoulos & Panagakis, 2013; Bu et al., 2010), a user and the music he/she listened along with the music lists compose a hyperedge. Similar problems, such as movie recommendation, interest recommendation (Yao et al., 2016), news recommendation (Liu & Dolan, 2010; Li & Li, 2013), image retrieval (Huang et al., 2010; Yu, Tao & Wang, 2012), user rating (Suo et al., 2015), scientific ranking (Liang & Jiang, 2016), can be solved based on the hypergraph framework. Experiments show hypergraph methods are powerful and efficient to model multi-relationship systems. In a hypergraph, an edge, called hyperedge, contains arbitrary number of nodes, rather than pair of nodes in ordinary graphs (Huang et al., 2010). A hypergraph is a group of H = (X, E) where X is a set of vertices and hyperedge E is a set of non-empty subsets of X. In our study, node set X is consisted of researcher set R, field set F and skill set S. Each hyperedge includes a researcher, a field and a skill, denoting the ternary relation among them.

SRModel

Our SRModel aims at ranking individual’s different skills using hypergraph model. We consider three kinds of objects and their relations in the model. The objects include researchers, fields and skills, forming the vertex set of the heterogeneous network. In this network, there are three-ary relations between skills, researchers and fields so that normal network structure cannot provide effective representation of this system. In this paper, we use a hypergraph to model the three-ary relations.

To construct the hypergraph model, we build a weighted heterogeneous network to calculate the weights of edges first, denoted as H = (V, E, w), where V = R∪F∪S denotes the set of vertices and E = {e|(vⁱ, v^j), i ≠ j} denotes the edge set. A simple example of a part of our heterogeneous network is showed in Fig. 2A. Vertex set includes three kinds of vertices, i.e., researcher v^r, field v^f and skill v^s. Edge set E includes three kinds of edges, i.e., edges between researchers and fields, researchers and skills, fields and skills. To understand the network clearly, the heterogeneous network can be regarded as a “tripartite network”, whose vertices can be divided into three independent and unconnected node sets R, F and S. And each edge e(i, j) in network connects vertices i and j in different node sets.

There are three types of edges indicating pairwise relationship between different types of nodes. To quantify the importance of the pairwise relationship, we set each kind of edge a weight to express the importance of a node to the other, which is calculated by the ranking of node’s attribute among the others in the same set.

One of the three types of edges is $e\left(v_j^s,v_i^f\right)$ between field i and skill j. The weight between a skill and a field is calculated by the percentile ranking of the skill in the field: (1)${\displaystyle W_1\left(v_j^s,v_i^f\right)=1-{\gamma }_{ji}^{sf}∕L_i^{sf},}$ where ${\gamma }_{ji}^{sf}$ denotes the ranking of the skill j that used in field i, which is calculated by ranking skills according to the times they are used in a field. $L_i^{sf}$ is the number of skills that used in the field i. We use the percentile ranking as the normalization to eliminate the influence of different number of skills in different fields. Besides, we subtract 1 from ${\gamma }_{ji}^{sf}$ to make the minimum weight not equal to zero. For example, suppose there are four skills in a field and their ranks are 1, 2, 3 and 4. The weights between those four skills and this field are 1.0, 0.75, 0.5 and 0.25, respectively.

Weight of edge $e\left(v_k^r,v_i^f\right)$ between researcher k and field i can be regarded as the importance of the researcher in the field, which is calculated by: (2)${\displaystyle W_2\left(v_k^r,v_i^f\right)=1-\sqrt{\left({\gamma }_{ki}^{rf}-1\right)∕L_i^{rf}},}$ where ${\gamma }_{ki}^{rf}$ denotes the ranking of researcher k in field i, which is calculated by ranking researchers according to the numbers of previous works that they have participated in a field. $L_i^{rf}$ is the total number of researchers in field i. Researchers get experience and knowledge of a field when they take part in works in this field. The more works they have done, the more they learned. We perform normalization by using the percentile ranking to eliminate the influence of different number of researchers in different fields, like in Eq. (1). There are differences between Eqs. (2) and (1) because we use the square root to re-scale the weight in Eq. (2). The re-scaling operation is to make the distribution of the weight wider because there are many researchers rank in the tail (a large number of beginners). For example, suppose a field m has five experts, and they have published 5, 3, 3, 1, 1 papers in this field respectively. Thus, the weights between those five experts and field m are 1.0, 0.553, 0.553, 0.225, 0.225, respectively.

Similarly, weight of edge $e\left(v_k^r,v_j^s\right)$ between researcher k and skill j represents how important a researcher is in a given skill, computed by: (3)${\displaystyle W_3\left(v_k^r,v_j^s\right)=1-\sqrt{\left({\gamma }_{kj}^{rs}-1\right)∕L_j^{rs}},}$ where ${\gamma }_{kj}^{rs}$ denotes the ranking of researcher k with skill j, which is calculated by ranking researchers according to the times they used this skill in their previous works. $L_j^{rs}$ denotes the number of researchers with skill j.

A hypergraph H^h = (X, E^h, w^h) is built after constructing the heterogeneous network and calculating the weights of edges. A hypergraph built on the heterogeneous network in Fig. 2A is demonstrated in Fig. 2B. In H^h, X is a set of hybrid nodes, composed by researcher, skill and field. E^h is hyperedge on X, and w^h is a weight function of hyperedge. For the relationships between researcher, field and skill, a hyperedge exists only if the researcher processes the skill in this field. We define the weight function of hyperedge inspired by the Laplace kernel function, and the distance calculation method in (Huang et al., 2010). The weight function is: (4)${\displaystyle W\left(v_k^r,v_i^f,v_j^s\right)=e^{\Theta },and}$ (5)${\displaystyle \Theta =\frac{W_1\left(v_j^s,v_i^f\right)}{2\cdot \overline{W_1}}+\frac{W_2\left(v_k^r,v_i^f\right)}{2\cdot \overline{W_2}}+\frac{W_3\left(v_k^r,v_j^s\right)}{2\cdot \overline{W_3}},}$ where $\overline{W_1},\overline{W_2}$ and $\overline{W_3}$ are the average edge weights between fields and skills, researchers and fields, researchers and skills, respectively.

Based on the above definition and introduction, our SRModel has been formulated. The value of skill ranking of a researcher is calculated by the weight of the hyperedge in the hypergraph.

Dataset construction

Previous studies on researcher evaluation usually use datasets of publications to evaluate their methods. These datasets include Digital Bibliography & Library Project (DBLP), American Physical Society (APS), Microsoft Academic Graph (MAG). However, authors’ skills are unavailable in those prevalent datasets. In the previous work where skills are required, such as team formation problem, team member replacement, collaborators recommendation, researchers usually use terms extracted from keywords, title (Farhadi et al., 2011) or conference and journal name (Li et al., 2017) instead of authors’ real skills. Either terms in titles or journal names have limitations to reflect skills of researchers, because the division of the work is not clear. However, there are several journals providing authors’ contribution statement, such as the British Medical Journal, Nature, Lancet. Besides, journals such as the FEBS Journal and PLOS journals require authors to declare their contributions in a specified format while submitting the paper.

We create a dataset by crawling papers’ information together with their research fields and authors’ individual contributions from the PLOS ONE journal website to validate our model, as we mentioned in the Background section. PLOS ONE journal is an open-access journal, so we can extract information from their website by parsing their HTML pages of each paper with Python. The contribution information of this journal has been used to analyze the authors’ contribution pattern recently (Corrêa Jr. et al., 2017; Sauermann & Haeussler, 2017). The details of our dataset are shown in Table 1. At first, we remove papers with incomplete information. In the raw dataset, 28,748 kinds of contributions are found, and most of them have similar meanings. PLOS journals recently adopted the CRediT Taxonomy providing standardized and fine-grained information of authors’ contribution (Atkins, 2016), but there are no standard rules for naming paper’s contributions in the early years. Using the raw contribution directly may bring inaccuracy. There are 100 items appearing more than 40 times accounting for the vast majority (about 96%). Thus, we cluster 100 kinds of contributions that appear frequently into 10 categories manually. The categories are shown in Table 2. Here, we regard these 10 categories of contributions as skills of researchers.

The acronyms of authors’ name followed the contributions on the PLOS ONE web pages. We match the acronyms up with authors’ full name in author list of each paper. Then we get the authors’ skills in every paper.

Another vital step is name disambiguation. Multiple authors may share the same name, and there is no unique identifier for each author in PLOS ONE journal. It can cause confusion and even fault if we regard those authors with the same name as one person. Many works have employed a variety of features to distinguish authors sharing the same name, such as affiliations, coauthors, topics or research interests (Kang et al., 2009; Ferreira, Gonçalves & Laender, 2012). To distinguish authors in PLOS ONE dataset, the following features are adopted: (i) Affiliation: authors with the same name and the same affiliation are regarded as the same authors; (ii) Coauthor: if two authors M₁ and M₂ with the same name coauthored one or more papers with a third author N, it is likely that M₁ and M₂ are the same author named M. We combine these two features to disambiguate the authors’ name. There are 676,852 authors in the raw dataset, and 684,844 authors after naming disambiguation.

Network construction

To construct the SRModel framework using PLOS ONE dataset, our experiment consists of four main parts as follows.

1. Construct a heterogeneous network with three kinds of vertices and three kinds of edges.

2. Compute edge weights according to Eqs(1)–(3).

3. Build a hypergraph H^h = (X, E^h, w^h) and compute hyperedge weights using Eq. (4).

4. Calculate the researchers’ yearly skill ranking Wⁱ from 2006 to 2016 respectively.

In the first step, we construct a heterogeneous network including three kinds of vertices and three kinds of relationships. Edge between researcher and field/skill exists only if he/she published paper in this field or used this skill at least once. A vertex denoting field $v_i^f$ and a vertex denoting skill $v_j^s$ are connected if one or more papers in field i use skill j. There are 684,844 authors, 11 fields and 10 skills in the heterogeneous network, forming 2,758,522 edges in total.

 
_______________________ 
 Algorithm 1: Algorithm to calculate the skill ranking of researchers      ____ 
    Input: Weight of edges; paper information 
    Output: Hyperedge weights 
  1  ____W1 = AVG(skills,fields); 
  2  ____W2 = AVG(researchers,fields); 
  3  ____W3 = AVG(researchers,skills); 
  4  hyperedges = list(); 
  5  hyperWeight = dict(); 
  6  for paper in dataset do 
     7   add (researcher, field, skill) to hyperedges; 
  8  for (r,f,s) in hyperedges do 
     9   Θ = w1[s,f] 
 2⋅__W1   + w2[r,f] 
 2⋅__W2   + w3[r,s] 
 2⋅__W3  ; 
10   weight = eΘ; 
11   hyperWeight[r,f,s] = weight; 
12  return hyperWeight;    

In the second step, we compute the weights of edges in the constructed network. We compute the weights of field-skill, researcher-field and researcher-skill by Eqs. (1)–(3), respectively. We rank the skills in field i by the numbers of papers in field i that using the skills, and for each skill j the ranking is denoted as ${\gamma }_{ji}^{sf}$. $L_i^{sf}$ is the total number skills in field i. When calculating the weights between researchers and fields, we assume the more paper a researcher published in a field, the more he/she is familiar with the field. Thus, we rank the researchers according to the numbers of papers they published in field i, and for researcher k the ranking is denoted as ${\gamma }_{ki}^{rf}$. Let the total number of researchers in field i as $L_i^{rf}$ in Eq. (2). Similarly, ${\gamma }_{kj}^{rs}$ is the ranking of researcher k using skill j by the number of papers he published using this skill, and $L_j^{rs}$ is the total number of researchers that using skill j in Eq. (3).

In the third step, we construct the hypergraph and calculate the skill proficiency of authors. A hyperedge connects a researcher, field and skill if the researcher published papers in this field using this skill. There are 9,501,866 hyperedges in the constructed hypergraph in total. And the weight of hyperedge is calculated by Eq. (4), representing a researcher’s skill proficiency in a field. The pseudo code of the algorithm to calculate the skill ranking is shown in Algorithm 1. Let E denote the number of hyperedges, so the time complexity of constructing a heterogeneous network of skill, field and researcher is O(E). Then, building hypergraph based on the network and calculating weights of hyperedges can be performed together. The time complexity of these two steps is also O(E). So the overall time complexity of our skill ranking method is O(E).

In fact, the skill ranking that we calculate in the third step is a snapshot at the time when the data are collected. In the last step, we take a snapshot every year to construct a series of hypergraphs and calculate the weights of hyperedges to represent the researchers’ skill rankings at different time, denoted as Wⁱ, where i ∈ [2006, 2016]. That is, the skill ranking of the researcher in a field at year i is calculated by all the papers he published until year i. We use the yearly skill ranking information to analyze the dynamic of researchers’ skill proficiency. Also, we can get the researchers’ abilities in the year when the paper was published.

Validation of SRModel

We computed the correlation coefficient between researchers’ skill ranking and their h-index and performed a hypothesis test to validate our model’s performance.

Correlation between skill ranking and h-index

To verify the validity of the method, and explore the potential relationship between skill ranking and h-index, we analyze how skill ranking changes with the increase of h-index. PLOS One is a journal mainly focusing on biology. Among the 182,007 papers that we obtained in their website, more than 169,150 papers are in the field of biology. Thus, we use the papers and authors in biology to perform the verification. Researchers’ skill rankings are calculated by the data in PLOS One. But only a fraction of researchers’ papers is published in this journal. It is not reasonable to use authors’ real h-indices to analyze the relationship with their skill rankings. Thus, we calculate the h-indices of all the authors in PLOS ONE dataset according to the papers they published in this journal and their citation counts. The h-indices of authors in PLOS One dataset range from 0 to 32. Pearson’s correlation coefficient quantifies the linear correlation between two groups of continuous variables (Pearson, 1895). Thus, we calculate the correlation coefficient between skill ranking and h-index, shown in Table 3. We notice that all the correlation coefficients are larger than 0.5, especially for the skill “Data analysis”, “Paper writing” and “Study and experiments design”, of which the correlation coefficients are larger than 0.8, which indicates a high correlation. This result means that a researcher with a higher h-index often has a higher skill ranking, which suggests that if a researcher is more skillful with academic skills, they can obtain outcomes with higher quality.

Table 3:

Correlation coefficient of skill ranking and h-index

Skill	Correlation coefficient
Experiment performing	0.7717
Data analysis	0.8036
Paper reviewing	0.7013
Paper writing	0.801
Supervision	0.5561
Theory analysis	0.5177
Funding acquisition	0.7582
Software and tools	0.7822
Study and experiments design	0.8155
Data collection	0.6173

DOI: 10.7717/peerjcs.182/table-3

We then analyze the distribution of skill ranking to prove the rationality of the model. The result is shown in Fig. 3. Both of two distribution curves subject to exponential distribution of function y = e^a+bx+cx2, and the R-square of these two fitting functions is close to 1, which indicates the fitting degree is high. It means that researchers’ h-index and skill ranking have the same kind of distribution, which also explains the high correlation coefficient between them and the rationality of the SRModel.

Analysis of skill ranking increase

We define the researcher’s academic age in PLOS ONE dataset as the number of years between his first paper and the last paper in this dataset. Although PLOS ONE is a comprehensive journal with a large number of papers, it cannot include all the papers published by every researcher. There still exist many researchers that published many papers but only one or two in this journal. Analyzing the skill rankings of those researchers is meaningless. In order to avoid the defect of the dataset as much as possible, we choose researchers whose academic ages and numbers of published papers are both in the top 10% respectively, named as “top researchers”, to carry out the analysis. In the PLOS ONE dataset, there are 26,978 “top researcher”.

We first calculate the yearly change rate of researchers’ skill ranking. The yearly change rate indicates that how much the skill ranking of a researcher increased or decreased compared to that in one year before, defined as ${\Delta }_i=\frac{W_i-W_{i-1}}{W_{i-1}}$, where W_i is the value of skill ranking in year i and W_i−1 ≠ 0. We show the distribution of yearly change rate between 0 to 100% in Fig. 4. The change rate of most researchers’ skills is less than 20%. The green bar demonstrates a slight decrease existing in the researchers’ skill ranking, because the progress of other researchers decreases the rankings of them. Observing the curve of researchers’ skills, we found the skills of most researchers are fluctuating upward, but the rising patterns are different. Then, we explore the commonalities and differences of the changes for researchers’ different skills.

First, we analyze the rising stages of skills. When a researcher’s skill ranking increased more than 20 % in a year, we call this year a “fast increase year” for his skill. The distribution of the longest continuous rapid growth years of each set of researcher-field-skill is presented in Fig. 5. In Fig. 5, we count the total number of researcher-field-skill sets for different length of time. About 434 thousand researcher-field-skill sets experience one-year rapid growth, and the number of that for two years reduced to around 220 thousand. A few skill sets rise continuously more than five years, including 1,254 sets for 5 years and 53 sets for 6 years. The average fast-growing years among the 730,789 researcher-field-skill sets for 26,798 researchers is 1.5 years. That is, it is generally for researchers to spend one or two years putting their energy on one skill to have a good command of it, and then they may focus on other disciplines or other skills. Several reasons account for this phenomenon. First, researchers usually change their research fields in their research careers, especially for those interdisciplinary researchers. The transformation of fields can bring researchers more inspiration and creativity. Second, for a researcher, as he becomes more and more skillful, the work he undertakes in the team may change. We then explore the skill transfer of researchers.

Changing patterns of skill ranking

Then, we explore the changing patterns of different skills. To investigate how skill ranking changes over time, we propose a method to count the average ranking for each skill in different periods. There are three problems needing to be considered. (1) Researchers’ academic ages are different, and the time when they enroll in a skill or field varies. (2) Many researchers didn’t publish paper on PLOS ONE at the beginning of their academic career. Studying the skill transfer of them is unconvincing. (3) Many researchers didn’t publish papers every year in PLOS ONE and the numbers of paper published by researchers are totally different. To solve these problems, we define a researcher’s “academic stage” as the period between years he has published papers. For example, a researcher m published two, five and one papers in 2013, 2015 and 2016, respectively, so he has three academic stages, which are 2013–2014, 2015–2015, 2016–2016. The skill ranking of each stage is indicated by the skill ranking in the first year of the stage.

After we got the skill ranking of all the stages for all the top researchers, we calculate the average value in each stage for different skills, and show the result in Fig. 6. We divide the skills into four groups according to their trend. In Fig. 6A we notice that there are five skills of which the ranking keep increasing throughout all academic stage, and the increase rate is higher than other groups, including “paper writing”, “data analysis”, “study and experiments design”, “experiment performing” and “software and tools”. It suggests that these five skills are the basic academic skills so that they are important for many researchers throughout their academic careers. The second group, as shown in Fig. 6B, has two skills, “supervision” and “theory analysis”. The rankings of these two skills change slowly in the early stages and have a sharp increase at the latter academic stages, which indicates that these two skills need more experience in academia and are harder to develop. Figure 6C is the third group of skills, including “paper reviewing” and “data collection”. These two skills’ ranking rise in the early stage and finally falling slightly, especially the skill “data collection”, whose increase rate is very small after the fourth stage. We assume that these two skills are easy to get started, and when researchers have more experience they will not use these skills frequently. There is one skill in Fig. 6D, the trend of which is not so obvious. We think “funding acquisition” has little correlation with time in our dataset. Maybe the time span is not long enough to find its changing law.

Thus, we find the changing patterns of different skills vary. Some skills are easy to get started. When researchers have more experience, they will transfer their interests to other skills. Some skills are harder to develop, researchers with these skills need to develop other skills first. Thus, they develop these skills in their later academic stage. There are also some basic skills are important for many researchers throughout their academic careers.