http://orcid.org/0000-0002-4930-748X
Figures
Abstract
Assessing the success and performance of researchers is a difficult task, as their grant output is influenced by a series of factors, including seniority, gender and geographical location of their host institution. In order to assess the effects of these factors, we analysed the publication and citation outputs, using Scopus and Web of Science, and the collaboration networks of European Research Council (ERC) starting (junior) and advanced (senior) grantees. For this study, we used a cohort of 355 grantees from the Life Sciences domain of years 2007–09. While senior grantees had overall greater publication output, junior grantees had a significantly greater pre-post grant award increase in their overall number of publications and in those on which they had last authorship. The collaboration networks size and the number of sub-communities increased for all grantees, although more pronounced for juniors, as they departed from smaller and more compact pre-award co-authorship networks. Both junior and senior grantees increased the size of the community within which they were collaborating in the post-award period. Pre-post grant award performance of grantees was not related to gender, although male junior grantees had more publications than female grantees before and after the grant award. Junior grantees located in lower research-performing countries published less and had less diverse collaboration networks than their peers located in higher research-performing countries. Our study suggests that research environment has greater influence on post-grant award publications than gender especially for junior grantees. Also, collaboration networks may be a useful complement to publication and citation outputs for assessing post-grant research performance, especially for grantees who already have a high publication output and who get highly competitive grants such as those from ERC.
Citation: Pina DG, Barać L, Buljan I, Grimaldo F, Marušić A (2019) Effects of seniority, gender and geography on the bibliometric output and collaboration networks of European Research Council (ERC) grant recipients. PLoS ONE 14(2): e0212286. https://doi.org/10.1371/journal.pone.0212286
Editor: Lutz Bornmann, Max Planck Society, GERMANY
Received: October 7, 2018; Accepted: January 30, 2019; Published: February 14, 2019
Copyright: © 2019 Pina et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data related to ERC grantees is available in at the ERC website and their bibliometric data is retrievable from Scopus and/or Web of Science. The methods for querying the database have been described within the Methods section of this paper.
Funding: Francisco Grimaldo is funded by the Spanish Ministry of Economy, Industry and Competitiveness project TIN2015-66972-C5-5-R. Ana Marušić is funded by the grant from the Croatian Research Foundation No. IP-2014-09-7672. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: DGP is a civil servant at the European Commission’s Research Executive Agency. AM occasionally acts as an ethics expert evaluator for ERC grants. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Peer review remains the core paradigm in assessing different research activities despite contradictory evidence on whether it is the best way of selecting grant proposals and judging articles' suitability for publication [1]. In addition, there are a few studies about the effects of receiving a grant on a researcher's performance [2–3]. Nevertheless, research into peer review of funding agencies is key as it can bring about policy changes or process improvements, possibly resulting in better use of resources [4].
Even though peer review should be objective, many biases were reported to affect it [1,5]. Among them, gender has been identified as a part of the general disparity in productivity between male and female researchers at all stages of their professional careers [6–10]. In several research fields, female researchers obtain lower funding [11–12] and, even when receiving similar grant funding, female faculty may lag behind their male colleagues in terms of publications and citations [13]. Longitudinal studies have shown that gender bias exists over professional research careers, even after controlling for other factors, such as research field and performance differences [6]. Another important bias–geographical location of the researcher host institution–has also been described as influencing the peer review process in scientific journals [14] and in research grants, even within a single country [15].
It is not clear whether standard bibliometric indices, such as publications and citations, are adequate measures of grant success and performance [16–17]. This uncertainty stems not only from the evidence that publications and citations outputs are diverse and vary between research disciplines [18], but also because there is conflicting data on their value as measures of grant success [19–20]. While some studies found a correlation between higher grant proposals review scores and grant output measured as citations and patents [3, 21–22], others have failed to directly confirm the importance of these outputs as a validation measure of the grant peer review process [2,23]. The assessment of grant success may be even more challenging for highly competitive grants, where the choice of the best among the best is very difficult. A possible alternative for assessing grant success could be the analysis of co-authorship networks, because it puts the individual research performance into a wider social context. Co-authorship networks establish collaboration patterns among scientists by using data drawn from their publication record [24–25].
The aim of our study was to assess seniority, gender and country differences in publication performance for a sample of grantees from the European Research Council (ERC), the EU flagship research funding agency. ERC's peer review process has been well described [26–28], with a recent study showing that it attracts high profile researchers [29]. However, that study did not establish a strong evidence for a major quantitative or qualitative impact on the publication output of grantees. Therefore, we analysed, for a cohort of junior and senior life sciences' ERC grantees from the years 2007–2009, both the publications/citations outputs and collaboration networks in the 5-year period before and after the award of their grants. We were particularly interested in the change of publication performance in relation to the grant award and whether the type of award (seniority), gender, and geographical location (higher vs. lower research performing countries) are associated with these differences.
Methods
For the purpose of this study, we used the data publicly available at the ERC website (https://erc.europa.eu) for a cohort of 355 grantees from the years 2007–2009, in order to select those who would have completed their grants by the end of 2015. We focused our study only on the life sciences grantees, because research output in this field is mostly observed through the publications in peer-reviewed journals, which are well covered by the Scopus and Web of Science Core Collection (WoS) databases [18]. This cohort was composed of Starting Grants (StG), designed to support junior researchers at the stage at which they are starting or consolidating their own independent research team, and Advanced Grants (AdG), reserved for leading senior investigators, having a track-record of significant research achievements in the last 10 years. Both grant types have a 5-year average duration (in our sample, 61% of grants lasted for 5 years, and 98% lasted for 4 to 6 years), and are awarded using a review process sharing common evaluation standards. For each grantee, a publication search was performed for the 5-year period before and after the year of the award of the grant (Fig 1). Individual grantee names from the list of the awarded ERC grants in the Life Sciences domain were used for “Author search” in the Scopus and WoS databases to identify publications for each individual (articles and reviews in English only). The gender of the grantees was judged by two authors (DGP and LB); in cases of ambiguity, a web search was performed to find possible identifying information, and a third author was consulted (AM) for final agreement. Geographical grouping was based on the host organisation location (not the grantee's nationality), and the split between the higher research-performing and lower research-performing countries was made according to the composite indicator for research excellence defined by the EU [30]. Countries considered as high-research performers were those with an indicator above the global EU value (Austria, Belgium, Denmark, Finland, France, Germany, Israel, Netherlands, Norway, Sweden, Switzerland, UK), whereas the ones considered low-research performers were those below that value (Bulgaria, Croatia, Cyprus, Czech Republic, Estonia, Greece, Hungary, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Iceland, Turkey).
Y indicates the year of the grant competition, and Y+1 the year of the grant award. Pre-award publications were extracted for 5 years, including the year of the grant competition. The post-award period was considered as the 5 years after Y+1, where the publications related to the grant could be expected. The same periods were used to collect the citations of the extracted publications.
The data were collected for StG and AdG recipients separately, and gender and country differences analysed separately for each group. Grouping of the two types of grants for the output analysis was not attempted because of large differences in publication and citations outputs.
For the analysis of the scientific collaborations established by individual grantees in the periods before and after the award of the grant, we constructed collaboration, i.e. co-authorship, networks [24–25, 31], derived from the publications retrieved in this study from Scopus for each period. We used the set of documents extracted from Scopus only since the results of searching this database are more complete and they allow us to build more robust networks. We had no access to the content of the proposals and could not establish whether the co-authors identified were also named as collaborators in the grant proposals. Nodes in these networks represent different researchers and edges connect two of them if they have co-authored a paper. The basic network indicators were:
1. Number of different co-authors. This indicator corresponds to the number of nodes in the network and, therefore, to the number of different co-authors found in the total number of publications for the period under consideration. It then measures the size of the research community the grantee is collaborating with before and after the grant.
2. Number of co-authorships. This indicator corresponds to the number of edges in the network or, similarly, to the sum of all different two-by-two relationships between researchers that can be generated from the list of co-authors of each paper published in the period under examination. This metric represents the global amount of collaboration generated by the papers published by the grantee.
We also calculated the following indicators:
3. Network density. This is the ratio between the number of edges in the network and the total number of edges if the network was completely connected. It measures how compact the co-authorship network is. The less compact it is, the more diverse the publication pattern will be.
4. Number of sub-communities. This is the number of densely connected clusters in the co-authorship network. We quantified how structured the community is by calculating the leading non-negative eigenvector of the community matrix [25].
5. Network modularity. This indicator measures how good the previous division into clusters is, or how separated are the different members of the sub-communities from each other. Modularity is calculated as the fraction of within-community edges minus the expected fraction if edges were distributed at random. The value of the modularity lies in the range −1 to 2.1 and, in practice, a value above 0.3 is a good indicator of significant community structure in a network [32].
6. Grantee eigencentrality. This is a measure of the influence of the grantee in the collaboration network. The normalized eigenvector centrality defines a ranking over the set of researchers in the network by assigning relative scores to all nodes based on the concept that connections to high-scoring nodes contribute more to the score of the node in question than equal connections to low-scoring nodes [33]. This means that a researcher is important if he or she is linked to other important researchers. In this paper we observe the evolution of the centrality that is assigned to grantees before and after the grant.
7. Network centralization. Centralization is a method for creating a network level centralization measure from the centrality scores of the researchers. It measures how much variation there is in the eigencentrality scores among the nodes [33]. Thus, in a similar way to the Gini index [34], it quantifies how equal the researchers are in the collaboration network defined by the papers co-authored with the grantee.
R programming language version 3.4.4 with iGraph library version 1.1.2 was used to perform the co-authorships networks analysis.
Statistical analysis was performed using MedCalc version 17.1 (MedCalc Software, Ostend, Belgium) and R programming language. The results were expressed as medians with 95% confidence intervals. Comparisons of indicators for individual groups at pre- or post-award time periods were based on the interpretation of 95% confidence intervals, where non-overlapping confidence intervals indicated that the measured values belonged to different populations and were therefore statistically significant. Comparison of differences between the pre- and post-award indicators for different groups was performed using Mann-Whitney U test. In a complementary analysis, we also used Bayesian t test for independent samples. Bayes factors were calculated using JASP 0.8.3.1 (JASP Team, 2017) assuming a default prior distribution [35]. Bayes Factors (BF10) with values which remained above 3 after sequential analysis and robustness check were considered to indicate substantial evidence for the alternative hypothesis [36]. In cases of discrepancies between frequentist and Bayesian statistics, we used the Bayesian approach to interpret the significance of the results, due to the inequalities in sample size and in order to quantify the size of the evidence for the tested hypothesis [35]. Moreover, due to the use of uninformative prior, we presumed that, if there was a real difference between the groups, the results of the frequentist and Bayesian analysis would not differ.
Results
Overall, most of the grantees were male (n = 291, 82%), both for StG (n = 144, 78%) and AdG groups (n = 147, 86%) (P = 0.931). Also, most of the grants were awarded to researchers from higher research-performing countries (n = 299, 84%). The median funding was 1.2 (95% CI 1.2–1.4) and 2.2 (95% CI 2.1–2.3) million € for StG and AdG, respectively. There was no difference between junior and senior grantees in the mean publication cost from the grant (total grant funding divided by the number of publications after grant award): €63,000 (95% CI €52,800-€70,300) vs €56,900 (95% CI €50,000-€62,500) (P = 0.080).
Effect of seniority
Most of the publications, for both types of grantees, fell in the first quartile of Scopus-indexed journals (median 93%, 95% CI 92–94% for StG; median 92%, 95% CI 81–94% for AdG). In absolute terms, senior grantees had more publications than junior grantees, both in the pre- and post-award periods (Table 1). However, in relative terms, junior grantees had a significantly greater increase in the number of publications from the pre- to the post-award period (Table 1). Before the grant award, senior grantees published more manuscripts as last (senior) author than junior grantees, but this difference disappeared after the grant award, as junior grantees had a significant increase in the proportion of publications with last authorships (Table 1). For all grantees, the increase in the number of publications in the post-award period was not accompanied by major changes in the median number of citations per publication (Table 1).
Table 2 summarises co-authorship networks established by the grantees with regard to several network indicators. Junior and senior grantees differed in network indices pre- and post-grant award, with senior grantees having bigger, denser and more modular collaboration communities. In relative terms, both junior and senior grantees increased the size of the community (indicator “No. of different co-authors”) within which they were collaborating in the post-award period. There was no statistically significant difference in this indicator between the two groups, despite the fact that junior grantees had a greater increase in the number of publications (Table 1). This could be explained by the fact that junior grantees worked in more compact groups both in the pre- and post-award periods, as indicated by their higher network densities in comparison to senior grantees (Table 2). The amount of collaboration generated by publications was similar for both groups, as visible in the median number of co-authorships. There was a decrease in the network densities in the post-award period, significantly more pronounced for junior grantees. Post-award collaboration networks increased significantly more for junior grantees, as measured by the increase in network modularity, without a difference in the number of emerging communities between junior and senior grantees. All modularity values were beyond 0.3, the reference indicator of significant community structures in a network [32]. Senior grantees had higher modularity values (over 0.5) but junior grantees showed a greater increase.
Fig 2 shows examples of pre- and post-award co-authorship networks for a junior and senior grantee to illustrate how the size of the collaboration network (number of nodes) and the number of sub-communities (coloured clusters) increased in the post-award period. In this example, the networks grew from 3 to 4 and from 3 to 6 communities for the junior and senior grantee, respectively. Also, the post-award co-authorship network represented a more structured collaboration pattern, where a variety of co-authors of different importance (node sizes) connect a more heterogeneous community (indicators “No. of communities” and “Network modularity” in Table 2). This effect was more pronounced for junior grantees, as they started from smaller and more compact pre-award co-authorship networks. The relative importance of the grantees within their community was reduced in the post-award period, mainly for senior grantees, who reduced their centrality in favour of other colleagues (Table 2). The post-award co-authorship networks shown in Fig 2 illustrate this situation, where higher centrality scores correspond to larger radius of nodes (emergence of researchers with intermediate importance). These nodes (researchers) became the link between the senior grantee and the others within the community, as reflected by the increase of post-award network modularity metrics. Due to the increase in the network heterogeneity (i.e., higher network modularity and lower grantee centrality), the overall network centralisation was augmented in a statistically meaningful way for junior grantees but not for seniors, who were already members of heterogeneous and robust collaboration networks in the pre-award period.
Each researcher (co-author) is depicted as a node whose size refers to the eigencentrality score and, thus, to his or her relative importance within the network. The colour of the node is assigned automatically for each individual network and cannot be used for comparisons. Edge widths represent the number of publications co-authored by the two linked researchers. Edge colours refer to the inter-cluster or intra-cluster connectivity: black edges correspond to links within the same community whereas red edges connect co-authors that have been assigned to different communities. The presentations are based on the publication data from Scopus bibliographical and citation database.
Effect of gender
Both in pre- and post-award periods, male junior grantees had more publications, but no difference in citations per publication or percent manuscripts with last authorship (Table 3). This difference was not observed for senior male and female grantees either before or after the grant award. In terms of pre- and post-award differences, we could not identify gender differences except for a greater increase in the number of publications as last authors for female senior grantees. The differences between junior and senior grantees of the same gender followed the same pattern observed for the whole StG or AdG group shown in Table 1.
Gender differences were not observed in pre- or post-award indicators of scientific networking, except for greater number of different co-authors after grant award for male junior grantees (median of 80 vs 40 for female junior grantees) (Table 4). This gender difference was not observed for senior grantees. In terms of pre-post award difference, there were no significant gender differences in the indicators of scientific networking. Changes in network density and modularity, as well as in grantee centrality and network centralisation, were similar between female and male grantees and followed the general pattern observed for the whole StG or AdG group shown in Table 2.
Effect of geography
Junior grantees from higher and lower research performing countries did not differ in publication indicators either before or after the grant award (Table 5). For senior grantees, those from higher research performing countries had more publications with last authorships than senior grantees from lower research performing countries. The changes between the pre- and post-award periods in terms of publications were similar for the senior grantees, irrespective of the country group. On the other hand, junior grantees from higher research performing countries had a greater increase in their number of publications compared with those from lower research performing countries.
The location of the researcher’s host organisation had an influence on the evolution of co-authorship networks only for junior grantees (Table 6). Junior grantees from higher research-performing countries increased the number of different co-authors after the grant award in comparison to their colleagues from lower research-performing countries; this difference was significant on frequentist but not Bayes statistical analysis. Furthermore, while all junior grantees expanded their collaboration networks (more co-authors and co-authorships), those located in higher research-performing countries became a part of more diverse (lower density) and robust (higher centralisation) networks than their counterparts from lower research-performing countries. This is illustrated with the examples of two junior grantees in Fig 3. Senior grantees from higher and lower research performing countries did not differ in pre- and post-award network indices and experienced similar pre-post award changes in these indices (Table 6).
Collaboration (co-authorship) networks for two representative junior (StG) ERC grantees (placed in the centre of the graphs) from higher (top) and lower (bottom) performing countries. Each researcher (co-author) is depicted as a node whose size refers to the eigencentrality score and, thus, to his or her relative importance within the network. The colour of the node is assigned automatically for each individual network and cannot be used for comparisons. Edge widths represent the number of publications co-authored by the two linked researchers. Edge colours refer to the inter-cluster or intra-cluster connectivity: black edges correspond to links within the same community whereas red edges connect co-authors that have been assigned to different communities. The presentations are based on the publication data from Scopus bibliographical and citation database.
Discussion
Our study demonstrated that seniority, gender and place of work are associated with the publication output and collaboration networks of ERC grantees, particularly for junior ones. Before the grant award, male junior grantees had more publications than female junior grantees and junior grantees from higher research performing countries collaborated with more other researchers that those from lower-research performing countries. In relation to the change in their performance from before the grant award to five years after the grant award, there were no major gender differences among both junior ERC grantees, although those working in lower research-performing countries did not publish and develop their collaboration networks to the same extent as their peers in higher research-performing countries. Gender and country differences were not observed for senior grantees to the same extent as for junior grantees. Senior grantees had significantly greater publication output than the junior grantees, both before and after the grant award. However, junior grantees had a greater increase in publications and last (senior) authorships. The post-award collaboration networks size and the number of sub-communities increased for both junior and senior grantees, but this change was greater for junior grantees, as they departed from smaller and more compact pre-award co-authorship networks.
The results of our study should be interpreted with potential limitations in mind. As we did not have access to the (non-public) data on unsuccessful grantees, we were not able to compare the grantees with a control group of researchers with similar profiles but lacking ERC funding. Similarly, it would also be interesting to check if different types of funding sources affect similarly the bibliometric outputs and collaboration networks or if it favours any cumulative advantage [37]. The groups compared by gender and geographical location were not comparable in size because there were much fewer women and researchers from lower research-performing countries as recipients of ERC grants. Because of the groups’ inequality, we used Bayes statistics due to its advantage of coherency and independency of the intention with which data are collected, as well as a minimum bias towards the null hypothesis [35]. We also did not explore the possible influence of the research field on the publication performance and collaboration networks. The subgroup analysis of the 9 panels from the ERC life sciences domain was not meaningful because of small number of grantees in some panels. However, recent studies of ERC panel review decision-making processes indicate that there are gender differences, which may be related to the social dynamics in different panels [38, 39].
An important finding of our study was the value of collaboration network analysis in assessing grant publication performance change, particularly in comparison to standard bibliometric indicators of publications and citations. This seems to be particularly true for researchers who were already highly productive before the grant award, such as those competing for advanced ERC grants.
The differences we observed between junior and senior grantees and the differences between the pre- and post-award periods for both grantee groups corroborate the value of combining bibliometric and collaboration networks analysis to assess scientific collaborative trends. Junior researchers benefited greatly from their ERC grants, confirmed by both the increase of number of publications, especially last author publications, and the expansion of their collaboration networks. They also positioned themselves within their respective scientific communities. For senior researchers, we observed that the centrality of their collaborative networks decreased after the grant. Such decentralisation can be understood as a positive effect for a growing scientific community because it results in more robust collaboration networks. This effect is also supported by the observation of an overall decrease in the proportion of publications as last author for senior grantees. A decentralised community can stay connected despite the disappearance of some of the nodes, which increases the robustness of the networks. This is similar to the way the Internet remains stable because there is no central server, but rather a decentralised structure, with many nodes on multiple receiving and forwarding networks [40].
While we observed gender differences both before and after the grant award in the number of publications, with male grantees publishing more papers than female grantees, the pre-post change in the publication output was similar for the two genders, both for junior and senior grantees. Senior female researchers, in contrast to their male counterparts, even increased their proportion of publications as last authors. The lack of major differences between male and female grantees in the post-award change in their publishing productivity and collaborative patterns suggests that researchers awarded an ERC grant, irrespective of their gender, have comparable output from the moment of grant award. Still, ERC granting results in significant underrepresentation of women (only 18% of our sample), but this may be related more to lower propensity in submitting grants proposals than to their research performance [41, 42], and biased peer review process [38, 39].
The place of work of the researchers also seemed to influence grant performance, both in terms of publications and positioning in the global scientific community. Junior grantees in lower research-performing countries performed worse than their peers in higher research-performing countries, suggesting that the environment in lower research-performing countries may not be beneficial for their career development, possibly due to the lack of additional support or leverage effects disposed at the regional or national level. On the other hand, senior grantees from lower research performing countries already had a higher publication track record before the award of the grant. It is important to keep in mind that the number of grantees from lower research performing countries was more than 3 times smaller than those from high research performing countries, reflecting also the difference in the applications for ERC grants: researchers from higher research performing countries represent the vast majority of applicants for ERC grants [43].
Our study is a preliminary study of publicly available data on ERC grants. It would be important to test the usefulness of collaboration networks analysis on full proposal datasets that include not only the grantees but also the (non-successful) applicants, as done in other granting systems. Funding agencies should follow the example of journals [44] and open their grant peer review data to meta-research in order to learn more about their processes, to develop better measures to validate peer review, and to evaluate grant success. Network analysis may add valuable information to standard publication and citation outputs, by providing insights not only into the structure of knowledge but also into the structure of the research community [24], which is linked to research performance [10, 45].
Acknowledgments
Disclaimer: All views expressed in this article are strictly those of the authors and may in no circumstances be regarded as an official position of the Research Executive Agency or the European Commission.
The authors would like to thank Brito Ferreira for valuable support in extracting and sorting the data.
Note: A part of this study was presented at the Eighth International Congress on Peer Review and Scientific Publication, Chicago, Illinois, USA; September 10–12, 2017, and PEERE2018 International Conference on Peer review, Rome, March 7–9, 2018.
References
- 1. Marsh HW, Jayasinghe UW, Bond NW, Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. Am Psychol. 2008;63:160–168. pmid:18377106
- 2. Gallo SA, Carpenter AS, Irwin D, McPartland CD, Travis J, Reynders S, et al. The validation of peer review through research impact measures and the implications for funding strategies. PLoS ONE. 2014;9:e106474. pmid:25184367
- 3. Li D, Agha L, Big names or big ideas: do peer-review panels select the best science proposals? Science. 2015;348:434–438. pmid:25908820
- 4. Pina DG, Hren D, Marušić A. Peer review evaluation process of Marie Curie Actions under EU’s Seventh Framework Programme for Research. PLoS ONE. 2015;10:e0130753. pmid:26126111
- 5. Park IU, Peacey MW, Munafò MR. Modelling the effects of subjective and objective decision making in scientific peer review. Nature. 2014;506:93–96. pmid:24305052
- 6. van den Besselaar P, Sandström U. Gender differences in research performance and its impact on careers: a longitudinal case study. Sceintometrics. 2016;106:143–162.
- 7. van den Besselaar P, Sandström U. Vicious circles of gender bias, lower positions, and lower performance: Gender differences in scholarly productivity and impact. PLoS ONE. 2017;12:e0183301. pmid:28841666
- 8. Tricco AC, Thomas SM, Antony J, Rios P, Robson R, Pattani R, et al. Strategies to prevent or reduce gender bias in peer review of research grants: A rapid scoping review. PLoS ONE. 2017;12:e0169718. pmid:28061509
- 9. Symonds MRE, Gemmell NJ, Braisher TL, Gorringe KL, Elgar MA. Gender differences in publication output: towards an unbiased metric of research performance. PLoS ONE. 2006;1:e127. pmid:17205131
- 10. Kyvik S, Reymert I. Research collaborations in groups and networks:differences across academic fields. Scientometrics. 2017;113:951–967. pmid:29081555
- 11. Svider PF, D'Aguillo CM, White PE, Pashkova AA, Bhagat N, Langer PD, et al. Gender differences in successful National Institutes of Health funding in ophthalmology. J Surg Educ. 2014;71:680–688. pmid:24776863
- 12. Eloy JA, Svider PF, Kovalerchik O, Baredes S, Kalyoussef E, Chandrasekhar SS. Gender differences in successful NIH grant funding in otolaryngology. Otolaryngol Head Neck Surg. 2013;149:77–83. pmid:23585148
- 13. Raj A, Carr PL, Kaplan SE, Terrin N, Breeze JL, Freund KM. Longitudinal analysis of gender differences in academic productivity among medical faculty across 24 medical schools in the United States. Acad Med. 2016;91:1074–1079. pmid:27276002
- 14. Thaler K, Kien C, Nussbaumer B, Van Noord MG, Griebler U, Klerings I, Gartlehner G, UNCOVER Project Consortium. Inadequate use and regulation of interventions against publication bias decreases their effectiveness: a systematic review. J Clin Epidemiol. 2015;68:792–802. pmid:25835490
- 15. Wals WP. Biases in grant proposal success rates, funding rates and award sizes affect the geographical distribution of funding for biomedical research. PeerJ. 2016;4:e1917. pmid:27077009
- 16. Bornmann L, Wallon G, Ledin A. Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European Molecular Biology Organization Programmes. PLoS ONE. 2008;3:e3480. pmid:18941530
- 17. Bornmann L. Measuring impact in research evaluations: a thorough discussion of methods for, effects of and problems with impact measurements. Higher Ed. 2017;73:775–787.
- 18. Mongeon P, Paul-Hus A. The journal coverage of Web of Science and Scopus: a comparative analysis. Scientometrics. 2016;106:213–228.
- 19. Catalini C, Lacetera N, Oettl A. The incidence and role of negative citations in science. Proc Natl Acad Sci USA. 2015;112:13823–13826. pmid:26504239
- 20. Fortin JM, Currie DJ. Big Science vs. Little Science: How scientific impact scales with funding. PLoS ONE. 2013;8:e65263. pmid:23840323
- 21. Sandström U. Research quality and diversity of funding: A model for relating research money to output of research. Scientometrics. 2009;79:341–349.
- 22. Jacob BA, Lefgren L. The impact of research grant funding on scientific productivity. J Pub Econ. 2011;95:1168–1177.
- 23. Fang FC, Bowen A, Casadevall A. NIH peer review percentile scores are poorly predictive of grant productivity. ELife. 2016;5:e13323. pmid:26880623
- 24. Newman MEJ, Co-authorship networks and patterns of scientific collaboration. Proc Natl Acad Sci USA. 2004;101:5200–5205. pmid:14745042
- 25. Newman MEJ.Finding community structure in networks using the eigenvectors of matrices. Phys Rev. 2006;E74:036104.
- 26. Luukkonen T. The European Research Council and the European research funding landscape. Sci Pub Policy. 2014;41:29–43.
- 27. Neufeld J, Huber N, Wegner A. Peer review-based selection decisions in individual research funding, applicants’ publication strategies and performance: The case of the ERC Starting Grants. Res Eval. 2013;22:237–247.
- 28. Scherngell T, Roche I, Hörlesberger M, Besagni D, Züger ME, Holste D. Initial comparative analysis of model and peer review process for ERC starting grant proposals. Res Eval. 2013;22:248–257.
- 29. Robitaille JP, Macaluso B, Pollitt A, Gunashekar S, Larivière V. Comparative scientometric assessment of the results of ERC-funded projects. Bibliometric Assessment Report (D5). 2015. Available from: https://erc.europa.eu/sites/default/files/document/file/ERC_Bibliometrics_report.pdf. Cited: 20 December 2018.
- 30.
European Commission, 2013. Research and Innovation performance in EU Member States and Associated countries: Innovation Union progress at country level. Available from: http://ec.europa.eu/research/innovation-union/pdf/state-of-the-union/2012/innovation_union_progress_at_country_level_2013.pdf. Cited: 20 December 2018.
- 31.
Wasserman S, Faust K. Social network analysis: methods and applications. Cambridge: Cambridge University Press; 1994.
- 32. Clauset A, Newman MEJ, Moore C. Finding community structure in very large networks. Phys Rev. 2004;E70:066111.
- 33. Freeman LC. Centrality in social networks: conceptual clarification. Social Networks. 1979;1:215–239.
- 34. Silber J. Factor components, population subgroups and the computation of the Gini index of inequality. Rev Econ Stat. 1989;71:107–115.
- 35. Wagenmakers EJ, Marsman M, Jamil T, Ly A, Verhagen J, Love J, et al. Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications. Psychol Bull Rev. 2018;25:35–57.
- 36.
Jeffreys H. Theory of probability. 3rd ed. Oxford University Press, UK; 1961.
- 37. Viner N, Powell P, Green R. Institutionalized biases in the award of research grants: a preliminary analysis revisiting the principle of accumulative advantage. Res Pol. 2004;33:443–454.
- 38. van den Besselaar P, Sandström U, Schiffbaenker H. Studying grant decision-making: a linguistic analysis of review reports. Scientometrics. 2018;117:313–329. pmid:30220747
- 39.
van den Besselaar P, Schiffbaenker J, Sandström U, Mom C. Explaining gender bias in ERC grant selection–a first exploration of the life sciences case. In: STI 2018 Conference Proceedings; 2018 Sep 12–14; Leiden, The Netherlands. Leiden: Centre for Science and Technology Studies (CWTS); 2018. p. 346–352.
- 40.
Spinello RA. Regulating cyberspace: The policies and technologies of control. Wesport: Quorum Books; 2002.
- 41. Hechtman LA, Moore NP, Schulkey CE, Miklos AC, Calcagno AM, Aragon R, et al. NIH funding longevity by gender. Proc Natl Acad Sci UaA. 2018;115:7943–7948.
- 42. Jappelli T, Nappi CA, Torrini R. Gender effects in research evaluation. Res Pol. 2017;46:911–924.
- 43. European Commission, 2015. ERC funding activities 2007–2013. Key facts, patterns and trends. Key facts, patterns and trends. Available from: https://erc.europa.eu/sites/default/files/publication/files/ERC_funding_activities_2007_2013.pdf. Cited: 20 December 2018.
- 44. Squazzoni F, Grimaldo F, Marušić A. Publishing: Journals could share peer-review data. Nature. 2017;546:352.
- 45. Abramo G, D’Angelo CA, Di Costa F. Research collaboration and productivity: is there a correlation? Higher Ed. 2009;57:155–171.