Skip to main content
SpringerLink
Log in

Do research assessment systems have the potential to hinder scientists from diversifying their research pursuits?

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

A possible side effect of several research evaluation schemes adopted by governments and research organizations is the discouragement of research diversification and cross-discipline fertilization, which are crucial for major global scientific challenges. This work intends to contribute to the literature stream investigating such initiatives’ effects on scientists’ specialization/diversification strategies. To this purpose, we analyze the response of academics to the introduction of the national scientific accreditation (ASN) for professorship introduced in Italy in 2012. We will conduct a longitudinal disciplinary analysis of the publications of each Italian university professor in the sciences (over 25.000), accounting for those individual and contextual variables that might moderate the opportunistic response to the ASN incentive scheme.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. Presidential decree 222 of 14/09/2011.

  2. Ministry of Education, Universities and Research Ministerial decree 76 of 07/06/2012.

  3. Initially the regulations required achievement of thresholds in all the three indicators, but after heated debate this was modified in consideration of the conditions of different sectors.

  4. The reason for introducing the CSs, unifying two or more SDSs into a single CS, is probably to reduce the number of accreditation committees, and therefore evaluators. SDSs were unified based on cognitive proximity and number of professors falling into them.

  5. For the “non-bibliometric” CSs, the indicators were not based on citations.

  6. For applicants to positions as evaluation committee members, the indicators were the number and the impact of overall scientific production, without normalization for years of academic seniority.

  7. Such data were retrieved from the ASN pages of the MIUR website http://abilitazione.miur.it/public/candidati.php?sersel=50&, but are no longer available. Subject to regulation, the ministry withdrew most data 60 days after the committee decisions, leaving only the lists of successful candidates.

  8. For each professor this database provides information on their name and surname, gender, affiliation, SDS and academic rank, at close of each year. http://cercauniversita.cineca.it/php5/docenti/cerca.php, last access on 9 January 2024.

  9. The harmonic average of precision and recall (F-measure) of authorships, as disambiguated by the algorithm, is around 97% (2% margin of error, 98% confidence interval).

  10. The sum of the values referred to the listed SCs is above 55, due to publications hosted in multi-SC journals.

  11. We observe a five year period to avoid the random fluctuations of yearly observations.

  12. To build the variable StatusASN, we refer to the 2012 academic rank, just before the ASN. If the professor has no records in 2012 (due to no publications in that year), we refer to the academic rank in the latest observed year. Note that the dataset does not distinguish whether an assistant professor would be seeking accreditation for advancement to associate or full professor, however the latter case would be so rare that it can be ignored.

  13. 0.226% = EXP(0.00226)—1.

References

  • Abramo, G., & D’Angelo, C. A. (2015). An assessment of the first “scientific accreditation” for university appointments in Italy. Economia Politica-Journal of Analytical and Institutional Economics, 32(3), 329–357.

    Article  Google Scholar 

  • Abramo, G., & D’Angelo, C.A. (2023). The effect of bibliometric research performance assessment on the specialization vs diversification strategies of scientists. Proceedings of the 19th International Society of Scientometrics and Informetrics Conference - (ISSI 2023) 2–5 July 2023, Bloomington, Indiana-US.

  • Abramo, G., D’Angelo, C. A., & Di Costa, F. (2018a). The effects of gender, age and academic rank on research diversification. Scientometrics, 114(2), 373–387.

    Article  Google Scholar 

  • Abramo, G., D’Angelo, C. A., & Di Costa, F. (2019). Diversification versus specialization in scientific research: Which strategy pays off? Technovation, 82–83, 51–57.

    Article  Google Scholar 

  • Abramo, G., D’Angelo, C. A., & Grilli, L. (2021). The effects of citation-based research evaluation schemes on self-citation behaviour. Journal of Informetrics, 15(4), 101204.

    Article  Google Scholar 

  • Abramo, G., D’Angelo, C. A., & Rosati, F. (2015). The determinants of academic career advancement: Evidence from Italy. Science and Public Policy, 42(6), 761–774.

    Google Scholar 

  • Abramo, G., D’Angelo, C. A., & Zhang, L. (2018b). A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference list. Journal of Informetrics, 12(4), 1182–1193. https://doi.org/10.1016/j.joi.2018.09.001

    Article  Google Scholar 

  • Allmendinger, J. (2015). Quests for interdisciplinarity: A challenge for the ERA and HORIZON 2020 Policy Brief by the Research, Innovation, and Science Policy Experts (RISE). Luxembourg: Publications Office of the European Union. https://op.europa.eu/en/publication-detail/-/publication/0309a87c-7276-11e5-9317-01aa75ed71a1/language-en. Last access 9 January 2024.

  • Bateman, T. S., & Hess, A. M. (2015). Different personal propensities among scientists relate to deeper vs broader knowledge contributions. Proceedings of the National Academy of Sciences, 112(12), 3653–3658.

    Article  ADS  CAS  Google Scholar 

  • Belkhouja, M., Fattoum, S., & Yoon, H. D. (2021). Does greater diversification increase individual productivity? The moderating effect of attention allocation. Research Policy, 50(6), 104256.

    Article  Google Scholar 

  • Chakraborty, T., Tammana, V., Ganguly, N., & Mukherjee, A. (2015). Understanding and modeling diverse scientific careers of researchers. Journal of Informetrics, 9(1), 69–78.

    Article  Google Scholar 

  • Cummings, J. N., & Kiesler, S. (2014). Organization theory and the changing nature of science. Journal of Organization Design, 3(3), 1–16.

    Article  Google Scholar 

  • D’Angelo, C. A., Giuffrida, C., & Abramo, G. (2011). A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments. Journal of the American Society for Information Science and Technology, 62(2), 257–269.

    Article  Google Scholar 

  • de Rijcke, S., Wouters, P. F., Rushforth, A. D., Franssen, T. P., & Hammarfelt, B. (2016). Evaluation practices and effects of indicator use-a literature review. Research Evaluation, 25(2), 161–169.

    Article  Google Scholar 

  • Della Sala, S., & Brooks, J. (2008). Multi-authors’ self-citation: A further impact factor bias? Cortex, 44(9), 1139–1145.

    Article  Google Scholar 

  • Edwards, M. A., & Roy, S. (2017). Academic research in the 21st Century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition. Environmental Engineering Science, 34(1), 51–61.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Fang, F. C., Steen, R. G., & Casadevall, A. (2012). Misconduct accounts for the majority of retracted scientific publications. Proceeding of the National Academy of Science, 109(42), 17028–17033.

    Article  ADS  CAS  Google Scholar 

  • Fontana, M., Iori, M., Sciabolazza, V. L., & Souza, D. (2022). The interdisciplinarity dilemma: Public versus private interests. Research Policy, 51(7), 104553.

    Article  Google Scholar 

  • Franzoni, C., & Rossi-Lamastra, C. (2017). Academic tenure, risk-taking and the diversification of scientific research. Industry and Innovation, 24(7), 691–712.

    Article  Google Scholar 

  • Gerosa, M. (2001). Competition for academic promotion in Italy. Lancet, 357(9263), 1208–1208.

    Article  CAS  PubMed  Google Scholar 

  • Haustein, S., & Larivière, V. (2015). The use of bibliometrics for assessing research: Possibilities, limitations and adverse effects. In I. M. Welpe, J. Wollersheim, S. Ringelhan, & M. Osterloh (Eds.), Incentives and Performance (pp. 121–139). Springer International Publishing.

    Chapter  Google Scholar 

  • Hazelkorn, E. (2010). Pros and cons of research assessment, in World Social Science Report. Knowledge Divides 2010 (UNESCO Press, 2010), pp. 255–58.

  • Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251–261.

    Article  Google Scholar 

  • Honig, B., & Bedi, A. (2012). The fox in the hen house: A critical examination of plagiarism among members of the academy of management. Academy of Management. Learning and Education, 11(1), 101–123.

    Article  ADS  Google Scholar 

  • Jamali, H. R., Abbasi, A., & Bornmann, L. (2020). Research diversification and its relationship with publication counts and impact: A case study based on Australian professors. Journal of Information Science, 46(1), 131–144.

    Article  Google Scholar 

  • Leahey, E., Beckman, C. M., & Stanko, T. L. (2017). Prominent but less productive: The impact of interdisciplinarity on scientists’ research. Administrative Science Quarterly, 62(1), 105–139.

    Article  Google Scholar 

  • Leahey, E., Keith, B., & Crockett, J. (2010). Specialization and promotion in an academic discipline. Research in Social Stratification and Mobility, 28(2), 135–155.

    Article  Google Scholar 

  • Nagin, D. S., Rebitzer, J. B., Sanders, S., & Lowell, J. T. (2002). Monitoring, motivation, and management: The determinants of opportunistic behaviour in a field experiment. American Economic Review, 92(4), 850–873.

    Article  Google Scholar 

  • Nagle, F., & Teodoridis, F. (2020). Jack of all trades and master of knowledge: The role of diversification in new distant knowledge integration. Strategic Management Journal, 41(1), 55–85.

    Article  Google Scholar 

  • National Academies of Science, National Academy of Engineering, and Institute of Medicine. (2005). Facilitating Interdisciplinary Research. The National Academies Press.

    Google Scholar 

  • Perotti, R. (2008). L’università truccata. Einaudi, Torino ISBN: 978–8–8061–9360–7.

  • Pinheiro, H., Durning, M., & Campbell, D. (2022). Do women undertake interdisciplinary research more than men, and do self-citations bias observed differences? Quantitative Science Studies, 3(2), 363–392.

    Article  Google Scholar 

  • Porter, A., & Rafols, I. (2009). Is science becoming more interdisciplinary? Measuring and mapping six research fields over time. Scientometrics, 81(3), 719–745.

    Article  Google Scholar 

  • Rafols, I., Leydesdorff, L., O’Hare, A., Nightingale, P., & Stirling, A. (2012). How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management. Research Policy, 41(7), 1262–1282.

    Article  Google Scholar 

  • Rekers, J. V., & Hansen, T. (2015). Interdisciplinary research and geography: Overcoming barriers through proximity. Science and Public Policy, 42(2), 242–254.

    Article  Google Scholar 

  • Schuitema, G., & Sintov, N. (2017). Should we quit our jobs? Challenges, barriers and recommendations for interdisciplinary energy research. Energy Policy, 101, 246–250.

    Article  Google Scholar 

  • Seeber, M., Cattaneo, M., Meoli, M., & Malighetti, P. (2019). Self-citations as strategic response to the use of metrics for career decisions. Research Policy, 48(2), 478–491.

    Article  Google Scholar 

  • Sidiropoulos, A., Katsaros, D., & Manolopoulos, Y. (2007). Generalized Hirsch h-index for disclosing latent facts in citation networks. Scientometrics, 72(2), 253–280.

    Article  CAS  Google Scholar 

  • Stephan, P. (2012). How economics shapes science. Harvard University Press.

    Book  Google Scholar 

  • Tripodi, G., Chiaromonte, F., & Lillo, F. (2020). Knowledge and social relatedness shape research portfolio diversification. Scientific Reports, 10(1), 1–12.

    Article  Google Scholar 

  • van Rijnsoever, F. J., & Hessels, L. K. (2011). Factors associated with disciplinary and interdisciplinary research collaboration. Research Policy, 40(3), 463–472.

    Article  Google Scholar 

  • Wilsdon, J. (2015). The metric tide: Independent review of the role of metrics in research assessment and management. Sage.

  • Wuchty, S., Jones, B. F., & Uzzi, B. (2007). The increasing dominance of teams in production of knowledge. Science, 316(5827), 1036–1039.

    Article  ADS  CAS  PubMed  Google Scholar 

  • Zagaria, C. (2007). Processo all’università. Cronache dagli atenei italiani tra inefficienze e malcostume. Dedalo.

  • Zeng, A., Shen, Z., Zhou, J., Fan, Y., Di, Z., Wang, Y., Havlin, S. (2019). Increasing trend of scientists to switch between topics. Nature Communications, 10(1), 1–11.

Download references

Acknowledgements

This manuscript is an extended version of a previous work presented at the 19th International Society of Scientometrics and Informetrics Conference—(ISSI 2023) 2-5 July 2023, Bloomington, Indiana, (Abramo & D’Angelo, 2023). This research was co-funded by the Italian Ministry of Education, Universities, and Research, BANDO PRIN 2017NKWYFC “The effects of evaluation on academic research: knowledge production and methodological issues”.

Author information

Authors and Affiliations

  1. Universitas Mercatorum, Piazza Mattei 10, 00186, Rome, Italy

    Giovanni Abramo

  2. Department of Engineering and Management, University of Rome “Tor Vergata”, Via del Politecnico 1, 00133, Rome, Italy

    Ciriaco Andrea D’Angelo

  3. Universitas Mercatorum, Piazza Mattei 10, 00186, Rome, Italy

    Flavia Di Costa

Authors
  1. Giovanni Abramo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ciriaco Andrea D’Angelo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Flavia Di Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giovanni Abramo.

Ethics declarations

Competing interests

None of the authors have a conflict of interest in the subject matter or materials discussed in this manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abramo, G., D’Angelo, C.A. & Di Costa, F. Do research assessment systems have the potential to hinder scientists from diversifying their research pursuits?. Scientometrics (2024). https://doi.org/10.1007/s11192-024-04959-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11192-024-04959-8

Keywords

Search

Navigation