Skip to main content
SpringerLink
Log in

Citations versus expert opinions: citation analysis of featured reviews of the American Mathematical Society

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Peer review and citation metrics are two means of gauging the value of scientific research, but the lack of publicly available peer review data makes the comparison of these methods difficult. Mathematics can serve as a useful laboratory for considering these questions because as an exact science, there is a narrow range of reasons for citations. In mathematics, virtually all published articles are post-publication reviewed by mathematicians in Mathematical Reviews (MathSciNet) and so the data set was essentially the Web of Science mathematics publications from 1993 to 2004. For a decade, especially important articles were singled out in Mathematical Reviews for featured reviews. In this study, we analyze the bibliometrics of elite articles selected by peer review and by citation count. We conclude that the two notions of significance described by being a featured review article and being highly cited are distinct. This indicates that peer review and citation counts give largely independent determinations of highly distinguished articles. We also consider whether hiring patterns of subfields and mathematicians’ interest in subfields reflect subfields of featured review or highly cited articles. We re-examine data from two earlier studies in light of our methods for implications on the peer review/citation count relationship to a diversity of disciplines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Note. Data of individual students in scatterplot (left). The same data in bins of size ten and in quartiles

Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. Others are based on altmetric measures or peer review such as mentions on social media, patented based measures, grant funding, or prizes and awards.

  2. See Gilbert (1977) for references.

  3. See Gilbert (1977) for references.

  4. For example, Bornmann and Osório write, “we use citations as a measure of ‘value’, because citations are usually applied to assess the usefulness and the value of publications for other researchers (Bornmann 2017)” (Bornmann and Osório 2019, p. 546).

  5. Article publication dates were 1993 to 2004.

  6. The list of featured review articles is no longer available from the American Mathematical Society. We found featured review articles through the analysis of the review texts.

  7. Using a sample of 6,000 and assuming that WOS mathematics category articles are included in MathSciNet.

  8. Formerly, the committee also included a representative of the Institute for Mathematical Statistics.

References

  • Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: An overview of basic concepts and theories. SAGE Open, 9(1), 1–17. https://doi.org/10.1177/2158244019829575.

    Article  Google Scholar 

  • American Mathematical Society. (1995). Editorial Statement. Mathematical Reviews, 95(a), 1.

  • American Mathematical Society. (2019). About MathSciNet. Retrieved August 12, 2019 from https://mathscinet.ams.org/mathscinet/help/about.html?version=2.

  • Barabási, A. L., & Albert, R. (1999). Emergence of scaling in random networks. Science, 286(5439), 509–512.

    Article  MathSciNet  Google Scholar 

  • Bensman, S. J., Smolinsky, L., & Pudovkin, A. I. (2010). Mean citation rate per article in mathematics journals: Differences from the scientific model. Journal of the American Society for Information Science and Technology, 61(7), 1440–1463.

    Article  Google Scholar 

  • Bertocchi, G., Gambardella, A., Jappellic, T., Nappi, C. A., & Peracchi, F. (2015). Bibliometric evaluation vs. informed peer review: Evidence from Italy. Research Policy, 44, 451–546.

    Article  Google Scholar 

  • Bornmann, L. (2017). Measuring impact in research evaluations: A thorough discussion of methods for, effects of and problems with impact measurements. Higher Education, 73(5), 775–787.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008a). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008b). The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173–7178.

    Article  Google Scholar 

  • Bornmann, L., & Osório, A. (2019). The value and credits of n-authors publications. Journal of Informetrics, 13(2019), 540–554.

    Article  Google Scholar 

  • Campanario, J. M. (1998). Peer review for journals as it stands today—part 1. Science Communication, 19(3), 181–211.

    Article  Google Scholar 

  • Chedzoy, O. B. (2006). Phi-Coefficient. Wiley Online Library. John Wiley & Sons, Inc.

    Book  Google Scholar 

  • Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104.

    Article  Google Scholar 

  • Cronin, B. (2005). The hand of science: Academic writing and its rewards. Scarecrow press.

  • Eysenck, H. J., & Eysenck, S. B. G. (1992). Peer review: Advice to referees and contributors. Personality and Individual Differences, 13, 393–399.

    Article  Google Scholar 

  • Garfield, E. (1962). Can citation indexing be automated? Essays of an Information Scientist, 1, 84–90.

    Google Scholar 

  • Garfield, E. (1979). Citation indexing, its theory and application in science, technology, and humanities. John Wiley & Sons Inc.

    Google Scholar 

  • Gilbert, G. N. (1977). Referencing as persuasion. Social Studies of Science, 7, 113–122.

    Article  Google Scholar 

  • Jackson, J. L., Srinivasan, M., Rea, J., Fletcher, K. E., & Kravitz, R. L. (2011). The validity of peer review in a general medicine journal. PLoS ONE, 6(7), e22475. https://doi.org/10.1371/journal.pone.0022475.

    Article  Google Scholar 

  • Kravitz, R. L., Franks, P., Feldman, M. D., Gerrity, M., Byrne, C., & Tierney, W. M. (2010). Editorial peer reviewers’ recommendations at a general medical journal: Are they reliable and do editors care? PLoS ONE, 5(4), e10072. https://doi.org/10.1371/journal.pone.0010072.

    Article  Google Scholar 

  • Langfeldt, L. (2001). The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science, 31, 820–841.

    Article  Google Scholar 

  • Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.

    Article  Google Scholar 

  • Li, X., & Thelwall, M. (2012). F1000, Mendeley and traditional bibliometric indicators. In Archambault, É., Gingras, Y., Larivière, V. (Eds.), Proceedings of 17th International Conference on Science and Technology Indicators (pp. 541–551). OST and Science-Metrix.

  • MacRoberts, M. H., & MacRoberts, B. R. (2018). The mismeasure of science: Citation analysis. Journal of the Association for Information Science and Technology, 69, 474–482. https://doi.org/10.1002/asi.23970.

    Article  Google Scholar 

  • Mallapaty, S. (2018, January 30). Paper authorship goes hyper. Retrieved August 14, 2019, from https://www.natureindex.com/news-blog/paper-authorship-goes-hyper.

  • McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282.

    Article  MathSciNet  Google Scholar 

  • McHugh, M. L. (2018). Phi Correlation Coefficient. In B. B. Frey (Ed.), The SAGE encyclopedia of educational research, measurement, and evaluation. SAGE Publications Inc.

    Google Scholar 

  • Merton, R. K. (1988). The Matthew effect in science, II: Cumulative advantage and the symbolism of intellectual property. Isis, 79(4), 606–623.

    Article  Google Scholar 

  • Moed, H. F. (2005). Citation analysis in research evaluation. Springer.

    Google Scholar 

  • Moed, H. F. (2017). Applied evaluative informetrics. Springer.

    Book  Google Scholar 

  • National Research Council. (2009). A Guide to the Methodology of the National Research Council Assessment of Doctorate Programs. The National Academies Press.

  • National Research Council. (2011). A data-based assessment of research doctoral programs in the United States. The National Academies Press.

  • National Science Board (2010). Science and Engineering Indicators 2010. (NSB 10–01). National Science Foundation.

  • Nicholas, D., Watkinson, A., Jamali, H. R., Herman, E., Tenopir, C., Volentine, R., et al. (2015). Peer review: Still king in the digital age. Learned Publishing, 28, 15–21. https://doi.org/10.1087/20150104.

    Article  Google Scholar 

  • Patterson, M. S., & Harris, S. (2009). The relationship between reviewers’ quality-scores and number of citations for papers published in the journal physics in medicine and biology from 2003–2005. Scientometrics, 80(2), 345–351.

    Article  Google Scholar 

  • Peters, D. P., & Ceci, S. J. (1982). Peer-Review Practices of Psychological Journals: The Fate Of Accepted, Published Articles, Submitted Again. Behavioral and Brain Sciences, 5(2), 187–195.

  • Price, D. D. S. (1976). A general theory of bibliometric and other cumulative advantage processes. Journal of the American Society for Information Science, 27(5), 292–306.

    Article  Google Scholar 

  • Rothwell, P. M., & Martyn, C. N. (2000). Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone? Brain, 123(9), 1964–1969.

    Article  Google Scholar 

  • Simon, H. A. (1955). On a class of skew distribution functions. Biometrika, 42(3/4), 425–440.

    Article  MathSciNet  Google Scholar 

  • Smolinsky, L., & Lercher, A. (2012). Citation rates in mathematics: a study of variation by subdiscipline. Scientometrics, 91, 911–924. https://doi.org/10.1007/s11192-012-0647-3.

    Article  Google Scholar 

  • Smolinsky, L., & Lercher, A. (2020). Co-author weighting in bibliometric methodology and subfields of a scientific discipline. Journal of Data and Information Science, 5(3), 84–96.

    Article  Google Scholar 

  • Smolinsky, L., Lercher, A., & McDaniel, A. (2015). Testing theories of preferential attachment in random networks of citations. Journal of the Association for Information Science and Technology, 66(10), 2132–2145.

    Article  Google Scholar 

  • Storer, N. W. (1973). Introduction. In R. K. Merton (Ed.), The sociology of science: Theoretical and empirical investigations. University of Chicago press.

    Google Scholar 

  • Tomkins, A., Zhang, M., & Heavlin, W. D. (2017). Reviewer bias in single-versus double-blind peer review. Proceedings of the National Academy of Sciences of the United States of America, 114(48), 12708–12713. doi:https://doi.org/10.1073/pnas.1707323114

  • Wainer, J., & Vieira, P. (2013). Correlations between bibliometrics and peer evaluation for all disciplines: The evaluation of Brazilian scientists. Scientometrics, 96, 395–410. https://doi.org/10.1007/s11192-013-0969-9.

    Article  Google Scholar 

  • Waltman, L., & Costas, R. (2014). F1000 recommendations as a potential new data source for research evaluation: a comparison with citations. Journal of the Association for Information Science and Technology, 65(3), 433–445.

    Article  Google Scholar 

  • Wouters, P. (1999). Beyond the Holy Grail: From citation theory to indicator theories. Scientometrics, 44, 561–580.

    Article  Google Scholar 

Download references

Acknowledgements

The authors are grateful to an anonymous reviewer whose questions and suggestions prompted the authors to clarify the implications of the study.

Funding

Daniel S. Sage was partially funded by the National Science Foundation (Grant No. DMS 1503555) and the Simons Foundation (Grant No. 637367).

Author information

Authors and Affiliations

  1. Department of Mathematics, Louisiana State University, Baton Rouge, LA, 70803, USA

    Lawrence Smolinsky & Daniel S. Sage

  2. LSU Library, Louisiana State University, Baton Rouge, LA, 70803, USA

    Aaron J. Lercher

  3. Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    Aaron Cao

Authors
  1. Lawrence Smolinsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel S. Sage
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aaron J. Lercher
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aaron Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lawrence Smolinsky.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Smolinsky, L., Sage, D.S., Lercher, A.J. et al. Citations versus expert opinions: citation analysis of featured reviews of the American Mathematical Society. Scientometrics 126, 3853–3870 (2021). https://doi.org/10.1007/s11192-021-03894-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-021-03894-2

Keywords

Search

Navigation