Skip to main content
SpringerLink
Log in

Computing a journal meta-ranking using paired comparisons and adaptive lasso estimators

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

In a “publish-or-perish culture”, the ranking of scientific journals plays a central role in assessing the performance in the current research environment. With a wide range of existing methods for deriving journal rankings, meta-rankings have gained popularity as a means of aggregating different information sources. In this paper, we propose a method to create a meta-ranking using heterogeneous journal rankings. Employing a parametric model for paired comparison data we estimate quality scores for 58 journals in the OR/MS/POM community, which together with a shrinkage procedure allows for the identification of clusters of journals with similar quality. The use of paired comparisons provides a flexible framework for deriving an aggregated score while eliminating the problem of missing data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Adler, N. J., & Harzing, A. W. (2009). When knowledge wins: Transcending the sense and nonsense of academic rankings. Academy of Management Learning and Education, 8(1), 72–95.

    Article  Google Scholar 

  • Agresti, A. (2010). Analysis of ordinal categorical data (2nd ed.). Hoboken: Wiley.

    Book  MATH  Google Scholar 

  • Barman, S., Tersine, R. J., & Buckley, M. R. (1991). An empirical assessment of the perceived relevance and quality of POM-related journals by academicians. Journal of Operations Management, 10(2), 194–212.

    Article  Google Scholar 

  • Barman, S., Hanna, M. D., & LaForge, R. L. (2001). Perceived relevance and quality of POM journals: A decade later. Journal of Operations Management, 19(3), 367–385.

    Article  Google Scholar 

  • Bergstrom, C., Jevin, D. W., & Wiseman, M. A. (2008). The Eigenfactor (TM) metrics. The Journal of Neuroscience, 28(45), 11,433–11,434.

    Article  Google Scholar 

  • Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345.

    Article  MathSciNet  MATH  Google Scholar 

  • Cattelan, M. (2012). Models for paired comparison data: A review with emphasis on dependent data. Statistical Science, 27(3), 412–433.

    Article  MathSciNet  Google Scholar 

  • Cheang, B., Chu, S. K. W., Li, C., & Lim, A. (2014). OR/MS journals evaluation based on a refined PageRank method: An updated and more comprehensive review. Scientometrics, 100(2), 339–361.

    Article  Google Scholar 

  • Cook, W. D., Raviv, T., & Richardson, A. J. (2010). Aggregating incomplete lists of journal rankings: An application to academic accounting journals. Accounting Perspectives, 9(3), 217–235.

    Article  Google Scholar 

  • de Borda, J. C. (1781). Mémoire sur les elections au scrutin. Histoire de l’Académie Royale des Sciences.

  • Dwork, C., Kumar, R., Naor, M., & Sivakumar, D. (2001). Rank aggregation methods for the web. In Proceedings of the 10th international conference on World Wide Web (pp. 613–622). ACM.

  • Emond, E. J., & Mason, D. W. (2002). A new rank correlation coefficient with application to the consensus ranking problem. Journal of Multi-Criteria Decision Analysis, 11(1), 17–28.

    Article  MATH  Google Scholar 

  • Fan, J., & Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96(456), 1348–1360.

    Article  MathSciNet  MATH  Google Scholar 

  • Firth, D. (2015) qvcalc: Quasi variances for factor effects in statistical models. http://CRAN.R-project.org/package=qvcalc. R package version 0.8-9.

  • Firth, D., & De Menezes, R. X. (2004). Quasi-variances. Biometrika, 91(1), 65–80.

    Article  MathSciNet  MATH  Google Scholar 

  • Frey, B. S. (2010). Withering academia? CESifo working paper series no 3209. SSRN. http://ssrn.com/abstract=1694448.

  • Frey, B. S., & Rost, K. (2010). Do rankings reflect research quality? Journal of Applied Economics, 13(1), 1–38.

    Article  Google Scholar 

  • Fry, T. D., & Donohue, J. M. (2013). Outlets for operations management research: A DEA assessment of journal quality and rankings. International Journal of Production Research, 51(23–24), 7501–7526.

    Article  Google Scholar 

  • Fry, T. D., & Donohue, J. M. (2014). Exploring the author affiliation index. Scientometrics, 98(3), 1647–1667.

    Article  Google Scholar 

  • Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178, 471–479.

    Article  Google Scholar 

  • Geary, J., Marriott, L., & Rowlinson, M. (2004). Journal rankings in business and management and the 2001 research assessment exercise in the UK. British Journal of Management, 15(2), 95–141.

    Article  Google Scholar 

  • Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193.

    Article  Google Scholar 

  • Glänzel, W., & Moed, H. F. (2013). Opinion paper: Thoughts and facts on bibliometric indicators. Scientometrics, 96(1), 381–394.

    Article  Google Scholar 

  • Goh, C., Holsapple, C., Johnson, L., & Tanner, J. (1996). An empirical assessment of influences on POM research. Omega, 24(3), 337–345.

    Article  Google Scholar 

  • González-Pereira, B., Guerrero-Bote, V. P., & Moya-Anegón, F. (2010). A new approach to the metric of journals scientific prestige: The SJR indicator. Journal of informetrics, 4(3), 379–391.

    Article  Google Scholar 

  • Halkos, G. E., & Tzeremes, N. G. (2011). Measuring economic journals citation efficiency: A data envelopment analysis approach. Scientometrics, 88(3), 979–1001.

    Article  Google Scholar 

  • Harzing, A. W. (2014). Journal quality list (52nd ed.). http://www.harzing.com.

  • Holsapple, C. W., & Lee-Post, A. (2010). Behavior-based analysis of knowledge dissemination channels in operations management. Omega, 38(3), 167–178.

    Article  Google Scholar 

  • Hornik, K., Jankowitsch, R., Lingo, M., Pichler, S., & Winkler, G. (2007). Validation of credit rating systems using multi-rater information. Journal of Credit Risk, 3(4), 1–27.

    Google Scholar 

  • Kemeny, J. G., & Snell, L. (1962). Mathematical models in the social sciences. Cambridge: MIT Press.

    MATH  Google Scholar 

  • List, C. (2013). Social choice theory. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy, (Winter 2013 ed.). Stanford: The Metaphysics Research Lab.

  • Lozano, S., & Salmerón, J. L. (2005). Data envelopment analysis of OR/MS journals. Scientometrics, 64(2), 133–150.

    Article  Google Scholar 

  • Masarotto, G., & Varin, C. (2012). The ranking lasso and its application to sport tournaments. The Annals of Applied Statistics, 6(4), 1949–1970.

    Article  MathSciNet  MATH  Google Scholar 

  • Meredith, J. R., Steward, M. D., & Lewis, B. R. (2011). Knowledge dissemination in operations management: Published perceptions versus academic reality. Omega, 39(4), 435–446.

    Article  Google Scholar 

  • Mingers, J., & Harzing, A. W. (2007). Ranking journals in business and management: A statistical analysis of the Harzing data set. European Journal of Information Systems, 16(4), 303–316.

    Article  Google Scholar 

  • Moed, F. (2011). The source normalized impact per paper is a valid and sophisticated indicator of journal citation impact. JASIST, 62, 211–213.

    Article  Google Scholar 

  • Olson, J. E. (2005). Top-25-business-school professors rate journals in operations management and related fields. Interfaces, 35(4), 323–338.

    Article  MathSciNet  Google Scholar 

  • Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The pageRank citation ranking: Bringing order to the web technical report. Stanford Digital Libraries Project.

  • Petersen, C. G., Aase, G. R., & Heiser, D. R. (2011). Journal ranking analyses of operations management research. International Journal of Operations and Production Management, 31(4), 405–422.

    Article  Google Scholar 

  • Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics. Information Processing and Management, 12(5), 297–312.

    Article  Google Scholar 

  • R Core Team. (2015). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/.

  • Serenko, A., & Bontis, N. (2013). Global ranking of knowledge management and intellectual capital academic journals: 2013 update. Journal of Knowledge Management, 17(2), 307–326.

    Article  Google Scholar 

  • Serenko, A., & Dohan, M. (2011). Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence. Journal of Informetrics, 5(4), 629–648.

    Article  Google Scholar 

  • Theußl, S., Reutterer, T., & Hornik, K. (2014). How to derive consensus among various marketing journal rankings? Journal of Business Research, 67(5), 998–1006.

    Article  Google Scholar 

  • Thomson Reuters. (2014). The 2014 Journal Citation Reports ® (JCR), Social Sciences Edition. Philadelphia: Thomson Reuters.

  • Tibshirani, R. J. (2011). The solution path of the generalized lasso. Redwood City: Stanford University.

    MATH  Google Scholar 

  • Tüselmann, H., Sinkovics, R. R., & Pishchulov, G. (2015). Towards a consolidation of worldwide journal rankings—A classification using random forests and aggregate rating via data envelopment analysis. Omega, 51, 11–23.

    Article  Google Scholar 

  • Vokurka, R. (1996). The relative importance of journals used in operations management research: A citation analysis. Journal of Operations Management, 14(4), 345–355.

    Article  Google Scholar 

  • Willmott, H. (2011). Journal list fetishism and the perversion of scholarship: Reactivity and the ABS list. Organization, 18(4), 429–442.

    Article  Google Scholar 

  • Xu, F., Liu, W., & Rousseau, R. (2015). Introducing sub-impact factor (SIF-) sequences and an aggregated SIF-indicator for journal ranking. Scientometrics, 102(2), 1577–1593.

    Article  Google Scholar 

  • Xu, Z., Cheang, B., Lim, A., & Wen, Q. (2011). Evaluating OR/MS journals via PageRank. Interfaces, 41(4), 375–388.

    Article  Google Scholar 

  • Young, H. P. (1986). Optimal ranking and choice from pairwise comparisons. Information pooling and group decision making, pp. 113–122.

  • Zhou, D., Ma, J., & Turban, E. (2001). Journal quality assessment: An integrated subjective and objective approach. IEEE Transactions on Engineering Management, 48(4), 479–490.

    Article  Google Scholar 

  • Zou, H. (2006). The adaptive lasso and its oracle properties. Journal of the American Statistical Association, 101(476), 1418–1429.

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

The authors thank the two reviewers for their insightful comments.

Author information

Authors and Affiliations

  1. Institute for Statistics and Mathematics, WU Vienna University of Economics and Business, Welthandelsplatz 1, 1020, Vienna, Austria

    Laura Vana, Ronald Hochreiter & Kurt Hornik

Authors
  1. Laura Vana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ronald Hochreiter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kurt Hornik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laura Vana.

Appendix: Journal abbreviations and ranking lists

Appendix: Journal abbreviations and ranking lists

See Tables 4 and 5.

Table 4 Journal abbreviations
Full size table
Table 5 Journal ranking lists
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vana, L., Hochreiter, R. & Hornik, K. Computing a journal meta-ranking using paired comparisons and adaptive lasso estimators. Scientometrics 106, 229–251 (2016). https://doi.org/10.1007/s11192-015-1772-6

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-015-1772-6

Keywords

Search

Navigation