Journal ratings as predictors of articles quality in Arts, Humanities and Social Sciences: an analysis based on the Italian Research Evaluation Exercise

Introduction

There is a large degree of agreement on the notion that research assessment in humanities and social sciences (HSS) is made more complex by a variety of factors. First, in these fields the structure of academic publication is largely different, with a large weight assigned to books and monographs, and to production in national language (Finkenstaedt, 1990). Consequently, the bibliometric approach is considered to be of limited usefulness (Nederhof et al., 1989), not only because journals are a small fraction of total production and indexed journals are a tiny fraction of the population of journals in HSS, but also because the meaning of citations is different (Frost, 1979). Third, and even more challenging, there is evidence that the number of research quality criteria is larger in HSS than in other fields, and also that there is less agreement on these criteria (Hemlin, 1996; Hemlin & Gustafsson, 1996; Hug et al., 2013; Hug et al., 2014; Ochsner et al., 2012; Ochsner et al., 2013).

Faced with these challenges, the state of the art of the assessment of research in HSS at international level has followed several directions. On the one hand, it is agreed that peer review is still the most important evaluation methodology, so large efforts are made in making it more sophisticated, methodologically controlled, based on sound principles of evaluation methodology in social sciences, and free from unwanted biases, distortions and unexpected side effects. Under this agenda, issues such as the notion of originality, unorthodox science, or interdisciplinarity are under examination (Guetzkow et al., 2004; Hammarfelt, 2011). On the other hand, there are many efforts to classify and evaluate non-indexed journals (mainly in national languages), as one of the main vehicles for academic communication. An additional line of work refers to the classification of books and publishers.

This paper reports on a large experiment in the classification of journals in HSS carried out in Italy in the 2012–2014 period for the National Scientific Habilitation (Abilitazione Scientifica Nazionale; ASN). The exercise was based upon a mandatory provision in the law to rate all journals, in order to calculate the overall academic production of all candidates to the national procedure to become associate professor or full professor. This exercise asked the National Agency for the Evaluation of Universities and Research Institutes (ANVUR) to evaluate all journals in which at least one Italian scholar published at least one paper in the 2002–2012 period, for a total of more than 60,000 titles.

While the rating of journals has been followed in several national contexts, it is only in the Italian exercise that there is the opportunity to carry out a controlled experiment in order to test the robustness of journal classification. In fact, we have two independent evaluations carried out on the same set of journals. On the one hand, a panel of experts classified all journals as academic and non-academic (i.e. popular, professional, technical, cultural and political etc.), and rated the subset of academic journals in A-rated and non A-rated. The rating exercise was done on the basis of the reputation, esteem, diffusion and impact of journals, that is, on a qualitative, expert-based, reputational basis. On the other hand, we also have the rating of individual articles published in those journals, which have been done by a large number of individual referees (not panels) and summarized with a consensus agreement approach by expert panels, who however acted independently from the other panels, and without exchange of information. This peculiarity of the Italian context and the time sequence of events creates a favorable condition for carrying out a controlled experiment.

This paper extends to all HSS, with the exception of economics and business, the analysis initiated by Ferrara & Bonaccorsi (2015) on journals in the area of philosophy and history. In the following, we first introduce the database used for the analysis and hence we test for the influence of the journal class on the article score. Some consideration on the results obtained will conclude the paper.

Methods

The paper is based on a dataset including data on all the journal articles submitted for evaluation by Italian scholars in the disciplinary areas of architecture, arts and humanities, history and philosophy, law and sociology and political science. Submissions for evaluation took place within the framework of VQR 2004–10, Italy’s national research assessment exercise involving all professors and researchers affiliated to the Italian universities and Public Research Organizations (PROs) as of November 2011. According to adopted rules, research evaluation in HSS was entirely based on peer review; research quality was assessed against the criteria of relevance, meaning contribution to the advancement of the state of art in the field, also in terms of adequacy, efficacy, timeliness and duration of impacts; originality and innovation, meaning contribution to the creation of new knowledge in the field; internationalization, meaning position in the international research landscape. Evaluation has been conducted by five Groups of Evaluation Experts (GEV in the Italian acronym), one for each area in HSS (Architecture; Arts and Humanities; History and Philosophy; Law; Sociology and political sciences); reviewers were instructed by GEV to evaluate articles only on the basis or their merit, regardless of the journal in which they are published in and of the language of publication. Each article had a possible rating of Excellent (A), Good (B), Fair (C) or Limited (D); to each class corresponded a score ranging from 1 (for articles A-rated) to zero (for articles deemed as limited). Negative scores were also assigned in case the article was deemed as non-academic (-1) or for plagiarism or fraud (-2, see Ancaiani et al., 2015 for details). Limited to the human and social sciences, a substantial fraction of articles – namely, 6,701 out of 11,660 (Table 1) – appeared on journals deemed as ‘A-class’ according to the procedure of ASN, intended to select the best researchers for the ranks of associate and full professors. Those journals, according to the relevant Ministerial Decree (No. 76/2012), were those ‘internationally recognized as excellent because of the rigor of their procedures of peer review and because of their diffusion among, esteem by, and impact on, the scholarly community of a field, as indicated also by their presence in the major national and international databases’ (our translation). Most of the remaining articles appeared on journals deemed as ‘academic’ for the purposes of the ASN, while a minority were published in journals that remained ‘uncategorized’. The main feature of the dataset, thus, is that it allows the comparison between the evaluations of journals and individual articles.

A preliminary analysis shows that there is a relationship between the evaluation of individual articles and that of journals where the article is published (Table 2). The non-parametric test for categorical data (Pearson χ²) is statistically significant at 1% (All the statistical analyses have been performed using the software STATA ver. 13 (http://www.stata.com/stata13/)), showing that the two distributions are not independent and hence the two ratings are mutually related. In the following, we will analyze more thoroughly this relationship, also controlling for a number of author-level and article-level variables.

The influence of journal classification on the article score

We assume that the probability for an article i, published in the journal j, of receiving a score equal to x ∈ {-2; 1} is influenced by the class assigned to the journal, once controlling for a number of characteristics of the article:

P (Score_i,j = x) = F(Journal class_i,j, Paper characteristics_i,j)       (1)

Among the controls, we consider the language of publication (Italian or not) and the age (distinguishing among 3 age classes, less than 40 years, between 41 and 55 years and more than 55 years), scientific sector of activity (Scientific Areas 8, 10, 11, 12, 14), academic status (full professor; associate professor; researcher; other) and gender of the researcher. We also add the consideration of two binary variables controlling for the existence of international co-author(s) and for the nationality of the referees (allowing for the possibility of international referees). We finally add a variable taking into account the size of the scientific area of the author. The model is estimated as an ordered probit, an extension of the standard binary probit model, used when the dependent variable takes the form of a ranked and multiple discrete variable, considering alternatively the whole sample or each scientific area; in the first case, we also control for possible area-specific effects. In order to avoid the “dummy trap”, we normalize with respect to articles written in Italian with no international co-author, evaluated by an Italian reviewer, presented by a female researcher in sociology and political science, aged less than 40: i.e. the statistical significance, sign and magnitude of estimated parameters are to be interpreted as differentials with respect to this control group. The total number of available observations amounts to 11,660 varying from a minimum of 918 in architecture to a maximum of 3,838 in law (Table 3).

The main result is that both at the aggregate level and in each scientific area the article score is higher as the journal ranking gets better: in other words, the probability of receiving a high score grows if the article is published in a high-ranking journal according to the evaluation of the ASN’s experts. As for the control variables, we confirm most of the results already emerged in a previous paper on the same data (Cicero et al., 2014), namely, that article scores are higher for papers not written in Italian, with international co-authors, published by an under-40, male full or associate professor. Moreover, we also find that at aggregate level and in most areas an international reviewer and a lower number of professors in the specific scientific sector (SSD) are associated with an higher article score: a possible interpretation of the first result is that the expert groups responsible for the evaluation (GEV) mostly assign to international reviewers more internationalized papers, that are considered to have an higher probability of receiving a high score, given also that the level of internationalization was one of the evaluation criteria according to VQR rules (see again Ancaiani et al., 2015). As for a negative relationship among area size and article score, this result emerged already in Ferrara & Bonaccorsi (2015) for the scientific fields in history and philosophy and is now extended to all HSS: a possible interpretation is that small fields may be favored by a “proximity bias” among authors and reviewers, thus resulting, ceteris paribus, in higher article scores.

As a final check, we concentrate on the probability of receiving an excellent score and relate it to the fact that the article is published in a top, A-Class journal, once controlling for the same variables considered in model 1:

P (Score_i,j = “E”) = F(Journal class_i,j = “A”, Paper characteristics_i,j)       (2)

In (2), F is the logistic function and the model is estimated as a logit, a class of models allowing to predict the binary response based on the specified predictors. A desirable feature of the logit model is that the regression coefficients may easily be transformed in odds ratio, expressing the change in the odds of the occurrence under scrutiny (in our case, the odds for a paper of receiving an ‘Excellent’ evaluation) due to a small change of a given predictor: in our case, we are particularly interested in the odds associated with the classification of a journal as a top, Class A journal. Estimation results for both the aggregate sample and each scientific area are presented in Table 4.

According to logit estimations, the probability of receiving an excellent evaluation is positively affected by the journal in which the paper is published in: more specifically, publishing in a class A journal almost doubles the probability of receiving an excellent evaluation. Looking at the results in each scientific area, the odds of receiving an excellent evaluation are more than doubled by the publication in a Class A journal in architecture and history and philosophy; the effect is somewhat lower, but still highly significant, in law, and arts and humanities, while disappearing in sociology and political sciences. Logit estimation also broadly confirms the results already emerging from the ordered probit model: the odds of receiving an excellent evaluation are increased by publishing in a foreign language, with an international co-author (albeit only in law and architecture) and when the submitting author is 40 years old or younger, an associate or full professor and a male. Gender effect is in fact significant at the aggregate level and in architecture and humanities, but not in the remaining areas. Also in this case, having an international reviewer and publishing in a SSD characterized by a lower number of full professors helps in obtaining an excellent evaluation.

Conclusions

Using a very large dataset of journal articles published in HSS, the paper proves that independent classifications of journals may be considered as good predictors of the score assigned to individual articles. More specifically, we find that, after controlling for a number of articles’ characteristics, the probability of receiving a better score grows with the quality profile of the journal the article is published in; moreover, the probability of receiving an excellent score almost doubles when the paper is published in a top, A-Class journal. The findings hold both at the aggregate level and for each specific sub-area considered in the analysis. While peer review has to remain the main evaluation methodology, our results indicate that journal classifications may be considered as a useful supporting tool in large evaluation exercise, since it may provide reviewers with valuable information apt to support expert evaluation.

Data availability

The authors hold the view that it is important to allow the free access to data used in the article in order to enable others to replicate the study. However, information used in the article were gathered by the national agency responsible for evaluation of the University and research system in Italy (ANVUR), in the framework of this VQR exercise. In this context, ANVUR asked Italian professors to provide access to their publications, assuming the commitment not to disclose to the public, unless in an aggregate form, any data concerning the publications submitted for the evaluation and, most importantly, the results of the evaluation itself. This is deemed as necessary in order to guarantee the full anonymity of evaluations performed on each individual publications and on each Italian professor. For this reason, as the public agency in charge of evaluating research of Italian universities, ANVUR does not allow to make information about individual evaluations available to the general public.

The information used to generate data in this article concerning journal classification is available to the public at the following URL: http://www.anvur.it/index.php?option=com_content&view=article&id=254&Itemid=315&lang=it.