Do social sciences and humanities behave like life and hard sciences?

Abstract

The quantitative evaluation of Social Science and Humanities (SSH) and the investigation of the existing similarities between SSH and Life and Hard Sciences (LHS) represent the forefront of scientometrics research. We analyse the scientific production of the universe of Italian academic scholars , over a 10-year period across 2002–2012, from a national database built by the Italian National Agency for the Evaluation of Universities and Research Institutes. We demonstrate that all Italian scholars of SSH and LHS are equals, as far as their publishing habits. They share the same general law, which is a lognormal. At the same time, however, they are different, because we measured their scientific production with different indicators required by the Italian law; we eliminated the “silent” scholars and obtained different scaling values—proxy of their productivity rates. Our findings may be useful to further develop indirect quali–quantitative comparative analysis across heterogeneous disciplines and, more broadly, to investigate on the generative mechanisms behind the observed empirical regularities.

Appendices

Appendix 1: List of disciplinary sectors (SSDs)

See Table 3.

Table 3 List of disciplinary sectors (SSDs)of the Italian academic system

Appendix 2: Sources and nature of data

Law no. 240/2010 has changed radically the way Italian professors have to be promoted, by creating a dual system of National Scientific Habilitation and local recruitment. The Ministerial Decree no. 76/2012 established that quantititive indicators should be used to qualify minimum requirements for candidates and delegated ANVUR to produce such indicators. On June 2012 ANVUR published in its website the Deliberation no. 50/2012, which specified in great detail the procedures for the computation of indicators based on publications. Based on the legislation that mandated the general criteria for the procedure, ANVUR announced that indicators would have been constructed on the basis of the data included by scholars in their personal homepage, called “loginmiur”. The Deliberation asked all Italian scholars to update the data included in their homepages before a deadline, following a set of instructions. These instructions clearly delimitated the kinds of publications that scholars should include in the upload of metadata. In all fields of Humanities and Socials Sciences the categories of publications were defined as follows: (a) books, including monographs, critical editions, critical translations, (b) chapters of books and journal articles, (c) articles published in A-rated journals.

The dataset was extracted on July 14, 2012, from the personal homepage of all Italian professors and researchers. The homepages are managed by Cineca, a university consortium in charge of creating and maintaining large scale computing facilities for scientific purposes, as well as information systems and services on behalf of the Ministry of University and Research and of universities. Homepages are self-administered by individual scholars and are not accessible externally. They are routinely used by the Ministry for the management of information related to academic staff, e.g. the submission of proposals for funding or of candidatures for committees or panels. With respect to the quality of data available, the following remarks are relevant. Individual scholars are responsible for updating the information. No administrative consequence or sanction is considered for not updating or filling in wrong information. The control on the quality of data provided by Cineca is a formal one. This means that Cineca, using automatic computer queries, ensures that the data uploaded by scholars are consistent with formal definitions. Among the formal controls there are the following:

• for books, the ISBN code is requested and the distinction between author and editor is recorded;

• for journals, the ISSN code is requested;

• all articles from journals are assigned to journals whose metadata are found in a Master list, managed by Cineca, based on controls of the titles;

• if a new journal title is included in the homepage, the system does not record the metadata until the new journal is recognized and included in the Master list.

With respect to the quality of data, we must distinguish between books and journal articles. With respect to books, the actual content of the category may include non-academic books. If a book has a ISBN code, its metadata are recorded in the homepage without filters. This means that the nature of the book recorded is left to the determination of scholars. While the homepage is clearly aimed at academic purposes, many scholars tend to use it as a personal repository for all their publications, including popular science, professional texts, books aimed at a political or general audience, and the like. While Deliberation 50 asked scholars to update only books of academic type, it is impossible to give an estimate of the inflation represented by non-academic books. On the contrary, for journal articles there has been a massive work of classification made by ANVUR upon mandate of the Decree 76/2012. In particular, among journals included in personal homepages, ANVUR selected those considered, following the same Deliberation n. 50, “scientific” and excluded non-scientific ones. Among scientific journals a small set (approximately 12% of the total) was included in the category of A-rated journals. As a consequence, for journals the possibility of inflation of data is almost non-existent, since all data have been filtered using formally approved lists of journals.

With respect to the issues of quality of data, it is useful to report two quality control procedures. These may give a hint of the magnitude of the problem of data quality, while they do not resolve the problem once and for all.

The first is a quality control of the data on books carried out by ANVUR during the procedure. In fact, it was possible for Full professors who were denied the access to the Habilitation panel in the first round to submit an appeal. After receiving the appeal, ANVUR re-examined all publications in order to verify whether there were errors or, most frequently omissions (in particular, omissions of actually existing ISBN codes) in the application, so that the candidate actually met the thresholds and could eventually be admitted, or rather there was a confirmation of the rejection. During the appeal procedures, the publications of more than 1500 Full professors were examined manually, of which 800 from SSH. Contrary to initial expectations, the cases in which the books declared in the homepages were clearly non-academic were extremely rare (among them, one case of tourist guide was prominent, however). In almost all cases, controversies were related to the following issues: date of publication (e.g. Professor $\times$ declares the book Y as published in year 2011, so that it is computed in the indicator, but then it is found that the date of publication has been postponed, or even it is not yet published)—wrong ISBN (e.g. Professor J includes a number not corresponding with the title of the book, leaving open the question whether it is a mistake or a fraud, i.e. the book does not have a ISBN code)—dubious academic qualification of publishers (e.g. the publisher is not member of a list of publishers associated to the National Association (AIE), has not a website or official address). It must be remembered that in the Italian legislation a provision for the precise qualification of “scientific book” is found, following a long-awaited initiative to create a transparent and open public archive of scientific publications established in 2009. However, this legislative provision has never been materialized, so that the concept of “scientific book” is left empty of specific content. Nevertheless, almost all cases treated during the appeal procedure had to deal with controversies over books whose academic content was beyond dispute. This does not exclude the fact that non-academic books have inflated the number of books in general. In fact, appeal procedures were typically submitted by Full professors whose overall production was below the thresholds by a little margin, so that recognizing even a small number of previously excluded publications might reverse the outcome. It is still possible that the list of books of those admitted since the beginning include either scientific and non-scientific books.

The second quality procedure was put in place immediately after the publication of the Deliberation n. 50, in order to check whether some senior scholar would cancel their full list in protest, perhaps with a goal of diminishing the aggregate figures. In order to detect this possibility, ANVUR controlled the figures of individual scholars daily, using a routine algorithm developed by Cineca. In the period 15 June–14 July 2012 there were no examples of massive cancelations.

Appendix 3: Detailed results

See Table 4.

Table 4 Detailed results for the LHS (a, c) and SSH (b, d) disciplinary sectors