ReviewerNet: A visualization platform for the selection of academic reviewers

Highlights

• An integrated visualization of scholarly data can support the academic reviewer search process.

• The visualization of scholarly data helps to avoid conflicts of interest and to build a fairly distributed pool of reviewers.

• A well-combined visualization of citations and co-authorship relations only, can reduce the need for complicated content analysis techniques.

• The platform evaluation with members from the Computer Graphics community demonstrated the improvement on the traditional process for searching reviewers.

• The evaluation confirmed that users were able to get acquainted with the system with a very limited training.

Abstract

We propose ReviewerNet, an online, interactive visualization system aimed to improve the reviewer selection process in the academic domain. Given a paper submitted for publication, we assume that good candidate reviewers can be chosen among the authors of a small set of pertinent papers; ReviewerNet supports the construction of such set of papers, by visualizing and exploring a literature citation network. The system helps journal editors and Program Committee members to select reviewers that do not have any conflict-of-interest and are representative of different research groups, by visualising the careers and co-authorship relations of candidate reviewers. The system is publicly available, and is demonstrated in the field of Computer Graphics.

Introduction

The number of digital academic documents, either newly published papers or documents resulting from digitization efforts, grows at a very fast pace: the Scopus digital repository counts more than 75 million records [1]; the Web of Science platform has more than 166 million records from journals, books, and proceedings [2]; Microsoft Academic collects about 232 million publications [3]. In 2019, over four hundred thousand new records in Computer Science were added to DBLP [4]. Bibliometric analysts estimate a doubling of global scientific output roughly every nine years [5]. Therefore, the volume, variety and velocity of scholarly documents generated satisfies the big data definition, so that we can now talk about big scholarly data [6].

Sensemaking in this huge reservoir of data calls for platforms adding an element of automation to standard procedures – such as literature search, expert finding, or collaborators discovery – to reduce the time and effort spent by scholars and researchers. In particular, there has been an increase in the number of visual approaches supporting the analysis of scholarly data. Visualization techniques were proposed to help stakeholders to get a general understanding of sets of documents, to navigate them, and to find patterns in publications and citations. Federico et al. [7] survey about 109 visual approaches for analysing scientific literature and patents published in-between 1991 and 2016. Most of the works focused on the visualization of document collections and citation networks. A more ambitious goal for visualization platforms would be to enable users to get enough understanding to make decisions.

In this paper, we focus on the problem of reviewer finding by journal editors or International Program Committee (IPC) members. On the one hand, reviewers are expected to know a subject well, and have enough expertise to fairly judge the work of colleagues. On the other hand, reviewers should not have any conflict-of-interest with the authors of the paper under scrutiny, where conflict means that the professional evaluation may be compromised due to a working relationship between reviewers and authors. Therefore, finding good candidate reviewers requires editors and IPC members to analyse many aspects simultaneously: topic coverage (possibly over time), stage of career, and past and ongoing collaborations. Every member of the community has his/her own approach to reviewer finding, which usually involves bibliographic research, and frequent visits to public repositories like DBLP [8] and researchers’ home pages. In any case, one has to confront possibly large collections of data to make decisions, and a user may easily get lost after following a few links.

We propose ReviewerNet, a visualization platform which facilitates the selection of reviewers. ReviewerNet builds on a reference database including papers, authors and citations from selected sources (journal articles and conference papers) taken from the Semantic Scholar Research Corpus [9]. ReviewerNet offers an interactive visualization of multiple, coordinated views about papers and researchers that help assessing the expertise and conflict of interest of candidate reviewers. The interface is shown in Fig. 1. The intuition behind ReviewerNet is that the authors of relevant papers are good candidate reviewers. One of the main advantages of ReviewerNet is that it only relies on citations, to analyse the literature, and on co-authorship relations, to analyse conflicts. Citations are an essential part of research: they represent a credible source of information about topic similarity and intellectual influence. Moreover, since citations have author-chosen reliability, they are a very robust cue to relatedness. Similar reasonings hold for co-authorship relations. Therefore, an important contribution is the demonstration that a well-combined visualization based on citation and co-authorship relations only can support the reviewer search process, without the need for more complicated content analysis techniques.

A preliminary version of the platform was presented at the 2019 edition of the Smart Tools and Applications in Graphics conference [10]. After validating the platform with a pool of senior researchers in Computer Graphics (see Section 5), we incorporated their feedback to improve the tool. Therefore, the present article describes the updated platform, which features both a revisited interface and new functionalities:

• improved visualization of the network of papers, with a different network layout for better investigation and interaction;

• improved visualization of the network of researchers, for better interactive community discovery;

• improved consistency of visual elements;

• a novel initialization procedure to complement the manual insertion of paper titles, by importing and automatically parsing a list of bibliographic references;

• the possibility for users to generate personalized instances of the platform, in any academic domain, by specifying a list of venues of interest.

We demonstrate the platform usage in the field of Computer Graphics, on a reference dataset containing 17.754 papers, 108.155 citations, 23,386 authors. We show how ReviewerNet can be used to search for reviewers who are expert on a certain topic, are at a certain career stage, who have a certain track of publishing records, who are not conflicting with neither the submitters nor other reviewers, and who are well-distributed in the scientific community.

The tool is free to use and open source; the source code is available at https://github.com/cnr-isti-vclab/ReviewerNet, while the demonstration platform is available at https://reviewernet.org/.