On the Mark? Responses to a Sting

It all seems very simple. A researcher writes up her findings to share them with her community. The findings are sent to a journal where the editor sends the manuscript out to peers to read. The reviewers comment and return the manuscript to the editor. The editor either asks the researcher to address the comments, or rejects the manuscript based on the reviews. The researcher sends their final manuscript to the editor, who in turn works with the publisher to ensure dissemination. Done.

Except that it's not that simple. The pressures of tenure and promotion, the rigor (or lack thereof ) in the peer review system, the cost of journal subscriptions, and other implications from the move to online scholarship aren't immediately visible in the scholarly communication environment, but they play a very real role. Scholarly publishing is ripe for change and involved-authors, reviewers, editors, publishers, librarians, and readersowe it to our future selves to start challenging legacy processes. We built the scholarly publishing system because research deserves to be shared, but that intention has been lost in favor of financial and status motives.
Published on October 4, 2013, in Science, "Who's Afraid of Peer Review?" 1 by John Bohannon attempted to evaluate the peer review system. Focusing solely on open access (OA) journals, Bohannon's findings were controversial. The lack of rigorous peer review in the 1 http://www.sciencemag.org/content/342/6154/60.full journals surveyed further bolstered the claims of OA detractors that open journals are lesser than their tollaccess counterparts. OA advocates pointed to the fact that many toll-access journals would have the same issues if they were studied similarly.
The current scholarly publishing system is unsustainable. Financially, libraries can't continue to pay rising subscription rates for big packages (there is no such thing as doing more with less). Ethically, using taxpayer money for research that taxpayers can't read without a subscription is unacceptable. If we all believe that research deserves to be shared (see my previous statement regarding why we built scholarly publishing in the first place), the system must evolve. Yes, business models will have to change-just as they have in many industries now that online life is the norm. Those who still believe this change doesn't affect scholarly publishing and academia are naïve, and need to quit putting up road blocks for the rest of us who understand how many opportunities are now open to publishing. Whether one agrees (or not) with its approach or conclusions, Bohannon's article serves as yet another call for the evolution of scholarly publishing. While the issue at hand may be the traditional process of peer review, the underlying issues-the purpose of publishing in academic life; the tension between public good and commercial ends; the necessity of transparency in the conduct and assessment of scholarship-are even more JL SC worthy of discussion and action. As a contribution to these discussions, JLSC invited several researchers to briefly comment on the Science article. Eve describes how publishing has become more about measuring and ranking academics than about communicating research findings. Steel calls for a re-evaluation of the publishing system in general and affirms the need for greater transparency. Gardy addresses the flaws and inconsistencies of peer review. And, finally, Salo tackles the "sting"-and the need for even more of them.
We hope that all of us involved in the scholarly publishing environment will heed the call of these authors-and of Bohannon-and continue to seek the necessary reformation of scholarly publishing and a return to our "first principle" of shared knowledge.
It becomes clear to most academics early in their career development that publication of scholarly material or scientific research fulfils two distinct and discrete functions. The first is to communicate research findings to a broader audience and to enter into the global conversation that forms the discipline. The second is less noble but nonetheless necessary: there is a need to ascribe relative value to work; to rank scholars and scientists; and for hiring, firing, and tenure committees to find proxy measures for this evaluative function. One of the outcomes of the Open Access movement, as a subset of its primary aims, has been to expose the fact that the former aspect, which should be the true purpose of academia, has been occluded in favour of the latter. We must dare to know better than this.
The reasons for this masking are clear. Under continuing contemporary conditions, in which a scarcity of academic jobs leads to intense competition and breeds precarious labour practices, it is imperative for senior faculty to have recourse to valid proxy metrics. For, while each piece of scholarship and scientific research carries with it a type of incomparable uniqueness akin to that found in art, without comparison there cannot be competition; without relative comparison it would be impossible to match applicants to the sparse number of jobs.
The true problem, however, is that these proxy measures have become drivers, rather than assessors. Publication venues (rather than articles) are assessed on their prestige factor (by which I mean the value they confer through their names; names-as-proxies) with which authors want their work to be associated, often formalised into the flawed Impact Factor. 1 Academic freedom to select a publication destination is then most commonly used to merely "select" the venue that has previously fared best as a proxy measure of quality. In this way, the sole prioritised criterion for publication becomes the perceived value of the proxy measure and neither reach nor public accessibility. 2 In his recent dubious Science sting piece, John Bohannon singled out open access journals to show that many had poor, or no, quality control measures. He did not conduct the same experiment on toll-access journals; his own "control" was missing in action. What I think might be interesting, however, is to consider that Bohannon's very method implies an assumption: that the role of journals and publishers is to shape themselves as valid proxy measures for value by grouping good material under one name. Yet, this type of pre-filtering to appease evaluation should not be driving us and may not even be relevant in the digital age. 3 Material should be published where it will have the furthest reach and societal impact, for better reasons than the appeasement of metrics, JL SC especially when that closes access and bankrupts our libraries. 4 It is only through this kind of realisation that we will free ourselves from the economic damnation of 4 I am well aware of the argument that impact is gained by the right people having access to knowledge and not through the broadest possible access. It seems clear to me that the right people will certainly have access when everybody does.
subscription-based commercial academic publishing and our own self-incurred minority of playing childish games. Indeed, part of our enlightenment must be an awareness of the bounds that constrain and structure our publication practices but also a return to the work itself rather than being governed by proxies for hiring and promotion.

Graham Steel Patient Advocate and Open Science Enthusiast
One of the first to comment on Bohannon's article was PLOS co-founder Michael Eisen 1 who tweeted, 2 "I confess. I wrote the Arsenic DNA paper to expose flaws in peer-review at Science http://www.michaeleisen.org/ blog/?p=1439 @sciencemagazine." A pull-quote from his blog post states, "There are deep problems with science publishing. But the way to fix this is not to curtain open access publishing. It is to fix peer review." Shortly after, Ivan Oransky 3 of Retraction Watch 4 tweeted, 5 "@sciencemagazine reporter spoofs hundreds of open access journals with fake papers http://bit. ly/1bAt5Hu." A pull-quote from that blog post notes: Bohannon's analysis, 6 which goes into far more depth, demonstrates an appalling lack of peer review and quality control at the journals he spoofed. But it's important to note, given the heated and endless debates between open access advocates and traditional publishers, that there was no control group.
In amongst all of the blog posts that followed, a key message is contained in the OASPA response 7 : There are currently 9,948 journals listed in the DOAJ 9 and in terms of the overall number of scientific journals, there are in excess of 25,000. As pointed out by many, crucially, there was no "control group" in this operation and several people saw this as Science scoring a significant own goal.
Regardless of the clear flaws and weaknesses of Bohannon's analysis, his article (and the OASPA and DOAJ statements) illustrate the need for greater openness and The ability of figshare to visualise research objects, such as datasets in the browser as well as providing detailed metrics on their use is very appealing to traditional publishers. This is the first collaboration between figshare and a publisher, but as the size and variety of research outputs continues to grow it is anticipated that publishers will be seeking out this

Jennifer Gardy Assistant Professor, School of Population and Public Health, University of British Columbia
If there is but an iota of good to come from Bohannon's sting, it's that it reinvigorated the debate around the current state of peer review, at least within an anecdotal un-controlled sample of scientific colleagues of low N (perhaps sufficient for a Science investigation). For a discipline that prides itself on reproducibility, precision, and accountability, science's peer review process is wildly variable with respect to quantity and quality of reviews, and is generally cloaked in mystery. Some journals employ two reviewers, others three, some four or more. Some reviews might be so brief as to almost resemble a haiku, while others might compete with the manuscript under review for length. A handful of reviews are signed by the referee, even fewer are made available with the article as part of its history. And, despite the refereeas-gatekeeper model meant to keep flawed studies out of the literature, duds sneak past. Perhaps this is due to inattention on the part of the reviewers but is more JL SC likely reflective of the fact that two or three individuals cannot reasonably be expected to be experts in the breadth of material contained in papers written in the modern atmosphere of team science, where a single work includes analyses spanning multiple disciplines.
Opinions on whither the peer review process should go are as varied as the calibre of reviews themselves. Some have proposed simple changes to the existing system: blinding referees to authors' names to reduce bias, both positive and negative; encouraging referees to sign their reviews, thereby ensuring they treat reviews with the same gravity as other knowledge products they'd sign their name to; or permitting deposition of drafts in pre-publication repositories like arXiv.org to solicit feedback from the community before submitting a refined manuscript to a peer-reviewed journal. Others support more dramatic reforms, such as rapid peer ratings of technical merit and fit to a given journal, followed by open post-publication commentary from the community, maintained over the lifespan of a published work.
Whatever the solution, virtually every scientist with at least a handful of publications to their name will agree that the system as it currently stands is fundamentally flawed and wildly inconsistent. Were peer review to be peer reviewed, it would almost certainly be rejected.

Dorothea Salo Faculty Associate, School of Library and Information Studies, University of Wisconsin-Madison
By now, many of us have received emailed come-ons from scammy outfits calling themselves open-access publishers. I don't understand why they bother targeting librarians-we of all people know better!-but target us they do, wasting our time and tarnishing academic librarianship's opinion of gold open access. That would be a mere annoyance rather than an outrage if these same shills didn't also exploit vulnerable populations such as naïve graduate students and new Ph.Ds under publish-or-perish pressure, as well as pollute the open web with pseudo-scholarly trash.
We have thus far responded to scam open-access publishing with educational initiatives. Many are the LibGuides on evaluating publisher quality and selecting a journal to publish in, for example. Ideally, these laudable efforts would starve the scammers of their prey, but that clearly hasn't happened. I'm doubtful it will, redouble our efforts as we may. Part of the problem is that we don't have enough mindshare among our faculty and graduate students to reach and protect them all. Worse, though, is the cognitive load of the evaluation methods we tend to offer. Would-be prey have a simple question, "helpful to my career y/n?" that they want an on-the-spot answer to. Asking them to look at funding, disciplinary spread, peer-review practices, and typography for every single journal under consideration does not help them.
Could we attack the problem from the demand side, trying to starve the scammers of willing readers? Unfortunately, at present this does them no damage whatever, as they do not presently have to prove readership to capture prey. We may hope that the alternative-metrics movement eventually offers a route out of this dilemma, by making more readership proxies more transparent, but we will not be able to rely on alternative metrics until they achieve considerably greater adoption by journals both open and toll-access.
The Directory of Open Access Journals 1 has taken a step toward easier answers by tightening its inclusion criteria and by instituting a Seal of Approval program. Once DOAJ finishes expelling listings that do not meet the new criteria, we can all feel safer about telling potential prey "if it's not in DOAJ, stay away!" This is progress, but a cursory examination of the new DOAJ criteria shows that they are crediting good practices such as peer review, rather than punishing bad practices such as email spam, falsely-listed editors, and junkety conferences. Moreover, DOAJ covers open-access journals only, not scammy "book" publishers who prey on recent dissertators. Its program simply does not suffice to eliminate all the scammers and scammy practices.
The central irony of DOAJ's new program is that JL SC despite lingering doubts about the quality of gold-OA journals, the DOAJ has now ensured that open access has more controls against poor quality and scamminess than subscription and hybrid journals do! If we rely on DOAJ, then, we have solved significantly less than half the scam-journal problem, never mind the scambook problem. We have not eliminated medical journals that accept industry-ghostwritten articles. 2 We have not eliminated subscription vanity journals clinging like parasites to Big Deals. 3 We have not eliminated double-dipping hybrid journals, or journals that impose excessive page charges and other author-side fees while simultaneously raking in library subscription money. We have not eliminated low-quality subscription journals invented purely to puff up the apparent size of Big Deals and abstracting/indexing coverage. We have not eliminated book publishers who sue librarian bloggers making informed comments about their quality. 4 What might eliminate more of the scammers, regardless of underlying business model, is a truly systematic approach to Bohannonesque sting operations. That Bohannon carried out a sting operation is not the problem with his article; the problem is that he didn't Stinging every journal from every journal publisher, never mind scammy book publishers, would be an impossible Augean-stable task, and that's before we quest for a reputable and lawsuit-resistant organization to do the work. How might we winnow the sample, without letting anyone entirely off the hook? The time-honored random sample could work, and the mere threat of it might encourage more responsible behavior; the difficulty is fielding a large enough random sample to credibly threaten every scammer with discovery. Less scientific but perhaps more effective is to guess that where there's smoke, there's fire. Journals that spam, or are caught plagiarizing or selling out, are top candidates for a quick sting; collecting evidence could be as simple as an email address that evidence can be forwarded to. DOAJ suggests a last promising possibility: create a voluntary seal-of-approval program under the aegis of a reputably objective organization, perhaps the Association for Research Libraries (though a firewall between this program and ARL's Scholarly Publishing and Academic Resources Coalition would be wise). Trade organizations simply will not do, as the somewhat checkered histories of the Open Access Scholarly Publishers Association and the PRISM Coalition strongly hint.
Let's sting the plagiarists. Let's sting the spammers. Let's sting the industry sellouts. Let's sting the copycats. Let's sting the Big Deal remoras. Let's sting the peer-review fakers. Let's sting the lazy editors. Let's sting the statuschasers who let bad articles through on the strength of impressive-looking author affiliations or inflated results. Let's sting greedy double-dippers. They waste our time, attention, and money. Sting them all to death!