A month after I and Matt published our paper “Why is vertebral pneumaticity in sauropod dinosaurs so variable?” at Qeios, we were bemoaning how difficult it was to get anyone to review it. But what a difference the last nineteen days have made!

In that time, we’ve had five reviews, and posted three revisions: revision 2 in response to a review by Mark McMenamin, version 3 in response to a review by Ferdinand Novas, and version 4 in response to reviews by Leonardo Cotts, by Alberto Collareta, and by Eduardo Jiménez-Hidalgo.

Taylor and Wedel (2021: Figure 2). Proximal tail skeleton (first 13 caudal vertebrate) of LACM Herpetology 166483, a juvenile specimen of the false gharial Tomistoma schlegelii. A: close-up of caudal vertebrae 4–6 in right lateral view, red circles highlighting vascular foramina: none in Ca4, two in Ca5 and one in Ca6. B: right lateral view. C: left lateral view (reversed). D: close-up of caudal vertebrae 4–6 in left lateral view (reversed), red circles highlighting vascular foramina: one each in Ca4, Ca5 and Ca6. In right lateral view, vascular foramina are apparent in the centra of caudal vertebrae 5–7 and 9–11; they are absent or too small to make out in vertebrae 1–4, 8 and 12–13. In left lateral view (reversed), vascular foramina are apparent in the centra of caudal vertebrae 4–7 and 9; they are absent or too small to make out in vertebrae 1–3, 8, and 10–13. Caudal centra 5–7 and 9 are therefore vascularised from both sides; 4 and 10–11 from one side only; and 1–3, 8 and 12–13 not at all.

There are a few things to say about this.

First, this is now among our most reviewed papers. Thinking back across all my publications, most have been reviewed by two people; the original Xenoposeidon description was reviewed by three; the same was true of my reassessment of Xenoposeidon as a rebbachisaur, and there may have been one or two more that escape me at the moment. But I definitely can’t think of any papers that have been under five sets of eyes apart from this one in Qeios.

Now I am not at all saying that all five of the reviews on this paper are as comprehensive and detailed as a typical solicited peer review at a traditional journal. Some of them have detailed observations; others are much more cursory. But they all have things to say — which I will return to in my third point.

Second, Qeios has further decoupled the functions of peer review. Traditional peer review combines three rather separate functions: A, Checking that the science is sound before publishing it; B, assessing whether it’s a good fit for the journal (often meaning whether it’s sexy enough); and C, helping the authors to improve the work. When PLOS ONE introduced correctness-only peer-review, they discarded B entirely, reasoning correctly that no-one knows which papers will prove influential[1]. Qeios goes further by also inverting A. By publishing before the peer reviews are in (or indeed solicited), it takes away the gatekeeper role of the reviewers, leaving them with only function C, helping the authors to improve the work. Which means it’s no surprise that …

Third, all five reviews have been constructive. As Matt has written elsewhere, “There’s no way to sugar-coat this: getting reviews back usually feels like getting kicked in the gut”. This is true, and we both have a disgraceful record of allowing harshly-reviewed projects to sit fallow for far too long before doing the hard work of addressing the points made by the reviewers and resubmitting[2].

The contrast with the reviews from Qeios has been striking. Each one has sent me scampering back to the manuscript, keen to make (most of) the suggested changes — hence the three revised versions that I’ve posted in the last fortnight. I think there are at least two reasons for this, a big one and a small one.

• The big reason, I think, is that the reviewers know their only role is to improve the paper. Well, that’s not quite true: they also have some influence over its evaluation, both in what they write and in assigning a 1-to-5 star score. But they know when they’re writing their reviews that whatever happens, they won’t block publication. This means, firstly, that there is no point in their writing something like “This paper should not be published until the authors do X”; but equally importantly, I think it puts reviewers in a different and more constructive mindset. They feel themselves to be allies of the authors rather than (as can happen) adversaries.
• The smaller reason is it’s easier to deal with one review at a time. I understand why journals solicit multiple reviews: so the handling editor can consider them all in reaching a decision. I understand why the authors get all the reviews back at once. But that process can’t help but be discouraging: because, once the decision has been made, they’re all on hand and there’s no point in stringing them out. One at a time may not be better, exactly; but it’s emotionally easier.

Is this all upside? Well, it’s too early to say. We’ve only done this once. The experience has certainly been more pleasant — and, crucially, much more efficient — than the traditional publishing lifecycle. But I’m aware of at least two potential drawbacks:

First, the publish-first lifecycle could be exploited by cranks. If the willingness to undergo peer-review is the mark of seriousness in a researcher — and if non-serious researchers are unwilling to face that gauntlet — then a venue that lets you make an end-run around peer-review is an obvious loophole. How serious a danger is this? Only time will tell, but I am inclined to think maybe not too serious. Bad papers on a site like Qeios will attract negative reviews and low scores, especially if they start to get noticed in the mainsteam media. They won’t be seen as having the stamp of having passed peer-review; rather, they will be branded with having publicly failed peer-review.

Second, it’s still not clear where reviewers will come from. We wrote about this problem in some detail last month, and although it’s worked out really well for our present paper, that’s no guarantee that it will always work out this well. We know that Qeios itself approached at least one reviewer to solicit their comments: that’s great, and if they can keep doing this then it will certainly help. But it probably won’t scale, so either a different reviewing culture will need to develop, or we will need people who — perhaps only on an informal basis — take it on themselves to solicit reviews from others. We’re interested to see how this develops.

Anyway, Matt and I have found our first Qeios experience really positive. We’ve come out of it with what I think is a good paper, relatively painlessly, and with much less friction than the usual process. I hope that some of you will try it, too. To help get the process rolling, I personally undertake to review any Qeios article posted by an SV-POW! reader. Just leave a comment here to let me know about your article when it’s up.

Notes

[1] “No-one knows which papers will prove influential”. As purely anecdotal evidence for this claim: when I wrote “Sauropod dinosaur research: a historical review” for the Geological Society volume Dinosaurs: A Historical Perspective, I thought it might become a citation monster. It’s done OK, but only OK. Conversely, it never occurred to me that “Head and neck posture in sauropod dinosaurs inferred from extant animals” would be of more than specialist interest, but it’s turned out to be my most cited paper. I bet most researchers can tell similar stories.

[2] One example: my 2015 preprint on the incompleteness of sauropod necks was submitted for publication in October 2015, and the reviews[3] came back that same month. Five and a half years later, I am only now working on the revision and resubmission. If you want other examples, we got ’em. I am not proud of this.

[3] I referred above to “harsh reviews” but in fact the reviews for this paper were not harsh; they were hard, but 100% fair, and I found myself agreeing with about 90% of the criticisms. That has certainly not been true of all the reviews I have found disheartening!