Notes from our Bibliographer – July 2021

On Open Peer Review in the IsisCB Pandemics Special Issue

by Stephen P. Weldon

This month I want to talk about how publishing the annual Isis Bibliography has opened doors to a relatively new form of academic essay production that is still experimental for most of us: namely, open peer review. It is a practice that is becoming more widely accepted in the sciences, so I think it is worth considering whether and when this form of review might be relevant for history of science.

When HSS first gave me the go-ahead to publish peer-reviewed bibliographical essays in 2018, the technical capabilities that I already had at my disposal gave me a lot of flexibility in the way that I produced them. The peer reviews of the first two bibliographic essays were managed in the normal way. In both cases, the Isis journal editor at the time (first Floris Cohen and then Matt Lavine) commissioned the reviews for me.

When it came to the special issue on pandemics, however, it quickly became evident that a different approach was in order. Right from the outset, my co-editor Neeraja Sankaran and I decided that we wanted rapid publication of the essays online. The exigencies of the COVID-19 pandemic made it clear that teachers and researchers might be wanting these bibliographies right away. So we took a page from the handbook of the COVID researchers themselves who were working collaboratively all over the world, sharing results as they wrote them up, many of them using public peer review. I turned to the Wellcome Open Research platform as a model because it seemed especially straightforward.

In open peer review, all versions of an essay and all reviews of essays are available for any reader to access. In addition to rapid publication, the open review model is ideal for highlighting multiple perspectives. As the essays work their way from first submission through final revision, readers can watch the essays develop and explore the scholarly conversation that ensues. Reading the review reports and author responses, one learns a lot more about the field than reading the essays alone. The final printed volume will not contain the peer reviews, but all of the documents will be preserved so that one can always go back to read them.

There are, of course, potential downsides to this practice. Initially, when we talked about it with our advisory board, some of the board members worried that publishing peer-reviews would not work because they would be either professionally damaging—because the criticism might undermine a scholar’s work—or worthless—because reviewers would not be honest about their criticisms. Indeed, we wondered whether anyone—authors or reviewers—would take us up on this offer? And if they did, would nasty fights emerge? With the help of our advisory board, we worked through the potential pitfalls and come up with language that we believe encourages transparent and productive scholarly exchange.

Here’s what we say to reviewers:

Our goal is to have a constructive and open review process that will help the authors develop a strong bibliographical essay that other researchers can use to better understand the literature on this topic. Primarily, we need a public comment on the article that will help the authors strengthen it for publication.

As it turned out, although there was some initial hesitation, over twenty authors and roughly twice as many reviewers took us up on the deal, some of them despite anticipating contention. Along the way, we realized that some discussion had to be done privately over email between editors, authors, and reviewers. Not everything could or should be part of the public record. Indeed, we explain this to reviewers as well when we tell them “You may also include, in addition, a more detailed report directed only to the authors if there are things you would like to indicate that you would prefer not be part of a public commentary.” On the whole, however, we have published nearly everything that we’ve been sent and have encouraged authors and reviewers to find ways to present their criticisms in constructive ways.

The bibliographic essay format seems to me to be a perfect venue for this kind of peer review. These essays are short (with a few exceptions), and their explanatory nature gives readers a sense of the literature. Some of the essays highlight historiographic differences, and serious disagreements do sometimes occur. By allowing those disagreements to reach the public, however, people new to these areas can better understand what the intellectual stakes are. While open review might not be suitable for all kinds of publication, I can say from my experience with this issue so far that it shows great promise.

More from the July 2021 Newsletter