Peer Review in the Google Age

Peer review in science has been the subject of significant controversy. Peggy Dominy has links to some recent articles discussing the matter. Dave Bradley points out that some journals are moving towards removing the anonymity of reviewers.

In a world where finding information is laborious, the knowledge that a document is from a "“trusted source" is valuable. I remember spending tremendous amounts of time as a graduate student in the early 90's using the Chemical Abstract books looking for articles about my research. An article from the Journal of Organic Chemistry was likely to have a reliable experimental section and it was worth my time to physically locate that article and photocopy it. Other sources might not have warranted the effort.

But the situation is different today. The only reason I have for visiting the library now is to see my librarian friends. The fact that I can download articles from my office or my home has made things easier. But just as importantly, my ability to quickly filter through databases using flexible queries saves a tremendous amount time. The source of the information is really not as important as my ability to find it online immediately.

The idea that peer review is useful to "authenticate" research has always seemed a bit strange to me. After all, the targeted audience for most scientific articles consists of (by definition) peers of the author. What makes the reviewers selected by an editor any more capable of validating an article than the targeted audience?

Of course reviewers do not repeat the experiments in an article and cannot check for fraud. This can only be determined over time, after other researchers have had a chance to try to use the reported techniques. Once in a while a big scientific fraud case makes the news and that would happen with or without peer review.

Sometimes it is clear that reviewers do not even read the articles, as was made evident from the green light given to a computer generated document of random jargon.

From an editor's perspective, peer review is a cost-effective way to maintain the quality and focus of a journal. And certainly for the well-known journals in a given field, it is essential. However in an age when anyone can start an online "journal" and select reviewers that remain anonymous, the term "peer review" is not a very good indicator of quality. Unfortunately it is still used a kind of gold standard in academic promotion and tenure when counting publications.

I think the real issue is that we have to separate the problem of efficiently communicating scientific information from the problem of convincing a committee of the impact of a faculty member's scholarship. And when doing open science, the first concern is the communication of the information. Heather Morrison, Peter Suber and others are also commenting on this. That's why we're blogging our entire undigested laboratory notebook as close as we can to real time.

When I was looking a synthesis of a key compound for our malaria project, I was not interested in peer review. Others who are thinking about repeating our experiments will probably not care either.

This does not prevent the subsequent write-up of a peer reviewed article summarizing the experiments, just like talking about experiments in a conference usually does not prevent publication in most journals. There is plenty of room for both types of communication.

First disclose, then discuss and finally convince when necessary.