Quality Control and Peer Review

Anyone following the current debate about Open Access scientific publication is well aware that peer review is a big factor in these discussions. For example, see Pedro Beltrao's recent discussion at Science Foo Camp.

I have found that this issue is of particular concern to librarians, who would like to be able to categorize all publications into two neat piles: reliable and unreliable. That makes it convenient to simply ignore anything that is not on the reliable list. Since librarians cannot be experts in every field, they have to rely on other criteria, such as a notice that a document has been peer reviewed.

Unfortunately, it is often assumed that peer reviewed information is reliable. This can be a dangerous assumption.

For example, in our research on the synthesis of anti-malarial compounds, we designed our experiments based on information found in peer-reviewed journal articles. Over time, we found some contradictory information in the literature about the solubility and spectroscopic properties of one of the compounds we were trying to make and this cost us some time.

I think that this was a great lesson for my students about how to use the chemical literature. The only way to confidence in science is to look for redundancy in independent reports. Peer review (from anonymous people who do not repeat experiments in a paper) cannot be used as a short cut to assessing reliability.

Of course if the papers that mislead us had links to the raw data of their experiments we would have been able to spot these errors much more quickly. This is a huge advantage of doing open source science.

With the proliferation of new forms of scholarship, it is no longer effective to teach our students that there is such a thing as reliable literature. There is uncertainty in every information source. We should teach them to find the same information from several different sources and then discuss the confidence we can derive from the aggregate results.

Finding the melting or boiling point of a compound is a great way to make that point. Use a company catalogue, Wikipedia, a textbook and Google and see if there is consensus. For example, the boiling point of ethanol is listed as 78.4 C in Wikipedia, 78.3 C on this website, 78.30 C on this website and 78.5 C here. Based on this we can be reasonably certain that the boiling point is between 78 C- 79 C and that may be all that we need for our application. Finding the accurate value for that first decimal place is going to require a lot more research.

Unfortunately most of these information sources do not reveal the details of the original experiments where the boiling point was determined, which would be extremely helpful in this case. However, before you start bashing Wikipedia and web pages as being inherently unreliable, take a look at the official information compilations that are out there. For example, MSDS (Material Safety Data Sheet) is considered by chemists to be the standard information source for chemical safety. In the course of our research, I was disturbed to find that benzene was listed as an incompatibility for sulfuric acid. This is the very first reaction I cover in the chemistry of aromatics!

I found this to be even more disturbing:

The quality and accuracy of MSDSs varies widely. One recent study showed that of 150 randomly selected MSDSs, information was accurately identified in Health Effects in 37%, in First Aid Procedures in 76%, in Personal Protective Clothing in 47%, and in Occupational Exposure Limits in 47%.

Unfortunately, there does not appear to be a simple way to get to the original experiments where the MSDS information was obtained. At least on Wikipedia you can ask the author directly for the source if it is not listed.

For a nice discussion on the issue of reproducibility in peer reviewed publications see Chris Surridge's post on the PLoS blog.