International Debate

A Nobel Laureate Explains Peer Review’s Ins and Outs

January 29, 2016

Peer review cycle

(Image: By Nikolaus Kriegeskorte [CC BY 3.0], via Wikimedia Commons)

If you are at all familiar with the operation of the Intergovernmental Panel on Climate Change (IPCC) you will know that, while the various authors are (unpaid) professionals of one sort or another with their own research programs, the IPCC itself neither commissions nor does research. Its conclusions are based solely on the information available in the published peer-reviewed scientific literature. What does that mean?

The Conversation logo

This article by Peter C. Doherty originally appeared at The Conversation, a Social Science Space partner site, under the title “Explainer: the ins and outs of peer review”

Every research paper that appears in print and/or online will have been read and critiqued (generally with complete anonymity) by at least two scientific peers. This mechanism was first put in place by the Royal Society in the 17th century, and has endured.

Sometimes, if the reviewers are at odds, a third or even fourth opinion will be sought. Though the tradition with law journals, for example, may be that the process is double blind – with neither the reviewers nor the authors being identified – the single-blind tradition in science means we generally know who wrote the paper, but can only speculate about the critics of our wonderful manuscript.

That may sound onerous but, in practice, it’s not hard to get almost anything published at some level in what’s broadly styled as the peer-reviewed scientific literature, especially if it is well written and gives the appearance of having been done properly.

One checkpoint most editors will look for is that the paper comes from a university or research institute with some substance and that the author is not using, say, a Gmail account. That doesn’t mean that the article should be immediately rejected, but it provides some measure of upfront quality evaluation.

Academic employers have standards and, in a formal sense, most research grants are awarded to the institution, not to the individual researcher.

The list
What do I mean by “published at some level”? There’s an A list of top journals – for instance, those with the word “nature” somewhere in the title – down to an E list or beyond to which researchers can drop if they are desperate to have an article appear in print and it has been rejected as it slid down the prestige ladder.

The leading journals, like Nature and Science, are read by a broad range of people from the physical to the biological sciences, while Nature Immunology and Nature Geoscience are obviously more specialized though of high stature.

In clinical medicine, the New England Journal of Medicine, The Lancet and the Journal of Clinical Investigation are very prestigious, while learned society publications like The Journal of Immunology and the Journal of Virology are thought of as “journals of record” for sound science within their particular specialty – or trade, if you like. Some of my most-cited papers are in these trade journals.

And there are journals for climate change sceptics. Editors of that mindset will, for instance, select appropriately friendly reviewers. Otherwise, the intervention of a serious scientist sceptic who makes a substantial and well-supported case will generally be welcomed by one or other leading journal.

The reasons are obvious. Informed critique and the debate that results are illuminating and interesting to read. Those of us who are outsiders both learn from, and are entertained by, reading a lively discussion. There’s nothing better than a good fight between intelligent protagonists! Then, nobody – and that includes journal editors – wants to persist with an erroneous view, so the alternatives must be aired.

Up for review
What of the anonymous, unpaid reviewers who work hard and do their best? Most are under a lot of pressure for all sorts of reasons, and they are likely to make the biggest effort with an article submitted to a high-profile journal.


Drawn from

This is an edited extract from The Knowledge Wars by Peter Doherty (RRP A$29.99, eBook A$19.99).


With a demanding and detailed critique for a paper that does seem to have basic merit, the back and forth between authors, editors and reviewers asking for changes and/or new experiments can take a year or more. A problem with such a protracted process can be that, through a mix of exhaustion and dilution, what was a clear message will be largely obscured by the time the article appears.

Peer review should also be seen for what it is: an agreement between the editors, reviewers and authors that the article is now fit for general consumption by a quality audience, not a validation of infallibility, a concept that doesn’t exist in science anyway!

What publication following peer review actually says about a piece of work is that, so far as anyone who has been asked to critique it can detect, what looks to be an interesting study has been done using appropriate techniques, and the data looks to be both valid and to support the conclusions that are reached.

Beyond that, the reviewers should ensure that the methods used have been presented with sufficient clarity so that the study can be repeated, or at least understood, by others.

First step

Peer-reviewed publication is, in fact, just the first step in the validation of any important research finding. It’s not an absolute; just, when done properly, the best evaluation that can be made without independent verification. What matters is that the findings and conclusions hold up as the field moves forward and that the work in question played a part in achieving a useful synthesis.

Even better, the new knowledge that appears first in a research paper is truly validated when it has the power to provide, or to facilitate, a practicable solution to a real problem.

Even so, trying to stay both topical and ahead of the game does carry some inherent risks for journal editors. Many of the fraudulent papers that are eventually outed in, particularly, the biomedical sciences appear in what are generally thought of as top journals.

The reason is obvious. Editors are prepared to chance their arm to publish novel, high-impact findings. Sometimes the review process is at fault, but more often the fraud is carefully hidden.

Even so, though bad (or dishonest) science does not usually have any real long-term negative effect within science, it can cause immense damage if wrong ideas about subjects like childhood vaccination become established in the public consciousness. That’s why it is so important that reviewers and editors do their best to see that potentially emotive, but erroneous, material does not seep through the process. The Conversation


5 1 vote
Article Rating

Peter Doherty, patron and namesake of the Peter Doherty Institute for Infection and Immunity, shared the Nobel Prize in Physiology or Medicine in 1996 with Swiss colleague Rolf Zinkernagel, for their discovery of how the immune system recognizes virus-infected cells. He was Australian of the Year in 1997, and has since been commuting between St Jude Children’s Research Hospital in Memphis and the Department of Microbiology and Immunology at the University of Melbourne. His research is mainly in the area of defense against viruses.

View all posts by Peter C. Doherty

Related Articles

Kohrra on Netflix – Policing and Everyday Life in Contemporary India
International Debate
August 14, 2023

Kohrra on Netflix – Policing and Everyday Life in Contemporary India

Read Now
The UK Pandemic Inquiry – Missing the Point?
International Debate
August 9, 2023

The UK Pandemic Inquiry – Missing the Point?

Read Now
NSF Guidelines Amidst Foreign Threats To Research Security
Investment
July 26, 2023

NSF Guidelines Amidst Foreign Threats To Research Security

Read Now
History Offers Way Forward for Ukrainian Scientific System
Infrastructure
July 12, 2023

History Offers Way Forward for Ukrainian Scientific System

Read Now
Carsten de Dreu on Why People Fight

Carsten de Dreu on Why People Fight

Trained as a social psychologist, Leiden University social psychologist Carsten de Dreu uses behavioral science, history, economics, archaeology, primatology and biology, among other disciplines to study the basis of conflict and cooperation among humans.

Read Now
Report: Latest Academic Freedom Index Sees Global Declines

Report: Latest Academic Freedom Index Sees Global Declines

The latest update of the global Academic Freedom Index finds improvements in only five countries

Read Now
Watch! Methodspace Roundtables Examine Threats To Intellectual And Academic Freedom

Watch! Methodspace Roundtables Examine Threats To Intellectual And Academic Freedom

Janet Salmons, the research community director of our sister site, Sage Methodspace, coordinated a series of research roundtables to discuss the obstacles facing academic freedom and how to navigate them.

Read Now
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments