International Debate

A Nobel Laureate Explains Peer Review’s Ins and Outs

January 29, 2016 1073

Peer review cycle

(Image: By Nikolaus Kriegeskorte [CC BY 3.0], via Wikimedia Commons)

If you are at all familiar with the operation of the Intergovernmental Panel on Climate Change (IPCC) you will know that, while the various authors are (unpaid) professionals of one sort or another with their own research programs, the IPCC itself neither commissions nor does research. Its conclusions are based solely on the information available in the published peer-reviewed scientific literature. What does that mean?

The Conversation logo

This article by Peter C. Doherty originally appeared at The Conversation, a Social Science Space partner site, under the title “Explainer: the ins and outs of peer review”

Every research paper that appears in print and/or online will have been read and critiqued (generally with complete anonymity) by at least two scientific peers. This mechanism was first put in place by the Royal Society in the 17th century, and has endured.

Sometimes, if the reviewers are at odds, a third or even fourth opinion will be sought. Though the tradition with law journals, for example, may be that the process is double blind – with neither the reviewers nor the authors being identified – the single-blind tradition in science means we generally know who wrote the paper, but can only speculate about the critics of our wonderful manuscript.

That may sound onerous but, in practice, it’s not hard to get almost anything published at some level in what’s broadly styled as the peer-reviewed scientific literature, especially if it is well written and gives the appearance of having been done properly.

One checkpoint most editors will look for is that the paper comes from a university or research institute with some substance and that the author is not using, say, a Gmail account. That doesn’t mean that the article should be immediately rejected, but it provides some measure of upfront quality evaluation.

Academic employers have standards and, in a formal sense, most research grants are awarded to the institution, not to the individual researcher.

The list
What do I mean by “published at some level”? There’s an A list of top journals – for instance, those with the word “nature” somewhere in the title – down to an E list or beyond to which researchers can drop if they are desperate to have an article appear in print and it has been rejected as it slid down the prestige ladder.

The leading journals, like Nature and Science, are read by a broad range of people from the physical to the biological sciences, while Nature Immunology and Nature Geoscience are obviously more specialized though of high stature.

In clinical medicine, the New England Journal of Medicine, The Lancet and the Journal of Clinical Investigation are very prestigious, while learned society publications like The Journal of Immunology and the Journal of Virology are thought of as “journals of record” for sound science within their particular specialty – or trade, if you like. Some of my most-cited papers are in these trade journals.

And there are journals for climate change sceptics. Editors of that mindset will, for instance, select appropriately friendly reviewers. Otherwise, the intervention of a serious scientist sceptic who makes a substantial and well-supported case will generally be welcomed by one or other leading journal.

The reasons are obvious. Informed critique and the debate that results are illuminating and interesting to read. Those of us who are outsiders both learn from, and are entertained by, reading a lively discussion. There’s nothing better than a good fight between intelligent protagonists! Then, nobody – and that includes journal editors – wants to persist with an erroneous view, so the alternatives must be aired.

Up for review
What of the anonymous, unpaid reviewers who work hard and do their best? Most are under a lot of pressure for all sorts of reasons, and they are likely to make the biggest effort with an article submitted to a high-profile journal.

Drawn from

This is an edited extract from The Knowledge Wars by Peter Doherty (RRP A$29.99, eBook A$19.99).

With a demanding and detailed critique for a paper that does seem to have basic merit, the back and forth between authors, editors and reviewers asking for changes and/or new experiments can take a year or more. A problem with such a protracted process can be that, through a mix of exhaustion and dilution, what was a clear message will be largely obscured by the time the article appears.

Peer review should also be seen for what it is: an agreement between the editors, reviewers and authors that the article is now fit for general consumption by a quality audience, not a validation of infallibility, a concept that doesn’t exist in science anyway!

What publication following peer review actually says about a piece of work is that, so far as anyone who has been asked to critique it can detect, what looks to be an interesting study has been done using appropriate techniques, and the data looks to be both valid and to support the conclusions that are reached.

Beyond that, the reviewers should ensure that the methods used have been presented with sufficient clarity so that the study can be repeated, or at least understood, by others.

First step

Peer-reviewed publication is, in fact, just the first step in the validation of any important research finding. It’s not an absolute; just, when done properly, the best evaluation that can be made without independent verification. What matters is that the findings and conclusions hold up as the field moves forward and that the work in question played a part in achieving a useful synthesis.

Even better, the new knowledge that appears first in a research paper is truly validated when it has the power to provide, or to facilitate, a practicable solution to a real problem.

Even so, trying to stay both topical and ahead of the game does carry some inherent risks for journal editors. Many of the fraudulent papers that are eventually outed in, particularly, the biomedical sciences appear in what are generally thought of as top journals.

The reason is obvious. Editors are prepared to chance their arm to publish novel, high-impact findings. Sometimes the review process is at fault, but more often the fraud is carefully hidden.

Even so, though bad (or dishonest) science does not usually have any real long-term negative effect within science, it can cause immense damage if wrong ideas about subjects like childhood vaccination become established in the public consciousness. That’s why it is so important that reviewers and editors do their best to see that potentially emotive, but erroneous, material does not seep through the process. The Conversation

Peter Doherty, patron and namesake of the Peter Doherty Institute for Infection and Immunity, shared the Nobel Prize in Physiology or Medicine in 1996 with Swiss colleague Rolf Zinkernagel, for their discovery of how the immune system recognizes virus-infected cells. He was Australian of the Year in 1997, and has since been commuting between St Jude Children’s Research Hospital in Memphis and the Department of Microbiology and Immunology at the University of Melbourne. His research is mainly in the area of defense against viruses.

View all posts by Peter C. Doherty

Related Articles

The Long Arm of Criminality
April 29, 2024

The Long Arm of Criminality

Read Now
Why Don’t Algorithms Agree With Each Other?
February 21, 2024

Why Don’t Algorithms Agree With Each Other?

Read Now
A Black History Addendum to the American Music Industry
February 6, 2024

A Black History Addendum to the American Music Industry

Read Now
When University Decolonization in Canada Mends Relationships with Indigenous Nations and Lands
Higher Education Reform
January 9, 2024

When University Decolonization in Canada Mends Relationships with Indigenous Nations and Lands

Read Now
Kohrra on Netflix – Policing and Everyday Life in Contemporary India

Kohrra on Netflix – Policing and Everyday Life in Contemporary India

Even Social Science Space bloggers occasionally have downtime when they log in to Netflix and crash out. One of my favourite themes […]

Read Now
The UK Pandemic Inquiry – Missing the Point?

The UK Pandemic Inquiry – Missing the Point?

The post-mortems on national governments’ management of the COVID-19 pandemic are getting under way. Some European countries have completed theirs, with rapid […]

Read Now
NSF Guidelines Amidst Foreign Threats To Research Security

NSF Guidelines Amidst Foreign Threats To Research Security

Reflecting concerns that foreign governments have illicitly obtained research developed by the United States, posing threats to research security and hindering international collaboration, the National Science Foundation released “NSF Guidelines for Research Security Analytics.”

Read Now
5 1 vote
Article Rating
Notify of

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Inline Feedbacks
View all comments