Research

Unknown Unknowns: The War on Null and Negative Results Research
Many times a negative can be very enlightening.

Unknown Unknowns: The War on Null and Negative Results

September 19, 2014 1432

X-ray

Many times a negative can be very enlightening.

There is a lot of pressure on academic researchers. Rightly or wrongly, it is widely held that researchers must publish prodigiously and secure funding consistently to advance their careers and that their research outputs must achieve high ‘impact’ to be of value.

An unintended consequence of this ‘impact culture’ may be a publication bias against research findings whose implications are not immediately obvious, or which may be deemed less novel, interesting, or eye-catching. Some research suggests that this bias may extend to research that yields negative and/or null findings.

Wellcome Trust logo

This post by Jonathon Kram and Adam Dinsmore originally appeared on the Wellcome Trust blog under the title, “Unknown unknowns: Is there a selection bias against null results?” It is reposted here under a Creative Commons Attribution-Non-Commercial 2.0 UK: England & Wales licence.

There are a few definitions of ‘negative finding’ currently in use, but the idea of significance is central to the concept. A negative result fails to reject the null hypothesis –the proposition that no significant effect is present – within the bounds of an experiment and to a level of statistical rigor. This can occur when a treatment is found to have no greater efficacy than placebo, or when no difference is found between two distinct groups of people (e.g. conservatives and liberals) on some variable of interest.

That said, the term is also used to refer to findings of slightly different but related kinds, including:

  • Incremental improvements: Minor refinements to previously reported hypotheses (e.g. where a previous result can only be replicated in certain conditions).
  • Refutations: Results that defy expectation, whether that’s the previous scientific consensus or a researcher’s original intent.

A bias against sharing statistically insignificant results (alongside incremental refinements and refutations) could have detrimental effects to the progression of science and our knowledge. Researchers could waste time pursuing hypotheses which have already been disconfirmed. Clinicians could make treatment decisions based on incomplete evidence. The time and energy of human participants could be wasted in research studies which were never subsequently reported.

Worryingly, there is some evidence that this is already happening.

Hopewell et al (2007) found that research papers which reported statistically significant results were approximately four times as likely to be published as their negative counterparts. The work of Daniele Fanelli (2012) suggests that the proportion of papers publishing negative findings may be decreasing over time; in 2007 just 14 percent of papers indexed by ISI which declared to have tested a hypothesis reported a negative finding, down from 30 percent in 1990. Recently Franco et al (2014) found that negative findings are less likely to be written up for publication than more positive results.

Several attempts have been made to encourage the publication of negative findings. BioMed Central’s Journal for Negative Results in BioMedicine has published articles which either report negative findings or discuss their place in science since 2002, while Open Access publishers PLOS One and PeerJ encourage the submission of any scientific results including negative findings.

Away from conventional publication the online datahub FigShare actively encourages the upload of datasets containing null and negative findings, without requiring that the researchers responsible for them prepare a full manuscript for submission.

As a research funder, we are keen that all research findings be accessible and presented to the world; including those that describe a breakthrough, disprove a theory or demonstrate a null result. Everyone in research has an opinion of what makes “good science”, from the researchers through to the readers of a paper. Researchers, journal editors and peer-reviewers all make value judgments about what’s worth telling the rest of the world about. However, if there are processes in play that inadvertently (or advertently) discriminate against the availability of a certain type of result and its data, we risk needless repetition of research and, potentially worse, proliferation of apparent knowledge that may be wrong.

Through the open access movement and the potential of digital technologies, we have the opportunity to enable accessibility to all the findings of research and reduce research waste. If the studies cited above accurately reflect a bias against negative findings in the biomedical literature, then it is the duty of all stakeholders in the delivery of science and knowledge to engage with the issue and explore ways of correcting it.

There is no benchmark to measure the progress of science against but there are hints that there may be a lot of wasted effort caused by our attitude towards negative findings and their unloved siblings. It is only by building a stronger evidence base around negative findings, and exploring the factors that influence publication bias, that we can explore the severity of the problem and make steps towards improving the health of science.

***

The Wellcome Trust policy position on research involving human participants states that negative results and full disclosure are expected. We are committed to maximising the availability of and value of research data.


Jonathon Kram is a research assistant, and Adam Dinsmore an evaluation officer, working within the Wellcome Trust’s Strategic Planning & Policy Unit. The unit provides analysis, research and advice on policy and evaluation issues relevant to the trust’s mission to achieve extraordinary improvements in human and animal health by supporting the brightest minds in biomedical research.

View all posts by Jonathon Kram and Adam Dinsmore

Related Articles

Digital Scholarly Records are Facing New Risks
Research
May 21, 2024

Digital Scholarly Records are Facing New Risks

Read Now
Analyzing the Impact: Social Media and Mental Health 
Research
May 15, 2024

Analyzing the Impact: Social Media and Mental Health 

Read Now
New Fellowship for Community-Led Development Research of Latin America and the Caribbean Now Open
Academic Funding
May 14, 2024

New Fellowship for Community-Led Development Research of Latin America and the Caribbean Now Open

Read Now
New Opportunity to Support Government Evaluation of Public Participation and Community Engagement Now Open
Featured
April 22, 2024

New Opportunity to Support Government Evaluation of Public Participation and Community Engagement Now Open

Read Now
Three Decades of Rural Health Research and a Bumper Crop of Insights from South Africa

Three Decades of Rural Health Research and a Bumper Crop of Insights from South Africa

A longitudinal research project project covering 31 villages in rural South Africa has led to groundbreaking research in many fields, including genomics, HIV/Aids, cardiovascular conditions and stroke, cognition and aging.

Read Now
Using Translational Research as a Model for Long-Term Impact

Using Translational Research as a Model for Long-Term Impact

Drawing on the findings of a workshop on making translational research design principles the norm for European research, Gabi Lombardo, Jonathan Deer, Anne-Charlotte Fauvel, Vicky Gardner and Lan Murdock discuss the characteristics of translational research, ways of supporting cross disciplinary collaboration, and the challenges and opportunities of adopting translational principles in the social sciences and humanities.

Read Now
Coping with Institutional Complexity and Voids: An Organization Design Perspective for Transnational Interorganizational Projects

Coping with Institutional Complexity and Voids: An Organization Design Perspective for Transnational Interorganizational Projects

Institutional complexity occurs when the structures, interests, and activities of separate but collaborating organizations—often across national and cultural boundaries—are not well aligned. Institutional voids in this context are gaps in function or capability, including skills gaps, lack of an effective regulatory regime, and weak contract-enforcing mechanisms.

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments