Research Ethics

Science’s Uphill Journey Out of Its Credibility Crisis

November 14, 2017 2345

Science’s credibility crisis is making headlines once more thanks to a paper from John P. A. Ioannidis and co-authors. Ioannidis, an expert in statistics, medicine and health policy at Stanford University, has done more than anyone else to ring the alarm bells on science’s quality control problems: scientific results are published which other researchers cannot reproduce.

When the crisis erupted in the media in 2013 The Economist devoted it’s cover to “Wrong Science.” Ionannidis’s work was an important part of the background material for the piece.

In previous papers Ioannidis had mapped the troubles of fields such as pre-clinical and clinical medical studies; commenting how, under market pressure, clinical medicine has been transformed to finance-based medicine.

The Conversation logo

This article by Andrea Saltelli originally appeared at The Conversation, a Social Science Space partner site, under the title “Science’s credibility crisis: why it will get worse before it can get better”

In this new work he and co-authors target empirical economics research. They conclude that the field is diseased, with one fifth of the subfields investigated showing a 90 percent incidence of under-powered studies – a good indicator of low-quality research – and a widespread bias in favor of positive effects.

The field of psychology had gone through a similar ordeal. Brian Nosek, professor of psychology at the University of Virginia and his co-workers ran a replication analysis of 100 high-profile psychology studies and reported that only about one third of the studies could be replicated.

Several other instances of bad science have gained attention in the media.
The problems in “priming research,” relevant to marketing and advertising, prompted Nobel Prize winner Daniel Kahneman to issue a publicized statement of concern about the wave of failed replication.

And a study on “power poses”, which claimed that body posture influences a person’s hormones level and “feelings of power” went first viral on TED when it was published – then again when its replication failed.

We are observing two new phenomena. On the one hand doubt is shed on the quality of entire scientific fields or sub-fields. On the other this doubt is played out in the open, in the media and blogosphere.

Fixes
In his newest work Ioannidis sets out a list of remedies that science needs to adopt urgently. These include fostering a culture of replication, data sharing and more collaborative works that pool together larger data sets; along with pre-specification of the protocol including model specifications and the analyses to be conducted.

Ioannidis has previously proposed additional remedies to “fix” science, as have other investigators. The list includes better statistical methods and better teaching of statistics as well as measures to restore the right system of incentives at all stages of the scientific production system – from peer review to academic careers.

Important work is already being done by committed individuals and communities, among them Nosek’s Reproducibility Project, Ioannidis’ Meta-research innovation centre, Ben Goldacre’s alltrials.net and the activities of Retraction Watch. These initiatives – which attracted private funding – are necessary and timely.

But what are the chances that these remedies will work? Will this crisis be solved any time soon?

Methods, incentives and introspection
Ioannidis and co-authors are aware of the interplay between methods and incentives. For example, they say they’d refrain from suggesting that underpowered studies go unpublished, “as such a strategy would put pressure on investigators to report unrealistic and inflated power estimates based on spurious assumptions”.

This is a crucial point. Better practices will only be adopted if new incentives gain traction. In turn the incentives will have traction only if they address the right set of science’s problems and contradictions.

Ethics is a crucial issue in this respect. And here is where research effort is lacking. The broader field of economics is aware of its ethical problems after Paul Romer – now chief economist of the World Bank – coined the new term “Mathiness,” to signify the use of mathematics to veil normative premises. Yet there seem to be some hesitation to join the dots from the methodology to the ethos of the discipline, or of science overall.

The book Science on the Verge has proposed an analysis of the root causes of the crisis, including its neglected ethical dimension. The formulation of remedial measures depends on understanding what happened to science and how this reflects on its social role, including when science feeds into evidence based policy.

These analyses are indebted to philosophers Silvio O. Funtowicz and Jerome R. Ravetz, who spent several decades studying the science’s quality control arrangements and how quality and uncertainty impacted the use of science for policy.

Ravetz’s book Scientific knowledge and its social problems published in 1971 predicted several relevant features of the present crisis.

For Ravetz it is possible for a field to be diseased, so that shoddy work is routinely produced and accepted. Yet, he notes, it will be far from easy to come to accept the existence of such a condition – and even more difficult to reform it.

Reforming a diseased field or arresting the incipient decline of another will be delicate tasks, adds Ravetz, which calls for a

sense of integrity, and a commitment to good work, among a significant section of the members of the field; and committed leaders with scientific ability and political skill. No quantity of published research reports, nor even an apparatus of institutional structures, can do anything to maintain or restore the health of a field in the absence of this essential ethical element operating through the interpersonal channel of communication.

Ravetz emphasises the loss of this essential ethical element. In later works he notes that the new social and ethical conditions of science are reflected in a set of “emerging contradictions.” These concern the cognitive dissonance between the official image of science as enlightened, egalitarian, protective and virtuous, against the current realities of scientific dogmatism, elitism and corruption; of science serving corporate interests and practices; of science used as an ersatz religion.

Echoes of Ravetz’s analysis can be found in many recent works, such as on the commodification of science, or on the present problems with trust in expertise.

A call to arms?
Ioannidis and co-authors are careful to stress the importance of a multidisciplinary approach, as both troubles and solutions may spill over from one discipline to the other. This would perhaps be a call to the arms for social scientists in general – and for those who study science itself – to tackle the crisis as a priority.

Here we clash with another of science’s contradictions: at this point in time, to study science as a scholar would mean to criticise its mainstream image and role. We do not see this happening any time soon. Because of the scars of “science wars” – whose spectre is periodically resuscitated – social scientists are wary of being seen as attacking science, or worse helping US President Donald Trump.

Scientists overall wish to use their moral authority and association with Enlightenment values, as seen in the recent marches for science.

The ConversationIf these contradictions are real, then we are condemned to see the present crisis becoming worse before it can become better.


Andrea Saltelli is an adjunct professor at the Centre for the Study of the Sciences and the Humanities at the University of Bergen. He has worked on physical chemistry, environmental sciences, applied statistics, impact assessment and science for policy. His main disciplinary focus is on sensitivity analysis of model output, a discipline where statistical tools are used to interpret the output from mathematical or computational models, and on sensitivity auditing, an extension of sensitivity analysis to the entire evidence-generating process in a policy context. At present he is in at the European Centre for Governance in Complexity, a joint undertaking of the Centre for the Study of the Sciences and the Humanities - University of Bergen, and of the Institut de Ciència i Tecnologia Ambientals -Universitat Autonoma de Barcelona .

View all posts by Andrea Saltelli

Related Articles

The Importance of Using Proper Research Citations to Encourage Trustworthy News Reporting
Impact
February 26, 2024

The Importance of Using Proper Research Citations to Encourage Trustworthy News Reporting

Read Now
Revolutionizing Management Research with Immersive Research Methods
Business and Management INK
February 26, 2024

Revolutionizing Management Research with Immersive Research Methods

Read Now
A Behavioral Scientist’s Take on the Dangers of Self-Censorship in Science
Interview
February 14, 2024

A Behavioral Scientist’s Take on the Dangers of Self-Censorship in Science

Read Now
Using Forensic Anthropology to Identify the Unknown Dead
Infrastructure
January 29, 2024

Using Forensic Anthropology to Identify the Unknown Dead

Read Now
Connecting Legislators and Researchers, Leads to Policies Based on Scientific Evidence

Connecting Legislators and Researchers, Leads to Policies Based on Scientific Evidence

The author’s team is developing ways to connect policymakers with university-based researchers – and studying what happens when these academics become the trusted sources, rather than those with special interests who stand to gain financially from various initiatives.

Read Now
The Risks Of Using Research-Based Evidence In Policymaking

The Risks Of Using Research-Based Evidence In Policymaking

With research-based evidence increasingly being seen in policy, we should acknowledge that there are risks that the research or ‘evidence’ used isn’t suitable or can be accidentally misused for a variety of reasons. 

Read Now
Our Academic-Industry ‘Research Sprints’ Can Solve Problems in 30 Days

Our Academic-Industry ‘Research Sprints’ Can Solve Problems in 30 Days

Inspired by ‘design sprints’ a Google where projects could create a prototype in five days, the authors started doing ‘research sprints’ in 2015.

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments