Research Ethics

Can We Encourage Public Self-Correction in the Scientific Record?

April 13, 2021 1881
Paper folder for errors
(Photo: Alpha Stock ImagesCC BY-SA 3.0)

The replication crisis has shaken our understanding of what rigorous research looks like in the social sciences. Practices that were once common—such as small samples, or extensive re-analysis of data until a significant effect is achieved—are now frowned upon. But, what does this mean for individual researchers who are confronted with flaws in their own research record? In psychology, it seems at least, many authors have decided to opt for silence. New studies may be conducted according to more rigorous standards, but what happened in the past stays in the past.

LSE-impact-blog-logo
This article by Julia Rohrer originally appeared on the LSE Impact of Social Sciences Blog as “A self-correcting fallacy – Why don’t researchers correct their own errors in the scientific record?” and is reposted under the Creative Commons license (CC BY 3.0).

This is not per se an obstacle to scientific self-correction, which can occur on the collective level even if authors remain silent about issues with past investigations, and even if they stubbornly cling to questionable prior findings. However, as we argue in our recent paper, scientific self-correction could be much more efficient if authors were willing to openly discuss problems with their past studies. For example, if somebody disclosed that a published finding was cherry-picked from a large number of statistical comparisons, this could inform others who were planning to build on said study, or who planned to replicate it. But are psychologists willing to disclose such information?

We launched a website on which we invited researchers to submit a statement describing how they lost confidence in one of their own published findings. We asked for cases in which the central result of an article was called into question, and in which there were theoretical or methodological problems for which the submitter took responsibility. The public reaction to our initiative was amazing: almost everybody agreed that such a project was urgently needed, and there was some early media coverage.

At the same time, barely anybody submitted an actual loss-of-confidence statement. Statements trickled in very slowly, and after repeated solicitation, we were able to collect 13 over the course of more than a year. The content of these statements was quite varied, with some common themes: miss-specified models, invalid inferences, and, in more than half of the statements, some form of p-hacking. All statements can be found in the published article.

READ MORE

This post draws on the author’s co-authored article, Putting the Self in Self-Correction: Findings From the Loss-of-Confidence Project, published in Perspectives on Psychological Science

In surveys, researchers routinely reveal how widespread certain questionable research practices were. So why did we receive so few statements? We conducted an anonymous follow-up survey, querying researchers across fields for their experiences. Our sample was not representative and so we cannot provide any precise estimates, but we still believe that the survey results shed some additional light on the culture of self-correction. Almost half of the respondents had lost confidence in a previously published finding, and of these, about half believed that this was due to a mistake or shortcoming in judgment on the part of themselves, the researchers.

The overwhelming majority reported that their loss of confidence was not a matter of public record in any way, and the reasons for this were diverse. More than half of the respondents were insufficiently sure about the subject matter to proceed in any form; almost half believed that public disclosure was unnecessary because their finding hadn’t attracted much attention; many were concerned about their coauthors’ feelings or didn’t know an appropriate venue. Overall, it seems like losses of confidence occur frequently, but are rarely reported due to uncertainties regarding both the substantive matter and the best way to proceed forward.

What could we do to encourage public self-correction? Currently, such behavior is actively discouraged by academic incentive structures. Time spent on correcting past work is time that cannot be spent on creating new work, and researchers are frequently evaluated based on the quantity of their output. Assuming that we cannot change much about the focus on quantity, it may thus make sense to establish critical commentaries on one’s own work as an article category. But maybe it is also possible to shift the focus of evaluation from quantity to quality—after all, expectations regarding the quantity of publications in psychology exceed those in other social sciences.

Reputation also plays a role. About a quarter of our survey respondents reported concerns about how a public disclosure of a loss of confidence would be perceived, reflecting the nature of self-correction as a collective action problem, rather than an individual failing. However, worries may be exaggerated. It is, for example, unclear whether self-retractions actually damage researchers’ reputations. Recent high-profile cases of self-correction in psychology have received positive reactions from within the psychological community, and we may try to foster an alternative narrative: scientists make errors, self-correction credibly signals that one cares about the correctness of the scientific record.

Beyond, there are more pragmatic questions that need to be addressed. Journals and publishers have often been reluctant to publish corrections and criticism, sometimes even if the instigator was the original author. There is currently no standardized protocol for what to do if one discovers a major mistake in one’s own published work. Retractions are a standard option but are often associated with the notion of deliberate fraud. Alternative labels have been suggested (“authorial expression of concern”, “voluntary withdrawal”), though in many cases adding to the research record may be the more transparent and productive way forward. The form of such amendments may vary, but they will only be of use if they are directly linked to the original work, be it in established databases (such as PubMed) or directly on the website of the journal. In the end, the static article format may be antithetical to the idea of self-correction, and more dynamic systems incorporating version control (such as the Springer Nature Living Reviews journal series) may be needed to ultimately improve science.

Julia Rohrer is personality psychologist by training and her work covers a broad range of topics, including the effects of birth order, age patterns in personality, and the correlates and determinants of subjective well-being. She recently finished a doctoral degree as a fellow of the International Max Planck Research School on the Life Course and is now a lecturer (Akademische Assistentin) at the Department of Psychology, University of Leipzig.

View all posts by Julia Rohrer

Related Articles

Reflections of a Former Student Body President: ‘Student Government is a Thankless Job’
Insights
July 1, 2024

Reflections of a Former Student Body President: ‘Student Government is a Thankless Job’

Read Now
Megan Stevenson on Why Interventions in the Criminal Justice System Don’t Work
Social Science Bites
July 1, 2024

Megan Stevenson on Why Interventions in the Criminal Justice System Don’t Work

Read Now
Why We’ve Had to Dramatically Shift How We Talk About UK Politics
Insights
June 25, 2024

Why We’ve Had to Dramatically Shift How We Talk About UK Politics

Read Now
How ‘Dad Jokes’ Help Children Learn How To Handle Embarrassment
Insights
June 14, 2024

How ‘Dad Jokes’ Help Children Learn How To Handle Embarrassment

Read Now
How Social Science Can Hurt Those It Loves

How Social Science Can Hurt Those It Loves

David Canter rues the way psychologists and other social scientists too often emasculate important questions by forcing them into the straitjacket of limited scientific methods.

Read Now
Digital Scholarly Records are Facing New Risks

Digital Scholarly Records are Facing New Risks

Drawing on a study of Crossref DOI data, Martin Eve finds evidence to suggest that the current standard of digital preservation could fall worryingly short of ensuring persistent accurate record of scholarly works.

Read Now
Analyzing the Impact: Social Media and Mental Health 

Analyzing the Impact: Social Media and Mental Health 

The social and behavioral sciences supply evidence-based research that enables us to make sense of the shifting online landscape pertaining to mental health. We’ll explore three freely accessible articles (listed below) that give us a fuller picture on how TikTok, Instagram, Snapchat, and online forums affect mental health. 

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments