Research

Using Prediction Markets to Forecast Replication Across Fields Research
An artist's conception of Human-Machine collaborative design

Using Prediction Markets to Forecast Replication Across Fields

August 27, 2020 1140

The U.S. Department of Defense is trying to find a way to assess and improve the credibility of social and behavioral science research using algorithms. There’s lots to unpack in that statement, but a key question is can machines really determine the credibility of research, ideally without conducting actual replications?

Replication is a key practice in scientific research, playing a role in controlling the impact of sampling error, questionable research practices, publication bias and fraud. Efforts to replicate studies are supposed to help establish credibility within a field (although the bravery of re-testing foundational findings can sometimes create a crisis mentality when too many fail to measure up). As the research community can offer its take about outcomes, we ask again, could those outcomes be forecast formally without actually conducting replications?

DARPA, the Pentagon’s Defense Advanced Research Projects Agency, funded the ‘Systematizing Confidence in Open Research and Evidence’ Program to answer these questions. SCORE aims to generate confidence scores for research claims from empirical studies in the social and behavioral sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication, helping DARPA pick winners for funding and application in the field.

In a paper published by Royal Society Open Science, a team of researchers ask a more detailed question of the process, “Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme.” Authors Michael Gordon, Domenico Viganola, Michael Bishop, Yiling Chen, Anna Dreber, Brandon Goldfedder, Felix Holzmeister, Magnus Johannesson, Yang Liu, Charles Twardy, Juntao Wang, and Thomas Pfeiffer used prediction markets and surveys of the research community to forecast replication outcomes for material of interest to DARPA. This was the authors’s second look at using prediction markets and surveys to predict replication; their initial proof-of-principle lead them to elicit information on a large set of research claims with only a few actually being replicated.

Most of the participants making predictions were drawn from academia, from undergraduate to emeritus professor roles, although the majority were in early or mid-career. Their disciplines fell into six groupings: economics, political science, psychology, education, sociology and criminology, and marketing, management and related areas, and respondents tended to focus their predictions on their own fields.

The paper reports that the participants expect replication rates to differ between fields, with the highest replication rate in economics and the lowest in psychology and education. Participants interested in economics were more optimistic about the replication rates in economics than those not in economics. No evidence was found for this effect with other topics. Moreover, participants who stated that they have been involved in a replication study before on average forecast a lower overall replication rate.

The authors note that participants expected replication rates to increase over time. “One plausible explanation might be that participants expect recent methodological changes in the social and behavioural sciences to have a positive impact on replication rates,” according to the paper. “This is also in line with an increased creation and use of study registries during this time period in the social and behavioural science.”

The forecasts presented in this study focus on field-specific and time-specific replication rates rather than the probability of replication for individual claims. Explicit forecasts for overall replication rates might be more reliable than what can be inferred from forecasts on individual replications.

See the original article here.

.

Kenalyn Ang is the social science communications intern with SAGE Publishing. She is also a communications student at the USC Annenberg School. Her research focuses on consumer behavior, identity-making and representation through the written word, and creative sensory marketing and branding strategies.

View all posts by Kenalyn Ang

Related Articles

Megan Stevenson on Why Interventions in the Criminal Justice System Don’t Work
Social Science Bites
July 1, 2024

Megan Stevenson on Why Interventions in the Criminal Justice System Don’t Work

Read Now
Pandemic Nemesis: Illich reconsidered
News
June 14, 2024

Pandemic Nemesis: Illich reconsidered

Read Now
How ‘Dad Jokes’ Help Children Learn How To Handle Embarrassment
Insights
June 14, 2024

How ‘Dad Jokes’ Help Children Learn How To Handle Embarrassment

Read Now
How Social Science Can Hurt Those It Loves
Ethics
June 4, 2024

How Social Science Can Hurt Those It Loves

Read Now
Digital Scholarly Records are Facing New Risks

Digital Scholarly Records are Facing New Risks

Drawing on a study of Crossref DOI data, Martin Eve finds evidence to suggest that the current standard of digital preservation could fall worryingly short of ensuring persistent accurate record of scholarly works.

Read Now
Analyzing the Impact: Social Media and Mental Health 

Analyzing the Impact: Social Media and Mental Health 

The social and behavioral sciences supply evidence-based research that enables us to make sense of the shifting online landscape pertaining to mental health. We’ll explore three freely accessible articles (listed below) that give us a fuller picture on how TikTok, Instagram, Snapchat, and online forums affect mental health. 

Read Now
New Fellowship for Community-Led Development Research of Latin America and the Caribbean Now Open

New Fellowship for Community-Led Development Research of Latin America and the Caribbean Now Open

Thanks to a collaboration between the Inter-American Foundation (IAF) and the Social Science Research Council (SSRC), applications are now being accepted for […]

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments