Higher Education Reform

Statistical Association Takes on Use, Abuse of P-values

March 7, 2016 2560

P-value_GraphEven as it insists it’s not really saying anything new, the American Statistical Association Board of Directors has laid down a marker in the debate over what constitutes “statistical significance.”

After months of discussion with a blue-chip panel of experts, the association has laid down six principles about what constitutes good practice when dealing with p-values, which it informally defines as “the probability under a specified statistical model that a statistical summary of the data (for example, the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.” It has been common shorthand to use P < 0.05 as meaning a result is significant. As physiologist Lewis Halsey explained in these pages, “Although P is never proof that there is a difference – scientific studies never prove things, they only provide a degree of evidence for them – studies with low P values are thought to be convincing, and so are not often repeated to be sure the results are correct.”

As Halsey further explained, this has proved an increasingly fraught decision with at least one journal banning the use of p-values in their accepted papers and debates over the replicability of research poking holes in using p-values and other ‘bright-line’ tests.

The principles include statements such as “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold” and “Proper inference requires full reporting and transparency,” statements that are both self-evident and yet striking at the same time.

Today’s statement is the first time the ASA – which describes itself as the world’s largest community of statisticians – has taken a position on a specific matter of statistical practice. “This statement does not seek to resolve all the issues relating to sound statistical practice, nor to settle foundational controversies,” the ASA statement reads. “Rather, the statement articulates in non-technical terms a few select principles that could improve the conduct or interpretation of quantitative science, according to widespread consensus in the statistical community.”

A release from the ASA to the statistics community does an admirable job of summing up what is already a concise statement (the full statement is available for free at The American Statistician website):

“Widespread use of ‘statistical significance’ (generally interpreted as ‘p < 0.05′) as a license for making a claim of a scientific finding (or implied truth) leads to considerable distortion of the scientific process,” says the ASA statement (in part). By putting the authority of the world’s largest community of statisticians behind such a statement, we seek to begin a broad-based discussion of how to more effectively and appropriately use statistical methods as part of the scientific reasoning process.

In short, we envision a new era, in which the broad scientific community recognizes what statisticians have been advocating for many years. In this “post p < .05 era,” the full power of statistical argumentation in all its nuance will be brought to bear to advance science, rather than making decisions simply by reducing complex models and methods to a single number and its relationship to an arbitrary threshold. This new era would be marked by radical change to how editorial decisions are made regarding what is publishable, removing the temptation to inappropriately hunt for statistical significance as a justification for publication. In such an era, every aspect of the investigative process would have its appropriate weight in the ultimate decision about the value of a research contribution.

The message from the ASA included a quick origin story by Ronald L. Wasserstein and Nicole A. Lazar about the statement, demonstrating the seriousness of the concern and the complexity of saying something that was, in itself, significant about the issue. “Though there was disagreement on exactly what the statement should say,” the message on ‘context, process, and purpose’ explained, “there was high agreement that the ASA should be speaking out about these matters.

“Let’s be clear. Nothing in the ASA statement is new. Statisticians and others have been sounding the alarm about these matters for decades, to little avail. We hoped that a statement from the world’s largest professional association of statisticians would open a fresh discussion and draw renewed and vigorous attention to changing the practice of science with regards to the use of statistical inference.”


Related Articles

AI Upskilling Can and Should Empower Business School Faculty
Higher Education Reform
July 10, 2024

AI Upskilling Can and Should Empower Business School Faculty

Read Now
Reflections of a Former Student Body President: ‘Student Government is a Thankless Job’
Insights
July 1, 2024

Reflections of a Former Student Body President: ‘Student Government is a Thankless Job’

Read Now
Pandemic Nemesis: Illich reconsidered
News
June 14, 2024

Pandemic Nemesis: Illich reconsidered

Read Now
How ‘Dad Jokes’ Help Children Learn How To Handle Embarrassment
Insights
June 14, 2024

How ‘Dad Jokes’ Help Children Learn How To Handle Embarrassment

Read Now
Universities Should Reimagine Governance Along Co-Operative Lines

Universities Should Reimagine Governance Along Co-Operative Lines

Instead of adhering to a corporate model based on individual achievement, the authors argue that universities need to shift towards co-operative governance that fosters collaborative approaches to teaching and research

Read Now
Biden Administration Releases ‘Blueprint’ For Using Social and Behavioral Science in Policy

Biden Administration Releases ‘Blueprint’ For Using Social and Behavioral Science in Policy

U.S. President Joseph Biden’s administration has laid down a marker buttressing the use of social and behavioral science in crafting policies for the federal government by releasing a 102-page Blueprint for the Use of Social and Behavioral Science to Advance Evidence-Based Policymaking.

Read Now
Striving for Linguistic Diversity in Scientific Research

Striving for Linguistic Diversity in Scientific Research

Each country has its own unique role to play in promoting greater linguistic diversity in scientific communication.

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments