Beyond p-values: Utilizing Multiple Estimates to Evaluate Evidence

Date created: | Last Updated:


Creating DOI. Please wait...

Create DOI

Category: Project

Description: Null hypothesis significance testing is frequently cited as a threat to the validity and reproducibility of the social sciences. While many individuals suggest we should focus on altering the *p*-value at which we deem an effect significant, we believe this suggestion is short-sighted. Alternative procedures (i.e., Bayesian analyses and Observation Oriented Modeling) can be more powerful and meaningful to our discipline. However, these methodologies are less frequently utilized and are rarely discussed in combination with NHST. Herein, we compare the possible interpretations of three analyses (ANOVA, Bayes Factor, and an Ordinal Pattern Analysis) in various data environments using a simulation study. The simulation generated 20000 unique datasets which varied sample size (*N*s of 10, 30, 100, 500, 1000), and effect sizes (*d*s of 0.10, 0.20, 0.05, 0.80). Through this simulation, we find that changing the threshold at which *p*-values are considered significant has little to no effect on conclusions. Further, we find that evaluating multiple estimates as evidence of an effect can allow for a more robust and nuanced report of findings. These findings suggest the need to redefine evidentiary value and reporting practices.

License: CC-By Attribution 4.0 International

This project represents a preprint. Learn more about how to work with preprint files. View preprint


View the Shiny Application: go here. Notes on how to recreate our work: - To recreate the manuscript, you will need to download the manuscript files folder and run the Rmarkdown within that folder (you can open this file with RStudio). The markdown includes the simulation code that created the files in the output folder. - We suggest starting by searching for "library" within our document to inst...


Loading files...



Recent Activity

Loading logs...


Start managing your projects on the OSF today.

Free and easy to use, the Open Science Framework supports the entire research lifecycle: planning, execution, reporting, archiving, and discovery.

Create an Account Learn More Hide this message