Date created: | Last Updated:

Category: Project

Description: The stereotype threat literature has become one of the latest in behavioural research to be accused of publication bias. By simulating datasets based on this literature, we examine how using different methods of statistical analysis affect the development of a field of research. Specifically, we consider how different analysis techniques can result in certainty or uncertainty about the true presence of an effect in a population. We simulated 30,000 datasets in total and compared four different analyses including commonly used frequentist methods (ANOVA and a generalized linear mixed model), as well as more novel Bayesian methods. We found that using posterior passing, a Bayesian approach in which past experiments inform subsequent analyses, allowed the true effect in the population to be found with higher certainty and accuracy than all other analysis types. We conclude that different statistical methods have important effects upon the ability of a literature to reliably come to accurate conclusions, in particular we suggest that using informative priors could help researchers to be more certain about the presence of a true effect in a population. We suggest that the use of informative priors better reflects the cumulative nature of scientific research than the current norm of null hypothesis significance testing.