Date created: | Last Updated:

Category: Project

Description: An important yet understudied area of researcher discretion is the use of covariates in statistical analysis. Researchers choose which covariates to include and their choices affect the size and statistical significance of their estimates. How often are researchers exploiting this discretion to achieve statistical significance unbeknownst to reviewers and readers? We use newly available replication data to shed light on this question, focusing primarily on observational studies. We find that, when researchers disclose a bivariate estimate of their key effect, adding covariates leaves it unchanged on average. When researchers do not, however, covariates frequently help them achieve statistical significance, and they do so by increasing the magnitude of the effect rather than reducing standard errors. Although such adjustments are potentially justifiable---to address suppressor variables---the articles did not justify them. Our findings do not demonstrate wrongdoing, but suggest that, because of a lack of transparency, the opportunities may be rife.