Achieving Statistical Significance with Covariates and without Transparency

Date created: | Last Updated:

: DOI | ARK

Creating DOI. Please wait...

Create DOI

Category: Project

Description: An important yet understudied area of researcher discretion is the use of covariates in statistical analysis. Researchers choose which covariates to include and their choices affect the size and statistical significance of their estimates. How often are researchers exploiting this discretion to achieve statistical significance unbeknownst to reviewers and readers? We use newly available replication data to shed light on this question, focusing primarily on observational studies. We find that, when researchers disclose a bivariate estimate of their key effect, adding covariates leaves it unchanged on average. When researchers do not, however, covariates frequently help them achieve statistical significance, and they do so by increasing the magnitude of the effect rather than reducing standard errors. Although such adjustments are potentially justifiable---to address suppressor variables---the articles did not justify them. Our findings do not demonstrate wrongdoing, but suggest that, because of a lack of transparency, the opportunities may be rife.

License: CC-By Attribution 4.0 International

Files

Loading files...

Citation

Recent Activity

Loading logs...

This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.
Accept
×

Start managing your projects on the OSF today.

Free and easy to use, the Open Science Framework supports the entire research lifecycle: planning, execution, reporting, archiving, and discovery.

Create an Account Learn More Hide this message