Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Affiliated institutions: University of Virginia, Center For Open Science

Date created: | Last Updated:

: DOI | ARK

Creating DOI. Please wait...

Create DOI

Category: Project

Description: Being able to replicate scientific findings is crucial for scientific progress. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.

License: CC0 1.0 Universal

This project represents an accepted paper submitted to SocArXiv . Learn more about how to work with paper files. View paper

Wiki

Social Sciences Replication Project This OSF repository accompanies the publication Camerer, C., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen, Y., Forsell, E., Gampa, A., Heikenstein, E., Hummer, L., Imai, T., Isaksson, S., Manfredi, D., Rose, J., Wagenmakers, E., and Wu, H. (2018). E...

Files

Loading files...

Citation

osf.io/pfdyw

Components

  • Materials


    Recent Activity

    Loading logs...

  • Replication of Ackerman et al (2010)


    Recent Activity

    Loading logs...

  • Replication of Derex et al (2013)

    Holzmeister, Ho & Wu

    Recent Activity

    Loading logs...

  • Replication of Gneezy et al (2012)


    Recent Activity

    Loading logs...

  • Replication of Janssen et al (2010)

    Holzmeister, Ho & Wu

    Recent Activity

    Loading logs...

  • Replication of Kovacs et al (2010)

    Holzmeister, Ho & Wu

    Recent Activity

    Loading logs...

  • Replication of Lee and Schwarz (2010)


    Recent Activity

    Loading logs...

  • Replication of Sparrow et al (2011)


    Recent Activity

    Loading logs...

  • Replication of Wilson et al (2014)


    Recent Activity

    Loading logs...

Tags

Recent Activity

Loading logs...

This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.
Accept
×

Start managing your projects on the OSF today.

Free and easy to use, the Open Science Framework supports the entire research lifecycle: planning, execution, reporting, archiving, and discovery.

Create an Account Learn More Hide this message