## The Preregistration Revolution ## ---------- Contacts: - David Mellor, david@cos.io, [@EvoMellor][3] - Jennifer Smith, jennifer@cos.io - Ian Sullivan, ian@cos.io ---------- #### Background #### Though scientists value transparency and reproducibility ([Anderson, Martinson, & De Vries, 2007][4]), we are rewarded for novelty and for presenting clean results. These rewards come from publications, grants, hiring, and promotion decisions that present a classic collective action problem. The incentives lead to behaviors that make scientific findings less reproducible than expected. This crisis in reproducibility can be addressed if scientists are rewarded for rigorous methods and best practices, instead of for presenting the most surprising and tidy results. **Increasing transparency improves rigor by allowing expert evaluation of every part of the process of science.** ---------- ### Workshop Outline ### - What are the problems that lead to irreproducible research? (10 min presentation, David) - Instructions for creating a preregistration on the OSF (5 min, Ian) - Group work on making a preregistration on the OSF (20 min, David and Ian roam) - Handout with instructions, suggested responses to five key points in a preregistration: 1) hypotheses, 2) data collection procedures or using existing data, 3) stopping rules, 4) creating variables, 5) clear analyses. - Group discussion, FAQs, and Q&A (15 min, David and Ian) - Link to presentation: https://osf.io/9sfta ---------- #### Major Initiatives #### Our presentation will cover three initiatives that the research and publishing community can adopt in order to improve the transparency and rigor of the work that they produce and publish. 1. The Transparency and Openness Promotion ([TOP][5]) Guidelines are a set of eight standards that publishers can adopt in order to reward best practices in transparent research. Journals can adopt each standard in one of three tiers of increasing rigor. This removes barriers to adoption, while still guiding future improvement. 2. [Badges][6] are a simple and effective way of rewarding best practices. When authors have the option of receiving a visual indicator on their work for sharing data and research materials, the rates of these behaviors increase ([Kidwell et. al., 2016][7]). ![badges][8] 3. [Registered Reports][9] are a publishing format in which peer review occurs before results are known. This focuses expert evaluation on the research questions and the proposed methods to answer them. Unlike traditional peer review, the findings are surfaced regardless of outcome, so the incentive is to address pressing questions as rigorously as possible. ---------- #### Supporting Materials #### - Badges: [project page][11] and [handout][12] - TOP Guidelines: [website][13] - Prereg Challenge: [website][14] and [handout][15] ---------- #### About COS #### The [Center for Open Science][16] was founded in 2013 with a mission to increase the openness, integrity, and reproducibility of scientific research. We do this in the following ways: - We work with the scientific community to align incentives with scientific values ([summary handout][17]). - We build tools for researchers that enable the actions we promote (e.g. the [Open Science Framework][18]) - We measure the extent of the problem (e.g. the Reproducibility Project: Psychology, [OSC, 2015][19]; and the [Reproducibility Project: Cancer Biology][20] ) and the effectiveness of our initiatives (e.g. [Kidwell et. al., 2016][21]). [1]: https://osf.io/8crmd/wiki/home [2]: https://www.force2017.org/home.html [3]: https://twitter.com/evomellor [4]: http://jre.sagepub.com/content/2/4/3.short [5]: http://cos.io/top [6]: https://osf.io/tvyxz/wiki/home/ [7]: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002456 [8]: http://i.imgur.com/wIINwZw.png [9]: https://osf.io/8mpji/wiki/home/ [10]: https://osf.io/s637e/ [11]: https://osf.io/tvyxz/wiki/home/ [12]: https://osf.io/k6fxs/ [13]: http://cos.io/top [14]: http://cos.io/prereg [15]: https://osf.io/2vu7m/ [16]: http://cos.io [17]: https://osf.io/vuzfb/ [18]: http://osf.io [19]: http://science.sciencemag.org/content/349/6251/aac4716 [20]: https://elifesciences.org/collections/reproducibility-project-cancer-biology [21]: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002456