## Examples of Large-Scale Replications ## - [Reproducibility Project][1] - [Many Labs Projects][2] - [Collaborative Research and Education Project (for Undergrads)][3] - [Red Card Study] [4] - [Pipeline Project] [5] - [Registered Replication Reports at *PPS*] [6] ## Lists of Individual Replications ## - [Psych File Drawer][7] - [Curate Science][8] - [Etienne LeBel's List][9] ## Discussions About Replications ## #### History & Philosophy #### - [Feynman, 1974, Cargo Cult lecture][10] - [Popper, 1935,1959, Logic of Scientific Discovery, pp. 23, 81][11] #### General #### - [APA Monitor Article on Replication][12] - [Asendorpf et al., 2013] [13] - [Brandt et al., 2014] [14] - [Funder et al., 2013] [15] - [Campbell et al., 2014] [16] - [Schmidt, 2009] [17] - [Valentine et al., 2011][18] - [Lykken, 1968] [19] - [Fetterman & Sassenberg, 2015] [20] - [Goodman et al., 2016][21] - [Kahneman Daisey Chain idea] [22] - [Makel et al., 2012] [23] - [Bonett, 2012] [24] - [Laws, 2016] [25] - [Dreber et al., 2015] [26] - [Crandall & Sherman, 2016] [27] - [Hüffmeier, Mazei, & Schultze, 2016][28] #### Pro-Replication #### - [LeBel et al.][29] - [Simons, 2014][30] - [LeBel, 2015] [31] - [Kraus, 2015] [32] - [Roberts, 2015] [33] - [Roberts, 2016] [34] #### Cautions About Replication #### - [Stroebe & Strack, 2014][35] - [Cesario, 2014][36] - [Schwarz & Clore, 2016][37] - [Coyne, 2016][38] - [Fabrigar & Wegener, 2016][39] - [Gilbert et al., 2016][40] - [Baumeister, 2016][41] - [Finkel et al., 2015] [42] - [Kahneman, 2014] [43] - [Barsalou, 2016] [44] - [Rotello et al., 2015] [45] - [Schmidt & Oh, 2016] [46] - [MacElreath & Smaldino, 2015][47] #### Statistical Issues - [Anderson & Maxwell, 2016] [48] - [Maxwell et al., 2015][49] - [Simonsohn, 2015] [50] - [Etz & Vandekerckhove, 2016] [51] - [Lakens, 2014] [52] - [Lakens & Evers, 2014] [53] - [Kashy et al., 2009] [54] - [Verhagen & Wagenmakers, 2014][55] - [Wagenmakers et al., 2016][56] ## Journals That Support Replications ## - Many open-access journals (e.g. [PLoS ONE][57], [Frontiers][58]) - [Collabra][59] - [Perspectives on Psychological Science][60] - [Journal of Personality and Social Psychology][61] - [Journal of Experimental Psychology: General][62] - [Archives of Scientific Psychology][63] - [Journal of Research in Personality][64] - [Social Psychological and Personality Science][65] - [Psychological Science][66] - [Judgment and Decision Making][67] - [Journal of Experimental Social Psychology][68] - [Psychonomic Bulletin & Review][69] ## Major Obstacles ## - Incentive structure (e.g., no incentives to execute & report internal direct replications; new badge discussed to help in this regard) - Lack of outlets - Negative reactions to replication attempts - Lack of details about original procedures - Challenges with certain designs (e.g., long-term longitudinal research, hard to reach populations) ## Things We Can Do To Promote Replication ## - Carefully describe our own studies so they can be replicated - Strive to be explicit about boundary conditions in our own reports - Archive research materials and data - Do replications (make them more normative) - Involve undergraduates - Make honors projects meaningful - Organize innovative replication studies - Pipeline Project ## Best Practices ## - Which replications to attempt - Replicate work that *should* replicate, not just work you expect to fail - Replicate your colleagues' work - Replicate studies that are important for your work - Replicate hard to conduct research - Preregister attempts - Work with original authors - Carefully consider sample size - Original effects are likely to be overestimates - Plan for high power - Consider sequential analysis - Carefully consider criteria for evaluating replication - Significant result - Significant result in meta-analysis with original effects - Original result outside confidence interval of new study - New result outside confidence interval of old study - Bayesian approaches - Simonsohn's "Small Telescopes" approach - Document all known discrepancies with original study - Be careful about tone - Commit to disseminating regardless of outcome ## Specific Goals for This Conference ## - Do we need a better framework for classifying replication attempts (e.g., "exact" vs. "direct" vs. "conceptual")?; See [here][70] for a recently proposed replication taxonomy - Do we need a review article summarizing objections and responses? - Do we need a review article summarizing best practices for replication studies? [1]: https://osf.io/ezcuj/ [2]: https://osf.io/89vqh/ [3]: https://osf.io/wfc6u/ [4]:https://osf.io/j5v8f/ [5]: http://www.sciencedirect.com/science/article/pii/S0022103115300019 [6]: http://pps.sagepub.com/content/9/5/552.short [7]: http://www.psychfiledrawer.org/ "Psych File Drawer" [8]: http://curatescience.org/ [9]: https://proveyourselfwrong.wordpress.com/2015/10/13/a-list-of-successful-and-unsuccessful-high-powered-direct-replications-of-social-psychology-findings/ [10]: http://calteches.library.caltech.edu/51/2/CargoCult.htm:// [11]: http://strangebeautiful.com/other-texts/popper-logic-scientific-discovery.pdf [12]: http://www.apa.org/monitor/2014/09/results.aspx [13]: https://biblio.ugent.be/publication/3048791/file/6780479 [14]: http://www.sciencedirect.com/science/article/pii/S0022103113001819 [15]: http://psr.sagepub.com/content/18/1/3 [16]: https://www.researchgate.net/profile/Timothy_Loving/publication/268236140_Enhancing_transparency_of_the_research_process_to_increase_accuracy_of_findings_A_guide_for_relationship_researchers/links/546aad260cf20dedafd38bc2.pdf [17]:http://psycnet.apa.org/journals/gpr/13/2/90/ [18]: http://link.springer.com/article/10.1007/s11121-011-0217-6 [19]: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.220.5553&rep=rep1&type=pdf [20]: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0143723 [21]: http://stm.sciencemag.org/content/8/341/341ps12.full [22]: http://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535 [23]: http://pps.sagepub.com/content/7/6/537.short [24]: http://cdp.sagepub.com/content/21/6/409.short [25]:http://bmcpsychology.biomedcentral.com/articles/10.1186/s40359-016-0135-2 [26]: http://www.pnas.org/content/112/50/15343.short [27]: http://www.sciencedirect.com/science/article/pii/S0022103115300020 [28]: http://www.sciencedirect.com/science/article/pii/S0022103115001195 [29]: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2616384 [30]: http://pps.sagepub.com/content/9/1/76.abstract [31]: http://collabra.org/articles/10.1525/collabra.23/ [32]: http://journal.frontiersin.org/article/10.3389/fpsyg.2015.01709/full [33]: https://pigee.wordpress.com/2015/04/23/be-your-own-replicator/ [34]:https://pigee.wordpress.com/2016/06/01/we-need-federally-funded-daisy-chains/ [35]: http://pps.sagepub.com/content/9/1/59.abstract [36]: http://pps.sagepub.com/content/9/1/40.abstract [37]: https://www.researchgate.net/profile/Norbert_Schwarz2/publication/303520356_Evaluating_psychological_research_requires_more_than_attention_to_the_N_A_comment_on_Simonsohn's_2015_small_telescopes/links/5746307908ae298602f9f122.pdf [38]: http://bmcpsychology.biomedcentral.com/articles/10.1186/s40359-016-0134-3 [39]: http://www.sciencedirect.com/science/article/pii/S0022103115000967 [40]: http://science.sciencemag.org/content/351/6277/1037.2 [41]: http://www.sciencedirect.com/science/article/pii/S002210311600007X [42]: http://psycnet.apa.org/journals/psp/108/2/275/ [43]: http://psycnet.apa.org/journals/zsp/45/4/310/ [44]: http://barsaloulab.org/Online_Articles/2016-Barsalou_CurrOpPsych-social_priming.pdf [45]: http://link.springer.com/article/10.3758/s13423-014-0759-2 [46]: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2773582 [47]: http://Replication,%20Communication,%20and%20the%20Population%20Dynamics%20of%20Scientific%20Discovery [48]: http://psycnet.apa.org/psycinfo/2015-34226-001/ [49]: http://psycnet.apa.org/journals/amp/70/6/487/ [50]: http://pss.sagepub.com/content/26/5/559 [51]: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0149794 [52]: http://onlinelibrary.wiley.com/doi/10.1002/ejsp.2023/full [53]:http://pps.sagepub.com/content/9/3/278.short [54]: http://www.collabra.org/ [55]: http://www.ejwagenmakers.com/2014/VerhagenWagenmakers2014.pdf [56]: http://www.ejwagenmakers.com/2016/WagenmakersEtAl2016LonelyBathing.pdf [57]: http://journals.plos.org/plosone/ [58]: http://journal.frontiersin.org/journal/psychology [59]: http://www.collabra.org/ [60]: http://www.psychologicalscience.org/index.php/replication [61]: http://www.apa.org/pubs/journals/psp/index.aspx?tab=4 [62]: http://www.apa.org/pubs/journals/xge/index.aspx [63]: http://www.apa.org/pubs/journals/arc/ [64]: https://www.elsevier.com/journals/journal-of-research-in-personality/0092-6566/guide-for-authors [65]: https://us.sagepub.com/en-us/nam/social-psychological-and-personality-science/journal201952 [66]: http://www.psychologicalscience.org/index.php/publications/journals/psychological_science/ps-submissions#ART [67]: http://journal.sjdm.org/ [68]: http://www.journals.elsevier.com/journal-of-experimental-social-psychology/ [69]: http://www.springer.com/psychology/cognitive+psychology/journal/13423 [70]: http://curatescience.org/logos/rep-taxonomy.png