Home

Menu

Loading wiki pages...

View
Wiki Version:
<h2>Examples of Large-Scale Replications</h2> <ul> <li><a href="https://osf.io/ezcuj/" rel="nofollow">Reproducibility Project</a></li> <li><a href="https://osf.io/89vqh/" rel="nofollow">Many Labs Projects</a></li> <li><a href="https://osf.io/wfc6u/" rel="nofollow">Collaborative Research and Education Project (for Undergrads)</a></li> <li><a href="https://osf.io/j5v8f/" rel="nofollow">Red Card Study</a></li> <li><a href="http://www.sciencedirect.com/science/article/pii/S0022103115300019" rel="nofollow">Pipeline Project</a></li> <li><a href="http://pps.sagepub.com/content/9/5/552.short" rel="nofollow">Registered Replication Reports at <em>PPS</em></a></li> </ul> <h2>Lists of Individual Replications</h2> <ul> <li><a href="http://www.psychfiledrawer.org/" rel="nofollow" title="Psych File Drawer">Psych File Drawer</a></li> <li><a href="http://curatescience.org/" rel="nofollow">Curate Science</a></li> <li><a href="https://proveyourselfwrong.wordpress.com/2015/10/13/a-list-of-successful-and-unsuccessful-high-powered-direct-replications-of-social-psychology-findings/" rel="nofollow">Etienne LeBel's List</a></li> </ul> <h2>Discussions About Replications ##</h2> <h4>History & Philosophy</h4> <ul> <li><a href="http://calteches.library.caltech.edu/51/2/CargoCult.htm://" rel="nofollow">Feynman, 1974, Cargo Cult lecture</a></li> <li><a href="http://strangebeautiful.com/other-texts/popper-logic-scientific-discovery.pdf" rel="nofollow">Popper, 1935,1959, Logic of Scientific Discovery, pp. 23, 81</a></li> </ul> <h4>General</h4> <ul> <li><a href="http://www.apa.org/monitor/2014/09/results.aspx" rel="nofollow">APA Monitor Article on Replication</a></li> <li><a href="https://biblio.ugent.be/publication/3048791/file/6780479" rel="nofollow">Asendorpf et al., 2013</a></li> <li><a href="http://www.sciencedirect.com/science/article/pii/S0022103113001819" rel="nofollow">Brandt et al., 2014</a></li> <li><a href="http://psr.sagepub.com/content/18/1/3" rel="nofollow">Funder et al., 2013</a></li> <li><a href="https://www.researchgate.net/profile/Timothy_Loving/publication/268236140_Enhancing_transparency_of_the_research_process_to_increase_accuracy_of_findings_A_guide_for_relationship_researchers/links/546aad260cf20dedafd38bc2.pdf" rel="nofollow">Campbell et al., 2014</a></li> <li><a href="http://psycnet.apa.org/journals/gpr/13/2/90/" rel="nofollow">Schmidt, 2009</a></li> <li><a href="http://link.springer.com/article/10.1007/s11121-011-0217-6" rel="nofollow">Valentine et al., 2011</a></li> <li><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.220.5553&rep=rep1&type=pdf" rel="nofollow">Lykken, 1968</a> </li> <li><a href="http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0143723" rel="nofollow">Fetterman & Sassenberg, 2015</a></li> <li><a href="http://stm.sciencemag.org/content/8/341/341ps12.full" rel="nofollow">Goodman et al., 2016</a></li> <li><a href="http://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535" rel="nofollow">Kahneman Daisey Chain idea</a></li> <li><a href="http://pps.sagepub.com/content/7/6/537.short" rel="nofollow">Makel et al., 2012</a> </li> <li><a href="http://cdp.sagepub.com/content/21/6/409.short" rel="nofollow">Bonett, 2012</a></li> <li><a href="http://bmcpsychology.biomedcentral.com/articles/10.1186/s40359-016-0135-2" rel="nofollow">Laws, 2016</a></li> <li><a href="http://www.pnas.org/content/112/50/15343.short" rel="nofollow">Dreber et al., 2015</a></li> <li><a href="http://www.sciencedirect.com/science/article/pii/S0022103115300020" rel="nofollow">Crandall & Sherman, 2016</a></li> <li><a href="http://www.sciencedirect.com/science/article/pii/S0022103115001195" rel="nofollow">H├╝ffmeier, Mazei, & Schultze, 2016</a></li> </ul> <h4>Pro-Replication</h4> <ul> <li><a href="http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2616384" rel="nofollow">LeBel et al.</a></li> <li><a href="http://pps.sagepub.com/content/9/1/76.abstract" rel="nofollow">Simons, 2014</a></li> <li><a href="http://collabra.org/articles/10.1525/collabra.23/" rel="nofollow">LeBel, 2015</a></li> <li><a href="http://journal.frontiersin.org/article/10.3389/fpsyg.2015.01709/full" rel="nofollow">Kraus, 2015</a></li> <li><a href="https://pigee.wordpress.com/2015/04/23/be-your-own-replicator/" rel="nofollow">Roberts, 2015</a></li> <li><a href="https://pigee.wordpress.com/2016/06/01/we-need-federally-funded-daisy-chains/" rel="nofollow">Roberts, 2016</a></li> </ul> <h4>Cautions About Replication</h4> <ul> <li><a href="http://pps.sagepub.com/content/9/1/59.abstract" rel="nofollow">Stroebe & Strack, 2014</a></li> <li><a href="http://pps.sagepub.com/content/9/1/40.abstract" rel="nofollow">Cesario, 2014</a></li> <li><a href="https://www.researchgate.net/profile/Norbert_Schwarz2/publication/303520356_Evaluating_psychological_research_requires_more_than_attention_to_the_N_A_comment_on_Simonsohn's_2015_small_telescopes/links/5746307908ae298602f9f122.pdf" rel="nofollow">Schwarz & Clore, 2016</a></li> <li><a href="http://bmcpsychology.biomedcentral.com/articles/10.1186/s40359-016-0134-3" rel="nofollow">Coyne, 2016</a></li> <li><a href="http://www.sciencedirect.com/science/article/pii/S0022103115000967" rel="nofollow">Fabrigar & Wegener, 2016</a></li> <li><a href="http://science.sciencemag.org/content/351/6277/1037.2" rel="nofollow">Gilbert et al., 2016</a></li> <li><a href="http://www.sciencedirect.com/science/article/pii/S002210311600007X" rel="nofollow">Baumeister, 2016</a></li> <li><a href="http://psycnet.apa.org/journals/psp/108/2/275/" rel="nofollow">Finkel et al., 2015</a></li> <li><a href="http://psycnet.apa.org/journals/zsp/45/4/310/" rel="nofollow">Kahneman, 2014</a> </li> <li><a href="http://barsaloulab.org/Online_Articles/2016-Barsalou_CurrOpPsych-social_priming.pdf" rel="nofollow">Barsalou, 2016</a></li> <li><a href="http://link.springer.com/article/10.3758/s13423-014-0759-2" rel="nofollow">Rotello et al., 2015</a></li> <li><a href="http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2773582" rel="nofollow">Schmidt & Oh, 2016</a></li> <li><a href="http://Replication,%20Communication,%20and%20the%20Population%20Dynamics%20of%20Scientific%20Discovery" rel="nofollow">MacElreath & Smaldino, 2015</a></li> </ul> <h4>Statistical Issues</h4> <ul> <li><a href="http://psycnet.apa.org/psycinfo/2015-34226-001/" rel="nofollow">Anderson & Maxwell, 2016</a></li> <li><a href="http://psycnet.apa.org/journals/amp/70/6/487/" rel="nofollow">Maxwell et al., 2015</a></li> <li><a href="http://pss.sagepub.com/content/26/5/559" rel="nofollow">Simonsohn, 2015</a></li> <li><a href="http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0149794" rel="nofollow">Etz & Vandekerckhove, 2016</a></li> <li><a href="http://onlinelibrary.wiley.com/doi/10.1002/ejsp.2023/full" rel="nofollow">Lakens, 2014</a></li> <li><a href="http://pps.sagepub.com/content/9/3/278.short" rel="nofollow">Lakens & Evers, 2014</a></li> <li><a href="http://www.collabra.org/" rel="nofollow">Kashy et al., 2009</a></li> <li><a href="http://www.ejwagenmakers.com/2014/VerhagenWagenmakers2014.pdf" rel="nofollow">Verhagen & Wagenmakers, 2014</a></li> <li><a href="http://www.ejwagenmakers.com/2016/WagenmakersEtAl2016LonelyBathing.pdf" rel="nofollow">Wagenmakers et al., 2016</a></li> </ul> <h2>Journals That Support Replications</h2> <ul> <li>Many open-access journals (e.g. <a href="http://journals.plos.org/plosone/" rel="nofollow">PLoS ONE</a>, <a href="http://journal.frontiersin.org/journal/psychology" rel="nofollow">Frontiers</a>)</li> <li><a href="http://www.collabra.org/" rel="nofollow">Collabra</a> </li> <li><a href="http://www.psychologicalscience.org/index.php/replication" rel="nofollow">Perspectives on Psychological Science</a></li> <li><a href="http://www.apa.org/pubs/journals/psp/index.aspx?tab=4" rel="nofollow">Journal of Personality and Social Psychology</a></li> <li><a href="http://www.apa.org/pubs/journals/xge/index.aspx" rel="nofollow">Journal of Experimental Psychology: General</a></li> <li><a href="http://www.apa.org/pubs/journals/arc/" rel="nofollow">Archives of Scientific Psychology</a></li> <li><a href="https://www.elsevier.com/journals/journal-of-research-in-personality/0092-6566/guide-for-authors" rel="nofollow">Journal of Research in Personality</a></li> <li><a href="https://us.sagepub.com/en-us/nam/social-psychological-and-personality-science/journal201952" rel="nofollow">Social Psychological and Personality Science</a></li> <li><a href="http://www.psychologicalscience.org/index.php/publications/journals/psychological_science/ps-submissions#ART" rel="nofollow">Psychological Science</a></li> <li><a href="http://journal.sjdm.org/" rel="nofollow">Judgment and Decision Making</a></li> <li><a href="http://www.journals.elsevier.com/journal-of-experimental-social-psychology/" rel="nofollow">Journal of Experimental Social Psychology</a></li> <li><a href="http://www.springer.com/psychology/cognitive+psychology/journal/13423" rel="nofollow">Psychonomic Bulletin & Review</a></li> </ul> <h2>Major Obstacles</h2> <ul> <li>Incentive structure (e.g., no incentives to execute & report internal direct replications; new badge discussed to help in this regard)</li> <li>Lack of outlets</li> <li>Negative reactions to replication attempts</li> <li>Lack of details about original procedures</li> <li>Challenges with certain designs (e.g., long-term longitudinal research, hard to reach populations)</li> </ul> <h2>Things We Can Do To Promote Replication ##</h2> <ul> <li>Carefully describe our own studies so they can be replicated</li> <li>Strive to be explicit about boundary conditions in our own reports</li> <li>Archive research materials and data</li> <li>Do replications (make them more normative)</li> <li>Involve undergraduates<ul> <li>Make honors projects meaningful</li> </ul> </li> <li>Organize innovative replication studies<ul> <li>Pipeline Project </li> </ul> </li> </ul> <h2>Best Practices</h2> <ul> <li>Which replications to attempt<ul> <li>Replicate work that <em>should</em> replicate, not just work you expect to fail</li> <li>Replicate your colleagues' work</li> <li>Replicate studies that are important for your work</li> <li>Replicate hard to conduct research</li> </ul> </li> <li>Preregister attempts</li> <li>Work with original authors</li> <li>Carefully consider sample size <ul> <li>Original effects are likely to be overestimates</li> <li>Plan for high power</li> <li>Consider sequential analysis</li> </ul> </li> <li>Carefully consider criteria for evaluating replication<ul> <li>Significant result</li> <li>Significant result in meta-analysis with original effects</li> <li>Original result outside confidence interval of new study</li> <li>New result outside confidence interval of old study</li> <li>Bayesian approaches</li> <li>Simonsohn's "Small Telescopes" approach</li> </ul> </li> <li>Document all known discrepancies with original study</li> <li>Be careful about tone</li> <li>Commit to disseminating regardless of outcome</li> </ul> <h2>Specific Goals for This Conference</h2> <ul> <li>Do we need a better framework for classifying replication attempts (e.g., "exact" vs. "direct" vs. "conceptual")?; See <a href="http://curatescience.org/logos/rep-taxonomy.png" rel="nofollow">here</a> for a recently proposed replication taxonomy </li> <li>Do we need a review article summarizing objections and responses?</li> <li>Do we need a review article summarizing best practices for replication studies?</li> </ul>
OSF does not support the use of Internet Explorer. For optimal performance, please switch to another browser.
Accept
This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.
Accept
×

Start managing your projects on the OSF today.

Free and easy to use, the Open Science Framework supports the entire research lifecycle: planning, execution, reporting, archiving, and discovery.