**Here you will find resources that we think might be helpful for teaching replications:** - Frank, M. C., & Saxe, R. (2012). Teaching replication. Perspectives on Psychological Science, 7, 600-604. doi: 10.1177/1745691612460686 [http://pps.sagepub.com/content/7/6/600.full.pdf+html][1] - Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F., Geller, J., Giner-Sorolla, R.,…Van 't Veer, A. (2014). The Replication Recipe: What Makes for a Convincing Replication? Journal of Experimental Social Psychology, 50, 217-224. doi: 10.1016/j.jesp.2013.10.005 [http://www.sciencedirect.com/science/article/pii/S0022103113001819][2] - Simons, D. J. (2014). The value of direct replication. Perspectives on Psychological Science, 9, 76-80. doi: 10.1177/1745691613514755 [http://pps.sagepub.com/content/9/1/76.abstract][3] - Grahe, J. E., Reifman, A., Hermann, A. D., Walker, M., Oleson, K. C., Nario-Redmond, M., & Wiebe, R. P. (2012). Harnessing the undiscovered resource of student research projects. Perspectives on Psychological Science, 7, 605-604. doi: 10.1177/1745691612459057 [http://pps.sagepub.com/content/7/6/605.full.pdf+html][4] - Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7, 608-614. doi: 10.1177/1745691612462586 [http://pps.sagepub.com/content/7/6/608.full.pdf+html][5] - Improving (Our) Science Syllabus [https://docs.google.com/document/d/1FLV7wHQ-89H3e38bFgmMI1HhB_T3k3EC_93PFq21OqY/edit][6] (note, this needs to be adapted for CREP purposes) - Syllabus for Psych 254 at Stanford - graduate-level methods class, focused on replication, taught by Mike Frank (Winter 2013) [https://osf.io/mbrwz/][7] **Articles from Improving Our Science syllabus:** Week 2: - Giner-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7(6), 562-571. doi: 10.1177/1745691612457576 [http://pps.sagepub.com/content/7/6/562.short][8] - Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia II. Restructuring incentives and practices to promote truth over publishability.Perspectives on Psychological Science, 7(6), 615-631. doi: 10.1177/1745691612459058 [http://pps.sagepub.com/content/7/6/615.short][9] Week 3: - Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23(3), 217-243. doi: 10.1080/1047840X.2012.692215 [http://www.tandfonline.com/doi/abs/10.1080/1047840X.2012.692215#.Uxjo8PldW54][10] - Nosek, B. A., & Bar-Anan, Y. (2012). Scientific Communication Is Changing and Scientists Should Lead the Way. Psychological Inquiry, 23(3), 308-314. doi: 10.1080/1047840X.2012.717907 [http://www.tandfonline.com/doi/abs/10.1080/1047840X.2012.717907?journalCode=hpli20#.UxjpOfldW54][11] Week 5: - Email exchanges among psychological science editors and other colleagues on Poldrack et al. (2008) and Simmons et al. (2011) articles. [http://psdlab.uoregon.edu/docs/psci_disc.pdf][12] - John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological science, 23(5), 524-532. doi:10.1177/0956797611430953 [http://pss.sagepub.com/content/23/5/524.short][13] - Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17(4), 551. doi: 10.1037/a0029487 [http://psycnet.apa.org/journals/met/17/4/551/][14] - Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366. doi:10.1177/0956797611417632 [http://pss.sagepub.com/content/22/11/1359.short][15] Week 6: - Bertamini, M., & Munafò, M. R. (2012). Bite-size science and its undesired side effects. Perspectives on Psychological Science, 7(1), 67-71. doi: 10.1177/1745691611429353 [http://pps.sagepub.com/content/7/1/67.short][16] - Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1. doi: 10.1037/h0076157 [http://psycnet.apa.org/journals/bul/82/1/1/][17] - Mathieu, S., Boutron, I., Moher, D., Altman, D. G., & Ravaud, P. (2009). Comparison of registered and published primary outcomes in randomized controlled trials. Jama, 302(9), 977-984. doi:10.1001/jama.2009.1242 [http://jama.jamanetwork.com/article.aspx?articleid=184503][18] - Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49(1), 108-112. doi:10.1080/00031305.1995.10476125 [http://www.jstor.org/stable/2684823][19] - Young, N. S., Ioannidis, J. P., & Al-Ubaydli, O. (2008). Why current publication practices may distort science. PLoS medicine, 5(10), e201. doi:10.1371/journal.pmed.0050201 [http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0050201][20] Week 7 - Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531-533. doi:10.1038/483531a [http://www.nature.com/nature/journal/v483/n7391/full/483531a.html][21] - Lehrer, J. (2010). The truth wears off. The New Yorker, 13, 52. For link search article title in Google scholar - Morin, A., Urban, J., Adams, P. D., Foster, I., Sali, A., Baker, D., & Sliz, P. (2012). Shining light into black boxes. Science, 336(6078), 159-160. [http://www.salilab.org/pdf/Morin_Science_2012.pdf][22] - Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657-660. doi: 10.1177/1745691612462588 [http://pps.sagepub.com/content/7/6/657.short][23] - Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology,13(2), 90. doi: 10.1037/a0015108 [http://psycnet.apa.org/journals/gpr/13/2/90/][24] - Schooler, J. (2011). Unpublished results hide the decline effect. Nature,470(7335), 437. doi: 10.1038/470437a [http://www.nature.com/news/2011/110223/full/470437a.html][25] Week 10: - Crocker, J., & Cooper, M. L. (2011). Addressing scientific fraud. Science,334(6060), 1182-1182. doi: 10.1126/science.1216775 [http://www.sciencemag.org/content/334/6060/1182.short][26] - Simonsohn, U. (2013). Just Post It The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone. Psychological science, 24(10), 1875-1888. doi:10.1177/0956797613480366 [http://pss.sagepub.com/content/24/10/1875.short][27] - Interim report regarding the breach of scientific integrity committed by Prof. D.A. Stapel [http://www.tilburguniversity.edu/upload/547aa461-6cd1-48cd-801b-61c434a73f79_interim-report.pdf][28] - Francis, G. (2012). Evidence that publication bias contaminated studies relating social class and unethical behavior. Proceedings of the National Academy of Sciences, 109(25), E1587-E1587. doi: 10.1073/pnas.1203591109 [http://www.pnas.org/content/109/25/E1587.short][29] - Piff, P. K., Stancato, D. M., Côté, S., Mendoza-Denton, R., & Keltner, D. (2012). Reply to Francis: Cumulative power calculations are faulty when based on observed power and a small sample of studies. Proceedings of the National Academy of Sciences, 109(25), E1588-E1588. doi: 10.1073/pnas.1205367109 [http://www.pnas.org/content/109/25/E1588.short][30] - Francis, G. (2012). Checking the counterarguments confirms that publication bias contaminated studies relating social class and unethical behavior. [http://www2.psych.purdue.edu/~gfrancis/Publications/FrancisRebuttal2012.pdf][31] - Francis, G. (2012). The same old New Look: Publication bias in a study of wishful seeing. i-Perception, 3(3), 176. doi: 10.1068/i0519ic [http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3485847/][32] - Balcetis, E., & Dunning, D. (2012). A false-positive error in search in selective reporting: A refutation of Francis. i. Perception, 3(3). - Francis, G. Areas of disagreement. Or Some clarity about publication bias and wishful seeing [http://i-perception.perceptionweb.com/misc/i03/i0519ic--greg_response.doc][33] - Francis, G. (2012). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19(2), 151-156. doi: 10.3758/s13423-012-0227-9 [http://link.springer.com/article/10.3758/s13423-012-0227-9][34] Week 11: - Cooper, H., & Patall, E. A. (2009). The relative benefits of meta-analysis conducted with individual participant data versus aggregated data.Psychological methods, 14(2), 165. doi: 10.1037/a0015565 [http://psycnet.apa.org/journals/met/14/2/165/][35] - Howard, G. S., Lau, M. Y., Maxwell, S. E., Venter, A., Lundy, R., & Sweeny, R. M. (2009). Do research literatures give correct answers?. Review of General Psychology, 13(2), 116. doi: 10.1037/a0015468 [http://psycnet.apa.org/journals/gpr/13/2/116/][36] - Howard, G. S., Hill, T. L., Maxwell, S. E., Baptista, T. M., Farias, M. H., Coelho, C., ... & Coulter-Kern, R. (2009). What’s wrong with research literatures? And how to make them right. Review of General Psychology, 13(2), 146. doi: 10.1037/a0015319 [http://psycnet.apa.org/journals/gpr/13/2/146/][37] - Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of consulting and clinical Psychology, 46(4), 806. doi: 10.1037/0022-006X.46.4.806 [http://psycnet.apa.org/journals/ccp/46/4/806/][38] Week 14: - Fiedler, K., Kutzner, F., & Krueger, J. I. (2012). The long way from α-error control to validity proper problems with a short-sighted false-positive debate.Perspectives on Psychological Science, 7(6), 661-669. doi: 10.1177/1745691612462587 [http://pps.sagepub.com/content/7/6/661.short][39] - Fuchs, H. M., Jenny, M., & Fiedler, S. (2012). Psychologists are open to change, yet wary of rules. Perspectives on Psychological Science, 7(6), 639-642. doi: 10.1177/1745691612459521 [http://pps.sagepub.com/content/7/6/639.short][40] - Klein, O., Doyen, S., Leys, C., Miller, S., Questienne, L., & Cleeremans, A. (2012). Low hopes, high expectations expectancy effects and the replicability of behavioral experiments. Perspectives on Psychological Science, 7(6), 572-584. doi: 10.1177/1745691612463704 [http://pps.sagepub.com/content/7/6/572.short][41] - Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in Psychology Research How Often Do They Really Occur?. Perspectives on Psychological Science, 7(6), 537-542. doi: 10.1177/1745691612460688 [http://pps.sagepub.com/content/7/6/537.short][42] [1]: http://pps.sagepub.com/content/7/6/600.full.pdf+html [2]: http://www.sciencedirect.com/science/article/pii/S0022103113001819 [3]: http://pps.sagepub.com/content/9/1/76.abstract [4]: http://pps.sagepub.com/content/7/6/605.full.pdf+html [5]: http://pps.sagepub.com/content/7/6/608.full.pdf+html [6]: https://docs.google.com/document/d/1FLV7wHQ-89H3e38bFgmMI1HhB_T3k3EC_93PFq21OqY/edit [7]: https://osf.io/mbrwz/ [8]: http://pps.sagepub.com/content/7/6/562.short [9]: http://pps.sagepub.com/content/7/6/615.short [10]: http://www.tandfonline.com/doi/abs/10.1080/1047840X.2012.692215#.Uxjo8PldW54 [11]: http://www.tandfonline.com/doi/abs/10.1080/1047840X.2012.717907?journalCode=hpli20#.UxjpOfldW54 [12]: http://psdlab.uoregon.edu/docs/psci_disc.pdf [13]: http://pss.sagepub.com/content/23/5/524.short [14]: http://psycnet.apa.org/journals/met/17/4/551/ [15]: http://pss.sagepub.com/content/22/11/1359.short [16]: http://pps.sagepub.com/content/7/1/67.short [17]: http://psycnet.apa.org/journals/bul/82/1/1/ [18]: http://jama.jamanetwork.com/article.aspx?articleid=184503 [19]: http://www.jstor.org/stable/2684823 [20]: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0050201 [21]: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html [22]: http://www.salilab.org/pdf/Morin_Science_2012.pdf [23]: http://pps.sagepub.com/content/7/6/657.short [24]: http://psycnet.apa.org/journals/gpr/13/2/90/ [25]: http://www.nature.com/news/2011/110223/full/470437a.html [26]: http://www.sciencemag.org/content/334/6060/1182.short [27]: http://pss.sagepub.com/content/24/10/1875.short [28]: http://www.tilburguniversity.edu/upload/547aa461-6cd1-48cd-801b-61c434a73f79_interim-report.pdf [29]: http://www.pnas.org/content/109/25/E1587.short [30]: http://www.pnas.org/content/109/25/E1588.short [31]: http://www2.psych.purdue.edu/~gfrancis/Publications/FrancisRebuttal2012.pdf [32]: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3485847/ [33]: http://i-perception.perceptionweb.com/misc/i03/i0519ic--greg_response.doc [34]: http://link.springer.com/article/10.3758/s13423-012-0227-9 [35]: http://psycnet.apa.org/journals/met/14/2/165/ [36]: http://psycnet.apa.org/journals/gpr/13/2/116/ [37]: http://psycnet.apa.org/journals/gpr/13/2/146/ [38]: http://psycnet.apa.org/journals/ccp/46/4/806/ [39]: http://pps.sagepub.com/content/7/6/661.short [40]: http://pps.sagepub.com/content/7/6/639.short [41]: http://pps.sagepub.com/content/7/6/572.short [42]: http://pps.sagepub.com/content/7/6/537.short