# Welcome to the ReproducibiliTea Toronto Journal Club! This journal club is based at York University, Toronto, Canada. We aim to be as inclusive as possible. If you want to join our conversation during a particular meeting, then **just follow the link on the [Google Calendar event][1] to join the bi-weekly meetings!** **For W2024, the zoom link will be: https://yorku.zoom.us/j/99113914933.** **For any general inquirites, or to request a specific article/source to discuss, please contact us at reproducibiliteatoronto@gmail.com.** **To be added to our mailing list (to receive emails to the club), please complete the following form: https://forms.gle/ri5gt3XANawjek419** *** ## **Readings** ### **Winter 2024** For the Winter semester, we'll be chatting about more specific topics as they relate to reproducibility, primarily the incentive structure in science, null hypothesis significance testing (NHST), and why common arguments made against reproducibility are misguided. **Once more, all articles are made freely available on the shared [Google Drive Folder][4]. Winter readings can be found within the "Winter" sub-folder. Below is a summary of each reading.** **Week 1** - Jan 23, 2024 @ 2:30 PM EST *The incentive structure in science: Scientific Utopia II: Restructuring Incentives and Practices to Promote Truth Over Publishability* A paper that explains what's wrong with the incentive structure in science, and how the structure can be changed to promote more open-scientific attitudes and practices. **Week 2** - Feb 6, 2024 @ 2:30 PM EST *Mindless Statistics* This paper describes the (disturbing) origins of null hypothesis significance testing (NHST), the statistical framework that dominates psychology. It makes the evocative argument that NHST is treated less like a statistical framework and more like a mindless ritual. **Week 3** - Mar 5, 2024 @ 2:30 PM EST *The Earth is Round (p < .05)* This classic paper describes the issues with null hypothesis significance testing as a framework in psychology, such as in how it makes a logically invalid argument and is highly prone to misinterpretations. **Week 4** - Mar 19, 2024 @ 2:30 PM EST *Life after NHST: How to Describe Data Without "p-ing" Everywhere* This paper explores ways of describing data without using NHST, one potential manner in which psychologists can move away from NHST in their own research. **Week 5** - Apr 2, 2024 @ 2:30 PM EST *Is the Replication Crisis Overblown? Three Arguments Examined* Paper examining potential counterarguments to the idea that the reproducibility crisis is overexaggerated, and which provides rebuttals to said common arguments. In doing so, it reinforces the importance of open science and reproducibility. ### **Fall 2023** For the re-launch of the ReproducibiliTea Toronto Club, we'll be going through a lot of the classic reproducibility papers, along with some goodies sprinkled in! :) **All articles are made freely available on the shared [Google Drive Folder][4]. Below is a summary of each reading.** **Week 1** - Oct 18, 2023 @ 10:00 AM EST 1. *A Manifesto for Reproducible Science* 2. *Estimating the Reproducibility of Psychology* Two excellent papers to get everyone acquainted with the idea and importance of reproducibility. **Week 2** - Nov 1, 2023 @ 10:00 AM EST *Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology* Paper on a recent attempt at computationally reproducing the statistical analyses within several journal articles. **Week 3** - Nov 15, 2023 @ 10:00 AM EST *Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling* Classic paper about the prevalence of questionable research practices (QRPs) in psychology and what it means for the discipline. **Week 4** - Nov. 29, 2023 1. *UK Psychology PhD researchers’ knowledge, perceptions, and experiences of open science* 2. *Attitudes Toward Open Science and Public Data Sharing* Papers exploring what students (paper 1) and professors (paper 2) think about the ideas of open science and reproducibility. **Week 5** - Dec. 13, 2023 *Is the Replication Crisis Overblown? Three Arguments Examined* Paper examining potential counterarguments to the idea that the reproducibility crisis is overexaggerated. *** ## Readings from Previous Sessions ### Fall 2020-Winter 2021 **Week 1** - Oct 30, 2020 @ 2:30 PM EST *When Personality Psychologists are High* Post from Dr. Ulrich Schimmack’s Blog about Replicability [https://replicationindex.com/2019/08/13/when-personality-psychologists-are-high/][5] **Week 2** - Nov 13, 2020 @ 2:30 PM EST *Personality Measurement with the Big Five Inventory* [https://replicationindex.com/2019/08/10/personality-measurement-with-the-big-five-inventory/][6] Post from Dr. Ulrich Schimmack’s Blog about Replicability **Week 3** - Nov 27, 2020 @ 2:30 PM EST *When a “Replication” Is Not a Replication. Commentary: Sequential Congruency Effects in Monolingual and Bilingual Adult* [https://www.frontiersin.org/articles/10.3389/fpsyg.2019.00797/full][7] Supplemental reading: [Null results in bilingualism research:What they tell us and what they don’t][8] Many of our readings this semester were pulled from a fantastic [Twitter thread][9] by [@zerdeve][10] highlighting some non-mainstream work on reproducibility, open science, replication crisis, meta-science by women **Week 4** - Jan 22, 2021 @ 2:30 PM EST *Paths in strange spaces: A comment on preregistration*: Part 1 A blog post by Daniel Navarro preserved as a preprint on psyarxiv. [https://psyarxiv.com/wxn58][11] **Week 5** - Feb 5, 2021 @ 2:30 PM EST *Paths in strange spaces: A comment on preregistration*: Part 2 A blog post by Daniel Navarro preserved as a preprint on psyarxiv. [https://psyarxiv.com/wxn58][12] **Week 6** - Feb 19, 2021 @ 1:oo PM EST Workshop on conducting Monte Carlo Simulations using the SimDesign package. [https://www.tqmp.org/RegularArticles/vol16-4/p248/p248.pdf][13] **Week 7** - Mar 5, 2021 @ 2:3o PM EST *Credibility of preprints: an interdisciplinary survey of researchers* [https://royalsocietypublishing.org/doi/10.1098/rsos.201520][14] **Week 8** - Mar 19, 2021 @ 2:30 PM EST *Mapping the discursive dimensions of the reproducibility crisis: A mixed methods analysis* [https://osf.io/preprints/metaarxiv/sbv3q][15] - Note might have to be read within your web broswer as the pdf had issues opening using some software. **Week 9** - Apr 2, 2021 @ 2:3o PM EST *An empirical analysis of journal policy effectiveness for computational reproducibility* [https://www.pnas.org/content/115/11/2584?ijkey=b785383d7b8d79876ed4fec4be93992ada5c50ca&keytype2=tf_ipsecsha][16] *** ### Winter 2020- Summer 2020 **Week 1** - Jan 17, 2020 @ 2:30 PM *A Manifesto for Reproducible Science* The Problem Defined: The general overview [https://www.nature.com/articles/s41562-016-0021][17] **Week 2** - Jan 24, 2020 @ 2:30 PM *False Positive Psychology* Examining Analytic Flexibility And why it is a problem [http://journals.sagepub.com/doi/abs/10.1177/0956797611417632][18] **Week 3** - Jan 31, 2020 @ 2:30 PM *Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling* Questionable Research Practices: are they really that common? And why are they problematic? [http://journals.sagepub.com/doi/abs/10.1177/0956797611430953][19] **Week 4** - Feb 7, 2020 @ 2:30 PM *Estimating the Reproducibility of Psychological Science* Reproducibility Now: Why many studies are not reproducible. [http://science.sciencemag.org/content/349/6251/aac4716][20] **Week 5** - Feb 14, 2020 @ 2:30 PM *Is the Replicability Crisis Overblown? Three Arguments Examined* How has the debate gone to far? Things will just turn out fine. [http://journals.sagepub.com/doi/pdf/10.1177/1745691612463401][21] **Week 6** - Feb 28, 2020 @ 2:30 PM *Open Science: What, Why and How Open Data and Materials* [https://psyarxiv.com/ak6jr][22] **Week 7** - Mar 6, 2020 @ 2:30 PM *The Natural Selection of Bad Science* And what about the future? [http://rsos.royalsocietypublishing.org/content/3/9/160384][23] **Week 8** - Mar 13, 2020 @ 2:30 PM *The Preregistration Revolution* Preregistration as a solution [http://www.pnas.org/content/pnas/early/2018/03/08/1708274114.full.pdf][24] **Week 9** - Mar 20, 2020 @ 2:30 PM *Introduction to Monte Carlo Simulation* A brief workshop about running Monte Carlo Simulations using the SimDesign package. [https://github.com/philchalmers/SimDesign/wiki][25] **Week 10** - Mar 27, 2020 @ 2:30 PM *The garden of forking paths* Researcher degrees of freedom [http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf][26] **Week 11** - Apr 3, 2020 @ 2:30 PM *Increasing Transparency Through a Multiverse Analysis* [http://www.stat.columbia.edu/~gelman/research/published/multiverse_published.pdf][27] **Week 12** - Apr 17, 2020 @ 1:30 PM *Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them* [https://psyarxiv.com/hs7wm/][28] **Week 13** - May 1, 2020 @ 1:30 PM *Good enough practices in scientific computing* [https://doi.org/10.1371/journal.pcbi.1005510][29] **Week 14** - May 8, 2020 @ 1:30 PM (GMT-4) *How (and Whether) to Teach Undergraduates About the Replication Crisis in Psychological Science* [https://journals.sagepub.com/doi/pdf/10.1177/0098628318762900?casa_token=-otISLbatjIAAAAA:ijtyknmdprbA5xYIsKF7_qzW8ugt4gdx0t2D8iw4yGAq0FsNJ5HSgXBvSxgBKGolBAqCNNsWWdw][30] **Week 15** - May 15, 2020 @ 1:30 PM (GMT-4) *Fallibility in Science: Responding to Errors in the Work of Oneself and Others* [https://journals.sagepub.com/doi/pdf/10.1177/2515245918776632?casa_token=1iG5YNtHYgIAAAAA:8HnvnF4JW5qTM95uqu6_Vplz3GyO5kiOJFtH19BgRPUO6ismdQiqRYGg41QRAtnFIfvMRTZSvCk][31] **Week 16** - May 22, 2020 @ 1:30 PM (GMT-4) *Practical Solutions for Sharing Data and Materials From Psychological Research* [https://journals.sagepub.com/doi/pdf/10.1177/2515245917746500][32] **Week 17** - June 5, 2020 @ 1:30 PM (GMT-4) *What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science* [https://journals.sagepub.com/doi/10.1177/1745691616646366][33] *** ### **ReproducibiliTea Toronto Mini Series:** A "conversation" between Tal Yarkoni and Daniël Lakens **Part 1 of 3** - June 12, 2020 @ 1:30 PM (GMT-4) (Week 18) [*The Generalizability Crisis*][34]; a preprint by Tal Yarkoni **Part 2 of 3** - June 19, 2020 @ 1:30 PM (GMT-4) (Week 19) [*Review of "The Generalizability Crisis" by Tal Yarkoni*][35]; a blog post by Daniël Lakens **Part 3 of 3** - June 26, 2020 @ 1:30 PM (GMT-4) (Week 20) [*Induction is not optional (if you’re using inferential statistics): reply to Lakens*][36]; a blog post by Tal Yarkoni *** [1]: https://calendar.google.com/calendar/u/0?cid=MzNiZjBlYWU4NzMyYWFiMTg0MTI2MWY5ZWY2NzlhN2JmNTgyMDAxNzBkNTdiZWI1MWQ4YTkwM2UwODYyNDI2OUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t [2]: https://calendar.google.com/calendar?cid=M3Uxb283bHZjZjU0MzExb2hkNWxma2s4MjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ [3]: https://osf.io/h3adt/ [4]: https://drive.google.com/drive/folders/1dNxV7TSIfxFSHvoPYqWRG87vcsJYaTLP?usp=sharing [5]: https://replicationindex.com/2019/08/13/when-personality-psychologists-are-high/ [6]: https://replicationindex.com/2019/08/13/when-personality-psychologists-are-high/ [7]: https://www.frontiersin.org/articles/10.3389/fpsyg.2019.00797/full [8]: https://journal.equinoxpub.com/JMTP/article/view/17104/20400 [9]: https://twitter.com/zerdeve/status/1234906668036608000 [10]: https://twitter.com/zerdeve [11]: https://psyarxiv.com/wxn58 [12]: https://psyarxiv.com/wxn58 [13]: https://www.tqmp.org/RegularArticles/vol16-4/p248/p248.pdf [14]: https://royalsocietypublishing.org/doi/10.1098/rsos.201520 [15]: https://osf.io/preprints/metaarxiv/sbv3q [16]: https://www.pnas.org/content/115/11/2584?ijkey=b785383d7b8d79876ed4fec4be93992ada5c50ca&keytype2=tf_ipsecsha [17]: https://www.nature.com/articles/s41562-016-0021 [18]: http://journals.sagepub.com/doi/abs/10.1177/0956797611417632 [19]: http://journals.sagepub.com/doi/abs/10.1177/0956797611430953 [20]: http://science.sciencemag.org/content/349/6251/aac4716 [21]: http://journals.sagepub.com/doi/pdf/10.1177/1745691612463401 [22]: https://psyarxiv.com/ak6jr [23]: http://rsos.royalsocietypublishing.org/content/3/9/160384 [24]: http://www.pnas.org/content/pnas/early/2018/03/08/1708274114.full.pdf [25]: https://github.com/philchalmers/SimDesign/wiki [26]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf [27]: http://www.stat.columbia.edu/~gelman/research/published/multiverse_published.pdf [28]: https://psyarxiv.com/hs7wm/ [29]: https://doi.org/10.1371/journal.pcbi.1005510 [30]: https://journals.sagepub.com/doi/pdf/10.1177/0098628318762900?casa_token=-otISLbatjIAAAAA:ijtyknmdprbA5xYIsKF7_qzW8ugt4gdx0t2D8iw4yGAq0FsNJ5HSgXBvSxgBKGolBAqCNNsWWdw [31]: https://journals.sagepub.com/doi/pdf/10.1177/2515245918776632?casa_token=1iG5YNtHYgIAAAAA:8HnvnF4JW5qTM95uqu6_Vplz3GyO5kiOJFtH19BgRPUO6ismdQiqRYGg41QRAtnFIfvMRTZSvCk [32]: https://journals.sagepub.com/doi/pdf/10.1177/2515245917746500 [33]: https://journals.sagepub.com/doi/10.1177/1745691616646366 [34]: https://psyarxiv.com/jqw35 [35]: http://daniellakens.blogspot.com/2020/01/review-of-generalizability-crisis-by.html [36]: https://www.talyarkoni.org/blog/2020/05/06/induction-is-not-optional-if-youre-using-inferential-statistics-reply-to-lakens/