# Welcome to the ReproducibiliTea Toronto Journal Club! This journal club is based at York University, Toronto, Canada. We aim to be as inclusive as possible. If you want to join our conversation during a particular meeting, then **just follow the link on the [Google Calendar event][1] to join the bi-weekly meetings!** **For Fall 2024, the zoom link will be: https://yorku.zoom.us/j/99113914933.** **For any general inquirites, or to request a specific article/source to discuss, please contact us at reproducibiliteatoronto@gmail.com.** **To be added to our mailing list (to receive emails to the club), please complete the following form: https://forms.gle/ri5gt3XANawjek419** *** ## **Readings** ### **Fall 2024** For the new Fall term, we have a unique selection of readings, with a large diversity of topics, from the merits of team science and statistical peer-review, to theory-salting and taking action on real-world issues such as climate change. As well, we have 3 guest speakers/facilitators this term! We hope you enjoy the selections and speakers! **The readings can be found on this OSF repository, in the *Fall 2024 Papers* folder.** ### Week 1 - October 2, 2024 @ 2:30 PM EST **Facilitator:** Gabriel Crone **Article**: Citizen science in the social sciences and humanities: the power of interdisciplinarity This paper introduces the idea of citizen science: having individuals contribute to large-scale research projects. It analyzes reports and challenges of citizen science in the social sciences, and concludes that while there are barriers, the payoffs to performing it well would be quite rewarding. ### Week 2 - October 16, 2024 @ 2:30 PM EST **Facilitator**: Gabriel Crone **Article:** Should Psychology Journals Adopt Specialized Statistical Review? This paper raises the possibility of including statistical peer-review, and if introduced, its merits as well as its challenges. ### Week 3 - October 30, 2024 @ 2:30 PM EST **Facilitator (Guest)**: Christine Moreau (Western ReproducibiliTea) **Article:** Conducting a Meta-Analysis in the Age of Open Science: Tools, Tips, and Practical Recommendations This preprint introduces the idea of open meta-analyses, meta-analyses designed using open-science principles. ### Week 4 - November 13, 2024 @ 2:30 PM EST **Facilitator (Guest)**: Lorne Hartman **Article:** Theory Salting in the Organizational Sciences This article discusses the idea of "theory salting", the notion that researchers might "sprinkle in" legitamate theories throughout their papers to give an impression of contributing to theory (when it really does not). ### Week 5 - November 27, 2024 @ 2:30 PM EST **Facilitator (Guest)**: Camille Proszanski **Article:** How can neuroscientists respond to the climate emergency? This applied article discusses neuroscientists' role in responding to the climate crisis. It will open up the discussion as to how other (seemingly unrelated) disciplines can weight in and affect positive change. # Previous Papers Archive: ### **Summer 2024** For the summer term, we have an excellent lineup of interesting talks, three of which will be facilitated by members of the previous semester's journal club! The topics are quite diverse, and include: running multiverse analyses in R, mis-citation in psychology articles, open-science platforms, expected non-results vs. HARKing, and the theoretical implications behind direct replications! We hope you find the selection to your liking! **Click [here][2] to read the articles! Below is a summary of each one:** **Week 1** - June 6, 2024 @ 3:00 PM EST *Increasing transparency through a multiverse analysis* This article introduces multiverse analyses. Researchers normally have a(n) (often unchecked) degree of freedom for the decisions they make: how you omit outliers, if at all? Which statistical model(s) should be run? How can variables be operationalized and/or computed? The idea behind the multiverse is that it runs the statistical model based on all of the possible configurations and decisions, providing a clear picture for how robust the final results really are. **Week 2** - June 20, 2024 @ 3:00 PM EST *The problem of mis-citation in psychological science: Righting the ship* Researchers cite articles to provide evidence in favor of their claims, and to support the basis behind their own research. But sometimes, articles might be *mis-cited*; that is, a citation might be inaccurately used or portrayed. Is this a problem of concern in psychology? According to Cobb et al., mis-citations are consequential, since they mislead and stifle research progress in the field. They argue that mis-citation should be considered a questionable research practice, and that when citing articles, psychologists should take care in ensuring that they accurately portray the cited sources. **Week 3** - July 4, 2024 @ 3:00 PM EST *Various open-science platforms: osf.io, aspredicted.org, PsyArXiv.* In this week, Ege Kamber, a member, will explore open-science platforms, including the open-science framework (OSF), as predicted, PsyArXiv, as well as many others! **Week 4** - July 18, 2024 @ 3:00 PM EST *HARKing: Hypothesizing After the Results are Known* and *HARKing can be good for science* The first article introduces the questionable research practice known as HARKing: hypothesizing after results are known. This occurs when researchers determine their results, then modify their manuscript in a way that made it seem as if the results are what they had predicted all along. The second article argues for a broader definition of HARKing: creating new hypotheses based on the found results. It further contends that hypothesizing after results are known, when done in an appropriate context, is essential for scientific progress. The facilitator, Lorne Hartman, will briefly discuss these articles, as well as a case illustration from his own research. **Week 5** - August 1, 2024 *The alleged crisis and the illusion of exact replication* This article discusses whether direct replications in psychology should be considered the essential "litmus test" for replicability. It contends that direct replication, while perceived as theoretically ideal, are in fact impossible to perform. Harley Glassman will facilitate this discussion. ### **Winter 2024** For the Winter semester, we'll be chatting about more specific topics as they relate to reproducibility, primarily the incentive structure in science, null hypothesis significance testing (NHST), and why common arguments made against reproducibility are misguided. **Once more, all articles are made freely available on the shared [Google Drive Folder][3]. Winter readings can be found within the "Winter" sub-folder. Below is a summary of each reading.** **Week 1** - Jan 23, 2024 @ 2:30 PM EST *The incentive structure in science: Scientific Utopia II: Restructuring Incentives and Practices to Promote Truth Over Publishability* A paper that explains what's wrong with the incentive structure in science, and how the structure can be changed to promote more open-scientific attitudes and practices. **Week 2** - Feb 6, 2024 @ 2:30 PM EST *Mindless Statistics* This paper describes the (disturbing) origins of null hypothesis significance testing (NHST), the statistical framework that dominates psychology. It makes the evocative argument that NHST is treated less like a statistical framework and more like a mindless ritual. **Week 3** - Mar 5, 2024 @ 2:30 PM EST *The Earth is Round (p < .05)* This classic paper describes the issues with null hypothesis significance testing as a framework in psychology, such as in how it makes a logically invalid argument and is highly prone to misinterpretations. **Week 4** - Mar 19, 2024 @ 2:30 PM EST *Life after NHST: How to Describe Data Without "p-ing" Everywhere* This paper explores ways of describing data without using NHST, one potential manner in which psychologists can move away from NHST in their own research. **Week 5** - Apr 2, 2024 @ 2:30 PM EST *Is the Replication Crisis Overblown? Three Arguments Examined* Paper examining potential counterarguments to the idea that the reproducibility crisis is overexaggerated, and which provides rebuttals to said common arguments. In doing so, it reinforces the importance of open science and reproducibility. ### **Fall 2023** For the re-launch of the ReproducibiliTea Toronto Club, we'll be going through a lot of the classic reproducibility papers, along with some goodies sprinkled in! :) **All articles are made freely available on the shared [Google Drive Folder][4]. Below is a summary of each reading.** **Week 1** - Oct 18, 2023 @ 10:00 AM EST 1. *A Manifesto for Reproducible Science* 2. *Estimating the Reproducibility of Psychology* Two excellent papers to get everyone acquainted with the idea and importance of reproducibility. **Week 2** - Nov 1, 2023 @ 10:00 AM EST *Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology* Paper on a recent attempt at computationally reproducing the statistical analyses within several journal articles. **Week 3** - Nov 15, 2023 @ 10:00 AM EST *Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling* Classic paper about the prevalence of questionable research practices (QRPs) in psychology and what it means for the discipline. **Week 4** - Nov. 29, 2023 1. *UK Psychology PhD researchers’ knowledge, perceptions, and experiences of open science* 2. *Attitudes Toward Open Science and Public Data Sharing* Papers exploring what students (paper 1) and professors (paper 2) think about the ideas of open science and reproducibility. **Week 5** - Dec. 13, 2023 *Is the Replication Crisis Overblown? Three Arguments Examined* Paper examining potential counterarguments to the idea that the reproducibility crisis is overexaggerated. *** ## Readings from Previous Sessions ### Fall 2020-Winter 2021 **Week 1** - Oct 30, 2020 @ 2:30 PM EST *When Personality Psychologists are High* Post from Dr. Ulrich Schimmack’s Blog about Replicability [https://replicationindex.com/2019/08/13/when-personality-psychologists-are-high/][5] **Week 2** - Nov 13, 2020 @ 2:30 PM EST *Personality Measurement with the Big Five Inventory* [https://replicationindex.com/2019/08/10/personality-measurement-with-the-big-five-inventory/][6] Post from Dr. Ulrich Schimmack’s Blog about Replicability **Week 3** - Nov 27, 2020 @ 2:30 PM EST *When a “Replication” Is Not a Replication. Commentary: Sequential Congruency Effects in Monolingual and Bilingual Adult* [https://www.frontiersin.org/articles/10.3389/fpsyg.2019.00797/full][7] Supplemental reading: [Null results in bilingualism research:What they tell us and what they don’t][8] Many of our readings this semester were pulled from a fantastic [Twitter thread][9] by [@zerdeve][10] highlighting some non-mainstream work on reproducibility, open science, replication crisis, meta-science by women **Week 4** - Jan 22, 2021 @ 2:30 PM EST *Paths in strange spaces: A comment on preregistration*: Part 1 A blog post by Daniel Navarro preserved as a preprint on psyarxiv. [https://psyarxiv.com/wxn58][11] **Week 5** - Feb 5, 2021 @ 2:30 PM EST *Paths in strange spaces: A comment on preregistration*: Part 2 A blog post by Daniel Navarro preserved as a preprint on psyarxiv. [https://psyarxiv.com/wxn58][12] **Week 6** - Feb 19, 2021 @ 1:oo PM EST Workshop on conducting Monte Carlo Simulations using the SimDesign package. [https://www.tqmp.org/RegularArticles/vol16-4/p248/p248.pdf][13] **Week 7** - Mar 5, 2021 @ 2:3o PM EST *Credibility of preprints: an interdisciplinary survey of researchers* [https://royalsocietypublishing.org/doi/10.1098/rsos.201520][14] **Week 8** - Mar 19, 2021 @ 2:30 PM EST *Mapping the discursive dimensions of the reproducibility crisis: A mixed methods analysis* [https://osf.io/preprints/metaarxiv/sbv3q][15] - Note might have to be read within your web broswer as the pdf had issues opening using some software. **Week 9** - Apr 2, 2021 @ 2:3o PM EST *An empirical analysis of journal policy effectiveness for computational reproducibility* [https://www.pnas.org/content/115/11/2584?ijkey=b785383d7b8d79876ed4fec4be93992ada5c50ca&keytype2=tf_ipsecsha][16] *** ### Winter 2020- Summer 2020 **Week 1** - Jan 17, 2020 @ 2:30 PM *A Manifesto for Reproducible Science* The Problem Defined: The general overview [https://www.nature.com/articles/s41562-016-0021][17] **Week 2** - Jan 24, 2020 @ 2:30 PM *False Positive Psychology* Examining Analytic Flexibility And why it is a problem [http://journals.sagepub.com/doi/abs/10.1177/0956797611417632][18] **Week 3** - Jan 31, 2020 @ 2:30 PM *Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling* Questionable Research Practices: are they really that common? And why are they problematic? [http://journals.sagepub.com/doi/abs/10.1177/0956797611430953][19] **Week 4** - Feb 7, 2020 @ 2:30 PM *Estimating the Reproducibility of Psychological Science* Reproducibility Now: Why many studies are not reproducible. [http://science.sciencemag.org/content/349/6251/aac4716][20] **Week 5** - Feb 14, 2020 @ 2:30 PM *Is the Replicability Crisis Overblown? Three Arguments Examined* How has the debate gone to far? Things will just turn out fine. [http://journals.sagepub.com/doi/pdf/10.1177/1745691612463401][21] **Week 6** - Feb 28, 2020 @ 2:30 PM *Open Science: What, Why and How Open Data and Materials* [https://psyarxiv.com/ak6jr][22] **Week 7** - Mar 6, 2020 @ 2:30 PM *The Natural Selection of Bad Science* And what about the future? [http://rsos.royalsocietypublishing.org/content/3/9/160384][23] **Week 8** - Mar 13, 2020 @ 2:30 PM *The Preregistration Revolution* Preregistration as a solution [http://www.pnas.org/content/pnas/early/2018/03/08/1708274114.full.pdf][24] **Week 9** - Mar 20, 2020 @ 2:30 PM *Introduction to Monte Carlo Simulation* A brief workshop about running Monte Carlo Simulations using the SimDesign package. [https://github.com/philchalmers/SimDesign/wiki][25] **Week 10** - Mar 27, 2020 @ 2:30 PM *The garden of forking paths* Researcher degrees of freedom [http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf][26] **Week 11** - Apr 3, 2020 @ 2:30 PM *Increasing Transparency Through a Multiverse Analysis* [http://www.stat.columbia.edu/~gelman/research/published/multiverse_published.pdf][27] **Week 12** - Apr 17, 2020 @ 1:30 PM *Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them* [https://psyarxiv.com/hs7wm/][28] **Week 13** - May 1, 2020 @ 1:30 PM *Good enough practices in scientific computing* [https://doi.org/10.1371/journal.pcbi.1005510][29] **Week 14** - May 8, 2020 @ 1:30 PM (GMT-4) *How (and Whether) to Teach Undergraduates About the Replication Crisis in Psychological Science* [https://journals.sagepub.com/doi/pdf/10.1177/0098628318762900?casa_token=-otISLbatjIAAAAA:ijtyknmdprbA5xYIsKF7_qzW8ugt4gdx0t2D8iw4yGAq0FsNJ5HSgXBvSxgBKGolBAqCNNsWWdw][30] **Week 15** - May 15, 2020 @ 1:30 PM (GMT-4) *Fallibility in Science: Responding to Errors in the Work of Oneself and Others* [https://journals.sagepub.com/doi/pdf/10.1177/2515245918776632?casa_token=1iG5YNtHYgIAAAAA:8HnvnF4JW5qTM95uqu6_Vplz3GyO5kiOJFtH19BgRPUO6ismdQiqRYGg41QRAtnFIfvMRTZSvCk][31] **Week 16** - May 22, 2020 @ 1:30 PM (GMT-4) *Practical Solutions for Sharing Data and Materials From Psychological Research* [https://journals.sagepub.com/doi/pdf/10.1177/2515245917746500][32] **Week 17** - June 5, 2020 @ 1:30 PM (GMT-4) *What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science* [https://journals.sagepub.com/doi/10.1177/1745691616646366][33] *** ### **ReproducibiliTea Toronto Mini Series:** A "conversation" between Tal Yarkoni and Daniël Lakens **Part 1 of 3** - June 12, 2020 @ 1:30 PM (GMT-4) (Week 18) [*The Generalizability Crisis*][34]; a preprint by Tal Yarkoni **Part 2 of 3** - June 19, 2020 @ 1:30 PM (GMT-4) (Week 19) [*Review of "The Generalizability Crisis" by Tal Yarkoni*][35]; a blog post by Daniël Lakens **Part 3 of 3** - June 26, 2020 @ 1:30 PM (GMT-4) (Week 20) [*Induction is not optional (if you’re using inferential statistics): reply to Lakens*][36]; a blog post by Tal Yarkoni *** [1]: https://calendar.google.com/calendar/u/0?cid=MzNiZjBlYWU4NzMyYWFiMTg0MTI2MWY5ZWY2NzlhN2JmNTgyMDAxNzBkNTdiZWI1MWQ4YTkwM2UwODYyNDI2OUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t [2]: https://drive.google.com/drive/folders/1dBhxNTpLNLhsvAZB5dzwVNdOSXIYkL07?usp=sharing [3]: https://drive.google.com/drive/folders/1dNxV7TSIfxFSHvoPYqWRG87vcsJYaTLP?usp=sharing [4]: https://drive.google.com/drive/folders/1dNxV7TSIfxFSHvoPYqWRG87vcsJYaTLP?usp=sharing [5]: https://replicationindex.com/2019/08/13/when-personality-psychologists-are-high/ [6]: https://replicationindex.com/2019/08/13/when-personality-psychologists-are-high/ [7]: https://www.frontiersin.org/articles/10.3389/fpsyg.2019.00797/full [8]: https://journal.equinoxpub.com/JMTP/article/view/17104/20400 [9]: https://twitter.com/zerdeve/status/1234906668036608000 [10]: https://twitter.com/zerdeve [11]: https://psyarxiv.com/wxn58 [12]: https://psyarxiv.com/wxn58 [13]: https://www.tqmp.org/RegularArticles/vol16-4/p248/p248.pdf [14]: https://royalsocietypublishing.org/doi/10.1098/rsos.201520 [15]: https://osf.io/preprints/metaarxiv/sbv3q [16]: https://www.pnas.org/content/115/11/2584?ijkey=b785383d7b8d79876ed4fec4be93992ada5c50ca&keytype2=tf_ipsecsha [17]: https://www.nature.com/articles/s41562-016-0021 [18]: http://journals.sagepub.com/doi/abs/10.1177/0956797611417632 [19]: http://journals.sagepub.com/doi/abs/10.1177/0956797611430953 [20]: http://science.sciencemag.org/content/349/6251/aac4716 [21]: http://journals.sagepub.com/doi/pdf/10.1177/1745691612463401 [22]: https://psyarxiv.com/ak6jr [23]: http://rsos.royalsocietypublishing.org/content/3/9/160384 [24]: http://www.pnas.org/content/pnas/early/2018/03/08/1708274114.full.pdf [25]: https://github.com/philchalmers/SimDesign/wiki [26]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf [27]: http://www.stat.columbia.edu/~gelman/research/published/multiverse_published.pdf [28]: https://psyarxiv.com/hs7wm/ [29]: https://doi.org/10.1371/journal.pcbi.1005510 [30]: https://journals.sagepub.com/doi/pdf/10.1177/0098628318762900?casa_token=-otISLbatjIAAAAA:ijtyknmdprbA5xYIsKF7_qzW8ugt4gdx0t2D8iw4yGAq0FsNJ5HSgXBvSxgBKGolBAqCNNsWWdw [31]: https://journals.sagepub.com/doi/pdf/10.1177/2515245918776632?casa_token=1iG5YNtHYgIAAAAA:8HnvnF4JW5qTM95uqu6_Vplz3GyO5kiOJFtH19BgRPUO6ismdQiqRYGg41QRAtnFIfvMRTZSvCk [32]: https://journals.sagepub.com/doi/pdf/10.1177/2515245917746500 [33]: https://journals.sagepub.com/doi/10.1177/1745691616646366 [34]: https://psyarxiv.com/jqw35 [35]: http://daniellakens.blogspot.com/2020/01/review-of-generalizability-crisis-by.html [36]: https://www.talyarkoni.org/blog/2020/05/06/induction-is-not-optional-if-youre-using-inferential-statistics-reply-to-lakens/