This page has resources for the "Training in Statistical Thinking" presentation by Bob Calin-Jageman at the Professional Development Workshop on "Neuroscience Training for the Future" given Sunday, October 6 at the 2024 Annual Meeting of the Society for Neuroscience. The presentation ---------------- * [Slides][1] * [Recording][2] Some excellent general sources ---------------- * The American Statistical Association has a set of guidelines and princples for statistics education (GAISE) for both primary instruction and for higher ed. This is an essential starting point! The current guidelines are from 2016 and the ASA is working on a revision, which is well worth following: https://www.amstat.org/education/guidelines-for-assessment-and-instruction-in-statistics-education-(gaise)-reports * This paper gives some general suggestions for statistical practice and offers lots of great ideas and references that would be useful for shaping a modern statistics curriculum.. Wagenmakers, Eric-Jan, Alexandra Sarafoglou, Sil Aarts, Casper Albers, Johannes Algermissen, Štěpán Bahník, Noah van Dongen, et al. “Seven Steps toward More Transparency in Statistical Practice.” Nature Human Behaviour 5, no. 11 (November 2021): 1473–80. https://doi.org/10.1038/s41562-021-01211-8. * This paper and talk give a bracing perspective on the issues of multiplicity. Benjamini, Yoav. “Selective Inference: The Silent Killer of Replicability.” Harvard Data Science Review 2, no. 4 (July 15, 2020). https://doi.org/10.1162/99608f92.fc62b261. * This paper summarizes a special issue of the American Statistician debating what can be done to improve or replace reliance on p values. There is lots of controversey about the value of p values, but the overall advice distilled here seems pretty darn reasonable regardless of your take on p values. Wasserstein, Ronald L., Allen L. Schirm, and Nicole A. Lazar. “Moving to a World Beyond ‘ p < 0.05.’” The American Statistician 73, no. sup1 (March 29, 2019): 1–19. https://doi.org/10.1080/00031305.2019.1583913. * The arrive guidelines provide some eminently sensible suggestions for how to report scientific research. In the ideal world, our neuroscience trainees would meet these standards as second nature, which means they should be baked into our stats curriculum: https://arriveguidelines.org/ * This paper is on how to improve reproducibility in fMRI studies. It highlights issues and challenges that are not always limited to fMRI specifically, and if we think of the recs in this paper as the end goal, it's worth thinking about what stats curriculum would produce trainees who meet these goals as a matter of course. Poldrack, Russell A., Chris I. Baker, Joke Durnez, Krzysztof J. Gorgolewski, Paul M. Matthews, Marcus R. Munafò, Thomas E. Nichols, Jean-Baptiste Poline, Edward Vul, and Tal Yarkoni. “Scanning the Horizon: Towards Transparent and Reproducible Neuroimaging Research.” Nature Reviews Neuroscience 18, no. 2 (February 2017): 115–26. https://doi.org/10.1038/nrn.2016.167. Estimation thinking ---------------- I'm partial to my own paper explaining why estimation is needed in the neurosciences (Calin-Jageman & Cumming, 2019; https://doi.org/10.1523/ENEURO.0205-19.2019). Estimation is completely compatible with Bayesian statistics; this overview is accessible and authoritative (Kruschke et al., 2018; https://doi.org/10.3758/s13423-016-1221-4). Textbooks: * I am co-author on an [undergraduate textbook][3] that puts estimation first, now in its second edition. * Kruschke's [textbook on Bayesian estimation][4] is fantastic. Software: * The statpsych package for R is elegant and authoritative. It can provide effects sizes and confidence intervals for many different types of studies. It also has functions for planning for precision. https://cran.r-project.org/web/packages/statpsych/index.html * esci is a package in R built on top of statpsych. It provides beautiful visualizations that focus on effect size. https://cran.r-project.org/web/packages/esci/ * DABEST is available for python, R, and online. It provides bootstrapped effect sizes and confidence intervals for many designs, along with outrageously gorgeous visualizations. https://acclab.github.io/DABEST-python/ Simulations for Learning and Planning ---------------- For learning: * The dance of the means and the dance of the R values from Cumming, Calin-Jageman and Moore. https://esci.thenewstatistics.com/ * A very cool version of the dance of the means by Kristoffer Magnusson: https://rpsychologist.com/d3/ci/ * OpenIntro has an open-source textbook that features simulation for developing understanding (Intro Statistics with Randomization and Simulation) that has now been updated (Introduction to Modern Statistics) * There has been a movement within statistics education to focus on simulation to teach primarily re-sampling methods. This is a cool approach, and there is some data suggesting it can improve student understanding. See, for example, Rossman & Chase (2014; https://doi.org/10.1002/wics.1302), and a blog they used to run which now throws a security error, but which has lots of good resources: https://www.causeweb.org/sbi/?page_id=176 For planning: * The faux package for R is amazing: https://debruine.github.io/faux/ * A great tutorial for simulation-based sample-size planning for tests using an interval null (Riesthuis, 2024; /doi.org/10.1177/25152459241240722). * A tutorial for simulation-based planning for mixed models, which can help you deal with nested data: Koch et al., 2023; https://doi.org/10.31234/osf.io/rpjem * This may also be useful: Zimmer et al.; 2022 https://doi.org/10.31234/osf.io/r9w6t Robustness Checking ---------------- * Multiverse analysis overview and tutorial: Steegan, Sara, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. “Increasing Transparency through a Multiverse Analysis Manuscript.” Perspectives on Psychological Science, 2016. https://doi.org/10.1177/1745691616658637. * Tutorial on cross-validation: Rooij, Mark de, and Wouter Weeda. “Cross-Validation: A Method Every Psychologist Should Know.” Advances in Methods and Practices in Psychological Science 3, no. 2 (June 1, 2020): 248–63. https://doi.org/10.1177/2515245919898466. * Model validation suggestions for neural networks: Gutzen, Robin, Michael von Papen, Guido Trensch, Pietro Quaglio, Sonja Grün, and Michael Denker. “Reproducible Neural Network Simulations: Statistical Methods for Model Validation on the Level of Network Activity Data.” Frontiers in Neuroinformatics 12 (December 19, 2018). https://doi.org/10.3389/fninf.2018.00090. * Two papers on how to clearly demarcate exploratory vs. confirmatory research, and approaches for deriving predictions from exploration to efficiently test via confirmation. Höfler, Michael, Brennan McDonald, Philipp Kanske, and Robert Miller. “Means to Valuable Exploration II: How to Explore Data to Modify Existing Claims and Create New Ones.” Meta-Psychology 7 (July 10, 2023). https://doi.org/10.15626/MP.2022.3270. Höfler, Michael, Stefan Scherbaum, Philipp Kanske, Brennan McDonald, and Robert Miller. “Means to Valuable Exploration: I. The Blending of Confirmation and Exploration and How to Resolve It.” Meta-Psychology 6 (November 8, 2022). https://doi.org/10.15626/MP.2021.2837. Multiplicity and Interdependence ---------------- * This paper and talk by Yoav Benjamini on is really useful: https://doi.org/10.1162/99608f92.fc62b261 * For hierarchical/nested data, the go to reference in neuroscience is Aarts et al., 2014; https://doi.org/10.1038/nn.3648; they reccomend a multi-level modeling approach. Note, though, that some of their simulated results are contested and likely erroneous. * McNabb et al., 2021 explain a limitation in Aarts' simulations and show that sufficient summary statistics approach can be viable; http://doi.org/10.1016/j.crneur.2021.100024 * A thorough discussion of the sufficient summary statistics approach is provided by Dowding & Haufe, 2018; https://10.3389/fnhum.2018.00103 * Hierarchical bootstrap may also be a useful strategy for hierarchical data: Saravanan et al., 2021; https://nbdt.scholasticahq.com/article/13927-application-of-the-hierarchical-bootstrap-to-multi-level-data-in-neuroscience Open Science ---------------- * The open science framework: https://osf.io/, a great tool and gateway to the resources and workshops from the Open Science Foundation. * OSF training modules: https://www.cos.io/services/training * NASA's open science training: https://www.nasa.gov/news-release/new-course-from-nasa-helps-build-open-inclusive-science-community/ * Open Science 101 from NeuroMatch: https://neuromatch.io/open-science-101-course/ The Neglected Factors ---------------- This is phrase from McShane et al. (2017) - that we should stop focusing so exclusively on p values and consider more broadly the quality of the design, relaibility and validity of the manipulatin and measurement, plausibility of the mechanism, and more-- that scientific truth should be a much broader concern than checking p values. http://www.stat.columbia.edu/~gelman/research/unpublished/abandon.pdf To me, this phrase also points to the fact that we should teach good design and measurment principles, and that we can create more interpretable and meaningful results when we better deploy good design (match controls, good positive and negative controls), improve our measurements, etc. Here are some readable sources on how you can improve the sensitivity of your studies without increasing sample size: * Kraemer, H C. “To Increase Power in Randomized Clinical Trials without Increasing Sample Size.” Psychopharmacology Bulletin 27, no. 3 (1991): 217–24. * Lazic, Stanley E. “Four Simple Ways to Increase Power without Increasing the Sample Size.” Laboratory Animals 52, no. 6 (December 1, 2018): 621–29. https://doi.org/10.1177/0023677218767478. * MacKinnon, Sean. “Increasing Statistical Power in Psychological Research without Increasing Sample Size.” Center for Open Science, 2013. http://osc.centerforopenscience.org/2013/11/03/Increasing-statistical-power/. * Meyvis, Tom, and Stijn M.J. Van Osselaer. “Increasing the Power of Your Study by Increasing the Effect Size.” Journal of Consumer Research 44, no. 5 (2018): 1157–73. https://doi.org/10.1093/jcr/ucx110. [1]: https://osf.io/jdqfb [2]: https://youtu.be/ZXnZ2NrNJz0 [3]: https://www.routledge.com/Introduction-to-the-New-Statistics-Estimation-Open-Science-and-Beyond/Cumming-Calin-Jageman/p/book/9780367531508 [4]: https://sites.google.com/site/doingbayesiandataanalysis/