**Module Description** Our definition of "good" scientific practices is constantly evolving. Over the last decade, many common practices have been found to negatively impact the replicability of psychological science. This module highlights currently recognized Questionable Research Practices (QRPs), their prevalence, and the problems they introduce. **Learning Objectives** - Be able to identify QRPs: - Leaving out participants, conditions, variables, or studies that are not in line with your hypothesis - Optional stopping - Presenting confirmatory findings as exploratory - Wrongly “rounding down” p-values (e.g., reporting p = .054 as p < .05) - HARKing - Gain insight into personal research practices that may be questionable - Understand the prevalence of QRPs - Be able to identify and discuss the problems that QRPs introduce - Understand reasons why researchers engage in QRPs - Are we all evil? (Perhaps this part would be brief; link to the other module on perverse incentives) **Readings** Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False positive psychology: Flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological science, 0956797611430953. Ebersole, C. R., Axt, J. R., & Nosek, B. A. (2016). Scientists’ reputations are based on getting it right, not being right. PLoS Biol, 14(5), e1002460. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543-554. doi:10.1177/1745691612459060 **Blogs** Schimmack, U. (2015, January). [Questionable research practices: Definition, detect, and recommendations for better practices.][1] Roberts, B. (2011, November). [Personality psychology has a serious problem (and so do many other Areas of psychology)][2]. Inzlicht, M. (2015, April). [Check yourself before you wreck yourself.][3] **Demonstrations** Aschwanden, C. (2015). [Science isn't broken.][4] Schoenbrodt, F. (2016). [Introducing the p-hacker app: Train your expert p-hacking skills.][5] Schönbrodt, F. D. (2015). [p-checker: One-for-all p-value analyzer.][6] - <b> Simulations on selective reporting </b> 1. Similated data from H0: Results are from H0 (no effect), only significant (i.e., false positive) results are published. 2. Simulated data from H1 (d=0.5): Results are from H1 (d=0.5), only significant results are published. 3. Simulated data from H0 + p-hacking: Results are from H0 (no effect), data are p-hacked with various techniques, only significant (i.e., false positive) results are published. **Assignments** - Anonymous survey: what QRPs have you engaged in? (pre-class exercise) - p-hacking exercise (link) - Reflective paper: considering your research area, what are the most relevant QRPs you can identify? Which could be avoided, and which could not? (post-class exercise) **Discussion Questions** - What are some of the reasons that researchers engage in QRPs? Are there different reasons in different sub-areas of psychology? Are there some common reasons across sub-areas? - What are the QRPs that you are most surprised by? Are there QRPs that you did not realize were questionable? - Should there be consequences for engaging in QRPs? If so, what would they be? - Should researchers be recognized or rewarded for not engaging in QRPs (i.e., for using better practices?) If so, how? - Now that you have learned about QRPs, how might this impact how you interpret the studies that you read? - At what stage of the research process can "flexibility" happen or is most present? (consider when reading the Simmons, Nelson, Simonsohn, 2011 article). **Slides** Schweinsberg, M., Bearden, N., Uhlmann, E. L. (2016). Crowdsourcing Science PhD Course slides and syllabus. Retrieved from https://osf.io/hj9zr/ Schönbrodt, F.D. (date unknown). Questionable research practices. Retrieved from https://osf.io/bh7zv/?action=download&version=1 [1]: https://replicationindex.wordpress.com/2015/01/24/questionable-research-practices-definition-detect-and-recommendations-for-better-practices/ [2]: http://www.personality-arp.org/html/newsletter06/problem.html [3]: http://sometimesimwrong.typepad.com/wrong/2015/04/guest-post-check-yourself-before-you-wreck-yourself.html [4]: https://fivethirtyeight.com/features/science-isnt-broken/#part1 [5]: http://www.nicebread.de/introducing-p-hacker/ [6]: http://shinyapps.org/apps/p-checker/