Stage 2 – Replication Study --------------------------- **Problem statement** Recent reports on the lack of reproducibility of important psychological findings and growing evidence for a systematic positive bias in the published research reports is often interpreted as a ‘confidence crisis’ in psychological science. One of the factors that has been suggested to lie behind this calamity is a collection of ‘questionable research practices’ (QRPs, Steneck, 2006). Studies testing controversial hypotheses suffer even more of the burden of the confidence crisis, because QRPs offer a convenient and parsimonious explanation for anomalous findings, especially if they do not fit the status quo theories. Recently, several publications reported positive results in support of ‘psi’ phenomena in high profile psychology journals (Bem, 2011; Storm, Tressoldi, & Di Risio, 2010) Nevertheless, these reports met poor reception. The interpretation that the results would be evidence for extrasensory perception (ESP) was criticized on several accounts (Fiedler & Krueger, 2013; Rouder & Morey, 2011; Schwarzkopf, 2014), and those who offered a counter-explanation for the positive findings usually mentioned some type of QRP, and problems with the execution of the studies (Wagenmakers, Wetzels, Borsboom, Kievit, & van der Maas, 2015; Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011). Such low level of confidence in the quality of studies can lead to the unwarranted dismissal of good research findings. This problem is prominent in ESP research, but it can be easily generalized to other controversial research areas and to science in general. One detrimental effect of low credibility for science is that researchers have to spend valuable resources on verifying published findings instead of being able to accept them as objective observations, or they may choose to disregard reports altogether because of unverifiable credibility, making the original study a waste of resources. Therefore, there is a critical need for methodological approaches that can increase the credibility and acceptability of research, and we need to set up clear criteria for credibility that can be adhered to in future scientific ventures. One possible solution to improve credibility of research findings is to eliminate opportunities for QRPs. Recently, there has been a rise in initiatives for transparency which could help reduce the prevalence of QRPs, such as encouragement of pre-registration of experiments by professional and governmental agencies, a call to publish research reports irrespective of their outcome, making reports available to a wider audience through self-archiving and open access publication, open publishing platforms, data repositories, and initiatives for large scale multi-lab replications. Furthermore, [best practice guidelines][1] have been set up to further improve credibility and integrity of research. However, inconsistencies in protocol delivery, result driven exclusion or imputation of data, and fraud are not prevented by these interventions and guidelines, because the study and data collection itself is still performed ‘in the dark’. So, to eliminate this last safe haven for QRPs and sloppy protocol execution, further innovations are in order. **Solution** In our project we implement methodologies that promote verifiable credibility. Specifically, we propose to use: - Real-time born open data: as data is being collected, it is immediately in real-time being pushed through to a publicly accessable database keeping an audit trail, ensuring data integrity. - Consensus Design Process. - Consensus-based pre-registration of conclusions for each possible outcome of the study related to the confirmatory hypothesis testing. - Verifying as-intended protocol delivery through video recordings of scripted trial sessions. - Validation of software codes by an independent party. - Involving independent research auditors to verify study integrity. **Specific Aim** In Stage 2 of our research project our primary aim is to carry out a replication of Bem’s (2011) Experiment 1 in a manner that incorporates the above mentioned credibility improving techniques, thus, making study integrity verifiable. [See our detailed research plan here][2] **Funding** The research program is funded by the Bial Foundation. ![Bial Foundation logo][3] **References** Bem, D. J. (2011). Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407-425. doi:10.1037/a0021524 Fiedler, K., & Krueger, J. I. (2013). Afterthoughts on precognition: No cogent evidence for anomalous influences of consequent events on preceding cognition. Theory & Psychology, 23(3), 323-333. doi:10.1177/0959354313485504 Rouder, J. N., & Morey, R. D. (2011). A Bayes factor meta-analysis of Bem’s ESP claim. Psychonomic Bulletin & Review, 18(4), 682-689. doi:10.3758/s13423-011-0088-7 Schwarzkopf, D. S. (2014). We should have seen this coming. Frontiers in human neuroscience, 8(332). doi:10.3389/fnhum.2014.00332 Steneck, N. H. (2006). Fostering integrity in research: Definitions, current knowledge, and future directions. Science and engineering ethics, 12(1), 53-74. Storm, L., Tressoldi, P. E., & Di Risio, L. (2010). Meta-analysis of free-response studies, 1992–2008: Assessing the noise reduction model in parapsychology. Psychological Bulletin, 136(4), 471-485. doi:http://dx.doi.org/10.1037/a0019457 Wagenmakers, E.-J., Wetzels, R., Borsboom, D., Kievit, R., & van der Maas, H. L. J. (2015). A skeptical eye on psi. In E. May & S. B. Marwaha (Eds.), Extrasensory Perception: Support, Skepticism, and Science (pp. 153-176). NB: ABC-CLIO. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. (2011). Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). Journal of Personality and Social Psychology, 100(3), 426-432. doi:10.1037/a0022790 [1]: https://osf.io/jk2zf/wiki/Best%20practice%20guidelines%20to%20improve%20research%20integrity/ [2]: https://osf.io/sb2dt/wiki/00%20Executive%20Summary/ [3]: https://static1.squarespace.com/static/533155cae4b08b6d16d34ec4/t/555c9b76e4b0e543f2e387b2/1432132471283/?format=300w