# Reproducipedia # Important terms related to reproducibility explained. by Simon Schwab @[toc] ## B ## ### Bias ### A bias is to lean towards a belief which is not necessarily factual. Science ideally is objective and based on facts. However, there are some well-known biases, see confirmation bias, publication bias, and reporting bias. ## C ## ### Code sharing ### Sharing of data cleaning code, analysis code, and software to generate the results. Ideally, the complete set of code is shared that generates the results in the publication, including all figures and tables. This is possible with modern approaches to data analysis such as analysis notebooks for popular languages (e.g, Jupyter Notebooks for Pythons, or R Notebooks). ### Conceptual replication ### A distinction between direct and conceptual replication has been made. In a conceptual replication, a different methodology is used to test the same hypothesis. Conceptual replication can establish confidence in results because they are independent of methodology. In contrast, a direct replication can demonstrate that results are reproducible, but this does not guarantee validity. For example, a methodology that is repeated could have confounds. See also direct replication. *Further reading:* Nosek, B. A., & Errington, T. M. (2017). Making sense of replications. *eLife*, 6. https://doi.org/10.7554/eLife.23383 ### Confirmation bias ### Scientists, and generally all human beings, have an innate tendency to confirm their expectations. Scientists tend to seek confirmatory evidence instead of trying to disprove their hypothesis. See also observer bias. ### Confounding ### Confounding (also confounder or confounding variable) is when a independent variable is mixed up with another variable, the confounding variable. As a result is is hard to determine the effect that the independent variable has on the outcome or dependent variable. For example, in an observational study with a patient and a control group where the patient group are older in average. Any difference between the patients and controls can not be clearly associated with the disease because age is a confound. ## D ## ### Data sharing ### ### Direct replication ### A distinction between direct and conceptual replication has been made. In a direct replication, protocols from the original study are followed with new data. See also conceptual replication. *Further reading:* Nosek, B. A., & Errington, T. M. (2017). Making sense of replications. *eLife*, 6. https://doi.org/10.7554/eLife.23383 ## E ## ### External validity ### The extent to which the conclusions of the study can be generalized to other people, i.e. the population. ## F ## ### File-drawer problem ### See publication bias. ### Flexibility in data analysis ### Also known as p-hacking, researcher degrees of freedom, data dredging, data fishing. It refers to the misuse of data analysis to produce a significant p-value. Exhaustive searching by testing large number of models with different subsets of variables, outlier removal, perform one-sided tests without justification, etc ## H ## ### Harking or HARKing ### Hypothesizing After the Results are Know. ## I ## ### Inferential reproducibility ### Refers to the drawing of qualitatively similar conclusions from either an independent replication study or a reanalysis of the original study. *Further reading:* Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science *Translational Medicine*, 8(341), 341ps12. https://doi.org/10.1126/scitranslmed.aaf5027 ## L ## ### Low power ### If the statistical power of a study is low, a study is unlikely to find an effect given that there really is a true effect. In other words, studies with low power tend to produce negative results (i.e., failure to reject the null hypothesis). This is in conflict with the fact that many disciplines report a large proportion of positive results, however, often are underpowered at the same time. See also publication bias and power analysis. *Further reading:* Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews. *Neuroscience*, 14(5), 365–376. https://doi.org/10.1038/nrn3475 ## M ## ### Method reproducibility ### Refers to the provision of enough methodological detail about study procedures and data so the analysis could be exactly repeated. This requires an understanding of the dataset and of which and how many analyses were performed and how the particular analyses were reported in the published paper. See also data sharing and code sharing. *Further reading:* Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science *Translational Medicine*, 8(341), 341ps12. https://doi.org/10.1126/scitranslmed.aaf5027 ## O ## ### Observer bias ### Occurs when researchers alter the outcome of a study due to their expectations. For example, a psychotherapist conducting a study to compare two types of therapies has prior beliefs about which therapy is supposed to be more effective. In turn, the psychotherapist may behave differently, for example, spend more time with one of the groups. The most effective way to avoid observer bias is to use double blind studies, where neither the researcher nor the subjects know what is being studied. This, however, is often impossible in psychological research. ## P ## ### p-hacking ### See flexibility in data analysis. ### Preregistration ### Submit hypotheses and plans for data analysis to a third party before performing experiments to prevent cherry picking statistically significant results later. ### Power analysis ### A power analysis is used to determine the sample size required to detect an effect of a given size with a given degree of confidence. Conversely, it allows us to determine the probability of detecting an effect of a given size with a given level of confidence, under sample size constraints. ### Publication bias ### Most publications report “positive” results, i.e. result with statistical significance. Negative results are usually not published. The problem is also described as the file-drawer problem. Studies that fail to reject the null hypothesis tend to remain unpublished and therefore remain in the file-drawer. Therefore, this bias leads to an overestimation of scientific evidence in the literature. *Further reading:* Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. *Trends in Cognitive Sciences*, 18(5), 235–241. https://doi.org/10.1016/j.tics.2014.02.010 ## R ## ### Reanalysis ### Reanalysis of a study that shares data. This can be achieved with own code or shared code. See also reproducibility and methods reproducibility. ### Replicability ### The ability of the researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected. See also replication study, conceptual replication and direct replication. ### Replication study ### A study that aims to redo a prior study by using the same methods and procedures but collects new data. Usually, good replication studies are larger and have higher power than the original studies. Often, the protocol of the replication study has been approved by the authors of the original study. ### Reporting bias ### Only reporting some of the results when more exists. Also known as selective reporting or selective analysis reporting. Examples of reporting bias or selective reporting are incomplete reporting of results, subgroup analyses, or substitution of non-significant primary outcomes with significant secondary outcomes. *Further reading:* Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. *Trends in Cognitive Sciences*, 18(5), 235–241. https://doi.org/10.1016/j.tics.2014.02.010 ### Reproducibility ### Refers to the ability of the researcher to produce the same results of a prior study by using the same materials and the same data that were used in the original investigation. In other words, reproducibility is the process when independent people analyze the same data to see if the results can be verified. This requires, at minimum, the sharing of data, relevant metadata, and code. See also the difference to replication. See also method reproducibility, result reproducibility, inferential reproducibility. *Further reading:* Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science *Translational Medicine*, 8(341), 341ps12. https://doi.org/10.1126/scitranslmed.aaf5027 ### Result reproducibility ### Refers to obtaining the same results from the conduct of an independent study with new data that closely follows the original study. *Further reading:* Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science *Translational Medicine*, 8(341), 341ps12. https://doi.org/10.1126/scitranslmed.aaf5027 ## S ## ### Sampling bias ### A bias in which a sample is collected in such a way that some members of the intended population are less likely to be included than others. For example, many studies in psychology are based on a sample of psychology university students. See also external validity. ### Selective reporting ### See reporting bias. ## U ## ### Underpowered study ### See low power.