Date created: | Last Updated:

Category: Project

Description: Meta-analyses are an important tool to evaluate the literature. Recent concerns about the reliability of single studies might also apply to meta-analyses, where the selection, inclusion, and coding of studies introduces substantial flexibility in the analysis and interpretation of results. It is essential that meta-analyses can easily be reproduced to allow researchers to evaluate the impact of subjective choices on meta-analytic effect sizes, but also to update meta-analyses as new data comes in, or as novel statistical techniques (for example to correct for publication bias) are developed. Regrettably, research in medicine reveals meta-analyses often cannot be reproduced. We plan to quantify the reproducibility of meta-analyses in psychology by reproducing twenty published meta-analyses. We will examine how much variability in effect size estimates is observed in reproduced meta-analyses, both as a function of subjective choices, as when novel statistical techniques to correct for bias are applied to published meta-analyses. Difficulties in reproducing meta-analyses, or substantial variation between original and reproduced meta-analyses, would highlight the importance of increasing the transparency and reproducibility of meta-analyses. This project has the potential to initiate a discussion about the need to increase the transparency of published meta-analyses, and results may have important implications for journal policies.