Date created: | Last Updated:

Category: Project

Description: Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias is widespread, how strongly it affects different scientific literatures is currently less well-known. We examine evidence of publication bias in a large-scale data set of meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and the Cochrane Database of Systematic Reviews (representing meta-analyses from medical research). Psychology is compared to medicine, because medicine has a longer history than psychology with respect to preregistration of studies as an effort to counter publication bias. The severity of publication bias and its inflating effects on effect size estimation were systematically studied by applying state-of-the-art publication bias tests and the p-uniform method for estimating effect size corrected for publication bias. Publication bias was detected in only a small amount of homogeneous subsets. The lack of evidence of publication bias was probably caused by the small number of effect sizes in most subsets resulting in low statistical power for publication bias tests. Regression analyses showed that small-study effects were actually present in the meta-analyses. That is, smaller studies were accompanied with larger effect sizes, possible due to publication bias. Overestimation of effect size due to publication bias was similar for meta-analyses published in Psychological Bulletin and Cochrane Database of Systematic Reviews.