Date created: | Last Updated:

Category: Project

Description: Most meta-analyses focus on meta-analytic means, testing whether they are significantly different from zero and how they depend on covariates. This mean is difficult to defend as a construct because the underlying distribution of studies reflects many factors such as how we choose to run experiments. We argue that the fundamental questions of meta-analysis should not be about the aggregated mean; instead, one should ask which relations are stable across all the studies. In a typical meta-analysis, there is a preferred or hypothesized direction (e.g., that violent video games increase, rather than decrease, agressive behavior). We ask whether all studies in a meta-analysis have true effects in a common direction. If so, this is an example of a stable relation across all the studies. We propose four models: (i) all studies are truly null; (ii) all studies share a single true nonzero effect; (iii) studies differ, but all true effects are in the same direction; and (iv) some study effects are truly positive while others are truly negative. We develop Bayes factor model comparison for these models and apply them to four extant meta-analyses to show their usefulness.