Multiple recent ‘many-analyst’ projects have demonstrated that when
different analysts seek to answer the same question with the same data,
they often produce a substantially heterogeneous set of answers. We
evaluated the extent of this phenomenon in ecology and evolutionary biology
with a registered report in which several hundred biologists analyzed data
from either of two data sets to answer a predetermined question associated
with each data set. We found substantial heterogeneity in results from
these analyses, but this heterogeneity differed qualitatively and
quantitatively between the two data sets. Surprisingly, very little of the
observed heterogeneity in results among analyses appeared related to the
assessments of the individual analyses submitted by peer reviewers. Our
results have major implications for how ecologists and evolutionary
biologists conduct and interpret statistical analyses.
