There are multiple approaches researchers have taken to handle dependent effect sizes (multiple effect sizes produced by the same sample) in meta-analyses. Researchers may choose to treat them as independent effect sizes, they may average the dependent effect sizes together, or they may make adjustments for dependent effect sizes, for example by using robust variance estimation or Cheung and Chan's method.
We adjusted for dependent performance measures using a method based on Cheung and Chan's (2004, 2008). Cheung and Chan's method adjusts the sample size to be between the sample N and the cumulative sample N, and applies this to the average of the dependent effect sizes. Their adjustment formula is as follows: adjusted N = ((N-1)/C)+1, where C accounts for the correlation between dependent effect sizes, in addition to the overall average effect size, and the number of dependent effect sizes per sample. We inadvertently used the formula as follows: adjusted N = (N -1)/(C+1) and then applied this formula to each individual effect size (rather than an average). We did not realize this until recently.
It happens that our lower N computed for each individual effect generally offset the higher N for the cumulative effect under the Cheung and Chan approach while obviating the limitations of combining effects. The formula and application we used produces values similar to those produced by robust variance estimation. Thus, while we inadvertently used this approach, it turns out to be within the range of common approaches, and remains in between treating dependent effects as independent, and averaging dependent effects together without adjustment. There was no practical effect on the results when we reanalyzed our findings using Cheung and Chan’s approach, and the changes have no impact whatsoever on our substantive findings and conclusions. We have removed the adjusted sample size column and variance columns for our data set so that anyone wishing to use our data for re-analysis can apply whichever approach they so choose. To view the adjusted samples and variances we originally used, select a prior version of the data set to view. Questions can be directed to Brooke N. Macnamara at email@example.com.
Cheung, S. F., & Chan, D. K-S. (2004). Dependent effect sizes in meta-analysis: Incorporating the degree of interdependence. Journal of Applied Psychology, 89, 780-791.
Cheung, S. F. & Chan, D. K-S. (2008). Dependent correlations in meta-analysis: The case of heterogeneous dependence. Educational and Psychological Measurement, 68(5), 760-777.