Meta-analysis is frequently used in ecology to synthesize data and draw conclusions across systems. However, many meta-analytic tools were developed for fields other than ecology, which is unique in its low sample size and high variability when comparing across multiple ecological systems. Under these conditions, commonly used metrics and weighting schemes can lead to erroneous conclusions. Here, we investigate bias associated with the common metric, Hedges' d. In particular, we investigate how different weighting schemes can lead to biased estimates of effects or altered coverage of confidence intervals when studies are heterogeneous. We simulated data from experiments, computed effect sizes and their variances, and performed random effects meta-analyses using three weighting schemes (inverse variance, inverse variance using a sample size-based estimate of variance, and unweighted) for varying levels of effect size, replication, number of studies in the meta-analysis, and among-study variance. We compared estimated effects from the meta-analyses to the true effect to evaluate bias; we also quantified coverage of the confidence intervals and examined efficiency of the estimators.
Results/Conclusions
Unweighted analyses or those weighted with the sample size-based variance were unbiased and yielded coverages that were close to the nominal level of 0.95. In contrast, the traditional inverse-variance weighting method led to bias and low coverage, especially for studies with small number of replicates, which are common in ecological meta-analyses. The number of studies in the meta-analysis did not affect bias, but did affect coverage when the average number of replicates for each study was small. As the among-study variance increased, bias increased, but coverage also increased. Bias and low coverage resulted from the correlation between estimated effect sizes and weights, when using the traditional inverse-variance weighting. This effect was worse when the average number of replicates for each study was small. Thus, in certain conditions, meta-analysists using Hedge’s d (which has been cautioned against on conceptual grounds in ecological meta-analyses as well) need to be aware of the bias introduced. In these cases, it may be advantageous to use unweighted analyses or weights based on a sample size.