How should we use meta-analysis to answer complex questions in ecology and evolution?

Background/Question/Methods

In ecology and evolution, some scientific questions may be addressed with meta-analysis by simply estimating an overall effect size, such as the average effect across studies of a particular experimental treatment.  However, many ecological processes are context dependent and meta-analyses which overlook such context dependence risk making false generalizations.  In principle, multi-factor meta-analysis (i.e., meta-regression), essentially a multiple regression approach to meta-analysis, can be used to explain variation in effect sizes among studies.  However, the proper application of meta-regression in ecology and evolution presents two significant challenges: First, meta-regression datasets are observational in nature, and thus explanatory variables are often correlated with each other and with un-measured variables.  Second, complex mixed models, in which the random effects may include a between-studies variance component as well as information on phylogenetic relationships among the species being analyzed, must be estimated and compared. The first of these problems points to the potential utility of model selection using information criteria (such as AIC or BIC) and multi-model inference.  But the second problem makes proper model fitting and model selection highly non-trivial and prone to pitfalls.  We illustrate these challenges through re-analysis of a large meta-regression dataset on context-dependency in responses of plants to inoculation with mycorrhizal fungi.

Results/Conclusions

Results show some of the conditions under which conclusions on the relative importance of predictor variables may be significantly influenced by the choice of parameter estimation methods, especially for fixed effects that are confounded with particular random effects among models.  We also show how parameter estimates can be strongly influenced by the choice of models used for inference, and how erroneous conclusions can be drawn if un-measured variables are not considered in interpretation. Overall, we suggest caution against over interpretation from meta-analyses of ecological data sets, that a richer set of statistical tools is needed to properly implement multi-factor meta-regression in ecology and evolution, and that simulation studies would substantially aid in choosing appropriate model selection methods.