Implications of using an optimal alpha approach for statistical hypothesis tests in ecology

Background/Question/Methods

The standard practice of setting alpha (the probability of Type I error in statistical hypothesis testing) at 0.05 is a problem in ecological research because it ignores the relationship between alpha and statistical power and it makes implicit assumptions about critical effect sizes and relative costs of Type I and Type II errors.  The optimal alpha level for a statistical test in ecology should be that which minimizes the overall probability (or cost) of committing an error for a given sample size and desired critical effect size.  Without information concerning the prior probabilities of null and alternate hypotheses, the overall probability of error is the average of alpha and beta (the probability of Type II error in statistical hypothesis testing), for which there is only one minimum value for any combination of sample size and critical effect size.  When the costs of Type I and II error are equal, error costs and probabilities are minimized at the same alpha level but if the costs differ, the overall cost of errors can be minimized using the cost-weighted average of alpha and beta. 

Results/Conclusions

We have examined the implications of our optimal alpha approach using data collected under the Canadian Environmental Effects Monitoring (EEM) program, which assists in regulation and mitigation of impacts of pulp and paper mill effluent on aquatic ecosystems.  The presence of significant statistical ecological differences informs a key decision in EEM as to whether a specific facility must continue to monitor fish populations in the next cycle of monitoring. Type I errors result in potentially unnecessary additional monitoring during the next cycle, while Type II errors result in undetected changes in key fish population characteristics that may affect the fish population and subsequently ecosystem function. Of 149 EEM pulp and paper mill studies comparing fish liver size between a reference and an exposure site, 10% would result in a different conclusion using an optimal alpha (set to minimize the overall probability of committing an error) as compared with the conclusion reached using alpha = 0.05.  The average Type I error/Type II error cost ratio at which the optimal alpha = 0.05 is 1.88. This implies that the pulp and paper industry considers detecting a false change in fish liver size (triggering unnecessary future monitoring) to be almost twice as serious as failing to detect a real change in fish liver size (resulting in unmitigated impacts on the fish population).