Adjusted AUC: A new method for evaluating ecological niche models for non-equilibrium species distributions

Background/Question/Methods

Accurate and consistent diagnostic tools for evaluating ecological niche models (ENMs) are necessary for understanding species range dynamics in the context of both native and non-native species. Over the past decade, The Receiver Operator Characteric  (ROC) curve and the Area Under the ROC curve (AUC) have gained popularity as simple tools for evaluating ENMs. However, the ROC curve lacks consistency for model evaluation for species in non-equilibrium states due to differences in the cost asymmetry of omission and commission errors, a critical problem for evaluating species range dynamics. To account for the relative cost asymmetry of these types of errors in non-equilibrium species distributions, I introduce a new metric, Adjusted AUC, wherein Adjusted AUC represents the projected AUC of the species in a saturated landscape (i.e., where all suitable locations are occupied by the species). Using Monte Carlo simulations of habitat suitability and suitability models with varying error rates, I assessed the relative importance of various model evaluation metrics (e.g., Kappa, AUC, Model Quality, Sorenson’s Index, Sensitivity, Specificity, Precision, Recall) in predicting both Adjusted AUC and the actual error rate. 

Results/Conclusions

Unlike AUC, Adjusted AUC is consistent across varying proportions of landscape suitability and percent occupied.  Adjusted AUC was highly negatively correlated with error (r=-0.932 and -0.995 for linear and cosine-transformed models) and commission error (1-specificity) for the optimal classification threshold. Although Adjusted AUC represents the potential AUC of a saturated model, in practice the precise Saturated AUC is not known. We show through simulations that a simple model based on the False Positive Rate at the threshold where Sensitivity+Specificity is maximized is within 1% of the Saturated AUC approximately 95% of the time.  These results show that derivations of the AUC curve can still be useful and consistent for use in assessing non-equilibrium models of habitat suitability.