Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You and Your Team.

Learn More →

Evaluating presence–absence models in ecology: the need to account for prevalence

Evaluating presence–absence models in ecology: the need to account for prevalence 1 Models for predicting the distribution of organisms from environmental data are widespread in ecology and conservation biology. Their performance is invariably evaluated from the percentage success at predicting occurrence at test locations. 2 Using logistic regression with real data from 34 families of aquatic invertebrates in 180 Himalayan streams, we illustrate how this widespread measure of predictive accuracy is affected systematically by the prevalence (i.e. the frequency of occurrence) of the target organism. Many evaluations of presence–absence models by ecologists are inherently misleading. 3 With the same invertebrate models, we examined alternative performance measures used in remote sensing and medical diagnostics. We particularly explored receiver‐operating characteristic (ROC) plots, from which were derived (i) the area under each curve (AUC), considered an effective indicator of model performance independent of the threshold probability at which the presence of the target organism is accepted, and (ii) optimized probability thresholds that maximize the percentage of true absences and presences that are correctly identified. We also evaluated Cohen's kappa, a measure of the proportion of all possible cases of presence or absence that are predicted correctly after accounting for chance effects. 4 AUC measures from ROC plots were independent of prevalence, but highly significantly correlated with the much more easily computed kappa. Moreover, when applied in predictive mode to test data, models with thresholds optimized by ROC erroneously overestimated true occurrence among scarcer organisms, often those of greatest conservation interest. We advocate caution in using ROC methods to optimize thresholds required for real prediction. 5 Our strongest recommendation is that ecologists reduce their reliance on prediction success as a performance measure in presence–absence modelling. Cohen's kappa provides a simple, effective, standardized and appropriate statistic for evaluating or comparing presence–absence models, even those based on different statistical algorithms. None of the performance measures we examined tests the statistical significance of predictive accuracy, and we identify this as a priority area for research and development. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Applied Ecology Wiley

Evaluating presence–absence models in ecology: the need to account for prevalence

Download PDF
11 pages
Read Article

Loading next page...
 
/lp/wiley/evaluating-presence-absence-models-in-ecology-the-need-to-account-for-NiLR5vJHUL
Publisher
Wiley
Copyright
Copyright © 2001 Wiley Subscription Services
ISSN
0021-8901
eISSN
1365-2664
DOI
10.1046/j.1365-2664.2001.00647.x
Publisher site
See Article on Publisher Site

Abstract

1 Models for predicting the distribution of organisms from environmental data are widespread in ecology and conservation biology. Their performance is invariably evaluated from the percentage success at predicting occurrence at test locations. 2 Using logistic regression with real data from 34 families of aquatic invertebrates in 180 Himalayan streams, we illustrate how this widespread measure of predictive accuracy is affected systematically by the prevalence (i.e. the frequency of occurrence) of the target organism. Many evaluations of presence–absence models by ecologists are inherently misleading. 3 With the same invertebrate models, we examined alternative performance measures used in remote sensing and medical diagnostics. We particularly explored receiver‐operating characteristic (ROC) plots, from which were derived (i) the area under each curve (AUC), considered an effective indicator of model performance independent of the threshold probability at which the presence of the target organism is accepted, and (ii) optimized probability thresholds that maximize the percentage of true absences and presences that are correctly identified. We also evaluated Cohen's kappa, a measure of the proportion of all possible cases of presence or absence that are predicted correctly after accounting for chance effects. 4 AUC measures from ROC plots were independent of prevalence, but highly significantly correlated with the much more easily computed kappa. Moreover, when applied in predictive mode to test data, models with thresholds optimized by ROC erroneously overestimated true occurrence among scarcer organisms, often those of greatest conservation interest. We advocate caution in using ROC methods to optimize thresholds required for real prediction. 5 Our strongest recommendation is that ecologists reduce their reliance on prediction success as a performance measure in presence–absence modelling. Cohen's kappa provides a simple, effective, standardized and appropriate statistic for evaluating or comparing presence–absence models, even those based on different statistical algorithms. None of the performance measures we examined tests the statistical significance of predictive accuracy, and we identify this as a priority area for research and development.

Journal

Journal of Applied Ecology – Wiley

Published: Jan 1, 2001

Keywords: ; ; ; ; ; ;

References

  • Multivariate Interpretation of Clinical Laboratory Data
    Albert, A.; Harris, E.K.
  • Selecting areas for species persistence using occurrence data
    Araujo, M.B.; Williams, P.H.
  • Current approaches to modelling the environmental niche of eucalyptus: implications for management of forest biodiversity
    Austin, M.P.; Meyers, J.A.
  • Habitat selection and breeding success of yellowhammers on lowland farmland
    Bradbury, R.B.; Kyrkos, A.; Morris, A.J.; Clark, S.C.; Perkins, A.J.; Wilson, J.D.
  • Predicting the likelihood of Eurasian water milfoil presence in lakes, a macrophyte monitoring tool.
    Buchan, L.A.J.; Padilla, D.K.
  • Empirical models for the spatial distribution of wildlife
    Buckland, S.T.; Elston, D.A.
  • Introduction: large‐scale processes in ecology and hydrology
    Caldow, R.W.G.; Racey, P.A.
  • Modelling Binary Data
    Collett, D.
  • Prediction of the spatial distribution on non‐indigenous weeds: issues of spatial scale and extent
    Collingham, Y.C.; Wadsworth, R.A.; Huntley, B.; Hulme, P.E.
  • The impact of agricultural land use on stream chemistry in the Middle Hills of the Himalayas, Nepal
    Collins, R.; Jenkins, A.
  • Habitat‐based statistical models for predicting the spatial distribution of butterflies and day‐flying moths in a fragmented landscape
    Cowley, M.J.R.; Wilson, R.J.; Leon‐Cortes, J.L.; Gutierrez, D.; Bulman, C.R.; Thomas, C.D.
  • Comparing the areas under two or more correlated receiver operating characteristics curves – a non‐parametric approach
    DeLong, E.R.; DeLong, D.M.; Clarke‐Pearson, D.I.
  • The impact of American mink Mustella vison on water birds in the upper Thames
    Ferreras, P.; Macdonald, D.W.
  • Machine Learning Methods for Ecological Applications
    Fielding, A.H.
  • A review of methods for the assessment of prediction errors in conservation presence/absence models
    Fielding, A.H.; Bell, J.F.
  • On the compensation for chance agreement in image classification accuracy assessment
    Foody, G.M.
  • Classification‐algorithm evaluation: five performance measures based on confusion matrices
    Forbes, A.D.
  • Mapping biodiversity using surrogates for species richness – macro‐scales and new world birds.
    Gaston, K.J.; Blackburn, T.M.
  • A critique for macroecology
    Gaston, K.J.; Blackburn, T.M.
  • Local extinction of British farmland birds and the prediction of further loss
    Gates, S.; Donald, P.F.
  • The distribution of passerine birds in hedgerows during the breeding season in relation to characteristics of hedgerows and adjacent farmlands.
    Green, R.E.; Osborne, P.E.; Sears, E.J.
  • Predictive habitat distribution models in ecology
    Guisan, A.; Zimmermann, N.E.
  • The uncertainty in simulations by a global biome model (BIOMES) to alternative parameter values
    Hallgren, W.S.; Pitman, A.J.
  • Mapping montane tropical forest successional stage and land‐use with multi‐date Landsat imagery
    Helmer, E.H.; Brown, S.; Cohen, W.B.
  • Modelling present and potential future ranges of some European higher plants using climate response surfaces
    Huntley, B.; Berry, P.M.; Cramer, W.; McDonald, A.P.
  • Data Analysis in Community and Landscape Ecology
    Jongman, R.H.G.; Ter Braak, C.J.F.; Van Tongeren, O.F.R.
  • Assessment of 2 × 2 associations: generalization of signal‐detection methodology
    Kraemer, H.C.
  • The measurements of observer agreement for categorical data
    Landis, J.R.; Koch, G.G.
  • Patterns in ecology.
    Lawton, J.
  • Designing the core zone in a biosphere reserve based on suitable habitats: Yanchang Biosphere Reserve and the red‐crowned crane (Grus japonensis)
    Li, W.J.; Wang, Z.J.; Ma, Z.J.; Tang, H.X.
  • Generalized Linear Models
    McCullagh, P.; Nelder, J.A.
  • Problems and possibilities in large‐scale surveys: the effects of land use on the habitats, invertebrates and birds of Himalayan rivers
    Manel, S.; Buckton, S.T.; Ormerod, S.J.
  • Alternative methods for predicting species distribution: an illustration with Himalayan river birds
    Manel, S.; Dias, J.M.; Buckton, S.T.; Ormerod, S.J.
  • Comparing discriminant analysis, neural networks and logistic regression for predicting species’ distributions: a case study with a Himalayan river bird
    Manel, S.; Dias, J.M.; Ormerod, S.J.
  • Habitat models of bird species distribution: an aid to the management of coastal grazing marshes
    Milsom, T.P.; Langton, S.D.; Parkin, W.K.; Peel, S.; Bishop, J.D.; Hart, J.D.; Moore, N.P.
  • Comparing global vegetation maps with the kappa statistic
    Monserud, R.A.; Leemans, R.
  • The statistical evaluation of ecological indicators
    Murtaugh, P.A.
  • Large‐scale ecology and hydrology: an introductory perspective from the editors of the Journal of Applied Ecology
    Ormerod, S.J.; Watkinson, A.R.
  • Communicating the value of ecology
    Ormerod, S.J.; Pienkowski, M.W.; Watkinson, A.R.
  • Evaluating the predictive performance of habitat models developed using logistic regression
    Pearce, J.; Ferrier, S.
  • Use of geographical information systems in the search for additional populations, or sites suitable for re‐establishment, of the endangered Northern Province endemic Euphorbia civicola
    Pfab, M.F.; Witowski, E.T.F.
  • A note on issues in meta‐analysis for behavioural genetic studies using categorical phenotypes
    Ridenour, T.A.; Heath, A.C.
  • Use of receiver operating curves to evaluate the clinical performance of analytical systems
    Robertson, E.A.; Zweig, M.H.
  • Evaluating the clinical accuracy of laboratory tests.
    Robertson, E.A.; Zweig, M.H.; Van Steirteghem, M.D.
  • Statistical power of presence absence data to detect population declines
    Strayer, D.L.
  • Nesting habitat selection by booted eagles Hieraaetus pennatus and implications for management
    Suarez, S.; Balbontin, J.; Ferrer, M.
  • Measuring the accuracy of diagnostic systems
    Swets, J.A.
  • Chance‐corrected classification for use in discriminant analysis: ecological applications
    Titus, K.; Mosher, J.A.; Williams, B.K.
  • Effects of environmental parameters on the distribution of bullhead Cottus gobio with particular consideration of the effects of obstructions
    Utzinger, J.; Roth, C.; Peter, A.
  • Comparison of the effectiveness of four clinical chemical assays in classifying patients’ chest pain
    Van Steirteghem, A.C.; Zweig, M.H.; Robertson, E.A.; Bernard, R.M.; Putzeys, G.A.; Bieva, C.
  • A spatial, climate‐determined risk rating for Scleroderris disease of pines in Ontario
    Venier, L.A.; Hopkin, A.A.; McKenny, D.W.; Wang, Y.
  • Satellite remote sensing of habitat suitability for ungulates and ostrich in the Kalahari of Botswana
    Verlinden, A.; Masogo, R.
  • Simulating the spread and management of alien riparian weeds: are they out of control?
    Wadsworth, R.A.; Collingham, Y.C.; Willis, S.G.; Huntley, B.; Hulme, P.E.
  • Visualisation of biomedical datasets by use of growing cell structure networks: a novel diagnostic classification technique
    Walker, A.J.; Cross, S.S.; Harrison, R.F.
  • Prediction of rare‐plant occurrence: a southern Appalachian example
    Wiser, S.; Peet, R.K.; White, P.S.
  • Predicting the distribution of European badger (Meles meles) setts over an urbanized landscape: a GIS approach
    Wright, A.; Fielding, A.H.; Wheater, C.P.
  • Spatial distribution and habitat preference of the endangered tarantula Brachypelma klaasi (Araneae: Theraphosidae) in Mexico
    Yanez, M.; Floater, G.
  • Predictive mapping of alpine grasslands in Switzerland: species versus community approach
    Zimmermann, N.E.; Kienast, F.
  • Receiver‐operating characteristics (ROC) plots: a fundamental evaluation tool in clinical medicine
    Zweig, M.H.; Campbell, G.
  • Why we need better test evaluations
    Zweig, M.H.; Robertson, E.A.
  • ROC curve analysis: an example showing the relationships among serum lipid and apolipoprotein concentration in identifying patients with coronary artery disease
    Zweig, M.H.; Broste, S.K.; Reinhart, R.A.

You’re reading a free preview. Subscribe to read the entire article.

Try 2 weeks free now

DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

or browse the journals available

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$499/year

Save searches from
Google Scholar,
PubMed

Create folders to
organize your research

Export folders, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

Sign up
for free
Start 14 day
Free Trial