Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Relationship between Obuchowski–Rockette–Hillis and Gallas methods for analyzing multi-reader diagnostic imaging data with empirical AUC as the reader performance measure

Relationship between Obuchowski–Rockette–Hillis and Gallas methods for analyzing multi-reader... For analyzing multireader multicase (MRMC) diagnostic imaging data when the reader performance measure of interest is the area under the receiver-operating-characteristic curve (AUC), two popular methods of analysis that allow conclusions to generalize to both the reader and case populations are the method developed by Obuchowski, Rockette and Hillis (ORH) and the method primarily developed by Gallas (Gallas). While the ORH method is a general method that is applicable to most reader performance metrics, the Gallas method is limited to those metrics for which an unbiased variance estimate exists. Previously it was not known if the ORH method could be adapted so as to produce the same variance estimate as the Gallas method. In this paper, I show that a recently proposed version of the OR method produces the same unconstrained variance statistic as the Gallas method. However, the two methods differ in their approaches to constraining the variance estimate to be nonnegative and in their degrees-of-freedom estimates. These two differences are discussed and recommendations given. In addition, several contributions to the development of the ORH method are made, including determining sufficient conditions for unbiased variance estimates and providing justification for the ORH variance constraints and covariance estimation method. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Biostatistics & Epidemiology Taylor & Francis

Relationship between Obuchowski–Rockette–Hillis and Gallas methods for analyzing multi-reader diagnostic imaging data with empirical AUC as the reader performance measure

Hillis, Stephen L.
Biostatistics & Epidemiology , Volume OnlineFirst: 38 – Jun 9, 2022
Download PDF

Relationship between Obuchowski–Rockette–Hillis and Gallas methods for analyzing multi-reader diagnostic imaging data with empirical AUC as the reader performance measure

Abstract

For analyzing multireader multicase (MRMC) diagnostic imaging data when the reader performance measure of interest is the area under the receiver-operating-characteristic curve (AUC), two popular methods of analysis that allow conclusions to generalize to both the reader and case populations are the method developed by Obuchowski, Rockette and Hillis (ORH) and the method primarily developed by Gallas (Gallas). While the ORH method is a general method that is applicable to most reader...
Loading next page...
 
/lp/taylor-francis/relationship-between-obuchowski-rockette-hillis-and-gallas-methods-for-3y74SjWseu
Publisher
Taylor & Francis
Copyright
© 2022 International Biometric Society – Chinese Region
ISSN
2470-9379
eISSN
2470-9360
DOI
10.1080/24709360.2022.2062115
Publisher site
See Article on Publisher Site

Abstract

For analyzing multireader multicase (MRMC) diagnostic imaging data when the reader performance measure of interest is the area under the receiver-operating-characteristic curve (AUC), two popular methods of analysis that allow conclusions to generalize to both the reader and case populations are the method developed by Obuchowski, Rockette and Hillis (ORH) and the method primarily developed by Gallas (Gallas). While the ORH method is a general method that is applicable to most reader performance metrics, the Gallas method is limited to those metrics for which an unbiased variance estimate exists. Previously it was not known if the ORH method could be adapted so as to produce the same variance estimate as the Gallas method. In this paper, I show that a recently proposed version of the OR method produces the same unconstrained variance statistic as the Gallas method. However, the two methods differ in their approaches to constraining the variance estimate to be nonnegative and in their degrees-of-freedom estimates. These two differences are discussed and recommendations given. In addition, several contributions to the development of the ORH method are made, including determining sufficient conditions for unbiased variance estimates and providing justification for the ORH variance constraints and covariance estimation method.

Journal

Biostatistics & Epidemiology – Taylor & Francis

Published: Jun 9, 2022

Keywords: Obuchowski–Rockette; receiver-operating-characteristic curve; area under the ROC curve; Gallas; diagnostic radiology

References

  • Area above ordinal dominance graph and area below receiver operating characteristic graph
  • The meaning and use of the area under a receiver operating characteristic (ROC) curve
  • Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: an ANOVA approach with dependent observations
  • A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data
  • A comparison of denominator degrees of freedom methods for multiple observer ROC analysis
  • A marginal-mean ANOVA approach for analyzing multireader multicase radiological imaging data
  • One-shot estimate of MRMC variance: AUC
  • A framework for random-effects ROC analysis: biases with the bootstrap and other variance estimators
  • A probabilistic model for the MRMC method, part 1: theoretical development
  • Generalized Roe and Metz receiver operating characteristic model: analytic link between simulated decision scores and empirical AUC variances and covariances
  • Relationship between Roe and Metz simulation model for multireader diagnostic data and Obuchowski-Rockette model parameters
  • Recent developments in the Dorfman-Berbaum-Metz procedure for multireader study analysis
  • Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design
  • Approximate tests of correlation in time series
  • Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach
  • Multireader sample size program for diagnostic studies: demonstration and methodology
  • Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method
  • Power estimation for multireader ROC methods: an updated and unified approach
  • Simulation of unequal-variance binormal multireader ROC decision data: an extension of the Roe and Metz simulation model
  • Multi-reader ROC studies with split-plot designs: a comparison of statistical methods
  • Multireader multicase reader studies with binary agreement data: simulation, analysis, validation, and sizing
  • Synthesis of variance
  • An approximate distribution of estimates of variance components
  • Multireader multicase variance analysis for binary data
  • Evaluation of a digital workstation for interpreting neonatal examinations: a receiver operating characteristic study
  • Resampling methods for the area under the ROC curve
  • Exact bootstrap variances of the area under ROC curve
  • Accuracy of bedside chest hard-copy screen-film versus hard-and soft-copy computed radiographs in a medical intensive care unit: receiver operating characteristic analysis
  • Generalized linear mixed models for multi-reader multi-case studies of diagnostic tests
  • Reader studies for validation of CAD systems
  • Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation