Browse > Article
http://dx.doi.org/10.5351/KJAS.2019.32.2.187

Partial AUC and optimal thresholds  

Hong, Chong Sun (Department of Statistics, Sungkyunkwan University)
Cho, Hyun Su (Department of Statistics, Sungkyunkwan University)
Publication Information
The Korean Journal of Applied Statistics / v.32, no.2, 2019 , pp. 187-198 More about this Journal
Abstract
Extensive literature exists on how to estimate optimal thresholds based on various accuracy measures using receiver operating characteristic (ROC) and cumulative accuracy profile (CAP) curves. This paper now proposes an alternative measure to represented the specific partial area under the ROC and CAP curves. The relationship between ROC and CAP functions is examined using differential equations of the new defined partial area under curves. In addition, the relationship with the optimal thresholds under conditions of various accuracy measures for the ROC and CAP functions is also derived. We assume there are two kinds of distribution functions composing the mixed distribution as various normal distributions before finding the optimal thresholds. Corresponding type 1 and 2 errors are also explored and discussed under various conditions for accuracy measures.
Keywords
accuracy; classification; confusion matrix; default; optimal threshold;
Citations & Related Records
Times Cited By KSCI : 3  (Citation Analysis)
연도 인용수 순위
1 Berry, M. J. A. and Linoff, G. (1999). Data Mining Techniques: For Marketing, Sales, and Customer Support (3rd ed), John Wiley & Sons, New York.
2 Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms, Pattern Recognition, 30, 1145-1159.   DOI
3 Brasil, P. (2010). DiagnosisMed: Diagnostic test accuracy evaluation for medical professionals, Package Diagnosis Med in R, from: http://www.CRAN.R-project.org/src/contrib/Archive/DiagnosisMed
4 Centor, R. M. (1991). Signal detectability: the use of ROC analysis, Med Decision Making, 11, 102-106.   DOI
5 Connell, F. A. and Koepsell, T. D. (1985). Measures of gain in certainty from a diagnostic test, American Journal of Epidemiology, 121, 744-753.   DOI
6 Dodd, L. E. and Pepe, M. S. (2003). Partial AUC Estimation and Regression, Biometrics, 59, 614-623.   DOI
7 Engelmann, B., Hayden, E. and Tasche, D. (2003). Measuring the discriminative power of rating systems, Discussion paper, Series 2: Banking and Financial Supervision, No. 01/2003.
8 Fawcett, T. (2003). ROC Graphs: Notes and Practical Considerations for Data Mining Researchers, HP Laboratories, Palo Alto, HPL-2003-4.
9 Fawcett, T. (2006). An introduction to ROC analysis, Pattern Recognition Letters, 27, 861-874.   DOI
10 Fawcett, T. and Provost, F. (1997). Analysis and visualization of classier performance comparison under imprecise class and cost distributions, Knowledge Discovery and Data Mining, 97, 43-48.
11 Fawcett, T. and Provost, F. (2001). Robust classification for imprecise environments, Machine Learning, 42, 203-231.   DOI
12 Hong, C. S. and Choi, J. S. (2009). Optimal Threshold from ROC and CAP Curves, The Korean Journal of Applied Statistics, 22, 911-921.   DOI
13 Hong, C. S., Joo, J. S., and Choi, J. S. (2010). Optimal threshold from mixture distributions, The Korean Journal of Applied Statistics, 23, 13-28.   DOI
14 Irwin, J. R. and Irwin, C. T. (2012). Appraising Credit Ratings: Does the CAP Fit Better than the ROC?, International Monetary Fund Working paper, WP. 12/122.
15 Jiang, Y., Metz, C., and Nishikawa, R. (1996). A receiver operating characteristic partial area index for highly sensitive diagnostic tests, Radiology, 201, 745-750.   DOI
16 Joseph, M. P. (2005). A PD validation framework for Basel II internal ratings-based systems, Credit Scoring and Credit Control IV.
17 Krzanowski, W. J. and Hand, D, J. (2009). ROC Curves for Continuous Data, CRC Press, New York.
18 Lambert, J. and Lipkovich, I. (2008). A macro for getting more out of your ROC curve, SAS Global Forum 2008, San Antonio, Paper 231-2008.
19 Moses, L. E., Shapiro, D., and Littenberg, B. (1993). Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations, Statistics in Medicine, 12, 1293-1316.   DOI
20 Pepe, M. S. (2000). Receiver operating characteristic methodology, Journal of the American Statistical Association, 95, 308-311.   DOI
21 Pepe, M. S. (2003). The Statistical Evaluation of Medical Test for Classification and Prediction (17th ed), Oxford University Press, Oxford.
22 Perkins, N. J. and Schisterman, E. F. (2006). The inconsistency of "Optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve, American Journal of Epidemiology, 163, 670-675.   DOI
23 Sobehart, J. R., Keenan, S. C., and Stein, R. M. (2000). Benchmarking quantitative default risk models: a validation methodology, Moody's Investors Service.
24 Velez, D. R., White, B. C., Motsinger, A. A., Bush, W. S., Ritchie, M. D., Williams, S. M., and Moore, J. H. (2007). A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction, Genetic Epidemiology, 31, 306-315.   DOI
25 Swets, J. A., Dawes, R. M., and Monahan, J. (2000). Better Decisions through Science, Scientific American, 82-87.
26 Swets, J. A. (1988). Measuring the accuracy of diagnostic systems, American Association for the Advancement of Science, 240, 1285-1293.   DOI
27 Tasche, D. (2006). Validation of internal rating systems and PD estimates, arXiv.org, eprint arXiv: physics/0606071.
28 Vuk, M. and Curk, T. (2006). ROC curve, lift chart and calibration plot, Metodoloski Zvezki, 3, 89-108.
29 Yoo, H. S. and Hong, C. S. (2011). Optimal criterion of classification accuracy measures for normal mixture, The Korean Journal of Applied Statistics, 18, 343-355.   DOI
30 Youden, W. J. (1950). Index for rating diagnostic tests, Cancer, 3, 32-35.   DOI
31 Zou, K. H. (2002). Receiver operating characteristic literature research, from: http://www.spl.harvard.edu/pages/ppl/zou/roc.html
32 Zweig, M. H. and Campbell, G. (1993). Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine, Clinical Chemistry, 39, 561-577.   DOI