Browse > Article
http://dx.doi.org/10.5351/KJAS.2020.33.4.395

Index of union and other accuracy measures  

Hong, Chong Sun (Department of Statistics, Sungkyunkwan University)
Choi, So Yeon (Department of Statistics, Sungkyunkwan University)
Lim, Dong Hui (Department of Statistics, Sungkyunkwan University)
Publication Information
The Korean Journal of Applied Statistics / v.33, no.4, 2020 , pp. 395-407 More about this Journal
Abstract
Most classification accuracy measures for optimal threshold are divided into two types: one is expressed with cumulative distribution functions and probability density functions, the other is based on ROC curve and AUC. Unal (2017) proposed the index of union (IU) as an accuracy measure that considers two types to get them. In this study, ten kinds of accuracy measures (including IU) are divided into six categories, and the advantages of the IU are studied by comparing the measures belonging to each category. The optimal thresholds of these measures are obtained by setting various normal mixture distributions; subsequently, the first and second type of errors as well as the error sums corresponding to each threshold are calculated. The properties and characteristics of the IU statistic are explored by comparing the discriminative power of other accuracy measures based on error values.The values of the first type error and error sum of IU statistic converge to those of the best accuracy measures of the second category as the mean difference between the two distributions increases. Therefore, IU could be an accuracy measure to evaluate the discriminant power of a model.
Keywords
accuracy; discriminant; sensitivity; specificity; threshold;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 Perkins, N. J. and Schisterman, E. F. (2006). The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve, American Journal of Epidemiology, 163, 670-675.   DOI
2 Provost, F. and Fawcett, T. (2001). Robust classification for imprecise environments, Machine Learning, 42, 203-231.   DOI
3 Spackman, K. A. (1989). Signal detection theory: valuable tools for evaluating inductive learning, The Analytics of Risk Model Validation, San Mateo, 160-163.
4 Tasche, D. (2006). Validation of internal rating systems and PD estimates, The Analytics of Risk Model Validation, 169-196.
5 Unal, I. (2017). Defining an optimal cut-point value in ROC analysis: an alternative approach, Computational & Mathematical Methods in Medicine, 2017, 1-14.   DOI
6 Youden, W. J. (1950). Index for rating diagnostic test, Cancer, 3, 32-35.   DOI
7 Zweig, M. and Campbell, G. (1993). Receiver-operating characteristics (ROC) plots: A fundamental evaluation tool in clinical medicine, Clinical Chemistry, 39, 561-577.   DOI
8 Vuk, M. and Curk, T. (2006). ROC curve, lift chart and calibration plot, Metodoloski Zvezki, 3, 89-108.
9 Yoo, H. S. and Hong, C. S. (2011). Optimal criterion of classification accuracy measures for normal mixture, Communications for Statistical Applications and Methods, 18, 343-355.   DOI
10 Brasil, P. (2010). Diagnostic test accuracy evaluation for medical professionals, Package DiagnosisMed in R.
11 Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms, Pattern Recognitions, 30, 1145-1159.   DOI
12 Cantor, S. B., Sun, C. C., Tortolero-Luna, G., Richards-Kortum, R., and Follen, M. (1999). A comparison of C/B ratios from studies using receiver operating characteristic curve analysis, Journal of Clinical Epidemiology, 52, 885-892.   DOI
13 Centor, R. N. (1991). Signal detectability: The use of ROC curves and their analyses, Medical Decision Making, 11, 102-106.   DOI
14 Connell, F. A. and Koepsell, T. D. (1985). Measures of gain in certainty from a diagnostic test, American Journal of Epidemiology, 121, 744-753.   DOI
15 Egan, J. P. (1975). Signal detection theory and ROC analysis, Academic Press, New York.
16 Engelmann, B., Hayden, E., and Tasche, D. (2003). Testing rating accuracy, Risk, 16, 82-86.
17 Fawcett, T. (2006). An introduction to ROC analysis, Pattern Recognition Letters, 27, 861-874.   DOI
18 Fawcett, T. and Provost, F. (1997). Adaptive fraud detection, Data Mining and Knowledge Discovery, 1, 291-316.   DOI
19 Freeman, E. A. and Moisen, G. G. (2008). A comparison of the performance of threshold criteria for binary classification in terms of predicted prevalence and kappa, Ecological Modelling, 217, 48-58.   DOI
20 Greiner, M. M. and Gardner, I. A. (2000). Epidemiologic issues in the validation of veterinary diagnostic tests, Preventive Veterinary Medicine, 45, 3-22.   DOI
21 Hong, C. S., Lin, M. H., Hong, S. W., and Kim, G. C. (2011). Classification accuracy measures with minimum error rate for normal mixture, Journal of the Korean Data & Information Science Society, 22, 619-630.
22 Hanley, J. A. and McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve, Radiology, 143, 29-36.   DOI
23 Hong, C. S. and Lee S. J. (2018). TROC curve and accuracy measures, Journal of the Korean Data & Information Science Society, 29, 861-872.   DOI
24 Hong, C. S., Joo, J. S., and Choi, J. S. (2010). Optimal thresholds from mixture distributions, The Korean Journal of Applied Statistics, 23(1), 13-28.   DOI
25 Hsieh, F. and Turnbull, B. W. (1996). Nonparametric and semiparametric estimation of the receiver operating characteristic curve, The Annals of Statistics, 24, 25-40.   DOI
26 Krzanowski, W. J. and Hand, D. J. (2009). ROC Curves for Continuous Data, Chapman & Hall/CRC, Boca Raton.
27 Lambert, J. and Lipkovich, I. (2008). A macro for getting more out of your ROC curve, SAS Global Forum, 231.
28 Liu, C., White, M., and Newell1, G. (2009). Measuring the accuracy of species distribution models: a review. In Proceedings 18th World IMACs/MODSIM Congress, 4241, 4247.
29 Metz, C. E. and Kronman H. B. (1980). Statistical significance tests for binormal ROC curves, Journal of Mathematical Psychology, 22, 218-243.   DOI
30 Moses, L. E., Shapiro, D., and Littenberg, B. (1993). Combining independent studies of a diagnostic test into a summary ROC curve: Data-analytic approaches and some additional considerations, Statistics in Medicine, 12, 1293-1316.   DOI
31 Pepe, M. S. (2003). The Statistical Evaluation of Medical Tests for Classication and Prediction, Oxford University Press, Oxford.
32 Pepe, M. S. (2000). Receiver operating characteristic methodology, Journal of the American Statistical Association, 95, 308-311.   DOI