Browse > Article
http://dx.doi.org/10.13088/jiis.2020.26.1.023

The Effect of Meta-Features of Multiclass Datasets on the Performance of Classification Algorithms  

Kim, Jeonghun (Department of Management, Kyung Hee University)
Kim, Min Yong (School of Management, Kyung Hee University)
Kwon, Ohbyung (School of Management, Kyung Hee University)
Publication Information
Journal of Intelligence and Information Systems / v.26, no.1, 2020 , pp. 23-45 More about this Journal
Abstract
Big data is creating in a wide variety of fields such as medical care, manufacturing, logistics, sales site, SNS, and the dataset characteristics are also diverse. In order to secure the competitiveness of companies, it is necessary to improve decision-making capacity using a classification algorithm. However, most of them do not have sufficient knowledge on what kind of classification algorithm is appropriate for a specific problem area. In other words, determining which classification algorithm is appropriate depending on the characteristics of the dataset was has been a task that required expertise and effort. This is because the relationship between the characteristics of datasets (called meta-features) and the performance of classification algorithms has not been fully understood. Moreover, there has been little research on meta-features reflecting the characteristics of multi-class. Therefore, the purpose of this study is to empirically analyze whether meta-features of multi-class datasets have a significant effect on the performance of classification algorithms. In this study, meta-features of multi-class datasets were identified into two factors, (the data structure and the data complexity,) and seven representative meta-features were selected. Among those, we included the Herfindahl-Hirschman Index (HHI), originally a market concentration measurement index, in the meta-features to replace IR(Imbalanced Ratio). Also, we developed a new index called Reverse ReLU Silhouette Score into the meta-feature set. Among the UCI Machine Learning Repository data, six representative datasets (Balance Scale, PageBlocks, Car Evaluation, User Knowledge-Modeling, Wine Quality(red), Contraceptive Method Choice) were selected. The class of each dataset was classified by using the classification algorithms (KNN, Logistic Regression, Nave Bayes, Random Forest, and SVM) selected in the study. For each dataset, we applied 10-fold cross validation method. 10% to 100% oversampling method is applied for each fold and meta-features of the dataset is measured. The meta-features selected are HHI, Number of Classes, Number of Features, Entropy, Reverse ReLU Silhouette Score, Nonlinearity of Linear Classifier, Hub Score. F1-score was selected as the dependent variable. As a result, the results of this study showed that the six meta-features including Reverse ReLU Silhouette Score and HHI proposed in this study have a significant effect on the classification performance. (1) The meta-features HHI proposed in this study was significant in the classification performance. (2) The number of variables has a significant effect on the classification performance, unlike the number of classes, but it has a positive effect. (3) The number of classes has a negative effect on the performance of classification. (4) Entropy has a significant effect on the performance of classification. (5) The Reverse ReLU Silhouette Score also significantly affects the classification performance at a significant level of 0.01. (6) The nonlinearity of linear classifiers has a significant negative effect on classification performance. In addition, the results of the analysis by the classification algorithms were also consistent. In the regression analysis by classification algorithm, Naïve Bayes algorithm does not have a significant effect on the number of variables unlike other classification algorithms. This study has two theoretical contributions: (1) two new meta-features (HHI, Reverse ReLU Silhouette score) was proved to be significant. (2) The effects of data characteristics on the performance of classification were investigated using meta-features. The practical contribution points (1) can be utilized in the development of classification algorithm recommendation system according to the characteristics of datasets. (2) Many data scientists are often testing by adjusting the parameters of the algorithm to find the optimal algorithm for the situation because the characteristics of the data are different. In this process, excessive waste of resources occurs due to hardware, cost, time, and manpower. This study is expected to be useful for machine learning, data mining researchers, practitioners, and machine learning-based system developers. The composition of this study consists of introduction, related research, research model, experiment, conclusion and discussion.
Keywords
Multi Class; Classification Algorithm; Meta-Feature; HHI; Data Complexity;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Japkowicz, N., and S. Stephen, "The Class Imbalance Problem: A Systematic Study," Intelligent Data Analysis, Vol.6, No.5(2002), 429-449.   DOI
2 Khoshgoftaar, T. M., J. Van Hulse, and A. Napolitano, "Comparing Boosting and Bagging Techniques with Noisy and Imbalanced data," IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, Vol.41, No.3(2010), 552-568.   DOI
3 Kiang, M. Y., "A Comparative Assessment of Classification Methods," Decision Support Systems, Vol.35, No.4(2003), 441-454.   DOI
4 Krawczyk, B. "Learning from Imbalanced Data: Open Challenges and Future Directions," Progress in Artificial Intelligence, Vol.5, No.4(2016), 221-232.   DOI
5 Krawczyk, B., M. Galar, Ł. Jelen, and F. Herrera, "Evolutionary Undersampling Boosting for Imbalanced Classification of Breast Cancer Malignancy," Applied Soft Computing, Vol.38 (2016), 714-726.   DOI
6 Kwon, O., and J. M. Sim, "Effects of Data Set Features on the Performances of Classification Algorithms," Expert Systems with Applications, Vol.40, No.5(2013), 1847-1857.   DOI
7 L'heureux, A., K. Grolinger, H. F. Elyamany, and M. A. Capretz, "Machine Learning with Big Data: Challenges and Approaches," IEEE Access, Vol.5(2017), 7776-7797.   DOI
8 Lopez, V., A. Fernandez, S. Garcia, V. Palade, and F. Herrera, "An Insight into Classification with Imbalanced Data: Empirical Results and Current Trends on using Data Intrinsic Characteristics," Information Sciences, Vol.250 (2013), 113-141.   DOI
9 Lorena, A. C., A. I. Maciel, P. B. de Miranda, I. G. Costa, and R. B. Prudencio, "Data Complexity Meta-Features for Regression Problems," Machine Learning, Vol.107, No.1 (2018), 209-246.   DOI
10 Lotte, F., L. Bougrain, A. Cichocki, M. Clerc, M. Congedo, A. Rakotomamonjy, and F. Yger, "A Review of Classification Algorithms for EEG-based Brain-Computer Interfaces: a 10 year update," Journal of Neural Engineering, Vol.15, No.3(2018), 031005.   DOI
11 Lu, C., J. Qiao, and J. Chang, "Herfindahl-Hirschman Index based Performance Analysis on the Convergence Development," Cluster Computing, Vol.20, No.1(2017), 121-129.   DOI
12 Matsumoto, A., U. Merlone, and F. Szidarovszky, "Some Notes on Applying the Herfindahl-Hirschman Index," Applied Economics Letters, Vol.19, No.2(2012), 181-184.   DOI
13 Nguyen, H. L., Y. K. Woon, and W. K. Ng, "A Survey on Data Stream Clustering and Classification," Knowledge and information systems, Vol.45, No.3(2015), 535-569.   DOI
14 Moran-Fernandez, L., V. Bolon-Canedo, and A. Alonso-Betanzos, "Can Classification Performance be Predicted by Complexity Measures? A Study using Microarray Data," Knowledge and Information Systems, Vol.51, No.3(2017), 1067-1090.   DOI
15 Napierala, K., and J. Stefanowski, "Types of Minority Class Examples and their Influence on Learning Classifiers from Imbalanced Data," Journal of Intelligent Information Systems, Vol.46, No.3(2016), 563-597.   DOI
16 Nemhauser, G., and L. Wolsey, "The Scope of Integer and Combinatorial Optimization," Integer and Combinatorial Optimization, (Eds.) G. Nemhauser, and L. Wolsey, John Wiley & Sons, Inc., 1988, 1-26.
17 Prabakaran, S., R. Sahu, and S. Verma, "Classification of Multi Class Dataset using Wavelet Power Spectrum," Data Mining and Knowledge Discovery, Vol.15, No.3(2007), 297-319.   DOI
18 Qiu, J., Q. Wu, G. Ding, Y. Xu, and S. Feng, "A Survey of Machine Learning for Big Data Processing," URASIP Journal on Advances in Signal Processing, No.1(2016), 67.
19 Sen, P. C., M. Hajra, and M. Ghosh, "Supervised Classification Algorithms in Machine Learning: A Survey and Review," In Emerging Technology in Modelling and Graphics. Springer, Singapore, 2020.
20 Shannon, C. E, "A Mathematical Theory of Communication," Bell System Technical Journal, Vol.27, No.3(1948), 379-423.   DOI
21 Sun, A., E.P. Lim, and Y. Liu, "On Strategies for Imbalanced Text Classification using SVM: A Comparative Study," Decision Support Systems, Vol.48, No.1(2009), 191-201.   DOI
22 Wu, G., and E. Y. Chang, "Aligning Boundary in Kernel Space for Learning Imbalanced Dataset," In Fourth IEEE International Conference on Data Mining (ICDM'04), 2004, November, 265-272.
23 Sun, Y., A. K. Wong, and M. S. Kamel "Classification of Imbalanced Data: A Review," International Journal of Pattern Recognition and Artificial Intelligence, Vol.23, No.04 (2009), 687-719.   DOI
24 Weiss, G. M., and F. Provost, "Learning when Training Data are Costly: The Effect of Class Distribution on Tree Induction," Journal of Artificial Intelligence Research, Vol.19(2003), 315-354.   DOI
25 Wolpert, D. H., and W. G. Macready, "No Free Lunch Theorems for Optimization," IEEE Transactions on Evolutionary Computation, Vol.1, No.1 (1997), 67-82.   DOI
26 Rok.B and L. Lusa, "Improved Shrunken Centroid Classifiers for High-Dimensional Class-Imbalanced Data," BMC bioinformatics, Vol.14, No.1(2013), 64.   DOI
27 Wu, X., V. Kumar, J. R. Quinlan, J. Ghosh, Q. Yang, H. Motoda, ... and Z. H. Zhou, "Top 10 Algorithms in Data Mining," Knowledge and Information Systems, Vol.14, No.1(2008), 1-37.   DOI
28 Zhang, M. L., J. M. Pena, and V. Robles, "Feature Selection for Multi-Label naive Bayes Classification," Information Sciences, Vol.179, No.19(2009), 3218-3229.   DOI
29 Zieba, M., and J. M. Tomczak, "Boosted SVM with Active Learning Strategy for Imbalanced Data," Soft Computing, Vol.19, No.12(2015), 3357-3368.   DOI
30 Zhang, X., R. Li, B. Zhang, Y. Yang, J. Guo, and X. Ji, "An Instance-based Learning Recommendation Algorithm of Imbalance Handling Methods," Applied Mathematics and Computation, Vol.351(2019), 204-218.   DOI
31 Del Rio, S., V. Lopez, J. M. Benitez, and F. Herrera, "On the use of MapReduce for Imbalanced Big Data using Random Forest," Information Sciences, Vol. 285(2014), 112-137.   DOI
32 Andric, K., D. Kalpic, and Z. Bohacek, "An Insight into the Effects of Class Imbalance and Sampling on Classification Accuracy in Credit Risk Assessment," Computer Science and Information Systems, Vol.16, No.1(2019), 155-178.   DOI
33 Bader-El-Den, M. E. Teitei, and T. Perry, "Biased Random Forest for Dealing with the Class Imbalance Problem," IEEE Transactions on Neural Networks and Learning Systems, Vol.30, No.7(2018), 2163-2172.   DOI
34 Brazdil, P., J.Gama, and B.Henery, "Characterizing the Applicability of Classification Algorithms using Meta-Level Learning," In European Conference on Machine Learning (1994, April), 83-102.
35 Brissaud, J. B, "The Meanings of Entropy," Entropy, Vol.7, No.1(2005), 68-96.   DOI
36 Chaitra, P. C., and D. R. S. Kumar, "A Review of Multi-Class Classification Algorithms," International Journal of Pure and Applied Mathematics, Vol.118, No.14(2018), 17-26.
37 Chan, P. P., Z. M. He, H. Li, and C. C.Hsu, "Data Sanitization Against Adversarial Label Contamination based on Data Complexity," International Journal of Machine Learning and Cybernetics, Vol.9, No.6(2018), 1039-1052.   DOI
38 Chawla, N. V., N. Japkowicz, and A. Kotcz, "Special Issue on Learning from Imbalanced Data Sets," ACM Sigkdd Explorations Newsletter, Vol.6, No.1(2004), 1-6.   DOI
39 Dogan, N., and Z. Tanrikulu, "A Comparative Analysis of Classification Algorithms in Data Mining for Accuracy, Speed and Robustness," Information Technology and Management, Vol.14, No.2(2013), 105-124.   DOI
40 Domingos, P. and M. Pazzani, "Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifier," in Proceedings of the 13th International Conference on Machine Learning, (1996) 105-122.
41 Galar, G., M. Fernandez, A. Barrenechea, E. Bustince, and F. Herrera, "A Review on Ensembles for the Class Imbalance Problem: Bagging-, Boosting-, and Hybrid-Based Approaches," IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), Vol.42, No.4(2011), 463-484.   DOI
42 Garcia, L. P., A. C. de Carvalho, and A. C. Lorena, "Effect of Label Noise in the Complexity of Classification Problems," Neurocomputing, Vol.160(2015), 108-119.   DOI
43 Guyon, I., and A. Elisseeff, "An Introduction to Variable and Feature Selection," Journal of Machine Learning Research, Vol.3(Mar, 2003), 1157-1182.
44 Garcia, L. P., A. C. Lorena, M. C.de Souto, and T. K. Ho, "Classifier Recommendation using Data Complexity Measures," In 2018 24th International Conference on Pattern Recognition (ICPR), (2018, August), 874-879.