The Korean Journal of Applied Statistics (응용통계연구)

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

A divide-oversampling and conquer algorithm based support vector machine for massive and highly imbalanced data

불균형의 대용량 범주형 자료에 대한 분할-과대추출 정복 서포트 벡터 머신

  • Bang, Sungwan (Department of Mathematics, Korea Military Academy) ;
  • Kim, Jaeoh (Department of Data Science, Inha University)
  • 방성완 (육군사관학교 수학과) ;
  • 김재오 (인하대학교 데이터사이언스학과)
  • Received : 2021.06.02
  • Accepted : 2021.08.27
  • Published : 2022.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

The support vector machine (SVM) has been successfully applied to various classification areas with a high level of classification accuracy. However, it is infeasible to use the SVM in analyzing massive data because of its significant computational problems. When analyzing imbalanced data with different class sizes, furthermore, the classification accuracy of SVM in minority class may drop significantly because its classifier could be biased toward the majority class. To overcome such a problem, we propose the DOC-SVM method, which uses divide-oversampling and conquers techniques. The proposed DOC-SVM divides the majority class into a few subsets and applies an oversampling technique to the minority class in order to produce the balanced subsets. And then the DOC-SVM obtains the final classifier by aggregating all SVM classifiers obtained from the balanced subsets. Simulation studies are presented to demonstrate the satisfactory performance of the proposed method.

일반적으로 support vector machine (SVM)은 높은 수준의 분류 정확도를 제공함으로써 다양한 분야의 분류분석에서 널리 사용되고 있다. 그러나 SVM은 최적화 계산식이 이차계획법(quadratic programming)으로 공식화되어 많은 계산 비용이 필요하므로 대용량 자료의 분류분석에는 그 사용이 제한된다. 또한 불균형 자료(imbalanced data)의 분류분석에서는 다수집단에 편향된 분류함수를 추정함으로써 대부분의 자료를 다수집단으로 분류하여 소수집단의 분류 정확도를 현저히 감소시키게 된다. 이러한 문제점들을 해결하기 위하여 본 논문에서는 다수집단을 분할(divide)하고, 소수집단을 과대추출(oversampling)하여 여러 분류함수들을 추정하고 이들을 통합(conquer)하는 DOC-SVM 분류기법을 제안한다. 제안한 DOC-SVM은 분할정복 알고리즘을 다수집단에 적용하여 SVM의 계산 효율을 향상시키고, 과대추출 알고리즘을 소수집단에 적용하여 SVM 분류함수의 편향을 줄이게 된다. 본 논문에서는 모의실험과 실제자료 분석을 통해 제안한 DOC-SVM의 효율적인 성능과 활용 가능성을 확인하였다.

Keywords

Acknowledgement

김재오의 연구는 인하대학교 연구비 지원을, 방성완의 연구는 정부(과학기술정보통신부)의 재원으로 한국연구재단의 지원을 받아 수행된 연구임(NO. 2020R1F1A1A01065107).

References

  1. Akbani R, Kwek S, and Japkowicz N (2004). Applying support vector machines to imbalanced datasets. In Proceedings of European Conference of Machine Learning, 3201, 39-50.
  2. Anand A, Pugalenthi G, Fogel GB, and Suganthan PN (2010). An approach for classification of highly imbalanced data using weighting and undersampling, Amino Acids, 39, 1385-1391. https://doi.org/10.1007/s00726-010-0595-2
  3. Bang S, Han SK, and Kim J (2021). Divide and conquer algorithm based support vector machine for massive data analysis, Journal of the Korean Data & Information Science Society, 32, 463-473. https://doi.org/10.7465/jkdi.2021.32.3.463
  4. Bang S and Jhun M (2014). Weighted support vector machine using k-means clustering, Communications in Statistics-Simulation and Computation, 43, 2307-2324. https://doi.org/10.1080/03610918.2012.762388
  5. Bang S and Kim J (2020a). Divide and conquer kernel quantile regression for massive dataset, The Korean Journal of Applied Statistics, 33, 569-578. https://doi.org/10.5351/KJAS.2020.33.5.569
  6. Bang S and Kim J (2020b). Sampling method using Gaussian mixture clustering for classification analysis of imbalanced data, Journal of the Korean Data Analysis Society, 22, 565-574. https://doi.org/10.37727/jkdas.2020.22.2.565
  7. Bunkhumpornpat C, Sinapiromsaran K, and Lursinsap C (2009). Safe-level-SMOTE: safe-level-synthetic minority over-sampling technique for handling the class imbalanced problem. In Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, 475--482.
  8. Bunkhumpornpat C, Sinapiromsaran K, and Lursinsap C (2012). DBSMOTE: density-based synthetic minority over-sampling technique, Applied Intelligence, 36, 664--684. https://doi.org/10.1007/s10489-011-0287-y
  9. Chawla N, Bowyer K, Hall L, and Kegelmeyer W (2002). SMOTE: synthetic minority over-sampling technique, Journal of Artificial Intelligence Research, 16, 321-357. https://doi.org/10.1613/jair.953
  10. Chen X, Liu W, and Zhang Y (2019). Quantile regression under memory constraint, Annals of Statistics, 47, 3244-3273.
  11. Chen X and Xie M (2014). A split-and-conquer approach for analysis of extraordinarily large data, Statistica Sinica, 24, 1655-1684.
  12. Chen L and Zhou Y (2020). Quantile regression in big data: A divide and conquer based strategy, Computational Statistics and Data Analysis, 144, 1-17.
  13. Cristianini N and Shawe-Taylor J (2000). An Introduction to Support Vector Machines, Cambridge University Press, Cambridge.
  14. Cortes C and Vapnik V (1995). Support vector networks, Machine Learning, 20, 273-297. https://doi.org/10.1007/BF00994018
  15. Datta S and Das S (2015). Near-Bayesian support vector machines for imbalanced data classification with equal or unequal misclassification costs, Neural Networks, 70, 39-52. https://doi.org/10.1016/j.neunet.2015.06.005
  16. Dua D and Graff C (2019). UCI Machine Learning Repository, Irvine, CA: University of California, School of Information and Computer Science.
  17. Fan T, Lin D, and Cheng K (2007). Regression analysis for massive datasets, Data and Knowledge Engineering, 61, 554-562. https://doi.org/10.1016/j.datak.2006.06.017
  18. Han H, Wang WY, and Mao BH (2005). Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning, Lecture Notes in Computer Science, 3644, 878-887.
  19. He H, Bai Y, Garcia EA, and Li S (2008). ADASYN: adaptive synthetic samplingapproach for imbalanced learning. In Proceedings of the 2008 IEEE International Joint Conference Neural Networks, 1322-1328.
  20. Hsieh C and Dhillon I (2014). A divide and conquer solver for kernel support vector machines. In Proceedings of the 31st International Conference on Machine Learning.
  21. Jeong H, Kang C, and Kim K (2008). The effect of oversampling method for imbalanced data, Journal of the Korean Data Analysis Society, 10, 2089-2098.
  22. Jiang R, Hu X, Yu K, and Qian W (2018). Composite quantile regression for massive datasets, Statistics, 52, 980-1004. https://doi.org/10.1080/02331888.2018.1500579
  23. Kang J and Jhun M (2020). Divide-and-conquer random sketched kernel ridge regression for large-scale data, Journal of the Korean Data & Information Science Society, 31, 15-23. https://doi.org/10.7465/jkdi.2020.31.1.15
  24. Kim E, Jhun M, and Bang S (2016). Hierarchically penalized support vector machine for the classification of imbalanced data with grouped variables, The Korea Journal of Applied Statistics, 29, 961-975. https://doi.org/10.5351/KJAS.2016.29.5.961
  25. Lian H and Fan Z (2018). Divide-and-conquer for debiased l1-norm support vector machine in ultra-high dimensions, Journal of Machine Learning Research, 18, 1-26.
  26. Lin Y, Lee Y, and Wahba G (2002). Support Vector Machines for Classification in Nonstandard Situations, Machine Learning, 46, 191-202. https://doi.org/10.1023/a:1012406528296
  27. Ling CX and Sheng VS (2008). Cost-sensitive learning and the class imbalance problem, Encyclopedia of Machine Learning, 2011, 231-235.
  28. Meyer D, Dimitriadou E, Hornik K, Weingessel A, Leisch F, Chang CC, and Meyer MD (2019). Package 'e1071', The R Journal.
  29. Oommen T, Baise LG, and Vogel RM (2011). Sampling bias and class imbalance in maximum-likelihood logistic regression, Mathematical Geosciences, 43, 99-120. https://doi.org/10.1007/s11004-010-9311-8
  30. Owen AB (2007). Infinitely imbalanced logistic regression, The Journal of Machine Learning Research, 8, 761-773.
  31. Park J and Bang S (2015). Logistic regression with sampling techniques for the classification of imbalanced data, Journal of The Korean Data Analysis Society, 17, 1877-1888.
  32. Tang Y, Zhang YQ, Chawla NV, and Krasser S (2009). SVMs modeling for highly imbalanced classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 39, 281-288. https://doi.org/10.1109/TSMCB.2008.2002909
  33. Vapnik VN (1998). Statistical Learning Theory, Wiley, New York.
  34. Veropoulos K, Campbell C, and Cristianini N (1999). Controlling the sensitivity of support vector machines. In Proceedings of the International Joint Conference on AI, 55-60.
  35. Xu Q, Cai C, Jiang C, Sun F, and Huang X (2020). Block average quantile regression for massive dataset, Statistical Papers, 61, 141-165. https://doi.org/10.1007/s00362-017-0932-6
  36. Zhang Y, Duchi J, and Wainwright M (2015). Divide and conquer kernel ridge regression: A distributed algorithm with minimax optimal rates, Journal of Machine Learning Research, 16, 3299-3340.
  37. Zhang YP, Zhang LN, and Wang YC (2010). Cluster-based majority under-sampling approaches for class imbalance learning. In Information and Financial Engineering (ICIFE) 2010 2nd IEEE International Conference, 400-404.