Industrial Engineering and Management Systems

Korean Institute of Industrial Engineers (대한산업공학회)
권호 표지
DOI QR코드

DOI QR Code

Optimization of Decision Tree for Classification Using a Particle Swarm

  • Cho, Yun-Ju (Department of Industrial and Management Engineering Pohang University of Science and Technology) ;
  • Lee, Hye-Seon (Department of Industrial and Management Engineering Pohang University of Science and Technolog) ;
  • Jun, Chi-Hyuck (Department of Industrial and Management Engineering Pohang University of Science and Technology)
  • Received : 2011.09.04
  • Accepted : 2011.10.31
  • Published : 2011.12.01
Download PDF
⟨ Previous Next ⟩

Abstract

Decision tree as a classification tool is being used successfully in many areas such as medical diagnosis, customer churn prediction, signal detection and so on. The main advantage of decision tree classifiers is their capability to break down a complex structure into a collection of simpler structures, thus providing a solution that is easy to interpret. Since decision tree is a top-down algorithm using a divide and conquer induction process, there is a risk of reaching a local optimal solution. This paper proposes a procedure of optimally determining thresholds of the chosen variables for a decision tree using an adaptive particle swarm optimization (APSO). The proposed algorithm consists of two phases. First, we construct a decision tree and choose the relevant variables. Second, we find the optimum thresholds simultaneously using an APSO for those selected variables. To validate the proposed algorithm, several artificial and real datasets are used. We compare our results with the original CART results and show that the proposed algorithm is promising for improving prediction accuracy.

Keywords

References

  1. Athanasios, P. and Dimitris, K. (2001), Breeding Decision Trees Using Evolutionary Techniques,Proceedings of the Eighteenth International Conference on Machine Learning, 393-400.
  2. Bennett, K. P. and Mangasarian O. L. (1994), Multicategory Discrimination via Linear Programming, Optimization Methods and Software, 3, 29-39.
  3. Breiman, L., Friedman, J. H., Olashen, R. A. and Stone, C. J. (1984), Classification and Regression Trees, Chapman and Hall/CRC, London, UK.
  4. Clerc, M. (1999), TheSwarm and the Queen: Towards a Deterministic and Adaptive Particle Swarm Optimization, Proceedings of the 1999 Congress on Evolutionary Computation, 1951-1957.
  5. Duda, R. and Hart, P. (1973), Pattern Classification and Scene Analysis, A Wiley-Interscience Publication, New York.
  6. Eberhart, R. C. and Shi, Y. (2001), Particle Swarm Optimization: Developments, Applications and Resources, Proceedings of the 2001 Congress on Evolutionary Computation, 81-86.
  7. Frank, A. and Asuncion, A. (2010), UCI Machine Learning Repository (http://archive.ics.uci.edu/ml), Irvine, CA.
  8. Hyafil, L. and Rivest, R. L. (1976), Constructing Optimal Binary Decision Trees is NP-Complete, Information Processing Letters, 5, 15-17. https://doi.org/10.1016/0020-0190(76)90095-8
  9. Kass, G. V. (1980), An Exploratory Technique for Investigating Large Quantities of Categorical Data, Applied Statistics, 29, 119-127. https://doi.org/10.2307/2986296
  10. Kennedy, J. and Eberhart, R. C. (1995), Particle Swarm Optimization, Proceedings of IEEE International Conference on Neural Networks, 1942-1948.
  11. Lior, R. and Oded, M. (2005), Top-Down Induction of Decision Trees Classifiers-A Survey, IEEE Transactions on Systems, Man, and Cybernetics-Part C: Applications and Reviews, 35, 476-487. https://doi.org/10.1109/TSMCC.2004.843247
  12. Murthy, S. K. (1998), Automatic Construction of Decision Trees from Data: A Multidisciplinary Survey, Data Mining and Knowledge Discovery, 2, 345-389. https://doi.org/10.1023/A:1009744630224
  13. Naumov, G. E. (1991), NP-Completeness of Problems of Construction of Optimal Decision Trees. Soviet Physics, 36, 270-271.
  14. Quinlan, J. R. (1993), C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers Inc, San Francisco, CA.
  15. Saher, E. and Shaul, M. (2007), Anytime Learning of Decision Trees, Journal of Machine Learning Research, 8, 891-933.
  16. Schuermann, J. and Doster, W. (1984), A Decision-theoretic Approach in Hierarchical Classifier Design, Pattern Recognition, 17, 359-369. https://doi.org/10.1016/0031-3203(84)90087-6
  17. Shi, Y. and Eberhart, R. C. (2001), Fuzzy Adaptive Particle Swarm Optimization, Proceedings of the 2001 Congress on Evolutionary Computation, 101-106.
  18. Utgoff, P. E. (1989), Perceptron Trees: A Case Study in Hybrid Concept Representations, Connection Science, 1, 377-391. https://doi.org/10.1080/09540098908915648
  19. Xie, X. F., Zhang, W. J., and Yang, Z. L. (2002), Adaptive Particle Swarm Optimization on Individual Level, International Conference of 2002 6th on Signal Processing, 1215-1218.
  20. Zhan, Z. H., Jun, Z., Yun, L. and Chung, H. S. (2009), Adaptive Particle Swarm Optimization, IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics, 39, 1362-1381. https://doi.org/10.1109/TSMCB.2009.2015956