Browse > Article
http://dx.doi.org/10.11627/jkise.2021.44.2.015

Interpretability Comparison of Popular Decision Tree Algorithms  

Hong, Jung-Sik (Dept. of Industrial Engineering, Seoul National University of Science and Technology)
Hwang, Geun-Seong (Dept. of Data Science, Graduate School, Seoul National University of Science and Technology)
Publication Information
Journal of Korean Society of Industrial and Systems Engineering / v.44, no.2, 2021 , pp. 15-23 More about this Journal
Abstract
Most of the open-source decision tree algorithms are based on three splitting criteria (Entropy, Gini Index, and Gain Ratio). Therefore, the advantages and disadvantages of these three popular algorithms need to be studied more thoroughly. Comparisons of the three algorithms were mainly performed with respect to the predictive performance. In this work, we conducted a comparative experiment on the splitting criteria of three decision trees, focusing on their interpretability. Depth, homogeneity, coverage, lift, and stability were used as indicators for measuring interpretability. To measure the stability of decision trees, we present a measure of the stability of the root node and the stability of the dominating rules based on a measure of the similarity of trees. Based on 10 data collected from UCI and Kaggle, we compare the interpretability of DT (Decision Tree) algorithms based on three splitting criteria. The results show that the GR (Gain Ratio) branch-based DT algorithm performs well in terms of lift and homogeneity, while the GINI (Gini Index) and ENT (Entropy) branch-based DT algorithms performs well in terms of coverage. With respect to stability, considering both the similarity of the dominating rule or the similarity of the root node, the DT algorithm according to the ENT splitting criterion shows the best results.
Keywords
Interpretability Comparison; Stability Measure; Decision Tree;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Li, R.-H. and Belford, G.G., Instability of Decision Tree Classification Algorithms, In Proceedings of the 8 th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Edmonton, Alberta, Canada, 2002, pp. 570-575.
2 Lustrek, M., Gams, M., and Martincic-Ipsic, S., What Makes Classification Trees Comprehensible?, Expert Syst. Appl., 2016, Vol. 62, pp. 333-346.   DOI
3 Martens, D., Vanthienen, J., Verbeke, W., and Baesens, B., Performance of Classification Models from a User Perspective, Decision Support Systems, 2011, Vol. 51, No. 4, pp. 782-793.   DOI
4 Murdoch, W.J., Singh, C., Kumbier, K., Abbasi-Asl, R., and Yu, B., Interpretable Machine Learning : Definitions, Methods, and Applications, arXiv e-prints, p. arXiv:1901.04592, 2019.
5 Wang, Y., Xia, S.-T., and Wu, J., A Less-Greedy Two-Term Tsallis Entropy Information Metric Approach for Decision Tree Classification, Knowledge-Based Systems, 2017, Vol. 120, pp. 34-42.
6 Zhao, Y. and Zhang, Y., Comparison of Decision Tree Methods for Finding Active Objects, Advances in Space Research, 2008, Vol. 41, No. 12, pp. 1955-1959.   DOI
7 Leiva, R.G., Anta, A.F., Mancuso, V., and Casari, P., A Novel Hyperparameter-Free Approach to Decision Tree Construction that Avoids Overfitting by Design, IEEE Access, 2019, Vol. 7, pp. 99978-99987.   DOI
8 An, A. and Cercone, N., Rule Quality Measures for Rule Induction Systems, Computational Intelligence, 2001, Vol. 17, No. 3, pp. 409-424.   DOI
9 Verbeke, W., Marteens, D., Mues, C., and Baesens, B., Building Comprehensible Customer Churn Prediction Models with Advanced Rule Induction Techniques, Expert Systems with Applications, 2011, Vol. 38, No. 3, pp. 2354-2364.   DOI
10 Dannegger, F., Tree Stability Diagnotics and Some Remedies for Instability, Statistics in Medicine, 2000, Vol. 19, No. 4, pp. 475-491.   DOI
11 Garcia, S., Fernandez, A., and Herrera, F., Enhancing the Effectiveness and Interpretability of Decision Tree and Rule Induction Classifiers with Evolutionary Training Set Selection over Imbalanced Problems, Applied Soft Computing, 2009, Vol. 9, No. 4, pp. 1304-1314.   DOI
12 Goldstein, A. and Buja, A., Penalized Split Criteria for Interpretable Trees, arXiv preprint arXiv:1310.5677, 2013.
13 Norouzi, M., Collins, M., Johnson, M.A., Fleet, D.J., and Kohli, P., Efficient Non-Greedy Optimization of Decision Trees, Proceedings of the 28th International Conference on Neural Informsation Processing, 2015, pp. 1729-1737.
14 Pazzani, M.J., Mani, S., and Shankle, W.R., Acceptance of Rules Generated by Machine Learning Among Medical Experts, Methods of Information in Medicine, 2001, Vol. 40, No. 5, pp. 380-385.   DOI
15 Quinlan, J.R., Induction of Decision Trees, Machine Learning, 1986, Vol. 1, No. 1, pp. 81-106.   DOI
16 Belle, V. and Papantonis, I., Principles and Practice of Explainable Machine Learning, arXiv preprint arXiv: 2009.11698, 2020.
17 Wang, Y. and Xia, S.-T., Unifying Attribute Splitting Criteria of Decision Trees by Tsallis Entropy, in Proc. IEEE Int. Conf. Acoust., Speech Signal Process(ICASSP), 2017, pp. 2507-2511.
18 Buntine, W. and Niblett, T., A Further Comparison of Splitting Rules for Decision-Tree Induction, Machine Learning, 1992, Vol. 8, No. 1, pp. 75-85.   DOI
19 Quinlan, J.R., C4.5 : Programs for Machine Learning, San Francisco, CA, USA : Morgan Kaufmann, 1993.
20 Baesens, B., Mues, C., De Backer, M., and Vanthienen, J., Building Intelligent Credit Scoring Systems Using Decision Tables, In : Enterprise Information Systems V, 2004, pp. 131-137.
21 Cano, A., Zafra, A., and Ventura, S., An Interpretable Classification Rule Mining Algorithm, Information Sciences, 2013, Vol. 240, pp. 1-20.   DOI
22 Breiman, L., Friedman, J.H., Olshen, R.A., and Stone, C.J., Classification and Regression Trees, Monterey, CA, USA : Wadsworth and Brooks, 1984.
23 Breiman, L., Heuristics of Instability and Stabilization in Model Selection, The Annals of Statistics, 1996, Vol. 24, No. 6, pp. 2350-2383.   DOI
24 Briand, B., Ducharme, G.R., Parache, V., and Mercat-Rommens, C., A Similarity Measureto Assess the Stability of Classification Trees, Computational Statistics and Data Analysis, 2009, Vol. 53, No. 4, pp. 1208-1217.   DOI
25 Chandra, B., Kothari, R., and Paul, P., A New Node Splitting Measure for Decision Tree Construction, Pattern Recognition, 2010, Vol. 43, No. 8, pp. 2725-2731.   DOI
26 Jacobucci, R., Decision Tree Stability and its Effect on Interpretation, Retrieved from osf.io/m5p2v, 2018.
27 Jaworski, M., Duda, P., and Rutkowski, L., New Splitting Criteria for Decision Trees in Stationary Data Streams, IEEE Trans. Neural Netw. Learn. Syst., 2018, Vol. 29, No. 6, pp. 2516-2529.   DOI
28 Kotsiantis, S.B., Supervised Machine Learning : A Review of Classification Techniques, Informatica, 2007, Vol. 31, No. 3, pp. 249-268.
29 Harris, E., Information Gain Versus Gain Ratio : A Study of Split Method Biases, The MITRE Corp., McLean, VI, USA, Tech. Rep., 2001.
30 Freitas, A.A., Comprehensible Classification Models : A Position Paper, ACM SIGKDD Explorations Newsletter, 2014, Vol. 15, No. 1, pp. 1-10.   DOI