The Journal of Society for e-Business Studies (한국전자거래학회지)

Society for e-Business Studies (한국전자거래학회)
권호 표지
DOI QR코드

DOI QR Code

A Comparative Analysis of Ensemble Learning-Based Classification Models for Explainable Term Deposit Subscription Forecasting

설명 가능한 정기예금 가입 여부 예측을 위한 앙상블 학습 기반 분류 모델들의 비교 분석

  • Shin, Zian (Department of Security Convergence Science, Chung-Ang University) ;
  • Moon, Jihoon (Chung-Ang University) ;
  • Rho, Seungmin (Department of Industrial Security, Chung-Ang University)
  • Received : 2021.07.26
  • Accepted : 2021.08.18
  • Published : 2021.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

Predicting term deposit subscriptions is one of representative financial marketing in banks, and banks can build a prediction model using various customer information. In order to improve the classification accuracy for term deposit subscriptions, many studies have been conducted based on machine learning techniques. However, even if these models can achieve satisfactory performance, utilizing them is not an easy task in the industry when their decision-making process is not adequately explained. To address this issue, this paper proposes an explainable scheme for term deposit subscription forecasting. For this, we first construct several classification models using decision tree-based ensemble learning methods, which yield excellent performance in tabular data, such as random forest, gradient boosting machine (GBM), extreme gradient boosting (XGB), and light gradient boosting machine (LightGBM). We then analyze their classification performance in depth through 10-fold cross-validation. After that, we provide the rationale for interpreting the influence of customer information and the decision-making process by applying Shapley additive explanation (SHAP), an explainable artificial intelligence technique, to the best classification model. To verify the practicality and validity of our scheme, experiments were conducted with the bank marketing dataset provided by Kaggle; we applied the SHAP to the GBM and LightGBM models, respectively, according to different dataset configurations and then performed their analysis and visualization for explainable term deposit subscriptions.

정기예금 가입 여부 예측은 은행의 대표적인 금융 마케팅 중 하나로, 은행은 다양한 고객 정보를 활용하여 예측 모델을 구성할 수 있다. 정기예금 가입 여부의 분류 정확도를 향상하기 위해, 많은 연구에서 기계학습 기법들을 이용하여 분류 모델들을 개발하였다. 하지만, 이러한 모델들이 만족스러운 성능을 보일지라도 모델의 의사결정 과정에 대한 근거가 적절하게 설명되지 않는다면 산업에서 활용하기가 쉽지 않다. 이러한 문제점을 해결하기 위해, 본 논문은 설명 가능한 정기예금 가입 여부 예측 기법을 제안한다. 먼저, 테이블 형식에서 우수한 성능을 도출하는 의사결정 나무 기반 앙상블 학습 기법인 랜덤 포레스트, GBM, XGBoost, LightGBM을 이용하여 분류 모델들을 개발하고, 10겹 교차검증을 통해 모델들의 분류 성능을 심층 분석한다. 다음으로, 가장 우수한 성능을 도출하는 모델에 설명 가능한 인공지능 기법인 SHAP을 적용하여 고객 정보의 영향도와 의사결정 과정 등을 해석할 수 있는 근거를 제공한다. 제안한 기법의 실용성과 타당성을 입증하기 위해, Kaggle에서 제공한 은행 마케팅 데이터 셋을 대상으로 모의실험을 진행하였으며, 데이터 셋 구성에 따라 GBM과 LightGBM 모델에 SHAP을 각기 적용하여 설명 가능한 정기예금 가입 여부를 위한 분석 및 시각화를 수행하였다.

Keywords

Acknowledgement

This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2021-2018-0-01799) supervised by the IITP (Institute for Information & communications Technology Planning & Evaluation) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2019R1F1A1060668).

References

  1. Adadi, A. and Berrada, M., "Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)," IEEE Access, Vol. 6, pp. 52138-52160, 2018. https://doi.org/10.1109/ACCESS.2018.2870052
  2. Ahmadi, A., Nabipour, M., MohammadiIvatloo, B., Amani, A. M., Rho, S., and Piran, M. J., "Long-Term Wind Power Forecasting Using Tree-Based Learning Algorithms," IEEE Access, Vol. 8, pp. 151511-151522, 2020. https://doi.org/10.1109/ACCESS.2020.3017442
  3. Altman, N. and Krzywinski, M., "Ensemble methods: bagging and random forests," Nature Methods, Vol. 14, No. 10, pp. 933-935, 2017. https://doi.org/10.1038/nmeth.4438
  4. Belgiu, M. and Dragut, L., "Random forest in remote sensing: A review of applications and future directions," ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 114, pp. 24-31, 2016. https://doi.org/10.1016/j.isprsjprs.2016.01.011
  5. Chen, T. and Guestrin, C., "XGBoost: A Scalable Tree Boosting System," in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785-794, 2016.
  6. Chun, Y. E., Kim, S. B., Lee, J. Y., and Woo, J. H., "Study on credit rating model using explainable AI," Journal of the Korean Data and Information Science Society, Vol. 32, No. 2, pp. 283-295, 2021. https://doi.org/10.7465/jkdi.2021.32.2.283
  7. Chun, Y. E., Park, Y., Sung, N., and Choi, J., "Model analysis using estimation of shapley value on classification of sentences explaining causes of changes in stock prices," KIISE Transactions on Computing Practices, Vol. 26, No. 4, pp. 195-201, 2020. https://doi.org/10.5626/ktcp.2020.26.4.195
  8. Jung, C. and Lee, H., "A comparative study of explainable AI techniques for process analysis," Journal of the Institute of Electronics and Information Engineers, Vol. 57, No. 8, pp. 51-59, 2020. https://doi.org/10.5573/ieie.2020.57.8.51
  9. Hung, P. D., Hanh, T. D., and Tung, T. D., "Term deposit subscription prediction using spark MLlib and ML packages," in Proceedings of the 2019 5th International Conference on E-Business and Applications, pp. 88-93, 2019.
  10. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y., "LightGBM: A highly efficient gradient boosting decision tree," in Proceedings of the 31st International Conference on Neural Information Processing Systems, Vol. 30, pp. 3146-3154, 2017.
  11. Kim, S., Kim, W., Jang, Y., and Kim, H., "Development of Explainable AI-Based Learning Support System," The Journal of Korean Association of Computer Education, Vol. 24, No. 1, pp. 107-115, 2021. https://doi.org/10.32431/KACE.2021.24.1.012
  12. Kwon, B. C., Choi, M.-J., Kim, J. T., Choi, E., Kim, Y. B., Kwon, S., Sun, J., and Choo, J., "RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records," IEEE Transactions on Visualization and Computer Graphics, Vol. 25, No. 1, pp. 299-309, 2018. https://doi.org/10.1109/TVCG.2018.2865027
  13. Landis, J. R. and Koch, G. G., "An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers," Biometrics, pp. 363-374, 1977.
  14. Lee, D. Y. and Hwang, B. S., "Performance comparison of algorithm for the prediction of time deposit," in Proceedings of the Korea Computer Congress, pp. 2074-2076, 2018.
  15. Lee, Y.-G., Oh, J.-Y., and Kim, G., "Interpretation of load forecasting using explainable artificial intelligence techniques," The Transactions of the Korean Institute of Electrical Engineers, Vol. 69, No. 3, pp. 480-485, 2020. https://doi.org/10.5370/kiee.2020.69.3.480
  16. Lim, M. and Jang, H., "A Study on the Risk Reduction Plan of Cryptocurrency Exchange," Journal of Platform Technology, Vol. 8, No. 4, pp. 29-37, 2020. https://doi.org/10.23023/JPT.2020.8.4.029
  17. Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., and Lee, S.-I., "From local explanations to global understanding with explainable AI for trees," Nature Machine Intelligence, Vol. 2, No. 1, pp. 56-67, 2020. https://doi.org/10.1038/s42256-019-0138-9
  18. Mangalathu, S., Hwang, S. H., and Jeon, J. S., "Failure mode and effects analysis of RC members based on machine-learning-based SHapley Additive exPlanations (SHAP) approach," Engineering Structures, Vol. 219, p. 110927, 2020. https://doi.org/10.1016/j.engstruct.2020.110927
  19. Moon, J., Jung, S., Rew, J., Rho, S., and Hwang, E., "Combination of short-term load forecasting models based on a stacking ensemble approach," Energy and Buildings, Vol. 216, p. 109921, 2020. https://doi.org/10.1016/j.enbuild.2020.109921
  20. Moon, J., Kim, J., Kang, P., and Hwang, E., "Solving the Cold-Start Problem in Short-Term Load Forecasting Using Tree-Based Methods," Energies, Vol. 13, No. 4, p. 886, 2020. https://doi.org/10.3390/en13040886
  21. Moon, J., Kim, Y., Son, M., and Hwang, E., "Hybrid short-term load forecasting scheme using random forest and multilayer perceptron," Energies, Vol. 11, No. 12, p. 3283, 2018. https://doi.org/10.3390/en11123283
  22. Moro, S., Cortez, P., and Rita, P., "A data-driven approach to predict the success of bank telemarketing," Decision Support Systems, Vol. 62, pp. 22-31, 2014. https://doi.org/10.1016/j.dss.2014.03.001
  23. Natekin, A. and Knoll, A., "Gradient boosting machines, a tutorial," Frontiers in Neurorobotics, Vol. 7, p. 21, 2013. https://doi.org/10.3389/fnbot.2013.00021
  24. Oh, H. R., Son, A. L., and Lee, Z., "Occupational accident prediction modeling and analysis using SHAP," Journal of Digital Contents Society, Vol. 22, No. 7, pp. 1115-1123, 2021. https://doi.org/10.9728/dcs.2021.22.7.1115
  25. Oshiro, T. M., Perez, P. S., and Baranauskas, J. A., "How Many Trees in a Random Forest?," Proceedings of the International Workshop on Machine Learning and Data Mining in Pattern Recognition, pp. 154-168, 2012.
  26. Park, J., Moon, J., Jung, S., and Hwang, E., "Multistep-ahead solar radiation forecasting scheme based on the light gradient boosting machine: A case study of Jeju Island," Remote Sensing, Vol. 12, No. 14, p. 2271, 2020. https://doi.org/10.3390/rs12142271
  27. Park, S. H., Lee, J. H., Jung, Y. W., and Won, Y. J., "Performance comparison of periodic deposit prediction using machine learning," Proceedings of the Korea Software Congress, pp. 2139-2141, 2018.
  28. Park, S., Moon, J., Jung, S., Jung, S., and Hwang, E., "SHAP-based Explainable Influenza Occurrence Forecasting using LightGBM," Proceedings of the Korea Software Congress, pp. 666-668, 2020.
  29. Park, S., Moon, J., and Hwang, E., "Explainable anomaly detection for district heating based on shapley additive explanations," Proceedings of the 2020 International Conference on Data Mining Workshops (ICDMW), pp. 762-765, 2020.
  30. Park, S., Moon, J., Jung, S., Rho, S., and Baik, S. W., Hwang, E., "A two-stage industrial load forecasting scheme for day-ahead combined cooling, heating and power scheduling," Energies, Vol. 13, No. 2, p. 443, 2020. https://doi.org/10.3390/en13020443
  31. Park, W. and Jang, H., "A study on implementing a priority tasks for invigoration of cloud in financial sector," Journal of Platform Technology, Vol. 8, No. 1, pp. 10-15, 2020. https://doi.org/10.23023/JPT.2020.8.1.010
  32. Parlar, T., "Using Data Mining Techniques for detecting the important features of the bank direct marketing data," International Journal of Economics and Financial Issues, Vol. 7, No. 2, p. 692, 2017.
  33. Rew, J., Cho, Y., Moon, J., and Hwang, E., "Habitat suitability estimation using a two-stage ensemble approach," Remote Sensing, Vol. 12, No. 9, p. 1475, 2020. https://doi.org/10.3390/rs12091475
  34. Rew, J., Kim, H., and Hwang, E., "Hybrid segmentation scheme for skin feature extraction using dermoscopy images," Computers, Materials & Continua, Vol. 69, No. 1, pp. 801-817, 2021. https://doi.org/10.32604/cmc.2021.017892
  35. Ribeiro, M. H. D. M., and dos Santos Coelho, L., "Ensemble approach based on bagging, boosting and stacking for short-term prediction in agribusiness time series," Applied Soft Computing, Vol. 86, p. 105837, 2020. https://doi.org/10.1016/j.asoc.2019.105837
  36. Rodriguez, J. D., Perez, A., and Lozano, J. A., "Sensitivity analysis of k-fold cross validation in prediction error estimation," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 3, pp. 569-575, 2009. https://doi.org/10.1109/TPAMI.2009.187
  37. Sun, J. C. and Kim, I. S., "Improvement of selective consent method in the collection process of personal information of financial institutions," The Journal of Society for e-Business Studies, Vol. 25, No. 1, pp. 123-134, 2020.