Journal of the Korea Society of Computer and Information (한국컴퓨터정보학회논문지)

Korean Society of Computer Information (한국컴퓨터정보학회)
권호 표지
DOI QR코드

DOI QR Code

An Efficient Deep Learning Ensemble Using a Distribution of Label Embedding

  • Park, Saerom (Dept. of Convergence Security Engineering, Sungshin Women's University)
  • Received : 2020.11.04
  • Accepted : 2021.01.04
  • Published : 2021.01.29
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose a new stacking ensemble framework for deep learning models which reflects the distribution of label embeddings. Our ensemble framework consists of two phases: training the baseline deep learning classifier, and training the sub-classifiers based on the clustering results of label embeddings. Our framework aims to divide a multi-class classification problem into small sub-problems based on the clustering results. The clustering is conducted on the label embeddings obtained from the weight of the last layer of the baseline classifier. After clustering, sub-classifiers are constructed to classify the sub-classes in each cluster. From the experimental results, we found that the label embeddings well reflect the relationships between classification labels, and our ensemble framework can improve the classification performance on a CIFAR 100 dataset.

본 연구에서는 레이블 임베딩의 분포를 반영하는 딥러닝 모형을 위한 새로운 스태킹 앙상블 방법론을 제안하였다. 제안된 앙상블 방법론은 기본 딥러닝 분류기를 학습하는 과정과 학습된 모형으로 부터 얻어진 레이블 임베딩을 이용한 군집화 결과로부터 소분류기들을 학습하는 과정으로 이루어져 있다. 본 방법론은 주어진 다중 분류 문제를 군집화 결과를 활용하여 소 문제들로 나누는 것을 기본으로 한다. 군집화에 사용되는 레이블 임베딩은 처음 학습한 기본 딥러닝 분류기의 마지막 층의 가중치로부터 얻어질 수 있다. 군집화 결과를 기반으로 군집화 내의 클래스들을 분류하는 소분류기들을 군집의 수만큼 구축하여 학습한다. 실험 결과 기본 분류기로부터의 레이블 임베딩이 클래스 간의 관계를 잘 반영한다는 것을 확인하였고, 이를 기반으로 한 앙상블 방법론이 CIFAR 100 데이터에 대해서 분류 성능을 향상시킬 수 있다는 것을 확인할 수 있었다.

Keywords

References

  1. Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Communications of the ACM Vol. 60, No. 6, pp.84-90 May. 2017. DOI: 10.1145/3065386
  2. Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. "Sequence to sequence learning with neural networks." Advances in neural information processing systems. Vol. 27, No. 1, pp.3104-3112, Dec. 2014.
  3. Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. "The elements of statistical learning," New York: Springer series in statistics, Vol. 1. No. 10. 2001.
  4. Malinin, Andrey, Bruno Mlodozeniec, and Mark Gales. "Ensemble distribution distillation." arXiv preprint arXiv:1905.00076, 2019.
  5. Ko, Hyungjin, et al. "Loss-Driven Adversarial Ensemble Deep Learning for On-Line Time Series Analysis." Sustainability Vol. 11, No. 12, pp. 1-24 Jun. 2019 DOI: DOI:10.3390/su11123489.
  6. Sesmero, M. Paz, Juan M. Alonso-Weber, and Araceli Sanchis. "CCE: An ensemble architecture based on coupled ANN for solving multiclass problems." Information Fusion Vol. 58, pp. 132-152. Jun. 2020. DOI: 10.1016/j.inffus.2019.12.015
  7. Park, S., J. Hah, and J. Lee. "Inductive ensemble clustering using kernel support matching." Electronics Letters Vol. 53, No. 25, pp. 1625-1626 Dec. 2017. DOI: 10.1049/el.2017.2159
  8. Sun, Xu, et al. "Label embedding network: Learning label representation for soft training of deep networks." arXiv preprint arXiv:1710.10393 , 2017.
  9. Sellah, Smail, and Vincent Hilaire. "Label Clustering for a Novel Problem Transformation in Multi-label Classification." J. UCS Vol.26 No.1 pp. 71-88. Jan. 2020,
  10. Tao, Sean. "Deep neural network ensembles." International Conference on Machine Learning, Optimization, and Data Science. Springer, Cham, 2019.
  11. Low, Cheng-Yaw, Jaewoo Park, and Andrew Beng-Jin Teoh. "Stacking-based deep neural network: Deep analytic network for pattern classification." IEEE Transactions on Cybernetics, Vol. 50, No. 12, Dec. 2019. DOI: 10.1109/TCYB.2019.2908387
  12. Liu, J. Z., Lin, Z., Padhy, S., Tran, D., Bedrax-Weiss, T., & Lakshminarayanan, B. "Simple and principled uncertainty estimation with deterministic deep learning via distance awareness." Advances in Neural Information Processing Systems Dec. 2020.
  13. Gabrie, M., Manoel, A., Luneau, C., Barbier, J., Macris, N., Krzakala, F., & Zdeborova, L. "Entropy and mutual information in models of deep neural networks." Journal of Statistical Mechanics: Theory and Experiment, Vol. 50, No. 12, pp. 5021-5034 Dec, 2020. DOI: 10.1109/TCYB.2019.2908387
  14. Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., ... & Nahavandi, S. A. "Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges." arXiv preprint arXiv:2011.06225. 2020