Browse > Article
http://dx.doi.org/10.9708/jksci.2021.26.01.027

An Efficient Deep Learning Ensemble Using a Distribution of Label Embedding  

Park, Saerom (Dept. of Convergence Security Engineering, Sungshin Women's University)
Publication Information
Journal of the Korea Society of Computer and Information / v.26, no.1, 2021 , pp. 27-35 More about this Journal
Abstract
In this paper, we propose a new stacking ensemble framework for deep learning models which reflects the distribution of label embeddings. Our ensemble framework consists of two phases: training the baseline deep learning classifier, and training the sub-classifiers based on the clustering results of label embeddings. Our framework aims to divide a multi-class classification problem into small sub-problems based on the clustering results. The clustering is conducted on the label embeddings obtained from the weight of the last layer of the baseline classifier. After clustering, sub-classifiers are constructed to classify the sub-classes in each cluster. From the experimental results, we found that the label embeddings well reflect the relationships between classification labels, and our ensemble framework can improve the classification performance on a CIFAR 100 dataset.
Keywords
Deep Ensemble Learning; Clustering; Multi-class classification; Label embedding; Stacking Ensemble Model;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Communications of the ACM Vol. 60, No. 6, pp.84-90 May. 2017. DOI: 10.1145/3065386   DOI
2 Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. "Sequence to sequence learning with neural networks." Advances in neural information processing systems. Vol. 27, No. 1, pp.3104-3112, Dec. 2014.
3 Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. "The elements of statistical learning," New York: Springer series in statistics, Vol. 1. No. 10. 2001.
4 Low, Cheng-Yaw, Jaewoo Park, and Andrew Beng-Jin Teoh. "Stacking-based deep neural network: Deep analytic network for pattern classification." IEEE Transactions on Cybernetics, Vol. 50, No. 12, Dec. 2019. DOI: 10.1109/TCYB.2019.2908387   DOI
5 Sesmero, M. Paz, Juan M. Alonso-Weber, and Araceli Sanchis. "CCE: An ensemble architecture based on coupled ANN for solving multiclass problems." Information Fusion Vol. 58, pp. 132-152. Jun. 2020. DOI: 10.1016/j.inffus.2019.12.015   DOI
6 Liu, J. Z., Lin, Z., Padhy, S., Tran, D., Bedrax-Weiss, T., & Lakshminarayanan, B. "Simple and principled uncertainty estimation with deterministic deep learning via distance awareness." Advances in Neural Information Processing Systems Dec. 2020.
7 Gabrie, M., Manoel, A., Luneau, C., Barbier, J., Macris, N., Krzakala, F., & Zdeborova, L. "Entropy and mutual information in models of deep neural networks." Journal of Statistical Mechanics: Theory and Experiment, Vol. 50, No. 12, pp. 5021-5034 Dec, 2020. DOI: 10.1109/TCYB.2019.2908387
8 Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., ... & Nahavandi, S. A. "Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges." arXiv preprint arXiv:2011.06225. 2020
9 Malinin, Andrey, Bruno Mlodozeniec, and Mark Gales. "Ensemble distribution distillation." arXiv preprint arXiv:1905.00076, 2019.
10 Ko, Hyungjin, et al. "Loss-Driven Adversarial Ensemble Deep Learning for On-Line Time Series Analysis." Sustainability Vol. 11, No. 12, pp. 1-24 Jun. 2019 DOI: DOI:10.3390/su11123489.   DOI
11 Park, S., J. Hah, and J. Lee. "Inductive ensemble clustering using kernel support matching." Electronics Letters Vol. 53, No. 25, pp. 1625-1626 Dec. 2017. DOI: 10.1049/el.2017.2159   DOI
12 Sun, Xu, et al. "Label embedding network: Learning label representation for soft training of deep networks." arXiv preprint arXiv:1710.10393 , 2017.
13 Sellah, Smail, and Vincent Hilaire. "Label Clustering for a Novel Problem Transformation in Multi-label Classification." J. UCS Vol.26 No.1 pp. 71-88. Jan. 2020,
14 Tao, Sean. "Deep neural network ensembles." International Conference on Machine Learning, Optimization, and Data Science. Springer, Cham, 2019.