Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

The Analysis of Semi-supervised Learning Technique of Deep Learning-based Classification Model

딥러닝 기반 분류 모델의 준 지도 학습 기법 분석

  • Park, Jae Hyeon (Department of Multimedia Engineering, Dongguk University) ;
  • Cho, Sung In (Department of Multimedia Engineering, Dongguk University)
  • 박재현 (동국대학교 멀티미디어공학과) ;
  • 조성인 (동국대학교 멀티미디어공학과)
  • Received : 2020.12.03
  • Accepted : 2021.01.07
  • Published : 2021.01.30
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we analysis the semi-supervised learning (SSL), which is adopted in order to train a deep learning-based classification model using the small number of labeled data. The conventional SSL techniques can be categorized into consistency regularization, entropy-based, and pseudo labeling. First, we describe the algorithm of each SSL technique. In the experimental results, we evaluate the classification accuracy of each SSL technique varying the number of labeled data. Finally, based on the experimental results, we describe the limitations of SSL technique, and suggest the research direction to improve the classification performance of SSL.

본 논문에서는 소량의 레이블 데이터로 딥러닝 기반 분류 모델을 훈련할 때 적용되는 준 지도 학습 기법 (semi-supervised learning: SSL)에 대해서 분석한다. 기존의 준 지도 학습 기법은 크게 일관성 정규화 (consistency regularization), 엔트로피 기반 (entropybased), 의사 레이블링 (pseudo labeling)으로 구분할 수 있다. 우선, 각 준 지도 학습 기법의 알고리즘에 대해서 서술한다. 실험에서는 준 지도학습 기법을 레이블 데이터의 수를 변화시키면서 훈련 후 분류 정확도를 평가한다. 최종적으로 실험 결과를 바탕으로 기존 준 지도 학습 기법의 한계에 대해서 서술하고, 분류 성능을 향상하기 위한 연구 방향을 제시한다.

Keywords

References

  1. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Proc. Adv. Neural Inf. Process. Syst., pp. 1097-1105, 2012.
  2. K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proc. Int. Conf. Learn. Represent., pp. 1-14, 2015.
  3. C. Szegedy et al., "Going deeper with convolutions," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1-9, 2015.
  4. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 770-778, 2016.
  5. P. Bachman, O. Alsharif, and D. Precup, "Learning with pseudo ensembles," in Proc. Advances Neural Inf. Process. Syst., pp. 3365-3373, 2014.
  6. M. Sajjadi, M. Javanmardi, and T. Tasdizen, "Regularization with stochastic transformations and perturbations for deep semi-supervised learning," in Proc. 30th Int. Conf. Neural Inf. Process. Syst., pp. 1171 -1179, 2016.
  7. S. Laine and T. Aila, "Temporal ensembling for semi-supervised learning," in Proc. Int. Conf. Learn. Represent., pp. 1-13, 2017.
  8. A. Tarvainen and H. Valpola, "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results," in Proc. Adv. Neural Inf. Process. Syst., pp. 1195-1204, 2017.
  9. T. Miyato, S.-I. Maeda, M. Koyama, and S. Ishii, ''Virtual adversarial training: A regularization method for supervised and semi-supervised learning,'' IEEE Trans. Pattern Anal. Mach. Intell., Vol. 41, No. 8, pp. 1979-1993, Aug 2019. https://doi.org/10.1109/TPAMI.2018.2858821
  10. Y. Grandvalet and Y. Bengio, "Semi-supervised learning by entropy minimization," in Proc. Adv. Neural Inf. Process. Syst., pp. 529-536, 2004.
  11. D.-H. Lee, "Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks," in Proc. Workshop Challenges Represent. Learn. (ICML), pp. 2-7, 2013.
  12. A. Oliver, A. Odena, C. Raffel, E. Cubuk, and I. Goodfellow, "Realistic Evaluation of Deep Semi-Supervised Learning Algorithms," in Adv. in Neural Inf. Process. Syst., pp. 3235-3246, 2018.
  13. A. Krizhevsky and G. Hinton, "Learning Multiple Layers of Features from Tiny Images," technical report, Univ. of Toronto, 2009.
  14. Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, "Reading digits in natural images with unsupervised feature learning," In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
  15. S. Zagoruyko and N. Komodakis, "Wide residual networks," in Proc. Brit. Mach. Vis. Conf., pp. 87.1-87.12, 2016.
  16. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." In Proc. IEEE Conf. Comput. Vis. Pattern Rcognit., pp. 770-778, 2016.
  17. S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep net- work training by reducing internal covariate shift," in Proc. Int. Conf. Mach. Learn., pp. 448-456, 2015.
  18. A. L. Maas, A. Y. Hannun, and A. Y. Ng, "Rectifier nonlinearities improve neural network acoustic models," in Proc. ICML, Vol. 30, No. 1, p. 3, Jun. 2013.