The Korean Journal of Applied Statistics (응용통계연구)

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

A study on the performance improvement of learning based on consistency regularization and unlabeled data augmentation

일치성규칙과 목표값이 없는 데이터 증대를 이용하는 학습의 성능 향상 방법에 관한 연구

  • Kim, Hyunwoong (Clinical Trial Center, Haeundae Paik Hospital Inje University) ;
  • Seok, Kyungha (Department of Statistics, Inje University)
  • 김현웅 (인제대학교 해운대백병원 임상시험센터) ;
  • 석경하 (인제대학교 통계학과)
  • Received : 2020.11.17
  • Accepted : 2021.01.04
  • Published : 2021.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

Semi-supervised learning uses both labeled data and unlabeled data. Recently consistency regularization is very popular in semi-supervised learning. Unsupervised data augmentation (UDA) that uses unlabeled data augmentation is also based on the consistency regularization. The Kullback-Leibler divergence is used for the loss of unlabeled data and cross-entropy for the loss of labeled data through UDA learning. UDA uses techniques such as training signal annealing (TSA) and confidence-based masking to promote performance. In this study, we propose to use Jensen-Shannon divergence instead of Kullback-Leibler divergence, reverse-TSA and not to use confidence-based masking for performance improvement. Through experiment, we show that the proposed technique yields better performance than those of UDA.

준지도학습(semi-supervised learning)은 목표값이 있는 데이터와 없는 데이터를 모두 이용하는 학습방법이다. 준지도학습에서 최근에 많은 관심을 받는 일치성규칙(consistency regularization)과 데이터 증대를 이용한 준지도학습(unsupervised data augmentation; UDA)은 목표값이 없는 데이터를 증대하여 학습에 이용한다. 그리고 성능 향상을 위해 훈련신호강화(training signal annealing; TSA)와 신뢰기반 마스킹(confidence based masking)을 이용한다. 본 연구에서는 UDA에서 사용하는 KL-정보량(Kullback-Leibler divergence)과 TSA 대신 JS-정보량(Jensen-Shanon divergene)과 역-TSA를 사용하고 신뢰기반 마스킹을 제거하는 방법을 제안한다. 실험을 통해 제안된 방법의 성능이 더 우수함을 보였다.

Keywords

Acknowledgement

이 논문은 본 논문은 2019학년도 인제대학교 학술연구조성비 보조에 의한 것임.

References

  1. Chapelle, O., Scholkopf, B., and Zien, A. (2006). Semi-supervised learning, Cambridge, Massachusetts: MIT Press.
  2. Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. (2018). AutoAugment: Learning augmentation policies from data, arXiv:1805.09501.
  3. Cubuk, E. D., Zoph, B., Shlens, J, and Quoc, V. L. (2019). RandAugment: Practical data augmentation with no separate search. arXiv:1909.13719.
  4. Grandvalet, Y. and Bengio, Y. (2004). Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems, 529-536.
  5. Freund, Y. and Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55, 119-139. https://doi.org/10.1006/jcss.1997.1504
  6. Laine, S. and Aila, T. (2016). Temporal ensembling for semi-supervised learning. arXiv:1610.02242.
  7. Miyato, T., Maeda, S., Ishii, S., and Koyama, M. (2018). Virtual adversarial training: a regularization method for supervised and semi-supervised learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1979-1993.
  8. Oliver, A., Odena, A., Raffel, C., Cubuk, E. D., and Goodfellow, I. (2018). Realistic evaluation of deep semisupervised learning algorithms, In Advances in Neural Information Processing Systems, 3235-3246.
  9. Tarvainen, A. and Valpola, H. (2017). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, 1195-1204.
  10. Xie, Q., Dai, Z., Hovy, E., Luong, M. T., and Le, Q. V. (2019). Unsupervised data augmentation for consistency training, arXiv:1904.12848.
  11. Zagoruyko, S. and Komodakis, N. (2016). Wide residual networks, arXiv:1605.07146.