The Korean Journal of Applied Statistics (응용통계연구)

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

Hybrid dropout

하이브리드 드롭아웃

  • Received : 2019.10.02
  • Accepted : 2019.11.05
  • Published : 2019.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Massive in-depth neural networks with numerous parameters are powerful machine learning methods, but they have overfitting problems due to the excessive flexibility of the models. Dropout is one methods to overcome the problem of oversized neural networks. It is also an effective method that randomly drops input and hidden nodes from the neural network during training. Every sample is fed to a thinned network from an exponential number of different networks. In this study, instead of feeding one sample for each thinned network, two or more samples are used in fitting for one thinned network known as a Hybrid Dropout. Simulation results using real data show that the new method improves the stability of estimates and reduces the minimum error for the verification data.

수 많은 모수들을 가지고 있는 방대한 심층신경망은 매우 강력한 기계학습 방법이지만 모형의 과도한 융통성으로 인하여 과적합문제를 내포하고 있다. 드롭아웃 방법은 크기가 큰 신경망의 과적합 문제를 해결하는 다양한 방법들 중 하나이며 매우 효과적인 방법으로 알려져 있다. 드롭아웃 방법은 훈련과정에서 각각의 표본에 다른 모형을 적용하는데 이들 모형은 입력과 은닉층의 노드들을 무작위로 제거한 모형들 중에 임의로 선택된다. 본 연구에서는 임의로 선택된 모형에 둘 이상의 표본을 적용하여 모형의 가중치들에 대한 추정치의 안정성을 높이는 하이브리드 드롭아웃 방법을 제시하였다. 실제 자료를 이용한 시뮬레이션 결과 노드의 선택확률과 모형의 적합에 사용되는 표본의 수를 적절하게 선택하여 기존의 방법에 비하여 추정치의 변동성이 감소시킬 수 있었으며 동시에 검증자료에 대한 최저오차도 줄일 수 있음을 보였다.

Keywords

References

  1. Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks, Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS) 2010, Chia Laguna Resort, Sardinia, Italy. Volume 9 of JMLR: W&CP 9.
  2. Helmbold, D. and Long, P. (2017). Surprising properties of dropout in deep networks, Proceedings of Machine Learning Research, 65, 1-24.
  3. Hinton, G., Srivastava, N., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. http://arxiv.org/abs/1207.0580.
  4. Ioffe, S. and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Proceeding ICML'15 Proceedings of the 32nd International Conference on International Conference on Machine Learning, 37, 448-456.
  5. Nowlan, S. and Hinton, G. (1992). Simplifying neural networks by soft weight-sharing, Neural Computation, 4, 473-493. https://doi.org/10.1162/neco.1992.4.4.473
  6. Salakhutdinov, R. and Mnih, A. (2008). Bayesian probabilistic matrix factorization using Markov chain Monte Carlo, In Proceedings of the 25th International Conference on Machine Learning, ACM.
  7. Srivastava, N. (2013). Improving Neural Networks with Dropout. Master's thesis, University of Toronto.
  8. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting, Journal of Machine Learning Research, 15, 1929-1958.
  9. Wan, L., Zeiler, M., Zhang, S., Cun, Y. and Fergus, R. (2013). Regularization of neural networks using DropConnect, Proceedings of the 30th International Conference on Machine Learning, 28, 1058-1066.
  10. Xiong, H., Barash, Y. and Frey, B. (2011). Bayesian prediction of tissue-regulated splicing using RNA sequence and cellular context, Bioinformatics, 27, 2554-2562. https://doi.org/10.1093/bioinformatics/btr444