KIISE Transactions on Computing Practices (정보과학회 컴퓨팅의 실제 논문지)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지
DOI QR코드

DOI QR Code

Neural Predictive Coding for Text Compression Using GPGPU

GPGPU를 활용한 인공신경망 예측기반 텍스트 압축기법

  • 김재주 (성균관대학교 반도체디스플레이공학과) ;
  • 한환수 (성균관대학교 소프트웨어공학과)
  • Received : 2015.09.02
  • Accepted : 2015.12.24
  • Published : 2016.03.15
⟨ Previous Next ⟩

Abstract

Several methods have been proposed to apply artificial neural networks to text compression in the past. However, the networks and targets are both limited to the small size due to hardware capability in the past. Modern GPUs have much better calculation capability than CPUs in an order of magnitude now, even though CPUs have become faster. It becomes possible now to train greater and complex neural networks in a shorter time. This paper proposed a method to transform the distribution of original data with a probabilistic neural predictor. Experiments were performed on a feedforward neural network and a recurrent neural network with gated-recurrent units. The recurrent neural network model outperformed feedforward network in compression rate and prediction accuracy.

인공신경망을 압축에 적용하여 더 높은 압축 성능을 보이기 위한 알고리즘들이 몇 가지 연구되어 있다. 그러나 그동안 이러한 알고리즘들은 한정된 계산 능력의 하드웨어를 가지고 있기에 작은 크기의 신경망을 사용할 수밖에 없었으며 적용하는 대상 역시 실제로 사용하기에는 너무 작은 크기의 파일들이었다. 본 논문에서는 GPGPU의 계산능력을 신경망 학습에 이용하여 만든 텍스트 문맥 기반 문자 등장 확률 예측기와 함께 허프만 부호화의 성능을 높일 수 있는 변환 방법을 제시한다. 앞먹임 신경망과 GRU 회귀 신경망에 대해 실험을 수행하였으며, 회귀 신경망 모델은 앞먹임 신경망에 비해 뛰어난 예측 성공률과 압축률을 보였다.

Keywords

Acknowledgement

Grant : 스마트TV 2.0 소프트웨어 플랫폼

Supported by : 정보통신기술진흥센터

References

  1. NVIDIA's Next Generation CUDA Compute Architecture: Fermi, v1.1, [Online] Available: http://www.nvidia.com (downloaded 2015, August 21)
  2. Patel, R., Zhang, Y., Mak, J., Davidson, A., & Owens, J. D., "Parallel lossless data compression on the GPU," Proc. of Innovative Parallel Computing, 2012.
  3. Huffman, David A., "A method for the construction of minimum redundancy codes," Proc. of the IRE, Vol. 40, No. 9, pp. 1098-1101, 1952.
  4. Schmidhuber, Jurgen, et al., "Predictive Coding with Neural Nets: Application to Text Compression," Advances in neural information processing systems, pp. 1047-1054. 1995.
  5. J. Kim, H. Han, "GPGPU-Accelerated Neural Predictive Coding for Text Compression," Proc. of the KIISE Computer Congress, 2015. (in Korean)
  6. Srivastava, Nitish, et al., "Dropout: A simple way to prevent neural networks from overfitting," The Journal of Machine Learning Research," Vol. 15, No. 1, pp. 1929-1958, 2014.
  7. Ioffe, Sergey, and Christian Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," arXiv preprint arXiv:1502.03167, 2015.
  8. He, Kaiming, et al., "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification," Proc. of the IEEE Int'l Conf. on Computer Vision, 2015.
  9. V. Nair and G.E. Hinton, "Rectified linear units improve restricted boltzmann machines," Proc. of the Int'l Conf. on Machine Learning, 2010.
  10. Bridle, John S., "Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition," Neurocomputing: Algorithms, Architectures and Applications, Springer, pp. 227-236, 1990.
  11. Hinton, Geoffrey, Simon Osindero, and Yee-Whye Teh, "A fast learning algorithm for deep belief nets," Neural Computation, Vol. 18, No. 7, pp. 1527-1554, 2006. https://doi.org/10.1162/neco.2006.18.7.1527
  12. Hochreiter, Sepp and Schmidhuber, Jurgen, "Long Short-Term Memory," Neural Computation, Vol. 9, No. 8, pp. 1735-1780, 1997. https://doi.org/10.1162/neco.1997.9.8.1735
  13. Sundermeyer, Martin, Ralf Schlüter, and Hermann Ney, "LSTM Neural Networks for Language Modeling," Proc. of the INTERSPEECH, pp. 194-197, 2012.
  14. Chung, Junyoung, et al., "Empirical evaluation of gated recurrent neural networks on sequence modeling," Poster Presented at Deep Learning and Representation Learning Workshop, 2014.
  15. Project Gutenberg, Project Gutenberg. [Online]. Availabile: https://www.gutenberg.org/
  16. Deorowicz, S., "Silesia corpus," Silesian University of Technology, Poland. 2003. [Online]. Available: http://www.data-compression.info/Corpora/SilesiaCorpus/
  17. Bergstra, James, et al., "Theano: a CPU and GPU math expression compiler," Proc. of the Python for Scientific Computing Conference (SciPy), 2010.
  18. Francois Chollet. Keras Project [Online]. Available: GitHub Repository, https://github.com/fchollet/keras. 2015.