KIISE Transactions on Computing Practices (정보과학회 컴퓨팅의 실제 논문지)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지
DOI QR코드

DOI QR Code

Large-Scale Text Classification with Deep Neural Networks

깊은 신경망 기반 대용량 텍스트 데이터 분류 기술

  • 조휘열 (서울대학교 공과대학 컴퓨터공학부) ;
  • 김진화 (서울대학교 인문대학 인지과학전공) ;
  • 김경민 (서울대학교 공과대학 컴퓨터공학부) ;
  • 장정호 (SK텔레콤) ;
  • 엄재홍 (SK텔레콤) ;
  • 장병탁 (서울대학교 공과대학 컴퓨터공학부)
  • Received : 2016.10.24
  • Accepted : 2017.01.18
  • Published : 2017.05.15
⟨ Previous Next ⟩

Abstract

The classification problem in the field of Natural Language Processing has been studied for a long time. Continuing forward with our previous research, which classifies large-scale text using Convolutional Neural Networks (CNN), we implemented Recurrent Neural Networks (RNN), Long-Short Term Memory (LSTM) and Gated Recurrent Units (GRU). The experiment's result revealed that the performance of classification algorithms was Multinomial Naïve Bayesian Classifier < Support Vector Machine (SVM) < LSTM < CNN < GRU, in order. The result can be interpreted as follows: First, the result of CNN was better than LSTM. Therefore, the text classification problem might be related more to feature extraction problem than to natural language understanding problems. Second, judging from the results the GRU showed better performance in feature extraction than LSTM. Finally, the result that the GRU was better than CNN implies that text classification algorithms should consider feature extraction and sequential information. We presented the results of fine-tuning in deep neural networks to provide some intuition regard natural language processing to future researchers.

문서 분류 문제는 오랜 기간 동안 자연어 처리 분야에서 연구되어 왔다. 우리는 기존 컨볼루션 신경망을 이용했던 연구에서 나아가, 순환 신경망에 기반을 둔 문서 분류를 수행하였고 그 결과를 종합하여 제시하려 한다. 컨볼루션 신경망은 단층 컨볼루션 신경망을 사용했으며, 순환 신경망은 가장 성능이 좋다고 알려져 있는 장기-단기 기억 신경망과 회로형 순환 유닛을 활용하였다. 실험 결과, 분류 정확도는 Multinomial Naïve Bayesian Classifier < SVM < LSTM < CNN < GRU의 순서로 나타났다. 따라서 텍스트 문서 분류 문제는 시퀀스를 고려하는 것 보다는 문서의 feature를 추출하여 분류하는 문제에 가깝다는 것을 확인할 수 있었다. 그리고 GRU가 LSTM보다 문서의 feature 추출에 더 적합하다는 것을 알 수 있었으며 적절한 feature와 시퀀스 정보를 함께 활용할 때 가장 성능이 잘 나온다는 것을 확인할 수 있었다.

Keywords

Acknowledgement

Supported by : SK텔레콤

References

  1. Leskovec, Jure, Anand Rajaraman, and Jeffrey David Ullman, "Mining of massive datasets, Cambridge University Press, 2014.
  2. Chai, Kian Ming Adam, Hai Leong Chieu, and Hwee Tou Ng, "Bayesian online classifiers for text classification and filtering," Proc. of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, ACM, 2002.
  3. Friedman, Nir, Dan Geiger, and Moises Goldszmidt, "Bayesian network classifiers," Machine learning 29(2-3), pp. 131-163, 1997. https://doi.org/10.1023/A:1007465528199
  4. Boser, Bernhard E., Isabelle M. Guyon, and Vladimir N. Vapnik, "A training algorithm for optimal margin classifiers," Proc. of the fifth annual workshop on Computational learning theory, ACM, 1992.
  5. LeCun, Yann, and Yoshua Bengio, "Convolutional networks for images, speech, and time series," M. A. Arbib (Ed.), The handbook of brain theory and neural networks, Cambridge, MA: MIT Press, pp. 255-258, 1995.
  6. Goller, Christoph, and Andreas Kuchler, "Learning task-dependent distributed representations by backpropagation through structure," Neural Networks, IEEE International Conference on, Vol. 1, IEEE, 1996.
  7. Hochreiter, Sepp, and Jürgen Schmidhuber, "Long short-term memory," Neural computation, Vol. 9, No. 8, pp. 1735-1780, 1997. https://doi.org/10.1162/neco.1997.9.8.1735
  8. Cho, Kyunghyun et al., "Learning phrase representations using RNN encoder-decoder for statistical machine translation," arXiv preprint, arXiv:1406.1078, 2014.
  9. Leonard, Nicholas, Sagar Waghmare, Yang Wang, and Jin-Hwa Kim, "rnn: Recurrent Library for Torch," arXiv preprint arXiv:1511.07889, 2015.
  10. Jo, Hwiyeol, Kim, Jin-Hwa, Yoon, Sangwoong, Kim, Kyung-Min and Zhang, Byoung-Tak, "Large-Scale Text Classification with a Convolutional Neural Network," 42th The Korean Institute of Information Scientists and Engineers Annual Meeting, 2015.
  11. Jo, Hwiyeol, Kim, Jin-Hwa, Kim, Kyung-Min, Chang, Jeong-Ho, Eom, Jae-Hong, and Zhang, Byoung-Tak, "Large-Scale Text Classification with Recurrent Neural Networks," Korea Computer Congress 2016, 2016.