Browse > Article
http://dx.doi.org/10.3745/KTSDE.2018.7.7.259

Document Classification using Recurrent Neural Network with Word Sense and Contexts  

Joo, Jong-Min (전남대학교 전자컴퓨터공학부)
Kim, Nam-Hun (전남대학교 전자컴퓨터공학부)
Yang, Hyung-Jeong (전남대학교 전자컴퓨터공학부)
Park, Hyuck-Ro (전남대학교 전자컴퓨터공학부)
Publication Information
KIPS Transactions on Software and Data Engineering / v.7, no.7, 2018 , pp. 259-266 More about this Journal
Abstract
In this paper, we propose a method to classify a document using a Recurrent Neural Network by extracting features considering word sense and contexts. Word2vec method is adopted to include the order and meaning of the words expressing the word in the document as a vector. Doc2vec is applied for considering the context to extract the feature of the document. RNN classifier, which includes the output of the previous node as the input of the next node, is used as the document classification method. RNN classifier presents good performance for document classification because it is suitable for sequence data among neural network classifiers. We applied GRU (Gated Recurrent Unit) model which solves the vanishing gradient problem of RNN. It also reduces computation speed. We used one Hangul document set and two English document sets for the experiments and GRU based document classifier improves performance by about 3.5% compared to CNN based document classifier.
Keywords
Document Classification; Word2vec; Doc2vec; Recurrent Neural Network; Gated Recurrent Unit(GRU);
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 J. H. Kim, J. H, Kim, K. M. Kim, and B. T. Zhang, "Large-Scale Text Classification with Convolution Neural Networks," Korean Information Science Society Conference Proceedings, pp.792-794, 2015.
2 C. H. Lee, Chang and S. C. Park, "BPNN Algorithm Using SVD for Korean Document Classification," Journal of the Korea Industrial Information System Society, Vol.15, No.2 pp.49-57, 2010.
3 G. Salton, A. Wong, and C. S. Yang, "A vector space model for automatic indexing," Communications of the ACM, Vol.18, No.11, pp.613-620, 1975.   DOI
4 J. W. Hwang and Y, J, Ko, "A Studyon Sentiment Features Extractionand Their Weight Boosting Methodfor Korean Document Sentiment Classification," Journal of KISS: Computing Practice and Letters, Vol.14, No.3, pp.336-340, 2008.
5 Y., Goldberg and O. Levy, "word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method," arXiv preprint arXiv:1402.3722, 2014.
6 A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," Advances in Neural Information Processing Systems, 2012.
7 J. M. Kim and J. H. Lee, "Text Document Classification Based on Recurrent Neural Network Using Word2vec," Journal of Korea Institute of Intelligent Systems, Vol.27, No.6, pp.560-565, 2017.   DOI
8 T. Mikolov, I. Sutskever, K. Chen, G. S., Corrado, and J. Dean, "Distributed representations of words and phrases and their compositionality," Advances in Neural Information Processing Systems, 2013.
9 P. Soucy and G. W. Mineau, "Beyond TFIDF weighting for text categorization in the vector space model," IJCAI, Vol. 5, 2005.
10 HANTEC Data Set [Internet], http://www.kristalinfo.com/TestCollections/#hkib
11 M. Cassel, and F. Lima, "Evaluating one-hot encoding finite state machines for SEU reliability in SRAM-based FPGAs," On-Line Testing Symposium, 2006. IOLTS 2006. 12th IEEE International, IEEE, 2006.
12 J. H. Lau, and T. Baldwin, "An empirical evaluation of doc2vec with practical insights into document embedding generation," arXiv preprint arXiv:1607.05368.
13 Q. Le and T. Mikolov, "Distributed representations of sentences and documents," International Conference on Machine Learning, 2014.
14 Text Classification Data Sets [Internet], http://goo.gl/JyCnZq
15 F. A. Gers, J. Schmidhuber, and F. Cummins, "Learning to forget: Continual prediction with LSTM," pp.850-855, 1999.
16 K. Cho, B. Van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase representations using RNN encoder-decoder for statistical machine translation," arXiv preprint arXiv:1406.1078, 2014.
17 20Newsgroups Data Set [Internet], http://qwone.com/-jason/20Newsgroups/
18 Python Package for Natural Language Processing [Internet], http://konlpy.org/en/v0.4.4/
19 J. Y. Lee, "A Study on the Improvement of Document Classification Performance of SVM Classifier Using Document Similarity," Journal of the Korean Society for Information Management, Vol.22 No.3, pp.261-287, 2005.   DOI
20 J. M. Kim and J. H. Lee, "A study on RNN based document classification using Word2vec," Journal of Korea Institute of Intelligent Systems, Vol.27, No.6, pp.560-565, 2017.   DOI
21 Jiang, Z., Zhang, S., & Zeng, J. "A hybrid generative/discriminative method for semi-supervised classification," Knowledge-Based Systems, Vol.37, pp.137-145, 2013.   DOI
22 N. H. Kim and H. J. Yang, "Classification of Hangul Documents Based on CNN Using Document Indexing Method Considering Meaning and Order of Words," Korean Computer Education Association Conference Paper, Vol.21, No.2, pp.41-45, 2017.