Browse > Article
http://dx.doi.org/10.33851/JMIS.2022.9.2.93

Proper Noun Embedding Model for the Korean Dependency Parsing  

Nam, Gyu-Hyeon (DeepBrain AI)
Lee, Hyun-Young (KT Corporation)
Kang, Seung-Shik (Department of Artificial Intelligence, Kookmin University)
Publication Information
Journal of Multimedia Information System / v.9, no.2, 2022 , pp. 93-102 More about this Journal
Abstract
Dependency parsing is a decision problem of the syntactic relation between words in a sentence. Recently, deep learning models are used for dependency parsing based on the word representations in a continuous vector space. However, it causes a mislabeled tagging problem for the proper nouns that rarely appear in the training corpus because it is difficult to express out-of-vocabulary (OOV) words in a continuous vector space. To solve the OOV problem in dependency parsing, we explored the proper noun embedding method according to the embedding unit. Before representing words in a continuous vector space, we replace the proper nouns with a special token and train them for the contextual features by using the multi-layer bidirectional LSTM. Two models of the syllable-based and morpheme-based unit are proposed for proper noun embedding and the performance of the dependency parsing is more improved in the ensemble model than each syllable and morpheme embedding model. The experimental results showed that our ensemble model improved 1.69%p in UAS and 2.17%p in LAS than the same arc-eager approach-based Malt parser.
Keywords
Dependency Parsing; LSTM; Proper Noun Embedding; Malt Parser; Transition-Based Model;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 D. Andor, C. Alberti, D. Weiss, A. Severyn, A. Presta, and K. Ganchev, et al., "Globally normalized transition-based neural networks," in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany, Aug. 2016. pp. 2442-2452.
2 J. Nivre, "Algorithms for deterministic incremental dependency parsing," Computational Linguistics, vol. 34, no. 4, pp. 513-553, Dec. 2007.   DOI
3 T. Kudo and J. Richardson, "Sentence piece: A simple and language independent subword tokenizer and detokenizer for neural text processing," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Brussels, Belgium, Nov. 2018, pp. 66-71.
4 T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient Estimation of Word Representations in Vector Space," https://arxiv.org/abs/1301.3781.
5 J. Devlin, M. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, Jun. 2019, pp. 4171-4186.
6 A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language models are unsupervised multitask learners," OpenAI Blog, vol. 1, no. 8, p. 9, 2019.
7 T. Brown, et al., "Language models are few-shot learners," in Advances in Neural Information Processing systems, Dec. 2020, pp. 1877-1901.
8 T. Lee and S. Kang, "Automatic text summarization based on selective OOV copy mechanism with BERT embedding," Journal of KIISE, vol. 47, no. 1, pp. 36- 44, Jan. 2020.   DOI
9 M. Ballesteros, C. Dyer, and N. A. Smith, "Improved transition-based parsing by modeling characters instead of words with LSTMs," in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, Sep. 2015, pp. 349-359.
10 D. Weiss, C. Alberti, M. Collins, and S. Petrov, "Structured training for neural network transition-based parsing," in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China, Jul. 2015, pp. 323-333.
11 C. Lee, J. Kim, and J. Kim, "Korean dependency parsing using deep learning," in Proceedings of the 26th Annual Conference on Human and Cognitive Language Technology, 2014, pp. 87-91.
12 S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. 8, pp. 1735-1780, Nov. 1997.   DOI
13 J. Li and J. H. Lee, "Korean transition-based dependency parsing with recurrent neural network," KIISE Transactions on Computing Practices, vol. 21, no. 8, pp. 567-571, 2015.   DOI
14 S. H. Na, K. Kim, and Y. K. Kim, "Stack LSTMs for transition-based Korean dependency parsing," in Proceedings of the Korea Computer Congress, 2016, pp. 732-734.
15 Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations," https://arxiv.org/abs/1909.11942.
16 J. W. Min,and S. H. Na, "SyntaxNet models using transition based recurrent unit for Korean dependency parsing," in Proceedings of the Korea Computer Congress'17, 2017, pp. 602-604.
17 C. Park and C. Lee, "Korean dependency parsing by using pointer networks," Journal of Korean Institute of Information Scientists and Engineers, vol. 44, no. 8, pp. 822-831, 2017.
18 G. H. Nam, H. Lee, and S. Kang, "Korean dependency parsing with proper noun encodin," in Proceedings of the 2019 4th International Conference on Intelligent Information Technology, Da Nang, Viet Nam, Feb. 2019, pp. 113-117.
19 N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," The Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929-1958, 2014.
20 J. Nivre, J. Hall, and J. Nilsson, "MaltParser: A data-driven parser-generator for dependency parsing," in Proceedings of the Fifth International Conference on Language Resources and Evaluation, Genoa, Italy, May 2016, pp. 2216-2219.
21 Z. Zhang, S. Liu, M. Li, M. Zhou, and E. Chen, "Stack-based multi-layer attention for transition-based de pendency parsing," in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, Sep. 2017, pp. 1677-1682.
22 D. Chen and C. D. Manning, "A fast and accurate dependency parser using neural networks," in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, Oct. 2014, pp. 740-750.
23 J. Nivre, "Non-projective dependency parsing in expected linear time," in Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, Suntec, Singapore, Aug. 2009, pp. 351-359, 2009.
24 T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, "Distributed representations of words and phrases and their compositionality," in Advances in Neural Information Processing Systems, USA, Dec. 2013, pp. 3111-3119.
25 A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, "Improving language understanding by generative pre-training," 2018.
26 Z. Hu, T. Chen, K. Chang, and Y. Sun, "Few-shot representation learning for out-of-vocabulary words," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, Jul. 2019, pp. 4102-4112.
27 C. Dyer, M. Ballesteros, W. Ling, A. Matthews, and N. A. Smith, "Transition-based dependency parsing with stack long short-term memory," in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China, Jul. 2015, pp. 334-343.
28 J. H. Lim, Y. C. Yoon, Y. J. Bae, S. J. Lim, H. K. Kim, and K. C. Lee, "Korean dependency parsing model based on transition system using head-final constraint," in Proceedings of the 26th Annual Conference on Human and Cognitive Language Technology, 2014, pp. 81-86.
29 H. Wang, H. Zhao, and Z. Zhang, "A transition-based system for universal dependency parsing," in Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, Vancouver, Canada, Aug. 2017, pp. 191-197.
30 Z. Zhang, H. Zhao, and L. Qin, "Probabilistic graph-based dependency parsing with convolutional neural network," in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany, Aug. 2016. pp. 1382-1392.
31 E. Kiperwasser and Y. Goldberg, "Simple and accurate dependency parsing using bidirectional LSTM feature representations," in Transactions of the Association for Computational Linguistics, vol. 4, pp. 313-327. 2016.   DOI
32 Z. Li, J. Cai, S. He, and H. Zhao, "Seq2seq dependency parsing," in Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA, Aug. 2018, pp. 3203-3214.
33 C. Lee, J Bae, C. Park, H. Hong, and S. Lee, "Korean information processing system competition: Korean dependency parsing," in Proceedings of the 30th Conference of Human and Language Technology, 2018, pp. 675-677.
34 G. H. Nam, "Transition-based deep learning approach to Korean dependency parsing," M.S. thesis, Kookmin University, Seoul, Korea, 2018.
35 W. Wang and B. Chang, "Graph-based dependency parsing with bidirectional LSTM," in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany, Aug. 2016, pp. 2306-2315.