Browse > Article
http://dx.doi.org/10.3745/KTSDE.2022.11.4.169

Korean Morphological Analysis Method Based on BERT-Fused Transformer Model  

Lee, Changjae (연세대학교 소프트웨어학부)
Ra, Dongyul (연세대학교 소프트웨어학부)
Publication Information
KIPS Transactions on Software and Data Engineering / v.11, no.4, 2022 , pp. 169-178 More about this Journal
Abstract
Morphemes are most primitive units in a language that lose their original meaning when segmented into smaller parts. In Korean, a sentence is a sequence of eojeols (words) separated by spaces. Each eojeol comprises one or more morphemes. Korean morphological analysis (KMA) is to divide eojeols in a given Korean sentence into morpheme units. It also includes assigning appropriate part-of-speech(POS) tags to the resulting morphemes. KMA is one of the most important tasks in Korean natural language processing (NLP). Improving the performance of KMA is closely related to increasing performance of Korean NLP tasks. Recent research on KMA has begun to adopt the approach of machine translation (MT) models. MT is to convert a sequence (sentence) of units of one domain into a sequence (sentence) of units of another domain. Neural machine translation (NMT) stands for the approaches of MT that exploit neural network models. From a perspective of MT, KMA is to transform an input sequence of units belonging to the eojeol domain into a sequence of units in the morpheme domain. In this paper, we propose a deep learning model for KMA. The backbone of our model is based on the BERT-fused model which was shown to achieve high performance on NMT. The BERT-fused model utilizes Transformer, a representative model employed by NMT, and BERT which is a language representation model that has enabled a significant advance in NLP. The experimental results show that our model achieves 98.24 F1-Score.
Keywords
Natural Language Processing; Morphological Analysis; Transfer Learning; Transformer; BERT-fused Model;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 T. Nguyen and J. Salazar, "Transformers without tears: Improving the normalization of self-attention," in Proceedings of the 16th International Workshop on Spoken Language Translation, 2019.
2 A. Graves, "Sequence transduction with recurrent neural networks," in Proceedings of the 29th International Conference on Machine Learning Workshop on Representation Learning, Edinburgh, Scotland, 2012.
3 D. Ra, M. Cho, and Y. Kim, "Enhancing a Korean part-of-speech tagger based on a maximum entropy model," Journal of the Korean Data Analysis Society, Vol.9, No.4, pp.1623-1638, 2007.
4 K. Cho, et al., "Learning phrase representations using RNN Encoder-decoder for statistical machine translation," in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.1724-1734, 2014.
5 A. Vaswani, et al., "Attention is all you need," in Advances in Neural Information Processing Systems, pp.6000-6010, 2017.
6 I. Sutskever, O. Vinyals, and Q. V. Le, "Sequence to sequence learning with neural networks," in Advances in Neural Information Processing Systems, pp.3104-3112, 2014.
7 D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," in Proceedings of the International Conference on Learning Representations, San Diego, California, 2015.
8 T. Luong, H. Pham, and C. D. Manning, "Effective approaches to attention-based neural machine translation," in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.1412-1421, 2015.
9 J. Zhu, et al., "Incorporating BERT into neural machine translation," in Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
10 Q. Wang, et al., "Learning Deep Transformer Models for Machine Translation," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp.1810-1822, 2019.
11 J. Devlin, M. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proceedings of NAACL-HLT, Minneapolis, Minnesota, pp.4171-4186, 2019.
12 S.-W. Kim and S.-P. Choi, "Research on joint models for Korean word spacing and POS (Part-Of-Speech) tagging based on bidirectional LSTM-CRF," Journal of Korean Institute of Information Scientists and Engineers, Vol.45, No.8, pp.792-800, 2018.
13 B. Choe, I.-h. Lee, and S.-g. Lee, "Korean morphological analyzer for neologism and spacing error based on sequence-to-sequence," Journal of Korean Institute of Information Scientists and Engineers, Vol.47, No.1, pp.70-77, 2020.
14 M. Freitag and Y. Al-Onaizan, "Beam search strategies for neural machine translation," in Proceedings of the First Workshop on Neural Machine Translation, Vancouver, Canada, pp.56-60, 2017.
15 E. Battenberg, et al., "Exploring neural transducers for end-to-end speech recognition," in Proceedings of 2017 IEEE Automatic Speech Recognition and Understanding Workshop, Okinawa, Japan, pp.206-213, 2017.
16 H. S. Hwang and C. K. Lee, "Korean morphological analysis using sequence-to-sequence learning with copying mechanism," in Proceedings of the Korea Computer Congress 2016, pp.443-445, 2016.
17 J. Li, E. H. Lee, and J.-H. Lee, "Sequence-to-sequence based morphological analysis and part-of-speech tagging for Korean language with convolutional features," Journal of Korean Institute of Information Scientists and Engineers, Vol.44, No.1, pp.57-62, 2017.
18 J. Min, S.-H. Na, J.-H. Shin, and Y.-K. Kim, "Stack pointer network for Korean morphological analysis," in Proceedings of the Korea Computer Congress 2020, pp.371-373, 2020.
19 J. Y. Youn and J. S. Lee, "A pipeline model for Korean morphological analysis and part-of-speech tagging using sequence-to-sequence and BERT-LSTM," in Proceedings of the 32nd Annual Conference on Human & Cognitive Language Technology, pp.414-417, 2020.
20 Y. Choi and K. J. Lee, "Performance analysis of Korean morphological analyzer based on transformer and BERT," Journal of Korean Institute of Information Scientists and Engineers, Vol.47, No.8, pp.730-741, 2020.