Journal of Information Technology Services (한국IT서비스학회지)

Korea Society of IT Services (한국IT서비스학회)
권호 표지
DOI QR코드

DOI QR Code

Explaining the Translation Error Factors of Machine Translation Services Using Self-Attention Visualization

Self-Attention 시각화를 사용한 기계번역 서비스의 번역 오류 요인 설명

  • 장청롱 (국민대학교 비즈니스IT전문대학원) ;
  • 안현철 (국민대학교 비즈니스IT전문대학원)
  • Received : 2021.12.06
  • Accepted : 2022.03.16
  • Published : 2022.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

This study analyzed the translation error factors of machine translation services such as Naver Papago and Google Translate through Self-Attention path visualization. Self-Attention is a key method of the Transformer and BERT NLP models and recently widely used in machine translation. We propose a method to explain translation error factors of machine translation algorithms by comparison the Self-Attention paths between ST(source text) and ST'(transformed ST) of which meaning is not changed, but the translation output is more accurate. Through this method, it is possible to gain explainability to analyze a machine translation algorithm's inside process, which is invisible like a black box. In our experiment, it was possible to explore the factors that caused translation errors by analyzing the difference in key word's attention path. The study used the XLM-RoBERTa multilingual NLP model provided by exBERT for Self-Attention visualization, and it was applied to two examples of Korean-Chinese and Korean-English translations.

Keywords

Acknowledgement

본 논문은 교육부 및 한국연구재단의 4단계 두뇌한국21 사업(4단계 BK21 사업)으로 지원된 연구입니다.

References

  1. 권도경, 문수정, 김정수, 허윤정, "2018년 인공신경망 기계번역의 중한 번역 현황: 구글 번역과 네이버 파파고를 중심으로", 중국문학, 제98권, 2019, 233-266. https://doi.org/10.21192/SCLL.98..201902.012
  2. 기유미, "한중 기계번역 오류의 문형별 비교분석: 네이버 파파고 번역기와 구글 번역기를 중심으로", 중국연구, 제74권, 2018, 3-32. https://doi.org/10.18077/CHSS.2018.74..001
  3. 남철진, "中韓(韓中) 번역을 통해 본 구글번역(GNMT)의 문제", 중국과 중국학, 제34권, 2018, 1-27.
  4. 손경호, "싱겁게 끝난 번역대결…인간, AI에 완승", ZDNet Korea, 2017. URL: https://zdnet.co.kr/view/?no=20170221160244.
  5. 시정곤, 고창수, 김원경, "영-한 기계번역 성능 평가방안 연구", 언어와 정보, 제4권, 제2호, 2000, 1-26.
  6. 악일비, 이길연, "한중, 한영 기계번역에서 동사 시제표기 활용의 번역 양상", 한국어교육연구, 제14권, 제2호, 2019, 36-61.
  7. 이준호, "신경망기계번역의 객관적 평가를 위한 예비연구: 자동평가와 수동평가의 균형점", 통번역학연구, 제23권, 제3호, 2019, 171-202. https://doi.org/10.22844/ITS.2019.23.3.171
  8. 정혜연, "번역의 자동평가: 기계번역 평가를 인간번역 평가에 적용해보기", 통번역학연구, 제22권, 제4호, 2018, 265-287. https://doi.org/10.22844/ITS.2018.22.4.265
  9. 황은하, "언어학적 지식에 기반한 한중 뉴스 표제의 기계번역", 번역학연구, 제15권, 제5호, 2014, 333-362. https://doi.org/10.15749/JTS.2014.15.5.012
  10. 황지연, 양슬아, "한국어-중국어 신경망 기계 번역에 대한 수동평가: 사회과학과 기술과학 카테고리에서 나타나는 구글 번역(GNMT)과 파파고(N2MT)의 정확도", 통번역학연구, 제24권, 제4호, 2020, 175-206. https://doi.org/10.22844/ITS.2020.24.4.175
  11. AIHub, 한국어-중국어 번역 말뭉치(기술과학), https://aihub.or.kr/aidata/30722.
  12. AIHub, 한국어-영어 번역 말뭉치(기술과학), https://aihub.or.kr/aidata/30719.
  13. Brunner, G., Y. Liu, D. Pascual, O. Richter, M. Ciaramita, and R. Wattenhofer, "On identifiability in transformers", arXiv preprint arXiv:1908.04211, 2019.
  14. Bach, S., A. Binder, G. Montavon, F. Klauschen, K.R. Muller, and W. Samek, "On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation", PloS one, Vol.10, No.7, 2015, e0130140. https://doi.org/10.1371/journal.pone.0130140
  15. Clinchant, S., K.W. Jung, and V. Nikoulina, "On the use of BERT for neural machine translation", arXiv preprint arXiv:1909.12744, 2019.
  16. Cho, W.I., S.M. Kim, H. Cho, and N.S. Kim, "Kosp2e: Korean Speech to English Translation Corpus", arXiv preprint arXiv:2107.02875, 2021.
  17. Choi, Y., Y. Lee, J. Cho, J. Baek, B. Kim, Y. Cha, … and J. Heo, "Towards an appropriate query, key, and value computation for knowledge tracing", In Proceedings of the Seventh ACM Conference on Learning @Scale, 2020, 341-344.
  18. Hoover, B., H. Strobelt, and S. Gehrmann, "exBERT: A visual analysis tool to explore learned representations in transformers models", arXiv preprint arXiv:1910.05276, 2019.
  19. Khandelwal, U., O. Levy, and C.D. Manning, "What does bert look at? An analysis of bert's attention", arXiv preprint arXiv:1906.04341, 2019.
  20. Kobayashi, G., T. Kuribayashi, S. Yokoi, and K. Inui, "Attention is not only a weight: Analyzing transformers with vector norms", arXiv preprint arXiv:2004.10102, 2020.
  21. Lundberg, S.M. and S.I. Lee, "Consistent feature attribution for tree ensembles", arXiv preprint arXiv:1706.06060, 2017.
  22. Park, J. and H. Zhao, "Korean-to-Chinese Machine Translation using Chinese Character as Pivot Clue", arXiv preprint arXiv:1911.11008, 2019.
  23. Park, J. and H. Zhao, "Korean Neural Machine Translation Using Hierarchical Word Structure", In 2020 International Conference on Asian Language Processing (IALP), IEEE. 2020, 294-298.
  24. Park, J. and H. Zhao, "Enhancing Language Generation with Effective Checkpoints of Pre-trained Language Model", In Findings of the Association for Computational Linguistics: ACL-IJCNLP, 2021, 2686-2694.
  25. Ribeiro, M.T., S. Singh, and C. Guestrin, "Why should i trust you? Explaining the predictions of any classifier", In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, 1135-1144.
  26. Selvaraju, R.R., M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, "Grad-cam: Visual explanations from deep networks via gradient-based localization", In Proceedings of the IEEE International Conference on Computer Vision, 2017, 618-626.
  27. Shin, D., Y. Shim, H. Yu, S. Lee, B. Kim, and Y. Choi, "Saint+: Integrating temporal features for ednet correctness prediction". In LAK21: 11th International Learning Analytics and Knowledge Conference, 2021, 490-496.
  28. Strobelt, H., S. Gehrmann, H. Pfister, and A.M. Rush, "Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks", IEEE Transactions on Visualization and Computer Graphics, Vol.24, No.1, 2017, 667-676. https://doi.org/10.1109/TVCG.2017.2744158
  29. Strobelt, H., S. Gehrmann, M. Behrisch, A. Perer, H. Pfister, and A.M. Rush, "Seq2seq-vis: A visual debugging tool for sequence-to-sequence models", IEEE Transactions on Visualization and Computer Graphics, Vol.25, No.1, 2018, 353-363. https://doi.org/10.1109/TVCG.2018.2865044
  30. Vig, J., "BertViz: A tool for visualizing multihead self-attention in the BERT model", In ICLR Workshop: Debugging Machine Learning Models, 2019.
  31. Wang, Z.J., R. Turko, and D.H. Chau, "Dodrio: Exploring Transformer Models with Interactive Visualization", arXiv preprint arXiv:2103.14 625, 2021.
  32. Yue, Y. and J. Li, "Translation of verb Tense Marks in Korean Chinese and Korean English Machine Translation", Journal of Korean Language Education, Vol.14, No.2, 2019, 36-61.