Journal of the Korea Convergence Society (한국융합학회논문지)

Korea Convergence Society (한국융합학회)
권호 표지
DOI QR코드

DOI QR Code

Study on Zero-shot based Quality Estimation

Zero-Shot 기반 기계번역 품질 예측 연구

  • Eo, Sugyeong (Combined Student, Department of Computer Science and Engineering, Korea University) ;
  • Park, Chanjun (Combined Student, Department of Computer Science and Engineering, Korea University) ;
  • Seo, Jaehyung (Combined Student, Department of Computer Science and Engineering, Korea University) ;
  • Moon, Hyeonseok (Combined Student, Department of Computer Science and Engineering, Korea University) ;
  • Lim, Heuiseok (Combined Student, Department of Computer Science and Engineering, Korea University)
  • 어수경 (고려대학교 컴퓨터학과) ;
  • 박찬준 (고려대학교 컴퓨터학과) ;
  • 서재형 (고려대학교 컴퓨터학과) ;
  • 문현석 (고려대학교 컴퓨터학과) ;
  • 임희석 (고려대학교 컴퓨터학과)
  • Received : 2021.08.10
  • Accepted : 2021.11.20
  • Published : 2021.11.28
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, there has been a growing interest in zero-shot cross-lingual transfer, which leverages cross-lingual language models (CLLMs) to perform downstream tasks that are not trained in a specific language. In this paper, we point out the limitations of the data-centric aspect of quality estimation (QE), and perform zero-shot cross-lingual transfer even in environments where it is difficult to construct QE data. Few studies have dealt with zero-shots in QE, and after fine-tuning the English-German QE dataset, we perform zero-shot transfer leveraging CLLMs. We conduct comparative analysis between various CLLMs. We also perform zero-shot transfer on language pairs with different sized resources and analyze results based on the linguistic characteristics of each language. Experimental results showed the highest performance in multilingual BART and multillingual BERT, and we induced QE to be performed even when QE learning for a specific language pair was not performed at all.

최근 다언어모델(Cross-lingual language model)을 활용하여 한 번도 보지 못한 특정 언어의 하위 태스크를 수행하는 제로샷 교차언어 전이(Zero-shot cross-lingual transfer)에 대한 관심이 증가하고 있다. 본 논문은 기계번역 품질 예측(Quality Estimation, QE)을 학습하기 위한 데이터 구축적 측면에서의 한계점을 지적하고, 데이터를 구축하기 어려운 상황에서도 QE를 수행할 수 있도록 제로샷 교차언어 전이를 수행한다. QE에서 제로샷을 다룬 연구는 드물며, 본 논문에서는 교차언어모델을 활용하여 영어-독일어 QE 데이터에 대해 미세조정을 실시한 후 다른 언어쌍으로의 제로샷 전이를 진행했고 이 과정에서 다양한 다언어모델을 활용하여 비교 연구를 수행했다. 또한 다양한 자원 크기로 구성된 언어쌍에 대해 제로샷 실험을 진행하고 실험 결과에 대해 언어별 언어학적 특성 관점으로의 분석을 수행하였다. 실험결과 multilingual BART와 multillingual BERT에서 가장 높은 성능을 보였으며, 특정 언어쌍에 대해 QE 학습을 전혀 진행하지 않은 상황에서도 QE를 수행할 수 있도록 유도하였다.

Keywords

Acknowledgement

"This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program(IITP-2018-0-01405) supervised by the IITP(Institute for Information & Communications Technology Planning & Evaluation)" and this research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(NRF-2021R1A6A1A03045425).

References

  1. T. Ranasinghe, C. Orasan & R. Mitkov. (2020). TransQuest at WMT2020: Sentence-Level Direct Assessment. arXiv preprint arXiv:2010.05318.
  2. Z. Chi, L. Dong, S. Ma, S. H. X. L. Mao, H. Huang & F. Wei. (2021). mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs. arXiv preprint arXiv:2104.08692.
  3. T. Pires, E. Schlinger & D. Garrette. (2019). How multilingual is multilingual BERT?. arXiv preprint arXiv:1906.01502. DOI : 10.18653/v1/p19-1493
  4. G. Chen et al. (2021). Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders. arXiv preprint arXiv:2104.08757.
  5. L. Specia, K. Shah, J. G. De Souza & T. Cohn (2013). QuEst-A translation quality estimation framework. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 79-84.
  6. K. Papineni, S. Roukos, T. Ward & W. J. Zhu. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311-318. DOI : 10.3115/1073083.1073135
  7. S. Banerjee & A. Lavie. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 65-72. DOI : 10.3115/1626355.1626389
  8. S. Eo, C. Park, H. Moon, J. Seo & H. Lim. (2021). Dealing with the Paradox of Quality Estimation. In Proceedings of the 4rd Workshop on Technologies for MT of Low Resource Languages, 1-10.
  9. G. Lample & A. Conneau. (2019). Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291.
  10. A. Conneau et al. (2019). Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. DOI : 10.18653/v1/P19-4007
  11. J. Devlin, M. W. Chang, K. Lee & K. Toutanova. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. DOI : 10.18653/v1/N19-1423
  12. Y. Liu et al. (2020). Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8, 726-742. https://doi.org/10.1162/tacl_a_00343
  13. J. Hu, S. Ruder, A. Siddhant, G. Neubig, O. Firat & M. Johnson. (2020, November). Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, 4411-4421.
  14. G. Campagna, A. Foryciarz, M. Moradshahi & M. S. Lam. (2020). Zero-shot transfer learning with synthesized data for multi-domain dialogue state tracking. arXiv preprint arXiv:2005.00891. DOI : 10.18653/v1/2020.acl-main.12
  15. A. Lauscher, V. Ravishankar, I. Vulic & G. Glavas, (2020). From zero to hero: On the limitations of zero-shot cross-lingual transfer with multilingual transformers. arXiv preprint arXiv:2005.00633. DOI : 10.18653/v1/2020.emnlp-main.363
  16. L. Zhou, L. Ding & K. Takeda. (2020). Zero-shot translation quality estimation with explicit cross-lingual patterns. arXiv preprint arXiv:2010.04989.
  17. Z. Chi, L. Dong, F. Wei, W. Wang, X. L. Mao & H. Huang. (2020). Cross-lingual natural language generation via pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, 7570-7577. DOI : 10.1609/aaai.v34i05.6256
  18. L. Specia, D. Raj & M. Turchi (2010). Machine translation evaluation versus quality estimation. Machine translation, 24(1), 39-50. DOI : 10.1007/s10590-010-9077-2
  19. D. Lee. (2020). Cross-lingual transformers for neural automatic post-editing. In Proceedings of the Fifth Conference on Machine Translation, 772-776.
  20. K. Shah, T. Cohn & L. Specia. (2015). A bayesian non-linear method for feature selection in machine translation quality estimation. Machine Translation, 29(2), 101-125. DOI : 10.1007/s10590-014-9164-x
  21. R. Soricut, N. Bach & Z. Wang. (2012). The SDL language weaver systems in the WMT12 quality estimation shared task. In Proceedings of the Seventh Workshop on Statistical Machine Translation, 145-151.
  22. H. Kim, J. H. Lee & S. H. Na. (2017). Predictor-estimator using multilevel task learning with stack propagation for neural quality estimation. In Proceedings of the Second Conference on Machine Translation, 562-568. DOI : 10.18653/v1/W17-4763
  23. R. N. Patel. (2016). Translation quality estimation using recurrent neural network. arXiv preprint arXiv:1610.04841. DOI : 10.18653/v1/W16-2389
  24. H. Kim, J. H. Lim, H. K. Kim & S. H. Na. (2019). QE BERT: bilingual BERT using multi-task learning for neural quality estimation. In Proceedings of the Fourth Conference on Machine Translation (3), 85-89. DOI : 10.18653/v1/W19-5407
  25. S. Eo, C. Park, H. Moon, J. Seo & H. Lim. (2021). Research on Recent Quality Estimation. Journal of the Korea Convergence Society, 12(7), 37-44. https://doi.org/10.15207/JKCS.2021.12.7.037
  26. M. Wang et al. (2020). Hw-tsc's participation at wmt 2020 quality estimation shared task. In Proceedings of the Fifth Conference on Machine Translation, 1056-1061.
  27. S. Eo, C. Park, H. Moon, J. Seo & H. Lim. (2021). Comparative Analysis of Current Approaches to Quality Estimation for Neural Machine Translation. Applied Sciences, 11(14), 6584. DOI : 10.3390/app11146584
  28. T. Wolf et al. (2019). Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  29. C. Park, Y. Yang, K. Park & H. Lim. (2020). Decoding strategies for improving low-resource machine translation. Electronics, 9(10), 1562. DOI : 10.3390/electronics9101562
  30. C. Park, S. Eo, H. Moon & H. S. Lim. (2021). Should we find another model?: Improving Neural Machine Translation Performance with ONE-Piece Tokenization Method without Model Modification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, 97-104. DOI : 10.18653/v1/2021.naacl-industry.13
  31. C. Park, J. Seo, S. Lee, C. Lee, H. Moon, S. Eo & H. Lim. (2021). BTS: Back TranScription for Speech-to-Text Post-Processor using Text-to-Speech-to-Text. In Proceedings of the 8th Workshop on Asian Translation, 106-116. DOI : 10.18653/v1/2021.wat-1.10
  32. H. Moon, C. Park, S. Eo, J. Park & H. Lim. (2021). Filter-mBART Based Neural Machine Translation Using Parallel Corpus Filtering. Journal of the Korea Convergence Society, 12(5), 1-7. DOI : 10.15207/JKCS.2021.12.5.001
  33. C. Park, Y. Lee, C. Lee & H. Lim. (2020). Quality, not quantity?: Effect of parallel corpus quantity and quality on neural machine translation. In The 32st Annual Conference on Human Cognitive Language Technology, 363-368.