KIPS Transactions on Software and Data Engineering (정보처리학회논문지:소프트웨어 및 데이터공학)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

A Study on Dataset Generation Method for Korean Language Information Extraction from Generative Large Language Model and Prompt Engineering

생성형 대규모 언어 모델과 프롬프트 엔지니어링을 통한 한국어 텍스트 기반 정보 추출 데이터셋 구축 방법

  • Received : 2023.10.04
  • Accepted : 2023.11.01
  • Published : 2023.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

This study explores how to build a Korean dataset to extract information from text using generative large language models. In modern society, mixed information circulates rapidly, and effectively categorizing and extracting it is crucial to the decision-making process. However, there is still a lack of Korean datasets for training. To overcome this, this study attempts to extract information using text-based zero-shot learning using a generative large language model to build a purposeful Korean dataset. In this study, the language model is instructed to output the desired result through prompt engineering in the form of "system"-"instruction"-"source input"-"output format", and the dataset is built by utilizing the in-context learning characteristics of the language model through input sentences. We validate our approach by comparing the generated dataset with the existing benchmark dataset, and achieve 25.47% higher performance compared to the KLUE-RoBERTa-large model for the relation information extraction task. The results of this study are expected to contribute to AI research by showing the feasibility of extracting knowledge elements from Korean text. Furthermore, this methodology can be utilized for various fields and purposes, and has potential for building various Korean datasets.

본 연구는 생성형 대규모 언어 모델을 활용하여 텍스트에서 정보를 추출하기 위한 한글 데이터셋 구축 방법을 탐구한다. 현대 사회에서는 혼합된 정보가 빠르게 유포되며, 이를 효과적으로 분류하고 추출하는 것은 의사결정 과정에 중요하다. 그러나 이에 대한 학습용 한국어 데이터셋은 아직 부족하다. 이를 극복하기 위해, 본 연구는 생성형 대규모 언어 모델을 사용하여 텍스트 기반 제로샷 학습(zero-shot learning)을 이용한 정보 추출을 시도하며, 이를 통해 목적에 맞는 한국어 데이터셋을 구축한다. 본 연구에서는 시스템-지침-소스입력-출력형식의 프롬프트 엔지니어링을 통해 언어 모델이 원하는 결과를 출력하도록 지시하며, 입력 문장을 통해 언어 모델의 In-Context Learning 특성을 활용하여 데이터셋을 구축한다. 생성된 데이터셋을 기존 데이터셋과 비교하여 본 연구 방법론을 검증하며, 관계 정보 추출 작업의 경우 KLUE-RoBERTa-large 모델 대비 25.47% 더 높은 성능을 달성했다. 이 연구 결과는 한국어 텍스트에서 지식 요소를 추출하는 가능성을 제시함으로써 인공지능 연구에 도움을 줄 것으로 기대된다. 더욱이, 이 방법론은 다양한 분야나 목적에 맞게 활용될 수 있어, 다양한 한국어 데이터셋 구축에 잠재력을 가진다고 볼 수 있다.

Keywords

Acknowledgement

이 연구는 2023년 정부(방위사업청)의 재원으로 국방과학연구소의 지원을 받아 수행된 미래도전국방기술 연구개발사업임(No.915026201)

References

  1. R. Grishman, "Information extraction," in IEEE Intelligent Systems, Vol.30, No.5, pp.8-15, 2015, doi: 10.1109/MIS. 2015.68.
  2. W. Xiang and B. Wang, "A survey of event extraction from text," in IEEE Access, Vol.7, pp.173111-173137, 2019, doi: 10.1109/ACCESS.2019.2956831.
  3. W. Liao and S. Veeramachaneni, "A simple semi-supervised algorithm for named entity recognition," In Proceedings of the NAACL HLT 2009 Workshop on Semi-Supervised Learning for Natural Language Processing, pp.58-65, 2009.
  4. D. Feng and H. Chen, "A small samples training framework for deep Learning-based automatic information extraction: Case study of construction accident news reports analysis," Advanced Engineering Informatics, Vol.47, pp.101256, 2021.
  5. Y. Chang et al., "A survey on evaluation of large language models," arXiv preprint arXiv:2307.03109, 2023.
  6. J. Wei et al., "Emergent abilities of large language models," arXiv preprint arXiv:2206.07682, 2022.
  7. M. T. R. Laskar, M. S. Bari, M. Rahman, M. A. H. Bhuiyan, S. Joty, and J. X. Huang, "A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets," arXiv preprint arXiv:2305.18486, 2023.
  8. J. Gou, B. Yu, S. J. Maybank, and D. Tao, "Knowledge distillation: A survey," International Journal of Computer Vision, Vol.129, No.6, pp.1789-1819, 2021.
  9. Y. Sari, M. F. Hassan, and N. Zamin, "Rule-based pattern extractor and named entity recognition: A hybrid approach," 2010 International Symposium on Information Technology, Kuala Lumpur, Malaysia, pp.563-568, 2010, doi: 10.1109/ITSIM.2010.5561392.
  10. C. Bizer et al., "DBpedia - A crystallization point for the web of data," Journal of Web Semantics: Science, Services and Agents on the WWW, Vol.7, No.3, pp.154-165, 2009. https://doi.org/10.1016/j.websem.2009.07.002
  11. Q. C. Bui, D. Campos, E. van Mulligen, and J. Kors, "A fast rule-based approach for biomedical event extraction," In Proceedings of the BioNLP Shared Task 2013 Workshop, pp.104-108, 2013.
  12. S. Rao, D. Marcu, K. Knight, and H. Daume III, "Biomedical event extraction using abstract meaning representation," In BioNLP 2017, pp.126-135, 2017.
  13. A. Vaswani et al., "Attention is all you need," Advances in Neural Information Processing Systems, Vol.30, 2017.
  14. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018.
  15. S. Liu, Y. Chen, K. Liu, and J. Zhao, "Exploiting argument information to improve event detection via supervised attention mechanisms," In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol.1: Long Papers), pp.1789-1798, 2017.
  16. L. Zhao, L. Li, X. Zheng, and J. Zhang, "A BERT based Sentiment Analysis and Key Entity Detection Approach for Online Financial Texts," 2021 IEEE 24th International Conference on Computer Supported Cooperative Work in Design (CSCWD), Dalian, China, pp.1233-1238, 2021, doi: 10.1109/CSCWD49262.2021.9437616.
  17. S. Park et al., "Klue: Korean language understanding evaluation," arXiv preprint arXiv:2105.09680, 2021.
  18. T. Brown et al., "Language models are few-shot learners," Advances in Neural Information Processing Systems, Vol.33, pp.1877-1901, 2020.
  19. J. Wei et al., "Chain-of-thought prompting elicits reasoning in large language models," Advances in Neural Information Processing Systems, Vol.35, pp.24824-24837, 2022.
  20. S. M. Xie, A. Raghunathan, P. Liang, and T. Ma, "An explanation of in-context learning as implicit bayesian inference," arXiv preprint arXiv:2111.02080, 2021.
  21. O. Sainz, H. Qiu, O. L. de Lacalle, E. Agirre, and B. Min, "ZS4IE: A toolkit for zero-shot information extraction with simple verbalizations," arXiv preprint arXiv:2203.13602, 2022.
  22. B. Sharma, Y. Gao, T. Miller, M. M. Churpek, M. Afshar, and D. Dligach, "Multi-Task Training with In-Domain Language Models for Diagnostic Reasoning," In Proceedings of the 5th Clinical Natural Language Processing Workshop, Toronto, Canada. Association for Computational Linguistics, pp.78-85, 2023.
  23. L. Ouyang et al., "Training language models to follow instructions with human feedback," Advances in Neural Information Processing Systems, Vol.35, pp.27730-27744, 2022.
  24. X. Wei et al., "Zero-shot information extraction via chatting with chatgpt," arXiv preprint arXiv:2302.10205, 2023.
  25. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. ACE 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57.
  26. M. Popovic, "chrF: character n-gram F-score for automatic MT evaluation," In Proceedings of the tenth workshop on statistical machine translation, pp.392-395, 2015.