스마트미디어저널 (Smart Media Journal)

한국스마트미디어학회 (THE KOREAN INSTITUTE OF SMART MEDIA)
권호 표지
DOI QR코드

DOI QR Code

InferSent를 활용한 오픈 도메인 기계독해

Open Domain Machine Reading Comprehension using InferSent

  • 김정훈 (순천대학교 IT-Bio융합시스템전공) ;
  • 김준영 (순천대학교 IT-Bio융합시스템전공) ;
  • 박준 (순천대학교 IT-Bio융합시스템전공) ;
  • 박성욱 (순천대학교 IT-Bio융합시스템전공) ;
  • 정세훈 (순천대학교 컴퓨터공학과) ;
  • 심춘보 (순천대학교 인공지능공학부)
  • 투고 : 2022.10.28
  • 심사 : 2022.12.14
  • 발행 : 2022.11.30
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

오픈 도메인 기계독해는 질문과 연관된 단락이 존재하지 않아 단락을 검색하는 검색 기능을 추가한 모델이다. 문서 검색은 단어 빈도 기반인 TF-IDF로 많은 연구가 진행됐으나 문서의 양이 많아지면 낮은 성능을 보이는 문제가 있다. 아울러 단락 선별은 단어 기반 임베딩으로 많은 연구가 진행됐으나 문장의 특징을 가지는 단락의 문맥을 정확히 추출하지 못하는 문제가 있다. 그리고 문서 독해는 BERT로 많은 연구가 진행됐으나 방대한 파라미터로 느린 학습 문제를 보였다. 본 논문에서는 언급한 3가지 문제를 해결하기 위해 문서의 길이까지 고려한 BM25를 이용하며 문장 문맥을 얻기 위해 InferSent를 사용하고, 파라미터 수를 줄이기 위해 ALBERT를 이용한 오픈 도메인 기계독해를 제안한다. SQuAD1.1 데이터셋으로 실험을 진행했다. 문서 검색은 BM25의 성능이 TF-IDF보다 3.2% 높았다. 단락 선별은 InferSent가 Transformer보다 0.9% 높았다. 마지막으로 문서 독해에서 단락의 수가 증가하면 ALBERT가 EM에서 0.4%, F1에서 0.2% 더 높았다.

An open domain machine reading comprehension is a model that adds a function to search paragraphs as there are no paragraphs related to a given question. Document searches have an issue of lower performance with a lot of documents despite abundant research with word frequency based TF-IDF. Paragraph selections also have an issue of not extracting paragraph contexts, including sentence characteristics accurately despite a lot of research with word-based embedding. Document reading comprehension has an issue of slow learning due to the growing number of parameters despite a lot of research on BERT. Trying to solve these three issues, this study used BM25 which considered even sentence length and InferSent to get sentence contexts, and proposed an open domain machine reading comprehension with ALBERT to reduce the number of parameters. An experiment was conducted with SQuAD1.1 datasets. BM25 recorded a higher performance of document research than TF-IDF by 3.2%. InferSent showed a higher performance in paragraph selection than Transformer by 0.9%. Finally, as the number of paragraphs increased in document comprehension, ALBERT was 0.4% higher in EM and 0.2% higher in F1.

키워드

과제정보

This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the Grand Information Technology Research Center support program (IITP-2022-2020-0-01489) supervised by the IITP(Institute for Information & communications Technology Planning & Evaluation) This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(2020R1I1A3054843) This work was supported by the BK21 plus program through the National Research Foundation (NRF) funded by the Ministry of Education of Korea(No. 5199990214660)

참고문헌

  1. D. Chen, A. Fisch, J. Weston and A. Bordes, "Reading wikipedia to answer open-domain questions," arXiv preprint, arXiv:1704.00051, Mar. 2017.
  2. M. Kim, S. Cho, D. Park, and H.C. Kwon, "Machine Reading Comprehension-based Question and Answering System for Search and Analysis of Safety Standards," Journal of Korea Multimedia Society, Vol. 23, No. 2, pp. 351-360, Feb. 2020.
  3. S.E. Robertson, S. Walker, S. Jones, M.H. Beaulieu and M. Gatford, "Okapi at TREC-3," Third Text REtrieval Conference, pp. 109-126, London, UK, 1995.
  4. M.J. Lim, W.H. Park and J.H. Shin, "Parting Lyrics Emotion Classification using Word2Vec and LSTM," Smart Media Journal, Vol. 9, No. 3, pp. 90-97, Sep. 2020. https://doi.org/10.30693/SMJ.2020.9.3.90
  5. J. Park, J.Y. Kim, S.W. Park, S.H. Jung and C.B. Sim, "Development of ResNet based Crop Growth Stage Estimation Model," Smart Media Journal, Vol. 11, No. 2, pp. 53-62, Mar. 2022.
  6. J.H. Bang, J. Park, S.W. Park J.Y. Kim, S.H. Jung and C.B. Sim, "A System for Determining the Growth Stage of Fruit Tree Using a Deep Learning-Based Object Detection Model," Smart Media Journal, Vol. 11, No. 4, pp. 9-18, May, 2022.
  7. S. Liu, X. Zhang, S. Zhang, H. Wang, and W. Zhang, "Neural Machine Reading Comprehension: Methods and Trends," Applied Sciences, Vol. 9, No. 18, pp. 3698, Sep. 2019.
  8. C. Zeng, S. Li, Q. Li, J. Hu, and J. Hu, "A Survey on Machine Reading Comprehension-Tasks, Evaluation Metrics and Benchmark Datasets," Applied Sciences, Vol. 10, No. 21, pp. 7640, Oct. 2020.
  9. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser and I. Polosukhin, "Attention is All you Need," Part of Advances in Neural Information Processing Systems 30 (NIPS 2017), Vol. 30, pp. 5998-6008, 2017.
  10. J. Devlin, M.W. Chang, K. Lee and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint, arXiv:1810.04805, Oct. 2018.
  11. A. Conneau, D. Kiela, H. Schwenk, L. Barrault and A. Bordes, "Supervised learning of universal sentence representations from natural language inference data," arXiv preprin, arXiv:1705.02364, May, 2017.
  12. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma and R. Soricut, "ALBERT: A Lite BERT for Self-supervised Learning of Language Representations," arXiv preprint, arXiv:1909.11942, Sep. 2019.
  13. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, "SQuAD: 100,000+ Questions for Machine Comprehension of Text," arXiv:1606.05250, pp. 2383-2392, Jun. 2016.
  14. J. Chung, C. Gulcehre, K.H. Cho, and Y. Bengio, "Empirical evaluation of gated recurrent neural networks on sequence modeling," arXiv preprint, arXiv:1412.3555, Dec. 2014.
  15. Y.H. Lee and S.B. Lee, "A Research on Enhancement of Text Categorization Performance by using Okapi BM25 Word Weight Method," Journal of the Korea Academia-Industrial cooperation Society, Vol. 11, No. 12, pp. 5089-5096, Dec. 2010. https://doi.org/10.5762/KAIS.2010.11.12.5089
  16. C. Xiong, V. Zhong, and R. Socher, "Dynamic coattention networks for question answering," arXiv preprint, arXiv:1611.01604, 2016.
  17. Z. Huang, W. Xu, and K. Yu, "Bidirectional LSTM-CRF models for sequence tagging," arXiv preprint, arXiv:1508.01991, 2015.
  18. M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi, "Bidirectional attention flow for machine comprehension," In International Conference on Learning Representations, Toulon, France, Apr. 2017.
  19. J. Pennington, R. Socher, and C.D. Manning, " Glove: Global vectors for word representation," In Proceedings of the 2014 conference on empirical methods in natural language processing, pp. 1532-1543. Doha,Qatar, Oct. 2014.
  20. W. Wang, N. Yang, F. Wei, B. Chang, and M. Zhou, "Gated self-matching networks for reading comprehension and question answering," In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vol. 1, pp. 189-198, Vancouver, Canada, Jul. 2017.
  21. Wikimedia Articles Download, https://dumps.wikimedia.org/enwikisource (accessed Aug., Jun. 30, 2022).