KIPS Transactions on Software and Data Engineering (정보처리학회논문지:소프트웨어 및 데이터공학)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Analyzing Korean Math Word Problem Data Classification Difficulty Level Using the KoEPT Model

KoEPT 기반 한국어 수학 문장제 문제 데이터 분류 난도 분석

  • 임상규 (서울대학교 지능정보융합학과) ;
  • 기경서 (서울대학교 지능정보융합학과) ;
  • 김부근 (서울대학교 인공지능혁신인재양성교육연구단) ;
  • 권가진 (서울대학교 지능정보융합학과)
  • Received : 2021.06.29
  • Accepted : 2022.04.26
  • Published : 2022.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose KoEPT, a Transformer-based generative model for automatic math word problems solving. A math word problem written in human language which describes everyday situations in a mathematical form. Math word problem solving requires an artificial intelligence model to understand the implied logic within the problem. Therefore, it is being studied variously across the world to improve the language understanding ability of artificial intelligence. In the case of the Korean language, studies so far have mainly attempted to solve problems by classifying them into templates, but there is a limitation in that these techniques are difficult to apply to datasets with high classification difficulty. To solve this problem, this paper used the KoEPT model which uses 'expression' tokens and pointer networks. To measure the performance of this model, the classification difficulty scores of IL, CC, and ALG514, which are existing Korean mathematical sentence problem datasets, were measured, and then the performance of KoEPT was evaluated using 5-fold cross-validation. For the Korean datasets used for evaluation, KoEPT obtained the state-of-the-art(SOTA) performance with 99.1% in CC, which is comparable to the existing SOTA performance, and 89.3% and 80.5% in IL and ALG514, respectively. In addition, as a result of evaluation, KoEPT showed a relatively improved performance for datasets with high classification difficulty. Through an ablation study, we uncovered that the use of the 'expression' tokens and pointer networks contributed to KoEPT's state of being less affected by classification difficulty while obtaining good performance.

이 논문에서는 자연어로 구성된 수학 문장제 문제 자동 풀이하기 위한 Transformer 기반의 생성 모델인 KoEPT를 제안한다. 수학 문장제 문제는 일상 상황을 수학적 형식으로 표현한 자연어 문제이다. 문장제 문제 풀이 기술은 함축된 논리를 인공지능이 파악해야 한다는 요구사항을 지녀 최근 인공지능의 언어 이해 능력을 증진하기 위해 국내외에서 다양하게 연구되고 있다. 한국어의 경우 문제를 유형으로 분류하여 풀이하는 기법들이 주로 시도되었으나, 이러한 기법은 다양한 수식을 포괄하여 분류 난도가 높은 데이터셋에 적용하기 어렵다는 한계가 있다. 본 논문은 이에 대해 '식' 토큰과 포인터 네트워크를 사용하는 KoEPT 모델을 사용했다. 이 모델의 성능을 측정하기 위해 현존하는 한국어 수학 문장제 문제 데이터셋인 IL, CC, ALG514의 분류 난도를 측정한 후 5겹 교차 검증 기법을 사용하여 KoEPT의 성능을 평가하였다. 평가에 사용된 한국어 데이터셋들에 대하여, KoEPT는 CC에서는 기존 최고 성능과 대등한 99.1%, IL과 ALG514에서 각각 89.3%, 80.5%로 새로운 최고 성능을 얻었다. 뿐만 아니라 평가 결과 KoEPT는 분류 난도가 높은 데이터셋에 대해 상대적으로 개선된 성능을 보였다. KoEPT가 분류 난도의 영향을 덜 받으며 좋은 성능을 얻게 된 이유를 '식' 토큰과 포인터 네트워크 때문이라는 것을 ablation study를 통해서 밝혔다.

Keywords

Acknowledgement

이 성과는 2021년도 정부(과학기술정보통신부)의 재원으로 한국연구재단 및 정보통신기획평가원의 지원을 받아 수행된 연구임(No.2020R1C1C1010162, No.2021-0-02146).

References

  1. C. Woo and G. Gweon, "solving automatically algebra math word problem in Korean," Annual Conference on Human and Language Technology, pp.310-315, 2018.
  2. K. Ki, D. Lee, and G. Gweon, "KoTAB: Korean template-based arithmetic solver with BERT," 2020 IEEE International Conference on Big Data and Smart Computing (BigComp), pp.279-282, 2020.
  3. J. Zhang, L. Wang, R. K. Lee, Y. Bin, Y. Wang, J. Shao, and E. Lim, "Graph-to-Tree learning for solving math word problems," Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.3928-3937, 2020.
  4. J. Zhang, R. K. Lee, E. Lim, W. Qin, L. Wang, J. Shao, and Q. Sun, "Teacher-Student networks with multiple decoders for solving math word problem," Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp.4011-4017, 2020.
  5. Y. Lan et al., "MWPToolkit: An open-source framework for deep learning-based math word problem solvers," [Internet], https://github.com/LYH-YF/MWPToolkit
  6. D. Hendrycks et al., "Measuring mathematical problem solving with the MATH dataset," 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks. 2021.
  7. N. Kushman, Y. Artzi, L. Zettlemoyer, and R. Barzilay, "Learning to automatically solve algebra word problems," Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Vol. 1: Long Papers, pp.271-281, 2014.
  8. D. Zhang, L. Wang, L. Zhang, B. T. Dai, and H. T. Shen, "The gap of semantic parsing: A survey on automatic math word problem solvers," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.42, No.9, pp.2287-2305, 2019. https://doi.org/10.1109/tpami.2019.2914054
  9. S. Roy and D. Roth, "Mapping to declarative knowledge for word problem solving," Transactions of the Association for Computational Linguistics, Vol.6, pp.159-172, 2018. https://doi.org/10.1162/tacl_a_00012
  10. L. Zhou, S. Dai, and L. Chen, "Learn to solve algebra word problems using quadratic programming," Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.817-822, 2015.
  11. J. D. Kenton, M. W. Chang, and L. K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," Proceedings of NAACL-HLT, pp.4171-4186. 2019.
  12. B. Kim, K. Ki, D. Lee, and G. Gweon, "Point to the expression: Solving algebraic word problems using the expression-pointer transformer model," Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp.3768-3779, 2020.
  13. A. Vaswani et al., "Attention is all you need," Advances in Neural Information Processing Systems, pp.5998-6008, 2017.
  14. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "ALBERT: A lite BERT for self-supervised learning of language representations," International Conference on Learning Representations, 2019.
  15. J. Lim, H. Kim, and Y. Kim, "Recent R&D trends for pretrained language model," Electronics and Telecommunications Trends, Vol.35, No.3, pp.9-19, 2020. https://doi.org/10.22648/ETRI.2020.J.350302
  16. J. Park, Pretrained ELECTRA Model for Korean [Internet], https://github.com/monologg/KoELECTRA.
  17. K. Clark, M. Luong, Q. V. Le, and C. D. Manning, "ELECTRA: Pre-training text encoders as discriminators rather than generators," International Conference on Learning Representations, 2019.
  18. D. Lee, J. Park, and S. Oh, "KB-ALBERT" [Internet], https://github.com/KB-AI-Research/KB-ALBERT
  19. O. Vinyals, M. Fortunato, and N. Jaitly, "Pointer networks," Advances in Neural Information Processing Systems, Vol.28, pp.2692-2700, 2015.
  20. A. Meurer et al., "SymPy: Symbolic computing in Python," PeerJ Computer Science, Vol.3, 2017.
  21. S. Roy, T. Vieira, and D. Roth, "Reasoning about quantities in natural language," Transactions of the Association for Computational Linguistics, Vol.3, pp.1-13, 2015. https://doi.org/10.1162/tacl_a_00118
  22. S. Roy and D. Roth, "Solving General Arithmetic Word Problems," In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.1743-1752, 2015.
  23. E. Collins, N. Rozanov, and B. Zhang, "Evolutionary data measures: Understanding the difficulty of text classification tasks," Proceedings of the 22nd Conference on Computational Natural Language Learning, pp.380-391, 2018.
  24. C. E. Shannon, "A mathematical theory of communication," ACM SIGMOBILE Mobile Computing and Communications Review, Vol.5, No.1, pp.3-55, 2001. https://doi.org/10.1145/584091.584093
  25. L. Le Cam and G. L. Yang, "Asymptotics in statistics: Some basic concepts," Springer Science and Business Media, 2012.