Browse > Article
http://dx.doi.org/10.3745/KTSDE.2022.11.8.315

Analyzing Korean Math Word Problem Data Classification Difficulty Level Using the KoEPT Model  

Rhim, Sangkyu (서울대학교 지능정보융합학과)
Ki, Kyung Seo (서울대학교 지능정보융합학과)
Kim, Bugeun (서울대학교 인공지능혁신인재양성교육연구단)
Gweon, Gahgene (서울대학교 지능정보융합학과)
Publication Information
KIPS Transactions on Software and Data Engineering / v.11, no.8, 2022 , pp. 315-324 More about this Journal
Abstract
In this paper, we propose KoEPT, a Transformer-based generative model for automatic math word problems solving. A math word problem written in human language which describes everyday situations in a mathematical form. Math word problem solving requires an artificial intelligence model to understand the implied logic within the problem. Therefore, it is being studied variously across the world to improve the language understanding ability of artificial intelligence. In the case of the Korean language, studies so far have mainly attempted to solve problems by classifying them into templates, but there is a limitation in that these techniques are difficult to apply to datasets with high classification difficulty. To solve this problem, this paper used the KoEPT model which uses 'expression' tokens and pointer networks. To measure the performance of this model, the classification difficulty scores of IL, CC, and ALG514, which are existing Korean mathematical sentence problem datasets, were measured, and then the performance of KoEPT was evaluated using 5-fold cross-validation. For the Korean datasets used for evaluation, KoEPT obtained the state-of-the-art(SOTA) performance with 99.1% in CC, which is comparable to the existing SOTA performance, and 89.3% and 80.5% in IL and ALG514, respectively. In addition, as a result of evaluation, KoEPT showed a relatively improved performance for datasets with high classification difficulty. Through an ablation study, we uncovered that the use of the 'expression' tokens and pointer networks contributed to KoEPT's state of being less affected by classification difficulty while obtaining good performance.
Keywords
Math Word Problems; Generation Model; Transformer; Pointer Network; Classification Difficulty;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "ALBERT: A lite BERT for self-supervised learning of language representations," International Conference on Learning Representations, 2019.
2 J. Lim, H. Kim, and Y. Kim, "Recent R&D trends for pretrained language model," Electronics and Telecommunications Trends, Vol.35, No.3, pp.9-19, 2020.   DOI
3 K. Clark, M. Luong, Q. V. Le, and C. D. Manning, "ELECTRA: Pre-training text encoders as discriminators rather than generators," International Conference on Learning Representations, 2019.
4 D. Lee, J. Park, and S. Oh, "KB-ALBERT" [Internet], https://github.com/KB-AI-Research/KB-ALBERT
5 O. Vinyals, M. Fortunato, and N. Jaitly, "Pointer networks," Advances in Neural Information Processing Systems, Vol.28, pp.2692-2700, 2015.
6 A. Meurer et al., "SymPy: Symbolic computing in Python," PeerJ Computer Science, Vol.3, 2017.
7 S. Roy, T. Vieira, and D. Roth, "Reasoning about quantities in natural language," Transactions of the Association for Computational Linguistics, Vol.3, pp.1-13, 2015.   DOI
8 S. Roy and D. Roth, "Solving General Arithmetic Word Problems," In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.1743-1752, 2015.
9 C. E. Shannon, "A mathematical theory of communication," ACM SIGMOBILE Mobile Computing and Communications Review, Vol.5, No.1, pp.3-55, 2001.   DOI
10 E. Collins, N. Rozanov, and B. Zhang, "Evolutionary data measures: Understanding the difficulty of text classification tasks," Proceedings of the 22nd Conference on Computational Natural Language Learning, pp.380-391, 2018.
11 C. Woo and G. Gweon, "solving automatically algebra math word problem in Korean," Annual Conference on Human and Language Technology, pp.310-315, 2018.
12 D. Zhang, L. Wang, L. Zhang, B. T. Dai, and H. T. Shen, "The gap of semantic parsing: A survey on automatic math word problem solvers," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.42, No.9, pp.2287-2305, 2019.   DOI
13 J. Zhang, R. K. Lee, E. Lim, W. Qin, L. Wang, J. Shao, and Q. Sun, "Teacher-Student networks with multiple decoders for solving math word problem," Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp.4011-4017, 2020.
14 N. Kushman, Y. Artzi, L. Zettlemoyer, and R. Barzilay, "Learning to automatically solve algebra word problems," Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Vol. 1: Long Papers, pp.271-281, 2014.
15 L. Zhou, S. Dai, and L. Chen, "Learn to solve algebra word problems using quadratic programming," Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.817-822, 2015.
16 J. Park, Pretrained ELECTRA Model for Korean [Internet], https://github.com/monologg/KoELECTRA.
17 K. Ki, D. Lee, and G. Gweon, "KoTAB: Korean template-based arithmetic solver with BERT," 2020 IEEE International Conference on Big Data and Smart Computing (BigComp), pp.279-282, 2020.
18 J. Zhang, L. Wang, R. K. Lee, Y. Bin, Y. Wang, J. Shao, and E. Lim, "Graph-to-Tree learning for solving math word problems," Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.3928-3937, 2020.
19 Y. Lan et al., "MWPToolkit: An open-source framework for deep learning-based math word problem solvers," [Internet], https://github.com/LYH-YF/MWPToolkit
20 D. Hendrycks et al., "Measuring mathematical problem solving with the MATH dataset," 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks. 2021.
21 S. Roy and D. Roth, "Mapping to declarative knowledge for word problem solving," Transactions of the Association for Computational Linguistics, Vol.6, pp.159-172, 2018.   DOI
22 J. D. Kenton, M. W. Chang, and L. K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," Proceedings of NAACL-HLT, pp.4171-4186. 2019.
23 B. Kim, K. Ki, D. Lee, and G. Gweon, "Point to the expression: Solving algebraic word problems using the expression-pointer transformer model," Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp.3768-3779, 2020.
24 A. Vaswani et al., "Attention is all you need," Advances in Neural Information Processing Systems, pp.5998-6008, 2017.
25 L. Le Cam and G. L. Yang, "Asymptotics in statistics: Some basic concepts," Springer Science and Business Media, 2012.