Browse > Article
http://dx.doi.org/10.3745/KTSDE.2016.5.11.527

Automated Scoring System for Korean Short-Answer Questions Using Predictability and Unanimity  

Cheon, Min-Ah (한국해양대학교 컴퓨터공학과)
Kim, Chang-Hyun (한국전자통신연구원 언어처리연구실)
Kim, Jae-Hoon (한국해양대학교 IT공학부)
Noh, Eun-Hee (한국교육과정평가원)
Sung, Kyung-Hee (한국교육과정평가원)
Song, Mi-Young (한국교육과정평가원)
Publication Information
KIPS Transactions on Software and Data Engineering / v.5, no.11, 2016 , pp. 527-534 More about this Journal
Abstract
The emergent information society requires the talent for creative thinking based on problem-solving skills and comprehensive thinking rather than simple memorization. Therefore, the Korean curriculum has also changed into the direction of the creative thinking through increasing short-answer questions that can determine the overall thinking of the students. However, their scoring results are a little bit inconsistency because scoring short-answer questions depends on the subjective scoring of human raters. In order to alleviate this point, an automated scoring system using a machine learning has been used as a scoring tool in overseas. Linguistically, Korean and English is totally different in the structure of the sentences. Thus, the automated scoring system used in English cannot be applied to Korean. In this paper, we introduce an automated scoring system for Korean short-answer questions using predictability and unanimity. We also verify the practicality of the automatic scoring system through the correlation coefficient between the results of the automated scoring system and those of human raters. In the experiment of this paper, the proposed system is evaluated for constructed-response items of Korean language, social studies, and science in the National Assessment of Educational Achievement. The analysis was used Pearson correlation coefficients and Kappa coefficient. Results of the experiment had showed a strong positive correlation with all the correlation coefficients at 0.7 or higher. Thus, the scoring results of the proposed scoring system are similar to those of human raters. Therefore, the automated scoring system should be found to be useful as a scoring tool.
Keywords
Machine Learning; Korean Automated-Scoring System; Unanimity; Predictability; Natural Language Processing;
Citations & Related Records
Times Cited By KSCI : 6  (Citation Analysis)
연도 인용수 순위
1 S-D. Choi, J-Y. Kim, S-J. Ban, K-J. Lee, S-J. Lee, and H-Y. Choi, "Education Strategy to Foster Creative Talent for the 21st Century," Korean Educational Development Institute Research Report, RR 2011-01, 2011.
2 Ministry of Education, Science, and Technology, "Introduction to 2009 Revised National Curriculum," Ministry of Education, Science, and Technology Notification (2009-41), 2009.
3 Ministry of Education, Science, and Technology, "The Future Korea to Open Using Creative Talent and Advanced Science and Technology," 2011 Business Report, 2010.
4 Ministry of Education, Science, and Technology, "The Plan for Improving Education Management for Secondary Schools," Press Release, 2011.
5 Ministry of Education, "Introduction to National Curriculum for Elementary and Secondary Schools," Ministry of Education Notification (2015-74), 2015.
6 Korean Society for Educational Evaluation, "Dictionary of Educational Evaluation Terms," Seoul: Hakjisa, 2004.
7 J-S. Kim, "Guidelines for Short-Answer Questions in Korean Subject," Secondary Schools Policy Division in Chungcheongnamdo Office of Education, p.7, 2009.
8 K-A. Jin, "Development of Automated Scoring System for English Writing," English Language & Literature Teaching, Vol.13, No.1, pp.236-237, 2007.
9 Y. Attali and J. Burstein, "Automated Essay Scoring with E-rator v.2.0," ETS Research Report RR-04-45, 2005.
10 M. D. Shermis and J. Burstein, "Automated Essay Scoring: A Cross-Disciplinary Perspective," Inc., Publishers. Mahawah, New Jersey, 2003.
11 L. M. Rudner, V. Garcia, and C. Welch, "An Evaluation of the IntelliMetricSM Essay Scoring System," The Journal of Technology, Learning, and Assessment, Vol.4, No.4, 2006.
12 ETS, ETS Automated Scoring Technologies, ETS Report, 2010.
13 Ministry of Education, Science, and Technology, "The Master Plan for Creativity-Character Education," Press Release, 2011.
14 N-H. Noh, S-H. Lee, E-Y. Lim, K-H. Sung, and S-Y. Park, "The Development and Evaluation for Automatic Scoring Programs in Korean Large-Scale Assessments," Korea Institute of Curriculum & Evaluation, Research Report RRE 2014-6, 2014.
15 M-A. Cheon, H-W. Seo, J-H. Kim, E-H. Noh, and K-H. Sung, "Effects of Human Raters on Results of an Automatic Scoring System Based on Semi-Supervised Learning," Proceedings of Korea Computer Congress 2015, pp.666-668, 2015.
16 E-H. Noh, M-Y. Song, K-H. Sung, and S-Y. Park, "Refinements and Application of Automatic Scoring Programs for Korean Large-scale Assessments," Korea Institute of Curriculum & Evaluation, Research Report RRE 2015-9, 2015.
17 M.-Y. Song, E.-H. Noh, and K.-H, Sung, "Analysis on the Accuracy of Automated Scoring for Korean Large-scale Assessment," The Journal of Curriculum and Evaluation, Vol.19, No.2, pp.255-274, 2016.
18 M-A. Cheon, H-W. Seo, J-H. Kim, E-H. Noh, K-H. Sung, and E-Y. Lim, "Semi-Automatic Scoring for Short Korean Free-Text Responses Using Semi-Supervised Learning," Korean Journal of Cognitive Science, Vol.26, No.2, pp.147-165, 2015.   DOI
19 D. Y. Jung, "Evaluation of Short and Long Essay Questions By Using Vector similarity and Thesaurus," Master's Thesis in Graduate School of Education Dongguk University, 2001.
20 H. J. Park and W. S. Kang, "Design and Implementation of a Subjective-type Evaluation System Using Natural Language Processing Technique," The Journal of Korean Association of Computer Education, Vol.6, No.3, pp.207-217, 2003.
21 W.-S. Kang, "Automatic Grading System for Subjective Questions Through Analyzing Question Type," The Journal of the Korea Contents Association, Vol.11, No.2, pp.13-21, 2011.   DOI
22 W. J. Cho, J. S. Oh, J. Y. Lee, and Y.-S. Kim, "An Intelligent Marking System based on Semantic Kernel and Korean WordNet," The KIPS Transactions: Part A., Vol.12, No.6, pp.539-546, 2005.
23 P. Harrigton, "Machine Learning in Action," Manning Publications, 2012.
24 K. S. Shim, "Automatic Word Spacing based on Conditional Random Fields," Korean Journal of Cognitive Science, Vol.22, No.2, pp.217-233, 2011.   DOI
25 A. Sogaard, "Semi-Supervised Learning and Domain Adaptation in Natural Language Processing," Morgan & Claypool Publishers, 2013.
26 S.-S. Kang, "Korean Morphological Analysis and Information Retrieval (Korean edition)," Hong Reunggwahakchulpansa, 2002.
27 Romoku, [Internet] http://blog.faroo.com/2012/06/07/improv ed-edit-distance-based-spelling-correction/.
28 M.-A. Cheon, "Morphological Analysis and Part-of-Speech Tagging for Applying Korean Automated Scoring of Short-Answer Questions," Master's Thesis in Graduate School of Korea Maritime and Ocean University, 2016.
29 The National Institute of The Korean Language, "Korean Grammar for Foreigners 1," Seoul: Communicationbooks, 2005.
30 J. Nivre, "Algorithms for Deterministic Incremental Dependency Parsing," Computational Linguistics, Vol.34, No.4, pp.513-553, 2008.   DOI
31 G. Casella, S. Fienberg and I. Olkin, An Introduction to Statistical Learning with Applications in R, Springer.
32 Korea Institute for Curriculum & Evaluation, "Test Paper and Answers in 2014 National Assessment of Educational Achievement of Korea," 2014. (http://www.kice.re.kr/board Cnts/list.do?type=default&page=2&searchStr=&m=030302&C06=&boardID=1500208&C05=&C04=&C03=&searchType=S&C02=&C01=&s=kice).
33 D. M. Corey, W. P. Dunlap, and M. J. Burke, "Averaging Correlations: Expected Values and Bias in Combined Pears rs and Fisher's z Transformations," The Journal of General Psychology, Vol.125, No.3, pp. 245-261, 1998.   DOI
34 J. L. Fleiss, B. Levin, and M. C. Paik, "Statical methods for rates and propositions 3rd Edition," John Wiley & Sons, Inc., pp.598-626, 2003.
35 J. Carletta, "Assessing Agreement on Classification Tasks: The Kappa Statistic," Computational Linguistics, Vol.22, No.2, pp.249-254, 1996.