KIPS Transactions on Software and Data Engineering (정보처리학회논문지:소프트웨어 및 데이터공학)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Automated Scoring System for Korean Short-Answer Questions Using Predictability and Unanimity

기계학습 분류기의 예측확률과 만장일치를 이용한 한국어 서답형 문항 자동채점 시스템

  • 천민아 (한국해양대학교 컴퓨터공학과) ;
  • 김창현 (한국전자통신연구원 언어처리연구실) ;
  • 김재훈 (한국해양대학교 IT공학부) ;
  • 노은희 (한국교육과정평가원) ;
  • 성경희 (한국교육과정평가원) ;
  • 송미영 (한국교육과정평가원)
  • Received : 2016.10.04
  • Accepted : 2016.10.12
  • Published : 2016.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

The emergent information society requires the talent for creative thinking based on problem-solving skills and comprehensive thinking rather than simple memorization. Therefore, the Korean curriculum has also changed into the direction of the creative thinking through increasing short-answer questions that can determine the overall thinking of the students. However, their scoring results are a little bit inconsistency because scoring short-answer questions depends on the subjective scoring of human raters. In order to alleviate this point, an automated scoring system using a machine learning has been used as a scoring tool in overseas. Linguistically, Korean and English is totally different in the structure of the sentences. Thus, the automated scoring system used in English cannot be applied to Korean. In this paper, we introduce an automated scoring system for Korean short-answer questions using predictability and unanimity. We also verify the practicality of the automatic scoring system through the correlation coefficient between the results of the automated scoring system and those of human raters. In the experiment of this paper, the proposed system is evaluated for constructed-response items of Korean language, social studies, and science in the National Assessment of Educational Achievement. The analysis was used Pearson correlation coefficients and Kappa coefficient. Results of the experiment had showed a strong positive correlation with all the correlation coefficients at 0.7 or higher. Thus, the scoring results of the proposed scoring system are similar to those of human raters. Therefore, the automated scoring system should be found to be useful as a scoring tool.

최근 정보화 사회에서는 단순 암기보다는 문제 해결 능력과 종합적인 사고력을 바탕으로 창의적인 생각을 할 수 있는 인재를 요구한다. 이에 따라 교육과정도 학생들의 종합적인 사고력을 판단할 수 있는 서답형 문항을 늘리는 방향으로 변하고 있다. 그러나 서답형 문항의 경우 채점자의 주관에 의존하여 채점이 진행되기 때문에, 채점 결과의 일관성을 확보하기 어렵다는 단점이 있다. 이런 점을 해결하기 위해 해외에서는 기계학습을 이용한 자동채점 시스템을 채점 도구로 사용하고 있다. 한국어는 영어와 언어학적으로 다른 분류에 속하므로 영어권에서 사용하는 자동채점 시스템을 한국어에 그대로 적용할 수 없다. 따라서 한국어 체계에 맞는 자동채점 시스템의 개발이 필요하다. 본 논문에서는 기계학습 분류기의 예측확률과 만장일치 방법을 사용한 한국어 서답형 문항 자동채점 시스템을 소개하고, 자동채점 시스템을 이용한 채점 결과와 교과 전문가의 채점 결과를 비교하여 자동채점 시스템의 실용성을 검증한다. 본 논문의 실험을 위해 2014년 국가수준 학업성취도 평가의 국어, 사회, 과학 교과의 서답형 문항을 사용했다. 평가 척도로 피어슨 상관계수와 카파계수를 사용했다. 채점자가 개입했을 때와 개입하지 않았을 때의 상관계수 모두 0.7 이상으로 강한 양의 상관관계를 보였다. 이는 자동채점 시스템이 교과 전문가가 채점한 결과와 유사한 방향으로 답안에 점수를 부여한 것이므로 자동채점 시스템을 채점 보조도구로서 충분히 사용할 수 있을 것이다.

Keywords

References

  1. S-D. Choi, J-Y. Kim, S-J. Ban, K-J. Lee, S-J. Lee, and H-Y. Choi, "Education Strategy to Foster Creative Talent for the 21st Century," Korean Educational Development Institute Research Report, RR 2011-01, 2011.
  2. Ministry of Education, Science, and Technology, "Introduction to 2009 Revised National Curriculum," Ministry of Education, Science, and Technology Notification (2009-41), 2009.
  3. Ministry of Education, Science, and Technology, "The Future Korea to Open Using Creative Talent and Advanced Science and Technology," 2011 Business Report, 2010.
  4. Ministry of Education, Science, and Technology, "The Master Plan for Creativity-Character Education," Press Release, 2011.
  5. Ministry of Education, Science, and Technology, "The Plan for Improving Education Management for Secondary Schools," Press Release, 2011.
  6. Ministry of Education, "Introduction to National Curriculum for Elementary and Secondary Schools," Ministry of Education Notification (2015-74), 2015.
  7. Korean Society for Educational Evaluation, "Dictionary of Educational Evaluation Terms," Seoul: Hakjisa, 2004.
  8. J-S. Kim, "Guidelines for Short-Answer Questions in Korean Subject," Secondary Schools Policy Division in Chungcheongnamdo Office of Education, p.7, 2009.
  9. K-A. Jin, "Development of Automated Scoring System for English Writing," English Language & Literature Teaching, Vol.13, No.1, pp.236-237, 2007.
  10. Y. Attali and J. Burstein, "Automated Essay Scoring with E-rator v.2.0," ETS Research Report RR-04-45, 2005.
  11. M. D. Shermis and J. Burstein, "Automated Essay Scoring: A Cross-Disciplinary Perspective," Inc., Publishers. Mahawah, New Jersey, 2003.
  12. L. M. Rudner, V. Garcia, and C. Welch, "An Evaluation of the IntelliMetricSM Essay Scoring System," The Journal of Technology, Learning, and Assessment, Vol.4, No.4, 2006.
  13. ETS, ETS Automated Scoring Technologies, ETS Report, 2010.
  14. N-H. Noh, S-H. Lee, E-Y. Lim, K-H. Sung, and S-Y. Park, "The Development and Evaluation for Automatic Scoring Programs in Korean Large-Scale Assessments," Korea Institute of Curriculum & Evaluation, Research Report RRE 2014-6, 2014.
  15. E-H. Noh, M-Y. Song, K-H. Sung, and S-Y. Park, "Refinements and Application of Automatic Scoring Programs for Korean Large-scale Assessments," Korea Institute of Curriculum & Evaluation, Research Report RRE 2015-9, 2015.
  16. M.-Y. Song, E.-H. Noh, and K.-H, Sung, "Analysis on the Accuracy of Automated Scoring for Korean Large-scale Assessment," The Journal of Curriculum and Evaluation, Vol.19, No.2, pp.255-274, 2016.
  17. M-A. Cheon, H-W. Seo, J-H. Kim, E-H. Noh, K-H. Sung, and E-Y. Lim, "Semi-Automatic Scoring for Short Korean Free-Text Responses Using Semi-Supervised Learning," Korean Journal of Cognitive Science, Vol.26, No.2, pp.147-165, 2015. https://doi.org/10.19066/cogsci.2015.26.2.002
  18. M-A. Cheon, H-W. Seo, J-H. Kim, E-H. Noh, and K-H. Sung, "Effects of Human Raters on Results of an Automatic Scoring System Based on Semi-Supervised Learning," Proceedings of Korea Computer Congress 2015, pp.666-668, 2015.
  19. D. Y. Jung, "Evaluation of Short and Long Essay Questions By Using Vector similarity and Thesaurus," Master's Thesis in Graduate School of Education Dongguk University, 2001.
  20. H. J. Park and W. S. Kang, "Design and Implementation of a Subjective-type Evaluation System Using Natural Language Processing Technique," The Journal of Korean Association of Computer Education, Vol.6, No.3, pp.207-217, 2003.
  21. W.-S. Kang, "Automatic Grading System for Subjective Questions Through Analyzing Question Type," The Journal of the Korea Contents Association, Vol.11, No.2, pp.13-21, 2011. https://doi.org/10.5392/JKCA.2011.11.2.013
  22. W. J. Cho, J. S. Oh, J. Y. Lee, and Y.-S. Kim, "An Intelligent Marking System based on Semantic Kernel and Korean WordNet," The KIPS Transactions: Part A., Vol.12, No.6, pp.539-546, 2005.
  23. P. Harrigton, "Machine Learning in Action," Manning Publications, 2012.
  24. A. Sogaard, "Semi-Supervised Learning and Domain Adaptation in Natural Language Processing," Morgan & Claypool Publishers, 2013.
  25. S.-S. Kang, "Korean Morphological Analysis and Information Retrieval (Korean edition)," Hong Reunggwahakchulpansa, 2002.
  26. Romoku, [Internet] http://blog.faroo.com/2012/06/07/improv ed-edit-distance-based-spelling-correction/.
  27. K. S. Shim, "Automatic Word Spacing based on Conditional Random Fields," Korean Journal of Cognitive Science, Vol.22, No.2, pp.217-233, 2011. https://doi.org/10.19066/cogsci.2011.22.2.007
  28. M.-A. Cheon, "Morphological Analysis and Part-of-Speech Tagging for Applying Korean Automated Scoring of Short-Answer Questions," Master's Thesis in Graduate School of Korea Maritime and Ocean University, 2016.
  29. The National Institute of The Korean Language, "Korean Grammar for Foreigners 1," Seoul: Communicationbooks, 2005.
  30. J. Nivre, "Algorithms for Deterministic Incremental Dependency Parsing," Computational Linguistics, Vol.34, No.4, pp.513-553, 2008. https://doi.org/10.1162/coli.07-056-R1-07-027
  31. G. Casella, S. Fienberg and I. Olkin, An Introduction to Statistical Learning with Applications in R, Springer.
  32. Korea Institute for Curriculum & Evaluation, "Test Paper and Answers in 2014 National Assessment of Educational Achievement of Korea," 2014. (http://www.kice.re.kr/board Cnts/list.do?type=default&page=2&searchStr=&m=030302&C06=&boardID=1500208&C05=&C04=&C03=&searchType=S&C02=&C01=&s=kice).
  33. D. M. Corey, W. P. Dunlap, and M. J. Burke, "Averaging Correlations: Expected Values and Bias in Combined Pears rs and Fisher's z Transformations," The Journal of General Psychology, Vol.125, No.3, pp. 245-261, 1998. https://doi.org/10.1080/00221309809595548
  34. J. Carletta, "Assessing Agreement on Classification Tasks: The Kappa Statistic," Computational Linguistics, Vol.22, No.2, pp.249-254, 1996.
  35. J. L. Fleiss, B. Levin, and M. C. Paik, "Statical methods for rates and propositions 3rd Edition," John Wiley & Sons, Inc., pp.598-626, 2003.