한국수학교육학회지시리즈A:수학교육 (The Mathematical Education)

  • 제61권4호
  • /
  • Pages.581-595
  • /
  • 2022
  • /
  • 1225-1380(pISSN)
  • /
  • 2287-9633(eISSN)
한국수학교육학회 (Korean Society of Mathematical Education)
권호 표지
DOI QR코드

DOI QR Code

수학교과에서 자동문항생성 기반의 디지털 평가 활용 방안: CAFA 시스템을 중심으로

The utility of digital evaluation based on automatic item generation in mathematics: Focusing on the CAFA system

  • 투고 : 2022.10.05
  • 심사 : 2022.10.21
  • 발행 : 2022.11.30
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

본 연구의 목적은 수학 교과에서 자동문항생성을 활용하여 지식의 핵심 구조인 온톨로지모형 기반의 문항모형을 CAFA 시스템을 통해 제작하는 절차를 명세하고, 생성된 문항 사례들을 탐색하는 데 있다. 이를 위한 사례로 수학 3의 대푯값과 산포도 단원의 평가준거 성취기준을 바탕으로 개념과 계산을 포함한 내용적 특성과 적용을 포함한 과정적 특성을 바탕으로 형성평가에 적합한 문항모형을 제작하였다. 하나의 문항모형에서 생성된 문항 유형은 최선답형, 정답형, 합답형, 미완성문장형, 부정형, 진위형, 배합형 등이었으며, 매체로는 Google Chart, HTML, TTS, 그림, 비디오 등을 활용할 수 있는 것으로 나타났다. 이를 바탕으로 자동문항생성 기반의 디지털 평가 활용방안에 대한 시사점을 학생, 예비교사, 일반교사 그리고 특수교육 측면에서 논의하고, 본 연구의 한계점과 향후 연구방향을 제시하였다.

The purpose of this study is to specify the procedure for making item models based on ontology models using automatic item generation in the mathematics subject through the CAFA system, and to explore the generated item instances. As an illustration for this, an item model was designed as a part of formative assessment based on the content characteristics, including concepts and calculations, and process characteristics, including application, using the representative values and the measures of dispersion in Mathematics of the 9th grade based on the evaluation criteria achievement standards. The item types generated in one item model were a best answer type, a correct answer type, a combined-response type, an incomplete statement type, a negative type, a true-false type, and a matching type. It was found that HTML, Google Charts, TTS, figures, videos and so on can be used as media. The implications of the use of digital evaluation based on automatic item generation were suggested in the aspects of students, pre-service teachers, general teachers, and special education, and the limitations of this study and future research directions were presented.

키워드

과제정보

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2022R1A2C1010310). I would like to thank three anonymous reviewers and the editor for their valuable comments and suggestions to improve this paper. I would also like to thank Professor Jaehwa Choi (The Geroge Washington University) and Kyongil Yoon (Notre Dame of Maryland Univeristy) for their dedicated support in using the CAFA system.

참고문헌

  1. Ala-Mutka, K. (2011). Mapping digital competence: Towards a conceptual understanding. Sevilla: Institute for Prospective Technological Studies, 7-60.
  2. Bejar, I. I. (2002). Generative testing: From conception to implementation. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp.199-217). Erlbaum.
  3. Bejar, I. I., Lawless, R. R., Morley, M. E., Wagner, M. E., Bennett, R. E., & Revuelta, J. (2002). A feasibility study of on-the-fly item generation in adaptive testing. ETS Research Report Series, 2002(2), 1-44. https://doi.org/10.1002/j.2333-8504.2002.tb01890.x
  4. Boo, J., & Vispoel, W. (2012). Computer versus paper-and-pencil assessment of educational development: A comparison of psychometric features and examinee preferences. Psychological Reports, 111(2), 443-460. https://doi.org/10.2466/10.03.11.PR0.111.5.443-460
  5. Carroll, S. N. (2021). Fourth grade student experiences in mathematics assessment: Computer versus paper-pencel testing modality. [Unpublished Doctoral Dissertation, Carson-Newman University].
  6. Choi, J. H. (2017). Introduction to Assessment Engineering. Assessment, Testing, and Measurement Technical Report Series. The George Washington University.
  7. Choi, J., Kim, H., & Pak, S. (2018). Evaluation of Automatic Item Generation Utilities in Formative Assessment Application for Korean High School Students. Journal of Educational Issues, 4(1), 68-89. https://doi.org/10.5296/jei.v4i1.12630
  8. Choi, J. H. (2019). Assessment engineering for learning analytics, Educational Development, 210, 42-45.
  9. Choi, J. H. (2020). Educational innovation in the digital age: A plan to implement an intelligent learning analysis platform based on big data. Educational Development, 214, 44-50.
  10. Choi, J. H., Oh, K. S., Youn, K. I., Lee, D. Y., Joung, J. M., Kim, S. Y., Youm, S. K., Lee, Y. E., Lee, E. J., Park, W. H. K & Lee, S. H. (2022). CAFA Model Example Book, Pubple.
  11. Choi, J. H., Kim, S. H., & Yoon, K. I. (2012-2022). CAFA Modeling Manual: Computer Adaptive Formative Assessment User's Guide [System Manual], 2nd edition, CAFA Lab, Inc.
  12. Chu, Y. G. (2014). Approach toward multicultural education and special education as inclusive education. Journal of Intellectual Disabilities, 16(2), 333-353.
  13. Dardick, W., & Choi, J. (2016). Teacher empowered assessment system: Assessment for the 21st century. Journal of Applied Educational and Policy Research, 2(2), 87-98.
  14. Deane, P., Graf, E. A., Higgins, D., Futagi, Y., & Lawless, R. (2006). Model analysis and model creation: Capturing the task-model structure of quantitative item domains. ETS Research Report Series, 2006(1), I-63. https://doi.org/10.1002/j.2333-8504.2006.tb02017.x
  15. Embretson, S. E., & Kingston, N. M. (2018). Automatic item generation: A more efficient process for developing mathematics achievement items?. Journal of Educational Measurement, 55(1), 112-131. https://doi.org/10.1111/jedm.12166
  16. Fridenfalk, M. (2013). System for automatic generation of examination papers in discrete mathematics. International Association for Development of the Information Society.
  17. Gierl, M. J., & Haladyna, T. M. (2012). Automatic item generation: Theory and practice. Routledge.
  18. Gierl, M. J., & Lai, H. (2013). Instructional topics in educational measurement (ITEMS) module: Using automated processes to generate test items. Educational Measurement: Issues and Practice, 32(3), 36-50. https://doi.org/10.1111/emip.12018
  19. Gierl, M. J., Zhou, J., & Alves, C. (2008). Developing a taxonomy of item model types to promote assessment engineering. Journal of Technology, Learning and Assessment, 7(2), 1-51.
  20. Holling, H., Bertling, J. P., & Zeuch, N. (2009). Automatic item generation of probability word problems. Studies in Educational Evaluation, 35(2-3), 71-76. https://doi.org/10.1016/j.stueduc.2009.10.004
  21. Jeong, J. C., Shin, K. A., Lee, S. H., & Yoo, W. H. (2008). Design and implementation of iterative contents based on SCORM in mathematics. Proceeding of the Winter Conference of the Korea Society of Computer and Information, 16, 153-158.
  22. Kang, N. R., Lim, D. K., & R, H. A. (2013). Analyzing the styles and types of math learning for middle school students. Journal of the Korean School Mathematics Society, 16(2), 363-381.
  23. Kang, S. K., & Choi, S. P. (2020). Research on mathematical automatic item generation based on dynamic knowledge-base. Proceeding of the Korean Information Science Society Conference, 302-304.
  24. Kellogg, M., Rauch, S., Leathers, R., Simpson, M. A., Lines, D., Bickel, L., & Elmore, J. (2015, April). Construction of a dynamic item generator for K-12 mathematics. In National Council on Measurement in Education Conference, Chicago, IL..
  25. Kim, C. H., Kim, B. Y., Moon, S. B., Kim, I. K., Kim, J. S., Park, J. H., Park, H. J., Lee, K. H., & Chae, J. E. (2015). A study on the development of educational indicators and indexes (IV). Korean Educational Development Institute.
  26. Kim, H. I., Heo, J. W., & Kim, K. S. (2000). Development of web-based mathematics learning model for self-directed learning. Proceeding of the Korean Information Science Society Conference, 27(1B), 678-680.
  27. Kim, H. R. (2021). New directions of english education course management for pre-service primary teachers in the post corona era. Educational Research, 81, 119-139. http://doi.org/10.17253/swueri.2021.81..006
  28. Kim, S. H., & Kang, H. Y. (2020). Exploring Ways to Improve Statistics Curriculum by Analyzing Secondary Curriculum in Foreign Countries. School Mathematics, 22(3), 631-650. https://doi.org/ 10.29275/sm.2020.09.22.3.631
  29. Kim, Y., Choi, G., & Noh, T. (2009). High school students' errors in constructing and interpreting science graph. Journal of The Korean Association For Science Education, 29(8), 978-989.
  30. Kim, W., & Lim, W. (2020). Research trends in mathematics teacher learning community: Literature review. Communications of Mathematical Education, 34(4), 439-464. https://doi.org/10.7468/jksmee.2020.34.4.439
  31. LaDuca, A., Staples, W. I., Templeton, B., & Holzman, G. B. (1986). Item modeling procedure for constructing content-equivalent multiple choice questions. Medical education, 20(1), 53-56. https://doi.org/10.1111/j.1365-2923.1986.tb01042.x
  32. Lee, S. G., Park, K. E., & Ham, Y. M. (2016). A study on 3D printed tactile mathematics textbook for visually impaired students. Communications of Mathematical Education, 30(4), 515-530. https://doi.org/10.7468/jksmee.2016.30.4.515-530
  33. Lee, E. H. (2010). A summary-concept based analysis of the representative values and the measures of dispersion in the 9th grade textbook of Korean middle school [Master Thesis, Ewha Womans University].
  34. Lee, S. M. (2020). Mathematical connection in teaching correlation-Analysis of 2015 Revised Mathematics Curriculum and textbooks- [Master Thesis, Seoul National University].
  35. Lim, H. J. (2017). Study on the design and properties of automatically generated items: focusing on polynomial factorization [Unpublished doctoral dissertation, Seoul National University].
  36. Luecht, R. M. (2012). An introduction to assessment engineering for automatic item generation. In Automatic item generation (pp. 69-86). Routledge.
  37. Ministry of Education (2015). The guide of revision of educational curriculum. Ministry of Education Notice No. 2015-80. Ministry of Education.
  38. Ministry of Education (2018). 2018 Multicultural education support plan. Ministry of Education
  39. Ministry of Education (2022). In order to strengthen digital public education, we will promote teacher competency enhancement and teacher recruitment. https://www.moe.go.kr/boardCnts/viewRenew.do?boardID=72771&lev=0&statusYN=W&s=moe&m=0315&opType=N&boardSeq=92581
  40. Mislevy, R. J., Behrens, J. T., Bennett, R. E., Demark, S. F., Frezzo, D. C., Levy, R., Robinson, D. H., Rutstein, D. W., Shute, V. J., Stanley, K., & Winters, F. I. (2010). On the roles of external knowledge representations in assessment design. Journal of Technology, Learning, and Assessment, 8(2), 1-56. https://doi.org/10.1037/e643752011-001
  41. Oh, J. S., Chu, S. W., Cho, W. J., Kim, Y.S., & Lee, J. Y. (2006). A design of dynamic question generation system using Korean wordnet. Journal of Korean Institute of Information Technology, 4(5), 37-44.
  42. Oh, K. S. (2022). A study on the development of reading assessment using automatic item generation -focused on reading comprehension item model-. Journal of CheongRam Korean Language Education, 87, 7-34.
  43. Paik, H. S. (2013). Categorization of middle school students' math learning style preferences and comparison of academic characteristics. School Mathematics, 15(1), 15-35.
  44. Park, K. M. et al., (2015). 2015 Revision mathematics curriculum proposal development study II. Ministry of Education. Research Report of Korea Foundation for the Advancement of Science and Creativity.
  45. Park, J. H. (2017). A study on teaching method for mathematics underachiever through step-by-step learning card-focused on 'number and operations' unit of 9th grade [Master thesis, Korea National University of Education].
  46. Seo, K. H. (2009). Teacher learning communities and professional development. The Journal of Korean Teacher Education, 26(2), 243-276. https://doi.org/10.24211/tjkte.2009.26.2.243
  47. Schwinning, N., Schypula, M., Striewe, M., & Goedicke, M. (2014). Concepts and Realizations of Flexible Exercise Design and Feedback Generation in an e-Assessment System for Mathematics. In CICM Workshops (pp. 1-10).
  48. Sinharay, S., & Johnson, M. S. (2012). Statistical modeling of automatically generated items. In Automatic item generation (pp. 193-205). Routledge.
  49. Singley, M. K., & Bennett, R. E. (2002). Item generation and beyond: Applications of schema theory to mathematics assessment. In S. Irvine & P. Kyllonen (Eds.) Item Generating for test development (pp.361-384). Lawrence Erlbaum.
  50. Vescio, V., Ross, D., & Adams, A. (2008). A review of research on the impact of professional learning communities on teaching practice and student learning. Teaching and Teacher Education, 24(1), 80-91. https://doi.org/10.1016/j.tate.2007.01.004
  51. Wilson, J., Morrison, K., & Embretson, S. E. (2014). Automatic item generator for mathematical achievement items: MathGen3.0. Technical report IES1005A-2014 for Institute of Educational Sciences Grant R305A100234. Cognitive Measurement Laboratory, Georgia Institute of Technology.
  52. Wikiedia (2022). Ontology, Wikipedia, Wikimedia Foundation, 23, February. 2022, https://ko.wikipedia.org/wiki/%EC%98%A8%ED%86%A8%EB%A1%9C%EC%A7%80
  53. Xinxin, Z. (2019). Using Automatic Item Generation to Create Content for Computerized Formative Assessment [Unpublished doctoral dissertation, University of Alberta].