Browse > Article
http://dx.doi.org/10.7468/mathedu.2022.61.4.581

The utility of digital evaluation based on automatic item generation in mathematics: Focusing on the CAFA system  

Kim, Sungyeun (Incheon National University)
Publication Information
The Mathematical Education / v.61, no.4, 2022 , pp. 581-595 More about this Journal
Abstract
The purpose of this study is to specify the procedure for making item models based on ontology models using automatic item generation in the mathematics subject through the CAFA system, and to explore the generated item instances. As an illustration for this, an item model was designed as a part of formative assessment based on the content characteristics, including concepts and calculations, and process characteristics, including application, using the representative values and the measures of dispersion in Mathematics of the 9th grade based on the evaluation criteria achievement standards. The item types generated in one item model were a best answer type, a correct answer type, a combined-response type, an incomplete statement type, a negative type, a true-false type, and a matching type. It was found that HTML, Google Charts, TTS, figures, videos and so on can be used as media. The implications of the use of digital evaluation based on automatic item generation were suggested in the aspects of students, pre-service teachers, general teachers, and special education, and the limitations of this study and future research directions were presented.
Keywords
Digital Assessment; Item Model; Mathematics; Automatic Item Generation; CAFA;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 Kellogg, M., Rauch, S., Leathers, R., Simpson, M. A., Lines, D., Bickel, L., & Elmore, J. (2015, April). Construction of a dynamic item generator for K-12 mathematics. In National Council on Measurement in Education Conference, Chicago, IL..
2 Kim, C. H., Kim, B. Y., Moon, S. B., Kim, I. K., Kim, J. S., Park, J. H., Park, H. J., Lee, K. H., & Chae, J. E. (2015). A study on the development of educational indicators and indexes (IV). Korean Educational Development Institute.
3 Kim, H. R. (2021). New directions of english education course management for pre-service primary teachers in the post corona era. Educational Research, 81, 119-139. http://doi.org/10.17253/swueri.2021.81..006   DOI
4 Kim, S. H., & Kang, H. Y. (2020). Exploring Ways to Improve Statistics Curriculum by Analyzing Secondary Curriculum in Foreign Countries. School Mathematics, 22(3), 631-650. https://doi.org/ 10.29275/sm.2020.09.22.3.631   DOI
5 Kim, W., & Lim, W. (2020). Research trends in mathematics teacher learning community: Literature review. Communications of Mathematical Education, 34(4), 439-464. https://doi.org/10.7468/jksmee.2020.34.4.439   DOI
6 Dardick, W., & Choi, J. (2016). Teacher empowered assessment system: Assessment for the 21st century. Journal of Applied Educational and Policy Research, 2(2), 87-98.
7 Wikiedia (2022). Ontology, Wikipedia, Wikimedia Foundation, 23, February. 2022, https://ko.wikipedia.org/wiki/%EC%98%A8%ED%86%A8%EB%A1%9C%EC%A7%80
8 Xinxin, Z. (2019). Using Automatic Item Generation to Create Content for Computerized Formative Assessment [Unpublished doctoral dissertation, University of Alberta].
9 Vescio, V., Ross, D., & Adams, A. (2008). A review of research on the impact of professional learning communities on teaching practice and student learning. Teaching and Teacher Education, 24(1), 80-91. https://doi.org/10.1016/j.tate.2007.01.004   DOI
10 Jeong, J. C., Shin, K. A., Lee, S. H., & Yoo, W. H. (2008). Design and implementation of iterative contents based on SCORM in mathematics. Proceeding of the Winter Conference of the Korea Society of Computer and Information, 16, 153-158.
11 Kang, S. K., & Choi, S. P. (2020). Research on mathematical automatic item generation based on dynamic knowledge-base. Proceeding of the Korean Information Science Society Conference, 302-304.
12 Kim, H. I., Heo, J. W., & Kim, K. S. (2000). Development of web-based mathematics learning model for self-directed learning. Proceeding of the Korean Information Science Society Conference, 27(1B), 678-680.
13 Kim, Y., Choi, G., & Noh, T. (2009). High school students' errors in constructing and interpreting science graph. Journal of The Korean Association For Science Education, 29(8), 978-989.
14 LaDuca, A., Staples, W. I., Templeton, B., & Holzman, G. B. (1986). Item modeling procedure for constructing content-equivalent multiple choice questions. Medical education, 20(1), 53-56. https://doi.org/10.1111/j.1365-2923.1986.tb01042.x   DOI
15 Lee, S. G., Park, K. E., & Ham, Y. M. (2016). A study on 3D printed tactile mathematics textbook for visually impaired students. Communications of Mathematical Education, 30(4), 515-530. https://doi.org/10.7468/jksmee.2016.30.4.515-530   DOI
16 Lee, E. H. (2010). A summary-concept based analysis of the representative values and the measures of dispersion in the 9th grade textbook of Korean middle school [Master Thesis, Ewha Womans University].
17 Lee, S. M. (2020). Mathematical connection in teaching correlation-Analysis of 2015 Revised Mathematics Curriculum and textbooks- [Master Thesis, Seoul National University].
18 Lim, H. J. (2017). Study on the design and properties of automatically generated items: focusing on polynomial factorization [Unpublished doctoral dissertation, Seoul National University].
19 Luecht, R. M. (2012). An introduction to assessment engineering for automatic item generation. In Automatic item generation (pp. 69-86). Routledge.
20 Ministry of Education (2015). The guide of revision of educational curriculum. Ministry of Education Notice No. 2015-80. Ministry of Education.
21 Ministry of Education (2018). 2018 Multicultural education support plan. Ministry of Education
22 Mislevy, R. J., Behrens, J. T., Bennett, R. E., Demark, S. F., Frezzo, D. C., Levy, R., Robinson, D. H., Rutstein, D. W., Shute, V. J., Stanley, K., & Winters, F. I. (2010). On the roles of external knowledge representations in assessment design. Journal of Technology, Learning, and Assessment, 8(2), 1-56. https://doi.org/10.1037/e643752011-001   DOI
23 Oh, J. S., Chu, S. W., Cho, W. J., Kim, Y.S., & Lee, J. Y. (2006). A design of dynamic question generation system using Korean wordnet. Journal of Korean Institute of Information Technology, 4(5), 37-44.
24 Ministry of Education (2022). In order to strengthen digital public education, we will promote teacher competency enhancement and teacher recruitment. https://www.moe.go.kr/boardCnts/viewRenew.do?boardID=72771&lev=0&statusYN=W&s=moe&m=0315&opType=N&boardSeq=92581
25 Oh, K. S. (2022). A study on the development of reading assessment using automatic item generation -focused on reading comprehension item model-. Journal of CheongRam Korean Language Education, 87, 7-34.
26 Paik, H. S. (2013). Categorization of middle school students' math learning style preferences and comparison of academic characteristics. School Mathematics, 15(1), 15-35.
27 Park, K. M. et al., (2015). 2015 Revision mathematics curriculum proposal development study II. Ministry of Education. Research Report of Korea Foundation for the Advancement of Science and Creativity.
28 Park, J. H. (2017). A study on teaching method for mathematics underachiever through step-by-step learning card-focused on 'number and operations' unit of 9th grade [Master thesis, Korea National University of Education].
29 Choi, J. H. (2017). Introduction to Assessment Engineering. Assessment, Testing, and Measurement Technical Report Series. The George Washington University.
30 Choi, J., Kim, H., & Pak, S. (2018). Evaluation of Automatic Item Generation Utilities in Formative Assessment Application for Korean High School Students. Journal of Educational Issues, 4(1), 68-89. https://doi.org/10.5296/jei.v4i1.12630   DOI
31 Choi, J. H., Kim, S. H., & Yoon, K. I. (2012-2022). CAFA Modeling Manual: Computer Adaptive Formative Assessment User's Guide [System Manual], 2nd edition, CAFA Lab, Inc.
32 Chu, Y. G. (2014). Approach toward multicultural education and special education as inclusive education. Journal of Intellectual Disabilities, 16(2), 333-353.
33 Deane, P., Graf, E. A., Higgins, D., Futagi, Y., & Lawless, R. (2006). Model analysis and model creation: Capturing the task-model structure of quantitative item domains. ETS Research Report Series, 2006(1), I-63. https://doi.org/10.1002/j.2333-8504.2006.tb02017.x   DOI
34 Seo, K. H. (2009). Teacher learning communities and professional development. The Journal of Korean Teacher Education, 26(2), 243-276.   DOI
35 Schwinning, N., Schypula, M., Striewe, M., & Goedicke, M. (2014). Concepts and Realizations of Flexible Exercise Design and Feedback Generation in an e-Assessment System for Mathematics. In CICM Workshops (pp. 1-10).
36 Sinharay, S., & Johnson, M. S. (2012). Statistical modeling of automatically generated items. In Automatic item generation (pp. 193-205). Routledge.
37 Singley, M. K., & Bennett, R. E. (2002). Item generation and beyond: Applications of schema theory to mathematics assessment. In S. Irvine & P. Kyllonen (Eds.) Item Generating for test development (pp.361-384). Lawrence Erlbaum.
38 Ala-Mutka, K. (2011). Mapping digital competence: Towards a conceptual understanding. Sevilla: Institute for Prospective Technological Studies, 7-60.
39 Bejar, I. I., Lawless, R. R., Morley, M. E., Wagner, M. E., Bennett, R. E., & Revuelta, J. (2002). A feasibility study of on-the-fly item generation in adaptive testing. ETS Research Report Series, 2002(2), 1-44. https://doi.org/10.1002/j.2333-8504.2002.tb01890.x   DOI
40 Boo, J., & Vispoel, W. (2012). Computer versus paper-and-pencil assessment of educational development: A comparison of psychometric features and examinee preferences. Psychological Reports, 111(2), 443-460. https://doi.org/10.2466/10.03.11.PR0.111.5.443-460   DOI
41 Wilson, J., Morrison, K., & Embretson, S. E. (2014). Automatic item generator for mathematical achievement items: MathGen3.0. Technical report IES1005A-2014 for Institute of Educational Sciences Grant R305A100234. Cognitive Measurement Laboratory, Georgia Institute of Technology.
42 Bejar, I. I. (2002). Generative testing: From conception to implementation. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp.199-217). Erlbaum.
43 Carroll, S. N. (2021). Fourth grade student experiences in mathematics assessment: Computer versus paper-pencel testing modality. [Unpublished Doctoral Dissertation, Carson-Newman University].
44 Choi, J. H. (2019). Assessment engineering for learning analytics, Educational Development, 210, 42-45.
45 Choi, J. H. (2020). Educational innovation in the digital age: A plan to implement an intelligent learning analysis platform based on big data. Educational Development, 214, 44-50.
46 Choi, J. H., Oh, K. S., Youn, K. I., Lee, D. Y., Joung, J. M., Kim, S. Y., Youm, S. K., Lee, Y. E., Lee, E. J., Park, W. H. K & Lee, S. H. (2022). CAFA Model Example Book, Pubple.
47 Embretson, S. E., & Kingston, N. M. (2018). Automatic item generation: A more efficient process for developing mathematics achievement items?. Journal of Educational Measurement, 55(1), 112-131. https://doi.org/10.1111/jedm.12166   DOI
48 Gierl, M. J., & Haladyna, T. M. (2012). Automatic item generation: Theory and practice. Routledge.
49 Gierl, M. J., & Lai, H. (2013). Instructional topics in educational measurement (ITEMS) module: Using automated processes to generate test items. Educational Measurement: Issues and Practice, 32(3), 36-50. https://doi.org/10.1111/emip.12018   DOI
50 Fridenfalk, M. (2013). System for automatic generation of examination papers in discrete mathematics. International Association for Development of the Information Society.
51 Gierl, M. J., Zhou, J., & Alves, C. (2008). Developing a taxonomy of item model types to promote assessment engineering. Journal of Technology, Learning and Assessment, 7(2), 1-51.
52 Kang, N. R., Lim, D. K., & R, H. A. (2013). Analyzing the styles and types of math learning for middle school students. Journal of the Korean School Mathematics Society, 16(2), 363-381.
53 Holling, H., Bertling, J. P., & Zeuch, N. (2009). Automatic item generation of probability word problems. Studies in Educational Evaluation, 35(2-3), 71-76. https://doi.org/10.1016/j.stueduc.2009.10.004   DOI