Acknowledgement
이 성과는 정부(과학기술정보통신부)의 재원으로 한국연구재단의 지원을 받아 수행된 연구임(No.2019R1F1A1059437, No.2022R1A2C1010310).
References
- Brennan, R. L. (2001a). Generalizability Theory. New York: Springer.
- Brennan, R. L. (2001b). Manual for mGENOVA (Version 2.1). [Computer software]. Iowa City, IA: University of Iowa.
- Choi, I. (2000). Prospects of computer adaptive testing and performance testing: Language testing in the 21st century. Language Research, 36(1), 205-241.
- Choi, J. (2020). Educational innovation in the digital age: A plan to implement an intelligent learning analysis platform based on big data. Educational Development, 214, 44-50.
- Choi, J., Kim, H., & Pak, S. (2018). Evaluation of automatic item generation utilities in formative assessment application for Korean high school students. Journal of Educational Issues, 4(1), 68-89.
- Choi, J., Kim, S., & Yoon, K. (2012-2023). CAFA Modeling Manual: Computer Adaptive Formative Assessment User's Guide [System Manual] (2nd edition). Clarksville, MD: CAFA Lab, Inc.
- Choi, J., Oh, K., Youn, K., Lee, D., Joung, J., Kim, S., Youm, S., Lee, Y., Lee, E., Park, W., & Lee, S. (2022). CAFA model example book. Seoul: Pubple.
- Churchill Jr, G. A. (1979). A paradigm for developing better measures of marketing constructs. Journal of marketing research, 16(1), 64-73. https://doi.org/10.1177/002224377901600110
- Cronbach, L. J., Linn, R. L., Brennan, R. L., & Haertel, E. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57(3), 373-399.
- Dunbar, S. B., Koretz, D. M., & Hoover, H. D. (1991). Quality control in the development and use of performance assessments. Applied Measurement in Education, 4(4), 289-303.
- Falcao, F., Pereira, D. M., Goncalves, N., De Champlain, A., Costa, P., & Pego, J. M. (2023). A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation. Advances in Health Sciences Education, 1-25.
- Gierl, M. J., & Lai, H. (2013). Instructional topics in educational measurement (ITEMS) module: Using automated processes to generate test items. Educational Measurement: Issues and Practice, 32(3), 36-50.
- Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. New York: Routledge.
- Jeong, H. (2009). A study on computer based test in education environment: Focused on students' experiences. Journal of Educational Technology, 25(4), 73-100. https://doi.org/10.17232/KSET.25.4.73
- Jeong, J., Shin, K., Lee, S., & Yoo, W. (2009). Design and implementation of iterative contents based on SCORM in mathematics. Proceeding of the Winter Conference of the Korea Society of Computer and Information, 16, 153-158.
- Kang, S., & Choi, S. (2020). Research on mathematical automatic item generation based on dynamic knowledge-base. Proceeding of the Korean Information Science Society Conference, 302-304.
- Keller, L. A., Clauser, B. E., & Swanson, D. B. (2010). Using multivariate generalizability theory to assess the effect of content stratification on the reliability of a performance assessment. Advances in Health Sciences Education, 15(5), 717-733.
- Kim, S. (2001). An analysis of sources of variation in the observational rating system-comparisons of observer agreement, interrater reliability, and generalizability theory. Journal of Educational Evaluation, 5(1), 37-56.
- Kim, S. (2014a). Exploring the application of generalizability theory to mathematics teacher evaluation for professional development in Korea based on the analysis of instructional quality assessment of mathematics teachers in the U.S. Communications of Mathematical Education, 28(4), 431-455. https://doi.org/10.7468/jksmee.2014.28.4.431
- Kim, S. (2014b). Exploring the application of multivariate generalizability theory to teacher evaluation for professional development in Korea based on the analysis of classroom observations in the US. The Journal of Korean Education, 41(1), 5-29.
- Kim, S. (2017). Multigroup generalizability analysis of Creative Attitude Scale-Korea for mathematically gifted and general students in middle schools. Communications of Mathematical Education, 31(1), 49-70. https://doi.org/10.7468/jksmee.2017.31.1.49
- Kim, S. (2022). The utility of digital evaluation based on automatic item generation in mathematics: focusing on the CAFA system. The Mathematical Education, 61(4), 581-596.
- Kim, S. (2023). Suggestions on the use of the CAFA system to promote teachers' digital competencies. The Journal of the Korea Contents Association, 23(4), 475-493. https://doi.org/10.5392/JKCA.2023.23.04.475
- Kim, S., & Berebitsky, D. (2016). An application of multivariate generalizability in selection of mathematically gifted students. Eurasia Journal of Mathematics, Science and Technology Education, 12(9), 2587-2598.
- Kim, S., & Choi, K. H. (2016). An investigation of efficient measurement conditions of the scientific creativity test. Secondary Education Research, 64(1), 49-75.
- Kim, S., & Chon, K. (2018). An Analysis of measurement errors and invariance properties by proficiency level in the non-cognitive measures. Journal of Curriculum Evaluation, 21(1), 153-172.
- Kim, S., & Han, K. (2014). Analysis of reliability coefficients depending on different domain weights in scoring teacher recommendation letters and self-introduction letters used in selection of mathematically gifted students. The Journal of the Korean Society for Gifted and Talented, 13(1), 43-66.
- Kim, S., & Kim, Y. (2001). Generalizability Theory. Seoul: Kyoyookbook.
- Kim, S., Song, M., & Park, I. (2012). Investigation on optimal conditions and error variance in standard setting using multivariate generalizability analysis. Journal of Educational Evaluation, 25(4), 679-700.
- Kim, S., Yeum, S., Chung, J., Yoon, K., & Park, S. (2023). Multivariate generalizability theory for reliability with item models: Industrial mathematics test example. NCME 2023 Annual Meeting, Chicago, IL: NCME.
- Lee, D. (2022, November 22). 4차 산업혁명 시대 따른 미래교육 모습은 [Future education in the era of the 4th industrial revolution]. Retrieved May 01, 2023, from http://m.wsobi.com/news/articleView.html?idxno=182590
- Lee, H. (2012). Multivariate generalizability analyses for mixed-format tests with various compositions of MC and CR item weights. Journal of Educational Evaluation, 25(1), 95-116.
- Lee, J., & Han, K. (2017). Finding optimal condition for conducting TTCT-Figural A (originality) using multivariate generalizability theory. The Journal of Creativity Education, 17(2), 57-77.
- Lee, J., Lee, S., & Ham, Y. (2022). Case study on college calculus education for vocational high school graduates with coding. Communications of Mathematical Education, 36(4), 611-626.
- Lee, S., Kim, S., Kim, J., Baek, K., & Lee, B. (2015). Analyses of the reliability of a preliminary creativity test using the multivariate generalizability theory. The Journal of Creativity Education, 15(3), 83-107.
- Lee, Y., & Shin, S. (2004). An investigation into the dependability of ratings in a German speaking test using the multivariate generalizability theory. Foreign Languages Education, 11(2), 259-265.
- Lim, H. (2017). Study on the design and properties of automatically generated items: focusing on polynomial factorization (Unpublished master thesis). Seoul National University, Seoul, Korea.
- Ministry of Education [MOE]. (2020). 코로나 이후, 미래교육 10대 정책과제 시안 발표 [Announcement of 10 policy tasks for future education after COVID-19]. Sejong, Korea. Retrieved from https://www.moe.go.kr/boardCnts/viewRenew.do?boardID=294&lev=0&statusYN=W&s=moe&m=020402&opType=N&boardSeq=82145
- Ministry of Education [MOE]. (2023). 디지털 기반 교육혁신 방안 [Digital-based education innovation plan]. Sejong, Korea. Retrieved from https://www.moe.go.kr/boardCnts/viewRenew.do?m=060209&s=moe&page=1&boardID=409&boardSeq=94072&lev=0&opType=N
- Oh, K. (2022). A study on the development of reading assessment using automatic item generation -Focused on reading comprehension item model-. Journal of CheongRam Korean Language Education, 87, 7-34.
- Shavelson, R. J., Baxter, G. P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30(3), 215-232.
- Song, I., & Kim, S. (2012). A validation study of epistemological belief using multivariate generalizability. The Korean Journal of Educational Methodology Studies, 24(1), 107-130.
- Vallevand, A. L. (2008). Reliability, validity and sources of errors in assessing physician performance in an Oobjective Structured Clinical Examination: A generalizability theory analysis (Unpublished doctoral dissertation). Calgary, Alberta, Canada.
- Webb, N. M., Shavelson, R. J., & Maddahian, E. (1983). Multivariate generalizability theory. In L. J. Pyans, Jr. (Eds.), Generalizability theory: Inferences and practical applications (pp.49-66). San Francisco, CA: Jossey-Bass.
- Wilhelm, A. G., & Kim, S. (2015). Generalizing from observations of mathematics teachers' instructional practice using the instructional quality assessment. Journal for Research in Mathematics Education, 46(3), 270-279.