Browse > Article
http://dx.doi.org/10.5392/JKCA.2021.21.03.410

Reliability Analysis and Improvement Plan for Evaluation of Program Outcomes among Demand-driven Raters  

Lee, Youngho (국립목포대학교 컴퓨터공학과)
Shin, Younghak (국립목포대학교 컴퓨터공학과)
Kim, Jonghwa (국립목포대학교 컴퓨터공학과)
Publication Information
The Journal of the Korea Contents Association / v.21, no.3, 2021 , pp. 410-418 More about this Journal
Abstract
In a program that runs an engineering education certification, program outcomes refer to the knowledge, skills, and attitudes a student must have until graduation. In general, capstone design is used as a tool for evaluating program outcomes. This paper applies the intraclass correlation coefficient (ICC) to measure the raters' reliability in assessing program outcomes. Several raters evaluate program outcomes, and the result is used to obtain the raters' ICC. ICC measures the reliability of ratings or measurements for clusters - data that has been collected as groups or sorted into groups. If the ICC is close to 1, it means that the reliability among the raters is high. We evaluated the proposed method's usefulness through case analysis. As a method for assessing an evaluation tool's objectivity, multiple raters measure the same evaluation tool. As a result, we measured the ICC values for all POs, and analyzed the cause for the low measured POs. We applied this method to evaluate program outcomes of the Department of Computer Engineering in the past two years. As a result, we derived guidelines for improvement and program outcomes.
Keywords
Engineering Education; Program Outcomes; Intraclass Correlation Coefficient; Assessment; Rubrics; Demand-driven Evaluator;
Citations & Related Records
연도 인용수 순위
  • Reference
1 한국공학교육인증원 2020년 적용 컴퓨터.정보공학교육인증기준 KCC2015, http://www.abeek.or.kr/intro/stendard, 2020.10.4.
2 한국공학교육인증원 2020년 적용 컴퓨터.정보공학교육인증기준 KCC2015 판정가이드, http://www.abeek.or.kr/intro/stendard, 2020.10.4.
3 이희원, 김성환, 박근, 김정엽, "캡스톤디자인 교과목 기반의 프로그램 학습성과 평가 연구," 공학교육연구, 제13권, 제6호, pp.143-151, 2010.   DOI
4 박윤국, "공학교육인증에서 교과목 학습성과와 프로그램 학습성과의 관계 - 실질적인 프로그램 습성과 평가를 지향하며," 공학교육연구, 제13권, 제4호, pp.87-92, 2010.
5 이영태, 임철일, "성찰저널을 활용한 프로그램 학습성과 평가체계 개발," 공학교육연구, 제16권, 제3호, pp.42-50, 2013.   DOI
6 T. K. Koo and M. Y. Li, "A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research," Journal of Chiropractic Medicine, Vol.15, No.2, pp.155-163, 2016. doi:10.1016/j.jcm.2016.02.012   DOI
7 J. J. Bartko, "The intraclass correlation coefficient as a measure of reliability," Psychol Rep, Vol.19, No.1, pp.3-11, 1966. doi:10.2466/pr0.1966.19.1.3   DOI
8 Jin-hong Kim and Bo-ram Choi, "Intra- and Inter-rater Reliability of BioRescue," 한국콘텐츠학회논문지, 제18권, 제11호, pp.348-352, 2018.   DOI
9 고현선, 오준석, 최금진, 박길문, "조선대학교 기계공학프로그램 학습성과 평가 사례 연구," 공학교육연구, 제16권, 제6호, pp.65-77, 2013.
10 김해진, 천의영, 김은경, "기본간호학실습에서 학습성과 평가도구의 적용 사례: 의사소통능력 평가를 중심으로," 한국산학기술학회 논문지, 제21권, 제4호, pp.173-180, 2020.