• Journal of The Korean Association For Science Education
• /
• v.30 no.2
• /
• pp.275-290
• /
• 2010

In this study, reliabilities of holistic scoring method and analytic scoring method were analyzed in performance assessments of middle school students' science investigation activity. Reliabilities of 2, 3, and 4~7-level rubric ratings for analytic scoring methods were compared to figure out optimized numbers of rubric ratings. Two trained raters rated four activity sheets of 60 students by two rating methods and three kinds of rubric ratings. Internal consistency reliabilities of holistic scoring methods were higher than those of analytic scoring methods, while intrarater reliabilities of analytic scoring were higher than those of holistic scoring methods. Internal consistency reliabilities and intra-rater reliabilities of 3-level rubric rating showed similar patterns of 4~7-level rubric ratings. But students' discriminations, item difficulties and item-response curves showed that the 3-level rubric ratings was reliable. These results suggest that holistic scoring method could be adapted to increase internal consistency reliabilities with improvement in intra-rater reliabilities by rater's conferences. Also, the 3-level rubric rating would be enough for good reliability in case of adapting analytic scoring methods.

• Annual Conference on Human and Language Technology
• /
• 2012.10a
• /
• pp.121-122
• /
• 2012

컴퓨터를 사용한 서술형 답안의 자동채점은 채점의 편의성과 객관성을 제고하기 위하여 많은 연구자들이 연구해 왔으며 자동채점의 성능을 향상시키기 위해 여러 가지 방법들이 제안되었다. 본 논문은 서술어 정규화를 통하여 서술형 답안의 자동채점 정확도를 높이고자 하였다. 기존의 다른 채점 방법들과 비교했을때 서술어 정규화 기법을 적용한 채점 방식은 기존의 방법들보다 유사도 계산 정확도가 향상되어 정답 판별 정확도가 향상되는 것을 확인할 수 있었다. 서술어 정규화는 기존의 모든 서술형 답안 채점 방법에 추가적으로 적용할 수 있는 범용성을 가지고 있다. 따라서 서술어 정규화는 기존 방법들의 자동채점 정확도를 향상시켜 보다 정확하게 서술형 답안을 채점할 수 있다.

• The Journal of Korean Association of Computer Education
• /
• v.10 no.2
• /
• pp.89-100
• /
• 2007

Scoring the constructed responses in large-scale assessments needs great efforts and time to reduce the various types of error in Paper-based training and scoring. For the purpose of eliminating the complexities and problems in Paper and pencil based training and scoring, many of countries including U.S.A and England already have applied online scoring system. There, however, has been few studies to develop the scoring system for the constructed responses items in Korea. The purpose of this study is to develop the basic design of the Internet-based scoring system for the constructed responses. This study suggested the algorithms for assigning scorers to constructed responses, employing methods for monitoring reliability, etc. This system can ensure reliable, quick scoring such as monitor scorer consistency through ongoing reliability checks and assess the quality of scorer decision making through frequent various checking procedures.

• Journal of The Korean Association For Science Education
• /
• v.25 no.2
• /
• pp.173-183
• /
• 2005

In this study, we questioned what assessment types and scoring methods of science performance assessment(SPA) were being used in middle and high school, and how much these SPA scores were reliable(generalizable). To answer these questions, SPA data obtained from the seven schools were classified according to assessment type and scoring method. Based upon this classification, we analyzed the reliability by applying generalizability theory. The result, from the classification of assessment type and scoring method, showed that SPA types of the seven schools were divided into two types: paper-pencil type and task type. Paper-pencil type included answer(content)-restricted essay-type test solely. Task type has two parts: process and outcome assessment. As the results of analyzing scoring methods of the seven schools, there were two cases in the way of scoring methods: one case is scoring all essay-type items and performance tasks by one teacher, the other is scoring assigned performance tasks by two teachers. But the case of scoring assigned essay-type items or the case of cross scoring by two or more teachers were not found. The findings of the reliability analysis are as follows: (1) Effect of essay-type item to SPA score was larger than that of performance task. (2) There was remarkable difference among the seven schools' interaction effect of person and rater in scoring performance tasks. (3) Most of generalizability(reliability) coefficients of SPA for the seven schools were smaller than the acceptable generalizability coefficient(0.80). Therefore, the population of statistical parameters such as number of item, task and rater, should be increased for approaching the acceptable generalizability level.

• The Journal of Korean Association of Computer Education
• /
• v.17 no.2
• /
• pp.39-51
• /
• 2014

In high-stakes tests for large groups, the efficiency with which students' responses are distributed to raters and how systematic scoring procedures are managed is important to the overall success of the testing program. In the scoring of constructed response items, it is important to understand whether the raters themselves are making consistent judgments on the responses, and whether these judgments are similar across all raters in order to establish measures of rater reliability. The purpose of this study was to design, develop and carry out a pilot test of an online scoring system for constructed response items administered in a paper-and-pencil test to large groups, and to verify the system's reliability. In this study, we show that this online system provided information on the scoring process of individual raters, including intra-rater and inter-rater consistency, compared to conventional scoring methods. We found this system to be especially effective for obtaining reliable and valid scores for constructed response items.

• Communications of Mathematical Education
• /
• v.2
• /
• pp.171-179
• /
• 1997

• 한국HCI학회:학술대회논문집
• /
• 2009.02a
• /
• pp.606-610
• /
• 2009

We propose a method utilizing BLEU(BiLingual Evaluation Understudy), which is widely used in automatic evaluation of machine translations, for an autonomous assessment of a short essay answer. BLEU evaluates translations with an assumption that the translation by a machine is supposed to be more accurate as it is getting to be more similar to the translation by a human. BLEU scores the translation by comparing the n-grams of translations by a machine and humans. Similarly we score students answers by comparing to multiple reference answers with BLEU. In the experiment, we compute correlation coefficient values between scores of our system and human instructors.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2005.05a
• /
• pp.1027-1030
• /
• 2005

기존 의미커널을 적용한 주관식 채점 시스템은 여러 답안과 말뭉치에서 추출한 색인어들과의 상관관계를 벡터방식으로 표현하여 자연어 처리에 대한 문제를 해결하려 하였다. 본 논문에서는 기존 시스템의 답안 및 색인어의 표현 한계로 인한 유사도 계산오차 가능성에 대한 문제를 해결하고자 시소러스를 이용한 임의 추출 방식의 답안 확장을 적용하였다. 서술형 주관식 평가에서는 문장의 문맥보다는 사용된 어휘에 채점가중치가 높다는 점을 착안, 출제자와 수험자 모두의 답안을 동의어, 유의어 그룹으로 확장하여 채점 성능을 향상시키려 하였다. 우선 두 답안을 형태소 분석기를 이용해 색인어를 추출한 후 워드넷을 이용하여 동의어, 유의어 그룹으로 확장한다. 이들을 말뭉치 색인을 이용하여 단어들 간 상관관계를 측정하기 위한 벡터로 구성하고 의미 커널을 적용하여 정답 유사도를 계산하였다. 출제자의 채점결과와 각 모델의 채점 점수의 상관계수 계산 결과 ELSA 모델이 가장 높은 유사도를 나타내었다..

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2013.07a
• /
• pp.227-228
• /
• 2013

대학에서 이루어진 강의에 대한 강의 지원시스템은 오래전부터 개발되어 사용되고 있으나 완벽한 강의지원이란 쉬운 일이 아니다. 따라서 본 시스템은 대학 강의에 적합한 온라인 강의지원 관리 시스템을 개발하는데 그 목적이 있다. 본 개발 시스템은 대학에서의 데이터베이스과목 관리에 목표를 두고 주로 시험과 채점관리에 초점을 맞추어 개발되었다. 강의관리를 위한 출석관리, 리포트관리, 퀴즈와 같은 수시시험관리 및 정규(중간, 기말)고사의 실시 및 채점을 주목적으로 하였다. 시험의 경우 객관식, 주관식, 단답식 그리고 SQL로 나누어 개발되었다. 특히 SQL의 경우 구분분석을 통해 채점할 수 있는 시스템을 개발함으로서 보다 효과적인 채점관리에 주력하였다. 주관식 및 단답형의 경우는 수작업을 통한 채점방식을 사용하였으며, 이 모든 과정을 학생 스스로 확인할 수 있도록 하여 채점과 관련한 문제를 해소하도록 하였다.

• Journal of the Korea Society of Computer and Information
• /
• v.19 no.12
• /
• pp.267-274
• /
• 2014

Even though computer based college lecture management system was developed long ago and has been used ever since, developing perfect lecture management is not simple. The main objective of this system is to develop appropriate online lecture supportive management program suitable for college lectures. This system [ADLEMS] mainly focuses on the management of college database lectures, exams, and grades. This system supports management and grading of attendance, reports, quizzes, mid-term, and final exams. Exam management categorizes into multiple choice questions, essay questions, short answer questions, and SQL. Especially for SQL, division analysis was applied when developing grading system for more effective grade management. For essay questions and short answer questions, manual [hand] grading method was used. Every student can verify the grading process in person to alleviate the problems occurring during the grading process.