• Science of Emotion and Sensibility
• /
• v.11 no.4
• /
• pp.553-564
• /
• 2008

Generalizability theory was employed to examine how the reliability of polygraph test is affected by the number of relevant questions, the number of repeated tests (the number of of charts), and the number of raters(scorers). The data consisted of the results of the polygraph tests administered to 31 crime suspects. The sample was drawn from the real polygraph tests based on Backster ZCT and archived by the Prosecutor's Office of the Republic of Korea. The numerical scores assigned by thirteen raters to the test charts were analyzed to determine the generalizability of the scores. The largest variance component was accounted for by the examinee factor(43.97%) and the residual variance component was 16.84% of the total variance. The variance component due to the interaction between the examinee and the chart factors was 12.17% and the variance component due to the three way interaction of the examinee, the repeated test, and the relevant question factors was 10.31%. The generalizability coefficient for the current measurement procedure as practiced by the Korean Prosecutor's Office was 0.74 which suggests that the current procedure is acceptable. However, measurement procedures with the combination of more than two relevant questions, more than three repeated tests, and more than two raters were generally found to yield generalizability coefficients larger than 0.80. Therefore, such procedures need to be considered seriously in order to significantly improve the reliability of polygraph test.

• Communications for Statistical Applications and Methods
• /
• v.1 no.1
• /
• pp.206-214
• /
• 1994

본 논문은 1994학년도 부터 부활된 대학별고사 실시에 따른 문항분석, 표준 점수제 그리고 검사동등화의 문제점을 지적하기 위하여 교육측정이론의 기본 개념을 소개하는데 있다. 대학별고사의 타당성과 신뢰성을 보장받기 위하여는 양질의 문항제작이 우선하여야하며, 이를 위하여 문항분석은 종전에 사용하던 고전검사이론 보다는 문항반응이론을 이용하는 것이 바람직하다. 문항반응이론에 의한 문항분석은 피험자 집단의 특성에 의하여 문항특성이 달리 분석되지 않는 특징을 지니고 있기 때문이다. 문항이 논술형일 경우 채점자간 신뢰도와 채점자 내 신뢰도를 간과하여서는 안될 것이다. 다양한 선택과목을 채택하는 대학별 고사에서 입학 사정을 위하여 원점수를 사용하거나, 표준점수 혹은 검사동등화 방법을 이용하고 있으나 이는 교육측정이론에 위배된다. 다른 과목에 대한 인가의 능력을 상대비교 할 수 없으며, 표준점수와 검사동등화는 동일 능력에 대한 상대비교를 위한 방법이다. 특히 검사동등화는 동일 특성, 공정성, 모교집단 불변성, 대칭성을 전제한다. 표준점수제에 의하여 수험생들의 다른 능력을 상대 비교하는 방법은 다른 능력이 점수로 표현되기 때문에 가능하나 그 점수가 무엇을 의미하는 가를 분석할 때는 교육평가의 기본 철학에도 위배된다.

• Journal of Korean Home Economics Education Association
• /
• v.32 no.3
• /
• pp.135-159
• /
• 2020

The purpose of this study was to develop items of an essay response assessment that could align with the instructions and assessments in the high school home economics curriculum. The contents of the study were as follows. First, to establish an assessment plan, 14 achievement standards were analyzed in the assessment area, and the elements of the questions were developed including the content elements of a total of 29 questions. Second, to develop the assessment tools, preliminary questions suited to the structure of essay questions were developed, and the method of presenting data and scoring criteria to be utilized in the questions was selected. Third, to prepare the answers and the scoring criteria tables, the answers to the sample questions for each score were prepared in form of a scoring criteria table, and the objectives of the assessment, the scoring items, and the scores for each item were reviewed. Fourth, the developed questions and answers were revised and supplemented by teachers of the professional learning community through preliminary and mutual review on the components of the questions, the embodiment of the assessment objectives, the implementation of the assessment intent, and the grading. This study can be used as a foundational study for the development of essay-type questions and scoring criteria in essay assessment in the field of education. Furthermore, the results of this study could help teachers enhance their learners' ability to apply knowledge in the future.

• KIPS Transactions on Software and Data Engineering
• /
• v.4 no.1
• /
• pp.45-56
• /
• 2015

In this paper, we present a method that can detect context-sensitive word errors and generate correction candidates. Spelling error detection is one of the most widespread research topics, however, the approach proposed in this paper is adjusted for an automated English scoring system. A common strategy in context-sensitive word error detection is using a pre-defined confusion set to generate correction candidates. We automatically generate a confusion set in order to consider the characteristics of sentences written by second-language learners. We define a word error that cannot be detected by a conventional grammar checker because of part-of-speech ambiguity, and propose how to detect the error and generate correction candidates for this kind of error. An experiment is performed on the English writings composed by junior-high school students whose mother tongue is Korean. The f1 value of the proposed method is 70.48%, which shows that our method is promising comparing to the current-state-of-the art.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2013.11a
• /
• pp.545-547
• /
• 2013

최근 대학 강의가 대형화 함에 따라 기존 교수방법으로는 출석체크, 퀴즈, 시험 등의 평가과정에서 발생하는 오버헤드를 피하기 어렵게 되었다. 교수는 대형강의에서 시행하는 퀴즈/시험 등에 따른 채점/평가에 대해서 상당한 부담감을 느끼며, 강의 중 교수-학생 상호작용/feedback 등을 시도하기도 어려운 상황이다. 이러한 문제를 해결하기 위해서, 본 연구에서는 대형강의 추세에 따른 적합한 교수법/교수지원도구(tool) 연구를 수행하였다. 대형강의에서 퀴즈/시험의 채점 및 평가에 소요되는 오버헤드를 감소시키는 방법을 고안하며, 대형강의에서 스마트폰을 활용하여 학생들과 실시간으로 상호작용할 수 있는 도구를 개발하였다. 특히, 본 연구에서는 HTML5 기술을 활용함으로써, 학생들이 소지한 스마트폰의 플랫폼에 독립적으로 설치과정의 번거로움 없이 활용할 수 있도록 하였다.

• School Mathematics
• /
• v.5 no.1
• /
• pp.1-25
• /
• 2003

This study develops a performance assessment system to improve grade 4-9 students' creativity. First, this study discusses its educational meaning, its task types, scoring methods and practical application methods. Then, this study provides the typical examples of performance assessment task classified by each of the types and its scoring rubrics. Finally, this study analyzes students' achievement levels for each task. Each task includes item informations such as content area, evaluation goal, evaluation procedure, preparatory material, characteristics, considering points, scoring rubric etc. for grade 4-9 teachers to use them in their evaluation processes directly without difficulties.

• The Mathematical Education
• /
• v.63 no.2
• /
• pp.165-186
• /
• 2024

Despite the growing attention on artificial intelligence-based automated scoring technology as a support method for the introduction of descriptive items in school environments and large-scale assessments, there is a noticeable lack of foundational research in mathematics compared to other subjects. This study developed an automated scoring model for two descriptive items in first-year middle school mathematics using the Random Forest algorithm, evaluated its performance, and explored ways to enhance this performance. The accuracy of the final models for the two items was found to be between 0.95 to 1.00 and 0.73 to 0.89, respectively, which is relatively high compared to automated scoring models in other subjects. We discovered that the strategic selection of the number of evaluation categories, taking into account the amount of data, is crucial for the effective development and performance of automated scoring models. Additionally, text preprocessing by mathematics education experts proved effective in improving both the performance and interpretability of the automated scoring model. Selecting a vectorization method that matches the characteristics of the items and data was identified as one way to enhance model performance. Furthermore, we confirmed that oversampling is a useful method to supplement performance in situations where practical limitations hinder balanced data collection. To enhance educational utility, further research is needed on how to utilize feature importance derived from the Random Forest-based automated scoring model to generate useful information for teaching and learning, such as feedback. This study is significant as foundational research in the field of mathematics descriptive automatic scoring, and there is a need for various subsequent studies through close collaboration between AI experts and math education experts.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.12 no.1
• /
• pp.318-326
• /
• 2011

The purpose of this study is to investigate whether standardized patient(SP) can be used as a reliable examiner in Objective Structured Clinical Examination(OSCE). 4 SPs and 4 faculties who have more than 2 years experience of OSCE scoring were selected. For 1 assignment 2 members of faculty and 2 SPs were designated as raters. SPs were educated for assessing 2 technical skills, male Foley catheter insertion and wound dressing, for 8 hours (4 hours / day, each topic). The definition, method, cautions and complications for each of procedural skills were covered in the education. Theoretical lectures, video learning, faculty demonstration and practical training on mannequins were employed. The 8 raters were standardized for an hour with simulated OSCE scoring using previous videos on the day before the OSCE. Each assessment was composed of 14 checklists and 1 global rate. The allotted time for each assignment was 5minutes and for evaluation time 2 minutes per student. The evaluation from the faculty and SPs were compared and analyzed with the GENOVA program. The overall generalizability coefficient (G coefficient) was 0.839 from two cases of OASTS. The reliability of the raters was high, 0.946. The inter-rater agreement between faculty group and SP group was 0.949 for checklist and 0.908 for global rating. Therefore SPs can play a role of raters in OSCE for procedural skills, if they are given the appropriate training.

• The KIPS Transactions:PartA
• /
• v.12A no.6 s.96
• /
• pp.539-546
• /
• 2005

Recently, as the number of Internet users are growing explosively, e-learning has been applied spread, as well as remote evaluation of intellectual capacity However, only the multiple choice and/or the objective tests have been applied to the e-learning, because of difficulty of natural language processing. For the intelligent marking of short-essay typed answer papers with rapidness and fairness, this work utilize heterogenous linguistic knowledges. Firstly, we construct the semantic kernel from un tagged corpus. Then the answer papers of students and instructors are transformed into the vector form. Finally, we evaluate the similarity between the papers by using the semantic kernel and decide whether the answer paper is correct or not, based on the similarity values. For the construction of the semantic kernel, we used latent semantic analysis based on the vector space model. Further we try to reduce the problem of information shortage, by integrating Korean Word Net. For the construction of the semantic kernel we collected 38,727 newspaper articles and extracted 75,175 indexed terms. In the experiment, about 0.894 correlation coefficient value, between the marking results from this system and the human instructors, was acquired.

• Proceedings of the Korea Multimedia Society Conference
• /
• 1998.04a
• /
• pp.296-301
• /
• 1998

본 논문에서는 클라이언트/서버 환경에서 교사가 평가 문제를 스스로 관리하는 동시에, 웹 상에서 학생들에게 학습 내용을 테스트하여 학생들의 학습 능력 수준을 손쉽게 파악할 수 있도록 하는 대화형 학생 평가 시스템을 개발하였다. 본 논문에서 구현한 시스템은 과목별 채점 결과를 학급별과 개인별로 통계 처리하여 보여주며, 또한 과목 주제별로 정답 비율을 통계 처리함으로써 학생 개개인 수준을 분석하는데 편리하도록 설계되어 있다. 또한 학생들에게는 웹 상에서 푼 문제에 대한 자신의 채점 결과와 문제 풀이 부분을 즉시 볼 수 있도록 하여 재학습 및 보충학습의 효과를 얻을 수 있도록 하였다. 이로써 교사는 학생 수준에 적합한 교수·학습 방법으로 학생들을 개별 지도할 수 있어 기존의 교사 주도적이고 획일화된 주입식 교육 방식의 문제점을 해결하고자 하였으며, 학생 역시 자신의 수준에 적합한 교육 내용을 제공받을 수 있어 학습 의욕을 향상시킬 수 있다.