• Annual Conference on Human and Language Technology
• /
• 한국정보과학회언어공학연구회 2012년도 제24회 한글 및 한국어 정보처리 학술대회
• /
• pp.69-72
• /
• 2012

본 논문은 한국어 서답형(단어, 구 수준) 문항 유형을 분석하고 실제 채점자가 채점 기준표를 보고 채점하는 방법을 컴퓨터가 인식할 수 있도록 정답 템플릿을 설계 및 개념 정의를 하여 한국어 서답형에 특화된 자동채점 시스템 방법을 제시한다. 본 시스템을 사용하여 1000개의 학생 답안지에 대한 유형 가지수 500개 이하의 2011년도 학업성취도 평가 과학 6개 문항에 대하여 채점 기준표 내용을 정답 템플릿으로 작성한 뒤 250개 학생 답안을 학습데이터로, 정답 템플릿을 업데이트로 사용, 750개 학생 답안에 대하여 자동채점한 결과, 평균 카파계수 0.84라는 수치로서 실제 사람 채점 결과와 거의 완벽히 일치라는 결과를 얻었다.

• Journal of the Korean earth science society
• /
• 제41권3호
• /
• pp.291-306
• /
• 2020

In this study, we developed a learning progression for the structure of the solar system using multi-tier supply form items and validated its appropriateness. To this end, by applying Wilson's (2005) construct modeling approach, we set up 'solar system components,' 'size and distance pattern of solar system planets,' and 'solar system modeling' as the progress variables of the learning progression and constructed multi-tier supply form items for each of these variables. The items were applied to 150 fifth graders before and after the classes that dealt with the 'solar system and star' unit. To describe the results of the assessment, the students' responses to each item were categorized into five levels. By analyzing the Wright map that was created by applying the partial credit Rasch model, we validated the appropriateness of the learning progression based on the students' responses. In addition, the validity of the hypothetical pathway that was established in the learning progression was verified by tracking changes in the developmental level of students before and after the classes. The results of the research are as follows. The bottom-up research method that used multi-tier supply form items was able to elaborately set the empirical learning progression for the conceptualization of the structure of the solar system that is taught in elementary school. In addition, the validity of the learning progression was high, and the development of students was found to change with the learning progression.

• KIPS Transactions on Software and Data Engineering
• /
• 제5권11호
• /
• pp.527-534
• /
• 2016

The emergent information society requires the talent for creative thinking based on problem-solving skills and comprehensive thinking rather than simple memorization. Therefore, the Korean curriculum has also changed into the direction of the creative thinking through increasing short-answer questions that can determine the overall thinking of the students. However, their scoring results are a little bit inconsistency because scoring short-answer questions depends on the subjective scoring of human raters. In order to alleviate this point, an automated scoring system using a machine learning has been used as a scoring tool in overseas. Linguistically, Korean and English is totally different in the structure of the sentences. Thus, the automated scoring system used in English cannot be applied to Korean. In this paper, we introduce an automated scoring system for Korean short-answer questions using predictability and unanimity. We also verify the practicality of the automatic scoring system through the correlation coefficient between the results of the automated scoring system and those of human raters. In the experiment of this paper, the proposed system is evaluated for constructed-response items of Korean language, social studies, and science in the National Assessment of Educational Achievement. The analysis was used Pearson correlation coefficients and Kappa coefficient. Results of the experiment had showed a strong positive correlation with all the correlation coefficients at 0.7 or higher. Thus, the scoring results of the proposed scoring system are similar to those of human raters. Therefore, the automated scoring system should be found to be useful as a scoring tool.

• Journal of Science Education
• /
• 제38권3호
• /
• pp.670-690
• /
• 2014

The purpose of this study was to analyze paper and pencil test items of life science I in high school to diagnose problems of the test items developed by teachers, and to provide some implication for better assessment. 690 selection-type items and 162 supply-type items in life science I were collected from 10 general high schools. In the analysis of test items, the ratio of the selection-type item and the supply-type item was 81:19 in the number of items based on item type, while the ratio was 74.4:25.6 in the distribution of marks, indicating that the distribution of marks compared to the number of items was higher in the supply-type items. In the analysis by the Bloom's revised taxonomy of educational objectives, the items of 'conceptual knowledge' in the knowledge and those of 'understanding' in the cognition process were shown most in both the selection-type item and the supply-type item. In the analysis by the science assessment frameworks of NAEA, the items of 'knowledge' were shown 9 times more than those of 'inquiry'. When compared to the level of difficulty presented in the two-way specification table and the percentage of correct answers in the selection-type item, the concurrence was 41.5%. When compared to the ratio of number of items based on the item type of the supply-type items, the short-answer items were 34.0%, the descriptive items were 61.1%, and the drawing items were 4.9%. The drawing items were mainly developed in the unit of 'Cells and Continuity of Life'. When the descriptive items were classified by the acceptance of response, all the items were 'response restricted' type, and the items of 'restricted in content range' type among them were highest. When the items were classified by presentation of data, the items of 'presentation of data' type were highest(65.4%), and when classified by type of question, the items of 'knowledge description' type were highest(80.4%). In conclusion, it is needed to develop items belonging to 'inquiry' area more in the school, and to increase the ratio of the descriptive items, presenting various types of items.

• Korean Journal of Cognitive Science
• /
• 제26권2호
• /
• pp.147-165
• /
• 2015

Through short-answer questions, we can reflect the depth of students' understanding and higher-order thinking skills. Scoring for short-answer questions may take long time and may be an issue on consistency of grading. To alleviate such the suffering, automated scoring systems are widely used in Europe and America, but are in the initial stage in research in Korea. In this paper, we propose a semi-automatic scoring system for short Korean free-text responses using semi-supervised learning. First of all, based on the similarity score between students' answers and model answers, the proposed system grades students' answers and the scored answers with high reliability have been included in the model answers through the thorough test. This process repeats until all answers are scored. The proposed system is used experimentally in Korean and social studies in Nationwide Scholastic Achievement Test. We have confirmed that the processing time and the consistency of grades are promisingly improved. Using the system, various assessment methods have got to be developed and comparative studies need to be performed before applying to school fields.

• Journal of the Korean earth science society
• /
• 제39권1호
• /
• pp.103-116
• /
• 2018

The purpose of this study was to explore $6^{th}$ graders learning progression for lunar phase change focusing astronomical systems thinking. By analyzing the results of previous studies, we developed the constructed-response items, set up the hypothetical learning progressions, and developed the item analysis framework based on the hypothetical learning progressions. Before and after the instruction on the lunar phase change, we collected test data using the constructed-response items. The results of the assessment were used to validate the hypothetical learning progression. Through this, we were able to explore the learning progression of the earth-moon system in a bottom-up. As a result of the study, elementary students seemed to have difficulty in the transformation between the earth-based perspective and the space-based perspective. In addition, based on the elementary school students' learning progression on lunar phase change, we concluded that the concept of the lunar phase change was a bit difficult for elementary students to learn in elementary science curriculum.

• Journal of the Korean Geographical Society
• /
• 제49권4호
• /
• pp.615-632
• /
• 2014

This study aims to explore 'bluffing', one of the most important topics in order to ensure the objectivity, validity, reliability of scoring of constructed-response items. The author identifies the conception of bluffing, and classifies major types of bluffing on the basis of previous studies on the theoretical level. Next, the author analyzes empirically the bluffing strategies and types of learners on key terms of Korean Geography subject. Compared with the existing research reports, the result of this study shows a significantly lower average bluffing score. On the other hand, it is consistent in results of previous studies reported that average bluffing score is similar between genders and that those students who got highest grades did least bluffing. Actually bluffing types are classified into four categories: 'repeating the question' type, 'depending on a well-known or existing knowledge' type, 'applying some basic concepts regardless of understanding' type, and 'inducing scorer's basic beliefs' type. Some comments and suggestions are as follows. First, it is necessary to continue researches of the relations among bluffing and age, gender, grade, intelligence and learning styles. Second, it is required for scorers who score constructed-response items to develop and supply the scoring guide including analysis contents of bluffing types and cases, and increase opportunity for training. Third, we need to inquire the domain-specific bluffing types in geography subject based on the generalizable sample size.

• Annual Conference on Human and Language Technology
• /
• 한국정보과학회언어공학연구회 2021년도 제33회 한글 및 한국어 정보처리 학술대회
• /
• pp.384-390
• /
• 2021

본 논문은 영어 에세이 자동 평가를 위한 딥러닝 기반의 새로운 평가 방법론을 제안한다. 어휘, 형태소, 구문, 의미 단계로 이루어진 평가 과정을 통해 자동화된 에세이 평가가 가능하다. 제안하는 방법의 객관성과 신뢰성을 검증하기 위하여 사람이 평가한 점수와 각 단계별 점수 사이의 상관관계 분석을 진행하였으며, 그 결과 제안하는 평가 방법이 유의미함을 알 수 있었다.

• Journal of the Korean earth science society
• /
• 제41권6호
• /
• pp.684-694
• /
• 2020

In this study, we developed a rubric for assessing middle school students' conceptual understanding about dew point. For this purpose, we analyzed 9th grade students' responses collected by using a multi-tiers constructed-response item of National Assessment of Educational Achievement (NAEA) and classified the types of the responses according to their characteristics. In addition, we analyzed the distribution of student response types according to mean achievement scores and developed an assessment rubric of conceptual understanding about dew point. The findings are as follows: First, the analysis of student responses to finding dew point in the saturation curve showed that many students had no or lack understanding of the scientific concept of dew point. Second, as a result of analyzing the student response to the water vapor condensation process at dew point, the proportion of scientific conception types was very low, while the proportion of misconception types was relatively high and the types varied as well. Third, a four- level assessment rubric was developed based on the analysis of the distribution of student response types according to the mean achievement scores. Based on the findings, we suggested the development and utilization of assessment rubric in the field of Earth science education.

• Communications of Mathematical Education
• /
• 제26권1호
• /
• pp.29-50
• /
• 2012

The purpose of this study is a comparison of proficient level and below basic level students' mathematical achievement in the National Achievement Evaluation and Assessment(NAEA). For the purpose, this study compared the proficient level and below basic level students' ratios, students' mathematical achievement of contents area and behavioral area in 6th, 9th, and 11th grades. This study found the change of proficient level and below basic level students' ratios, and the proficient level and below basic level students' characteristics on mathematical achievement of contents area and behavioral area in 6th, 9th, and 11th grades.