• Journal of the Korean Geographical Society
• /
• v.47 no.6
• /
• pp.934-954
• /
• 2012

Constructed-response items are useful to estimate a higher-order thinking abilities such as sense of place, graphicacy, and geographical imaginations which are aimed by geography education. This study aims to identify the conception of the constructed-response items, to classify those feasible items in geography education, and to propose some comments and suggestions for development and application in Korea. The author tries to classify the constructed-response items by analyzing the NAEP items in the formal aspect and the aspect of cognitive dimension respectively. In the formal aspect, they are classified as 'question-standalone' type and 'material-presenting' type. In the aspect of cognitive dimension, they are classified into three types as 'knowledge-requirement', 'understanding-requirement', and 'applying-requirement'. Some comments and suggestions are as follows. First, it is necessary to develope the constructed-response items that coherent to both content and cognitive dimensions. Second, it is required for students who take an examination to use a great variety of graphics, photos, thematic maps related to geography. Third, we need to inquire the rubric, grade, process to scoring because they are vital to success of the constructed-response items.

• The Journal of Korean Association of Computer Education
• /
• v.17 no.2
• /
• pp.39-51
• /
• 2014

In high-stakes tests for large groups, the efficiency with which students' responses are distributed to raters and how systematic scoring procedures are managed is important to the overall success of the testing program. In the scoring of constructed response items, it is important to understand whether the raters themselves are making consistent judgments on the responses, and whether these judgments are similar across all raters in order to establish measures of rater reliability. The purpose of this study was to design, develop and carry out a pilot test of an online scoring system for constructed response items administered in a paper-and-pencil test to large groups, and to verify the system's reliability. In this study, we show that this online system provided information on the scoring process of individual raters, including intra-rater and inter-rater consistency, compared to conventional scoring methods. We found this system to be especially effective for obtaining reliable and valid scores for constructed response items.

• The Journal of Korean Association of Computer Education
• /
• v.10 no.2
• /
• pp.89-100
• /
• 2007

Scoring the constructed responses in large-scale assessments needs great efforts and time to reduce the various types of error in Paper-based training and scoring. For the purpose of eliminating the complexities and problems in Paper and pencil based training and scoring, many of countries including U.S.A and England already have applied online scoring system. There, however, has been few studies to develop the scoring system for the constructed responses items in Korea. The purpose of this study is to develop the basic design of the Internet-based scoring system for the constructed responses. This study suggested the algorithms for assigning scorers to constructed responses, employing methods for monitoring reliability, etc. This system can ensure reliable, quick scoring such as monitor scorer consistency through ongoing reliability checks and assess the quality of scorer decision making through frequent various checking procedures.

• 한국어정보학회:학술대회논문집
• /
• 2016.10a
• /
• pp.181-185
• /
• 2016

최근 교육과정에서 학생들의 능력 평가는 단순 암기보다 학생들의 종합적인 사고력을 판단할 수 있는 서답형 문항을 늘리는 방향으로 변하고 있다. 그러나 서답형 문항의 경우 채점하는 데 시간과 비용이 많이 들고, 채점자의 주관에 따라 채점 결과의 일관성과 신뢰성을 보장하기 어렵다는 문제가 있다. 이런 점을 해결하기 위해 해외의 사례를 참고하여 국내에서도 서답형 문항에 자동채점 시스템을 적용하는 연구를 진행하고 있다. 본 논문에서는 2014년도에 개발된 '한국어 문장 수준 서답형 문항 자동채점 시스템'의 성능분석을 바탕으로 언어 처리 기능과 자동채점 성능을 개선한 2015년도 자동채점 시스템을 간략하게 소개하고, 각 자동채점 시스템의 성능을 비교 분석한다. 성능 분석 대상으로는 2014년도 국가수준 학업성취도평가의 서답형 문항을 사용했다. 실험 결과, 개선한 시스템의 평균 완전 일치도와 평균 정확률이 기존의 시스템보다 각각 9.4%p, 8.9%p 증가했다. 자동채점 시스템의 목적은 가능한 채점 시간을 단축하면서 채점 기준의 일관성과 신뢰성을 확보하는 데 있으므로, 보완한 2015년 자동채점 시스템의 성능이 향상되었다고 판단할 수 있다.

• Annual Conference on Human and Language Technology
• /
• 2016.10a
• /
• pp.181-185
• /
• 2016

최근 교육과정에서 학생들의 능력 평가는 단순 암기보다 학생들의 종합적인 사고력을 판단할 수 있는 서답형 문항을 늘리는 방향으로 변하고 있다. 그러나 서답형 문항의 경우 채점하는 데 시간과 비용이 많이 들고, 채점자의 주관에 따라 채점 결과의 일관성과 신뢰성을 보장하기 어렵다는 문제가 있다. 이런 점을 해결하기 위해 해외의 사례를 참고하여 국내에서도 서답형 문항에 자동채점 시스템을 적용하는 연구를 진행하고 있다. 본 논문에서는 2014년도에 개발된 '한국어 문장 수준 서답형 문항 자동채점 시스템'의 성능분석을 바탕으로 언어 처리 기능과 자동채점 성능을 개선한 2015년도 자동채점 시스템을 간략하게 소개하고, 각 자동채점 시스템의 성능을 비교 분석한다. 성능 분석 대상으로는 2014년도 국가수준 학업성취도평가의 서답형 문항을 사용했다. 실험 결과, 개선한 시스템의 평균 완전 일치도와 평균 정확률이 기존의 시스템보다 각각 9.4%p, 8.9%p 증가했다. 자동채점 시스템의 목적은 가능한 채점 시간을 단축하면서 채점 기준의 일관성과 신뢰성을 확보하는 데 있으므로, 보완한 2015년 자동채점 시스템의 성능이 향상되었다고 판단할 수 있다.

• Journal of Educational Research in Mathematics
• /
• v.25 no.3
• /
• pp.281-302
• /
• 2015

In the current study, answer sheets from 8007 students in 236 Korean schools were selected and analyzed to examine errors that emerge in the process of solving descriptive questions of the National Educational Achievement Assessment in mathematics. Questions used in the analysis were response assessment covering middle school mathematics topics: "mathematical symbols and equations" and "functions." The behavioral domain of the questions was that of "problem solving and computation," which requires establishing an equation for a word problem and allows the calculation of an answer that meets a certain condition. The analysis results revealed various errors in each stage of each question, from understanding to solving; the study attempts to conjecture causes for these errors and draw pedagogical implications.

• Annual Conference on Human and Language Technology
• /
• 2012.10a
• /
• pp.138-141
• /
• 2012

한국어 서답형 문항의 자동채점 프로그램을 개발하기 위해서는 모범답안, 오답, 부분점수 부여를 위한 세부적인 내용을 채점 기준표로 기술해야 한다. 자동채점에 필요한 구체적인 사항들을 기술하기 위하여 XML 형식으로 정답 템플릿을 정의하였다. 채점에 필요한 내용을 단위 개념으로 정의하고 이를 컴퓨터가 엑세스 가능한 형태의 정답 템플릿을 설계하였다. 정답 템플릿 형식에 맞게 편리하게 템플릿을 작성할 수 있는 작성 도구를 이용하여 학업 성취도평가 각 문항에 대한 채점 기준표를 정답 템플릿으로 작성하여 채점기준표를 작성하는 실험을 수행하였다.

• Journal of the Korean Chemical Society
• /
• v.65 no.1
• /
• pp.48-57
• /
• 2021

This study analyzed the science assessment items that were conducted by the paper and the computer-based assessment in the TIMSS 2019 pre-test, which is an international academic achievement assessment. We examined whether the test mode influenced the percentage of items answered correctly by each graders. As a result, there was no significant difference in the correct answer rates of science items according to the test media in 4th and 8th graders. Looking at the each graders, the difference in the percentage of correct answers was relatively large in the fourth grader rather than the eighth grader. In terms of items, the difference in the percentage of correct answers was relatively larger for the constructed items than the multiple choice questions. As a result of analyzing the content of science and assessment items on the influence of the test media, the multiple choice items showed little difference between the paper and the computer-based assessment items, but the answer-type items tended to have a lower percentage of correct answers than the paper-based assessment. In addition, by grade, 4th graders showed lower percentages of correct answers in the constructed items than 8th graders. This study provides implications related to the development and introduction of computer-based assessment in Korea, and suggests an educational implications for computer-based assessment to be well established as school evaluation.

• Journal of the Korean earth science society
• /
• v.41 no.3
• /
• pp.291-306
• /
• 2020

In this study, we developed a learning progression for the structure of the solar system using multi-tier supply form items and validated its appropriateness. To this end, by applying Wilson's (2005) construct modeling approach, we set up 'solar system components,' 'size and distance pattern of solar system planets,' and 'solar system modeling' as the progress variables of the learning progression and constructed multi-tier supply form items for each of these variables. The items were applied to 150 fifth graders before and after the classes that dealt with the 'solar system and star' unit. To describe the results of the assessment, the students' responses to each item were categorized into five levels. By analyzing the Wright map that was created by applying the partial credit Rasch model, we validated the appropriateness of the learning progression based on the students' responses. In addition, the validity of the hypothetical pathway that was established in the learning progression was verified by tracking changes in the developmental level of students before and after the classes. The results of the research are as follows. The bottom-up research method that used multi-tier supply form items was able to elaborately set the empirical learning progression for the conceptualization of the structure of the solar system that is taught in elementary school. In addition, the validity of the learning progression was high, and the development of students was found to change with the learning progression.

• Journal of Science Education
• /
• v.38 no.3
• /
• pp.670-690
• /
• 2014

The purpose of this study was to analyze paper and pencil test items of life science I in high school to diagnose problems of the test items developed by teachers, and to provide some implication for better assessment. 690 selection-type items and 162 supply-type items in life science I were collected from 10 general high schools. In the analysis of test items, the ratio of the selection-type item and the supply-type item was 81:19 in the number of items based on item type, while the ratio was 74.4:25.6 in the distribution of marks, indicating that the distribution of marks compared to the number of items was higher in the supply-type items. In the analysis by the Bloom's revised taxonomy of educational objectives, the items of 'conceptual knowledge' in the knowledge and those of 'understanding' in the cognition process were shown most in both the selection-type item and the supply-type item. In the analysis by the science assessment frameworks of NAEA, the items of 'knowledge' were shown 9 times more than those of 'inquiry'. When compared to the level of difficulty presented in the two-way specification table and the percentage of correct answers in the selection-type item, the concurrence was 41.5%. When compared to the ratio of number of items based on the item type of the supply-type items, the short-answer items were 34.0%, the descriptive items were 61.1%, and the drawing items were 4.9%. The drawing items were mainly developed in the unit of 'Cells and Continuity of Life'. When the descriptive items were classified by the acceptance of response, all the items were 'response restricted' type, and the items of 'restricted in content range' type among them were highest. When the items were classified by presentation of data, the items of 'presentation of data' type were highest(65.4%), and when classified by type of question, the items of 'knowledge description' type were highest(80.4%). In conclusion, it is needed to develop items belonging to 'inquiry' area more in the school, and to increase the ratio of the descriptive items, presenting various types of items.