• Journal of the Korean earth science society
• /
• v.31 no.1
• /
• pp.95-105
• /
• 2010

The purpose of this study lies in applying generalizability theory depending on the aim of the usage of GALT to analyze the sources of error of single-facet considering item and person only and to analyze the sources of error of multi-facet considering item, person and domain. The study was conducted with 1016 students of local elementary, middle, and high schools. The 21 items of a full version were answered for 40 minute and then the 12 items of short version were sampled to analyze reliability using generalizability theory. Both the full version and the short version of the items were analyzed using Cronbach's alpha for data analysis, and we applied generalizability theory and separate $p{\times}i$ design and $p{\times}(i:h)$ design, G study and D study were performed. Results of analysis are as follows: First, the result of D study after $p{\times}I$ design both on the full version and the short version showed that in the case of the full version, the generalizability coefficient was 0.87 exceeding a normal level of 0.80, and the normal level of generalizability coefficient was achieved in 13 items as well. In case of short version, when 12 items were evaluated, generalizability coefficient was 0.77 not reaching the normal level, and the normal level was achieved in case of more than 15 items. Second, the result of D study after $p{\times}(I:H)$ design on the short version showed that once one domain consists of 2 items in 6 domains, generalizability coefficient was 0.71 which is lower than the normal level of 0.80, the normal level was achieved in more than 5 item cases.

• Science of Emotion and Sensibility
• /
• v.11 no.4
• /
• pp.553-564
• /
• 2008

Generalizability theory was employed to examine how the reliability of polygraph test is affected by the number of relevant questions, the number of repeated tests (the number of of charts), and the number of raters(scorers). The data consisted of the results of the polygraph tests administered to 31 crime suspects. The sample was drawn from the real polygraph tests based on Backster ZCT and archived by the Prosecutor's Office of the Republic of Korea. The numerical scores assigned by thirteen raters to the test charts were analyzed to determine the generalizability of the scores. The largest variance component was accounted for by the examinee factor(43.97%) and the residual variance component was 16.84% of the total variance. The variance component due to the interaction between the examinee and the chart factors was 12.17% and the variance component due to the three way interaction of the examinee, the repeated test, and the relevant question factors was 10.31%. The generalizability coefficient for the current measurement procedure as practiced by the Korean Prosecutor's Office was 0.74 which suggests that the current procedure is acceptable. However, measurement procedures with the combination of more than two relevant questions, more than three repeated tests, and more than two raters were generally found to yield generalizability coefficients larger than 0.80. Therefore, such procedures need to be considered seriously in order to significantly improve the reliability of polygraph test.

• Journal of Science Education
• /
• v.47 no.2
• /
• pp.211-224
• /
• 2023

The purpose of this study is to suggest how to investigate the reliability of the assessment, which consists of items generated by automatic item generation using empirical example data. To achieve this, we analyzed the illustrative assessment data by applying the multivariate generalizability theory, which can reflect the design of responding to different items for each student and multiple error sources in the assessment score. The result of the G-study showed that, in most designs, the student effect corresponding to the true score of the classical test theory was relatively large after residual effects. In addition, in the design where the content domain was fixed, the ranking of students did not change depending on the item types or items. Similarly, in the design where the item format was fixed, the difficulty showed little variation depending on the content domains. The result of the D-study indicated that the original assessment data achieved a sufficient level of reliability. It was also found that higher reliability than the original assessment data could be obtained by reducing the number of items in the content domains of operation, geometry, and probability and statistics, or by assigning higher weights to the domains of letters and formulas, and function. The efficient measurement conditions presented in this study are limited to the illustrative assessment data. However, the method applied in this study can be utilized to determine the reliability and to find efficient measurement conditions for the various assessment situations using automatic item generation based on measurement traits.

• Journal of the Korean earth science society
• /
• v.26 no.6
• /
• pp.511-523
• /
• 2005

The purpose of this study is to analyze the 1st and 2nd Korea Earth Science Olympiad (KESO) items, in order to find informations to enhance item quality. To do this, internal and external item classification frameworks are developed. Item difficulty (P), discrimination index (DI), correlation, and reliability are estimated by using classical test theory. Generalizability is also estimated by applying the generalizability theory. The results of item classification are as follows: (1) ‘Geology’, ‘astronomy’ and ‘data analysis and interpretation’ are dominant in content and inquiry process domain, respectively. Nearly every item has textbook context. (2) There is no difference between the preliminary and final tests in terms of their thinking skills sections. (3) As a whole, the ratio of items with pictures is high in item representation. However, multiple-choice and short answer items are more common in preliminary competition, and essay type items are found more often in final competition. The ratio of simple items is high in middle school section and preliminary competition, but composite items are dominant in high school section and final competition. The findings of item analysis are as follows: (1) In the middle school section, P is low and DI is moderate. But in the high school section, there is a considerable differences between science high schools and other high schools in general. (2) The highest correlation is reported between the scores of meteorology domain and total score in middle school, whereas in high school astronomy domain and total score show the highest correlation. (3) General high school section show the highest Cronbach $\alpha$ and generalizability. (4) General high school section show acceptable generalizability coefficient (> 0.80), but middle and science high school section should increase the number of items to reach acceptable generalizability level.

• Journal of Gifted/Talented Education
• /
• v.23 no.5
• /
• pp.671-695
• /
• 2013

This study provides an illustrative example of using the multivariate generalizability theory. Specifically, it investigates relative effects of each error source, and finds optimal measurement conditions for the number of items within each content domain that maximizes the reliability-like coefficients, such as a generalizability coefficient and an index of dependability. The method is based on teacher recommendation letters and self-introduction letters, using an analytic scoring method in the context of selection of mathematically gifted students by observation and nomination. This study analyzed data from the 2011 academic year in the science education institute for the gifted, which is attached to the university located in the Seoul metropolitan area. It should be noted that the optimal scoring structures of this study are not generalizable to other selection instruments. However, the methodology applied in this study can be utilized to find optimal measurement conditions for the number of raters, the number of content domains, and the number of items in other selection instruments self-developed by many institutions including: the education institutes for the gifted at provincial offices of education, gifted classes, and the science education institutes for the gifted attached to universities in general. In addition, the methodology will provide bases for making informed decisions in selection instruments of the gifted based on measurement traits.

• Journal of The Korean Association For Science Education
• /
• v.25 no.2
• /
• pp.173-183
• /
• 2005

In this study, we questioned what assessment types and scoring methods of science performance assessment(SPA) were being used in middle and high school, and how much these SPA scores were reliable(generalizable). To answer these questions, SPA data obtained from the seven schools were classified according to assessment type and scoring method. Based upon this classification, we analyzed the reliability by applying generalizability theory. The result, from the classification of assessment type and scoring method, showed that SPA types of the seven schools were divided into two types: paper-pencil type and task type. Paper-pencil type included answer(content)-restricted essay-type test solely. Task type has two parts: process and outcome assessment. As the results of analyzing scoring methods of the seven schools, there were two cases in the way of scoring methods: one case is scoring all essay-type items and performance tasks by one teacher, the other is scoring assigned performance tasks by two teachers. But the case of scoring assigned essay-type items or the case of cross scoring by two or more teachers were not found. The findings of the reliability analysis are as follows: (1) Effect of essay-type item to SPA score was larger than that of performance task. (2) There was remarkable difference among the seven schools' interaction effect of person and rater in scoring performance tasks. (3) Most of generalizability(reliability) coefficients of SPA for the seven schools were smaller than the acceptable generalizability coefficient(0.80). Therefore, the population of statistical parameters such as number of item, task and rater, should be increased for approaching the acceptable generalizability level.

• Communications of Mathematical Education
• /
• v.28 no.4
• /
• pp.431-455
• /
• 2014

The purpose of this study was to suggest methods to apply generalizability theory to mathematics teacher evaluation using classroom observations in Korea by analysing mathematics teachers in the U.S. using the instructional quality of assessment instrument as an illustrative example. The subjects were 96 teachers participating in Year 3 and Year 4 from the Middle-school Mathematics and the Institutional Setting of Teaching (MIST) project funded by the National Science Foundation since 2007. The MIST project investigates the following question: What does it takes to support mathematics teachers' development of ambitious and equitable instructional practices on a large scale (MIST, 2007). This study examined data based on both the univariate generalizability analysis using GENOVA program and the multivariate generalizability analysis using mGENOVA program. Specifically, this study determined the relative effects of each error source and investigated optimal measuring conditions to obtain the suitable generalizability coefficients. The methodology applied in this study can be utilized to find effective optimal measurement conditions for the mathematics teacher evaluation for professional development in Korea. Finally, this study discussed limitations of the results and suggested directions for future research.

• Proceedings of the Korean Information Science Society Conference
• /
• 2005.11b
• /
• pp.955-957
• /
• 2005

Ada 언어의 일반화(Generic) 메커니즘은 복잡한 대형 시스템의 설계에 있어 소프트웨어의 재사용성을 높이기 위한 효과적인 방편으로 사용되어 왔다. 일반화 모듈은 기능 또는 알고리즘은 정의되었으나 파라미터의 구체적인 속성이 아직 정해지지 않은 한 개의 소프트웨어 모듈을 다양한 파라미터를 적용하여 여러 곳에서 반복적으로 사용할 수 있도록 한 개념이다. 정형이론 및 구조적 설계기법에 기반한 설계도구인 Stalemate 또한 설계 모듈의 반복적인 재사용을 위해 일반화 차트(Generic Chart)를 제공하고 있다. ATCS(Air Traffic Control System) 설계 모듈에 적용한 결과 두 가지 모두 소프트웨어의 재사용과 설계의 효율성 향상을 위한 도구로 활용할 수 있으며 Statemate의 경우 일반화 차트를 다양한 자료형으로 실례화(Instantiation)할 수 있다면 보다 효과적인 설계모듈의 재사용이 가능할 것이다.

• Journal of the Korean Data and Information Science Society
• /
• v.27 no.2
• /
• pp.395-407
• /
• 2016

This study aims to apply the G-theory for estimation of reliability of evaluation scores between raters on Taekwondo Poomsae rating categories. Selecting a number of game days and raters as multiple error sources, we analyzed the error sources caused by relative magnitude of error variances of interaction between the factors and proceeded with D-study based on the results of G-study for optimal determination of measurement condition. The results showed below. The estimated outcomes of variance component for accuracy among the Taekwondo Poomsae categories with G-theory showed that impact of error was the biggest influence factor in raters conditions and in order of interaction in subjects and between subjects, also impact of variance component estimation error on expression category was the major influence factor in interaction and in order of the between subjects and raters. Finally, the result of generalizability coefficient estimation via D-study showed that measurement condition of optimal level depend on the number of raters was 8 persons of raters on accuracy category, and stable reliability on expression category was gained when the raters were 7 persons.

• Proceedings of the Korea Institutes of Information Security and Cryptology Conference
• /
• 2003.07a
• /
• pp.57-61
• /
• 2003

Iwata 등은 SPN 구조에 기반한 블록 암호들 중 Serpent에 대한 의사 난수성을 분석하였다 ［2］. 그들은 Serpent의 구조를 최대한 보존한 상태에서 의사 난수성을 분석하기 위하여 Serpent의 Diffusion layer의 특성을 그대로 보존하여 일반화 한 후 이론을 전개하였다. 본 논문에서는 Serpent가 취한 Diffusion layer 뿐만 아니라 SPN 구조에 기반한 블록 암호들이 취할 수 있는 임의의 Diffusion layer에 대하여 적용 가능한 일반적인 이론을 도출해낼 것이다.