• The Journal of the Korean dental association
• /
• v.57 no.8
• /
• pp.448-455
• /
• 2019

This study aims to apply the Generalizability Theory (G-theory) for estimation of reliability of evaluation scores between raters on Patient Dentist Interaction. Selecting a number of raters as multiple error sources, this study was analyzed the error sources caused by relative magnitude of error variances of interaction between the factors and proceeded with D-study based on the results of G-study for optimal determination of measurement condition. The estimated outcomes of variance component for accuracy among the Patient Dentist Interaction evaluation with G-theory showed that impact of error was the biggest influence factor in students. The second influence was the item effect, and the rater effect was relatively small. The Generalizability coefficients for case1 and case2 which were estimated through the D- study were calculated relatively low.

• Proceedings of the Safety Management and Science Conference
• /
• 2011.04a
• /
• pp.701-705
• /
• 2011

The paper reviews Cronbaha's coefficient to measure a single source of error. On the contrary to classical measurement theory, the generalizability study can be used in the social survey design to calculate Generalizability Index (GI) and Dependability Index (DI) for measuring multiple sources of errors of behavior evaluation. The study proposes application guidelines to implement R:($A{\times}B$) mixed models that are composed of random factor and fixed factor.

• Journal of the Korean earth science society
• /
• v.31 no.1
• /
• pp.95-105
• /
• 2010

The purpose of this study lies in applying generalizability theory depending on the aim of the usage of GALT to analyze the sources of error of single-facet considering item and person only and to analyze the sources of error of multi-facet considering item, person and domain. The study was conducted with 1016 students of local elementary, middle, and high schools. The 21 items of a full version were answered for 40 minute and then the 12 items of short version were sampled to analyze reliability using generalizability theory. Both the full version and the short version of the items were analyzed using Cronbach's alpha for data analysis, and we applied generalizability theory and separate $p{\times}i$ design and $p{\times}(i:h)$ design, G study and D study were performed. Results of analysis are as follows: First, the result of D study after $p{\times}I$ design both on the full version and the short version showed that in the case of the full version, the generalizability coefficient was 0.87 exceeding a normal level of 0.80, and the normal level of generalizability coefficient was achieved in 13 items as well. In case of short version, when 12 items were evaluated, generalizability coefficient was 0.77 not reaching the normal level, and the normal level was achieved in case of more than 15 items. Second, the result of D study after $p{\times}(I:H)$ design on the short version showed that once one domain consists of 2 items in 6 domains, generalizability coefficient was 0.71 which is lower than the normal level of 0.80, the normal level was achieved in more than 5 item cases.

• Proceedings of the Safety Management and Science Conference
• /
• 2011.04a
• /
• pp.695-699
• /
• 2011

The study presents an efficient design method of Venn Diagram that can be used when implementing the quality design of experiments based on generalizability theory. The paper examines four mixed and combined models that are designed by fixed factor, random factor, crossed factor and nested factor. The models considered in this research are $A^*{\times}B^*{\times}C$, (B: $A^*$)${\times}C$, $A{\times}B{\times}C$ and (B: A)${\times}C$.

• Communications of Mathematical Education
• /
• v.28 no.4
• /
• pp.431-455
• /
• 2014

The purpose of this study was to suggest methods to apply generalizability theory to mathematics teacher evaluation using classroom observations in Korea by analysing mathematics teachers in the U.S. using the instructional quality of assessment instrument as an illustrative example. The subjects were 96 teachers participating in Year 3 and Year 4 from the Middle-school Mathematics and the Institutional Setting of Teaching (MIST) project funded by the National Science Foundation since 2007. The MIST project investigates the following question: What does it takes to support mathematics teachers' development of ambitious and equitable instructional practices on a large scale (MIST, 2007). This study examined data based on both the univariate generalizability analysis using GENOVA program and the multivariate generalizability analysis using mGENOVA program. Specifically, this study determined the relative effects of each error source and investigated optimal measuring conditions to obtain the suitable generalizability coefficients. The methodology applied in this study can be utilized to find effective optimal measurement conditions for the mathematics teacher evaluation for professional development in Korea. Finally, this study discussed limitations of the results and suggested directions for future research.

• Science of Emotion and Sensibility
• /
• v.11 no.4
• /
• pp.553-564
• /
• 2008

Generalizability theory was employed to examine how the reliability of polygraph test is affected by the number of relevant questions, the number of repeated tests (the number of of charts), and the number of raters(scorers). The data consisted of the results of the polygraph tests administered to 31 crime suspects. The sample was drawn from the real polygraph tests based on Backster ZCT and archived by the Prosecutor's Office of the Republic of Korea. The numerical scores assigned by thirteen raters to the test charts were analyzed to determine the generalizability of the scores. The largest variance component was accounted for by the examinee factor(43.97%) and the residual variance component was 16.84% of the total variance. The variance component due to the interaction between the examinee and the chart factors was 12.17% and the variance component due to the three way interaction of the examinee, the repeated test, and the relevant question factors was 10.31%. The generalizability coefficient for the current measurement procedure as practiced by the Korean Prosecutor's Office was 0.74 which suggests that the current procedure is acceptable. However, measurement procedures with the combination of more than two relevant questions, more than three repeated tests, and more than two raters were generally found to yield generalizability coefficients larger than 0.80. Therefore, such procedures need to be considered seriously in order to significantly improve the reliability of polygraph test.

• Journal of the Korean earth science society
• /
• v.26 no.6
• /
• pp.511-523
• /
• 2005

The purpose of this study is to analyze the 1st and 2nd Korea Earth Science Olympiad (KESO) items, in order to find informations to enhance item quality. To do this, internal and external item classification frameworks are developed. Item difficulty (P), discrimination index (DI), correlation, and reliability are estimated by using classical test theory. Generalizability is also estimated by applying the generalizability theory. The results of item classification are as follows: (1) ‘Geology’, ‘astronomy’ and ‘data analysis and interpretation’ are dominant in content and inquiry process domain, respectively. Nearly every item has textbook context. (2) There is no difference between the preliminary and final tests in terms of their thinking skills sections. (3) As a whole, the ratio of items with pictures is high in item representation. However, multiple-choice and short answer items are more common in preliminary competition, and essay type items are found more often in final competition. The ratio of simple items is high in middle school section and preliminary competition, but composite items are dominant in high school section and final competition. The findings of item analysis are as follows: (1) In the middle school section, P is low and DI is moderate. But in the high school section, there is a considerable differences between science high schools and other high schools in general. (2) The highest correlation is reported between the scores of meteorology domain and total score in middle school, whereas in high school astronomy domain and total score show the highest correlation. (3) General high school section show the highest Cronbach $\alpha$ and generalizability. (4) General high school section show acceptable generalizability coefficient (> 0.80), but middle and science high school section should increase the number of items to reach acceptable generalizability level.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.18 no.1
• /
• pp.565-571
• /
• 2017

The purpose of this study is to determine the effects of error sources in emotional intelligence assessment of young children. The emotional intelligence of 198 five years old children was assessed using the Emotional Intelligence Scale developed by Lee[14]. Evaluation results were analyzed using G study for generalizability theory. G study results show that parents can be effective evaluators for emotional intelligence in young children. Strategies to reduce error effects are discussed.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.17 no.11
• /
• pp.507-512
• /
• 2016

The purpose of this study is to determine the error sources and effects of each error source in self-control assessment of young children. The self-control of 259 five-year-old children was assessed using the Self-Control Rating Scale(SCRS) developed by Kendall and Wilcox[1]. The evaluation results were analyzed using G study of generalizability theory. The results of G study shows that parents can be effective evaluators for the self-control assessment of young children. The strategies to reduce the effects of error are also discussed.

• Journal of The Korean Association For Science Education
• /
• v.25 no.2
• /
• pp.173-183
• /
• 2005

In this study, we questioned what assessment types and scoring methods of science performance assessment(SPA) were being used in middle and high school, and how much these SPA scores were reliable(generalizable). To answer these questions, SPA data obtained from the seven schools were classified according to assessment type and scoring method. Based upon this classification, we analyzed the reliability by applying generalizability theory. The result, from the classification of assessment type and scoring method, showed that SPA types of the seven schools were divided into two types: paper-pencil type and task type. Paper-pencil type included answer(content)-restricted essay-type test solely. Task type has two parts: process and outcome assessment. As the results of analyzing scoring methods of the seven schools, there were two cases in the way of scoring methods: one case is scoring all essay-type items and performance tasks by one teacher, the other is scoring assigned performance tasks by two teachers. But the case of scoring assigned essay-type items or the case of cross scoring by two or more teachers were not found. The findings of the reliability analysis are as follows: (1) Effect of essay-type item to SPA score was larger than that of performance task. (2) There was remarkable difference among the seven schools' interaction effect of person and rater in scoring performance tasks. (3) Most of generalizability(reliability) coefficients of SPA for the seven schools were smaller than the acceptable generalizability coefficient(0.80). Therefore, the population of statistical parameters such as number of item, task and rater, should be increased for approaching the acceptable generalizability level.