Browse > Article
http://dx.doi.org/10.7465/jkdi.2016.27.2.395

Analysis of error source in subjective evaluation results on Taekwondo Poomsae: Application of generalizability theory  

Cho, Eun Hyung (The Research Institution for Physical Fitness and Sport Science, Sungkyunkwan University)
Publication Information
Journal of the Korean Data and Information Science Society / v.27, no.2, 2016 , pp. 395-407 More about this Journal
Abstract
This study aims to apply the G-theory for estimation of reliability of evaluation scores between raters on Taekwondo Poomsae rating categories. Selecting a number of game days and raters as multiple error sources, we analyzed the error sources caused by relative magnitude of error variances of interaction between the factors and proceeded with D-study based on the results of G-study for optimal determination of measurement condition. The results showed below. The estimated outcomes of variance component for accuracy among the Taekwondo Poomsae categories with G-theory showed that impact of error was the biggest influence factor in raters conditions and in order of interaction in subjects and between subjects, also impact of variance component estimation error on expression category was the major influence factor in interaction and in order of the between subjects and raters. Finally, the result of generalizability coefficient estimation via D-study showed that measurement condition of optimal level depend on the number of raters was 8 persons of raters on accuracy category, and stable reliability on expression category was gained when the raters were 7 persons.
Keywords
Generalizability theory; g-coefficient; multiple source error; reliability; variance component;
Citations & Related Records
Times Cited By KSCI : 6  (Citation Analysis)
연도 인용수 순위
1 Cho, J. H. (1991). Genralizability coefficient estimation of skinfold thickness test and basketball skill test, Ph. D. Thesis, Korean National College of Physical Education, Seoul.
2 Cho, J. H. (2006). An analysis of sources of error and optimal condition of generalizability coefficient in the Korean national bowling player selective match. The Korean Journal of Physical Education, 45, 457-167.
3 Choi, J. S. (2014). Projection analysis for two-way variance components. Journal of the Korean Data & Information Science Society, 25, 547-554.   DOI
4 Choi, Y. L. (2013). The olympic experience and expertise of taekwondo national representative coach, Ph. D. Thesis, Korea National Sport University, Seoul.
5 Cronbach, L. J., Gleser, G. C., Nanda, H. and Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalization for scores and profiles, Wiley, New York.
6 Hagtvet, K.nut A. (1997). The function of indicators and errors in construct measures: An application of generalizability theory. Journal of Vocational Education Research, 22, 247-266.
7 Heo, C. K. (2003) Generalizability coefficient estimation of subjectivity volleyball test, Ph. D. Thesis, Kookmin University, Seoul.
8 Jung, Y. H. (2009). Analysis on reliability of referees judgment in Taekwondo competition applying generalizbility theory, Ph. D. Thesis, Kookmin University, Seoul.
9 Kang, S. J. and Ann, A. J. (1999). Rater's objectivity in the assessment of gymnastic performance. The Korean Journal of Physical Education, 38, 641-650.
10 KERIS (2015). http://www.riss.kr.
11 MacMillan, P. D. (2000). Classical, generalizability, and multifaceted rasch detection of interrater variability in large, sparse data sets. The Journal of Experimental Education, 68, 167-190.   DOI
12 Nunally, J. C. (1978). Psychometric theory, 2nd ed., McGraw-Hill, New York.
13 Park, H, C. (2014). Comparison of confidence measures useful for classification model building. Journal of the Korean Data & Information Science Society, 25, 365-371.   DOI
14 Park, H. J. and Pi, S. Y. (2013). Study on the K-scale reflecting the confidence of survey responses. Journal of the Korean Data & Information Science Society, 24, 41-51.   DOI
15 Safrit, M. J., Atwater, A. E., Baumgarther, T. A andWest, C. (1976). Reliability theory, American Alliance for Health, Physical Education and Recreation, Washington.
16 Shavelson, R. J and Webb, N. M. (1981). Generalizability theory: 1973-1980. British Journal of Mathematical and Statistical Psychology, 34, 133-166.   DOI
17 Shavelson, R. J., Baxter. G. P. and Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30, 215-232.   DOI
18 Sun, A., Michael J. V. and Gao, X. (1997). Using generalizability theory to assess the reliability of student ratings of academic advising. The Journal of Experimental Education, 65, 367-379.   DOI
19 WTF (2014). http://www.wtf.org.
20 Zhu, W., Ennis, C. D. and Chin, A. (1998). Many faceted rasch modeling expert judgement in test development. Measurement in physical education and exercise science, 2, 21-39.   DOI
21 Ahn, A. J. (1997). Application of generalizability in analyzing sources of error variance of gymnastic judgement, Ph. D. Thesis, Myungji University, Kyeonggido.
22 Allen, M. J. and Yen, W. M. (1979). Introduction to measurement theory, 1st ed., Waveland Press, Illinois.
23 Brennan, R. L. (1983). Elements of generalizability theory, ACT Publication, Iowa.
24 Cho, E. H. (2014). Development of computer-based assessment instrument and movement classification for poomsae skills in taekwondo, Ph. D. Thesis, Sungkyunkwan University, Kyeonggido.
25 Cho, E. H., Kwon, J. H., Cha, C. H. and Kim, K. S. (2005). The severity and consistency of the judge's decision in taekwondo. Journal of Sport Leisure Studies, 25, 85-95.
26 Kim, D. Y. and Heo, C. K. (2002). Generalizability coefficient estimation of subjectively volleyball test. The Korean Journal of Measurement and Evaluation in Physical Education and Sport Science, 4, 15-28.
27 Kim, M. Y. and Kang, S. J. (2003). Examining rater errors in performance assessment with a many-faceted rasch model. The Korean Journal of Physical Education, 42, 1019-1027.
28 Kim, S. S. (1989). An analysis of sources of variation in teacher behaviors using generalizability theory, Ph. D. Thesis, University of Virginia, Virginia.
29 Kim, S. S. (1993). The generalizability of student ratings of instructors across sections. The Korean Journal of Educational Research, 31, 23-40.
30 Kim, Y. B. and Kim, S. S. (2001). Generalizability theory, Kyohaksa of Publisher, Seoul.
31 Ko, B. G. (1997). A Generalizability of physical performance tests for apparatus exercises in elementary school. Journal of the Research Institute of Physical Education, 18, 111-126.
32 Kukkiwon (2015). The countries geups & dans situation, Retrieved March 5, 2015, from http://www.kukkiwon.or.kr.
33 Lee, J. T. (2010). A Study on the performance pattern in the elite decathlon. Journal of the Korean Data & Information Science Society, 21, 1071-1079.
34 Lee, J. Y. (2005). Estimating the reliability of the selection method of the national team bowlers; The application of generalizability theory, Ph. D. Thesis, Korea National Sport University, Seoul.
35 Looney, M. (1997). Objectivity measurement of figure skating performance. Journal of outcome measurement, 1, 143-163.