[KSCI] Korea Science Citation Index Service

http://dx.doi.org/10.9708/jksci.2015.20.12.169

Permutation P-values for Inter-rater Agreement Measures

Um, Yonghwan (Division of Industrial and Management Engineering, Sungkyul University)

Publication Information

Journal of the Korea Society of Computer and Information / v.20, no.12, 2015 , pp. 169-174 More about this Journal

Abstract

Permutation p-values are provided for the agreement measures for multivariate interval data among many raters. Three agreement measures, Berry and Mielke's measure, Janson and Olsson's measure, and Um's measure are described and compared. Exact and resampling permutation methods are utilized to compute p-values and empirical quantile limits for three measures. Comparisons of p-values demonstrate that resampling permutation methods provide close approximations to exact p-values, and Berry and Mielke's measure and Um's measure show similar performance in terms of measuring agreement.

Keywords

Permutation Method; agreement measure; Multivariate data;

Citations & Related Records

Times Cited By KSCI : 1 (Citation Analysis)

Reference
Cited By KSCI

1	J. Cohen, A coefficient of agreement for nominal scales, Educational and Psychological Measurement, Vol. 20, pp. 37-46, 1960. DOI
2	G. W. Willam, Comparing the joint agreement of several raters with another rater, Biometrics, Vol. 32, pp. 619-627, 1976. DOI
3	R. J. Light, Measures of response agreement for aualitative data: some generalizations and alternatives, Psychological Bulletin,Vol. 76, pp. 365-377, 1971. DOI
4	L. Hubert, Kappa revisited, Psychological Bulletin, Vol. 36, pp. 207-216, 1983.
5	A. J. Cogner, Integration and generalization of kappas for multiple raters, Psychological Bulletin, Vol. 88, pp. 322-328, 1980. DOI
6	K. J. Berry, and P. W. Mielke Jr. A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement, Vol. 48, pp. 921-933, 1988. DOI
7	H. Janson, and U. Olsson, A measure of agreement for interva or nominal multivariate observations, Educational and Psychological Measurement, Vol. 61, No. 2,pp. 277-289. 2001. DOI
8	Y. H. Um, A new agreement measure for interval multivariate observations, Journal of Korean Data & Information Science Society, Vol. 15, pp. 263-271, 2004.
9	E. J. G. Pitman, Significance tests which may be applied to sample from any populations, III. The analysis of variance test. Biometrika, Vol. 29, pp. 322-335, 1938.
10	A. F. Hayes, Permustat: randomization tests for the Machintosh, Behavior Research Methods, Instruments, & Computers, Vol 28, pp. 473-475, 1996. DOI
11	R. S. Chen and W. P. Dunlap, SAS procedures for approximate randomizatio tests, Behavior Research Methods, Instruments, & Computers, Vol. 25, pp. 406-409, 1993. DOI
12	P. S. Maxim, Quantative research Methods in the Social Sciences. New York: Oxford University Press, 1999.
13	J. E. Johnston, K. J. Berry, and P. W. Mielke, Permutation tests: precision in estimating probability values., Perceptual and Motor Skills, Vol. 105, pp. 915-920, 2007. DOI
14	P. W. Mielke Jr. and K. J. Berry, Permutation methods: a distance function approach. (2nd ed.) New York: Springer-Verlag, 2007.