• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.10 no.9
• /
• pp.2424-2434
• /
• 2009

This study, centering on public enterprises, has been conducted from 4 different perspectives such as grasping the effects that the factor precedent of the multi-rater feedback affects on the usefulness of evaluation and the effects that the usefulness of evaluation affects on the organizational achievement, checking on whether there is any discrepancy among the effects affecting on the usefulness of evaluation in accordance with the degree of advance training on the evaluation system, and being what kind of effects the usefulness of valuation that the appraised eventually understand affects on the organizational achievement. As result of the study, the factor precedent that are confidence of the appraisers and the support from CEO reasonably affected on the usefulness of evaluation, but the standard referred to evaluation was not reasonable. It became resultantly clear that the confidence of the appraisers held the effect of control from the verification of whether there is the effect of control between the factor precedent and the usefulness of evaluation in accordance with the degree of advance training. However, it turned out that the standard referred to evaluation and the support from CEO held no effect of control. As result, It was possible to confirm that the usefulness of evaluation in this study affects directly or indirectly to the organizational achievement such as organizational immersion, job satisfaction, self development opportunity, and organizational revitalization as influenced by the prior factors, and the advance training was partially effective on the adjusting effect.

• The Journal of Korean Medicine Ophthalmology and Otolaryngology and Dermatology
• /
• v.26 no.2
• /
• pp.10-18
• /
• 2013

Objectives : We performed a pilot study to investigate inter- and intra-rater reliability of pattern identification using nasal endoscopy for allergic rhinitis(AR). Methods : Eight experts of ophthalmology, otolaryngology and dermatology of Korean medicine evaluated 20 nasal endoscopy photograph cases of AR patients with pattern identification index using nasal endoscopy for AR including the nasal membrane color(pale / hyperemia), nasal membrane humidity(dryness / dampness), rhinorrhea(watery / yellow), and membrane edema (atrophic / edematous) on nasal endoscopy. Results : Intra-rater agreement(%) and Kappa coefficient was generally from 'moderate' to 'good'(% agreement: 73.13-90% / Kappa coefficient: 0.547-0.748). Inter-rater agreement(%) and Kappa coefficient was also from 'moderate' to 'good' (% agreement: 65-85% / Kappa: 0.475-0.778) except 'humidity(dryness / dampness)' item (% agreement: 55.98% / Kappa: 0.340). In findings of subgroup analysis according to affiliation of raters, Inter-rater agreement(%) and Kappa coefficient of raters in same affiliation was higher than inter-rater agreement(%) and Kappa coefficient of raters in different affiliation except 'dryness / dampness' item. Conclusions : It is necessary to improve objectivity and reproducibility of pattern identification using nasal endoscopy for allergic rhinitis(AR) through the development of detail-oriented criteria and enhanced training of clinicians with development of standard operating procedures(SOPs).

• The Journal of the Korea Contents Association
• /
• v.11 no.9
• /
• pp.407-413
• /
• 2011

Distinguishing aphasia in stroke patients and observing the subtle linguistic characteristics associated with it primarily requires the use of instruments that provide reliable assessment results. Additionally, examiners should be fully aware of how to use those instruments. This study examined 46 stroke patients for aphasia and assessed the reliability of their diagnoses according to examiners whose medical fields were different from each other. Furthermore, a comparison was made between the reliability before training and that after training. To this end, 46 stroke patients were tested for aphasia and in terms of their speech disorder degree by 3 groups, each of which consisted of 12 professionals (3 SLP, 3 neurologist, and 3 nurse). In the result, a rating of 'acceptable' was given for speech intelligibility tasks and the voice quality of /ah-/ prolongation, and other sub-tests were marked as 'good-excellent' by the experts with different areas of medical expertise. For the tasks marked as 'acceptable', the subjects were video-trained for 3 weeks and the differences were compared before and after their training. Consequently, the differences in the examiners' ratings in the speech intelligibility tasks showed a significant decrease and the accuracy of their voice quality ratings showed a significant increase. In the result of research on the correlation between the accuracy of the sub-test ratings and the amount of clinic experience, speech therapists developed more accuracy in rating a picture description task and a speech intelligibility task as their experience accumulated. Meanwhile, doctors and nurses showed more accurate ratings in picture description tasks with greater clinical experience. The results of this study suggest that assessing the neurologic-communicative disorders of stroke patients requires ongoing training and experience, especially for speech disorders. It was also found that the rating reliability in this case could be improved by training.

• Transactions of the Korean Society of Mechanical Engineers B
• /
• v.41 no.2
• /
• pp.125-133
• /
• 2017

This paper presents the inter-rater reliability of finger spasticity assessment tested realized by using finger simulator that mimics finger spasticity of patients after a stroke. For controlling the simulator torque, finger spasticity was modeled, and the model parameters were obtained by measuring quantitative data while grading based on Modified Ashworth Scale (MAS). A robotic finger simulator was designed for mimicking finger spasticity. Evaluation of this simulator with the help of seven rehabilitation doctors showed that the simulator had a Cohen's kappa value of 0.619 for Metacarpophalangeal Joint and 0.514 for Proximal Interphalangeal Joint. Fleiss' kappa between raters is 0.513 for Metacarpophalangeal Joint and 0.486 for Proximal Interphalangeal Joint. Therefore, the spasticity assessment made by MAS grade system is not reliable owing to the subjectivity of the assessment. The proposed robotic simulator can be used as a training tool for improving the reliability of the spasticity assessment.

• The Journal of Korean Physical Therapy
• /
• v.22 no.5
• /
• pp.83-93
• /
• 2010

Purpose: The purpose of this study was to determine the reliability and validity of the Functional Movement Screen (FMS) for assessing Korean athletes. Methods: A total of 48 patients (37 males and 11 females) participated in this study. Data were acquired after translation of the FMS from English to Korean and cross-cultural adaptation of the this questionnaire. To determine inter-rater reliability, the relationship between the FMS scores obtained by two raters was evaluated using the Kappa coefficient, which was in total agreement with the Intra-Correlation Coefficient (ICC3,1). Concurrent validity was examined by correlating the FMS scores with the Oswestry disability index (ODI) scores and Visual analogue scale (VAS) scores. Results: The raters demonstrated excellent agreement on 7 (above 90%) of the 17 test (72.9 to 97.9%) components. Substantial agreement was seen in 11 of the 17 tests. Two components of the In-line lunge and rotatory stability tests demonstrated moderate agreement. It showed good inter-rater reliability: the Kappa coefficients ranged from 0.42 to 0.97. ICC3,1=0.42~0.99 and 0.93 (total FMS score). Cronbach's alpha for FMS was 0.80. It was not correlated with ODI or VAS. Conclusion: The Korean version of the FMS is a reliable instrument for measuring movement patterns of Korean athletes and for making decisions related to interventions for performance enhancement.

• Clinics in Shoulder and Elbow
• /
• v.25 no.4
• /
• pp.321-327
• /
• 2022

Background: Scapular dyskinesis is considered a risk factor for the shoulder pain that may warrant screening for prevention. Clinicians of all experience screen scapular dyskinesis using the scapular dyskinesis test yes-no classification (Y-N), yet its reliability in asymptomatic individuals is unknown. We aimed to establish Y-N's intra- and inter-reliability between students and expert physical therapists. Methods: We utilized a cross-sectional design using consecutive asymptomatic subjects. Six students and two experts rated 100 subjects using the Y-N. Cohen's kappa (κ) and Krippendorff's alpha (K-α) were calculated to determine intra- and inter-rater reliability. Results: Intra- and inter-rater values for experts were κ=0.92 (95% confidence interval [CI], 0.91-0.93) and 0.85 (95% CI, 0.84-0.87) respectively; students were κ=0.77 (95% CI, 0.75-0.78) and K-α=0.63 (95% CI, 0.58-0.67). Conclusions: The Y-N is reliable in detecting scapular dyskinesis in asymptomatic individuals regardless of experience.

• The Journal of Advanced Prosthodontics
• /
• v.14 no.4
• /
• pp.203-211
• /
• 2022

PURPOSE. The aim of this study was to introduce rating criteria to evaluate student performance in a newly developed, digital wax-up preclinical program for computer-aided design (CAD) of full-coverage crowns and preliminarily investigate the reliability and internal consistency of the rating system. MATERIALS AND METHODS. This study, conducted in 2017, enrolled 47 fifth-year dental students of Okayama University Dental School. Digital wax-up training included a fundamental practice using computer graphics (CG), multipurpose CAD software programs, and an advanced practice to execute a digital wax-up of the right mandibular second molar (#47). Each student's digital wax-up work (stereolithography data) was evaluated by two instructors using seven qualitative criteria. The total qualitative score (0-90) of the criteria was calculated. The total volumetric discrepancy between each student's digital wax-up work and a reference prepared by an instructor was automatically measured by the CAD software. The inter-rater reliability of each criterion was analyzed using a weighted kappa index. The relationship between the total volume discrepancy and the total qualitative score was analyzed using Spearman's correlation. RESULTS. The weighted kappa values for the seven qualitative criteria ranged from 0.62 - 0.93. The total qualitative score and the total volumetric discrepancy were negatively correlated (ρ = -0.27, P = .09, respectively); however, this was not statistically significant. CONCLUSION. The established qualitative criteria to evaluate students' work showed sufficiently high inter-rater reliability; however, the digitally measured volumetric discrepancy could not sufficiently predict the total qualitative score.

• The Journal of Korean Academic Society of Nursing Education
• /
• v.22 no.4
• /
• pp.549-558
• /
• 2016

Purpose: The purpose of this study was to develop and validate a Nursing Competency Scale in Simulation (NCSS) for nursing students. Methods: A preliminary version of the NCSS of 14 items was derived from the literature. A panel of seven experts reviewed the preliminary version for content validation and developed 15 items scale. A convenient sample of 195 nursing students completed the survey and two evaluators measured the performances of nine teams in a simulation scenario with NCSS. The data were analyzed using exploratory factor analysis, descriptive statistics, independent t-test and Cronbach's alpha. Intraclass correlation coefficient (ICC) was used in order to estimate the degree of inter-rater reliability. Results: An exploratory factor analysis demonstrated that two factor structures of the NCSS explained 51.1% of the total variance. Two factors were named psychosocial skills and cognitive and psychomotor skills. The mean scores of NCSS between third and 4th grade were significantly different providing support for its known-group validity. Cronbach's alpha was .90 and ranged from .79 to .88. The overall ICC for inter-rater agreement was 0.89 (95% CI 0.03 to 0.98). Conclusion: This scale shows preliminary evidence for validity and reliability. It could be a useful instrument for measuring learning outcome in simulation for nursing students' clinical competency.

• Journal of the Korean Society of Physical Medicine
• /
• v.8 no.1
• /
• pp.137-145
• /
• 2013

PURPOSE: The purpose of this study is to examine assessment of test-retest reliability for 13 items of pediatric balance scales, then the intrarater reliability among the raters. METHODS: Study participants included 6 children with spastic cerebral palsy who could walk. Raters were consist of seventy-four, 2nd year physical therapy major students. The children's ability to achieve physical balance was wideotaped for PBS items. The raters watched the tapes and evlauated each child twice. Test-retest reliability was analyzed using the Spearman correlation, and interrater reliability was analyzed using the Kendall's coefficient of concordance for ranks. RESULTS: The total PBS scores averaged 49.22 and 50.06 for first and second tests. Test-retest reliability of PBS individual items were between low and high. The low itmes were 3 items, and high were 4 itmes. Interrater reliability were low agreement. CONCLUSION: Based on the study, the pediatric balance scale is an effective evaluating tool for measuring functional balance of school age child. However, the agreement of students or beginner's interpretation is necessary in conjunction to this study results. On top of that, it is important to provide detailed explanation and repetitive training, which improves the analysis reliability. Also, each category subscores level for the pediatric balance scale must be examined carefully in conjunction to performing test, only after improving match quality via preceding practice.

• The Journal of Korean Physical Therapy
• /
• v.27 no.4
• /
• pp.175-182
• /
• 2015

Purpose: The purposes of this study were to develop the Korean version of the trunk impairment scale (K-TIS) and to examine reliability and responsiveness of the K-TIS in patients with stroke. Methods Subjects of the study were 51 stroke patients (mean age: 57.78 years) recruited from two stroke clinics. For the interrater and test-retest reliability, two raters measured the K-TIS two times using video clips with an interval of 2 weeks. For the responsiveness, intensive physical therapy training was provided to all participants 2 times a day for one month or three months depending on the onset of the stroke and the admission rules of the two clinics. Inter-rater reliability and test-retest reliability of the K-TIS three subscales (static sitting balance, dynamic sitting balance, and coordination) scores and total scores were examined using intra-correlation coefficient ($ICC_{3,1}$) and Pearson's correlation coefficient (r). To examine responsiveness, the minimally important difference (MID) was calculated with effect size. Results: Inter-rater reliability of the K-TIS subscales and total scores were all high (ICC3,1=0.920-0.983 and r=0.924-0.984). For the test-retest reliability, $ICC_{3,1}$=0.805-0.901 and r=0.806-0.903, and the MID for acute and post-acute as well as chronic stroke patients remained in the mean change range. Conclusion: It is suggested that the K-TIS might be used for clinical and research purposes as a standardized tool for stroke patients. In addition, it can also be useful in establishment of treatment goal(s) and planning treatment program(s) for patients with stroke.