• Korean Journal of Radiology
• /
• v.24 no.4
• /
• pp.284-293
• /
• 2023

Objective: To validate a simplified ordinal scoring method, referred to as modified length-based grading, for assessing coronary artery calcium (CAC) severity on non-electrocardiogram (ECG)-gated chest computed tomography (CT). Materials and Methods: This retrospective study enrolled 120 patients (mean age ± standard deviation [SD], 63.1 ± 14.5 years; male, 64) who underwent both non-ECG-gated chest CT and ECG-gated cardiac CT between January 2011 and December 2021. Six radiologists independently assessed CAC severity on chest CT using two scoring methods (visual assessment and modified length-based grading) and categorized the results as none, mild, moderate, or severe. The CAC category on cardiac CT assessed using the Agatston score was used as the reference standard. Agreement among the six observers for CAC category classification was assessed using Fleiss kappa statistics. Agreement between CAC categories on chest CT obtained using either method and the Agatston score categories on cardiac CT was assessed using Cohen's kappa. The time taken to evaluate CAC grading was compared between the observers and two grading methods. Results: For differentiation of the four CAC categories, interobserver agreement was moderate for visual assessment (Fleiss kappa, 0.553 [95% confidence interval {CI}: 0.496-0.610]) and good for modified length-based grading (Fleiss kappa, 0.695 [95% CI: 0.636-0.754]). The modified length-based grading demonstrated better agreement with the reference standard categorization with cardiac CT than visual assessment (Cohen's kappa, 0.565 [95% CI: 0.511-0.619 for visual assessment vs. 0.695 [95% CI: 0.638-0.752] for modified length-based grading). The overall time for evaluating CAC grading was slightly shorter in visual assessment (mean ± SD, 41.8 ± 38.9 s) than in modified length-based grading (43.5 ± 33.2 s) (P < 0.001). Conclusion: The modified length-based grading worked well for evaluating CAC on non-ECG-gated chest CT with better interobserver agreement and agreement with cardiac CT than visual assessment.

• Transactions of the Korean Society of Mechanical Engineers B
• /
• v.41 no.2
• /
• pp.125-133
• /
• 2017

This paper presents the inter-rater reliability of finger spasticity assessment tested realized by using finger simulator that mimics finger spasticity of patients after a stroke. For controlling the simulator torque, finger spasticity was modeled, and the model parameters were obtained by measuring quantitative data while grading based on Modified Ashworth Scale (MAS). A robotic finger simulator was designed for mimicking finger spasticity. Evaluation of this simulator with the help of seven rehabilitation doctors showed that the simulator had a Cohen's kappa value of 0.619 for Metacarpophalangeal Joint and 0.514 for Proximal Interphalangeal Joint. Fleiss' kappa between raters is 0.513 for Metacarpophalangeal Joint and 0.486 for Proximal Interphalangeal Joint. Therefore, the spasticity assessment made by MAS grade system is not reliable owing to the subjectivity of the assessment. The proposed robotic simulator can be used as a training tool for improving the reliability of the spasticity assessment.

• The Korean Journal of Applied Statistics
• /
• v.25 no.5
• /
• pp.719-731
• /
• 2012

Agreement analysis is conducted to assess reliability among rating results performed repeatedly on the same subjects by one or more raters. The kappa statistic is commonly used when rating scales are categorical. The simple and weighted kappa statistics are used to measure the degree of agreement between two raters, and the generalized kappa statistics to measure the degree of agreement among more than two raters. In this paper, we compare the performance of four different generalized kappa statistics proposed by Fleiss (1971), Conger (1980), Randolph (2005), and Gwet (2008a). We also examine how sensitive each of four generalized kappa statistics can be to the marginal probability distribution as to whether marginal balancedness and/or homogeneity hold or not. The performance of the four methods is compared in terms of the relative bias and coverage rate through simulation studies in various scenarios with different numbers of raters, subjects, and categories. A real data example is also presented to illustrate the four methods.

• KSCE Journal of Civil and Environmental Engineering Research
• /
• v.39 no.6
• /
• pp.913-921
• /
• 2019

The government introduced the comprehensive evaluation bidding system with the goal of pursuing the best value and the global standard in 2016. However, for the evaluation criteria on the construction evaluation reflected to the comprehensive evaluation bidding system, the problems of the objectivity insufficiency, the inclusion of multiple subjective evaluation items, and the irrationality of the weight for each evaluation item are continue to be presented. The central office group, the local government, the relevant industry, and the expert group share recognition, but the solution is not derived. Hence, the major evaluation items to be improved were derived with the characteristics analyzed to secure the objectivity of the construction evaluation. For the analysis method, the standard deviation and the Fleiss Kappa analysis method were used by utilizing the characteristics that the construction evaluation criteria consist of all 4-point measures (good, average, insufficient, and poor). According to the result, the 10 evaluation items of the total 25 construction evaluation items were derived as the evaluation items to be improved. It was found in the analysis on the major characteristics of the derived evaluation items that the qualitative evaluation criteria such as 'Very Suitable' and 'Suitable' were commonly included in the detailed evaluation guidelines. Hence, as far as the future construction evaluation standards are concerned, the qualitative evaluation standards are sublated, and the improvement should be made mainly for the quantitative evaluation criteria enabling the objectivity assurance.

• Clinics in Shoulder and Elbow
• /
• v.24 no.2
• /
• pp.98-105
• /
• 2021

Background: Our aim is to determine the interobserver reliability for surgeons to detect Hill-Sachs lesions on magnetic resonance imaging (MRI), the certainty of judgement, and the effects of surgeon characteristics on agreement. Methods: Twenty-nine patients with Hill-Sachs lesions or other lesions with a similar appearance on MRIs were presented to 20 surgeons without any patient characteristics. The surgeons answered questions on the presence of Hill-Sachs lesions and the certainty of diagnosis. Interobserver agreement was assessed using the Fleiss' kappa (κ) and percentage of agreement. Agreement between surgeons was compared using a technique similar to the pairwise t-test for means, based on large-sample linear approximation of Fleiss' kappa, with Bonferroni correction. Results: The agreement between surgeons in detecting Hill-Sachs lesions on MRI was fair (69% agreement; κ, 0.304; p<0.001). In 84% of the cases, surgeons were certain or highly certain about the presence of a Hill-Sachs lesion. Conclusions: Although surgeons reported high levels of certainty for their ability to detect Hill-Sachs lesions, there was only a fair amount of agreement between surgeons in detecting Hill-Sachs lesions on MRI. This indicates that clear criteria for defining Hill-Sachs lesions are lacking, which hampers accurate diagnosis and can compromise treatment.

• Journal of the Korean Society of Earth Science Education
• /
• v.16 no.1
• /
• pp.87-102
• /
• 2023

The purpose of this study is to propose an analytical framework for the essential contents that must be included in a climate change education program for elementary school students based on community issues, which can be used by citizen instructors in the community. To develop the analytical framework, 24 climate environmental education specialists were consulted seven times. The content validity of the final analysis framework was statistically verified using I-CVI and S-CVI coefficients, and the reliability of the expert panel was verified using Fleiss' Kappa coefficient. The final analysis framework consists of three analytical areas (program objectives, program content, program evaluation), seven analysis items, seven analysis indicators, and detailed explanations of the analysis indicators. In particular, by adding detailed explanations for the analysis indicators, the content validity and reliability were increased, and the objective nature of the analysis framework was firmly established. It is expected that the proposed analytical framework for a community-based climate change education program for elementary school students in this study will contribute to the systematic development of the program by citizen instructors.

• The Journal of the Korean dental association
• /
• v.58 no.3
• /
• pp.140-151
• /
• 2020

Purpose: To evaluate reliability of Q-ray view (Aiobio Inc,. Seoul, Korea) for assessing retention status of pit and fissure sealants. Methods: Pit and fissure sealants of 58 permanent molars from 15 third-grade students were examined. Posterior teeth with ≥1 pit and fissure sealants applied to the occlusal surface for >6 months were examined. The teeth were examined using traditional visual-tactile assessments and combined Q-ray view. Pit and fissure sealants were evaluated by assessing marginal plaque, marginal discoloration, marginal integrity, retention, and presence of caries. Fleiss kappa and Cohen's kappa values were calculated to compare inter- and intrarater agreements between visual-tactile and combined Q-ray view assessments. Results: Regarding interrater agreement in visual-tactile assessments, K values of Cohen's kappa for marginal plaque, marginal discoloration, and presence of caries were 0.22-0.57, 0.36-0.57, and 0.43-0.61, respectively, and agreements ranged from slight to moderate. When combined with Q-ray view, the values were 0.81-0.89, 0.69-0.88, and 0.80-0.90, respectively, and agreements ranged from substantial to nearly perfect level, indicating statistical significance. Marginal plaque (0.81-0.83), marginal discoloration (0.57-0.89), and presence of caries (0.69-0.91) showed higher agreements in combined Q-ray view than in visual-tactile assessments, and kappa values of marginal plaques were significantly higher in combined Q-ray view than in visual-tactile assessments. Conclusion: Evaluating retention status of pit and fissure sealants using Q-ray view showed higher reliability than using visual/tactile assessments for marginal plaque, marginal discoloration, and presence of caries. Therefore, Q-ray view may be used to assess the retention status of pit and fissure sealants.

• Asian Pacific Journal of Cancer Prevention
• /
• v.17 no.2
• /
• pp.491-495
• /
• 2016

Background: Small B-cell non-Hodgkins lymphoma (NHL) is difficult to be distinguished from non-neoplastic reactive processes using conventional haematoxylin-eosin (HE) staining due to different interpretations among pathologists with diagnosis based on morphologic features. Ancillary examinations such as immunohistochemical (IHC) staining are essential. However, negative or doubtful results are still sometimes obtained due to unsatisfactory tissue processing or IHC technique. The polymerase chain reaction (PCR) as a molecular diagnostic technique is very sensitive and specific. Clonality detection of heavy chain immunoglobulin (IgH) gene rearrangement has been widely used to establish diagnosis of B-cell NHL. Aims: To elaborate interobserver variation in small B-cell NHL diagnosis based on morphologic features only and to confirm sensitivity and specificity of the PCR technique as an ancillary method. Materials and Methods: A toptal of 28 samples of small B cell NHL and suspicious lymphoma were interpreted by 3 pathologists in Sardjito General Hospital based on their morphology only. The reliability of assessment and the coefficient of interobserver agreement were calculated by Fleiss kappa statistics. Interpretation results were confirmed with IHC staining (CD20, CD3, Bcl2). PCR was performed to analyze the clonality of IgH gene rearrangement. Results: Interobserver agreement in morphologic evalution of small B cell NHL and chronic lymphadenitis revealed kappa coefficient 0.69 included in the substantial agreement category. The cases were divided into 3 groups based on morphology and IHC results; lymphoma, reactive process and undetermined group. PCR analysis showed 90% sensitivity and 60% specificity. Conclusions: The present study revealed a substantial agreement among pathologists in small B-cell NHL diagnosis. For difficult cases, PCR is useful as complementary method to morphologic and IHC examinations to establish definitive diagnosis.

• Journal of health informatics and statistics
• /
• v.41 no.1
• /
• pp.49-56
• /
• 2016

Objectives: Carotid intima-media thickness (CIMT) and the presence of carotid artery plaque are widely used as preclinical markers of atherosclerosis. Due to operator dependency in measuring CIMT, it is important to evaluate the reliability of measuring CIMT and plaque between centers in a multicenter study. The purpose of this study is to evaluate the inter-rater reliability of CIMT and plaque presence among three clinical centers of the Cardiovascular and Metabolic Disease Etiology Research Center (CMERC). Methods: Twenty people without known cardiovascular disease (age 37-64) were enrolled during 2014-2015, and their left and right carotid arteries were examined repeatedly with ultrasonography for CIMT measurements at three clinical centers according to a predetermined protocol. Maximum and mean values of CIMT at distal common carotid artery were recorded. Plaque presence at a carotid artery was checked by an operator. The reliability of CIMT and carotid plaque presence was assessed using an intraclass correlation coefficient (ICC) and kappa statistics, respectively. Results: Calculated ICC was 0.647 (95% CI: 0.487-0.779) for maximum CIMT, and 0.758 (95% CI: 0.632- 0.854) for mean CIMT. In Bland Altman plot, most observed values were distributed within mean difference ${\pm}1.96$ SD ranges. Kappa statistics of plaque presence between two centers were 0.304 (center 1 and 2), 0.507 (center 1 and 3), and 0.606 (center 2 and 3), respectively, while Fleiss kappa for overall agreement was 0.445. Conclusions: The inter-rater reliability of CIMT measurements among three clinical centers turned out to be high, and the agreement of measuring carotid plaque presence was fair.

• Journal of radiological science and technology
• /
• v.40 no.2
• /
• pp.187-195
• /
• 2017

Early diagnosis for upper facial trauma is difficult by using the standard Water's view (S-Water's) in general radiograph due to overlapping of anatomical structures, the uncertainty of patient positioning, and specific patients with obese, pediatric, old, or high-risk. The purpose of this study was to analyze appropriate exposure angles through a comparison of two different protocols (S-Water's vs. reverse Water's view (R-Water's)) by using a head phantom. A head phantom and general radiograph with 75 kVp, 400 mA, 45 ms 18 mAs, and SID 100 cm. Images of R-Water's were obtained by different angles in the range of $0^{\circ}$ to $50^{\circ}$, which adjusted an angle at 1 degree interval in supine position. Survey elements were developed and three observers were evaluated with four elements including the maxillary sinus, zygomatic arch, petrous ridge, and image distortion. Statistical significant analysis were used the Krippendorff's alpha and Fleiss' kappa. The intra-class correlation (ICC) coefficient for three observers were high with maxillary, 0.957 (0.903, 0.995); zygomatic arch, 0.939 (0.866, 0.987); petrous ridge, 0.972 (0.897, 1.000); and image distortion, 0.949 (0.830, 1.000). The high-quality image (HI) and perfect agreement (PA) for acquired exposure angles were high in range of the maxillary sinus ($36^{\circ}-44^{\circ}C$), zygomatic arch ($33^{\circ}-40^{\circ}$), petrous ridge ($32^{\circ}-50^{\circ}$), and image distortion ($44^{\circ}-50^{\circ}$). Consequently, an appropriate exposure angles for the R-Water's view in the supine position for patients with facial trauma are in the from $36^{\circ}$ to $40^{\circ}$ in this phantom study. The results of this study will be helpful for the rapid diagnosis of facial fractures by simple radiography.