• Title/Summary/Keyword: Inter-Rater reliability

Search Result 197, Processing Time 0.027 seconds

Reliability of Standardized Patients as Raters in Objective Structured Clinical Examination (객관 구조화 절차 기술 평가에서 채점자로서의 표준화환자의 신뢰도)

  • Son, Hee-Jeong;Moon, Joong-Bum;Lee, Hyang-Ah;Roh, Hye-Rin
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.12 no.1
    • /
    • pp.318-326
    • /
    • 2011
  • The purpose of this study is to investigate whether standardized patient(SP) can be used as a reliable examiner in Objective Structured Clinical Examination(OSCE). 4 SPs and 4 faculties who have more than 2 years experience of OSCE scoring were selected. For 1 assignment 2 members of faculty and 2 SPs were designated as raters. SPs were educated for assessing 2 technical skills, male Foley catheter insertion and wound dressing, for 8 hours (4 hours / day, each topic). The definition, method, cautions and complications for each of procedural skills were covered in the education. Theoretical lectures, video learning, faculty demonstration and practical training on mannequins were employed. The 8 raters were standardized for an hour with simulated OSCE scoring using previous videos on the day before the OSCE. Each assessment was composed of 14 checklists and 1 global rate. The allotted time for each assignment was 5minutes and for evaluation time 2 minutes per student. The evaluation from the faculty and SPs were compared and analyzed with the GENOVA program. The overall generalizability coefficient (G coefficient) was 0.839 from two cases of OASTS. The reliability of the raters was high, 0.946. The inter-rater agreement between faculty group and SP group was 0.949 for checklist and 0.908 for global rating. Therefore SPs can play a role of raters in OSCE for procedural skills, if they are given the appropriate training.

Appropriateness Evaluation of Hospitalization for the Cerebral Ischemia Patients (허혈성 뇌졸중 환자의 재원적절성 평가)

  • Yoem, Hyo-Young;Kim, Soon-Lae
    • Research in Community and Public Health Nursing
    • /
    • v.10 no.1
    • /
    • pp.80-92
    • /
    • 1999
  • The purpose of this study was to survey appropriateness of admission and days of care for the cerebral ischemia patients as a basis to provide an effectiveness of hospitalization. The authors retrospectively reviewed the medical records of cerebral ischemia patients in two hospitals from November 1997 to February 1998. Of 194 medical records reviewed, there were 2108 medical days. I t is used a 'Appropriateness Evaluation Protocol' previously developed by Gertman and Restuccia (1981) and translated by Department of Health Management, Seoul National University and Korea Institute for Health Services Management (1993), It was found that the 'Appropriateness Evaluation Protocol' had a high inter-rater reliability(k=.92), Statistical significant was tested by using the percentage, mean, and logistic regression by SAS 6.12. The results were as follows; 1. The appropriate admissions were 87.6%, days of care 63.4%, and the average length of stay $10.9{\pm}6.7$ days. 2. The reasons of inappropriate admissions were for work-up(75.0%) and conservative care (25.0%). Major reasons of inappropriate days of care were 'cases in which the medical purpose of hospitalization has been accomplish or can be addressed in a less setting(45.0%)', and 'cases in which there is a delay in performing the work-up or treatment which required patients is hospitalized (44.4%)'. 3. Appropriate days of care were higher as ageing. Appropriate days of care were higher in patients with lower accademic back ground than those of upper college graduates, and in the patients who enter a hospital via emergency room than out-patients department. Appropriate days of care were higher in the patient with MCA infarction, and lower in the patient with cerebellar infarction than the patient with lacunar infarction. Appropriate days of care were higher in attack first than attack above second, in nomortension patients than hypertensive, and lower in groups who engaged in semi-private room and public room than private room in hospital. Appropriate days of care were higher in shorter length of stay than longer length of stay. 4. Diagnosis, admission path, and appropriate days of care explained appropriate admissions. Diagnosis, appropriate admissions, hypertension explained appropriate days of care. According to the above results, author confirms the substantial amount of inappropriate hospital bed utilization. To reduce inappropriateness, it is necessary to develop some alternative services such as home care services or nursing home with which can be replaced inpatient services and to introduce policy such as case management which includes Critical Pathway for consistent management. And, it should be followed the further study for the effectiveness.

  • PDF

The Development of a Checklist for Quantitative Assessment of Risk Factors and Management of Cumulative Trauma Disorders : Application to Automobile Assembly Lines (누적외상성질환 위험 요인의 정량적 평가 및 관리를 위한 점검표 개발 -자동차 조립 작업을 중심으로-)

  • Lee, Yun-Keun;Kim, Hyun-Wook;Yim, Shang-Hyuk;Park, Hee-Sok
    • Journal of Korean Society of Occupational and Environmental Hygiene
    • /
    • v.11 no.1
    • /
    • pp.56-69
    • /
    • 2001
  • Objectives : This study was designed to develop and standardize a checklist for ergonomic risk factors, and to provide ergonomic guidelines for managing cumulative trauma disorders (CTDs) in automobile assembly lines. Methods : The Checklist for Ergonomic Risk Factors (CERF-1) was developed based on the results of previous studies, and then modified after performing pilot study. Information on the symptoms possibly related with CTDs was obtained using a self-reported Questionnaire from 465 automobile assembly workers. Their job conditions were examined to assess risk factors through both direct observation and video analysis. Results : Rate of detecting risky job through CERF-1 was 85.6%, and was similar to that (88.8%) by Occupational Safety and Health Adminstration(OSHA) checklist but higher than that (63.7%) by American National Standards Institute(ANSI) Z-365. Relationship of the exposure scores derived from CERF-1 with levels of symptom was greater (r=0.49) than OSHA (r=0.28) and ANSI Z-365 (r=0.22). Considering the relationship, jobs scoring higher than 16 could be classified as the Risk Job. and lower than 16 as the Low Risk Job. Sensitivity and specificity of the Risk Job were 92.5 % and 31.5 %, respectively. Odds ratio (OR) after age adjustment was 5.69 (95 % confidence interval 3.15-10.29) for the Risk Job, and these ORs were significantly different from those of the Low Risk Job. The exposure scores were Quite valid, in that the scores at the main survey were significantly correlated with those at the follow-up survey, as suggested by test-retest(r=0.88) and inter-rater reliability(r=0.80). Conclusions : The CERF-1, developed in this study, will be an efficient tool for evaluation of risk jobs for CTDs in automobile assembly lines, and can be used easily by health care providers.

  • PDF

A Classification of Car-related Mobile Apps: For App Development from a Convergence Perspective (차량용 모바일 앱의 분류: 융복합 관점의 앱 개발을 위해)

  • Zhang, Chao;Wan, Lili;Min, Daihwan
    • Journal of Digital Convergence
    • /
    • v.15 no.3
    • /
    • pp.77-86
    • /
    • 2017
  • This study selected car-related mobile apps for app developers suffering from low revenue and classified car apps assisting users in driving or managing a car. A total of 697 car apps were classified into eight categories. Most apps are in four categories: car news & information (28%), locating service (23%), car rental service (15%), safe/efficient driving service (12%). The remaining categories are buying & selling, driver's communication, maintenance management, and expenses monitoring. Many apps are simple and too similar in their main functions. Only a few apps are designed to be more comprehensive and have functions in two or more categories. For the practicality of the categorization scheme, this study checked the inter-rater reliability in two tests and got 0.886 and 0.828. The result from this study suggests functions that are not implemented yet or need to be combined. Future research will focus on identifying promising car apps or designing multi-functional car apps.

Scoring Korean Written Responses Using English-Based Automated Computer Scoring Models and Machine Translation: A Case of Natural Selection Concept Test (영어기반 컴퓨터자동채점모델과 기계번역을 활용한 서술형 한국어 응답 채점 -자연선택개념평가 사례-)

  • Ha, Minsu
    • Journal of The Korean Association For Science Education
    • /
    • v.36 no.3
    • /
    • pp.389-397
    • /
    • 2016
  • This study aims to test the efficacy of English-based automated computer scoring models and machine translation to score Korean college students' written responses on natural selection concept items. To this end, I collected 128 pre-service biology teachers' written responses on four-item instrument (total 512 written responses). The machine translation software (i.e., Google Translate) translated both original responses and spell-corrected responses. The presence/absence of five scientific ideas and three $na{\ddot{i}}ve$ ideas in both translated responses were judged by the automated computer scoring models (i.e., EvoGrader). The computer-scored results (4096 predictions) were compared with expert-scored results. The results illustrated that no significant differences in both average scores and statistical results using average scores was found between the computer-scored result and experts-scored result. The Pearson correlation coefficients of composite scores for each student between computer scoring and experts scoring were 0.848 for scientific ideas and 0.776 for $na{\ddot{i}}ve$ ideas. The inter-rater reliability indices (Cohen kappa) between computer scoring and experts scoring for linguistically simple concepts (e.g., variation, competition, and limited resources) were over 0.8. These findings reveal that the English-based automated computer scoring models and machine translation can be a promising method in scoring Korean college students' written responses on natural selection concept items.

Variation of Seasonal Groundwater Recharge Analyzed Using Landsat-8 OLI Data and a CART Algorithm (CART알고리즘과 Landsat-8 위성영상 분석을 통한 계절별 지하수함양량 변화)

  • Park, Seunghyuk;Jeong, Gyo-Cheol
    • The Journal of Engineering Geology
    • /
    • v.31 no.3
    • /
    • pp.395-432
    • /
    • 2021
  • Groundwater recharge rates vary widely by location and with time. They are difficult to measure directly and are thus often estimated using simulations. This study employed frequency and regression analysis and a classification and regression tree (CART) algorithm in a machine learning method to estimate groundwater recharge. CART algorithms are considered for the distribution of precipitation by subbasin (PCP), geomorphological data, indices of the relationship between vegetation and landuse, and soil type. The considered geomorphological data were digital elevaion model (DEM), surface slope (SLOP), surface aspect (ASPT), and indices were the perpendicular vegetation index (PVI), normalized difference vegetation index (NDVI), normalized difference tillage index (NDTI), normalized difference residue index (NDRI). The spatio-temperal distribution of groundwater recharge in the SWAT-MOD-FLOW program, was classified as group 4, run in R, sampled for random and a model trained its groundwater recharge was predicted by CART condidering modified PVI, NDVI, NDTI, NDRI, PCP, and geomorphological data. To assess inter-rater reliability for group 4 groundwater recharge, the Kappa coefficient and overall accuracy and confusion matrix using K-fold cross-validation were calculated. The model obtained a Kappa coefficient of 0.3-0.6 and an overall accuracy of 0.5-0.7, indicating that the proposed model for estimating groundwater recharge with respect to soil type and vegetation cover is quite reliable.

Stress, Social Support and Coping of Adults According to Level of Self-Efficacy (성인의 스트레스, 사회적 지원과 대처: 자기효능감 수준별 분석)

  • Young-Shin Park;Ju-Yeon Son;Ok-Ran Song
    • Korean Journal of Culture and Social Issue
    • /
    • v.23 no.2
    • /
    • pp.295-332
    • /
    • 2017
  • The main purpose of this research is to analyze stress, social support and coping behavior of adults based on their level of self-efficacy. A total of 899 adults (399 male; 500 female), each with a child attending either elementary and secondary school, participated in the study. The inter-rater reliability for the open-ended questionnaire utilized in the study was 93.4%, with a Kappa coefficient of .92. The range of Cronbach α for the variables measured through a quantitative method was .87~.92. The results were as follows: First, the representative responses to the question about their most painful stress experiences were, financial difficulties, child rearing and duties of workplace. The Lower Efficacy group, compared to the Upper Efficacy group, responded much more with financial difficulties related responses. There were significant differences in the level of stress symptoms according to level of self-efficacy. The Lower Efficacy group expressed stronger levels of stress symptoms when compared to the Upper Efficacy group. Second, in terms of social support, the participants responded that they received the most help from their family members, followed by none(self), and friends. When comparing the two efficacy groups, the Upper Efficacy group responded most frequently that they received social support from their family members, whereas the Lower Efficacy group indicated none. There were significant differences in the level of relational conflicts according to the level of self-efficacy. The Upper Efficacy group showed much less conflict in parent-child relations, spousal relations and relations with their boss, compared to the Lower Efficacy group. Third, for the type of social support participants received, the most frequent response was emotional support, followed by none, and advice. Relatively, when comparing the two groups with each other, the Lower Efficacy group responded more frequently with none, whereas for the Upper Efficacy group responded more frequently with advice. There were significant differences in the amount of emotional support received according to level of self-efficacy. The Upper Efficacy group received much more emotional support from their spouses and their bosses compared to the Lower Efficacy group. Fourth, the most frequently adopted coping style to stress was self-regulation, followed by direct problem solving, and nothing(none). The most frequent response for the Upper Efficacy group was direct problem solving, whereas for the Lower Efficacy group was nothing(none). There was a significant difference in coping efficiency to stress according to level of self-efficacy. The Upper Efficacy group coped more efficiently with stress than the Lower Efficacy group.