• Journal of Korean Medical Science
• /
• v.33 no.43
• /
• pp.239.1-239.12
• /
• 2018

Background: We described a novel multi-step retinal fundus image reading system for providing high-quality large data for machine learning algorithms, and assessed the grader variability in the large-scale dataset generated with this system. Methods: A 5-step retinal fundus image reading tool was developed that rates image quality, presence of abnormality, findings with location information, diagnoses, and clinical significance. Each image was evaluated by 3 different graders. Agreements among graders for each decision were evaluated. Results: The 234,242 readings of 79,458 images were collected from 55 licensed ophthalmologists during 6 months. The 34,364 images were graded as abnormal by at-least one rater. Of these, all three raters agreed in 46.6% in abnormality, while 69.9% of the images were rated as abnormal by two or more raters. Agreement rate of at-least two raters on a certain finding was 26.7%-65.2%, and complete agreement rate of all-three raters was 5.7%-43.3%. As for diagnoses, agreement of at-least two raters was 35.6%-65.6%, and complete agreement rate was 11.0%-40.0%. Agreement of findings and diagnoses were higher when restricted to images with prior complete agreement on abnormality. Retinal/glaucoma specialists showed higher agreements on findings and diagnoses of their corresponding subspecialties. Conclusion: This novel reading tool for retinal fundus images generated a large-scale dataset with high level of information, which can be utilized in future development of machine learning-based algorithms for automated identification of abnormal conditions and clinical decision supporting system. These results emphasize the importance of addressing grader variability in algorithm developments.

• Physical Therapy Korea
• /
• v.29 no.4
• /
• pp.262-268
• /
• 2022

Background: Virtual reality (VR) programs based on motion capture camera are the most convenient and cost-effective approaches for remote rehabilitation. Assessment of physical function is critical for providing optimal VR rehabilitation training; however, direct muscle strength measurement using camera-based kinematic data is impracticable. Therefore, it is necessary to develop a method to indirectly estimate the muscle strength of users from the value obtained using a motion capture camera. Objects: The purpose of this study was to determine whether the pedaling speed converted using the VR engine from the captured foot position data in the VR environment can be used as an indirect way to evaluate knee muscle strength, and to investigate the validity and reliability of a camera-based VR program. Methods: Thirty healthy adults were included in this study. Each subject performed a 15-second maximum pedaling test in the VR and built-in speedometer modes. In the VR speedometer mode, a motion capture camera was used to detect the position of the ankle joints and automatically calculate the pedaling speed. An isokinetic dynamometer was used to assess the isometric and isokinetic peak torques of knee flexion and extension. Results: The pedaling speeds in VR and built-in speedometer modes revealed a significantly high positive correlation (r = 0.922). In addition, the intra-rater reliability of the pedaling speed in the VR speedometer mode was good (ICC [intraclass correlation coefficient] = 0.685). The results of the Pearson correlation analysis revealed a significant moderate positive correlation between the pedaling speed of the VR speedometer and the peak torque of knee isokinetic flexion (r = 0.639) and extension (r = 0.598). Conclusion: This study suggests the potential benefits of measuring the maximum pedaling speed using 3D depth camera in a VR environment as an indirect assessment of muscle strength. However, technological improvements must be followed to obtain more accurate estimation of muscle strength from the VR cycling test.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.12 no.1
• /
• pp.318-326
• /
• 2011

The purpose of this study is to investigate whether standardized patient(SP) can be used as a reliable examiner in Objective Structured Clinical Examination(OSCE). 4 SPs and 4 faculties who have more than 2 years experience of OSCE scoring were selected. For 1 assignment 2 members of faculty and 2 SPs were designated as raters. SPs were educated for assessing 2 technical skills, male Foley catheter insertion and wound dressing, for 8 hours (4 hours / day, each topic). The definition, method, cautions and complications for each of procedural skills were covered in the education. Theoretical lectures, video learning, faculty demonstration and practical training on mannequins were employed. The 8 raters were standardized for an hour with simulated OSCE scoring using previous videos on the day before the OSCE. Each assessment was composed of 14 checklists and 1 global rate. The allotted time for each assignment was 5minutes and for evaluation time 2 minutes per student. The evaluation from the faculty and SPs were compared and analyzed with the GENOVA program. The overall generalizability coefficient (G coefficient) was 0.839 from two cases of OASTS. The reliability of the raters was high, 0.946. The inter-rater agreement between faculty group and SP group was 0.949 for checklist and 0.908 for global rating. Therefore SPs can play a role of raters in OSCE for procedural skills, if they are given the appropriate training.

• Korean Medical Education Review
• /
• v.25 no.2
• /
• pp.139-158
• /
• 2023

Defining a competent doctor is important for educating and training doctors. However, competency frameworks have rarely been validated during the process of their development in Korea. The purpose of this study was to validate the patient-centered doctor's competency framework, which had been developed by our expert working group (EWG). Two rounds of Delphi questionnaire surveys were conducted among a panel of experts on medicine and medical education. The panel members were provided with six core competencies, 17 sub-competencies, and 53 enabling competencies, and were asked to rate the importance of these competencies on a 5-point Likert scale. Between April and July 2021, a total of 28 experts completed both rounds. The data of the Delphi study were analyzed for the mean, standard deviation, median, inter-rater agreement (IRA), and content validity ratio (CVR). A CVR >0.36 and IRA ≥0.75 were deemed to indicate validity and agreement. This study found that five enabling competencies were not valid, and agreement was not reached for three sub-competencies and two enabling competencies. In consideration of CVR and the individual opinions of panel members at each session, the final competencies were extracted through consensus meetings of the EWG. The competencies were modified into six core competencies, 16 sub-competencies, and 47 enabling competencies. This study is meaningful in that it proposes patient-centered doctor's competencies enabling the development of residents' milestone competencies, an assessment system, and educational programs.