Browse > Article
http://dx.doi.org/10.5762/KAIS.2011.12.1.318

Reliability of Standardized Patients as Raters in Objective Structured Clinical Examination  

Son, Hee-Jeong (Department of Anesthesiology, School of Medicine, Kangwon National University)
Moon, Joong-Bum (Emergency Medicine, School of Medicine, Kangwon National University)
Lee, Hyang-Ah (Obstetrics & Gynecology, School of Medicine, Kangwon National University)
Roh, Hye-Rin (Surgery, School of Medicine, Kangwon National University)
Publication Information
Journal of the Korea Academia-Industrial cooperation Society / v.12, no.1, 2011 , pp. 318-326 More about this Journal
Abstract
The purpose of this study is to investigate whether standardized patient(SP) can be used as a reliable examiner in Objective Structured Clinical Examination(OSCE). 4 SPs and 4 faculties who have more than 2 years experience of OSCE scoring were selected. For 1 assignment 2 members of faculty and 2 SPs were designated as raters. SPs were educated for assessing 2 technical skills, male Foley catheter insertion and wound dressing, for 8 hours (4 hours / day, each topic). The definition, method, cautions and complications for each of procedural skills were covered in the education. Theoretical lectures, video learning, faculty demonstration and practical training on mannequins were employed. The 8 raters were standardized for an hour with simulated OSCE scoring using previous videos on the day before the OSCE. Each assessment was composed of 14 checklists and 1 global rate. The allotted time for each assignment was 5minutes and for evaluation time 2 minutes per student. The evaluation from the faculty and SPs were compared and analyzed with the GENOVA program. The overall generalizability coefficient (G coefficient) was 0.839 from two cases of OASTS. The reliability of the raters was high, 0.946. The inter-rater agreement between faculty group and SP group was 0.949 for checklist and 0.908 for global rating. Therefore SPs can play a role of raters in OSCE for procedural skills, if they are given the appropriate training.
Keywords
OSCE; Procedural skill; Standardized patient; Generalizability;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Regehr G, MacRae H, Reznick RK, Szalay D, "Comparing the psychometric properties of checklists and global scales for assessing performance on an OSCE-format Examination", Acad Med, 73(9), pp. 993-997, 1998.   DOI
2 Friedman Z, Siddiqui N, Katznelson R, Devito I, Davies S, "Experience is not enough: repeated breaches in epidural anesthesia aseptic technique by novice operators despite improved skill", Anesthesiology, 108, pp. 914-920, 2008.   DOI
3 Newble DI, Hoare J, Sheldrake PF, "The selection and training of examiners for clinical examinations",. Med Edu, 14(5), pp. 345-349, 1980.   DOI
4 Heine N, Garman K, Wallace P, Bartos R, Richards A, "An analysis of standardized patient checklist errors and their effect on student scores", Med Educ, 37, pp. 99-104, 2003.   DOI
5 Humphrey-Murto S, Smee S, Touchie C, Wood TJ, Blackmore DE, "A comparison of physician examiners and trained assessors in a high-stakes OSCE setting", Acad Med, 80(10), s59-s62, 2005.   DOI
6 Downing SM, "Reliability: on the reproducibility of assessment data", Med Educ, 38, pp. 1006-1012, 2004.   DOI
7 성태제, "타당도와 신뢰도", 학지사, pp 137-163, 2002.
8 Bullock G, kovacs G, Macdonald K, Story BA, "Evaluating procedural skills competence: inter-rater reliability of expert and non-expert observers", Acad Med, 74(1), pp. 76-78, 1999.   DOI
9 강승호, "신뢰도", 교육과학사, pp 73-80, 2004.
10 Crick JE, Brennan RL, "Manual of GENOVA: A generalized analysis of variance system", American College Testing Program, USA, 1983.
11 Martin JA, Reznick RK, Rothman A, Tamblyn RM, Regehr G, "Who should rate candidates in an objective structured clinical examination?", Acad Med, 71, pp. 170-175, 1996.   DOI
12 Stillman PL, "Technical issues: Logistics", Acad Med, 68, pp. 464-468, 1993.   DOI
13 Peggy Wallace, "Coaching Standardized Patients for Use in the Assessment of Clinical Competence", Springer Publishing Company. 2007.
14 Harden RM, Stevenson M, Downie WW, Wilson GM, "Assessment of clinical competence using objective structured examination", BMJ, 1, pp. 447-451, 1975.   DOI
15 Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M, "Objective structured assessment of technical skill (OSATS) for surgical residents", Br J Surg, 84, pp. 273-278, 1997.   DOI   ScienceOn
16 Friedlich M, Wood T, Regehr G, Hurst C, Shamji F, "Structured assessment of minor surgical skills (SAMSS) for clinical clerks", Acad Med, 77(10), S39-S41, 2002.   DOI
17 Goff BA, Lentz GM, Lee D, Houmard B, Mandel LS, "Development of an objective structured assessment of technical skills for obstetric and gynecology residents", Obstet Gynecol, 96, pp. 146-150, 2000.   DOI
18 Morgan PJ, Cleave-Hogg D, Guest CB, "A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator", Acad Med, 76(10), pp. 1053-1055, 2001.   DOI
19 Bould MD, Crabtree NA, Naik VN, "Assessment of procedural skills in anaesthesia", Br J Anaesth, 103, pp. 472-483, 2009.   DOI   ScienceOn
20 Lammers RL, Davenport M, Korley F, et al, "Teaching and assessing procedural skills using simulation: metrics and methodology", Acad Emerg Med, 15, pp. 1079-87, 2008.   DOI
21 Barrows HS, "An overview of the uses of standardized patients for teaching and evaluating clinical skills", Acad Med, 68(6), pp. 443-451, 1993.   DOI
22 Regehr G, Freeman R, Robb A, Misshiha N, Heisey R, "OSCE performance evaluations made by standardized patients: comparing checklist and global rating scores", Acad Med, 10(Oct suppl), S135-S137, 1999.
23 Barrows HS, Abrahamson S, "The programmed patient: a technique for appraising student performance in clinical neurology", J Med Educ, 39, pp. 802-805, 1964.
24 Lee B, "Recent world trend in performance-based assessments and application of the standardized patient program in Korean medical education", Korean J Med Educ, 12(2), pp. 377-392, 2000.
25 Stillman PL, Ruggill JS, Sabers DL, "The use of practical instructors to evaluate a complete physical examination", Eval Heal Prof, 1, pp. 49-54, 1978.   DOI
26 Whelan GP, Boulet JR, McKinley DW, Norcini JJ, van Zanten M, Hambleton RK, Burdick WP, Peitzman SJ, "Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA${\circledR}$)", Med Teach, 27(3), pp. 200-20, 2005.   DOI