Korean Journal of Radiology

  • 제11권2호
  • /
  • Pages.149-155
  • /
  • 2010
  • /
  • 1229-6929(pISSN)
  • /
  • 2005-8330(eISSN)
대한영상의학회 (The Korean Society of Radiology)
권호 표지
DOI QR코드

DOI QR Code

Observer Variability and the Performance between Faculties and Residents: US Criteria for Benign and Malignant Thyroid Nodules

  • Kim, Sung-Hun (Department of Radiology, College of Medicine, The Catholic University of Korea) ;
  • Park, Chang-Suk (Department of Radiology, College of Medicine, The Catholic University of Korea) ;
  • Jung, So-Lyung (Department of Radiology, College of Medicine, The Catholic University of Korea) ;
  • Kang, Bong-Joo (Department of Radiology, College of Medicine, The Catholic University of Korea) ;
  • Kim, Jee-Young (Department of Radiology, College of Medicine, The Catholic University of Korea) ;
  • Choi, Jae-Jung (Department of Radiology, College of Medicine, The Catholic University of Korea) ;
  • Kim, Ye-Il (Department of Radiology, College of Medicine, The Catholic University of Korea) ;
  • Oh, Jin-Kyung (Department of Radiology, College of Medicine, The Catholic University of Korea) ;
  • Oh, Jung-Suk (Department of Radiology, College of Medicine, The Catholic University of Korea) ;
  • Kim, Han-Na (Department of Radiology, College of Medicine, The Catholic University of Korea) ;
  • Jeong, Seung-Hee (Department of Preventive Medicine, College of Medicine, The Catholic University of Korea) ;
  • Yim, Hyeon-Woo (Department of Preventive Medicine, College of Medicine, The Catholic University of Korea)
  • 투고 : 2009.08.28
  • 심사 : 2009.11.18
  • 발행 : 2010.04.01
⟨ 이전 논문 다음 논문 ⟩

초록

Objective: To evaluate the interobserver variability and performance in the interpretation of ultrasonographic (US) findings of thyroid nodules. Materials and Methods: 72 malignant nodules and 61 benign nodules were enrolled as part of this study. Five faculty radiologists and four residents independently performed a retrospective analysis of the US images. The observers received one training session after the first interpretation and then performed a secondary interpretation. Agreement was analyzed by Cohen's kappa statistic. Degree of performance was analyzed using receiver operating characteristic (ROC) curves. Results: Agreement between the faculties was fair-to-good for all criteria; however, between residents, agreement was poor-to-fair. The area under the ROC curves was 0.72, 0.62, and 0.60 for the faculties, senior residents, and junior residents, respectively. There was a significant difference in performance between the faculties and the residents (p < 0.05). There was a significant increase in the agreement for some criteria in the faculties and the senior residents after the training session, but no significant increase in the junior residents. Conclusion: Independent reporting of thyroid US performed by residents is undesirable. A continuous and specialized resident training is essential to enhance the degree of agreement and performance.

키워드

참고문헌

  1. Frates MC, Benson CB, Charboneau JW, Cibas ES, Clark OH, Coleman BG, et al. Management of thyroid nodules detected at US: Society of Radiologists in Ultrasound consensus conference statement. Radiology 2005;237:794-800 https://doi.org/10.1148/radiol.2373050220
  2. Kim EK, Park CS, Chung WY, Oh KK, Kim DI, Lee JT, et al. New sonographic criteria for recommending fine-needle aspiration biopsy of nonpalpable solid nodules of the thyroid. AJR Am J Roentgenol 2002;178:687-691 https://doi.org/10.2214/ajr.178.3.1780687
  3. Chan BK, Desser TS, McDougall IR, Weigel RJ, Jeffrey RB Jr. Common and uncommon sonographic features of papillary thyroid carcinoma. J Ultrasound Med 2003;22:1083-1090 https://doi.org/10.7863/jum.2003.22.10.1083
  4. Iannuccilli JD, Cronan JJ, Monchik JM. Risk for malignancy of thyroid nodules as assessed by sonographic criteria: the need for biopsy. J Ultrasound Med 2004;23:1455-1464 https://doi.org/10.7863/jum.2004.23.11.1455
  5. Shimura H, Haraguchi K, Hiejima Y, Fukunari N, Fujimoto Y, Katagiri M, et al. Distinct diagnostic criteria for ultrasonographic examination of papillary thyroid carcinoma: a multicenter study. Thyroid 2005;15:251-258 https://doi.org/10.1089/thy.2005.15.251
  6. Moon WJ, Jung SL, Lee JH, Na DG, Baek JH, Lee YH, et al. Benign and malignant thyroid nodules: US differentiation-multicenter retrospective study. Radiology 2008;247:762-770 https://doi.org/10.1148/radiol.2473070944
  7. Nam-Goong IS, Kim HY, Gong G, Lee HK, Hong SJ, Kim WB, et al. Ultrasonography-guided fine-needle aspiration of thyroid incidentaloma: correlation with pathological findings. Clin Endocrinol (Oxf) 2004;60:21-28 https://doi.org/10.1046/j.1365-2265.2003.01912.x
  8. Frates MC, Benson CB, Doubilet PM, Kunreuther E, Contreras M, Cibas ES, et al. Prevalence and distribution of carcinoma in patients with solitary and multiple thyroid nodules on sonography. J Clin Endocrinol Metab 2006;91:3411-3417 https://doi.org/10.1210/jc.2006-0690
  9. Thyroid Study Group of the Korean Society of Neuroradiology and Head and Neck Radiology (KSNRHNR). Thyroid gland: imaging diagnosis and intervention, 1st ed. Seoul: Ilchokak, 2008
  10. Fleiss JL. Statistical methods for rates and proportions, 2nd ed. New York: John Wiley & Sons Inc., 1981
  11. Cohen J. A coefficient of agreement for nominal scales. Educat Psychol Meas 1960;20:37-46 https://doi.org/10.1177/001316446002000104
  12. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-174 https://doi.org/10.2307/2529310
  13. Wysoki MG, Nassar CJ, Koenigsberg RA, Novelline RA, Faro SH, Faerber EN. Head trauma: CT scan interpretation by radiology residents versus staff radiologists. Radiology 1998;208:125-128 https://doi.org/10.1148/radiology.208.1.9646802
  14. Erly WK, Berger WG, Krupinski E, Seeger JF, Guisto JA. Radiology resident evaluation of head CT scan orders in the emergency department. AJNR Am J Neuroradiol 2002;23:103-107
  15. Filippi CG, Schneider B, Burbank HN, Alsofrom GF, Linnell G, Ratkovits B. Discrepancy rates of radiology resident interpretations of on-call neuroradiology MR imaging studies. Radiology 2008;249:972-979 https://doi.org/10.1148/radiol.2493071543
  16. Carney E, Kempf J, DeCarvalho V, Yudd A, Nosher J. Preliminary interpretations of after-hours CT and sonography by radiology residents versus final interpretations by body imaging radiologists at a level 1 trauma center. AJR Am J Roentgenol 2003;181:367-373 https://doi.org/10.2214/ajr.181.2.1810367
  17. Yavas US, Calisir C, Ozkan IR. The interobserver agreement between residents and experienced radiologists for detecting pulmonary embolism and DVT with using CT pulmonary angiography and indirect CT venography. Korean J Radiol 2008;9:498-502 https://doi.org/10.3348/kjr.2008.9.6.498
  18. Wienke JR, Chong WK, Fielding JR, Zou KH, Mittelstaedt CA. Sonographic features of benign thyroid nodules: interobserver reliability and overlap with malignancy. J Ultrasound Med 2003;22:1027-1031 https://doi.org/10.7863/jum.2003.22.10.1027
  19. Gur D, Bandos AI, Cohen CS, Hakim CM, Hardesty LA, Ganott MA, et al. The "laboratory" effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 2008;249:47-53 https://doi.org/10.1148/radiol.2491072025

피인용 문헌

  1. Radiofrequency Ablation of Thyroid Nodules: Basic Principles and Clinical Application vol.2012, pp.None, 2012, https://doi.org/10.1155/2012/919650
  2. Radiofrequency Ablation of Benign Thyroid Nodules and Recurrent Thyroid Cancers: Consensus Statement and Recommendations vol.13, pp.2, 2012, https://doi.org/10.3348/kjr.2012.13.2.117
  3. Man to man training: Can it help improve the diagnostic performances and interobserver variabilities of thyroid ultrasonography in residents? vol.81, pp.3, 2010, https://doi.org/10.1016/j.ejrad.2011.11.011
  4. Proper Indication of BRAF V600E Mutation Testing in Fine-Needle Aspirates of Thyroid Nodules vol.8, pp.5, 2010, https://doi.org/10.1371/journal.pone.0064505
  5. Preoperative Assessment of Extrathyroidal Extension of Papillary Thyroid Carcinoma : Comparison of 2- and 3-Dimensional Sonography vol.33, pp.5, 2010, https://doi.org/10.7863/ultra.33.5.819
  6. Diagnostic Role of Conventional Ultrasonography and Shearwave Elastography in Asymptomatic Patients with Diffuse Thyroid Disease: Initial Experience with 57 Patients vol.55, pp.1, 2010, https://doi.org/10.3349/ymj.2014.55.1.247
  7. Application of the Thyroid Imaging Reporting and Data System in thyroid ultrasonography interpretation by less experienced physicians vol.33, pp.1, 2010, https://doi.org/10.14366/usg.13016
  8. Diagnostic Performance of Ultrasound and Ultrasound Elastography with Respect to Physician Experience vol.40, pp.5, 2010, https://doi.org/10.1016/j.ultrasmedbio.2013.10.005
  9. Vascular and interventional radiology radiofrequency ablation of benign thyroid nodules and recurrent thyroid cancers: literature review vol.119, pp.7, 2010, https://doi.org/10.1007/s11547-014-0411-2
  10. Can Ultrasound Be as a Surrogate Marker for Diagnosing a Papillary Thyroid Cancer? Comparison with BRAF Mutation Analysis vol.55, pp.4, 2010, https://doi.org/10.3349/ymj.2014.55.4.871
  11. Ex Vivo Estimation of Photoacoustic Imaging for Detecting Thyroid Microcalcifications vol.9, pp.11, 2010, https://doi.org/10.1371/journal.pone.0113358
  12. All Thyroid Ultrasound Evaluations are Not Equal: Sonographers Specialized in Thyroid Cancer Correctly Label Clinical N0 Disease in Well Differentiated Thyroid Cancer vol.22, pp.2, 2015, https://doi.org/10.1245/s10434-014-4089-4
  13. Application of Texture Analysis in the Differential Diagnosis of Benign and Malignant Thyroid Nodules: Comparison With Gray-Scale Ultrasound and Elastography vol.205, pp.3, 2010, https://doi.org/10.2214/ajr.14.13825
  14. Off-site evaluation of three-dimensional ultrasound for the diagnosis of thyroid nodules: comparison with two-dimensional ultrasound vol.26, pp.10, 2010, https://doi.org/10.1007/s00330-015-4193-2
  15. Application of Thyroid Imaging Reporting and Data System in the Ultrasound Assessment of Thyroid Nodules According to Physician Experience vol.32, pp.2, 2016, https://doi.org/10.1097/ruq.0000000000000189
  16. Evaluation of Underlying Lymphocytic Thyroiditis With Histogram Analysis Using Grayscale Ultrasound Images vol.35, pp.3, 2010, https://doi.org/10.7863/ultra.15.04014
  17. Quantitative Evaluation for Differentiating Malignant and Benign Thyroid Nodules Using Histogram Analysis of Grayscale Sonograms vol.35, pp.4, 2016, https://doi.org/10.7863/ultra.15.05055
  18. Validation of the modified 4‐tiered categorization system through comparison with the 5‐tiered categorization system of the 2015 American Thyroid Association guidelines for classifying sma vol.39, pp.11, 2010, https://doi.org/10.1002/hed.24888
  19. Qualitative and Semiquantitative Elastography for the Diagnosis of Intermediate Suspicious Thyroid Nodules Based on the 2015 American Thyroid Association Guidelines : Elastography for the Diagnosis of vol.37, pp.4, 2010, https://doi.org/10.1002/jum.14449
  20. Competency‐based assessment in surgeon‐performed head and neck ultrasonography: A validity study vol.128, pp.6, 2018, https://doi.org/10.1002/lary.26841
  21. Interobserver Variability of Sonographic Features Used in the American College of Radiology Thyroid Imaging Reporting and Data System vol.211, pp.1, 2010, https://doi.org/10.2214/ajr.17.19192
  22. Interobserver agreement of various thyroid imaging reporting and data systems vol.7, pp.1, 2010, https://doi.org/10.1530/ec-17-0336
  23. Diagnostic Value of Machine Learning-Based Quantitative Texture Analysis in Differentiating Benign and Malignant Thyroid Nodules vol.2019, pp.None, 2019, https://doi.org/10.1155/2019/6328329
  24. Deep convolutional neural network for the diagnosis of thyroid nodules on ultrasound vol.41, pp.4, 2010, https://doi.org/10.1002/hed.25415
  25. Computer-aided diagnosis system for thyroid nodules on ultrasonography: diagnostic performance and reproducibility based on the experience level of operators vol.29, pp.4, 2010, https://doi.org/10.1007/s00330-018-5772-9
  26. Differentiation of thyroid nodules on US using features learned and extracted from various convolutional neural networks vol.9, pp.1, 2019, https://doi.org/10.1038/s41598-019-56395-x
  27. Computer-Aided Diagnosis System for the Evaluation of Thyroid Nodules on Ultrasonography: Prospective Non-Inferiority Study according to the Experience Level of Radiologists vol.21, pp.3, 2010, https://doi.org/10.3348/kjr.2019.0581
  28. A Clinical Assessment of an Ultrasound Computer-Aided Diagnosis System in Differentiating Thyroid Nodules With Radiologists of Different Diagnostic Experience vol.10, pp.None, 2010, https://doi.org/10.3389/fonc.2020.557169
  29. Interobserver agreement and efficacy of consensus reading in Kwak-, EU-, and ACR-thyroid imaging recording and data systems and ATA guidelines for the ultrasound risk stratification of thyroid nodules vol.67, pp.1, 2010, https://doi.org/10.1007/s12020-019-02134-1
  30. Inter- and Intraobserver Agreement in the Assessment of Thyroid Nodule Ultrasound Features and Classification Systems: A Blinded Multicenter Study vol.30, pp.2, 2010, https://doi.org/10.1089/thy.2019.0360
  31. The Thyroid Nodule Conundrum: Evaluate or Leave it Alone? vol.105, pp.3, 2010, https://doi.org/10.1210/clinem/dgz124
  32. Interreader Concordance of the TI-RADS: Impact of Radiologist Experience vol.214, pp.5, 2010, https://doi.org/10.2214/ajr.19.21913
  33. False-Positive Malignant Diagnosis of Nodule Mimicking Lesions by Computer-Aided Thyroid Nodule Analysis in Clinical Ultrasonography Practice vol.10, pp.6, 2010, https://doi.org/10.3390/diagnostics10060378
  34. Computer-aided diagnostic system for thyroid nodule sonographic evaluation outperforms the specificity of less experienced examiners vol.23, pp.2, 2020, https://doi.org/10.1007/s40477-020-00453-y
  35. The Usefulness of the Thyroid Imaging Reporting and Data System in Determining Thyroid Malignancy vol.130, pp.8, 2010, https://doi.org/10.1002/lary.28476
  36. Interobserver Variability of Ultrasound Features Based on American College of Radiology Thyroid Imaging Reporting and Data System Lexicon in American College of Radiology Thyroid Imaging Reporting and vol.37, pp.4, 2010, https://doi.org/10.1097/ruq.0000000000000512
  37. 2021 Korean Thyroid Imaging Reporting and Data System and Imaging-Based Management of Thyroid Nodules: Korean Society of Thyroid Radiology Consensus Statement and Recommendations vol.22, pp.None, 2021, https://doi.org/10.3348/kjr.2021.0713
  38. Update on ACR TI-RADS: Successes, Challenges, and Future Directions, From the AJR Special Series on Radiology Reporting and Data Systems vol.216, pp.3, 2021, https://doi.org/10.2214/ajr.20.24608
  39. Deep learning-based artificial intelligence model to assist thyroid nodule diagnosis and management: a multicentre diagnostic study vol.3, pp.4, 2010, https://doi.org/10.1016/s2589-7500(21)00041-8
  40. Inter-Reader Agreement of ATA Sonographic Risk in Thyroid Nodules with Bethesda Category III Indeterminate Cytology vol.2, pp.2, 2021, https://doi.org/10.3390/endocrines2020009
  41. Implications of US radiomics signature for predicting malignancy in thyroid nodules with indeterminate cytology vol.31, pp.7, 2010, https://doi.org/10.1007/s00330-020-07670-3
  42. A practical approach for the management of small thyroid nodules referred for biopsy vol.75, pp.11, 2010, https://doi.org/10.1111/ijcp.14757