Korean Journal of Radiology

  • 제23권10호
  • /
  • Pages.1009-1018
  • /
  • 2022
  • /
  • 1229-6929(pISSN)
  • /
  • 2005-8330(eISSN)
대한영상의학회 (The Korean Society of Radiology)
권호 표지
DOI QR코드

DOI QR Code

Artificial Intelligence-Based Identification of Normal Chest Radiographs: A Simulation Study in a Multicenter Health Screening Cohort

  • Hyunsuk Yoo (Lunit Inc) ;
  • Eun Young Kim (Department of Radiology, Gil Medical Center, Gachon University College of Medicine) ;
  • Hyungjin Kim (Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital) ;
  • Ye Ra Choi (Department of Radiology, Seoul National University-Seoul Metropolitan Government Boramae Medical Center) ;
  • Moon Young Kim (Department of Radiology, Seoul National University-Seoul Metropolitan Government Boramae Medical Center) ;
  • Sung Ho Hwang (Department of Radiology, Korea University Anam Hospital) ;
  • Young Joong Kim (Department of Radiology, Konyang University Hospital, Konyang University College of Medicine) ;
  • Young Jun Cho (Department of Radiology, Konyang University Hospital, Konyang University College of Medicine) ;
  • Kwang Nam Jin (Department of Radiology, Seoul National University-Seoul Metropolitan Government Boramae Medical Center)
  • 투고 : 2022.03.20
  • 심사 : 2022.08.12
  • 발행 : 2022.10.01
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

Objective: This study aimed to investigate the feasibility of using artificial intelligence (AI) to identify normal chest radiography (CXR) from the worklist of radiologists in a health-screening environment. Materials and Methods: This retrospective simulation study was conducted using the CXRs of 5887 adults (mean age ± standard deviation, 55.4 ± 11.8 years; male, 4329) from three health screening centers in South Korea using a commercial AI (Lunit INSIGHT CXR3, version 3.5.8.8). Three board-certified thoracic radiologists reviewed CXR images for referable thoracic abnormalities and grouped the images into those with visible referable abnormalities (identified as abnormal by at least one reader) and those with clearly visible referable abnormalities (identified as abnormal by at least two readers). With AI-based simulated exclusion of normal CXR images, the percentages of normal images sorted and abnormal images erroneously removed were analyzed. Additionally, in a random subsample of 480 patients, the ability to identify visible referable abnormalities was compared among AI-unassisted reading (i.e., all images read by human readers without AI), AI-assisted reading (i.e., all images read by human readers with AI assistance as concurrent readers), and reading with AI triage (i.e., human reading of only those rendered abnormal by AI). Results: Of 5887 CXR images, 405 (6.9%) and 227 (3.9%) contained visible and clearly visible abnormalities, respectively. With AI-based triage, 42.9% (2354/5482) of normal CXR images were removed at the cost of erroneous removal of 3.5% (14/405) and 1.8% (4/227) of CXR images with visible and clearly visible abnormalities, respectively. In the diagnostic performance study, AI triage removed 41.6% (188/452) of normal images from the worklist without missing visible abnormalities and increased the specificity for some readers without decreasing sensitivity. Conclusion: This study suggests the feasibility of sorting and removing normal CXRs using AI with a tailored cut-off to increase efficiency and reduce the workload of radiologists.

키워드

과제정보

Lunit provided technical support in building a customized web-based image review tool for this study. We thank Soo Young Kwak for providing advice on statistical interpretation of the data and Sang Hyup Lee for reviewing the images in the internal validation set.

참고문헌

  1. Speets AM, van der Graaf Y, Hoes AW, Kalmijn S, Sachs AP, Rutten MJ, et al. Chest radiography in general practice: indications, diagnostic yield and consequences for patient management. Br J Gen Pract 2006;56:574-578 
  2. Watanabe Y, Nakagawa T, Fukai K, Honda T, Furuya H, Hayashi T, et al. Descriptive study of chest x-ray examination in mandatory annual health examinations at the workplace in Japan. PLoS One 2022;17:e0262404 
  3. Basi SK, Marrie TJ, Huang JQ, Majumdar SR. Patients admitted to hospital with suspected pneumonia and normal chest radiographs: epidemiology, microbiology, and outcomes. Am J Med 2004;117:305-311 
  4. Pinsky PF, Freedman M, Kvale P, Oken M, Caporaso N, Gohagan J. Abnormalities on chest radiograph reported in subjects in a cancer screening trial. Chest 2006;130:688-693 
  5. Joshi R, Patil S, Kalantri S, Schwartzman K, Menzies D, Pai M. Prevalence of abnormal radiological findings in health care workers with latent tuberculosis infection and correlations with T cell immune response. PLoS One 2007;2:e805 
  6. Harolds JA, Parikh JR, Bluth EI, Dutton SC, Recht MP. Burnout of radiologists: frequency, risk factors, and remedies: a report of the ACR Commission on Human Resources. J Am Coll Radiol 2016;13:411-416 
  7. Courtiol P, Maussion C, Moarii M, Pronier E, Pilcer S, Sefta M, et al. Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nat Med 2019;25:1519-1525 
  8. Hwang EJ, Nam JG, Lim WH, Park SJ, Jeong YS, Kang JH, et al. Deep learning for chest radiograph diagnosis in the emergency department. Radiology 2019;293:573-580 
  9. Hwang EJ, Park S, Jin KN, Kim JI, Choi SY, Lee JH, et al. Development and validation of a deep learning-based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open 2019;2:e191095 
  10. Jang S, Song H, Shin YJ, Kim J, Kim J, Lee KW, et al. Deep learning-based automatic detection algorithm for reducing overlooked lung cancers on chest radiographs. Radiology 2020;296:652-661 
  11. Khan FA, Majidulla A, Tavaziva G, Nazish A, Abidi SK, Benedetti A, et al. Chest x-ray analysis with deep learning-based software as a triage test for pulmonary tuberculosis: a prospective study of diagnostic accuracy for culture-confirmed disease. Lancet Digit Health 2020;2:e573-e581 
  12. Lee JH, Park S, Hwang EJ, Goo JM, Lee WY, Lee S, et al. Deep learning-based automated detection algorithm for active pulmonary tuberculosis on chest radiographs: diagnostic performance in systematic screening of asymptomatic individuals. Eur Radiol 2021;31:1069-1080 
  13. Lee JH, Sun HY, Park S, Kim H, Hwang EJ, Goo JM, et al. Performance of a deep learning algorithm compared with radiologic interpretation for lung cancer detection on chest radiographs in a health screening population. Radiology 2020;297:687-696 
  14. Nam JG, Park S, Hwang EJ, Lee JH, Jin KN, Lim KY, et al. Development and validation of deep learning-based automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology 2019;290:218-228 
  15. Sim Y, Chung MJ, Kotter E, Yune S, Kim M, Do S, et al. Deep convolutional neural network-based software improves radiologist detection of malignant lung nodules on chest radiographs. Radiology 2020;294:199-209 
  16. Yoo H, Kim KH, Singh R, Digumarthy SR, Kalra MK. Validation of a deep learning algorithm for the detection of malignant pulmonary nodules in chest radiographs. JAMA Netw Open 2020;3:e2017135 
  17. Recht M, Bryan RN. Artificial intelligence: threat or boon to radiologists? J Am Coll Radiol 2017;14:1476-1480 
  18. Thrall JH, Li X, Li Q, Cruz C, Do S, Dreyer K, et al. Artificial intelligence and machine learning in radiology: opportunities, challenges, pitfalls, and criteria for success. J Am Coll Radiol 2018;15(3 Pt B):504-508 
  19. Annarumma M, Withey SJ, Bakewell RJ, Pesce E, Goh V, Montana G. Automated triaging of adult chest radiographs with deep artificial neural networks. Radiology 2019;291:272 
  20. Park SH. Diagnostic case-control versus diagnostic cohort studies for clinical validation of artificial intelligence algorithm performance. Radiology 2019;290:272-273 
  21. Kim EY, Kim YJ, Choi WJ, Lee GP, Choi YR, Jin KN, et al. Performance of a deep-learning algorithm for referable thoracic abnormalities on chest radiographs: a multicenter study of a health screening cohort. PLoS One 2021;16:e0246472 
  22. Hansell DM, Bankier AA, MacMahon H, McLoud TC, Muller NL, Remy J. Fleischner society: glossary of terms for thoracic imaging. Radiology 2008;246:697-722 
  23. Johnson AEW, Pollard TJ, Berkowitz SJ, Greenbaum NR, Lungren MP, Deng CY, et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with freetext reports. Sci Data 2019;6:317 
  24. Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers R. ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017 Jul 21-26; Honolulu, HI, USA: IEEE; 2017. p.2097-2106 
  25. Bramer GR. International statistical classification of diseases and related health problems. Tenth revision. World Health Stat Q 1988;41:32-36 
  26. Dembrower K, Wahlin E, Liu Y, Salim M, Smith K, Lindholm P, et al. Effect of artificial intelligence-based triaging of breast cancer screening mammograms on cancer detection and radiologist workload: a retrospective simulation study. Lancet Digit Health 2020;2:e468-e474 
  27. Lang K, Dustler M, Dahlblom V, Akesson A, Andersson I, Zackrisson S. Identifying normal mammograms in a large screening population using artificial intelligence. Eur Radiol 2021;31:1687-1692 
  28. Yala A, Schuster T, Miles R, Barzilay R, Lehman C. A deep learning model to triage screening mammograms: a simulation study. Radiology 2019;293:38-46 
  29. Youk JH, Kim EK. Research highlight: artificial intelligence for ruling out negative examinations in screening breast MRI. Korean J Radiol 2022;23:153-155