Journal of the Korean Society of Radiology (대한영상의학회지)

The Korean Society of Radiology (대한영상의학회)
권호 표지
DOI QR코드

DOI QR Code

Construction of a Standard Dataset for Liver Tumors for Testing the Performance and Safety of Artificial Intelligence-Based Clinical Decision Support Systems

인공지능 기반 임상의학 결정 지원 시스템 의료기기의 성능 및 안전성 검증을 위한 간 종양 표준 데이터셋 구축

  • Seung-seob Kim (Department of Radiology and Research Institute of Radiological Science, Severance Hospital, Yonsei University College of Medicine) ;
  • Dong Ho Lee (Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine) ;
  • Min Woo Lee (Department of Radiology and Center for Imaging Science, Samsung Medical Center, Sungkyunkwan University School of Medicine) ;
  • So Yeon Kim (Department of Radiology, Asan Medical Center, University of Ulsan College of Medicine) ;
  • Jaeseung Shin (Department of Radiology and Research Institute of Radiological Science, Severance Hospital, Yonsei University College of Medicine) ;
  • Jin‑Young Choi (Department of Radiology and Research Institute of Radiological Science, Severance Hospital, Yonsei University College of Medicine) ;
  • Byoung Wook Choi (Department of Radiology and Research Institute of Radiological Science, Severance Hospital, Yonsei University College of Medicine)
  • 김승섭 (연세대학교 의과대학 세브란스병원 영상의학과, 방사선의과학연구소) ;
  • 이동호 (서울대학교 의과대학 서울대학교병원 영상의학과) ;
  • 이민우 (성균관대학교 의과대학 삼성서울병원 영상의학과) ;
  • 김소연 (울산대학교 의과대학 서울아산병원 영상의학과) ;
  • 신재승 (연세대학교 의과대학 세브란스병원 영상의학과, 방사선의과학연구소) ;
  • 최진영 (연세대학교 의과대학 세브란스병원 영상의학과, 방사선의과학연구소) ;
  • 최병욱 (연세대학교 의과대학 세브란스병원 영상의학과, 방사선의과학연구소)
  • Received : 2020.10.13
  • Accepted : 2021.02.04
  • Published : 2021.09.01
Download PDF
⟨ Previous Next ⟩

Abstract

Purpose To construct a standard dataset of contrast-enhanced CT images of liver tumors to test the performance and safety of artificial intelligence (AI)-based algorithms for clinical decision support systems (CDSSs). Materials and Methods A consensus group of medical experts in gastrointestinal radiology from four national tertiary institutions discussed the conditions to be included in a standard dataset. Seventy-five cases of hepatocellular carcinoma, 75 cases of metastasis, and 30-50 cases of benign lesions were retrieved from each institution, and the final dataset consisted of 300 cases of hepatocellular carcinoma, 300 cases of metastasis, and 183 cases of benign lesions. Only pathologically confirmed cases of hepatocellular carcinomas and metastases were enrolled. The medical experts retrieved the medical records of the patients and manually labeled the CT images. The CT images were saved as Digital Imaging and Communications in Medicine (DICOM) files. Results The medical experts in gastrointestinal radiology constructed the standard dataset of contrast-enhanced CT images for 783 cases of liver tumors. The performance and safety of the AI algorithm can be evaluated by calculating the sensitivity and specificity for detecting and characterizing the lesions. Conclusion The constructed standard dataset can be utilized for evaluating the machine-learning-based AI algorithm for CDSS.

목적 간 종양의 조영증강 컴퓨터단층촬영(이하 CT) 영상에 관한 인공지능 알고리즘의 성능과 안전성을 검증할 수 있는 표준 테스팅 데이터셋을 구축하고자 하였다. 대상과 방법 국내 4개 3차 의료기관의 복부 영상의학 전문가 4인이 모여 간 종양 진단 알고리즘의 성능과 안전성을 검증하기 위해 표준 데이터셋이 갖춰야 할 조건을 논의하였다. 각 기관마다 간세포암 75예, 전이암 75예, 그리고 양성 병변 30-50예씩 수집하여, 총 783명 환자의 CT 영상을 대상으로 하였다. 간세포암과 전이암의 경우 병리학적으로 확진된 경우만을 대상으로 하였다. 각 기관의 복부 영상의학 전문가들이 직접 환자의 임상정보를 추출하고 CT 영상에 관한 데이터 라벨링(labeling)을 수기로 시행하였다. CT 영상은 의료용 디지털 영상 및 통신(Digital Imaging and Communications in Medicine, DICOM) 파일로 저장하였다. 결과 복부 영상의학 전문가들이 수기 데이터 라벨링을 시행한 총 783 증례의 간 종양 조영증강 CT의 표준 데이터셋을 구축하였다. 알고리즘의 성능 및 안전성은 병변의 발견 여부 및 특성화의 정확도에 대해 민감도와 특이도를 계산하여 평가할 수 있다. 결론 본 연구에서 구축한 간 종양 조영증강 CT 영상의 표준 데이터셋은 임상의학 결정 지원시스템을 위한 기계학습 기반 인공지능 알고리즘을 평가하는 데에 활용될 수 있다.

Keywords

Acknowledgement

This research was supported by a grant (18173의료평331-1[DY0002258200]) from Ministry of Food and Drug Safety in 2020.

References

  1. Wichmann JL, Willemink MJ, De Cecco CN. Artificial intelligence and machine learning in radiology: current state and considerations for routine clinical implementation. Invest Radiol 2020;55:619-627 
  2. Data Bridge Market Research. Global artificial intelligence in medical imaging market-industry trends-forecast to 2026. Available at. https://www.databridgemarketresearch.com/reports/global-artificial-intelligence-in-medical-imaging-market. Accessed Sep 22, 2020 
  3. Willemink MJ, Koszek WA, Hardell C, Wu J, Fleischmann D, Harvey H, et al. Preparing medical imaging data for machine learning. Radiology 2020;295:4-15 
  4. Oakden-Rayner L. Exploring large-scale public medical image datasets. Acad Radiol 2020;27:106-112 
  5. Park SH, Han K. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 2018;286:800-809 
  6. Weikert T, Cyriac J, Yang S, Nesic I, Parmar V, Stieltjes B. A practical guide to artificial intelligence-based image analysis in radiology. Invest Radiol 2020;55:1-7 
  7. Lloyd, K. Bias amplification in artificial intelligence systems. ArXiv preprint 2018;arXiv:1809.07842 
  8. Yu AC, Eng J. One algorithm may not fit all: how selection bias affects machine learning performance. Radiographics 2020;40:1932-1937 
  9. Song TJ, Fong Y, Cho SJ, Gonen M, Hezel M, Tuorto S, et al. Comparison of hepatocellular carcinoma in American and Asian patients by tissue array analysis. J Surg Oncol 2012;106:84-88 
  10. Pasquinelli MM, Kovitz KL, Koshy M, Menchaca MG, Liu L, Winn R, et al. Outcomes from a minority-based lung cancer screening program vs the National Lung Screening Trial. JAMA Oncol 2018;4:1291-1293 
  11. Jaeger S, Candemir S, Antani S, Wang YX, Lu PX, Thoma G. Two public chest X-ray datasets for computer-aided screening of pulmonary diseases. Quant Imaging Med Surg 2014;4:475-477 
  12. Pons E, Braun LM, Hunink MG, Kors JA. Natural language processing in radiology: a systematic review. Radiology 2016;279:329-343 
  13. Gera S, Ettel M, Acosta-Gonzalez G, Xu R. Clinical features, histology, and histogenesis of combined hepatocellular-cholangiocarcinoma. World J Hepatol 2017;9:300-309 
  14. Fasel JH, Selle D, Evertsz CJ, Terrier F, Peitgen HO, Gailloud P. Segmental anatomy of the liver: poor correlation with CT. Radiology 1998;206:151-156 
  15. FDA. Artificial intelligence and machine learning in software as a medical device. Available at. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device. Accessed Sep 24, 2020 
  16. Azer SA. Deep learning with convolutional neural networks for identification of liver masses and hepatocellular carcinoma: a systematic review. World J Gastrointest Oncol 2019;11:1218-1230 
  17. Wang W, Iwamoto Y, Han X, Chen YW, Chen Q, Liang D, et al. Classification of focal liver lesions using deep learning with fine-tuning. Proceedings of the 2018 International Conference on Digital Medicine and Image Processing; 2018 Nov; Okinawa, Japan: Association for Computing Machinery; 2018:56-60 
  18. Liang D, Lin L, Hu H, Zhang Q, Chen Q, Han X, et al. Combining convolutional and recurrent neural networks for classification of focal liver lesions in multi-phase CT images. International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer 2018:666-675 
  19. Das A, Acharya UR, Panda SS, Sabut S. Deep learning based liver cancer detection using watershed transform and Gaussian mixture model techniques. Cogn Syst Res 2019;54:165-175 
  20. Gletsos M, Mougiakakou SG, Matsopoulos GK, Nikita KS, Nikita AS, Kelekis D. A computer-aided diagnostic system to characterize CT focal liver lesions: design and optimization of a neural network classifier. IEEE Trans Inf Technol Biomed 2003;7:153-162 
  21. Cao SE, Zhang LQ, Kuang SC, Shi WQ, Hu B, Xie SD, et al. Multiphase convolutional dense network for the classification of focal liver lesions on dynamic contrast-enhanced computed tomography. World J Gastroenterol 2020;26:3660-3672 
  22. Xu Y, Lin L, Hu H, Wang D, Zhu W, Wang J, et al. Texture-specific bag of visual words model and spatial cone matching-based method for the retrieval of focal liver lesions using multiphase contrast-enhanced CT images. Int J Comput Assist Radiol Surg 2018;13:151-164 
  23. Mougiakakou SG, Valavanis IK, Nikita A, Nikita KS. Differential diagnosis of CT focal liver lesions using texture features, feature selection and ensemble driven classifiers. Artif Intell Med 2007;41:25-37 
  24. Nayantara PV, Kamath S, Manjunath KN, Rajagopal KV. Computer-aided diagnosis of liver lesions using CT images: a systematic review. Comput Biol Med 2020;127:104035 
  25. Ben-Cohen A, Klang E, Diamant I, Rozendorn N, Raskin SP, Konen E, et al. CT image-based decision support system for categorization of liver metastases into primary cancer sites: initial results. Acad Radiol 2017;24:1501-1509 
  26. Yasaka K, Akai H, Abe O, Kiryu S. Deep learning with convolutional neural network for differentiation of liver masses at dynamic contrast-enhanced CT: a preliminary study. Radiology 2018;286:887-896 
  27. Park HJ, Park B, Lee SS. Radiomics and deep learning: hepatic applications. Korean J Radiol 2020;21:387-401