Journal of the Korean Society of Radiology (대한영상의학회지)

The Korean Society of Radiology (대한영상의학회)
권호 표지
DOI QR코드

DOI QR Code

The Latest Trends in Attention Mechanisms and Their Application in Medical Imaging

어텐션 기법 및 의료 영상에의 적용에 관한 최신 동향

  • Hyungseob Shin (Department of Electrical and Electronic Engineering, Yonsei University) ;
  • Jeongryong Lee (Department of Electrical and Electronic Engineering, Yonsei University) ;
  • Taejoon Eo (Department of Electrical and Electronic Engineering, Yonsei University) ;
  • Yohan Jun (Department of Electrical and Electronic Engineering, Yonsei University) ;
  • Sewon Kim (Department of Electrical and Electronic Engineering, Yonsei University) ;
  • Dosik Hwang (Department of Electrical and Electronic Engineering, Yonsei University)
  • 신형섭 (연세대학교 공과대학 전기전자공학과) ;
  • 이정룡 (연세대학교 공과대학 전기전자공학과) ;
  • 어태준 (연세대학교 공과대학 전기전자공학과) ;
  • 전요한 (연세대학교 공과대학 전기전자공학과) ;
  • 김세원 (연세대학교 공과대학 전기전자공학과) ;
  • 황도식 (연세대학교 공과대학 전기전자공학과)
  • Received : 2020.08.13
  • Accepted : 2020.11.07
  • Published : 2020.11.01
Download PDF
⟨ Previous Next ⟩

Abstract

Deep learning has recently achieved remarkable results in the field of medical imaging. However, as a deep learning network becomes deeper to improve its performance, it becomes more difficult to interpret the processes within. This can especially be a critical problem in medical fields where diagnostic decisions are directly related to a patient's survival. In order to solve this, explainable artificial intelligence techniques are being widely studied, and an attention mechanism was developed as part of this approach. In this paper, attention techniques are divided into two types: post hoc attention, which aims to analyze a network that has already been trained, and trainable attention, which further improves network performance. Detailed comparisons of each method, examples of applications in medical imaging, and future perspectives will be covered.

딥러닝 기술은 빅데이터 및 컴퓨팅 파워를 기반으로 최근 영상의학 분야의 연구에서 괄목할만한 성과를 이루어 내고 있다. 하지만 성능 향상을 위해 딥러닝 네트워크가 깊어질수록 그 내부의 계산 과정을 해석하기 어려워졌는데, 이는 환자의 생명과 직결되는 의료분야의 의사결정 과정에서는 매우 심각한 문제이다. 이를 해결하기 위해 "설명 가능한 인공지능 기술"이 연구되고 있으며, 그중 하나로 개발된 것이 바로 어텐션(attention) 기법이다. 본 종설에서는 이미 학습이 완료된 네트워크를 분석하기 위한 Post-hoc attention과, 네트워크 성능의 추가적인 향상을 위한 Trainable attention 두 종류의 기법에 대해 각각의 방법 및 의료 영상 연구에 적용된 사례, 그리고 향후 전망 등에 대해 자세히 다루고자 한다.

Keywords

Acknowledgement

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (2019R1A2B5B01070488). This research was results of a study on the "HPC Support" Project, supported by the 'Ministry of Science and ICT' and NIPA. This work has been supported by Y-BASE R&E Institute a Brain Korea 21, Yonsei University.

References

  1. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521:436-444
  2. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 2016;316:2402-2410
  3. Kickingereder P, Isensee F, Tursunova I, Petersen J, Neuberger U, Bonekamp D, et al. Automated quantitative tumour response assessment of MRI in neuro-oncology with artificial neural networks: a multicentre, retrospective study. Lancet Oncol 2019;20:728-740
  4. Bello GA, Dawes TJW, Duan J, Biffi C, De Marvao A, Howard LSGE, et al. Deep learning cardiac motion analysis for human survival prediction. Nat Mach Intell 2019;1:95-104
  5. Eo T, Jun Y, Kim T, Jang J, Lee HJ, Hwang D. KIKI-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images. Magn Reson Med 2018;80:2188-2201
  6. Jun Y, Eo T, Shin H, Kim T, Lee HJ, Hwang D. Parallel imaging in time-of-flight magnetic resonance angiography using deep multistream convolutional neural networks. Magn Reson Med 2019;81:3840-3853
  7. Eo T, Shin H, Jun Y, Kim T, Hwang D. Accelerating Cartesian MRI by domain-transform manifold learning in phase-encoding direction. Med Image Anal 2020;63:101689
  8. Han Y, Ye JC. Framing U-Net via deep convolutional framelets: application to sparse-view CT. IEEE Trans Med Imaging 2018;37:1418-1429
  9. Kim S, Jang H, Jang J, Lee YH, Hwang D. Deep-learned short tau inversion recovery imaging using multi-contrast MR images. Magn Reson Med 2020 [in press] doi: https://doi.org/10.1002/mrm.28327
  10. Jun Y, Eo T, Kim T, Shin H, Hwang D, Bae SH, et al. Deep-learned 3D black-blood imaging using automatic labelling technique and 3D convolutional neural networks for detecting metastatic brain tumors. Sci Rep 2018;8:9450
  11. Yala A, Schuster T, Miles R, Barzilay R, Lehman C. A deep learning model to triage screening mammograms: a simulation study. Radiology 2019;293:38-46
  12. De Fauw J, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med 2018;24:1342-1350
  13. Collins FS, Varmus H. A new initiative on precision medicine. N Engl J Med 2015;372:793-795
  14. Kang D, Park JE, Kim YH, Kim JH, Oh JY, Kim J, et al. Diffusion radiomics as a diagnostic model for atypical manifestation of primary central nervous system lymphoma: development and multicenter external validation. Neuro Oncol 2018;20:1251-1261
  15. Lao J, Chen Y, Li ZC, Li Q, Zhang J, Liu J, et al. A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme. Sci Rep 2017;7:10353
  16. Ardila D, Kiraly AP, Bharadwaj S, Choi B, Reicher JJ, Peng L, et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat Med 2019;25:954-961
  17. Schelb P, Kohl S, Radtke JP, Wiesenfarth M, Kickingereder P, Bickelhaupt S, et al. Classification of cancer at prostate MRI: deep learning versus clinical PI-RADS assessment. Radiology 2019;293:607-617
  18. Lindsey R, Daluiski A, Chopra S, Lachapelle A, Mozer M, Sicular S, et al. Deep neural network improves fracture detection by clinicians. Proc Natl Acad Sci U S A 2018;115:11591-11596
  19. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017;542:115-118
  20. Rauschecker AM, Rudie JD, Xie L, Wang J, Duong MT, Botzolakis EJ, et al. Artificial intelligence system approaching neuroradiologist-level differential diagnosis accuracy at brain MRI. Radiology 2020;295:626-637
  21. Baltruschat IM, Nickisch H, Grass M, Knopp T, Saalbach A. Comparison of deep learning approaches for multi-label chest X-ray classification. Sci Rep 2019;9:6381
  22. Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017 Jul 21-26; Honolulu, HI, USA: IEEE; 2017:2097-2106
  23. Kim H, Jung DC, Choi BW. Exploiting the vulnerability of deep learning-based artificial intelligence models in medical imaging: adversarial attacks. J Korean Soc Radiol 2019;80:259-273
  24. Jetley S, Lord NA, Lee N, Torr PH. Learn to pay attention. ArXiv Preprint 2018;arXiv:1804.02391
  25. Wen PY, Macdonald DR, Reardon DA, Cloughesy TF, Sorensen AG, Galanis E, et al. Updated response assessment criteria for high-grade gliomas: response assessment in neuro-oncology working group. J Clin Oncol 2010;28:1963-1972
  26. Theiler R, Stucki G, Schutz R, Hofer H, Seifert B, Tyndall A, et al. Parametric and non-parametric measures in the assessment of knee and hip osteoarthritis: interobserver reliability and correlation with radiology. Osteoarthritis Cartilage 1996;4:35-42
  27. Mnih V, Heess N, Graves A. Recurrent models of visual attention. Proceedings of Advances in Neural Information Processing Systems 27 (NIPS 2014); 2014 Dec 8-13; Montreal, Canada: NIPS; 2014:2204-2212
  28. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. Learning deep features for discriminative localization. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016 Jun 27-30; Las Vegas, NV, USA: IEEE; 2016:2921-2929
  29. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision (ICCV); 2017 Oct 22-29; Venice, Italy: IEEE; 2017:618-626
  30. Simonyan K, Vedaldi A, Zisserman A. Deep inside convolutional networks: visualising image classification models and saliency maps. ArXiv Preprint 2013;arXiv:1312.6034
  31. Bach S, Binder A, Montavon G, Klauschen F, Muller KR, Samek W. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One 2015;10:e0130140
  32. Bien N, Rajpurkar P, Ball RL, Irvin J, Park A, Jones E, et al. Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet. PLoS Med 2018;15:e1002699
  33. Cheng CT, Ho TY, Lee TY, Chang CC, Chou CC, Chen CC, et al. Application of a deep learning algorithm for detection and visualization of hip fractures on plain pelvic radiographs. Eur Radiol 2019;29:5469-5477
  34. Kim B, Seo J, Jeon S, Koo J, Choe J, Jeon T. Why are saliency maps noisy? Cause of and solution to noisy saliency maps. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW); 2019 Oct 27-28; Seoul, Korea (South): IEEE; 2019:4149-4157
  35. Pasa F, Golkov V, Pfeiffer F, Cremers D, Pfeiffer D. Efficient deep network architectures for fast chest X-ray tuberculosis screening and visualization. Sci Rep 2019;9:6268
  36. Bohle M, Eitel F, Weygandt M, Ritter K. Layer-wise relevance propagation for explaining deep neural network decisions in MRI-based Alzheimer's disease classification. Front Aging Neurosci 2019;11:194
  37. Radiopaedia. Rad_doc, rID 47997, "Childhood Pneumonia". Available at. https://radiopaedia.org/cases/childhood-pneumonia-1?lang=us. Published Sep 13, 2016. Accessed Aug 1, 2020
  38. Ypsilantis PP, Montana G. Learning what to look in chest X-rays with a recurrent visual attention model. ArXiv Preprint 2017;arXiv:1701.06452
  39. Guan Q, Huang Y, Zhong Z, Zheng Z, Zheng L, Yang Y. Thorax disease classification with attention guided convolutional neural network. Pattern Recognit Lett 2020;131:38-45
  40. Schlemper J, Oktay O, Schaap M, Heinrich M, Kainz B, Glocker B, et al. Attention gated networks: learning to leverage salient regions in medical images. Med Image Anal 2019;53:197-207
  41. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging 2014;34:1993-2024
  42. Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby JS, et al. Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci Data 2017;4:170117
  43. Bakas S, Reyes M, Jakab A, Bauer S, Rempfler M, Crimi A, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. ArXiv Preprint 2018;arXiv:1811.02629
  44. Hu J, Shen L, Sun G. Squeeze-and-excitation networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018); 2018 Jun 18-22; Salt Lake City, UT, USA: IEEE; 2018:7132-7141
  45. Rundo L, Han C, Nagano Y, Zhang J, Hataya R, Militello C, et al. USE-Net: Incorporating Squeeze-and-Excitation blocks into U-Net for prostate zonal segmentation of multi-institutional MRI datasets. Neurocomputing 2019;365:31-43
  46. Guha Roy A, Siddiqui S, Polsterl S, Navab N, Wachinger C. 'Squeeze & excite' guided few-shot segmentation of volumetric images. Med Image Anal 2020;59:101587
  47. Zhou C, Ding C, Wang X, Lu Z, Tao D. One-pass multi-task networks with cross-task guided attention for brain tumor segmentation. IEEE Trans Image Process 2020 [in press] doi: https://doi.org/10.1109/ TIP.2020.2973510
  48. Li S, Dong M, Du G, Mu X. Attention dense-u-net for automatic breast mass segmentation in digital mammogram. IEEE Access 2019;7:59037-59047
  49. Pesce E, Joseph Withey S, Ypsilantis PP, Bakewell R, Goh V, Montana G. Learning to detect chest radiographs containing pulmonary lesions using visual attention networks. Med Image Anal 2019;53:26-38
  50. Zhang J, Xie Y, Xia Y, Shen C. Attention residual learning for skin lesion classification. IEEE Trans Med Imaging 2019;38:2092-2103
  51. Ni ZL, Bian GB, Xie XL, Hou ZG, Zhou XH, Zhou YJ. RASNet: segmentation for tracking surgical instruments in surgical videos using refined attention segmentation network. Annu Int Conf IEEE Eng Med Biol Soc 2019:5735-5738
  52. Yune S, Lee H, Kim M, Tajmir SH, Gee MS, Do S. Beyond human perception: sexual dimorphism in hand and wrist radiographs is discernible by a deep learning model. J Digit Imaging 2019;32:665-671
  53. Rajpurkar P, Irvin J, Ball RL, Zhu K, Yang B, Mehta H, et al. Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med 2018;15:e1002686
  54. Lee H, Yune S, Mansouri M, Kim M, Tajmir SH, Guerrier CE, et al. An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat Biomed Eng 2019;3:173-182
  55. Oh Y, Park S, Ye JC. Deep learning COVID-19 features on CXR using limited training data sets. IEEE Trans Med Imaging 2020;39:2688-2700
  56. Li L, Qin L, Xu Z, Yin Y, Wang X, Kong B, et al. Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT: evaluation of the diagnostic accuracy. Radiology 2020;296:E65-E71
  57. Cohen JP, Dao L, Morrison P, Roth K, Bengio Y, Shen B, et al. Predicting covid-19 pneumonia severity on chest x-ray with deep learning. ArXiv Preprint 2020;arXiv:2005.11856 2005
  58. Shaikh M, Kollerathu VA, Krishnamurthi G. Recurrent attention mechanism networks for enhanced classification of biomedical images. Proceedings of the 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019); 2019 Apr 8-11; Venice, Italy: IEEE; 2019:1260-1264
  59. Wu L, Wang Y, Gao J, Li X. Where-and-when to look: deep siamese attention networks for video-based person re-identification. IEEE T Multimedia 2018;21:1412-1424