Browse > Article
http://dx.doi.org/10.13104/imri.2019.23.2.81

Deep Learning in MR Image Processing  

Lee, Doohee (Laboratory for Imaging Science and Technology, Department of Electrical and Computer Engineering, Institute of Engineering Research, Seoul National University)
Lee, Jingu (Laboratory for Imaging Science and Technology, Department of Electrical and Computer Engineering, Institute of Engineering Research, Seoul National University)
Ko, Jingyu (Laboratory for Imaging Science and Technology, Department of Electrical and Computer Engineering, Institute of Engineering Research, Seoul National University)
Yoon, Jaeyeon (Laboratory for Imaging Science and Technology, Department of Electrical and Computer Engineering, Institute of Engineering Research, Seoul National University)
Ryu, Kanghyun (Department of Electrical and Electronic Engineering, Yonsei University)
Nam, Yoonho (Department of Radiology, Seoul St. Mary's Hospital, College of Medicine, The Catholic University of Korea)
Publication Information
Investigative Magnetic Resonance Imaging / v.23, no.2, 2019 , pp. 81-99 More about this Journal
Abstract
Recently, deep learning methods have shown great potential in various tasks that involve handling large amounts of digital data. In the field of MR imaging research, deep learning methods are also rapidly being applied in a wide range of areas to complement or replace traditional model-based methods. Deep learning methods have shown remarkable improvements in several MR image processing areas such as image reconstruction, image quality improvement, parameter mapping, image contrast conversion, and image segmentation. With the current rapid development of deep learning technologies, the importance of the role of deep learning in MR imaging research appears to be growing. In this article, we introduce the basic concepts of deep learning and review recent studies on various MR image processing applications.
Keywords
Deep learning; Machine learning; Image processing;
Citations & Related Records
연도 인용수 순위
  • Reference
1 LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE 1998;86:2278-2324   DOI
2 Goodfellow I, Bengio Y, Courville A, Bengio Y. Deep learning. Cambridge: MIT Press, 2016
3 Schmidhuber J. Deep learning in neural networks: an overview. Neural Netw 2015;61:85-117   DOI
4 Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2015:1-9
5 Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014;2:2672-2680
6 Isola P, Zhu J-Y, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017:1125-1134
7 Zhu J-Y, Park T, Isola P, Efros AA. Unpaired image-toimage translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2017:2223-2232
8 Zhu B, Liu JZ, Cauley SF, Rosen BR, Rosen MS. Image reconstruction by domain-transform manifold learning. Nature 2018;555:487-492   DOI
9 Lauterbur PC. Image formation by induced local interactions. Examples employing nuclear magnetic resonance. 1973. Clin Orthop Relat Res 1989:3-6
10 Shmueli K, de Zwart JA, van Gelderen P, Li TQ, Dodd SJ, Duyn JH. Magnetic susceptibility mapping of brain tissue in vivo using MRI phase data. Magn Reson Med 2009;62:1510-1522   DOI
11 Liu T, Spincemaille P, de Rochefort L, Kressler B, Wang Y. Calculation of susceptibility through multiple orientation sampling (COSMOS): a method for conditioning the inverse problem from measured magnetic field map to susceptibility source image in MRI. Magn Reson Med 2009;61:196-204   DOI
12 Liu T, Liu J, de Rochefort L, et al. Morphology enabled dipole inversion (MEDI) from a single-angle acquisition: comparison with COSMOS in human brain imaging. Magn Reson Med 2011;66:777-783   DOI
13 Wharton S, Schafer A, Bowtell R. Susceptibility mapping in the human brain using threshold-based k-space division. Magn Reson Med 2010;63:1292-1304   DOI
14 de Rochefort L, Liu T, Kressler B, et al. Quantitative susceptibility map reconstruction from MR phase data using bayesian regularization: validation and application to brain imaging. Magn Reson Med 2010;63:194-206   DOI
15 Papyan V, Romano Y, Elad M. Convolutional Neural Networks Analyzed via Convolutional Sparse Coding. J Mach Learn Res 2017;83:1-52
16 Sodickson DK, Manning WJ. Simultaneous acquisition of spatial harmonics (SMASH): fast imaging with radiofrequency coil arrays. Magn Reson Med 1997;38:591-603   DOI
17 Mansfield P, Maudsley AA. Medical imaging by NMR. Br J Radiol 1977;50:188-194   DOI
18 Griswold MA, Jakob PM, Heidemann RM, et al. Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magn Reson Med 2002;47:1202-1210   DOI
19 Wiatowski T, Bolcskei H. A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction. IEEE Trans Inf Theory 2018;64:1845-1866   DOI
20 Ye JC, Han Y, Cha E. Deep Convolutional Framelets: A General Deep Learning Framework for Inverse Problems. SIAM J Imaging Sci 2018;11:991-1048   DOI
21 Lustig M, Donoho D, Pauly JM. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn Reson Med 2007;58:1182-1195   DOI
22 Wang G, Ye JC, Mueller K, Fessler JA. Image reconstruction is a new frontier of machine learning. IEEE Trans Med Imaging 2018;37:1289-1296   DOI
23 Boyd S, Vandenberghe L. Convex optimization. New York: Cambridge University Press, 2004:127-214
24 Han Y, Yoo J, Kim HH, Shin HJ, Sung K, Ye JC. Deep learning with domain adaptation for accelerated projection-reconstruction MR. Magn Reson Med 2018;80:1189-1205   DOI
25 Ye JC, Han Y, Cha E. Deep convolutional framelets: a general deep learning framework for inverse problems. SIAM J Imaging Sci 2018;11:991-1048   DOI
26 Lee D, Yoo J, Ye JC. Deep residual learning for compressed sensing MRI. In IEEE 14th International Symposium on Biomedical Imaging (ISBI). 2017:15-18
27 Hammernik K, Klatzer T, Kobler E, et al. Learning a variational network for reconstruction of accelerated MRI data. Magn Reson Med 2018;79:3055-3071   DOI
28 Knoll F, Hammernik K, Kobler E, Pock T, Recht MP, Sodickson DK. Assessment of the generalization of learned image reconstruction and the potential for transfer learning. Magn Reson Med 2019;81:116-128   DOI
29 Razavian AS, Azizpour H, Sullivan J, Carlsson S. CNN features off-the-shelf: an astounding baseline for recognition. arXiv preprint arXiv 2014:1403.6382
30 Yosinski J, Clune J, Bengio Y, Lipson H. How transferable are features in deep neural networks? In Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds.). Advances in neural information processing systems 27. Curran Associates, Inc. 2014:3320-3328
31 Fong RC, Vedaldi A. Interpretable explanations of black boxes by meaningful perturbation. IEEE International Conference on Computer Vision (ICCV). 2017:3449-3457
32 Zeiler MD, Fergus R. Visualizing and understanding convolutional networks. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Volume 8689 LNCS, No. PART 1. Springer Verlag. 2014:818-833
33 Liu F, Jang H, Kijowski R, Bradshaw T, McMillan AB. Deep learning MR imaging-based attenuation correction for PET/MR imaging. Radiology 2018;286:676-684   DOI
34 Han X. MR-based synthetic CT generation using a deep convolutional neural network method. Med Phys 2017;44:1408-1419   DOI
35 Xiang L, Wang Q, Nie D, et al. Deep embedding convolutional neural network for synthesizing CT image from T1-Weighted MR image. Med Image Anal 2018;47:31-44   DOI
36 Yang G, Yu S, Dong H, et al. DAGAN: Deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction. IEEE Trans Med Imaging 2018;37:1310-1321   DOI
37 Pruessmann KP, Weiger M, Scheidegger MB, Boesiger P. SENSE: sensitivity encoding for fast MRI. Magn Reson Med 1999;42:952-962   DOI
38 Knoll F, Hammernik K, Kobler E, Pock T, Recht MP, Sodickson DK. Assessment of the generalization of learned image reconstruction and the potential for transfer learning. Magn Reson Med 2019;81:116-128   DOI
39 Akcakaya M, Moeller S, Weingartner S, Ugurbil K. Scanspecific robust artificial-neural-networks for k-space interpolation (RAKI) reconstruction: Database-free deep learning for fast imaging. Magn Reson Med 2019;81:439-453   DOI
40 Wolterink JM, Dinkla AM, Savenije MHF, Seevinck PR, van den Berg CAT, Isgum I. Deep MR to CT synthesis using unpaired data. International Workshop on Simulation and Synthesis in Medical Imaging. Springer, Cham, 2017:14-23
41 Quan TM, Nguyen-Duc T, Jeong WK. Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss. IEEE Trans Med Imaging 2018;37:1488-1497   DOI
42 Mardani M, Gong E, Cheng JY, et al. Deep generative adversarial neural networks for compressive sensing MRI. IEEE Trans Med Imaging 2019;38:167-179   DOI
43 Hyun CM, Kim HP, Lee SM, Lee S, Seo JK. Deep learning for undersampled MRI reconstruction. Phys Med Biol 2018;63:135007   DOI
44 Eo T, Jun Y, Kim T, Jang J, Lee HJ, Hwang D. KIKInet: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images. Magn Reson Med 2018;80:2188-2201   DOI
45 Aggarwal HK, Mani MP, Jacob M. MoDL: Model-based deep learning architecture for inverse problems. IEEE Trans Med Imaging 2019;38:394-405   DOI
46 Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-assisted Invervention. 2015:234-241
47 Jun Y, Eo T, Kim T, et al. Deep-learned 3D black-blood imaging using automatic labelling technique and 3D convolutional neural networks for detecting metastatic brain tumors. Sci Rep 2018;8:9450   DOI
48 Gong E, Pauly JM, Wintermark M, Zaharchuk G. Deep learning enables reduced gadolinium dose for contrast-enhanced brain MRI. J Magn Reson Imaging 2018;48:330-340   DOI
49 Ryu K, Shin NY, Kim DH, Nam Y. Synthesizing T1 weighted MPRAGE image from multi echo GRE images via deep neural network. Magn Reson Imaging 2019 [Epub ahead of print]
50 Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Springer, Cham, 2015:3431-3440
51 Yang Q, Yan P, Zhang Y, et al. Low-Dose CT Image Denoising Using a Generative Adversarial Network With Wasserstein Distance and Perceptual Loss. IEEE Trans Med Imaging 2018;37:1348-1357   DOI
52 Nie D, Trullo R, Lian J, et al. Medical image synthesis with context-aware generative adversarial networks. Medical Image Computing and Computer-Assisted Intervention-MICCAI 2017-20th International Conference, Proceedings. Springer Verlag, 2017:417-425
53 Fischl B. FreeSurfer. Neuroimage 2012;62:774-781   DOI
54 Jenkinson M, Beckmann CF, Behrens TE, Woolrich MW, Smith SM. Fsl. Neuroimage 2012;62:782-790   DOI
55 Zoran D, Weiss Y. From learning models of natural image patches to whole image restoration. In IEEE International Conference on Computer Vision (ICCV). 2011:479-486
56 Benou A, Veksler R, Friedman A, Riklin Raviv T. Ensemble of expert deep neural networks for spatio-temporal denoising of contrast-enhanced MRI sequences. Med Image Anal 2017;42:145-159   DOI
57 Dabov K, Foi A, Katkovnik V, Egiazarian K. Image restoration by sparse 3D transform-domain collaborative filtering. Image Processing: Algorithms and Systems VI. International Society for Optics and Photonics 2008:681207
58 Elad M, Aharon M. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans Image Process 2006;15:3736-3745   DOI
59 Jin KH, McCann MT, Froustey E, Unser M. Deep convolutional neural network for inverse problems in imaging. IEEE Trans Image Process 2017;26:4509-4522   DOI
60 Gu S, Zhang L, Zuo W, Feng X. Weighted nuclear norm minimization with application to image denoising. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2014:2862-2869
61 Pereira S, Pinto A, Alves V, Silva CA. Brain tumor segmentation using convolutional neural networks in MRI images. IEEE Trans Med Imaging 2016;35:1240-1251   DOI
62 Kamnitsas K, Chen L, Ledig C, Rueckert D, Glocker B. Multi-scale 3D convolutional neural networks for lesion segmentation in brain MRI. Ischemic Stroke Lesion Segment 2015;13:46
63 Rajchl M, Pawlowski N, Rueckert D, Matthews PM, Glocker B. NeuroNet: Fast and robust reproduction of multiple brain image segmentation pipelines. arXiv preprint arXiv 2018:1806.04224
64 Havaei M, Davy A, Warde-Farley D, et al. Brain tumor segmentation with deep neural networks. Med Image Anal 2017;35:18-31   DOI
65 Akkus Z, Galimzianova A, Hoogi A, Rubin DL, Erickson BJ. Deep learning for brain MRI segmentation: state of the art and future directions. J Digit Imaging 2017;30:449-459   DOI
66 Liu S, Zheng H, Feng Y, Li W. Prostate cancer diagnosis using deep learning with 3D multiparametric MRI. SPIE Med Imaging 2017;10134:1-4
67 Song Y, Zhang YD, Yan X, et al. Computer-aided diagnosis of prostate cancer using a deep convolutional neural network from multiparametric MRI. J Magn Reson Imaging 2018;48:1570-1577   DOI
68 Milletari F, Navab N, Ahmadi S-A. V-net: fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision (3DV). IEEE 2016:565-571
69 Guerrero R, Qin C, Oktay O, et al. White matter hyperintensity and stroke lesion segmentation and differentiation using convolutional neural networks. Neuroimage Clin 2018;17:918-934   DOI
70 Lee D, Yoo J, Tak S, Ye J. Deep residual learning for accelerated mri using magnitude and phase networks. IEEE Trans Biomed Eng 2018;65:1985-1995   DOI
71 Kyathanahally SP, Doring A, Kreis R. Deep learning approaches for detection and removal of ghosting artifacts in MR spectroscopy. Magn Reson Med 2018;80:851-863   DOI
72 Atkinson D, Hill DL, Stoyle PN, Summers PE, Keevil SF. Automatic correction of motion artifacts in magnetic resonance images using an entropy focus criterion. IEEE Trans Med Imaging 1997;16:903-910   DOI
73 Loktyushin A, Nickisch H, Pohmann R, Scholkopf B. Blind retrospective motion correction of MR images. Magn Reson Med 2013;70:1608-1618   DOI
74 Ooi MB, Krueger S, Thomas WJ, Swaminathan SV, Brown TR. Prospective real-time correction for arbitrary head motion using active markers. Magn Reson Med 2009;62:943-954   DOI
75 Maclaren J, Armstrong BS, Barrows RT, et al. Measurement and correction of microscopic head motion during magnetic resonance imaging of the brain. PLoS One 2012;7:e48088   DOI
76 Haskell MW, Cauley SF, Wald LL. TArgeted Motion Estimation and Reduction (TAMER): Data Consistency Based Motion Mitigation for MRI Using a Reduced Model Joint Optimization. IEEE Trans Med Imaging 2018;37:1253-1265   DOI
77 Kober T, Marques JP, Gruetter R, Krueger G. Head motion detection using FID navigators. Magn Reson Med 2011;66:135-143   DOI
78 Zhou Z, Zhao G, Kijowski R, Liu F. Deep convolutional neural network for segmentation of knee joint anatomy. Magn Reson Med 2018;80:2759-2770   DOI
79 Norman B, Pedoia V, Majumdar S. Use of 2D U-net convolutional neural networks for automated cartilage and meniscus segmentation of knee MR imaging data to determine relaxometry and morphometry. Radiology 2018;288:177-185   DOI
80 Prasoon A, Petersen K, Igel C, Lauze F, Dam E, Nielsen M. Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. Med image Comput Comput Assist Interv 2013;16:246-253
81 Tamada D, Kromrey M-L, Onishi H, Motosugi U. Method for motion artifact reduction using a convolutional neural network for dynamic contrast enhanced MRI of the liver. arXiv preprint arXiv 2018:1807.06956
82 Johnson PM, Drangova M. Motion correction in MRI using deep learning. In Proceeding ISMRM Scientific Meeting & Exhibition. 2018:4098
83 Pawar K, Chen Z, Shah J, Egan GF. Motion correction in MRI using deep convolutional neural network. In Proceeding ISMRM Scientific Meeting & Exhibition. 2018:1174
84 Lee H, Ryu K, Nam Y, Lee J, Kim DH. Reduction of respiratory motion artifact in c-spine imaging using deep learning: Is substitution of navigator possible? In Proceeding ISMRM Scientific Meeting & Exhibition. 2018:2660
85 Meding K, Loktyushin A, Hirsch M. Automatic detection of motion artifacts in MR images using CNNS. 42nd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2017:811-815
86 Pham CH, Ducournau A, Fablet R, Rousseau F. Brain MRI super-resolution using deep 3D convolutional networks. IEEE 14th International Symposium on Biomedical Imaging (ISBI). 2017:197-200
87 Shi J, Liu Q, Wang C, Zhang Q, Ying S, Xu H. Superresolution reconstruction of MR image with a novel residual learning network algorithm. Phys Med Biol 2018;63:085011   DOI
88 Chen Y, Xie Y, Zhou Z, Shi F, Christodoulou AG, Li D. Brain MRI super resolution using 3D deep densely connected neural networks. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). 2018:739-742
89 Sommer K, Brosch T, Rafael W, et al. Correction of motion artifacts using a multi-resolution fully convolutional neural network. In Proceeding ISMRM Scientific Meeting & Exhibition. 2018:1175
90 Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada, 2012:1097-1105
91 Hinton G, Deng I, Yu D, et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Processing Magazine 2012;29:82-97   DOI
92 Sutskever I, Vinyals O, Le QV. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Vol 2. 2014:3104-3112
93 Ryu K, Nam Y, Gho SM, et al. Data-driven synthetic MRI FLAIR artifact correction via deep neural network. J Magn Reson Imaging 2019 [Epub ahead of print]
94 Jifara W, Jiang F, Rho S, Cheng M, Liu S. Medical image denoising using convolutional neural network: a residual learning approach. J Supercomput 2017:1-15
95 Kim KH, Do WJ, Park SH. Improving resolution of MR images with an adversarial network incorporating images with different contrast. Med Phys 2018;45:3120-3131   DOI
96 Chaudhari AS, Fang Z, Kogan F, et al. Super-resolution musculoskeletal MRI using deep learning. Magn Reson Med 2018;80:2139-2154   DOI
97 Dong C, Loy CC, He K, Tang X. Image Super-Resolution Using Deep Convolutional Networks. IEEE Trans Pattern Anal Mach Intell 2016;38:295-307   DOI
98 Tanenbaum LN, Tsiouris AJ, Johnson AN, et al. Synthetic MRI for Clinical Neuroimaging: Results of the Magnetic Resonance Image Compilation (MAGiC) Prospective, Multicenter, Multireader Trial. AJNR Am J Neuroradiol 2017;38:1103-1110   DOI
99 Lee J, Han Y, Ye JC. k-Space Deep Learning for Reference-free EPI Ghost Correction. arXiv preprint arXiv 2018:1806.00153v2
100 Kim KH, Choi SH, Park SH. Improving Arterial Spin Labeling by Using Deep Learning. Radiology 2018;287:658-666   DOI
101 Golkov V, Dosovitskiy A, Sperl JI, et al. q-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI Scans. IEEE Trans Med Imaging 2016;35:1344-1351   DOI
102 Bertleff M, Domsch S, Weingartner S, et al. Diffusion parameter mapping with the combined intravoxel incoherent motion and kurtosis model using artificial neural networks at 3 T. NMR Biomed 2017;30 [Epub ahead of print]
103 Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014
104 Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res 2010;11:3371-3408
105 Zhang K, Zuo W, Chen Y, Meng D, Zhang L. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans Image Process 2017;26:3142-3155   DOI
106 He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016:770-778
107 Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell 2018;40:834-848   DOI
108 Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2015:1520-1528
109 LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521:436-444   DOI
110 Glorot X, Bordes A, Bengio Y. Deep sparse rectifier neural networks. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), 2011. Volume 15 of JMLR: W&CP 15. 2011:315-323
111 LeCun Y, Boser B, Denker JS, et al. Handwritten digit recognition with a back-propagation network. In Advances in neural information processing system 2. 1990:396-404
112 Yoon J, Gong E, Chatnuntawech I, et al. Quantitative susceptibility mapping using deep neural network: QSMnet. Neuroimage 2018;179:199-206   DOI
113 Domsch S, Murle B, Weingartner S, Zapp J, Wenz F, Schad LR. Oxygen extraction fraction mapping at 3 Tesla using an artificial neural network: a feasibility study. Magn Reson Med 2018;79:890-899   DOI
114 Lee D, Jung W, Lee J, et al. SafeNet: Artificial neural network for real-time T2 mapping with quality assurance. Joint Annual Meeting ISMRM-ESMRMB. ISMRM 2018:2277
115 Ma D, Gulani V, Seiberlich N, et al. Magnetic resonance fingerprinting. Nature 2013;495:187-192   DOI
116 Cohen O, Zhu B, Rosen MS. MR fingerprinting Deep RecOnstruction NEtwork (DRONE). Magn Reson Med 2018;80:885-894   DOI