Browse > Article
http://dx.doi.org/10.9717/kmms.2019.22.10.1149

Multi-task Architecture for Singe Image Dynamic Blur Restoration and Motion Estimation  

Jung, Hyungjoo (School of Electrical Electronic Engineering, Yonsei University)
Jang, Hyunsung (EO/IR R&D Lab., LIG Nex1 Co., Ltd.)
Ha, Namkoo (EO/IR R&D Lab., LIG Nex1 Co., Ltd.)
Yeon, Yoonmo (EO/IR R&D Lab., LIG Nex1 Co., Ltd.)
Kwon, Ku yong (EO/IR R&D Lab., LIG Nex1 Co., Ltd.)
Sohn, Kwanghoon (School of Electrical Electronic Engineering, Yonsei University)
Publication Information
Journal of Korea Multimedia Society / v.22, no.10, 2019 , pp. 1149-1159 More about this Journal
Abstract
We present a novel deep learning architecture for obtaining a latent image from a single blurry image, which contains dynamic motion blurs through object/camera movements. The proposed architecture consists of two sub-modules: blur image restoration and optical flow estimation. The tasks are highly related in that object/camera movements make cause blurry artifacts, whereas they are estimated through optical flow. The ablation study demonstrates that training multi-task architecture simultaneously improves both tasks compared to handling them separately. Objective and subjective evaluations show that our method outperforms the state-of-the-arts deep learning based techniques.
Keywords
Dynamic Motion Deblurring; CNN; Motion Estimation; Multi-task Architecture;
Citations & Related Records
Times Cited By KSCI : 3  (Citation Analysis)
연도 인용수 순위
1 K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
2 J. Long, E. Shelhamer, and T. Darrell, "Fully Convolutional Networks for Semantic Segmentation," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431-3440, 2015.
3 S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks," Advances in Neural Information Processing Systems, pp. 91-99, 2015.
4 R. Fergus, B. Singh, A. Hertzmann, S.T. Roweis, and W.T. Freeman, “Removing Camera Shake from a Single Photograph,” Association for Computiong Machinery Transactions on Graphics, Vol. 25, No. 3, pp. 787-794, 2006.
5 D. Krishnan and R. Fergus, "Fast Image Deconvolution using Hyper-Laplacian Priors," Advances in Neural Information Processing Systems, pp. 1033-1041, 2009.
6 D. Krishnan, T. Tay, and R. Fergus, "Blind Deconvolution Using a Normalized Sparsity Measure," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 233-240, 2011.
7 L. Xu, S. Zheng, and J. Jia, "Unnatural L0 Sparse Representation for Natural Image Deblurring," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1107-1114, 2013.
8 T.H. Kim, B. Ahn, and K.M. Lee, "Dynamic Scene Deblurring," Proceeding of IEEE International Conference on Computer Vision, pp. 3883-3891, 2013.
9 T.H. Kim and K.M. Lee, "Segmentation-free Dynamic Scene Deblurring," Proceeding of IEEE International Conference on Computer Vision, pp. 2766-2773, 2014.
10 H. Kim, K. Kwon, and J. Kim, "Adaptive Unsharp Masking Filter Design based on Multi-scale Retinex for Image Enhancement," Journal of Korea Multimedia Society, Vol. 21, No. 2, pp. 108-116, 2018.   DOI
11 L. Xu, S.J. Ren, C. Liu, and J. Jia, "Deep Convolutional Neural Network for Image Deconvolution," Advances in Neural Information Processing Systems, pp. 1790-1798, 2014.
12 O. Kwon, "Face Recognition based on Super-resolution Method Using Sparse Representation and Deep Learning," Journal of Korea Multimedia Society, Vol. 21, No. 2, pp. 173-180, 2018.   DOI
13 J. Cho, H. Jang, N. Ha, S. Lee, S. Park, and K. Sohn, “Deep Unsupervised Learning for Rain Streak Removal Using Time-varing Rain Streak Scene,” Journal of Korea Multimedia Society, Vol. 22, No. 1, pp. 1-9, 2019.   DOI
14 S. Nah, T. Kim, and K.M. Lee, "Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3883-3891, 2017.
15 O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, "DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 8183-8192, 2018.
16 C.J. Schuler, M. Hirsch, S. Harmeling, and B. Scholkopf, “Learning to Deblur,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38, No. 7, pp. 1433-1451, 2016.
17 J. Sun, W. Cao, Z. Xu, and J. Ponce, "Learning a Convolutional Neural Network for Non-uniform Motion Blur Removal," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 769-777, 2015.
18 I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, et al., "Generative Adversarial Nets," Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
19 J. Zhang, J. Pan, J. Ren, Y. Song, L. Bao, R.W. Lau, et al., "Dynamic Scene Deblurring Using Spatially Variant Recurrent Neural Networks," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2521-2529, 2018.
20 X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia, "Scale-recurrent Network for Deep Image Deblurring," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 8174-8182, 2018.
21 T. Brox and J. Malik, “Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 33, No. 3, pp. 500-513, 2011.   DOI
22 C. Liu, J. Yuen, and A. Torralba, “SIFT Flow: Dense Correspondence Across Scenes and Its Applications,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 33, No. 5, pp. 978-994, 2010.   DOI
23 C. Barnex, E. Shechtman, D.B. Goldman, and A. Finkelstein, "The Generalized PatchMatch Correspondence Algorithm," Proceeding European Conference on Computer Vision, pp. 29-43, 2010.
24 L. Xu, J. Jia, and Y. Matsushita, “Motion Detail Preserving Optical Flow Estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, No. 9, pp. 1744-1757, 2011.   DOI
25 J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid, "Epicflow: Edge-preserving Interpolation of Correspondences for Optical Flow," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1164-1172, 2015.
26 J. Thewlis, S. Zheng, P.H. Torr, and A. Vedaldi, "Fully Trainable Deep Matching," Proceeding of British Machine Vision Conference, pp. 145.1-145.12, 2016.
27 D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, "Deep End-to-end Voxel-to-voxel Prediction," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 17-24, 2016.
28 P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid, "DeepFlow: Large Displacement Optical Flow with Deep Matching," Proceeding of IEEE International Conference on Computer Vision, pp. 1385-1392, 2013.
29 D. Gadot and L. Wolf, "Patchbatch: A Batch Augmented Loss for Optical Flow," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 4236-4245, 2016.
30 A. Dosovitskiy, P. Fischer, E. Ilg, H. Philip, H. Caner, and G. Vladimir, "FlowNet: Learning Optical Flow with Convolutional Networks," Proceeding of IEEE International Conference on Computer Vision, pp. 2758-2766, 2015.
31 A. Ahmadi and I. Patras, "Unsupervised Convolutional Neural Networks for Motion Estimation," Proceeding of IEEE International Conference on Image Processing, pp. 1629-1633, 2016.
32 D. Teny and M. Hebert, "Learning to Extract Motion from Videos in Convolutional Neural Networks," arXiv Preprint arXiv:1601.07532, 2016.
33 A. Ranjan and M.J. Black, "Optical Flow Estimation Using a Spatial Pyramid Network," arXiv Preprint arXiv:1611.00850, 2016.
34 H. Noh, S. Hong, and B. Han, "Learning Deconvolution Network for Semantic Segmentation," Proceeding of IEEE International Conference on Image Processing, pp. 1520-1528, 2015.
35 S. Ioffe and C. Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift," arXiv Preprint arXiv: 1502.03167, 2015.
36 K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-scale Image Recognition," arXiv Preprint arXiv: 1409.1556, 2014.
37 E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, "Flownet2.0: Evolution of Optical Flow Estimation with Deep Networks," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2462-2470, 2017.
38 M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein GAN," arXiv Preprint arXiv: 1701.07875, 2017.
39 D.P. Kingma and J. Ba, "Adam: A Method for Stochastic Optimization," arXiv Preprint arXiv: 1412.6980, 2014.