Browse > Article
http://dx.doi.org/10.5909/JBE.2020.25.5.672

Adversarial Framework for Joint Light Field Super-resolution and Deblurring  

Lumentut, Jonathan Samuel (Inha University, Department of Information and Communication Engineering)
Baek, Hyungsun (Inha University, Department of Information and Communication Engineering)
Park, In Kyu (Inha University, Department of Information and Communication Engineering)
Publication Information
Journal of Broadcast Engineering / v.25, no.5, 2020 , pp. 672-684 More about this Journal
Abstract
Restoring a low resolution and motion blurred light field has become essential due to the growing works on parallax-based image processing. These tasks are known as light-field enhancement process. Unfortunately, only a few state-of-the-art methods are introduced to solve the multiple problems jointly. In this work, we design a framework that jointly solves light field spatial super-resolution and motion deblurring tasks. Particularly, we generate a straight-forward neural network that is trained under low-resolution and 6-degree-of-freedom (6-DOF) motion-blurred light field dataset. Furthermore, we propose the strategy of local region optimization on the adversarial network to boost the performance. We evaluate our method through both quantitative and qualitative measurements and exhibit superior performance compared to the state-of-the-art methods.
Keywords
Deep learning; light field; deblurring; super-resolution;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Y. Bok, H. G. Jeon, and I. S. Kweon, "Geometric calibration of micro-lense-based light field cameras using line features," IEEE Trans. on Pattern Analysis and Machine Intelligence, 39(2):287-300, February 2017.   DOI
2 I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," In Proc. of Advances in Neural Information Processing Systems, pages 2672-2680, December 2014.
3 I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, "Improved training of Wasserstein GANs," In Proc. of Advances in Neural Information Processing Systems, pages 5767-5777, December 2017.
4 K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, June 2016.
5 T. H. Kim, K. M. Lee, B. Scholkopf, and M. Hirsch, "Online video deblurring via dynamic temporal blending network," In Proc. of IEEE International Conference on Computer Vision, pages 4058-4067, 2017.
6 S. Iizuka, E. Simo-Serra, and H. Ishikawa, "Globally and locally consistent image completion," ACM Transactions on Graphics, 36(4):107, Article No. 107, July 2017.
7 P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional adversarial networks," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1125-1134, July 2017.
8 H. G. Jeon, J. Park, G. Choe, J. Park, Y. Bok, Y. W. Tai, and I. S. Kweon, "Accurate depth map estimation from a lenslet light field camera," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1547-1555, June 2015.
9 N. K. Kalantari, T. C. Wang, and R. Ramamoorthi, "Learning-based view synthesis for light field cameras," ACM Transactions on Graphics, 35(6), Article No. 193, November 2016.
10 J. Kim, J. K. Lee, and K. M. Lee, "Accurate image super-resolution using very deep convolutional networks," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1646-1654, June 2016.
11 D. Krishnan, T. Tay, and R. Fergus, "Blind deconvolution using a normalized sparsity measure," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 233-240, June 2011.
12 O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, "DeblurGAN: Blind motion deblurring using conditional adversarial networks," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 8183-8192, June 2018.
13 W. S. Lai, J. B. Huang, Z. Hu, N. Ahuja, and M. H. Yang, "A comparative study for single image blind deblurring," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1701-1709, June 2016.
14 D. Lee, H. Park, I. K. Park, and K. M. Lee, "Joint blind motion deblurring and depth estimation of light field," In Proc. of European Conference on Computer Vision, pp. 300-316, September 2018.
15 M. Levoy and P. Hanrahan, "Light field rendering," In Proc. of SIGGRAPH, pages 31-42, August 1996.
16 J. S. Lumentut, T. H. Kim, R. Ramamoorthi, and I. K. Park, "Fast and full-resolution light field deblurring using a deep neural network," IEEE Signal of Processing Letters, 26(12), pages 1788-1792, December 2019.   DOI
17 J.S. Lumentut, Williem, and I. K. Park, "6-DOF Motion Blur Synthesis and Performance Evaluation of Light Field Deblurring," Multimedia Tools and Application, 78(23), pages 33723-33746, December 2019.   DOI
18 M. R. Mahesh Mohan and A. N. Rajagopalan, "Divide and conquer for full-resolution light field deblurring," In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6421-6429, June 2018.
19 S. Nah, T. H. Kim, and K. M. Lee, "Deep multi-scale convolutional neural network for dynamic scene deblurring," In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, volume 1, pages 3883-3891, July 2017.
20 R. Ng, "Fourier slice photography," ACM Trans. on Graphics, 24(3):735-744, July 2005.   DOI
21 J. Pan, Z. Hu, Z. Su, and M. H. Yang, "Deblurring text images via L0-regularized intensity and gradient prior," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2908, June 2014.
22 W. Qiu, F. Zhong, Y. Zhang, S. Qiao, Z. Xiao, T. S. Kim, and Y. Wang, "UnrealCV: Virtual worlds for computer vision." In Proc. of ACM International Conference on Multimedia, pages 1221-1224, October 2017.
23 Z. Shen, W. S. Lai, T. Xu, J. Kautz, and M. H. Yang, "Deep semantic face deblurring," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 8260-8269, June 2018.
24 K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," In Proc. of International Conference on Learning Representations, April 2014.
25 P. P. Srinivasan, R. Ng, and R. Ramamoorthi, "Light field blind motion deblurring," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 3958-3966, July 2017.
26 P. P. Srinivasan, R. Tucker, J. T. Barron, R. Ramamoorthi, R. Ng, and N. Snavely, "Pushing the boundaries of view extrapolation with multiplane images," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 175-184, June 2019.
27 P. P. Srinivasan, T. Wang, A. Sreelal, R. Ramamoorthi, and R. Ng, "Learning to synthesize a 4D RGBD light field from a single image," In Proc. of IEEE International Conference on Computer Vision, pages 2262-2270, October 2017.
28 S. Su, M. Delbracio, J. Wang, G. Sapiro, W. Heidrich, and O. Wang, "Deep video deblurring for hand-held cameras," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages 237-246, July 2017.
29 M. W. Tao, S. Hadap, J. Malik, and R. Ramamoorthi, "Depth from combining defocus and correspondence using light-field cameras," In Proc. of the IEEE International Conference on Computer Vision, pages 673-680, December 2013.
30 Y. W. Tai, P. Tan, and M. S. Brown, "Richardson-Lucy deblurring for scenes under a projective motion path," IEEE Trans. on Pattern Analysis and Machine Intelligence, 33(8):1603-1618, August 2011.   DOI
31 M. W. Tao, P. P Srinivasan, J. Malik, S. Rusinkiewicz, and R. Ramamoorthi, "Depth from shading, defocus, and correspondence using light-field angular coherence," In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1940-1948, June 2015.
32 D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Instance normalization: The missing ingredient for fast stylization," arXiv preprint arXiv:1607.08022, 2016.
33 T. C Wang, A. A. Efros, and Ravi Ramamoorthi, "Depth estimation with occlusion modeling using light-field cameras," IEEE Trans. on Pattern Analysis and Machine Intelligence, 38(11):2170-2181, November 2016.   DOI
34 Williem, I. K. Park, and K. M. Lee, "Robust light field depth estimation using occlusion-noise aware data costs," IEEE Trans. on Pattern Analysis and Machine Intelligence, 40(10):2484-2497, October 2018.   DOI
35 Y. Yoon, H. G. Jeon, D. Yoo, J. Y. Lee, and I. S. Kweon, "Learning a deep convolutional network for light-field image super-resolution," In Proc. of IEEE International Conference on Computer Vision Workshop, pages 57-65, June 2015.
36 S. Zhang, Y. Lin, and H. Sheng, "Residual networks for light field image super-resolution," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 11046-11055, June 2019.
37 T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, "Stereo magnification: Learning view synthesis using multiplane images," ACM Trans. on Graphics, 37(4), Article No. 65, July 2018.