Browse > Article
http://dx.doi.org/10.5909/JBE.2019.24.5.755

5D Light Field Synthesis from a Monocular Video  

Bae, Kyuho (Inha University, Department of Information and Communication Engineering)
Ivan, Andre (Inha University, Department of Information and Communication Engineering)
Park, In Kyu (Inha University, Department of Information and Communication Engineering)
Publication Information
Journal of Broadcast Engineering / v.24, no.5, 2019 , pp. 755-764 More about this Journal
Abstract
Currently commercially available light field cameras are difficult to acquire 5D light field video since it can only acquire the still images or high price of the device. In order to solve these problems, we propose a deep learning based method for synthesizing the light field video from monocular video. To solve the problem of obtaining the light field video training data, we use UnrealCV to acquire synthetic light field data by realistic rendering of 3D graphic scene and use it for training. The proposed deep running framework synthesizes the light field video with each sub-aperture image (SAI) of $9{\times}9$ from the input monocular video. The proposed network consists of a network for predicting the appearance flow from the input image converted to the luminance image, and a network for predicting the optical flow between the adjacent light field video frames obtained from the appearance flow.
Keywords
Deep learning; Light field; Video synthesis; View synthesis;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Raytrix 3D Light Field Cameras, https://raytrix.de/products
2 M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, and M. Kudlur, "Tensor-flow: A system for large-scale machine learning," In Proc. of 12th Symposium on Operating Systems Design and Implementation, volume 16, pages 265-283, 2016.
3 N. K. Kalantari, T.-C. Wang, and R. Ramamoorthi, "Learning-based view synthesis for light field cameras," ACM Transactions on Graphics, 35(6): 193, 2016.
4 P. P. Srinivasan, T. Wang, A. Sreelal, R. Ramamoorthi, and R. Ng, "Learning to synthesize a 4D RGBD light field from a single image," In Proc. of IEEE International Conference on Computer Vision, pages 2243-2251, 2017.
5 A. Ivan, Willem, and I. K. Park, "Synthesizing a 4D spatio-angular consistent light field from a single image," arXiv preprint arXiv:1903.12364, 2019.
6 T.-C. Wang, J.-Y. Zhu, N. K. Kalantari, A. A. Efros, and R. Ramamoorthi, "Light field video capture using a learning-based hybrid imaging system," ACM Transactions on Graphics, 36(4): 133, 2017.
7 B. Wilburn, N. Joshi, V. Vaish, E. -V. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy, "High performance imaging using large camera arrays," ACM Transactions on Graphics, 24(3), pages 765-776, 2005.   DOI
8 W. Qiu and Y. Alan,"UnrealCV: Connecting computer vision to unreal engine," In Proc. of European Conference on Computer Vision, pages 909-916, 2016.
9 M. Jaderberg, K. Simonyan, and A. Zisserman, "Spatial transformer networks," In Proc. of Advances in Neural Information Processing Systems, pages 2017-2025, 2015.
10 D. P. Kingma and J. B. Adam, "Adam: A method for stochastic optimization," In Proc. of International Conference on Machine Learning, 2015.
11 G. Wu, M. Zhao, L. Wang, Q. Dai, T. Chai, and Y. Liu, "Light field reconstruction using deep convolutional network on EPI," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 6319-6327, 2017.
12 H. Wing Fung Yeung, J. Hou, J. Chen, Y. Ying Chung, and X. Chen, "Fast light field reconstruction with deep coarse-to-fine modelling of spatial-angular clues," In Proc. of European Conference on Computer Vision, pages 137-152, 2018.
13 T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, "Stereo magnification: Learning view synthesis using multiplane images," ACM Transactions on Graphics, 37(4):65:1-65:12, 2018.
14 Williem, and I. K. Park, "Robust light field depth estimation for noisy scene with occlusion," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 4396-4404, 2016.
15 P. P. Srinivasan, R. Tucker, J. T. Barron, R. Ramamoorthi, R. Ng, and N. Snavely, "Pushing the boundaries of view extrapolation with multiplane images," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 175-184, 2019.
16 B. Mildenhall, P. P. Srinivasan, R. Ortiz-Cayon, N. K. Kalantari, R. Ramamoorthi, R. Ng, and A. Kar, "Local light field fusion: Practical view synthesis with prescriptive sampling guidelines," ACM Transactions on Graphics, 38(4):29:1-29:14, 2019.
17 T. Zhou, S. Tulsiani, W. Sun, J Malik, and A. A. Efros, "View synthesis by appearance flow," In Proc. of European Conference on Computer Vision, pages 286-301, 2016.
18 H. Schilling, M. Diebold, C. Rother, and B. Jhne, "Trust your model: Light field depth estimation with inline occlusion handling," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 4530-4538, 2018
19 C. Shin, H.-G. Jeon, Y. Yoon, I. S. Kweon, and S. J. Kim, "Epinet: A fully-convolutional neural network using epipolar geometry for depth from light field images," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 4748-4757, 2018.
20 Williem, I. K. Park, and K. M. Lee, "Robust light field depth estimation using occlusion-noise aware data costs," IEEE Transactions on Pattern Analysis and Machine Intelligence, (10):2484-2497, 2018.
21 R. Ng, M. Levoy, M. Brdif, G. Duval, M. Horowitz, and P. Hanrahan, "Light field photography with a hand-held plenoptic camera," Computer Science Technical Report CSTR, 2(11):1-11, 2005.