Browse > Article
http://dx.doi.org/10.4218/etrij.2021-0205

View synthesis with sparse light field for 6DoF immersive video  

Kwak, Sangwoon (Media Research Division, Electronics and Telecommunications Research Institute)
Yun, Joungil (Media Research Division, Electronics and Telecommunications Research Institute)
Jeong, Jun-Young (Media Research Division, Electronics and Telecommunications Research Institute)
Kim, Youngwook (Department of Computer Science and Engineering, Sogang University)
Ihm, Insung (Department of Computer Science and Engineering, Sogang University)
Cheong, Won-Sik (Media Research Division, Electronics and Telecommunications Research Institute)
Seo, Jeongil (Media Research Division, Electronics and Telecommunications Research Institute)
Publication Information
ETRI Journal / v.44, no.1, 2022 , pp. 24-37 More about this Journal
Abstract
Virtual view synthesis, which generates novel views similar to the characteristics of actually acquired images, is an essential technical component for delivering an immersive video with realistic binocular disparity and smooth motion parallax. This is typically achieved in sequence by warping the given images to the designated viewing position, blending warped images, and filling the remaining holes. When considering 6DoF use cases with huge motion, the warping method in patch unit is more preferable than other conventional methods running in pixel unit. Regarding the prior case, the quality of synthesized image is highly relevant to the means of blending. Based on such aspect, we proposed a novel blending architecture that exploits the similarity of the directions of rays and the distribution of depth values. By further employing the proposed method, results showed that more enhanced view was synthesized compared with the well-designed synthesizers used within moving picture expert group (MPEG-I). Moreover, we explained the GPU-based implementation synthesizing and rendering views in the level of real time by considering the applicability for immersive video service.
Keywords
depth image-based rendering; image blending; immersive video; view synthesis; warping;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 ISO/IEC JTC1 SC29/WG11, Reference view synthesizer (RVS) manual, N18068, 2018.
2 ISO/IEC JTC1 SC29/WG4, Test model 9 for MPEG immersive video, N00084, 2021.
3 A. Dziembowski et al., Multiview synthesis-Improved view synthesis for virtual navigation, in Proc. Picture Coding Symp. (Nuremberg, Germany), Dec. 2016, pp. 1-5.
4 J. C. Seong, K. A. Mulcahy, and E. L. Usery, The sinusoidal projection: A new importance in relation to global image data, Prof. Geogr. 52 (2002), 218-225.   DOI
5 N. Max, Optical models for direct volume rendering, IEEE Trans. Vis. Comput. Graph. 1 (1995), 99-108.   DOI
6 Y. Sun, A. Lu, and L. Yu, Weighted-to-spherically uniform quality evaluation for omnidirectional video, IEEE Signal Process. Lett. 24 (2017), no. 9, 1408-1412.   DOI
7 ISO/IEC JTC1 SC29/WG4, Common test conditions for MPEG immersive video, N00085, 2021.
8 H. Shum and S. B. Kang, Review of image-based rendering techniques, in Proc. SPIE Visual Commun. Image Process. (Perth, Australia), 2000, pp. 2-13.
9 B. Mildenhall et al., Local light field fusion: Practical view synthesis with prescriptive sampling guidelines, ACM Trans. Graph. 38 (2019), no. 4, 1-14.   DOI
10 M. Broxton et al., Immersive light field video with a layered mesh representation, ACM Trans. Graph. 39 (2020), no. 4 86, 1-15.
11 M. Levoy and P. Hanrahan, Light field rendering, in Proc. Therapeutic 23rd Annu. Conf. Comput. Graphics Interactive Techniques, Aug. 1996, pp. 31-42.
12 S. J. Gortler et al., The lumigraph, in Proc. Therapeutic 23rd Annu. Conf. Comput. Graphics Interactive Techniques, Aug. 1996, pp. 43-54.
13 C. Buehler et al., Unstructured lumigraph rendering, in Proc. Therapeutic 28th Annu. Conf. Comput. Graphics Interactive Techniques (Los Angeles, CA, USA), Aug. 2001, pp. 425-432.
14 Y. Mori et al., View generation with 3D warping using depth information for FTV, Signal Process.: Image Commun. 24 (2009), no. 1-2, 65-72. https://doi.org/10.1016/j.image.2008.10.013   DOI
15 M. L. Davis and F. Durand, Unstructured light fields, in Comput, Graph. Forum 31 (2012), 305-314.   DOI
16 J. T. Barron et al., Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields, in Proc. Int. Conf. Comput. Vision, Oct. 2021, pp. 5855-5864.
17 K. Muller et al., View synthesis for advanced 3D video systems, EURASIP J. Image Video Process. 2008 (2009), 438148:1-11.   DOI
18 Y. J. Jeong, Y. J. Jung, and D.-S. Park, Depth image-based rendering for multiview generation, J. Soc. Inf. Disp. 18 (2010), no. 4, 310-316.   DOI
19 J. M. Boyce et al., MPEG immersive video coding standard, Proc. IEEE 109 (2021), no. 9, 1521-1536.   DOI
20 B. Mildenhall et al., NeRF: Representing scenes as neural radiance fields for view synthesis, in Proc. Eur. Conf. Comput. Vision, Aug. 2020, pp. 405-421.
21 ISO/IEC JTC1 SC29/WG11, Versatile view synthesizer (VVS) 2.0 manual, N18172, 2019.
22 C. Fehn, Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV, in Proc. SPIE Stereoscopic Displays Virtual Reality Syst. (San Jose, CA, USA), 2004, pp. 93-104.
23 D. Tian et al., View synthesis techniques for 3D video, in Proc. SPIE Applicat. Digital Image Process. XXXII (San Diego, CA, USA), 2009, pp. 74430T:1-11.
24 S.-W. Nam et al., Hole-filling methods using depth and color information for generating multiview images, ETRI J. 38 (2016), no. 5, 996-1007.   DOI
25 S. Fachada et al., Depth image based view synthesis with multiple reference views for virtual reality, in Proc. 2018-3DTVConf.: True Vision-Capture, Transmission Display of 3D (video), pp. 1-4.
26 P. Srinivasan et al., NeRV: Neural reflectance and visibility fields for relighting and view synthesis, in Proc. IEEE Int. Conf. Comput. Vision Pattern Recognit. (Nashville, TN, USA), Nov. 2021, pp. 7495-7504.
27 C. Reiser et al., KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs, in Proc. Int. Conf. Comput. Vision, Oct. 2021, pp. 14335-14345.
28 N. Sabater et al., Dataset and pipeline for multi-view light-field video, in Proc. IEEE Int. Conf. Comput. Vision Pattern Recognit. Workshops (Honolulu, HI, USA), July 2017, pp. 30-40.
29 R. S. Overbeck et al., A system for acquiring, processing, and rendering panoramic light field stills for virtual reality, ACM Trans. Graph. 37 (2019), no. 6, 1-15.
30 E. Penner and L. Zhang, Soft 3D reconstruction for view synthesis, ACM Trans. Graph. 36 (2017), no. 6, 1-11.
31 T. Tezuka, K. Takahashi, and T. Fujii, Superpixel-based 3D warping using view plus depth data from multiple viewpoints, in Proc. SPIE Stereoscopic Displays Applicat. XXV (San Francisco), 2014 pp. 90111V:1-9.
32 M. P. Tehrani et al., Free-viewpoint image synthesis using super-pixel segmentation, APSIPA Trans. Signal Inf. Process. 6 (2017), 1-12.