Browse > Article
http://dx.doi.org/10.4218/etrij.16.0116.0062

Hole-Filling Methods Using Depth and Color Information for Generating Multiview Images  

Nam, Seung-Woo (SW & Contents Research Laboratory, ETRI)
Jang, Kyung-Ho (SW & Contents Research Laboratory, ETRI)
Ban, Yun-Ji (SW & Contents Research Laboratory, ETRI)
Kim, Hye-Sun (SW & Contents Research Laboratory, ETRI)
Chien, Sung-Il (School of Electronics Engineering, Kyungpook National University)
Publication Information
ETRI Journal / v.38, no.5, 2016 , pp. 996-1007 More about this Journal
Abstract
This paper presents new hole-filling methods for generating multiview images by using depth image based rendering (DIBR). Holes appear in a depth image captured from 3D sensors and in the multiview images rendered by DIBR. The holes are often found around the background regions of the images because the background is prone to occlusions by the foreground objects. Background-oriented priority and gradient-oriented priority are also introduced to find the order of hole-filling after the DIBR process. In addition, to obtain a sample to fill the hole region, we propose the fusing of depth and color information to obtain a weighted sum of two patches for the depth (or rendered depth) images and a new distance measure to find the best-matched patch for the rendered color images. The conventional method produces jagged edges and a blurry phenomenon in the final results, whereas the proposed method can minimize them, which is quite important for high fidelity in stereo imaging. The experimental results show that, by reducing these errors, the proposed methods can significantly improve the hole-filling quality in the multiview images generated.
Keywords
Multiview images; Hole-filling; Depth image based rendering; Inpainting;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 P.J. Lee and Effendi, "Nongeometric Distortion Smoothing Approach for Depth Map Preprocessing," IEEE Trans. Multimedia, vol. 13, no. 2, Apr. 2011, pp. 246-254.   DOI
2 P. Ndjiki-Nya et al., "Depth Image-Based Rendering with Advanced Texture Synthesis for 3-D Video," IEEE Trans. Multimedia, vol. 13, no. 3, June 2011, pp. 453-465.   DOI
3 F. Zilly, J. Kluger, and P. Kauff, "Production Rules for Stereo Acquisition," Proc. IEEE, vol. 99, no. 4, Apr. 2011, pp. 590-606.   DOI
4 S.-W. Nam "Multiview Image Generation Technique Based on Depth Image by Hole-Filling Method," Ph.D. Thesis, Kyungpook National University, Rep. of Korea, June 2015.
5 T. Shibata et al., "The Zone of Comfort: Predicting Visual Discomfort with Stereo Displays," J. Vis., vol. 11, no. 8, July 2011, pp. 1-29.
6 A. Smolic et al., "3D Video and Free Viewpoint Video: Technologies, Applications and MPEG Standards," IEEE Intell. Conf. Multimedia Expo, Toronto, Canada, July 9-12, 2006, pp. 2161-2164.
7 K. Yun et al., "Repetitive Delivery Scheme for Left and Right Views in Service-Compatible 3D Video Service," ETRI J., vol. 36, no. 2, Apr. 2014, pp. 264-270.   DOI
8 M. Park et al., "Digital Holographic Display System with Large Screen Based on Viewing Window Movement for 3D Video Service," ETRI J., vol. 36, no. 2, Apr. 2014, pp. 232-241.   DOI
9 K. Muller, P. Merkle, and T. Wiegand, "3-D Video Representation Using Depth Maps," Proc. IEEE, vol. 99, no. 4, Apr. 2011, pp. 643-656.   DOI
10 C. Fehn, "Depth-Image-Based Rendering (DIBR), Compression, and Transmission for a New Approach on 3DTV," Proc. SPIE, vol. 5291, May 2004, pp. 93-104.
11 H. Urey et al., "State of the Art in Stereoscopic and Autostereoscopic Displays," Proc. IEEE, vol. 99, no. 4, Apr. 2011, pp. 540-555.   DOI
12 L. Zhang and W.J. Tam, "Stereoscopic Image Generation Based on Depth Images for 3D TV," IEEE Trans. Broadcast., vol. 51, no. 2, June 2005, pp. 191-199.   DOI
13 The Middlebury Stereo Datasets, Accessed Feb. 2, 2015. http://vision.Middlebury.edu/stereo/data.
14 D. Scharstein and R. Szeliski, "High-Accuracy Stereo Depth Maps Using Structured Light," IEEE Comput. Soc. Conf. Comput. Vision Pattern Recogn., Madision, WI, USA, 2003, pp. 195-202.
15 D. Scharstein and C. Pal, "Learning Conditional Random Fields for Stereo," IEEE Conf. Comput. Vision Pattern Recogn., Minneapolis, MN, USA, June 17-22, 2007, pp. 1-8.
16 H. Hirschmuller and D. Scharstein, "Evaluation of Cost Functions for Stereo Matching," IEEE Conf. Comput. Vision Pattern Recogn., Minneapolis, MN, USA, 2007, pp. 1-8.
17 S. Liu et al., "Kinect Depth Restoration via Energy Minimization with TV21 Regularization," IEEE Intell. Conf. Image Process., 2003, pp. 724-727.
18 K. Xu, J. Zhou, and Z. Wang, "A Method of Hole-Filling for the Depth Map Generated by Kinect with Moving Objects Detection," IEEE Int. Symp. Broadband Multimedia Syst. Broadcast., Seoul, Rep. of Korea, June 27-29, 2012, pp. 1-5.
19 A. Criminisi, P. Perez, and K. Toyama, "Region Filling and Object Removal by Exemplar-Based Image Inpainting," IEEE Trans. Image Process., vol. 13, no. 9, Sept. 2004, pp. 1200-1212.   DOI
20 A. Telea, "An Image Inpainting Technique Based on the Fast Marching Method," J. Graphics Tools, vol. 9, no. 1, 2004, pp. 23-34.   DOI
21 H. Lim et al., "Bi-Layer Inpainting for Novel View Synthesis," IEEE Int. Conf. Image Process., Brussels, Belgium, Sept. 11-14, 2011, pp. 1089-1092.
22 K.J. Oh, S. Yea, and Y.S. Ho, "Hole Filling Method Using Depth Based Inpainting for View Synthesis in Free Viewpoint Television and 3D Video," Proc. Conf. Picture Coding Symp., Chicago, II, USA, May 6-8, 2009, pp. 233-236.
23 M. Solh and G. AlRegib, "Hierarchical Hole-Filling for Depth-Based View Synthesis in FTV and 3D Video," IEEE J. Sel. Topics Signal Process., vol. 6, no. 5, Sept. 2012, pp. 495-504.   DOI