Browse > Article
http://dx.doi.org/10.7471/ikeee.2016.20.1.031

Hole-filling Algorithm Based on Extrapolating Spatial-Temporal Background Information for View Synthesis in Free Viewpoint Television  

Kim, Beomsu (A School of Electronic Engineering, Soongsil University)
Nguyen, Tien-Dat (A School of Electronic Engineering, Soongsil University)
Hong, Min-cheol (A School of Electronic Engineering, Soongsil University)
Publication Information
Journal of IKEEE / v.20, no.1, 2016 , pp. 31-44 More about this Journal
Abstract
This paper presents a hole-filling algorithm based on extrapolating spatial-temporal background information used in view synthesis for free-viewpoint television. A new background codebook is constructed and updated in order to extract reliable temporal background information. In addition, an estimation of spatial local background values is conducted to discriminate an adaptive boundary between the background region and the foreground region as well as to update the information about the hole region. The holes then are filled by combining the spatial background information and the temporal background information. In addition, an exemplar-based inpainting technique is used to fill the rest of holes, in which a priority function using background-depth information is defined to determine the order in which the holes are filled. The experimental results demonstrated that the proposed algorithm outperformed the other comparative methods about average 0.3-0.6 dB, and that it synthesized satisfactory views regardless of video characteristics and type of hole region.
Keywords
hole-filling; spatial-temporal background information; background codebook; priority function; DIBR; View Synthesis;
Citations & Related Records
연도 인용수 순위
  • Reference
1 A. Smolic, P. Kauff, S. Knorr, A. Hornung, M. Kunter, M. Muller, and M. Lang, "Three-dimensional video postproduction and processing," Proc. of the IEEE, vol. 99, no. 4, pp. 607-625, Apr. 2011.   DOI
2 N. A. Dodgson, "Autostereoscopic 3D displays", Computer, vol. 38, no. 8, pp.31-36, Aug. 2005.   DOI
3 C. Fehn, "Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D TV," SPIE Proc. Stereoscopic Image Proc. and Rendering, pp. 93-104, May 2004.
4 P.-J. Lee, and Effendi, "Nongeometric distortion smoothing approach for depth map preprocessing," IEEE Trans. Multimedia, vol. 13, no. 2, pp. 246-254, Apr. 2011.   DOI
5 S. Zinger, D. Ruijters, L. Do and P. H. N. de With, "View interpolation for medical images on autostereoscopic displays," IEEE Trans. Circuits and Systems for Video Tech., vol. 21, no. 5-6, pp. 533-541, Jan. 2012.
6 S. Zinger, L. Do and P. H. N. de With, "Free-viewpoint depth image based rendering," J. of Visual Commu. and Image Representation, vol. 21, no. 5-6, pp. 533-541, Jul. 2012.   DOI
7 M. Solh and G. Alregib, "Hierarchical hole-filling for depth-based view synthesis in FTV and 3D video," IEEE J. of Selected Topics in Signal Processing, vol. 6, no. 5, pp. 495-504, Sep. 2012   DOI
8 B. Marcelo, S. Guillermo, C. Vincent and B. Coloma, "Image inpainting," The 27th Annual Conf. on Comput. Graphics and Interactive Techniques, pp. 417-424, 2000.
9 K. Oh, S. Yea and Y.-S. Ho, "Hole-filling method using depth based inpainting for view synthesis in free viewpoint television and 3D video," Picture Coding Symposium, pp. 1-4, June 2009.
10 I. Daribo, H. Saito, "A novel inpainting-based layered depth video for 3D TV," IEEE Trans. Broadcasting, vol. 57, no. 2, pp. 533-541, June 2011.   DOI
11 A. Criminisi, P. Perez and K. Toyama, "Object removal by exemplar-based inpainting," IEEE Conf. on Computer Vision and Pattern Recognition, pp. 721-728, June 2003.
12 M. Koppel, X. Wang, D. Doshkov, T. Wiegand and P. Ndjiki-Nya, "Consitent spatio-temporal filling of disocclusions in the multiview video plus depth format," IEEE Int. Workshop on Multimedia Signal Process., pp. 25-30, Sept. 2012.
13 M. Xi, L.-H. Wang, Q.-Q. Yang, D.-X. Li and M. Zhang, "Depth image based rendering with spatial and temporal texture synthesis for 3DTV," EURASIP J. on Image and Video Process., vol. 2013, no. 1, DOI:10.1186/1687-5281-2013-9, Feb. 2013.   DOI
14 C. Yao, T. Tillo, Y. Zhao, J. Xiao, H. Bai and C. Lin, "Depth map driven hole filling algorithm exploiting temporal correlation information," IEEE Trans. Broadcasting, vol. 60, no. 2, pp. 394-404, June 2014.   DOI
15 K. Kim, T. H. Chalidabhongse, D. Harwood and L. Davis, "Background modeling and subtraction by codebook construction," IEEE Int. Conf. on Image Processing, vol. 5, pp. 3061-3064, Oct. 2004.
16 E. J. Fernandez-Sanchez, J.r Diaz and E. Ros, "Background subtraction based on color and depth using active sensors", Sensors, vol. 13, no. 7, pp. 8895-8915, July 2013.   DOI
17 A. S. Glassner, Graphics Gems, Academic Press, Cambridge, MA, 1990.
18 Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE Trans. Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004.   DOI