Browse > Article
http://dx.doi.org/10.5573/ieie.2017.54.8.67

Hole-Filling Method Using Extrapolated Spatio-temporal Background Information  

Kim, Beomsu (School of Electronic Engineering, Soongsil University)
Nguyen, Tien Dat (School of Electronic Engineering, Soongsil University)
Hong, Min-Cheol (School of Electronic Engineering, Soongsil University)
Publication Information
Journal of the Institute of Electronics and Information Engineers / v.54, no.8, 2017 , pp. 67-80 More about this Journal
Abstract
This paper presents a hole-filling method using extrapolated spatio-temporal background information to obtain a synthesized view. A new temporal background model using non-overlapped patch based background codebook is introduced to extrapolate temporal background information In addition, a depth-map driven spatial local background estimation is addressed to define spatial background constraints that represent the lower and upper bounds of a background candidate. Background holes are filled by comparing the similarities between the temporal background information and the spatial background constraints. Additionally, a depth map-based ghost removal filter is described to solve the problem of the non-fit between a color image and the corresponding depth map of a virtual view after 3-D warping. Finally, an inpainting is applied to fill in the remaining holes with the priority function that includes a new depth term. The experimental results demonstrated that the proposed method led to results that promised subjective and objective improvement over the state-of-the-art methods.
Keywords
Hole-filling; Temporal background modeling; Local background estimation; Ghost removal; Inpainting;
Citations & Related Records
연도 인용수 순위
  • Reference
1 C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, "High-quality video view interpolation using a layered representation," ACM Trans. Graphics, vol. 23, no. 3, pp. 600-608, Aug. 2004.   DOI
2 M. Solh and G. AlRegib, "Hierarchical hole-filling for depth-based view synthesis in FTV and 3D video," IEEE. J. Selected Topic in Signal Process., vol.6, no.5, pp. 495-504, Sep. 2012.   DOI
3 M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, "Image inpainting," in Proc. ACM the 27th Int. Conf. Computer Graphics and Interactive Tech. (SIGGRAPH), pp. 417-424, July 2000.
4 A. Criminisi, P. Perez, and K. Toyama, "Object removal by exemplarbased inpainting," in Proc. IEEE Int. Conf. Computer Vision and Pattern Recog. (CVPR), doi:10.1109/CVPR.2003.1211538, July 2003.
5 C. Guillemot and O. Le Meur, "Image inpainting: Overview and recent advances," IEEE Signal Process. Mag., vol. 31, no. 1, pp. 127-144, Jan. 2014.   DOI
6 Y. Mori, N. Fukushima, T. Fujii, and M. Tanimoto, "View generation with 3D warping using depth information for FTV," Signal Process.: Image Commu., vol. 24, no. 1-2, pp. 65-72, Jan. 2009.   DOI
7 K. Oh, S. Yea, and Y.-S. Ho, "Hole filling method using depth based inpainting for view synthesis in free viewpoint television and 3-D video," in Proc. Picture Coding Symp., doi:10.1109/PCS.2009.5167450, July 2009.
8 S. Zinger, L. Do, and P. H. N. de With, "Free-viewpoint depth image based rendering," J. Vis. Comm. Image Represent., vol. 21, no. 5-6, pp. 533-541, July 2010.   DOI
9 I. Daribo and H. Saito, "A novel inpainting-based layered depth video for 3DTV," IEEE Trans. Broadcast., vol. 57, no. 2, pp. 533-541, June 2011.   DOI
10 I. Ahn and C. Kim, "A depth-based disocclusion filling method for virtual viewpoint image synthesis," Journal of the Institute of Electronics Engineers of Korea SP, vol. 48, no. 6, pp. 48-60, Nov. 2011.
11 S. Choi, B. Ham, and K. Sohn, "Space-time hole filling with random walks in view extrapolation for 3D video," IEEE. Trans. Image Process., vol. 22, no. 6, pp. 2429-2441, June 2013.   DOI
12 S. M. Muddala, M. Sjostrom, R. Olsson, "Virtual view synthesis using layered depth image generation and depth-based inpainting for filling disocclusions and translucent disocclusion," J. Vis. Commu. Image Represent., vol. 38, pp. 351-366, July 2016.   DOI
13 P. Mdjiki-Nya, M. Koppel, D. Doshkov, H. Lakshman, P. Merkle, K. Muller, and T. Wiegand, "Depth image-based rendering with advanced texture synthesis for 3-D Video," IEEE Trans. Multimedia, vol. 13, no. 3, pp. 453-465, June 2011.   DOI
14 M. Koppel, X. Wang, D. Doshkov, T. Wiegand, and P. Ndjiki-Nya, "Consistent spatio-temporal filling of disocclusions in the multiview-video-plus-depth format," in Proc. IEEE Int. Workshop Multimedia Signal Process., pp. 25-30, Sep. 2012.
15 M. Xi, L.-H. Wang, Q.-Q. Yang, D.-X. Li, and M. Zhang, "Depthimage-based rendering with spatial and temporal texture synthesis for 3DTV," EURASIP J. Image and Video Process., doi:10.1186/1687-5281-2013-9, Dec. 2013.   DOI
16 C. Yao, T. Tillo, X. Jimin, "Depth map driven hole filling algorithm exploiting temporal correlation information," IEEE Trans. Broadcast., vol. 60, no. 2, pp. 394-404, June 2014.   DOI
17 K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis, "Background modeling and subtraction by codebook construction," in IEEE Int. Conf. Image Process., pp. 3061-3064, Oct. 2004.
18 W. Zhou, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE Trans. Image Process., vol. 13, no. 4, pp. 600-612, Apr. 2004.   DOI
19 E. J. Fernandez-Sanchez, D. Javier, and R. Eduardo, "Background subtraction based on color and depth using active sensors," Sensors, vol. 13, no. 7, pp. 8895-8915, July 2013.   DOI
20 A. S. Glassner, Ed., Graphics Gems I, Academic Press, Cambridge, MA, 1990.
21 L. Zhang and W. J. Tam, "Stereoscopic image generation based on depth images for 3D TV," IEEE Trans. Broadcast., vol. 51, no. 2, pp. 191-199, June 2005.   DOI
22 C. Fehn, "Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D TV," in SPIE Int. Conf. Stereoscopic Display and Virtual Reality Syst. XI, pp. 93-104, Jan. 2004,
23 P. Kauff, N. Atzpadin, C. Fehn, M. Muller, O. Schreer, A. Smolic, and R. Tanger, "Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability," Signal Process.: Image Commu., vol. 22, no. 2, pp. 217-234, Feb. 2007.   DOI
24 C. Zhu and S. Li, "Depth image based view synthesis: new insights and perspectives on hole generation and filling," IEEE Trans. Broadcast., vol. 62, no. 1, pp. 82-93, Mar. 2016.   DOI