Journal of IKEEE (전기전자학회논문지)

Institute of Korean Electrical and Electronics Engineers (한국전기전자학회)
권호 표지
DOI QR코드

DOI QR Code

Hole-filling Algorithm Based on Extrapolating Spatial-Temporal Background Information for View Synthesis in Free Viewpoint Television

자유 시점 TV에서 시점 합성을 위한 시공간적 배경 정보 추정 기반 홀 채움 방식

  • Kim, Beomsu (A School of Electronic Engineering, Soongsil University) ;
  • Nguyen, Tien-Dat (A School of Electronic Engineering, Soongsil University) ;
  • Hong, Min-cheol (A School of Electronic Engineering, Soongsil University)
  • Received : 2016.01.21
  • Accepted : 2016.02.22
  • Published : 2016.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

This paper presents a hole-filling algorithm based on extrapolating spatial-temporal background information used in view synthesis for free-viewpoint television. A new background codebook is constructed and updated in order to extract reliable temporal background information. In addition, an estimation of spatial local background values is conducted to discriminate an adaptive boundary between the background region and the foreground region as well as to update the information about the hole region. The holes then are filled by combining the spatial background information and the temporal background information. In addition, an exemplar-based inpainting technique is used to fill the rest of holes, in which a priority function using background-depth information is defined to determine the order in which the holes are filled. The experimental results demonstrated that the proposed algorithm outperformed the other comparative methods about average 0.3-0.6 dB, and that it synthesized satisfactory views regardless of video characteristics and type of hole region.

본 논문에서는 자유 시점 텔레비전에서 시점 합성 영상 획득을 위해 시공간적 배경 정보 추정 기반 홀 채움 방식을 제안한다. 신뢰할 수 있는 시간적 배경 정보를 획득하기 위해 새로운 배경 코드북의 구성 및 갱신하는 과정을 수행한다. 더불어, 공간적인 국부 배경 정보 추정을 위해 홀 영역의 배경 및 전경 영역의 구별 및 갱신 과정을 수행한다. 추정된 시공간 배경 정보를 조합하여 홀 채움 과정을 수행하고, 잔여 홀 채움을 수행하기 위해 깊이 배경 정보를 이용한 우선순위 함수를 결정하여 표본 기반 인페인팅 기법을 적용한다. 실험 결과를 통해 제안 방식은 기존방식들과 비교하여 평균 0.3~0.6dB의 성능 향상이 있음을 확인하였으며, 동영상 특성 및 홀 형태에 관계없이 제안된 방식이 새로운 시점 영상을 효과적으로 합성할 수 있음을 확인할 수 있었다.

Keywords

References

  1. A. Smolic, P. Kauff, S. Knorr, A. Hornung, M. Kunter, M. Muller, and M. Lang, "Three-dimensional video postproduction and processing," Proc. of the IEEE, vol. 99, no. 4, pp. 607-625, Apr. 2011. https://doi.org/10.1109/JPROC.2010.2098350
  2. N. A. Dodgson, "Autostereoscopic 3D displays", Computer, vol. 38, no. 8, pp.31-36, Aug. 2005. https://doi.org/10.1109/MC.2005.252
  3. C. Fehn, "Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D TV," SPIE Proc. Stereoscopic Image Proc. and Rendering, pp. 93-104, May 2004.
  4. P.-J. Lee, and Effendi, "Nongeometric distortion smoothing approach for depth map preprocessing," IEEE Trans. Multimedia, vol. 13, no. 2, pp. 246-254, Apr. 2011. https://doi.org/10.1109/TMM.2010.2100372
  5. S. Zinger, D. Ruijters, L. Do and P. H. N. de With, "View interpolation for medical images on autostereoscopic displays," IEEE Trans. Circuits and Systems for Video Tech., vol. 21, no. 5-6, pp. 533-541, Jan. 2012.
  6. S. Zinger, L. Do and P. H. N. de With, "Free-viewpoint depth image based rendering," J. of Visual Commu. and Image Representation, vol. 21, no. 5-6, pp. 533-541, Jul. 2012. https://doi.org/10.1016/j.jvcir.2010.01.004
  7. M. Solh and G. Alregib, "Hierarchical hole-filling for depth-based view synthesis in FTV and 3D video," IEEE J. of Selected Topics in Signal Processing, vol. 6, no. 5, pp. 495-504, Sep. 2012 https://doi.org/10.1109/JSTSP.2012.2204723
  8. B. Marcelo, S. Guillermo, C. Vincent and B. Coloma, "Image inpainting," The 27th Annual Conf. on Comput. Graphics and Interactive Techniques, pp. 417-424, 2000.
  9. K. Oh, S. Yea and Y.-S. Ho, "Hole-filling method using depth based inpainting for view synthesis in free viewpoint television and 3D video," Picture Coding Symposium, pp. 1-4, June 2009.
  10. I. Daribo, H. Saito, "A novel inpainting-based layered depth video for 3D TV," IEEE Trans. Broadcasting, vol. 57, no. 2, pp. 533-541, June 2011. https://doi.org/10.1109/TBC.2011.2125110
  11. A. Criminisi, P. Perez and K. Toyama, "Object removal by exemplar-based inpainting," IEEE Conf. on Computer Vision and Pattern Recognition, pp. 721-728, June 2003.
  12. M. Koppel, X. Wang, D. Doshkov, T. Wiegand and P. Ndjiki-Nya, "Consitent spatio-temporal filling of disocclusions in the multiview video plus depth format," IEEE Int. Workshop on Multimedia Signal Process., pp. 25-30, Sept. 2012.
  13. M. Xi, L.-H. Wang, Q.-Q. Yang, D.-X. Li and M. Zhang, "Depth image based rendering with spatial and temporal texture synthesis for 3DTV," EURASIP J. on Image and Video Process., vol. 2013, no. 1, DOI:10.1186/1687-5281-2013-9, Feb. 2013.
  14. C. Yao, T. Tillo, Y. Zhao, J. Xiao, H. Bai and C. Lin, "Depth map driven hole filling algorithm exploiting temporal correlation information," IEEE Trans. Broadcasting, vol. 60, no. 2, pp. 394-404, June 2014. https://doi.org/10.1109/TBC.2014.2321671
  15. K. Kim, T. H. Chalidabhongse, D. Harwood and L. Davis, "Background modeling and subtraction by codebook construction," IEEE Int. Conf. on Image Processing, vol. 5, pp. 3061-3064, Oct. 2004.
  16. E. J. Fernandez-Sanchez, J.r Diaz and E. Ros, "Background subtraction based on color and depth using active sensors", Sensors, vol. 13, no. 7, pp. 8895-8915, July 2013. https://doi.org/10.3390/s130708895
  17. A. S. Glassner, Graphics Gems, Academic Press, Cambridge, MA, 1990.
  18. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE Trans. Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004. https://doi.org/10.1109/TIP.2003.819861