ETRI Journal

  • 제38권5호
  • /
  • Pages.996-1007
  • /
  • 2016
  • /
  • 1225-6463(pISSN)
  • /
  • 2233-7326(eISSN)
한국전자통신연구원 (Electronics and Telecommunications Research Institute)
권호 표지
DOI QR코드

DOI QR Code

Hole-Filling Methods Using Depth and Color Information for Generating Multiview Images

  • 투고 : 2016.02.01
  • 심사 : 2016.07.04
  • 발행 : 2016.10.01
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

This paper presents new hole-filling methods for generating multiview images by using depth image based rendering (DIBR). Holes appear in a depth image captured from 3D sensors and in the multiview images rendered by DIBR. The holes are often found around the background regions of the images because the background is prone to occlusions by the foreground objects. Background-oriented priority and gradient-oriented priority are also introduced to find the order of hole-filling after the DIBR process. In addition, to obtain a sample to fill the hole region, we propose the fusing of depth and color information to obtain a weighted sum of two patches for the depth (or rendered depth) images and a new distance measure to find the best-matched patch for the rendered color images. The conventional method produces jagged edges and a blurry phenomenon in the final results, whereas the proposed method can minimize them, which is quite important for high fidelity in stereo imaging. The experimental results show that, by reducing these errors, the proposed methods can significantly improve the hole-filling quality in the multiview images generated.

키워드

참고문헌

  1. A. Smolic et al., "3D Video and Free Viewpoint Video: Technologies, Applications and MPEG Standards," IEEE Intell. Conf. Multimedia Expo, Toronto, Canada, July 9-12, 2006, pp. 2161-2164.
  2. K. Yun et al., "Repetitive Delivery Scheme for Left and Right Views in Service-Compatible 3D Video Service," ETRI J., vol. 36, no. 2, Apr. 2014, pp. 264-270. https://doi.org/10.4218/etrij.14.2113.0043
  3. M. Park et al., "Digital Holographic Display System with Large Screen Based on Viewing Window Movement for 3D Video Service," ETRI J., vol. 36, no. 2, Apr. 2014, pp. 232-241. https://doi.org/10.4218/etrij.14.2113.0086
  4. K. Muller, P. Merkle, and T. Wiegand, "3-D Video Representation Using Depth Maps," Proc. IEEE, vol. 99, no. 4, Apr. 2011, pp. 643-656. https://doi.org/10.1109/JPROC.2010.2091090
  5. C. Fehn, "Depth-Image-Based Rendering (DIBR), Compression, and Transmission for a New Approach on 3DTV," Proc. SPIE, vol. 5291, May 2004, pp. 93-104.
  6. H. Urey et al., "State of the Art in Stereoscopic and Autostereoscopic Displays," Proc. IEEE, vol. 99, no. 4, Apr. 2011, pp. 540-555. https://doi.org/10.1109/JPROC.2010.2098351
  7. L. Zhang and W.J. Tam, "Stereoscopic Image Generation Based on Depth Images for 3D TV," IEEE Trans. Broadcast., vol. 51, no. 2, June 2005, pp. 191-199. https://doi.org/10.1109/TBC.2005.846190
  8. D. Scharstein and R. Szeliski, "High-Accuracy Stereo Depth Maps Using Structured Light," IEEE Comput. Soc. Conf. Comput. Vision Pattern Recogn., Madision, WI, USA, 2003, pp. 195-202.
  9. D. Scharstein and C. Pal, "Learning Conditional Random Fields for Stereo," IEEE Conf. Comput. Vision Pattern Recogn., Minneapolis, MN, USA, June 17-22, 2007, pp. 1-8.
  10. H. Hirschmuller and D. Scharstein, "Evaluation of Cost Functions for Stereo Matching," IEEE Conf. Comput. Vision Pattern Recogn., Minneapolis, MN, USA, 2007, pp. 1-8.
  11. The Middlebury Stereo Datasets, Accessed Feb. 2, 2015. http://vision.Middlebury.edu/stereo/data.
  12. S. Liu et al., "Kinect Depth Restoration via Energy Minimization with TV21 Regularization," IEEE Intell. Conf. Image Process., 2003, pp. 724-727.
  13. K. Xu, J. Zhou, and Z. Wang, "A Method of Hole-Filling for the Depth Map Generated by Kinect with Moving Objects Detection," IEEE Int. Symp. Broadband Multimedia Syst. Broadcast., Seoul, Rep. of Korea, June 27-29, 2012, pp. 1-5.
  14. A. Criminisi, P. Perez, and K. Toyama, "Region Filling and Object Removal by Exemplar-Based Image Inpainting," IEEE Trans. Image Process., vol. 13, no. 9, Sept. 2004, pp. 1200-1212. https://doi.org/10.1109/TIP.2004.833105
  15. A. Telea, "An Image Inpainting Technique Based on the Fast Marching Method," J. Graphics Tools, vol. 9, no. 1, 2004, pp. 23-34. https://doi.org/10.1080/10867651.2004.10487596
  16. K.J. Oh, S. Yea, and Y.S. Ho, "Hole Filling Method Using Depth Based Inpainting for View Synthesis in Free Viewpoint Television and 3D Video," Proc. Conf. Picture Coding Symp., Chicago, II, USA, May 6-8, 2009, pp. 233-236.
  17. M. Solh and G. AlRegib, "Hierarchical Hole-Filling for Depth-Based View Synthesis in FTV and 3D Video," IEEE J. Sel. Topics Signal Process., vol. 6, no. 5, Sept. 2012, pp. 495-504. https://doi.org/10.1109/JSTSP.2012.2204723
  18. H. Lim et al., "Bi-Layer Inpainting for Novel View Synthesis," IEEE Int. Conf. Image Process., Brussels, Belgium, Sept. 11-14, 2011, pp. 1089-1092.
  19. P.J. Lee and Effendi, "Nongeometric Distortion Smoothing Approach for Depth Map Preprocessing," IEEE Trans. Multimedia, vol. 13, no. 2, Apr. 2011, pp. 246-254. https://doi.org/10.1109/TMM.2010.2100372
  20. P. Ndjiki-Nya et al., "Depth Image-Based Rendering with Advanced Texture Synthesis for 3-D Video," IEEE Trans. Multimedia, vol. 13, no. 3, June 2011, pp. 453-465. https://doi.org/10.1109/TMM.2011.2128862
  21. F. Zilly, J. Kluger, and P. Kauff, "Production Rules for Stereo Acquisition," Proc. IEEE, vol. 99, no. 4, Apr. 2011, pp. 590-606. https://doi.org/10.1109/JPROC.2010.2095810
  22. S.-W. Nam "Multiview Image Generation Technique Based on Depth Image by Hole-Filling Method," Ph.D. Thesis, Kyungpook National University, Rep. of Korea, June 2015.
  23. T. Shibata et al., "The Zone of Comfort: Predicting Visual Discomfort with Stereo Displays," J. Vis., vol. 11, no. 8, July 2011, pp. 1-29.

피인용 문헌

  1. 360-degree color hologram generation for real 3D objects vol.57, pp.1, 2016, https://doi.org/10.1364/ao.57.000a91