Browse > Article
http://dx.doi.org/10.7840/kics.2013.38C.5.409

Real-Time Virtual-View Image Synthesis Algorithm Using Kinect Camera  

Lee, Gyu-Cheol (광운대학교 전자공학과 디지털미디어 연구실)
Yoo, Jisang (광운대학교 전자공학과 디지털미디어 연구실)
Publication Information
The Journal of Korean Institute of Communications and Information Sciences / v.38C, no.5, 2013 , pp. 409-419 More about this Journal
Abstract
Kinect released by Microsoft in November 2010 is a motion sensing camera in xbox360 and gives depth and color images. However, Kinect camera also generates holes and noise around object boundaries in the obtained images because it uses infrared pattern. Also, boundary flickering phenomenon occurs. Therefore, we propose a real-time virtual-view video synthesis algorithm which results in a high-quality virtual view by solving these problems. In the proposed algorithm, holes around the boundary are filled by using the joint bilateral filter. Color image is converted into intensity image and then flickering pixels are searched by analyzing the variation of intensity and depth images. Finally, boundary flickering phenomenon can be reduced by converting values of flickering pixels into the maximum pixel value of a previous depth image and virtual views are generated by applying 3D warping technique. Holes existing on regions that are not part of occlusion region are also filled with a center pixel value of the highest reliability block after the final block reliability is calculated by using a block based gradient searching algorithm with block reliability. The experimental results show that the proposed algorithm generated the virtual view image in real-time.
Keywords
Kinect; Virtual viewpoint image; Depth image enhancement; Hole filling; 3D Warping;
Citations & Related Records
Times Cited By KSCI : 8  (Citation Analysis)
연도 인용수 순위
1 Retrevo Corporation, Could low interest in 3DTV hurt the TV business?, Retrieved Nov., 2011, from http://www.retrevo.com/content/node/1915.
2 G. M. Um, G. H. Cheong, W. S. Cheong, and N. H. Hur, "Technical development and standardization trends of multi-view 3D and free-viewpoint video," IEEK Mag., vol. 38, no. 2, pp. 18-23, Feb. 2011.
3 T. J. Kim and J. S. Yoo, "Hierarchical stereo matching with color information," J. KICS, vol. 34, no. 3, pp. 279-287, Mar. 2009.   과학기술학회마을
4 T. Leyvand, C. Meekhof, Y. C. Wei, J. Sun, and B. Guo, "Kinect identity: technology and experience," Computer (IEEE Comput. Soc.), vol. 44, no. 4, pp. 94-96, Apr. 2011.
5 Y. S. Park, S. M. Yun, and C. S. Won, "Hole filling for kinect depth image according to the causes of the holes," in Proc. Conf. Korean Soc. Broadcast Eng. (KOSBE), pp. 75-80, Jeju Island, Korea, July 2012.
6 C. J. Park, J. H. Ko, and E. S. Kim, "A new intermediate view reconstruction scheme based-on stereo image rectification algorithm," J. KICS, vol. 29, no. 5C, pp. 632-641, May. 2004.   과학기술학회마을
7 M. S. Ko and J. S. Yoo, "Boundary noise removal and hole filling algorithm for virtual viewpoint image generation," J. KICS, vol. 37, no. 8, pp. 679-688, Aug. 2012.   과학기술학회마을   DOI   ScienceOn
8 L. Zhao and H. Wang, "Image denoising using trivariate shrinkage filter in the wavelet domain and joint bilateral filter in the spatial Domain," IEEE Trans. Image Process., vol. 18, no. 10, pp. 2364-2369, Oct. 2009.   DOI   ScienceOn
9 A. Telea, "An image inpainting technique based on the fast marching method," J. Graphics Tools, vol. 9, no. 1, pp. 25-36, Dec. 2004.
10 A. Criminisi, P. Perez, and K. Toyama, "Region filling and object removal by exemplar-based image in-painting," IEEE Trans. Image Process., vol. 13, no. 9, pp. 1200-1212, Sept. 2004.   DOI   ScienceOn
11 T. Y. Kim, Y. G. Jeon, and J. C. Jeong, "Adaptive linear interpolation using the new distance weight and local patterns," J. KICS, vol. 31, no. 12C, pp. 1184-1193, Dec. 2006.   과학기술학회마을
12 G. C. Lee, Y. H. Seo, and J. S. Yoo, "GPGPU-based multiview synthesis using kinect depth image," in Proc. Conf. KICS, pp. 152-154, Yongpyong, Korea, Jan. 2012.
13 Y. Mori, N. Fukushima, T. Yendoa, T. Fujii, and M. Tanimotoa, "View generation with 3D warping using depth information for FTV," Signal Processing : Image Communication (ELSEVIER), vol. 24, no. 1-2, pp. 65-72, Jan. 2009.   DOI   ScienceOn
14 Y. J. Kim, S. H. Lee, and J. I. Park, "A high-quality occlusion filling method using image inpainting," J. Korean Soc. Broadcast Eng. (KOSBE), vol. 15, no. 1, pp. 3-13, Jan. 2010.   과학기술학회마을   DOI   ScienceOn
15 S. B. Lee and Y. S. Ho, "Real time eye contact system using a kinect depth camera for realistic telepresence," J. KICS, vol. 37, no. 4, pp. 277-282, Apr. 2012.   과학기술학회마을   DOI   ScienceOn
16 W. J. Tam, G. Alain, L. Zhang, T. Martin, and R. Renaud, "Smoothing depth maps for improved stereoscopic image quality," in Proc. SPIE Conf. Three-Dimensional TV, Video, Display III, vol. 5599, pp. 162-172, Philadelphia, U.S.A., Oct. 2004.
17 G. C. Lee and J. S. Yoo, "Kinect depth map enhancement using boundary flickering compensation," in Proc. Conf. Korean Soc. Broadcast Eng. (KOSBE), pp. 25-28, Seoul, Korea, Nov. 2012.
18 T. J. Kim, E. Y. Chang, N. H. Hur, J. W. Kim, and J. S. Yoo, "Virtual viewpoint image synthesis algorithm using multi-view geometry," J. KICS, vol. 34, no. 12, pp. 1154-1166, Dec. 2009.   과학기술학회마을
19 J. H. Park and C. G. Song, "Effective shadow removal from aerial image of golf course to extract components," J. Korean Inst. of Inform. Sci. Eng. (KIISE), vol. 39, no. 7, pp. 577-582, July 2012   과학기술학회마을
20 N. E. Yang, Y. G. Kim, and R. H. Hong, "Depth hole filling using the depth distribution of neighboring regions of depth holes in the kinect sensor," in Proc. IEEE Int. Conf. Signal Process., Commun. Comput. (ICSPCC), pp. 658-661, Hong Kong, China, Aug. 2012
21 S.-B. Lee and Y.-S. Ho, "Real-time stereo view generation using kinect depth camera," in Proc. Asia-Pacific Signal Inform. Process. Assoc. Annu. Summit Conf. (APSIPA ASC) 2011, pp. 1-4, Xi'an, China, Oct. 2011.