Browse > Article

A Robust Depth Map Upsampling Against Camera Calibration Errors  

Kim, Jae-Kwang (Dept. of Electrical Engineering, Korea Advanced Institute of Science and Technology)
Lee, Jae-Ho (Dept. of Electrical Engineering, Korea Advanced Institute of Science and Technology)
Kim, Chang-Ick (Dept. of Electrical Engineering, Korea Advanced Institute of Science and Technology)
Publication Information
Journal of the Institute of Electronics Engineers of Korea SP / v.48, no.6, 2011 , pp. 8-17 More about this Journal
Abstract
Recently, fusion camera systems that consist of depth sensors and color cameras have been widely developed with the advent of a new type of sensor, time-of-flight (TOF) depth sensor. The physical limitation of depth sensors usually generates low resolution images compared to corresponding color images. Therefore, the pre-processing module, such as camera calibration, three dimensional warping, and hole filling, is necessary to generate the high resolution depth map that is placed in the image plane of the color image. However, the result of the pre-processing step is usually inaccurate due to errors from the camera calibration and the depth measurement. Therefore, in this paper, we present a depth map upsampling method robust these errors. First, the confidence of the measured depth value is estimated by the interrelation between the color image and the pre-upsampled depth map. Then, the detailed depth map can be generated by the modified kernel regression method which exclude depth values having low confidence. Our proposed algorithm guarantees the high quality result in the presence of the camera calibration errors. Experimental comparison with other data fusion techniques shows the superiority of our proposed method.
Keywords
Depth Map Upsampling; Hybrid Camera System; Joint Bilateral Filter; Time-of-Flight Camera;
Citations & Related Records
연도 인용수 순위
  • Reference
1 J. Zhu, L. Wang, R. Yang, and J. Davis, "Fusion of Time-of-Flight Depth and Stereo for High Accuracy Depth Maps," in Proc. IEEE International Conference on Computer Vision and Pattern Recognition, 2008.
2 Y. Kim, D. Chen, C, Theobalt, and S. Thrun, "Design and Calibration of a Multi-view TOF Sensor Fusion System," in Proc. IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1-7, 2008.
3 J. Diebel and S. Thrun, "An Application of Markov Random Fields to Range Sensing," Advances in Neural Information Processing Systems, 18:291-298, 2005.
4 J. Zhu, L. Wang, J. Gao, and R. Yang, "Spatial-Temporal Fusion for High Accuracy Depth Maps Using Dynamic MRFs," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 32, no. 5, pp. 899-909, 2010.   DOI
5 E. Eisemann and F. Durand, "Flash Photography Enhancement via Intrinsic Relighting," ACM Trans. on Graphics, vol. 23, no. 3, pp. 673-678. 2004.   DOI
6 G. Petschnigg, M. Agrawala, and H. Hoppe, "Digital Photography with Flash and No-Flash Image Pairs," ACM Trans. on Graphics, vol. 23, no. 3, pp. 664-672. 2004.   DOI
7 C. Tomasi and R. Manduchi, "Bilateral Filtering for Gray and Color Images," in Proc. IEEE International Conference on Computer Vision, pp. 836-846, 1998.
8 J. Kopf, M. Cohen, D. Lischinski, and M. Uyttendaele, "Joint Bilateral Upsampling," ACM Trans. on Graphics, vol. 26, no. 3, 2007.
9 D. Chan, H. Buisman, C. Thebalt, and S. Thrun, "A Noise-Aware Filter for Real-Time Depth Upsampling," in Workshop on M2SFA2, ECCV, 2008.
10 C. Fehn, "Depth-Image-Based Rendering (DIBR), Compression and Transmission for a New Approach on 3D-TV," in Proc. of SPIE, Stereoscopic Displays and Virtual Reality Systems XI, vol. 5291, pp. 93-104, 2004.
11 Camera calibration toolbox program for Matlab provided by Caltech, http://www.vision.caltech. edu/bouguetj/calib_doc
12 SY. Kim, EK. Lee, and YS. Ho, "Generation of ROI Enhanced Depth Maps Using Stereoscopic Cameras and a Depth Camera," IEEE Trans. on Broadcasting, vol. 54, no. 4, pp. 732-740, 2008.   DOI
13 Y. Qingxiong, Y. Ruigang, J. Davis, and D. Nister, "Spatial-Depth Super Resolution for Range Images," in Proc. IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1-8. 2007.
14 http://www.middleburry.edu/stereo.
15 A. Reimens, O. Gangwal, B. Barenbrug, and RM. Berretty, "Multi-step Joint Bilateral Depth Upsampling," in Proc. SPIE Visual Communication and Image Processing, 2009.