Browse > Article
http://dx.doi.org/10.7746/jkros.2018.13.4.230

Fusion System of Time-of-Flight Sensor and Stereo Cameras Considering Single Photon Avalanche Diode and Convolutional Neural Network  

Kim, Dong Yeop (Intelligent Robotics Research Center, Korea Electronics Technology Institute (KETI))
Lee, Jae Min (Intelligent Robotics Research Center, Korea Electronics Technology Institute (KETI))
Jun, Sewoong (Intelligent Robotics Research Center, Korea Electronics Technology Institute (KETI))
Publication Information
The Journal of Korea Robotics Society / v.13, no.4, 2018 , pp. 230-236 More about this Journal
Abstract
3D depth perception has played an important role in robotics, and many sensory methods have also proposed for it. As a photodetector for 3D sensing, single photon avalanche diode (SPAD) is suggested due to sensitivity and accuracy. We have researched for applying a SPAD chip in our fusion system of time-of-fight (ToF) sensor and stereo camera. Our goal is to upsample of SPAD resolution using RGB stereo camera. Currently, we have 64 x 32 resolution SPAD ToF Sensor, even though there are higher resolution depth sensors such as Kinect V2 and Cube-Eye. This may be a weak point of our system, however we exploit this gap using a transition of idea. A convolution neural network (CNN) is designed to upsample our low resolution depth map using the data of the higher resolution depth as label data. Then, the upsampled depth data using CNN and stereo camera depth data are fused using semi-global matching (SGM) algorithm. We proposed simplified fusion method created for the embedded system.
Keywords
Sensor Fusion; Time-of-Flight; Stereo camera; Single Photon Avalanche Diodes; Convolution Neural Network;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 S.-H. Kim, D.-K. Lee, J.-H. Cheon, S.-J. Kim, and K.-H. Yu, "Design and Flight Tests of a Drone for Delivery Service," Journal of Institute of Control, Robotics and System, vol. 22, no. 3, pp. 204-209, Mar., 2016.   DOI
2 S. Cho, D. Lee, Y. Jung, U. Lee, and D. H. Shim, "Development of a Cooperative Heterogeneous Unmanned System for Delivery Services," Journal of Institute of Control, Robotics and System, vol. 20, no. 12, pp. 1181-1188, Dec., 2014.   DOI
3 V. Magnanimo, S. Walther, L Tecchia. C. Natale, and T. Guhl, "Safeguarding a mobile manipulator using dynamic safety fields," 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, South Korea, pp. 2972-2977, 2016.
4 J.-H. Moon, H.-D. Choi, N.-H. Park, C.-H. Kim, Y.-W. Park, and E.-T. Kim, "Database based global positioning system correction," Journal of Korea Robotics Society, vol. 7, No. 3, pp. 205-215, Sep., 2012.   DOI
5 C. Song and S.-H. Kim, "Robust vehicle occupant detection based on rgb-depth-thermal camera," Journal of Korea Robotics Society, vol. 13, No.1, pp. 31-37, Mar., 2018.   DOI
6 D. Shin, B. Park, Y. Chae, and I. Yun, "Structure variation effects on device reliability of single photon avalanche diodes," Microelectronics Reliability, vol. 76-77, pp. 610-613, Sep., 2014.
7 Y. Choi, N. Kim, S. Hwang, and I. S. Kwon, "Thermal image enhancement using convolutional neural network," 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, South Korea, pp. 223-230, 2016.
8 S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, and D. Liang, "Accelerating magnetic resonance imaging via deep learning," 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), Prague, Czech Republic, pp. 514-517, 2016.
9 B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen, and M. S. Rosen, "Image reconstruction by domain-transform manifold learning," Nature, vol. 555, pp. 487-492, Mar., 2018.   DOI
10 J. Park, H. Kim, Y.-W. Tai, M. S. Brown, and I. Kwon, "High quality depth map upsampling for 3d-tof cameras," 2011 International Conference on Computer Vision, Barcelona, Spain, pp. 1623-1630, 2011.
11 R. Nair, F. Lenzen, S. Meister, H. Schafer, C. Garbe, and D. Kondermann, "High accuracy tof and stereo sensor fusion at interactive rates," European Conference on Computer Vision, pp. 1-11, 2012.
12 G. Agresti, L. Minto, G. Marlin, and P. Zanuttigh, "Deep learning for confidence information in stereo and TOF data fusion," 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy, pp. 697-705, 2017.
13 Z. Zhang, "A flexible new technique for camera calibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp. 1330-1334, Dec., 2000.   DOI
14 http://www.cube-eye.co.kr/en/#/spec/product_MDC600S.html, [Online] Accessed: Jul. 24, 2018.
15 J. M. Lee, D. Y. Kim, I. Kim, S. Jun, and C.-W. Park, "A study on heterogeneous sensor calibration between ToF sensor and RGB camera," Korean Society for Precision Engineering Spring Conference, Jeju, Korea, pp. 108-109, 2016.
16 D. Y. Kim, Y.-O. Kim, C.-W. Park, J.-M. Lee, and S. Jun, "System and method for concurrent calibration of camera and depth sensor," Korea Patent, Application No. 10-2015-0095091, Jul., 3, 2015. DOI: 10.8080/1020150095091.
17 D. Y. Kim, C.-W. Park, J.-M. Lee, Y.-O. Kim, I. Kim, S. Jun, and J.-H. Hwang, "System and method for 3-dimensional calibration of camera and depth sensor, and calibration device and calibration jig thereof," Korea Patent, Registration No. 10-1803705, Nov., 2017. DOI: 10.8080/1020160057145.
18 D. Hernandez-Juarez, A. Chacon, A. Espinosa, D. Vazquez, J. C. Moure, and A. M. Lopez, "Embedded real-time stereo estimation via semi-global matching on the GPU," Procedia Computer Science, Vol. 80, pp. 143-153, Jun., 2016.   DOI
19 C. Dong, C. C. Loy, and X. Tang, "Accelerating the super-resolution convolutional neural network," European Conference on Computer Vision, Amsterdam, The Netherlands, pp. 391-407, 2016.