로봇학회논문지 (The Journal of Korea Robotics Society)

  • 제13권4호
  • /
  • Pages.230-236
  • /
  • 2018
  • /
  • 1975-6291(pISSN)
  • /
  • 2287-3961(eISSN)
한국로봇학회 (Korea Robotics Society)
권호 표지
DOI QR코드

DOI QR Code

SPAD과 CNN의 특성을 반영한 ToF 센서와 스테레오 카메라 융합 시스템

Fusion System of Time-of-Flight Sensor and Stereo Cameras Considering Single Photon Avalanche Diode and Convolutional Neural Network

  • Kim, Dong Yeop (Intelligent Robotics Research Center, Korea Electronics Technology Institute (KETI)) ;
  • Lee, Jae Min (Intelligent Robotics Research Center, Korea Electronics Technology Institute (KETI)) ;
  • Jun, Sewoong (Intelligent Robotics Research Center, Korea Electronics Technology Institute (KETI))
  • 투고 : 2018.09.06
  • 심사 : 2018.11.14
  • 발행 : 2018.11.30
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

3D depth perception has played an important role in robotics, and many sensory methods have also proposed for it. As a photodetector for 3D sensing, single photon avalanche diode (SPAD) is suggested due to sensitivity and accuracy. We have researched for applying a SPAD chip in our fusion system of time-of-fight (ToF) sensor and stereo camera. Our goal is to upsample of SPAD resolution using RGB stereo camera. Currently, we have 64 x 32 resolution SPAD ToF Sensor, even though there are higher resolution depth sensors such as Kinect V2 and Cube-Eye. This may be a weak point of our system, however we exploit this gap using a transition of idea. A convolution neural network (CNN) is designed to upsample our low resolution depth map using the data of the higher resolution depth as label data. Then, the upsampled depth data using CNN and stereo camera depth data are fused using semi-global matching (SGM) algorithm. We proposed simplified fusion method created for the embedded system.

키워드

참고문헌

  1. S.-H. Kim, D.-K. Lee, J.-H. Cheon, S.-J. Kim, and K.-H. Yu, "Design and Flight Tests of a Drone for Delivery Service," Journal of Institute of Control, Robotics and System, vol. 22, no. 3, pp. 204-209, Mar., 2016. https://doi.org/10.5302/J.ICROS.2016.16.8001
  2. S. Cho, D. Lee, Y. Jung, U. Lee, and D. H. Shim, "Development of a Cooperative Heterogeneous Unmanned System for Delivery Services," Journal of Institute of Control, Robotics and System, vol. 20, no. 12, pp. 1181-1188, Dec., 2014. https://doi.org/10.5302/J.ICROS.2014.14.9046
  3. V. Magnanimo, S. Walther, L Tecchia. C. Natale, and T. Guhl, "Safeguarding a mobile manipulator using dynamic safety fields," 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, South Korea, pp. 2972-2977, 2016.
  4. J.-H. Moon, H.-D. Choi, N.-H. Park, C.-H. Kim, Y.-W. Park, and E.-T. Kim, "Database based global positioning system correction," Journal of Korea Robotics Society, vol. 7, No. 3, pp. 205-215, Sep., 2012. https://doi.org/10.7746/jkros.2012.7.3.205
  5. C. Song and S.-H. Kim, "Robust vehicle occupant detection based on rgb-depth-thermal camera," Journal of Korea Robotics Society, vol. 13, No.1, pp. 31-37, Mar., 2018. https://doi.org/10.7746/jkros.2018.13.1.031
  6. D. Shin, B. Park, Y. Chae, and I. Yun, "Structure variation effects on device reliability of single photon avalanche diodes," Microelectronics Reliability, vol. 76-77, pp. 610-613, Sep., 2014.
  7. Y. Choi, N. Kim, S. Hwang, and I. S. Kwon, "Thermal image enhancement using convolutional neural network," 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, South Korea, pp. 223-230, 2016.
  8. S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, and D. Liang, "Accelerating magnetic resonance imaging via deep learning," 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), Prague, Czech Republic, pp. 514-517, 2016.
  9. B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen, and M. S. Rosen, "Image reconstruction by domain-transform manifold learning," Nature, vol. 555, pp. 487-492, Mar., 2018. https://doi.org/10.1038/nature25988
  10. J. Park, H. Kim, Y.-W. Tai, M. S. Brown, and I. Kwon, "High quality depth map upsampling for 3d-tof cameras," 2011 International Conference on Computer Vision, Barcelona, Spain, pp. 1623-1630, 2011.
  11. R. Nair, F. Lenzen, S. Meister, H. Schafer, C. Garbe, and D. Kondermann, "High accuracy tof and stereo sensor fusion at interactive rates," European Conference on Computer Vision, pp. 1-11, 2012.
  12. G. Agresti, L. Minto, G. Marlin, and P. Zanuttigh, "Deep learning for confidence information in stereo and TOF data fusion," 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy, pp. 697-705, 2017.
  13. Z. Zhang, "A flexible new technique for camera calibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp. 1330-1334, Dec., 2000. https://doi.org/10.1109/34.888718
  14. J. M. Lee, D. Y. Kim, I. Kim, S. Jun, and C.-W. Park, "A study on heterogeneous sensor calibration between ToF sensor and RGB camera," Korean Society for Precision Engineering Spring Conference, Jeju, Korea, pp. 108-109, 2016.
  15. D. Y. Kim, Y.-O. Kim, C.-W. Park, J.-M. Lee, and S. Jun, "System and method for concurrent calibration of camera and depth sensor," Korea Patent, Application No. 10-2015-0095091, Jul., 3, 2015. DOI: 10.8080/1020150095091.
  16. D. Y. Kim, C.-W. Park, J.-M. Lee, Y.-O. Kim, I. Kim, S. Jun, and J.-H. Hwang, "System and method for 3-dimensional calibration of camera and depth sensor, and calibration device and calibration jig thereof," Korea Patent, Registration No. 10-1803705, Nov., 2017. DOI: 10.8080/1020160057145.
  17. http://www.cube-eye.co.kr/en/#/spec/product_MDC600S.html, [Online] Accessed: Jul. 24, 2018.
  18. D. Hernandez-Juarez, A. Chacon, A. Espinosa, D. Vazquez, J. C. Moure, and A. M. Lopez, "Embedded real-time stereo estimation via semi-global matching on the GPU," Procedia Computer Science, Vol. 80, pp. 143-153, Jun., 2016. https://doi.org/10.1016/j.procs.2016.05.305
  19. C. Dong, C. C. Loy, and X. Tang, "Accelerating the super-resolution convolutional neural network," European Conference on Computer Vision, Amsterdam, The Netherlands, pp. 391-407, 2016.