Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Depth Map Completion using Nearest Neighbor Kernel

최근접 이웃 커널을 이용한 깊이 영상 완성 기술

  • 정태현 (한국항공대학교 항공전자정보공학부) ;
  • 우딘 쿠툽 (한국항공대학교 항공전자정보공학부) ;
  • 오병태 (한국항공대학교 항공전자정보공학부)
  • Received : 2022.09.22
  • Accepted : 2022.10.19
  • Published : 2022.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose a new deep network architecture using nearest neighbor kernel for the estimation of dense depth map from its sparse map and corresponding color information. First, we propose to decompose the depth map signal into the structure and details for easier prediction. We then propose two separate subnetworks for prediction of both structure and details using classification and regression approaches, respectively. Moreover, the nearest neighboring kernel method has been newly proposed for accurate prediction of structure signal. As a result, the proposed method showed better results than other methods quantitatively and qualitatively.

본 논문에서는 희소 깊이 영상과 컬러 영상을 이용해 조밀한 깊이 영상을 추정하는 깊이 완성을 수행하기 위해 최근접 이웃 커널 기술을 사용하는 방식의 네트워크를 제안한다. 먼저 예측하고자 하는 깊이 영상을 대략적인 깊이 정보의 구조 정보를 포함하는 부분과 세밀한 깊이 정보를 가지는 상세 부분으로 분할하여 예측하는 방식을 제안한다. 이 과정에서 깊이 영상의 구조 및 상세 정보는 분류 기법과 회귀 기법을 활용하여 각각 추정하였으며, 특히 분류 과정에서 최근접 이웃 커널 정보를 활용하여 주변 정보를 통해 분류를 진행하는 방식을 제안하였다. 제안 방식은 기존의 희소 깊이 완성 방식과 비교하여 우수한 성능을 나타냈고, 시각적으로도 만족할만한 결과를 보이게 됨을 확인하였다.

Keywords

Acknowledgement

This research was supported in part by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of ICT (NRF-2022R1A2C1005769), and by the GRRC program of Gyeonggi Province [2017-B02, Study on 3D Point Cloud Processing and Application Technology].

References

  1. Uhrig, Jonas, et al. "Sparsity invariant cnns." 2017 international conference on 3D Vision (3DV). IEEE, 2017. doi: https://doi.org/10.1109/3DV.2017.00012  
  2. Chodosh, Nathaniel, Chaoyang Wang, and Simon Lucey. "Deep convolutional compressed sensing for lidar depth completion."Asian Conference on Computer Vision. Springer, Cham, 2018. doi: https://doi.org/10.1007/978-3-030-20887-5_31  
  3. Liu, Sifei, et al. "Learning affinity via spatial propagation networks."Advances in Neural Information Processing Systems30 (2017). doi: https://doi.org/10.48550/arXiv.1710.01020  
  4. Ma, Fangchang, Guilherme Venturelli Cavalheiro, and Sertac Karaman. "Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera."2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019. doi: https://doi.org/10.1109/ICRA.2019.8793637  
  5. He, Kaiming, et al. "Deep residual learning for image recognition. " Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. doi: https://doi.org/10.1109/CVPR.2016.90  
  6. Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computerassisted intervention. Springer, Cham, 2015. doi: https://doi.org/10.48550/arXiv.1505.04597  
  7. Qiu, Jiaxiong, et al. "Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image."Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. doi: https://doi.org/10.48550/arXiv.1812.00488  
  8. Eldesokey, Abdelrahman, Michael Felsberg, and Fahad Shahbaz Khan. "Confidence propagation through cnns for guided sparse depth regression."IEEE transactions on pattern analysis and machine intelligence 42.10 (2019): 2423-2436. doi: https://doi.org/10.1109/TPAMI.2019.2929170  
  9. Huang, Zixuan, et al. "Hms-net: Hierarchical multi-scale sparsityinvariant network for sparse depth completion."IEEE Transactions on Image Processing 29 (2019): 3429-3441. doi: https://doi.org/10.48550/arXiv.1808.08685  
  10. Zhang, Chongzhen, et al. "Multitask gans for semantic segmentation and depth completion with cycle consistency."IEEE Transactions on Neural Networks and Learning Systems32.12 (2021): 5404-5415. doi: https://doi.org/10.1109/TNNLS.2021.3072883  
  11. Nazir, Danish, et al. "SemAttNet: Towards Attention-based Semantic Aware Guided Depth Completion."arXiv preprint arXiv:2204.13635 (2022). doi: https://doi.org/10.1109/ACCESS.2022.3214316  
  12. Khan, Muhammad Ahmed Ullah, et al. "A Comprehensive Survey of Depth Completion Approaches." (2022). doi: https://doi.org/10.20944/preprints202205.0343.v1  
  13. Tang, Jie, et al. "Learning guided convolutional network for depth completion."IEEE Transactions on Image Processing 30 (2020): 1116-1129. doi: https://doi.org/10.1109/TIP.2020.3040528  
  14. Hu, Mu, et al. "Penet: Towards precise and efficient image guided depth completion."2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021. doi: https://doi.org/10.48550/arXiv.2103.00783  
  15. Yan, Zhiqiang, et al. "RigNet: Repetitive image guided network for depth completion."arXiv preprint arXiv:2107.13802(2021).  
  16. Cheng, Xinjing, Peng Wang, and Ruigang Yang. "Learning depth with convolutional spatial propagation network."IEEE transactions on pattern analysis and machine intelligence 42.10 (2019): 2361-2379. doi: https://doi.org/10.48550/arXiv.1810.02695  
  17. Cheng, Xinjing, et al. "Cspn++: Learning context and resource aware convolutional spatial propagation networks for depth completion." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020. doi: https://doi.org/10.48550/arXiv.1911.05377  
  18. Xu, Zheyuan, Hongche Yin, and Jian Yao. "Deformable spatial propagation networks for depth completion."2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020. doi: https://doi.org/10.48550/arXiv.2007.04251  
  19. Park, Jinsun, et al. "Non-local spatial propagation network for depth completion."European Conference on Computer Vision. Springer, Cham, 2020. doi: https://doi.org/10.1007/978-3-030-58601-0_8  
  20. Lin, Yuankai, et al. "Dynamic spatial propagation network for depth completion."arXiv preprint arXiv:2202.09769(2022). doi: https://doi.org/10.48550/arXiv.2202.09769  
  21. Zhang, Kai, et al. "Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising."IEEE transactions on image processing 26.7 (2017): 3142-3155. doi: https://doi.org/10.1109/TIP.2017.2662206  
  22. Kim, Jiwon, Jung Kwon Lee, and Kyoung Mu Lee. "Accurate image super-resolution using very deep convolutional networks."Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. doi: https://doi.org/10.48550/arXiv.1511.04587  
  23. Liu, Lina, et al. "Fcfr-net: Feature fusion based coarse-to-fine residual learning for depth completion."Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 3. 2021. doi: https://doi.org/10.1609/aaai.v35i3.16311  
  24. Gu, Jiaqi, et al. "Denselidar: A real-time pseudo dense depth guided depth completion network."IEEE Robotics and Automation Letters6.2 (2021): 1808-1815. doi: https://doi.org/10.1109/LRA.2021.3060396  
  25. Zhu, Yufan, et al. "Robust depth completion with uncertainty-driven loss functions."Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 3. 2022. doi: https://doi.org/10.1609/aaai.v36i3.20275  
  26. Lee, Byeong-Uk, Kyunghyun Lee, and In So Kweon. "Depth completion using plane-residual representation."Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. doi: https://doi.org/10.48550/arXiv.2104.07350  
  27. Silberman, Nathan, et al. "Indoor segmentation and support inference from rgbd images."European conference on computer vision. Springer, Berlin, Heidelberg, 2012. doi: https://doi.org/10.1007/978-3-642-33715-4_54