ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지
DOI QR코드

DOI QR Code

Real-time 3D multi-pedestrian detection and tracking using 3D LiDAR point cloud for mobile robot

  • Ki-In Na (Field Robotics Research Section, Mobility Robot Research Division, Electronics and Telecommunications of Research Institute) ;
  • Byungjae Park (School of Mechanical Engineering, Korea University of Technology and Education)
  • Received : 2023.03.24
  • Accepted : 2023.08.09
  • Published : 2023.10.20
Download PDF
⟨ Previous Next ⟩

Abstract

Mobile robots are used in modern life; however, object recognition is still insufficient to realize robot navigation in crowded environments. Mobile robots must rapidly and accurately recognize the movements and shapes of pedestrians to navigate safely in pedestrian-rich spaces. This study proposes real-time, accurate, three-dimensional (3D) multi-pedestrian detection and tracking using a 3D light detection and ranging (LiDAR) point cloud in crowded environments. The pedestrian detection quickly segments a sparse 3D point cloud into individual pedestrians using a lightweight convolutional autoencoder and connected-component algorithm. The multi-pedestrian tracking identifies the same pedestrians considering motion and appearance cues in continuing frames. In addition, it estimates pedestrians' dynamic movements with various patterns by adaptively mixing heterogeneous motion models. We evaluate the computational speed and accuracy of each module using the KITTI dataset. We demonstrate that our integrated system, which rapidly and accurately recognizes pedestrian movement and appearance using a sparse 3D LiDAR, is applicable for robot navigation in crowded spaces.

Keywords

Acknowledgement

This work was supported by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2023-00215760, Guide Dog: Development of Navigation AI Technology of a Guidance Robot for the Visually Impaired Person).

References

  1. S. Seo and H. Jung, A robust collision prediction and detection method based on neural network for autonomous delivery robots, ETRI J. 45 (2023), 329-337. https://doi.org/10.4218/etrij.2021-0397
  2. S.-J. Han, J. Kang, K.-W. Min, and J. Choi, DiLO: direct light detection and ranging odometry based on spherical range images for autonomous driving, ETRI J. 43 (2021), no. 4, 603-616. https://doi.org/10.4218/etrij.2021-0088
  3. H. Shin, K.-I. Na, J. Chang, and T. Uhm, Multimodal layer surveillance map based on anomaly detection using multi-agents for smart city security, ETRI J. 44 (2022), no. 2, 183-193. https://doi.org/10.4218/etrij.2021-0395
  4. A. Shenoi, M. Patel, J. Gwak, P. Goebel, A. Sadeghian, H. Rezatofighi, R. Martin-Martin, and S. Savarese, JRMOT: a real-time 3D multi-object tracker and a new large-scale dataset, (IEEE/RSJ Int. Conf. Intell. Robots Syst., Las Vegas, NV, USA), 2020, pp. 10335-10342.
  5. C. R. Qi, L. Yi, H. Su, and L. Guibas, PointNet++: deep hierarchical feature learning on point sets in a metric space, (Conf. Neural Inf. Process. Syst., Long Beach, CA, USA), 2017, pp. 5105-5114.
  6. B. Wu, A. Wan, X. Yue, and K. Keutzer, SqueezeSeg: convolutional neural nets with recurrent CRF for real-time roadobject segmentation from 3D LiDAR point cloud, (IEEE Int. Conf. Robot. Autom., Brisbane, Australia), 2018, pp. 1887-1893.
  7. H. K Chiu, A. Prioletti, J. Li, and J. Bohg, Probabilistic 3D multi-object tracking for autonomous driving, arXive preprint, 2020. https://doi.org/10.48550/arXiv.2001.05673
  8. X. Weng, J. Wang, D. Held, and K. Kitani, 3D multi-object tracking: a baseline and new evaluation metrics, (IEEE/RSJ Int. Conf. Intell. Robots Syst., Las Vegas, NV, USA), 2020, pp. 10359-10366.
  9. K.-I. Na, S. Choi, and J.-H. Kim, Adaptive target tracking with interacting heterogeneous motion models, IEEE Trans. Intell. Transport. Syst. 23 (2022), no. 11, 21301-21313. https://doi.org/10.1109/TITS.2022.3191814
  10. K.-I. Na, B. Park, and J.-H. Kim, SPriorSeg: fast road-object segmentation using deep semantic prior for sparse 3D point clouds, (IEEE Int. Conf. Syst. Man Cybern., Toronto, Canada), 2020, pp. 3928-3933.
  11. M. Himmelsbach, F. V. Hundelshausen, and H. Wuensche, Fast segmentation of 3D point clouds for ground vehicles, (IEEE Intell. Veh. Symp., La Jolla, CA, USA), 2010, pp. 560-565.
  12. M. Shin, G. Oh, S. Kim, and S. Seo, Real-time and accurate segmentation of 3-D point clouds based on gaussian process regression, IEEE Trans. Intell. Transp. Syst. 18 (2017), no. 12, 3363-3377. https://doi.org/10.1109/TITS.2017.2685523
  13. I. Bogoslavskyi and C. Stachniss, Fast range image-based segmentation of sparse 3D laser scans for online operation, (IEEE Int. Conf. Intell. Robots Syst., Daejeon, Republic of Korea), 2016, pp. 163-169.
  14. F. Moosmann, O. Pink, and C. Stiller, Segmentation of 3D lidar data in non-flat urban environments using a local convexity criterion, (IEEE Intell. Veh. Symp., Xi'an, China), 2009, pp. 215-220.
  15. A. Geiger, P. Lenz, and R. Urtasun, Are we ready for autonomous driving? The KITTI vision benchmark suite, (IEEE Conf. Comput. Vis. Pattern Recognit., Providence, RI, USA), 2012, pp. 3354-3361.
  16. J. Kang, S.-J. Han, N. Kim, and K.-W. Min, ETLi: efficiently annotated traffic LiDAR dataset using incremental and suggestive annotation, ETRI J. 43 (2021), no. 4, 630-639. https://doi.org/10.4218/etrij.2021-0055
  17. Y. Zhou and O. Tuzel, VoxelNet: end-to-end learning for point cloud based 3D object detection, (IEEE Conf. Comput. Vis. Pattern Recognit., Salt Lake City, UT, USA), 2018, pp. 4490-4499.
  18. S. Shi, X. Wang, and H. Li, PointRCNN: 3D object proposal generation and detection from point cloud, (IEEE Conf. Comput. Vis. Pattern Recognit., Long Beach, CA, USA), 2019, pp. 770-779.
  19. J. Long, E. Shelhamer, and T. Darrell, Fully convolutional networks for semantic segmentation, (IEEE Conf. Comput. Vis. Pattern Recognit., Boston, MA, USA), 2015, pp. 3431-3440.
  20. O. Ronneberger, P. Fischer, and T. Brox, U-Net: convolutional networks for biomedical image segmentation, Medical image computing and computer-assisted intervention, Springer, 2015, pp. 234-241.
  21. B. Wu, X. Zhou, S. Zhao, X. Yue, and K. Keutzer, SqueezeSegV2: improved model structure and unsupervised domain adaptation for road-object segmentation from a LiDAR point cloud, (IEEE Int. Conf. Robot. Autom., Montreal, Canada), 2019, pp. 4376-4382.
  22. J. Fei, W. Chen, P. Heidenreich, S. Wirges, and C. Stiller, SemanticVoxels: sequential fusion for 3D pedestrian detection using LiDAR point cloud and semantic segmentation, (IEEE Int. Conf. Multisens. Fusion Integr. Intell. Syst., Karlsruhe, Germany), 2020, pp. 185-190.
  23. T. Chen, R. Wang, B. Dai, D. Liu, and J. Song, Likelihood-field-model-based dynamic vehicle detection and tracking for self-driving, IEEE Trans. Intell. Transp. Syst. 17 (2016), no. 11, 3142-3158. https://doi.org/10.1109/TITS.2016.2542258
  24. Y. Ye, L. Fu, and B. Li, Object detection and tracking using multi-layer laser for autonomous urban driving, (IEEE Int. Conf. Intell. Transp. Syst., Rio de Janeiro, Brazil), 2016, pp. 259-264.
  25. M. Sualeh and G.-W. Kim, Dynamic multi-LiDAR based multiple object detection and tracking, Sensors 19 (2019), 1474.
  26. M. Fischler and R. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Commun. ACM 24 (1981), no. 6, 381-395. https://doi.org/10.1145/358669.358692
  27. L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, Rethinking atrous convolution for semantic image segmentation, arXive preprint, 2017. https://doi.org/10.48550/arXiv.1706.05587 arXiv abs/1706.05587.
  28. M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Ng, ROS: an open-source robot operating system, (ICRA Workshop on Open Source Software, Kobe, Japan), 2009.
  29. K. Bernardin and R. Stiefelhagen, Evaluating multiple object tracking performance: the CLEAR MOT metrics, EURASIP J. Image Video Process. 2008 (2008), 246309.
  30. J. Luiten, A. Osep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixe, and B. Leibe, HOTA: a higher order metric for evaluating multi-object tracking, Int. J. Comp. Vision 129 (2021), 548-578. https://doi.org/10.1007/s11263-020-01375-2