The Journal of The Korea Institute of Intelligent Transport Systems (한국ITS학회 논문지)

The Korea Institute of Intelligent Transport Systems (한국ITS학회)
권호 표지
DOI QR코드

DOI QR Code

Detection Algorithm of Road Damage and Obstacle Based on Joint Deep Learning for Driving Safety

주행 안전을 위한 joint deep learning 기반의 도로 노면 파손 및 장애물 탐지 알고리즘

  • Shim, Seungbo (Korea Institute of Civil Engineering and Building Technology) ;
  • Jeong, Jae-Jin (Daegu, Catholic University, Department of Electronic & Electrical Engineering)
  • 심승보 (한국건설기술연구원 인프라안전연구본부) ;
  • 정재진 (대구가톨릭대학교 전자전기공학부)
  • Received : 2021.02.04
  • Accepted : 2021.03.02
  • Published : 2021.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

As the population decreases in an aging society, the average age of drivers increases. Accordingly, the elderly at high risk of being in an accident need autonomous-driving vehicles. In order to secure driving safety on the road, several technologies to respond to various obstacles are required in those vehicles. Among them, technology is required to recognize static obstacles, such as poor road conditions, as well as dynamic obstacles, such as vehicles, bicycles, and people, that may be encountered while driving. In this study, we propose a deep neural network algorithm capable of simultaneously detecting these two types of obstacle. For this algorithm, we used 1,418 road images and produced annotation data that marks seven categories of dynamic obstacles and labels images to indicate road damage. As a result of training, dynamic obstacles were detected with an average accuracy of 46.22%, and road surface damage was detected with a mean intersection over union of 74.71%. In addition, the average elapsed time required to process a single image is 89ms, and this algorithm is suitable for personal mobility vehicles that are slower than ordinary vehicles. In the future, it is expected that driving safety with personal mobility vehicles will be improved by utilizing technology that detects road obstacles.

인구의 감소 및 고령화 사회가 진행되면서 운전자의 평균 연령은 높아지게 된다. 그에 따라 잠재적인 사고의 위험성이 높은 고령 운전자들은 자율 주행형 개인 이동체가 필요하게 된다. 이러한 이동체가 도로 주행 중에 안전성을 확보하기 위하여 여러 장애물에 대응할 기술이 요구된다. 그 중에서도 주행 중에 마주할 수 있는 차량, 자전거, 사람과 같은 동적 장애물뿐만 아니라 도로 노면의 불량 상태와 같은 정적 장애물을 인식하는 기술이 가장 우선적으로 필요하다. 이를 위해서 본 논문에서는 두 종류의 장애물을 동시에 탐지할 수 있는 심층 신경망 알고리즘을 제안했다. 이 알고리즘을 개발하기 위해서 1,418장의 영상을 이용하여 7종의 동적 장애물에 표기한 annotation data와 도로 노면 파손을 표시한 label 영상을 확보했다. 이를 이용하여 학습한 결과, 46.22%의 평균 정확도로 동적 장애물을 탐지하고 74.71%의 mean intersection over union으로 도로 노면 파손을 탐지했다. 또한 한 장의 영상을 처리하는데 평균 소요시간은 89ms로 일반 차량보다 느린 개인 이동 차량에 사용하기 적합한 알고리즘을 개발했다. 향후 주행 중 마주할 있는 도로 장애물을 탐지하는 기술을 활용하여 개인 이동 차량의 주행 안전성이 향상되길 기대한다.

Keywords

References

  1. Argyros A., Georgiadis P., Trahanias P. and Tsakiris D.(2002), "Semi-autonomous navigation of a robotic wheelchair," Journal of Intelligent and Robotic Systems, vol. 34, no. 3, pp.315-329. https://doi.org/10.1023/A:1016371922451
  2. Bang S., Park S., Kim H. and Kim H.(2019), "Encoder-decoder network for pixel-level road crack detection in black-box images," Computer Aided Civil and Infrastructure Engineering, vol. 34, no. 8, pp.713-727. https://doi.org/10.1111/mice.12440
  3. Borowsky A., Shinar D. and Oron-Gilad T.(2010), "Age, skill, and hazard perception in driving," Accident Analysis & Prevention, vol. 42, no. 4, pp.1240-1249. https://doi.org/10.1016/j.aap.2010.02.001
  4. Buza E., Omanovic S. and Huseinovic A.(2013), "A pothole detection with image processing and spectral clustering," In Proc. the 2nd International Conference on Information Technology and Computer Networks, Antalya, Turkeys, pp.48-53.
  5. Chen L., Yang Z., Ma J. and Luo Z.(2018), "Driving scene perception network: Real-time joint detection, depth estimation and semantic segmentation," In Proc. 2018 IEEE Winter Conference on Applications of Computer Vision(WACV), Lake Tahoe, NV, USA, pp.1283-1291.
  6. Chen X., Kundu K., Zhu Y., Ma H., Fidler S. and Urtasun R.(2017), "3d object proposals using stereo imagery for accurate object class detection," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 5, pp.1259-1272. https://doi.org/10.1109/TPAMI.2017.2706685
  7. Dumoulin V. and Visin F.(2016), A guide to convolution arithmetic for deep learning, arXiv:1603.07285. Available at https://arxiv.org/abs/1603.07285
  8. Fayyad J., Jaradat M. A., Gruyer D. and Najjaran H.(2020), "Deep learning sensor fusion for autonomous vehicle perception and localization: A review," Sensors, vol. 20, no. 15, 4220. https://doi.org/10.3390/s20154220
  9. Feng D., Haase-Schuetz C., Rosenbaum L., Hertlein H., Glaeser C., Timm F., Wiesbeck W. and Dietmayer K.(2020), "Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges," IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 3, pp.1-20.
  10. Girshick R.(2015), "Fast r-cnn," In Proc. the IEEE International Conference on Computer Vision(ICCV), Sangtiago, Chile, pp.1440-1448.
  11. Glorot X. and Bengio Y.(2010), "Understanding the difficulty of training deep feedforward neural networks," In Proc. 13th International Conference on Artificial Intelligence and Statistics(AISTATS), Sardinia, Italy, pp.249-256.
  12. He K., Gkioxari G., Dollar P. and Girshick R.(2017), "Mask r-cnn," In Proc. the IEEE International Conference on Computer Vision(ICCV), Venice, Italy, pp.2961-2969.
  13. He K., Zhang X., Ren S. and Sun J.(2016), "Deep residual learning for image recognition," In Proc. the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Las Vegas, NV, 2016, pp.770-778.
  14. Ilas C.(2013), "Electronic sensing technologies for autonomous ground vehicles: A review," In Proc. 8th International Symposium on Advanced Topics in Electrical Engineering(ATEE), Bucharest, Romania, pp.1-6.
  15. Jenkins M. D., Carr T. A., Iglesias M. I., Buggy T. and Morison G.(2018), "A deep convolutional neural network for semantic pixel-wise segmentation of road and pavement surface cracks," In Proc. 26th European Signal Processing Conference(EUSIPCO), Rome, Italy, pp.2120-2124.
  16. Jo Y., Ryu S. K. and Kim Y. R.(2016), "Pothole detection based on the features of intensity and motion," Journal of the Transportation Research Board, no. 2595, pp.18-28.
  17. Kingma D. P. and Ba J.(2014), Adam: A method for stochastic optimization, arXiv:1412.6980. Available at https://arxiv.org/abs/1412.6980
  18. Kobayashi Y., Kinpara Y., Shibusawa T. and Kuno Y.(2009), "Robotic wheelchair based on observations of people using integrated sensors," In Proc. 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, St. Louis, USA, pp.2013-2018.
  19. Koch C. and Brilakis I.(2011), "Pothole detection in asphalt pavement images," Advanced Engineering Information, vol. 25, no. 1, pp.507-515. https://doi.org/10.1016/j.aei.2011.01.002
  20. Li P. and Qin T.(2018), "Stereo vision-based semantic 3d object and ego-motion tracking for autonomous driving," In Proc. the European Conference on Computer Vision(ECCV), Munich, Germany, pp.646-661.
  21. Li P., Chen X. and Shen S.(2019), "Stereo r-cnn based 3d object detection for autonomous driving," In Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Long Beach, CA, USA, pp.7644-7652.
  22. Liu W., Anguelov D., Erhan D., Szegedy C., Reed S., Fu C. Y. and Berg A. C.(2016), "Ssd: Single shot multibox detector," In Proc. European Conference on Computer Vision (ECCV), Amsterdam, Netherlands, pp.21-37.
  23. Madli R., Hebbar S., Pattar P. and Golla V.(2015), "Automatic detection and notification of potholes and humps on roads to aid drivers," IEEE Sensors Journal, vol. 15, no. 8, pp.4313-4318. https://doi.org/10.1109/JSEN.2015.2417579
  24. Maeda H., Sekimoto Y., Seto T., Kashiyama T. and Omata H.(2018), "Road damage detection and classification using deep neural networks with smart phone images," Computer-Aided Civil and Infrastructure Engineering, vol. 33, no. 12, pp.1127-1141. https://doi.org/10.1111/mice.12387
  25. Muramatsu N. and Akiyama H.(2011), "Japan: Super-aging society preparing for the future," The Gerontologist, vol. 51, no. 4, pp.425-432. https://doi.org/10.1093/geront/gnr067
  26. Nakane J. and Farevaag M.(2004), "Elder care in Japan," Perspectives(Gerontological Nursing Association(Canada)), vol. 28, no. 1, pp.17-24.
  27. Redmon J. and Farhadi A.(2018), Yolov3: An incremental improvement, arXiv:1804.02767. Available at https://arxiv.org/abs/1804.02767
  28. Ren S., He K., Girshick R. and Sun J.(2015), Faster r-cnn: Towards real-time object detection with region proposal networks, arXiv:1506.01497. Available at https://arxiv.org/abs/1506.01497
  29. Ronneberger O., Fischer P. and Brox T.(2015), "U-net: Convolutional networks for biomedical image segmentation," In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention(MICCAI), Munich, Germany, pp.234-241.
  30. Shi Y., Cui L., Qi Z., Meng F. and Chen Z.(2016), "Automatic road crack detection using random structured forests," IEEE Transactions on Intelligent Transportation Systems, vol. 17, no. 12, pp.3434-3445. https://doi.org/10.1109/TITS.2016.2552248
  31. Shim S. and Cho G. C.(2020), "Lightweight semantic segmentation for road-surface damage recognition based on multiscale learning," IEEE Access, vol. 8, pp.102680-102690. https://doi.org/10.1109/access.2020.2998427
  32. Singh S.(2015), "Critical reasons for crashes investigated in the national motor vehicle crash causation survey," Traffic Safety Facts Crash Stats. Report No. DOT HS 812 115; National Center for Statistics and Analysis, Washington, DC, USA.
  33. Sistu G., Leang I. and Yogamani S.(2019), Real-time joint object detection and semantic segmentation network for automated driving, arXiv:1901.03912. Available at https://arxiv.org/abs/1901.03912
  34. Tinnila M. and Kalli J.(2015), "Impact of future trends on personal mobility services," International Journal of Automotive Technology and Management, vol. 15, no. 4, pp.401-417. https://doi.org/10.1504/IJATM.2015.072876
  35. Zhang S., Wen L., Bian X., Lei Z. and Li S. Z.(2018), "Single-shot refinement neural network for object detection," In Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Salt Lake City, UT, USA, pp.4203-4212.
  36. Zhao Q., Sheng T., Wang Y., Tang Z., Chen Y., Cai L. and Ling H.(2019), "M2det: A single-shot object detector based on multi-level feature pyramid network," In Proc. the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, vol. 33, no. 1, pp.9259-9266.
  37. Zou Q., Zhang Z., Li Q., Qi X., Wang Q. and Wang S.(2019), "DeepCrack: Learning hierarchical convolutional features for crack detection," IEEE Transactions on Image Processing, vol. 28, no. 3, pp.1498-1512. https://doi.org/10.1109/tip.2018.2878966