Browse > Article
http://dx.doi.org/10.12815/kits.2021.20.2.95

Detection Algorithm of Road Damage and Obstacle Based on Joint Deep Learning for Driving Safety  

Shim, Seungbo (Korea Institute of Civil Engineering and Building Technology)
Jeong, Jae-Jin (Daegu, Catholic University, Department of Electronic & Electrical Engineering)
Publication Information
The Journal of The Korea Institute of Intelligent Transport Systems / v.20, no.2, 2021 , pp. 95-111 More about this Journal
Abstract
As the population decreases in an aging society, the average age of drivers increases. Accordingly, the elderly at high risk of being in an accident need autonomous-driving vehicles. In order to secure driving safety on the road, several technologies to respond to various obstacles are required in those vehicles. Among them, technology is required to recognize static obstacles, such as poor road conditions, as well as dynamic obstacles, such as vehicles, bicycles, and people, that may be encountered while driving. In this study, we propose a deep neural network algorithm capable of simultaneously detecting these two types of obstacle. For this algorithm, we used 1,418 road images and produced annotation data that marks seven categories of dynamic obstacles and labels images to indicate road damage. As a result of training, dynamic obstacles were detected with an average accuracy of 46.22%, and road surface damage was detected with a mean intersection over union of 74.71%. In addition, the average elapsed time required to process a single image is 89ms, and this algorithm is suitable for personal mobility vehicles that are slower than ordinary vehicles. In the future, it is expected that driving safety with personal mobility vehicles will be improved by utilizing technology that detects road obstacles.
Keywords
Personal mobility vehicle; Deep learning; Semantic segmentation; Object detection; Drinving safety;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Sistu G., Leang I. and Yogamani S.(2019), Real-time joint object detection and semantic segmentation network for automated driving, arXiv:1901.03912. Available at https://arxiv.org/abs/1901.03912
2 Tinnila M. and Kalli J.(2015), "Impact of future trends on personal mobility services," International Journal of Automotive Technology and Management, vol. 15, no. 4, pp.401-417.   DOI
3 Zhang S., Wen L., Bian X., Lei Z. and Li S. Z.(2018), "Single-shot refinement neural network for object detection," In Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Salt Lake City, UT, USA, pp.4203-4212.
4 Zhao Q., Sheng T., Wang Y., Tang Z., Chen Y., Cai L. and Ling H.(2019), "M2det: A single-shot object detector based on multi-level feature pyramid network," In Proc. the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, vol. 33, no. 1, pp.9259-9266.
5 Zou Q., Zhang Z., Li Q., Qi X., Wang Q. and Wang S.(2019), "DeepCrack: Learning hierarchical convolutional features for crack detection," IEEE Transactions on Image Processing, vol. 28, no. 3, pp.1498-1512.   DOI
6 Shi Y., Cui L., Qi Z., Meng F. and Chen Z.(2016), "Automatic road crack detection using random structured forests," IEEE Transactions on Intelligent Transportation Systems, vol. 17, no. 12, pp.3434-3445.   DOI
7 Shim S. and Cho G. C.(2020), "Lightweight semantic segmentation for road-surface damage recognition based on multiscale learning," IEEE Access, vol. 8, pp.102680-102690.   DOI
8 Jenkins M. D., Carr T. A., Iglesias M. I., Buggy T. and Morison G.(2018), "A deep convolutional neural network for semantic pixel-wise segmentation of road and pavement surface cracks," In Proc. 26th European Signal Processing Conference(EUSIPCO), Rome, Italy, pp.2120-2124.
9 Singh S.(2015), "Critical reasons for crashes investigated in the national motor vehicle crash causation survey," Traffic Safety Facts Crash Stats. Report No. DOT HS 812 115; National Center for Statistics and Analysis, Washington, DC, USA.
10 Jo Y., Ryu S. K. and Kim Y. R.(2016), "Pothole detection based on the features of intensity and motion," Journal of the Transportation Research Board, no. 2595, pp.18-28.
11 Li P. and Qin T.(2018), "Stereo vision-based semantic 3d object and ego-motion tracking for autonomous driving," In Proc. the European Conference on Computer Vision(ECCV), Munich, Germany, pp.646-661.
12 Kingma D. P. and Ba J.(2014), Adam: A method for stochastic optimization, arXiv:1412.6980. Available at https://arxiv.org/abs/1412.6980
13 Kobayashi Y., Kinpara Y., Shibusawa T. and Kuno Y.(2009), "Robotic wheelchair based on observations of people using integrated sensors," In Proc. 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, St. Louis, USA, pp.2013-2018.
14 Koch C. and Brilakis I.(2011), "Pothole detection in asphalt pavement images," Advanced Engineering Information, vol. 25, no. 1, pp.507-515.   DOI
15 Li P., Chen X. and Shen S.(2019), "Stereo r-cnn based 3d object detection for autonomous driving," In Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Long Beach, CA, USA, pp.7644-7652.
16 Liu W., Anguelov D., Erhan D., Szegedy C., Reed S., Fu C. Y. and Berg A. C.(2016), "Ssd: Single shot multibox detector," In Proc. European Conference on Computer Vision (ECCV), Amsterdam, Netherlands, pp.21-37.
17 Madli R., Hebbar S., Pattar P. and Golla V.(2015), "Automatic detection and notification of potholes and humps on roads to aid drivers," IEEE Sensors Journal, vol. 15, no. 8, pp.4313-4318.   DOI
18 Maeda H., Sekimoto Y., Seto T., Kashiyama T. and Omata H.(2018), "Road damage detection and classification using deep neural networks with smart phone images," Computer-Aided Civil and Infrastructure Engineering, vol. 33, no. 12, pp.1127-1141.   DOI
19 Muramatsu N. and Akiyama H.(2011), "Japan: Super-aging society preparing for the future," The Gerontologist, vol. 51, no. 4, pp.425-432.   DOI
20 Nakane J. and Farevaag M.(2004), "Elder care in Japan," Perspectives(Gerontological Nursing Association(Canada)), vol. 28, no. 1, pp.17-24.
21 Redmon J. and Farhadi A.(2018), Yolov3: An incremental improvement, arXiv:1804.02767. Available at https://arxiv.org/abs/1804.02767
22 Ren S., He K., Girshick R. and Sun J.(2015), Faster r-cnn: Towards real-time object detection with region proposal networks, arXiv:1506.01497. Available at https://arxiv.org/abs/1506.01497
23 Ronneberger O., Fischer P. and Brox T.(2015), "U-net: Convolutional networks for biomedical image segmentation," In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention(MICCAI), Munich, Germany, pp.234-241.
24 Borowsky A., Shinar D. and Oron-Gilad T.(2010), "Age, skill, and hazard perception in driving," Accident Analysis & Prevention, vol. 42, no. 4, pp.1240-1249.   DOI
25 Buza E., Omanovic S. and Huseinovic A.(2013), "A pothole detection with image processing and spectral clustering," In Proc. the 2nd International Conference on Information Technology and Computer Networks, Antalya, Turkeys, pp.48-53.
26 Chen L., Yang Z., Ma J. and Luo Z.(2018), "Driving scene perception network: Real-time joint detection, depth estimation and semantic segmentation," In Proc. 2018 IEEE Winter Conference on Applications of Computer Vision(WACV), Lake Tahoe, NV, USA, pp.1283-1291.
27 Chen X., Kundu K., Zhu Y., Ma H., Fidler S. and Urtasun R.(2017), "3d object proposals using stereo imagery for accurate object class detection," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 5, pp.1259-1272.   DOI
28 Feng D., Haase-Schuetz C., Rosenbaum L., Hertlein H., Glaeser C., Timm F., Wiesbeck W. and Dietmayer K.(2020), "Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges," IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 3, pp.1-20.
29 Dumoulin V. and Visin F.(2016), A guide to convolution arithmetic for deep learning, arXiv:1603.07285. Available at https://arxiv.org/abs/1603.07285
30 Fayyad J., Jaradat M. A., Gruyer D. and Najjaran H.(2020), "Deep learning sensor fusion for autonomous vehicle perception and localization: A review," Sensors, vol. 20, no. 15, 4220.   DOI
31 Girshick R.(2015), "Fast r-cnn," In Proc. the IEEE International Conference on Computer Vision(ICCV), Sangtiago, Chile, pp.1440-1448.
32 Glorot X. and Bengio Y.(2010), "Understanding the difficulty of training deep feedforward neural networks," In Proc. 13th International Conference on Artificial Intelligence and Statistics(AISTATS), Sardinia, Italy, pp.249-256.
33 Argyros A., Georgiadis P., Trahanias P. and Tsakiris D.(2002), "Semi-autonomous navigation of a robotic wheelchair," Journal of Intelligent and Robotic Systems, vol. 34, no. 3, pp.315-329.   DOI
34 Bang S., Park S., Kim H. and Kim H.(2019), "Encoder-decoder network for pixel-level road crack detection in black-box images," Computer Aided Civil and Infrastructure Engineering, vol. 34, no. 8, pp.713-727.   DOI
35 He K., Gkioxari G., Dollar P. and Girshick R.(2017), "Mask r-cnn," In Proc. the IEEE International Conference on Computer Vision(ICCV), Venice, Italy, pp.2961-2969.
36 He K., Zhang X., Ren S. and Sun J.(2016), "Deep residual learning for image recognition," In Proc. the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Las Vegas, NV, 2016, pp.770-778.
37 Ilas C.(2013), "Electronic sensing technologies for autonomous ground vehicles: A review," In Proc. 8th International Symposium on Advanced Topics in Electrical Engineering(ATEE), Bucharest, Romania, pp.1-6.