Browse > Article
http://dx.doi.org/10.12989/sss.2022.29.1.221

Crack segmentation in high-resolution images using cascaded deep convolutional neural networks and Bayesian data fusion  

Tang, Wen (Lyles School of Civil Engineering, Purdue University)
Wu, Rih-Teng (Department of Civil Engineering, National Taiwan University)
Jahanshahi, Mohammad R. (Lyles School of Civil Engineering, Purdue University)
Publication Information
Smart Structures and Systems / v.29, no.1, 2022 , pp. 221-235 More about this Journal
Abstract
Manual inspection of steel box girders on long span bridges is time-consuming and labor-intensive. The quality of inspection relies on the subjective judgements of the inspectors. This study proposes an automated approach to detect and segment cracks in high-resolution images. An end-to-end cascaded framework is proposed to first detect the existence of cracks using a deep convolutional neural network (CNN) and then segment the crack using a modified U-Net encoder-decoder architecture. A Naïve Bayes data fusion scheme is proposed to reduce the false positives and false negatives effectively. To generate the binary crack mask, first, the original images are divided into 448 × 448 overlapping image patches where these image patches are classified as cracks versus non-cracks using a deep CNN. Next, a modified U-Net is trained from scratch using only the crack patches for segmentation. A customized loss function that consists of binary cross entropy loss and the Dice loss is introduced to enhance the segmentation performance. Additionally, a Naïve Bayes fusion strategy is employed to integrate the crack score maps from different overlapping crack patches and to decide whether a pixel is crack or not. Comprehensive experiments have demonstrated that the proposed approach achieves an 81.71% mean intersection over union (mIoU) score across 5 different training/test splits, which is 7.29% higher than the baseline reference implemented with the original U-Net.
Keywords
Bayesian data fusion; crack detection; deep learning; semantic segmentation; structural health monitoring;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Long, J., Shelhamer, E. and Darrell, T. (2015), "Fully convolutional networks for semantic segmentation", Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA, June, pp. 3431-3440.
2 Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T. and Omata, H. (2018), "Road damage detection and classification using deep neural networks with smartphone images", Comput.-Aided Civil Infrastr. Eng., 33(12), 1127-1141. https://doi.org/10.1111/mice.12387   DOI
3 Nair, V. and Hinton, G.E. (2010), "Rectified linear units improve restricted boltzmann machines", Proceedings of International Conference on Machine Learning, Haifa, Israel, June.
4 Oh, J.K., Jang, G., Oh, S., Lee, J.H., Yi, B.J., Moon, Y.S., Lee, J.S. and Choi, Y. (2009), "Bridge inspection robot system with machine vision", Automat. Constr., 18(7), 929-941. https://doi.org/10.1016/j.autcon.2009.04.003   DOI
5 Ren, S., He, K., Girshick, R. and Sun, J. (2015), "Faster r-cnn: Towards real-time object detection with region proposal networks", Adv. Neural Inform. Process. Syst., 28, 91-99. https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf
6 Liu, Y., Yao, J., Lu, X., Xie, R. and Li, L. (2019a), "DeepCrack: A deep hierarchical feature learning architecture for crack segmentation", Neurocomputing, 338, 139-153. https://doi.org/10.1016/j.neucom.2019.01.036   DOI
7 He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NA, USA, June, pp. 770-778.
8 Zhang, X., Rajan, D. and Story, B. (2019), "Concrete crack detection using context-aware deep semantic segmentation network", Comput.-Aided Civil Infrastr. Eng, 34(11), 951-971. https://doi.org/10.1111/mice.12477   DOI
9 Bao, Y. and Li, H. (2021), "Machine learning paradigm for structural health monitoring", Struct. Health Monitor., 20(4), 1353-1372. https://doi.org/10.1177/1475921720972416   DOI
10 Yang, X., Li, H., Yu, Y., Luo, X., Huang, T. and Yang, X. (2018), "Automatic pixel-level crack detection and measurement using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 33(12), 1090-1109. https://doi.org/10.1111/mice.12412   DOI
11 Beckman, G.H., Polyzois, D. and Cha, Y.J. (2019), "Deep learning-based automatic volumetric damage quantification using depth camera", Automat. Constr., 99, 114-124. https://doi.org/10.1016/j.autcon.2018.12.006   DOI
12 Bao, Y., Chen, Z., Wei, S., Xu, Y., Tang, Z. and Li, H. (2019), "The state of the art of data science and engineering in structural health monitoring", Engineering, 5(2), 234-242. https://doi.org/10.1016/j.eng.2018.11.027   DOI
13 Chen, L.C., Papandreou, G., Schroff, F. and Adam, H. (2017b), "Rethinking atrous convolution for semantic image segmentation", arXiv. https://arxiv.org/abs/1706.05587
14 Bao, Y., Li, J., Nagayama, T., Xu, Y., Spencer Jr, B.F. and Li, H. (2021), "The 1st International Project Competition for Structural Health Monitoring (IPC-SHM, 2020): A summary and benchmark problem", Struct. Health Monitor., 20(4), 2229-2239. https://doi.org/10.1177/14759217211006485   DOI
15 Cha, Y.J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastr. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263   DOI
16 Ronneberger, O., Fischer, P. and Brox, T. (2015), "U-net: Convolutional networks for biomedical image segmentation", International Conference on Medical Image Computing and Computer-Assisted Intervention, October, pp. 234-241.
17 Spencer Jr, B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Engineering, 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030   DOI
18 Tasar, O., Tarabalka, Y. and Alliez, P. (2019), "Incremental learning for semantic segmentation of large-scale remote sensing data", IEEE J. Select. Topics Appl. Earth Observ. Remote Sensing, 12(9), 3524-3537. https://doi.org/10.1109/JSTARS.2019.2925416   DOI
19 Xu, Y., Bao, Y., Chen, J., Zuo, W. and Li, H. (2019), "Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images", Struct. Health Monitor., 18(3), 653-674. https://doi.org/10.1177/1475921718764873   DOI
20 Xu, Y., Li, S., Zhang, D., Jin, Y., Zhang, F., Li, N. and Li, H. (2018), "Identification framework for cracks on a steel structure surface by a restricted Boltzmann machines algorithm based on consumer-grade camera images", Struct. Control Health Monitor., 25(2), e2075. https://doi.org/10.1002/stc.2075   DOI
21 Xue, Y. and Li, Y. (2018), "A fast detection method via region-based fully convolutional neural networks for shield tunnel lining defects", Comput.-Aided Civil Infrastr. Eng., 33(8), 638-654. https://doi.org/10.1111/mice.12367   DOI
22 Yamaguchi, T. and Hashimoto, S. (2010), "Fast method for crack detection surface concrete large-size images using percolation-based image processing", Mach. Vis. Appl., 21, 797-809. https://doi.org/10.1007/s00138-009-0189-8   DOI
23 Cha, Y.J., Choi, W., Suh, G., Mahmoudkhani, S. and Buyukozturk, O. (2018), "Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types", Comput.-Aided Civil Infrastr. Eng., 33(9), 731-747. https://doi.org/10.1111/mice.12334   DOI
24 Chen, F.C. and Jahanshahi, M.R. (2019), "NB-FCN: Real-time accurate crack detection in inspection videos using deep fully convolutional network and parametric data fusion", IEEE Transact. Instrument. Measur., 69(8), 5325-5334. https://doi.org/10.1109/TIM.2019.2959292   DOI
25 Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L. (2017a), "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs", IEEE Transact. Pattern Anal. Mach. Intell., 40(4), 834-848. https://doi.org/10.1109/TPAMI.2017.2699184   DOI
26 Chen, F.C. and Jahanshahi, M.R. (2017), "NB-CNN: Deep learning-based crack detection using convolutional neural network and Naive Bayes data fusion", IEEE Transact. Indust. Electron., 65(5), 4392-4400. https://doi.org/10.1109/TIE.2017.2764844   DOI
27 Li, X., Chen, S., Hu, X. and Yang, J. (2019), "Understanding the disharmony between dropout and batch normalization by variance shift", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, June, pp. 2682-2690.
28 Dung, C.V. (2019), "Autonomous concrete crack detection using deep fully convolutional neural network", Automat. Constr., 99, 52-58. https://doi.org/10.1016/j.autcon.2018.11.028   DOI
29 Abdel-Qader, I., Abudayyeh, O. and Kelly, M.E. (2003), "Analysis of edge-detection techniques for crack identification in bridges", J. Comput. Civil Eng., 17(4), 255-263. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(255)   DOI
30 Lin, T.Y., Goyal, P., Girshick, R., He, K. and Dollar, P. (2017), "Focal loss for dense object detection", Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, October, pp. 2980-2988.
31 Fujita, Y. and Hamamoto, Y. (2011), "A robust automatic crack detection method from noisy concrete surfaces", Mach. Vis. Applicat., 22(2), 245-254. https://doi.org/10.1007/s00138-009-0244-5   DOI
32 Ding, L., Zhang, J. and Bruzzone, L. (2020), "Semantic segmentation of large-size VHR remote sensing images using a two-stage multiscale training architecture", IEEE Transact. Geosci. Remote Sensing, 58(8), 5367-5376. https://doi.org/10.1109/TGRS.2020.2964675   DOI
33 Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F. and Adam, H. (2018), "Encoder-decoder with atrous separable convolution for semantic image segmentation", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 801-818.
34 Cheng, H.D., Chen, J.R., Glazier, C. and Hu, Y.G. (1999), "Novel approach to pavement cracking detection based on fuzzy set theory", J. Comput. Civil Eng., 13(4), 270-280. https://doi.org/10.1061/(ASCE)0887-3801(1999)13:4(270)   DOI
35 Cheng, H.D., Shi, X.J. and Glazier, C. (2003), "Real-time image thresholding based on sample space reduction and interpolation approach", J. Comput. Civil Eng., 17(4), 264-272. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(264)   DOI
36 Badrinarayanan, V., Kendall, A. and Cipolla, R. (2017), "Segnet: A deep convolutional encoder-decoder architecture for image segmentation", IEEE Transact. Pattern Anal. Mach. Intell., 39(12), 2481-2495. https://doi.org/10.1109/TPAMI.2016.2644615   DOI
37 Bang, S., Park, S., Kim, H. and Kim, H. (2019), "Encoder-decoder network for pixel-level road crack detection in black-box images", Comput.-Aided Civil Infrastr. Eng., 34(8), 713-727. https://doi.org/10.1111/mice.12440   DOI
38 Ciregan, D., Meier, U. and Schmidhuber, J. (2012), "Multi-column deep neural networks for image classification", IEEE Conference on Computer Vision and Pattern Recognition, Providence, Rhode Island, USA, June, pp. 3642-3649.
39 Deng, J., Lu, Y. and Lee, V.C.S. (2020), "Concrete crack detection with handwriting script interferences using faster region-based convolutional neural network", Comput.-Aided Civil Infrastr. Eng., 35(4), 373-388. https://doi.org/10.1111/mice.12497   DOI
40 Dumoulin, V. and Visin, F. (2016), "A guide to convolution arithmetic for deep learning", arXiv preprint. https://arxiv.org/abs/1603.07285
41 He, K., Zhang, X., Ren, S. and Sun, J. (2015), "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification", Proceedings of the IEEE International Conference on Computer Vision, Boston, MA, USA, June, pp. 1026-1034.
42 Choi, W. and Cha, Y.J. (2019), "SDDNet: Real-time crack segmentation", IEEE Transact. Industr. Electron., 67(9), 8016-8025. https://doi.org/10.1109/TIE.2019.2945265   DOI
43 Lee, B.J., Shin, D.H., Seo, J.W., Jung, J.D. and Lee, J.Y. (2011), "Intelligent bridge inspection using remote controlled robot and image processing technique", International Symposium on Automation and Robotics in Construction, Seoul, Korea, June.
44 Ioffe, S. and Szegedy, C. (2015), "Batch normalization: Accelerating deep network training by reducing internal covariate shift", Proceedings of International Conference on Machine Learning, Lille, France, June, pp. 448-456.
45 Jahanshahi, M.R., Kelly, J.S., Masri, S.F. and Sukhatme, G.S. (2009), "A survey and evaluation of promising approaches for automatic image-based defect detection of bridge structures", Struct. Infrastr. Eng., 5(6), 455-486. https://doi.org/10.1080/15732470801945930   DOI
46 Jahanshahi, M.R., Masri, S.F., Padgett, C.W. and Sukhatme, G.S. (2013), "An innovative methodology for detection and quantification of cracks through incorporation of depth perception", Mach. Vis. Applicat., 24(2), 227-241. https://doi.org/10.1007/s00138-011-0394-0   DOI
47 Liu, Y., Ren, Q., Geng, J., Ding, M. and Li, J. (2018), "Efficient patch-wise semantic segmentation for large-scale remote sensing images", Sensors, 18(10), 3232. https://doi.org/10.3390/s18103232   DOI
48 Ji, A., Xue, X., Wang, Y., Luo, X. and Xue, W. (2020), "An integrated approach to automatic pixel-level crack detection and quantification of asphalt pavement", Automat. Constr., 114, 103176. https://doi.org/10.1016/j.autcon.2020.103176   DOI
49 Kingma, D.P. and Ba, J. (2014), "Adam: A method for stochastic optimization", arXiv preprint. https://arxiv.org/abs/1412.6980
50 LeCun, Y., Bengio, Y. and Hinton, G. (2015), "Deep learning", Nature, 521(7553), 436-444. https://doi.org/10.1038/nature14539   DOI
51 Lim, R.S., La, H.M. and Sheng, W. (2014), "A robotic crack inspection and mapping system for bridge deck maintenance", IEEE Transact. Automat. Sci. Eng., 11(2), 367-378. https://doi.org/10.1109/TASE.2013.2294687   DOI
52 Liu, J., Yang, X., Lau, S., Wang, X., Luo, S., Lee, V.C.S. and Ding, L. (2020), "Automated pavement crack detection and segmentation based on two-step convolutional neural network", Comput.-Aided Civil Infrastr. Eng., 35(11), 1291-1305. https://doi.org/10.1111/mice.12622   DOI
53 Lin, M., Chen, Q. and Yan, S. (2013), "Network in network", arXiv preprint. https://arxiv.org/abs/1312.4400
54 Liu, Z., Cao, Y., Wang, Y. and Wang, W. (2019b), "Computer vision-based concrete crack detection using U-net fully convolutional networks", Automat. Constr., 104, 129-139. https://doi.org/10.1016/j.autcon.2019.04.005   DOI