Browse > Article
http://dx.doi.org/10.12989/sss.2022.30.1.017

Semantic crack-image identification framework for steel structures using atrous convolution-based Deeplabv3+ Network  

Ta, Quoc-Bao (Department of Ocean Eng., Pukyong National University)
Dang, Ngoc-Loi (Urban Infrastructure Faculty, Mien Tay Construction University)
Kim, Yoon-Chul (Department of Civil and Environmental Eng., Yonsei University)
Kam, Hyeon-Dong (Department of Ocean Eng., Pukyong National University)
Kim, Jeong-Tae (Department of Ocean Eng., Pukyong National University)
Publication Information
Smart Structures and Systems / v.30, no.1, 2022 , pp. 17-34 More about this Journal
Abstract
For steel structures, fatigue cracks are critical damage induced by long-term cycle loading and distortion effects. Vision-based crack detection can be a solution to ensure structural integrity and performance by continuous monitoring and non-destructive assessment. A critical issue is to distinguish cracks from other features in captured images which possibly consist of complex backgrounds such as handwritings and marks, which were made to record crack patterns and lengths during periodic visual inspections. This study presents a parametric study on image-based crack identification for orthotropic steel bridge decks using captured images with complicated backgrounds. Firstly, a framework for vision-based crack segmentation using the atrous convolution-based Deeplapv3+ network (ACDN) is designed. Secondly, features on crack images are labeled to build three databanks by consideration of objects in the backgrounds. Thirdly, evaluation metrics computed from the trained ACDN models are utilized to evaluate the effects of obstacles on crack detection results. Finally, various training parameters, including image sizes, hyper-parameters, and the number of training images, are optimized for the ACDN model of crack detection. The result demonstrated that fatigue cracks could be identified by the trained ACDN models, and the accuracy of the crack-detection result was improved by optimizing the training parameters. It enables the applicability of the vision-based technique for early detecting tiny fatigue cracks in steel structures.
Keywords
atrous convolution; Deeplabv3+ network; fatigue crack; image processing technique; semantic segmentation; steel structures;
Citations & Related Records
Times Cited By KSCI : 7  (Citation Analysis)
연도 인용수 순위
1 Habtour, E., Cole, D.P., Riddick, J.C., Weiss, V., Robeson, M., Sridharan, R. and Dasgupta, A. (2016), "Detection of fatigue damage precursor using a nonlinear vibration approach", Struct. Control Health Monitor., 23(12), 1442-1463. https://doi.org/10.1002/stc.1844   DOI
2 He, K., Zhang, X., Ren, S. and Sun, J. (2015), "Spatial pyramid pooling in deep convolutional networks for visual recognition", IEEE Transact. Pattern Anal. Mach. Intell., 37(9), 1904-1916. https://doi.org/10.1109/TPAMI.2015.2389824   DOI
3 Jeong, S., Kim, H., Lee, J. and Sim, S.H. (2020), "Automated wireless monitoring system for cable tension forces using deep learning", Struct. Health Monitor., 20(4), 1805-1821. https://doi.org/10.1177/1475921720935837   DOI
4 Jin, S.S., Jeong, S., Sim, S.H., Seo, D.W. and Park, Y.S. (2021), "Fully automated peak-picking method for an autonomous stay-cable monitoring system in cable-stayed bridges", Automat. Constr., 126, 103628. https://doi.org/10.1016/j.autcon.2021.103628   DOI
5 Bhalla, S., Vittal, P.A. and Veljkovic, M. (2012), "Piezo-impedance transducers for residual fatigue life assessment of bolted steel joints", Struct. Health Monitor., Int. J., 11(6), 733-750. https://doi.org/10.1177/1475921712458708   DOI
6 Cha, Y.J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastruct. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263   DOI
7 Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L. (2014), "Semantic image segmentation with deep convolutional nets and fully connected crfs", Computer Vision and Pattern Recognition, arXiv preprint arXiv:1412.7062. https://doi.org/10.48550/arXiv.1412.7062
8 Huynh, T.C. (2021), "Vision-based autonomous bolt-looseness detection method for splice connections: Design, lab-scale evaluation, and field application", Automat. Constr, 124, 103591. https://doi.org/10.1016/j.autcon.2021.103591   DOI
9 Barbedo, J.G.A. (2018), "Impact of dataset size and variety on the effectiveness of deep learning and transfer learning for plant disease classification", Comput. Electron. Agricult., 153, 46-53. https://doi.org/10.1016/j.compag.2018.08.013   DOI
10 Bastani, A., Amindavar, H., Shamshirsaz, M. and Sepehry, N. (2011), "Identification of temperature variation and vibration disturbance in impedance-based structural health monitoring using piezoelectric sensor array method", Struct. Health Monitor., Int. J., 11(3), 305-314. https://doi.org/10.1177/1475921711427486   DOI
11 Bao, Y., Li, J., Nagayama, T., Xu, Y., Spencer, B.F. and Li, H. (2021), "The 1st international project competition for structural health monitoring (IPC-SHM, 2020): a summary and benchmark problem", Struct. Health Monitor., 20(4), 2229-2239. https://doi.org/10.1177/14759217211006485   DOI
12 Sun, Y., Yang, Y., Yao, G., Wei, F. and Wong, M. (2021), "Autonomous crack and bughole detection for concrete surface image based on deep learning", IEEE Access, 9, 85709-85720. https://doi.org/10.1109/ACCESS.2021.3088292   DOI
13 He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR.2016.90   DOI
14 Hou, Y., Yue, P., Xin, Q., Pauli, T., Sun, W. and Wang, L. (2013), "Fracture failure of asphalt binder in mixed mode (Modes I and II) by using phase-field model", Road Mater. Pave. Des., 15(1), 167-181. https://doi.org/10.1080/14680629.2013.866155   DOI
15 Yang, X., Li, H., Yu, Y., Luo, X., Huang, T. and Yang, X. (2018), "Automatic pixel-level crack detection and measurement using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 33(12), 1090-1109. https://doi.org/10.1111/mice.12412   DOI
16 Di, J., Ruan, X., Zhou, X., Wang, J. and Peng, X. (2021), "Fatigue assessment of orthotropic steel bridge decks based on strain monitoring data", Eng. Struct., 228, 111437. https://doi.org/10.1016/j.engstruct.2020.111437   DOI
17 Chollet, F. (2017), "Xception: Deep learning with depthwise separable convolutions", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.48550/arXiv.1610.02357   DOI
18 Ta, Q.B. and Kim, J.T. (2020), "Monitoring of corroded and loosened bolts in steel structures via deep learning and hough transforms", Sensors, 20(23). https://doi.org/10.3390/s20236888   DOI
19 Xu, Y., Bao, Y., Chen, J., Zuo, W. and Li, H. (2018a), "Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumerg-rade camera images", Struct. Health Monitor, 18(3), 653-674. https://doi.org/10.1177/1475921718764873   DOI
20 Xu, Y., Li, S., Zhang, D., Jin, Y., Zhang, F., Li, N. and Li, H. (2018b), "Identification framework for cracks on a steel structure surface by a restricted Boltzmann machines algorithm based on consumer-grade camera images", Struct. Control Health Monitor., 25(2), 2075. https://doi.org/10.1002/stc.2075   DOI
21 Yao, L., Dong, Q., Jiang, J. and Ni, F. (2020), "Deep reinforcement learning for long-term pavement maintenance planning", Comput.-Aided Civil Infrastr. Eng., 35(11), 1230-1245. https://doi.org/10.1111/mice.12558   DOI
22 Csurka, G., Larlus, D., Perronnin, F. and Meylan, F. (2013), "What is a good evaluation measure for semantic segmentation?", In: Bmvc, Vol. 27, pp. 10-5244. http://dx.doi.org/10.5244/C.27.32   DOI
23 Zhao, X., Zhang, Y. and Wang, N. (2019), "Bolt loosening angle detection technology using deep learning", Struct. Control Health Monitor., 26(1), 2292. https://doi.org/10.1002/stc.2292   DOI
24 Li, M., Suzuki, Y., Hashimoto, K. and Sugiura, K. (2018), "Experimental study on fatigue resistance of rib-to-deck joint in orthotropic steel bridge deck", J. Bridge Eng., 23(2), 04017128. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001175   DOI
25 Ly, C.D., Vo, T.H., Mondal, S., Park, S., Choi, J., Vu, T.T.H., Kim, C.S. and Oh, J. (2021), "Full-view in vivo skin and blood vessels profile segmentation in photoacoustic imaging based on deep learning", Photoacoustics, 25, 100310. https://doi.org/10.1016/j.pacs.2021.100310   DOI
26 Lee, Y.F., Lu, Y. and Guan, R. (2020), "Nonlinear guided waves for fatigue crack evaluation in steel joints with digital image correlation validation", Smart Mater. Struct., 29(3), 035031. https://doi.org/10.1088/1361-665X/ab6fe7   DOI
27 Papadimitriou, C., Fritzen, C.P., Kraemer, P. and Ntotsios, E. (2011), "Fatigue predictions in entire body of metallic structures from a limited number of vibration sensors using Kalman filtering", Struct. Control Health Monitor., 18(5), 554-573. https://doi.org/10.1002/stc.395   DOI
28 Park, J.-H., Huynh, T.-C., Choi, S.-H. and Kim, J.-T. (2015), "Vision-based technique for bolt-loosening detection in wind turbine tower", Wind Struct., Int. J., 21(6), 709-726. https://doi.org/10.12989/was.2015.21.6.709   DOI
29 Pham, H.C., Ta, Q.B., Kim, J.T., Ho, D.D., Tran, X.L. and Huynh, T.C. (2020), "Bolt-loosening monitoring framework using an image-based deep learning and graphical model", Sensors, 20(12). https://doi.org/10.3390/s20123382   DOI
30 Shorten, C. and Khoshgoftaar, T.M. (2019), "A survey on image data augmentation for deep learning", J. Big Data, 6(1). https://doi.org/10.1186/s40537-019-0197-0   DOI
31 Huynh, T.C., Park, Y.H., Park, J.H., Hong, D.S. and Kim, J.T. (2015), "Effect of temperature variation on vibration monitoring of prestressed concrete girders", Shock Vib., 1-9. https://doi.org/10.1155/2015/741618   DOI
32 Dhivya, J.J. and Ramaswami, M. (2018), "A perusal analysis on hybrid spectrum handoff schemes in cognitive radio networks", International Conference on Intelligent Systems Design and Applications, pp. 312-321. https://doi.org/10.1007/978-3-030-16660-1_31   DOI
33 Ye, X., Jin, T. and Yun, C. (2019), "A review on deep learning-based structural health monitoring of civil infrastructures", Smart Struct. Syst.., Int. J., 24(5), 567-586. https://doi.org/10.12989/sss.2019.24.5.567   DOI
34 Fasl, J., Helwig, T. and Wood, S.L. (2016), "Fatigue response of a fracture-critical bridge at the end of service life", J. Perform. Constr. Facil, 30(5), 04016019. https://doi.org/10.1061/(ASCE)CF.1943-5509.0000871   DOI
35 Sim, H.B. and Uang, C.M. (2012), "Stress analyses and parametric study on full-scale fatigue tests of rib-to-deck welded joints in steel orthotropic decks", J. Bridge Eng., 17(5), 765-773. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000307   DOI
36 Campbell, L.E., Connor, R.J., Whitehead, J.M. and Washer, G.A. (2020), "Benchmark for evaluating performance in visual inspection of fatigue cracking in steel bridges", J. Bridge Eng., 25(1), 04019128. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001507   DOI
37 Chen, L.C., Papandreou, G., Schroff, F. and Adam, H. (2017), "Rethinking atrous convolution for semantic image segmentation", arXiv preprint arXiv:1706.05587. https://doi.org/10.48550/arXiv.1706.05587
38 Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L. (2018a), "DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs", IEEE Transact. Pattern Anal. Machine Intell., 40(4), 834-848. https://doi.org/10.1109/TPAMI.2017.2699184   DOI
39 Connor, R.J. (2012), "Manual for design, construction, and maintenance of orthotropic steel deck bridges", No. FHWA-IF-12-027; United States, Federal Highway Administration. https://rosap.ntl.bts.gov/view/dot/41395
40 Dellenbaugh, L., Kong, X., Al-Salih, H., Collins, W., Bennett, C., Li, J. and Sutley, E.J. (2020), "Development of a distortion-induced fatigue crack characterization methodology using digital image correlation", J. Bridge Eng., 25(9), 04020063. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001598   DOI
41 Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F. and Adam, H. (2018b), "Encoder-decoder with atrous separable convolution for semantic image segmentation", Proceedings of the European Conference on Computer Vision (ECCV). https://doi.org/10.48550/arXiv.1802.02611   DOI
42 Soh, C.K. and Lim, Y.Y. (2009), "Detection and characterization of fatigue induced damage using electromechanical impedance technique", Adv. Mater. Res., 79-82, 2031-2034. https://doi.org/10.4028/www.scientific.net/AMR.79-82.2031   DOI
43 Spencer, B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Engineering, 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030   DOI
44 Sun, L.M., Zhang, W. and Nagarajaiah, S. (2019), "Bridge real-time damage identification method using inclination and strain measurements in the presence of temperature variation", J. Bridge Eng., 24(2), 04018111. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001325   DOI
45 Ronneberger, O., Fischer, P. and Brox, T. (2015), "U-net: convolutional networks for biomedical image segmentation", In: International Conference on Medical image computing and Computer-assisted Intervention, 9351. https://doi.org/10.1007/978-3-319-24574-4_28   DOI
46 Huynh, T.C., Dang, N.L. and Kim, J.T. (2017), "Advances and challenges in impedance-based structural health monitoring", Struct. Monitor. Maint., Int. J., 4(4), 301-329. https://doi.org/10.12989/smm.2017.4.4.301   DOI
47 Ya, S., Yamada, K. and Ishikawa, T. (2011), "Fatigue evaluation of rib-to-deck welded joints of orthotropic steel bridge deck", J. Bridge Eng., 16(4), 492-499. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000181   DOI
48 Huynh, T.C., Nguyen, T.T., Kim, J.T., Ta, Q.B., Ho, D.D. and Phan, T.T.V. (2021), "Deep learning-based functional assessment of piezoelectric-based smart interface under various degradations", Smart Struct. Syst., Int. J., 28(1), 69-87. https://doi.org/10.12989/sss.2021.28.1.069   DOI
49 Huynh, T.C., Dang, N.L. and Kim, J.T. (2018), "PCA-based filtering of temperature effect on impedance monitoring in prestressed tendon anchorage", Smart Struct. Syst., Int. J., 22(1), 57-70. https://doi.org/10.12989/sss.2018.22.1.057   DOI
50 Huynh, T.C., Park, J.H., Jung, H.J. and Kim, J.T. (2019), "Quasi-autonomous bolt-loosening detection method using vision-based deep learning and image processing", Automat. Constr., 105, 102844. https://doi.org/10.1016/j.autcon.2019.102844   DOI
51 Khuc, T. and Catbas, F.N. (2016), "Computer vision-based displacement and vibration monitoring without using physical target on structures", Struct. Infrastr. Eng., 13(4), 505-516. https://doi.org/10.1080/15732479.2016.1164729   DOI
52 Kim, H., Yoon, J. and Sim, S.H. (2020), "Automated bridge component recognition from point clouds using deep learning", Struct. Control Health Monitor, 27(9). https://doi.org/10.1002/stc.2591   DOI
53 Kong, X., Li, J., Collins, W., Bennett, C., Laflamme, S. and Jo, H. (2018), "Sensing distortion-induced fatigue cracks in steel bridges with capacitive skin sensor arrays", Smart Mater. Struct, 27(11), 115008. https://doi.org/10.1088/1361-665X/aadbfb   DOI
54 Dung, C.V. and Anh, L.D. (2019), "Autonomous concrete crack detection using deep fully convolutional neural network", Automat. Constr., 99, 52-58. https://doi.org/10.1016/j.autcon.2018.11.028   DOI
55 Dong, C.Z. and Catbas, F.N. (2020), "A review of computer vision-based structural health monitoring at local and global levels", Struct. Health Monitor., 20(2), 692-743. https://doi.org/10.1177/1475921720935585   DOI
56 Dong, C.Z., Bas, S. and Catbas, F.N. (2020), "Investigation of vibration serviceability of a footbridge using computer vision-based methods", Eng. Struct., 224, 111224. https://doi.org/10.1016/j.engstruct.2020.111224   DOI
57 Dong, C., Li, L., Yan, J., Zhang, Z., Pan, H. and Catbas, F.N. (2021), "Pixel-level fatigue crack segmentation in large-scale images of steel structures using an encoder-decoder network", Sensors, 21(12), 4135. https://doi.org/10.3390/s21124135   DOI
58 Erdogan, Y.S. and Ada, M. (2020), "A computer-vision based vibration transducer scheme for structural health monitoring applications", Smart Mater. Struct., 29(8), 085007. https://doi.org/10.1088/1361-665X/ab9062   DOI
59 Ghahremani, K., Sadhu, A., Walbridge, S. and Narasimhan, S. (2013), "Fatigue testing and structural health monitoring of retrofitted web stiffeners on steel highway bridges", Transport. Res. Record: J. Transport. Res. Board, 2360(1), 27-35. https://doi.org/10.3141/2360-04   DOI