Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

A label-free high precision automated crack detection method based on unsupervised generative attentional networks and swin-crackformer

  • Shiqiao Meng (State Key Laboratory of Disaster Reduction in Civil Engineering, College of Civil Engineering, Tongji University) ;
  • Lezhi Gu (State Key Laboratory of Disaster Reduction in Civil Engineering, College of Civil Engineering, Tongji University) ;
  • Ying Zhou (State Key Laboratory of Disaster Reduction in Civil Engineering, College of Civil Engineering, Tongji University) ;
  • Abouzar Jafari (State Key Laboratory of Disaster Reduction in Civil Engineering, College of Civil Engineering, Tongji University)
  • Received : 2023.05.23
  • Accepted : 2024.08.22
  • Published : 2024.06.25
⟨ Previous Next ⟩

Abstract

Automated crack detection is crucial for structural health monitoring and post-earthquake rapid damage detection. However, realizing high precision automatic crack detection in the absence of corresponding manual labeling presents a formidable challenge. This paper presents a novel crack segmentation transfer learning method and a novel crack segmentation model called Swin-CrackFormer. The proposed method facilitates efficient crack image style transfer through a meticulously designed data preprocessing technique, followed by the utilization of a GAN model for image style transfer. Moreover, the proposed Swin-CrackFormer combines the advantages of Transformer and convolution operations to achieve effective local and global feature extraction. To verify the effectiveness of the proposed method, this study validates the proposed method on three unlabeled crack datasets and evaluates the Swin-CrackFormer model on the METU dataset. Experimental results demonstrate that the crack transfer learning method significantly improves the crack segmentation performance on unlabeled crack datasets. Moreover, the Swin-CrackFormer model achieved the best detection result on the METU dataset, surpassing existing crack segmentation models.

Keywords

Acknowledgement

The research described in this paper was financially supported by the Distinguished Young Scientist Fund of National Natural Science Foundation of China (Grant No. 52025083), the Shanghai Social Development Science and Technology Research Project (Grant No. 22dz1201400), and the National Natural Science Foundation of China (Grant No. U2139209)

References

  1. Alqahtani, H., Kavakli-Thorne, M. and Kumar, G. (2021), "Applications of generative adversarial networks (gans): An updated review", Arch. Computat. Methods Eng., 28, 525-552. https://doi.org/10.1007/s11831-019-09388-y
  2. Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q. and Wang, M. (2023), "Swin-unet: Unet-like pure transformer for medical image segmentation", Proceedings of Computer Vision-ECCV 2022 Workshops (Part III), Tel Aviv, Israel, October.
  3. Cha, Y.J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastr. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263
  4. Chen, F.C. and Jahanshahi, M.R. (2017), "NB-CNN: Deep learning-based crack detection using convolutional neural network and Nave Bayes data fusion", IEEE Transact. Indust. Electr., 65(5), 4392-4400. https://doi.org/10.1109/TIE.2017.2764844
  5. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. and Adam, H. (2018), "Encoder-decoder with atrous separable convolution for semantic image segmentation", Proceedings of the European Conference on Computer Vision (ECCV).
  6. Chen, C., Xie, W., Huang, W., Rong, Y., Ding, X., Huang, Y., Xu, T. and Huang, J. (2019), "Progressive feature alignment for unsupervised domain adaptation", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  7. Chen, G., Teng, S., Lin, M., Yang, X. and Sun, X. (2022), "Crack detection based on generative adversarial networks and deep learning", KSCE J. Civil Eng., 26(4), 1803-1816. https://doi.org/10.1007/s12205-022-0518-2
  8. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G. and Gelly, S. (2020), "An image is worth 16x16 words: Transformers for image recognition at scale", arXiv preprint arXiv:2010.11929.
  9. Du, Y., Pan, N., Xu, Z., Deng, F., Shen, Y. and Kang, H. (2021), "Pavement distress detection and classification based on YOLO network", Int. J. Pave. Eng., 22(13), 1659-1672. https://doi.org/10.1080/10298436.2020.1714047
  10. Dunphy, K., Sadhu, A. and Wang, J. (2022), "Multiclass damage detection in concrete structures using a transfer learning-based generative adversarial networks", Struct. Control Health Monitor., 29(11), e3079. https://doi.org/10.1002/stc.3079
  11. Fernandes, K. and Ciobanu, L. (2014), "Pavement pathologies classification using graph-based features", In: 2014 IEEE International Conference on Image Processing (ICIP).
  12. Ganin, Y. and Lempitsky, V. (2015), "Unsupervised domain adaptation by backpropagation", International Conference on Machine Learning.
  13. Ghifary, M., Kleijn, W.B., Zhang, M., Balduzzi, D. and Li, W. (2016), "Deep reconstruction-classification networks for unsupervised domain adaptation", Proceedings of the 14th European Conference of Computer Vision-ECCV 2016 (Part IV 14), Amsterdam, The Netherlands, October.
  14. Gou, C., Peng, B., Li, T. and Gao, Z. (2019), "Pavement crack detection based on the improved faster-rcnn", In: 2019 IEEE 14th International Conference on Intelligent Systems and Knowledge Engineering (ISKE).
  15. Guo, F., Qian, Y., Liu, J. and Yu, H. (2023), "Pavement crack detection based on transformer network", Autom. Const., 145, 104646. https://doi.org/10.1016/j.autcon.2022.104646
  16. He, K., Zhang, X., Ren, S. and Sun, J. (2015), "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification", Proceedings of the IEEE International Conference on Computer Vision.
  17. Hu, D., Chen, J. and Li, S. (2022), "Reconstructing unseen spaces in collapsed structures for search and rescue via deep learning based radargram inversion", Autom. Const., 140, 104380. https://doi.org/10.1016/j.autcon.2022.104380
  18. Huang, X. and Belongie, S. (2017), "Arbitrary style transfer in real-time with adaptive instance normalization", Proceedings of the IEEE International Conference on Computer Vision.
  19. Jahanshahi, M.R., Masri, S.F., Padgett, C.W. and Sukhatme, G.S. (2013), "An innovative methodology for detection and quantification of cracks through incorporation of depth perception", Mach. Vis. Appl., 24, 227-241. https://doi.org/10.1007/s00138-011-0394-0
  20. Jenkins, M.D., Carr, T.A., Iglesias, M.I., Buggy, T. and Morison, G. (2018), "A deep convolutional neural network for semantic pixel-wise segmentation of road and pavement surface cracks", In: 2018 26th European Signal Processing Conference (EUSIPCO).
  21. Johnson, J., Alahi, A. and Fei-Fei, L. (2016), "Perceptual losses for real-time style transfer and super-resolution", Proceedings of the 14th European Conference of Computer Vision-ECCV 2016 (Part II 14), Amsterdam, The Netherlands, October.
  22. Kim, J., Kim, M., Kang, H. and Lee, K. (2019), "U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation", arXiv preprint arXiv:1907.10830.
  23. Kingma, D.P. and Ba, J. (2014), "Adam: A method for stochastic optimization", arXiv preprint arXiv:1412.6980.
  24. Liu, Y., Yao, J., Lu, X., Xie, R. and Li, L. (2019), "DeepCrack: A deep hierarchical feature learning architecture for crack segmentation", Neurocomputing, 338, 139-153. https://doi.org/10.1016/j.neucom.2019.01.036
  25. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S. and Guo, B. (2021), "Swin transformer: Hierarchical vision transformer using shifted windows", Proceedings of the IEEE/CVF International Conference on Computer Vision.
  26. Liu, G., Niu, Y., Zhao, W., Duan, Y. and Shu, J. (2022), "Data anomaly detection for structural health monitoring using a combination network of GANomaly and CNN", Smart Struct. Syst., Int. J., 29(1), 53-62. https://doi.org/10.12989/sss.2022.29.1.053
  27. Long, M., Cao, Y., Wang, J. and Jordan, M. (2015), "Learning transferable features with deep adaptation networks", International Conference on Machine Learning.
  28. Long, M.S., Zhu, H., Wang, J.M. and Jordan, M.I. (2017), "Deep Transfer Learning with Joint Adaptation Networks", In: International Conference on Machine Learning, 2208-2217.
  29. Ma, D., Fang, H., Wang, N., Lu, H., Matthews, J. and Zhang, C. (2023), "Transformer-optimized generation, detection, and tracking network for images with drainage pipeline defects", Comput.-Aided Civil Infrastruct. Eng., 38(15), 2109-2127. https://doi.org/10.1111/mice.12970
  30. Makantasis, K., Protopapadakis, E., Doulamis, A., Doulamis, N. and Loupos, C. (2015), "Deep convolutional neural networks for efficient vision-based tunnel inspection", In: 2015 IEEE International Conference on Intelligent Computer Communication and Processing (ICCP).
  31. Meng, S., Zhang, X., Qiao, S. and Zhou, Y. (2020), "Research on grid optimized crack detection model based on deep learning", J. Build. Struct., 41(S2), 404-410.
  32. Meng, S., Gao, Z., Zhou, Y., He, B. and Djerrad, A. (2022a), "Real-time automatic crack detection method based on drone", Comput.-Aided Civil Infrastruct. Eng., 38(7), 849-872. https://doi.org/10.1111/mice.12918
  33. Meng, S., Gao, Z., Zhou, Y., He, B. and Kong, Q. (2022b), "A three-stage deep-learning-based method for crack detection of high-resolution steel box girder image", Smart Struct. Syst., Int. J., 29(1), 29-39. https://doi.org/10.12989/sss.2022.29.1.029
  34. Mirza, M. and Osindero, S. (2014), "Conditional generative adversarial nets", arXiv preprint arXiv:1411.1784.
  35. Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y. and Kainz, B. (2018), "Attention u-net: Learning where to look for the pancreas", arXiv preprint arXiv:1804.03999.
  36. Oliveira, H. and Correia, P.L. (2009), "Automatic road crack segmentation using entropy and image dynamic thresholding", In: 2009 17th European Signal Processing Conference.
  37. Ozgenel, C.F. (2019), "Concrete crack segmentation dataset", Mendeley Data, 1, p. 2019.
  38. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N. and Antiga, L. (2019), "Pytorch: An imperative style, high-performance deep learning library", Adv. Neural Inform. Process. Syst., 32.
  39. Pei, L., Sun, Z., Xiao, L., Li, W., Sun, J. and Zhang, H. (2021), "Virtual generation of pavement crack images based on improved deep convolutional generative adversarial network", Eng. Applicat. Artif. Intell., 104, 104376. https://doi.org/10.1016/j.engappai.2021.104376
  40. Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C. and Dosovitskiy, A. (2021), "Do vision transformers see like convolutional neural networks?", Adv. Neural Inform. Process. Syst., 34, 12116-12128.
  41. Ronneberger, O., Fischer, P. and Brox, T. (2015), "U-net: Convolutional networks for biomedical image segmentation", Proceedings of 18th International Conference on Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015 (Part III 18), Munich, Germany, October.
  42. Saito, K., Ushiku, Y. and Harada, T. (2017), "Asymmetric tri-training for unsupervised domain adaptation", In: International Conference on Machine Learning.
  43. Salman, M., Mathavan, S., Kamal, K. and Rahman, M. (2013), "Pavement crack detection using the Gabor filter", In: The 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013).
  44. Shi, Y., Cui, L., Qi, Z., Meng, F. and Chen, Z. (2016), "Automatic road crack detection using random structured forests", IEEE Transact. Intell. Transport. Syst., 17(12), 3434-3445. https://doi.org/10.1109/TITS.2016.2552248
  45. Siu, C., Wang, M. and Cheng, J.C. (2022), "A framework for synthetic image generation and augmentation for improving automatic sewer pipe defect detection", Autom. Const., 137, 104213. https://doi.org/10.1016/j.autcon.2022.104213
  46. Subirats, P., Dumoulin, J., Legeay, V. and Barba, D. (2006), "Automation of pavement surface crack detection using the continuous wavelet transform", In: 2006 International Conference on Image Processing.
  47. Sun, W., Zhou, Y., Xiang, J., Chen, B. and Feng, W. (2022), "Crack detection in concrete slabs by graph-based anomalies calculation", Smart Struct. Syst., Int. J., 29(3), 421-431. https://doi.org/10.12989/sss.2022.29.3.421
  48. Tyagi, S. and Yadav, D. (2021), "A comprehensive review on image synthesis with adversarial networks: Theory, literature, and applications", Arch. Computat. Methods Eng., 29, 2685-2705. https://doi.org/10.1007/s11831-021-09672-w
  49. Tzeng, E., Hoffman, J., Zhang, N., Saenko, K. and Darrell, T. (2014), "Deep domain confusion: Maximizing for domain invariance", arXiv preprint arXiv:1412.3474.
  50. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L. and Polosukhin, I. (2017), "Attention is all you need", Adv. Neural Inform. Process. Syst., 30.
  51. Wang, M. and Deng, W. (2018), "Deep visual domain adaptation: A survey", Neurocomputing, 312, 135-153. https://doi.org/10.1016/j.neucom.2018.05.083
  52. Wang, G. and Xiang, J. (2021), "Railway sleeper crack recognition based on edge detection and CNN", Smart Struct. Syst., Int. J., 28(6), 779-789. https://doi.org/10.12989/sss.2021.28.6.779
  53. Wang, K.C., Li, Q. and Gong, W. (2007), "Wavelet-based pavement distress image edge detection with a trous algorithm", Transport. Res. Record, 2024(1), 73-81. https://doi.org/10.3141/2024-09
  54. Zhang, L., Yang, F., Zhang, Y.D. and Zhu, Y.J. (2016), "Road crack detection using deep convolutional neural network", In: 2016 IEEE International Conference on Image Processing (ICIP).
  55. Zhang, W., Ouyang, W., Li, W. and Xu, D. (2018), "Collaborative and adversarial network for unsupervised domain adaptation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  56. Zhang, K., Zhang, Y. and Cheng, H. (2020), "Self-supervised structure learning for crack detection based on cycle-consistent generative adversarial networks", J. Comput. Civil Eng., 34(3), 04020004. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000883
  57. Zhang, E., Shao, L. and Wang, Y. (2023), "Unifying transformer and convolution for dam crack detection", Autom. Const., 147, 104712. https://doi.org/10.1016/j.autcon.2022.104712
  58. Zhao, H., Qin, G. and Wang, X. (2010), "Improvement of canny algorithm based on pavement edge detection", In: 2010 3rd International Congress on Image and Signal Processing.
  59. Zhao, H., Shi, J., Qi, X., Wang, X. and Jia, J. (2017), "Pyramid scene parsing network", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  60. Zhu, J.-Y., Park, T., Isola, P. and Efros, A.A. (2017), "Unpaired image-to-image translation using cycle-consistent adversarial networks", Proceedings of the IEEE International Conference on Computer Vision.
  61. Zhu, H., Li, Z., Huang, M., Ji, P. and Zhang, Q. (2022), "One-step deep learning-based method for pixel-level detection of fine cracks in steel girder images", Smart Struct. Syst., Int. J., 29(1), 153-166. https://doi.org/10.12989/sss.2022.29.1.153
  62. Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H. and He, Q. (2020), "A comprehensive survey on transfer learning", Proceedings of the IEEE, 109(1), 43-76. https://doi.org/10.1109/JPROC.2020.3004555
  63. Zou, Q., Zhang, Z., Li, Q., Qi, X., Wang, Q. and Wang, S. (2018), "Deepcrack: Learning hierarchical convolutional features for crack detection", IEEE Transact. Image Process., 28(3), 1498-1512. https://doi.org/10.1109/TIP.2018.2878966
  64. Zou, D., Zhang, M., Bai, Z., Liu, T., Zhou, A., Wang, X., Cui, W. and Zhang, S. (2022), "Multicategory damage detection and safety assessment of post-earthquake reinforced concrete structures using deep learning", Comput.-Aided Civil Infrastruct. Eng., 37(9), 1188-1204. https://doi.org/10.1111/mice.12815