Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Twin models for high-resolution visual inspections

  • Seyedomid Sajedi (Department of Civil, Structural and Environmental Engineering, University at Buffalo, the State University of New York) ;
  • Kareem A. Eltouny (Department of Civil, Structural and Environmental Engineering, University at Buffalo, the State University of New York) ;
  • Xiao Liang (Department of Civil, Structural and Environmental Engineering, University at Buffalo, the State University of New York)
  • Received : 2022.08.30
  • Accepted : 2023.02.02
  • Published : 2023.04.25
⟨ Previous Next ⟩

Abstract

Visual structural inspections are an inseparable part of post-earthquake damage assessments. With unmanned aerial vehicles (UAVs) establishing a new frontier in visual inspections, there are major computational challenges in processing the collected massive amounts of high-resolution visual data. We propose twin deep learning models that can provide accurate high-resolution structural components and damage segmentation masks efficiently. The traditional approach to cope with high memory computational demands is to either uniformly downsample the raw images at the price of losing fine local details or cropping smaller parts of the images leading to a loss of global contextual information. Therefore, our twin models comprising Trainable Resizing for high-resolution Segmentation Network (TRS-Net) and DmgFormer approaches the global and local semantics from different perspectives. TRS-Net is a compound, high-resolution segmentation architecture equipped with learnable downsampler and upsampler modules to minimize information loss for optimal performance and efficiency. DmgFormer utilizes a transformer backbone and a convolutional decoder head with skip connections on a grid of crops aiming for high precision learning without downsizing. An augmented inference technique is used to boost performance further and reduce the possible loss of context due to grid cropping. Comprehensive experiments have been performed on the 3D physics-based graphics models (PBGMs) synthetic environments in the QuakeCity dataset. The proposed framework is evaluated using several metrics on three segmentation tasks: component type, component damage state, and global damage (crack, rebar, spalling). The models were developed as part of the 2nd International Competition for Structural Health Monitoring.

Keywords

References

  1. Abdel-Qader, I., Abudayyeh, O. and Kelly, M.E. (2003), "Analysis of edge-detection techniques for crack identification in bridges", J. Comput. Civil Eng., 17(4), 255-263. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(255)
  2. Agyemang, I.O., Zhang, X., Mensah, I.A., Mawuli, B.C., Agbley, B.L.Y. and Arhin, J.R. (2021), "Enhanced deep convolutional neural network for building component detection towards structural health monitoring", Proceedings of the 4th International Conference on Pattern Recognition and Artificial Intelligence (PRAI), Yibin, China, August. https://doi.org/10.1109/PRAI53619.2021.9551102
  3. Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M. and Kalinin, A.A. (2020), "Albumentations: fast and flexible image augmentations", Information, 11(2), 125. https://doi.org/10.3390/info11020125
  4. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L. (2017), "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs", IEEE Transact. Pattern Anal. Mach. Intell., 40(4), 834-848. https://doi.org/10.1109/TPAMI.2017.2699184
  5. Dorafshan, S., Thomas, R.J. and Maguire, M. (2018), "Fatigue crack detection using unmanned aerial systems in fracture critical inspection of steel bridges", J. Bridge Eng., 23(10), 04018078. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001291
  6. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J. and Houlsby, N. (2021), "An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale", Proceedings of 2021 International Conference on Learning Representations (ICLR).
  7. Guo, J., Wang, Q. and Li, Y. (2021), "Semi-supervised learning based on convolutional neural network and uncertainty filter for facade defects classification", Comput.-Aided Civil Infrastr. Eng., 36(3), 302-317. https://doi.org/10.1111/mice.12632
  8. He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778.
  9. He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J. and Li, M. (2019), "Bag of tricks for image classification with convolutional neural networks", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  10. Hinton, G.E., Osindero, S. and Teh, Y.-W. (2006), "A fast learning algorithm for deep belief nets", Neural Computat., 18(7), 1527-1554. https://doi.org/10.1162/neco.2006.18.7.1527
  11. Hoskere, V., Narazaki, Y., Hoang, T. and Spencer Jr, B. (2017), "Vision-based structural inspection using multiscale deep convolutional neural networks", Proceedings of the 3rd Huixian International Forum on Earthquake Engineering for Young Researchers, Urbana, IL, USA. https://doi.org/10.48550/arXiv.1805.01055
  12. Hoskere, V., Narazaki, Y., Hoang, T.A. and Spencer Jr, B. (2020), "MaDnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure", J. Civil Struct. Health Monitor., 10(5), 757-773. https://doi.org/10.1007/s13349-020-00409-0
  13. Hoskere, V., Narazaki, Y. and Spencer Jr, B.F. (2022), "Physics-based graphics models in 3D synthetic environments as autonomous vision-based inspection testbeds", Sensors, 22(2), 532. https://doi.org/10.3390/s22020532
  14. Hou, X., Zeng, Y. and Xue, J. (2020), "Detecting structural components of building engineering based on deep-learning method", J. Constr. Eng. Manag., 146(2), 04019097. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001751
  15. Hu, J., Shen, L. and Sun, G. (2018), "Squeeze-and-excitation networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132-7141.
  16. IC-SHM (2021), Structures and Artificial Intelligence Lab. Retrieved 2022/04/30 from https://sail.cive.uh.edu/ic-shm2021/
  17. Jahanshahi, M.R. and Masri, S.F. (2012), "Adaptive vision-based crack detection using 3D scene reconstruction for condition assessment of structures", Automat. Constr., 22, 567-576. https://doi.org/10.1016/j.autcon.2011.11.018
  18. Jegou, S., Drozdzal, M., Vazquez, D., Romero, A. and Bengio, Y. (2017), "The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 11-19.
  19. Katharopoulos, A. and Fleuret, F. (2019), "Processing megapixel images with deep attention-sampling models", Proceedings of the 36th International Conference on Machine Learning, pp. 3282-3291.
  20. Kingma, D.P. and Ba, J. (2014), "Adam: A method for stochastic optimization", arXiv preprint arXiv:1412.6980. https://doi.org/10.48550/arXiv.1412.6980
  21. Li, L., Wang, Q., Zhang, G., Shi, L., Dong, J. and Jia, P. (2018), "A method of detecting the cracks of concrete undergo high-temperature", Constr. Build. Mater., 162, 345-358. https://doi.org/10.1016/j.conbuildmat.2017.12.010
  22. Liang, X. (2019), "Image-based post-disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization", Comput.-Aided Civil Infrastr. Eng, 34(5), 415-430. https://doi.org/10.1111/mice.12425
  23. Lin, T.-Y., Goyal, P., Girshick, R., He, K. and Dollar, P. (2017), "Focal loss for dense object detection", Proceedings of the IEEE International Conference on Computer Vision, pp. 2980-2988.
  24. Liu, Z., Cao, Y., Wang, Y. and Wang, W. (2019), "Computer vision-based concrete crack detection using U-net fully convolutional networks", Automat. Constr., 104, 129-139. https://doi.org/10.1016/j.autcon.2019.04.005
  25. Liu, Y.F., Nie, X., Fan, J.S. and Liu, X.G. (2020), "Image-based crack assessment of bridge piers using unmanned aerial vehicles and three-dimensional scene reconstruction", Comput.-Aided Civil Infrastr. Eng., 35(5), 511-529. https://doi.org/10.1111/mice.12501
  26. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S. and Guo, B. (2021), "Swin transformer: Hierarchical vision transformer using shifted windows", arXiv preprint arXiv:2103.14030. pp. 10012-10022
  27. Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T. and Xie, S. (2022), "A convnet for the 2020s". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976-11986.
  28. Munawar, H.S., Hammad, A.W., Haddad, A., Soares, C.A.P. and Waller, S.T. (2021), "Image-based crack detection methods: A review", Infrastructures, 6(8), 115. https://doi.org/10.3390/infrastructures6080115
  29. Narazaki, Y., Hoskere, V., Hoang, T.A., Fujino, Y., Sakurai, A. and Spencer Jr, B.F. (2020), "Vision-based automated bridge component recognition with high-level scene consistency", Comput.-Aided Civil Infrastr. Eng., 35(5), 465-482. https://doi.org/10.1111/mice.12505
  30. Narazaki, Y., Hoskere, V., Chowdhary, G. and Spencer Jr, B.F. (2022), "Vision-based navigation planning for autonomous post-earthquake inspection of reinforced concrete railway viaducts using unmanned aerial vehicles", Automat. Constr., 137, 104214. https://doi.org/10.1016/j.autcon.2022.104214
  31. PapersWithCode (2022), Semantic Segmentation. https://paperswithcode.com/task/semantic-segmentation
  32. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N. and Antiga, L. (2019), "Pytorch: An imperative style, high-performance deep learning library", Adv. Neural Inform. Process. Syst., 32, 8026-8037.
  33. Sajedi, S.O. and Liang, X. (2019), "A convolutional cost-sensitive crack localization algorithm for automated and reliable RC bridge inspection", In: Risk-Based Bridge Engineering, pp. 229-235.
  34. Sajedi, S.O. and Liang, X. (2021), "Uncertainty-assisted deep vision structural health monitoring", Comput.-Aided Civil Infrastr. Eng., 36(2), 126-142. https://doi.org/10.1111/mice.12580
  35. Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D. and Wang, Z. (2016), "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874-1883.
  36. Talab, A.M.A., Huang, Z., Xi, F. and HaiMing, L. (2016), "Detection crack in image using Otsu method and multiple filtering in image processing techniques", Optik, 127(3), 1030-1033. https://doi.org/10.1016/j.ijleo.2015.09.147
  37. Tan, M. and Le, Q. (2019), "Efficientnet: Rethinking model scaling for convolutional neural networks", International Conference on Machine Learning, pp. 6105-6114.
  38. Tang, W., Wu, R.-T. and Jahanshahi, M.R. (2022), "Crack segmentation in high-resolution images using cascaded deep convolutional neural networks and Bayesian data fusion", Smart Struct. Syst., Int. J., 29(1), 221-235. https://doi.org/10.12989/sss.2022.29.1.221
  39. Teng, S., Liu, Z., Chen, G. and Cheng, L. (2021), "Concrete crack detection based on well-known feature extractor model and the YOLO_v2 network", Appl. Sci., 11(2), 813. https://doi.org/10.3390/app11020813
  40. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L. and Polosukhin, I. (2017), "Attention is all you need", Adv. Neural Inform. Process. Syst.
  41. Wightman, R. (2019), PyTorch Image Models, GitHub. https://github.com/rwightman/pytorch-image-models
  42. Wu, Y., Qin, Y., Qian, Y., Guo, F., Wang, Z. and Jia, L. (2022), "Hybrid deep learning architecture for rail surface segmentation and surface defect detection", Comput.-Aided Civil Infrastr. Eng., 37(2), 227-244. https://doi.org/10.1111/mice.12710
  43. Yakubovskiy, P. (2020), Segmentation Models Pytorch. GitHub. https://github.com/qubvel/segmentation_models.pytorch
  44. Zhang, A., Wang, K.C., Li, B., Yang, E., Dai, X., Peng, Y., Fei, Y., Liu, Y., Li, J.Q. and Chen, C. (2017), "Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network", Comput.-Aided Civil Infrastr. Eng., 32(10), 805-819. https://doi.org/10.1111/mice.12297
  45. Zhang, K., Cheng, H. and Zhang, B. (2018), "Unified approach to pavement crack and sealed crack detection using preclassification based on transfer learning", J. Comput. Civil Eng., 32(2), 04018001. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000736
  46. Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Lin, H., Zhang, Z., Sun, Y., He, T., Mueller, J. and Manmatha, R. (2020), "Resnest: Split-attention networks", arXiv preprint arXiv:2004.08955.
  47. Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Lin, H., Zhang, Z., Sun, Y., He, T., Mueller, J. and Manmatha, R. (2022), "Resnest: Split-attention networks", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2736-2746.
  48. Zhang, Y., Qin, J., Park, D.S., Han, W., Chiu, C.-C., Pang, R., Le, Q.V. and Wu, Y. (2020), "Pushing the limits of semi-supervised learning for automatic speech recognition", arXiv preprint arXiv:2010.10504. https://doi.org/10.48550/arXiv.2010.10504
  49. Zheng, Y., Gao, Y., Lu, S. and Mosalam, K.M. (2022), "Multistage semisupervised active learning framework for crack identification, segmentation, and measurement of bridges", Comput.-Aided Civil Infrastr. Eng., 37(9), 1089-1108. https://doi.org/10.1111/mice.12851
  50. Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N. and Liang, J. (2018), "Unet++: A nested u-net architecture for medical image segmentation", In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 3-11. https://doi.org/10.1007/978-3-030-00889-5_1
  51. Zhou, Z., Zhang, J. and Gong, C. (2022), "Automatic detection method of tunnel lining multi-defects via an enhanced You Only Look Once network", Comput.-Aided Civil Infrastr. Eng., 37(6), 762-780. https://doi.org/https://doi.org/10.1111/mice.12836