Browse > Article
http://dx.doi.org/10.12989/sss.2020.25.6.733

Autonomous vision-based damage chronology for spatiotemporal condition assessment of civil infrastructure using unmanned aerial vehicle  

Mondal, Tarutal Ghosh (Lyles School of Civil Engineering, Purdue University)
Jahanshahi, Mohammad R. (Lyles School of Civil Engineering, Purdue University)
Publication Information
Smart Structures and Systems / v.25, no.6, 2020 , pp. 733-749 More about this Journal
Abstract
This study presents a computer vision-based approach for representing time evolution of structural damages leveraging a database of inspection images. Spatially incoherent but temporally sorted archival images captured by robotic cameras are exploited to represent the damage evolution over a long period of time. An access to a sequence of time-stamped inspection data recording the damage growth dynamics is premised to this end. Identification of a structural defect in the most recent inspection data set triggers an exhaustive search into the images collected during the previous inspections looking for correspondences based on spatial proximity. This is followed by a view synthesis from multiple candidate images resulting in a single reconstruction for each inspection round. Cracks on concrete surface are used as a case study to demonstrate the feasibility of this approach. Once the chronology is established, the damage severity is quantified at various levels of time scale documenting its progression through time. The proposed scheme enables the prediction of damage severity at a future point in time providing a scope for preemptive measures against imminent structural failure. On the whole, it is believed that the present study will immensely benefit the structural inspectors by introducing the time dimension into the autonomous condition assessment pipeline.
Keywords
damage chronology; vision-based inspection; autonomous condition assessment; UAV; civil infrastructures;
Citations & Related Records
Times Cited By KSCI : 5  (Citation Analysis)
연도 인용수 순위
1 Wu, R.T., Singla, A., Jahanshahi, M.R., Bertino, E., Ko, B.J., Verma, D. (2019), "Pruning deep convolutional neural networks for efficient edge computing in condition assessment of infrastructures", Comput.-Aided Civil Infrastruct. Eng., 34(9), 774-789. https://doi.org/10.1111/mice.12449   DOI
2 Yamaguchi, T. and Hashimoto, S. (2010), "Fast crack detection method for large-size concrete surface images using percolation-based image processing", Mach. Vision Applicat., 21(5), 797-809. https://doi.org/10.1007/s00138-009-0189-8   DOI
3 Yeum, C.M., Choi, J. and Dyke, S.J. (2017), "Autonomous image localization for visual inspection of civil infrastructure", Smart Mater. Struct., 26(3), 035051. https://doi.org/10.1088/1361-665X/aa510e   DOI
4 Yeum, C.M., Choi, J. and Dyke, S.J. (2019), "Automated region-of-interest localization and classification for vision-based visual assessment of civil infrastructure", Struct. Health Monitor., 18(3), 675-689. https://doi.org/10.1177/1475921718765419   DOI
5 Yu, S.N., Jang, J.H. and Han, C.S. (2007), "Auto inspection system using a mobile robot for detecting concrete cracks in a tunnel", Automat. Constr., 16(3), 255-261. https://doi.org/10.1016/j.autcon.2006.05.003   DOI
6 Zhu, Z., German, S. and Brilakis, I. (2011), "Visual retrieval of concrete crack properties for automated post-earthquake structural safety evaluation", Automat. Constr., 20(7), 874-883. https://doi.org/10.1016/j.autcon.2011.03.004   DOI
7 Glorot, X. and Bengio, Y. (2010), "Understanding the difficulty of training deep feedforward neural networks", Proceedings of the 13th International Conference on Artificial Intelligence and Statistics.
8 Ghorbani, R., Matta, F. and Sutton, M.A. (2015), "Full-field deformation measurement and crack mapping on confined masonry walls using digital image correlation", Experim. Mech., 55(1), 227-243. https://doi.org/10.1007/s11340-014-9906-y   DOI
9 Girshick, R. (2015), "Fast R-CNN", Proceedings of the IEEE International Conference on Computer Vision.
10 Girshick, R., Donahue, J., Darrell, T. and Malik, J. (2014), "Rich feature hierarchies for accurate object detection and semantic segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
11 He, K., Zhang, X., Ren, S. and Sun, J. (2001), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
12 Hosang, J.H., Benenson, R. and Schiele, B. (2017), "Learning non-maximum suppression", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
13 Jahanshahi, M.R. and Masri, S.F. (2013), "A new methodology for non-contact accurate crack width measurement through photogrammetry for automated structural safety evaluation", Smart Mater. Struct., 22(3), 035019. https://doi.org/10.1088/0964-1726/22/3/035019   DOI
14 Jahanshahi, M.R., Chen, F.C., Ansar, A., Padgett, C.W., Clouse, D. and Bayard, D.S. (2017b), "Accurate and robust scene reconstruction in the presence of misassociated features for aerial sensing", J. Comput. Civil Eng., 31(6), 04017056. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000702   DOI
15 Jahanshahi, M.R. and Masri, S.F. (2014), "Image-based crack detection", Google Patents, 22(3), 8,873,837.
16 Jahanshahi, M.R., Kelly, J.S., Masri, S.F. and Sukhatme, G.S. (2009), "A survey and evaluation of promising approaches for automatic image-based defect detection of bridge structures", Struct. Infrastruct. Eng., 5(6), 455-486. https://doi.org/10.1080/15732470801945930   DOI
17 Jahanshahi, M.R., Masri, S.F. and Sukhatme, G.S. (2011), "Multiimage stitching and scene reconstruction for evaluating defect evolution in structures", Struct. Health Monitor., 10(6), 643-657. https://doi.org/10.1177/1475921710395809   DOI
18 Jahanshahi, M.R., Masri, S.F., Padgett, C.W. and Sukhatme, G.S. (2013), "An innovative methodology for detection and quantification of cracks through incorporation of depth perception", Mach. Vision Applicat., 24(2), 227-241. https://doi.org/10.1007/s00138-011-0394-0   DOI
19 Jahanshahi, M.R., Chen, F.C., Joffe, C. and Masri, S.F. (2017a), "Vision-based quantitative assessment of microcracks on reactor internal components of nuclear power plants", Struct. Infrastruct. Eng., 13(8), 1013-1026. https://doi.org/10.1080/15732479.2016.1231207   DOI
20 Jahanshahi, M.R., Shen, W.M., Mondal, T.G., Abdelbarr, M., Masri, S.F. and Qidwai, U.A. (2017c), "Reconfigurable swarm robots for structural health monitoring: a brief review", Int. J. Intel. Robot. Applicat., 1(3), 287-305. https://doi.org/10.1007/s41315-017-0024-8   DOI
21 Kim, H., Ahn, E., Shin, M. and Sim, S.H. (2019), "Crack and noncrack classification from concrete surface images using machine learning", Struct. Health Monitor., 18(3), 725-738. https://doi.org/10.1177/1475921718768747   DOI
22 Lowe, D.G. (2004), "Distinctive image features from scaleinvariant keypoints", Int. J. Comput. Vision, 60(2), 91-110. https://doi.org/10.1023/B:VISI.0000029664.99615.94   DOI
23 Kong, X. and Li, J. (2018), "Vision-based fatigue crack detection of steel structures using video feature tracking", Comput.-Aided Civil Infrastruct. Eng., 33(9), 783-799. https://doi.org/10.1111/mice.12353   DOI
24 Kong, X. and Li, J. (2019), "Non-contact fatigue crack detection in civil infrastructure through image overlapping and crack breathing sensing", Automat. Constr., 99, 125-139. https://doi.org/10.1016/j.autcon.2018.12.011   DOI
25 Adhikari, R.S., Bagchi, A. and Moselhi, O. (2014), "Automated condition assessment of concrete bridges with digital imaging", Smart Struct. Syst., Int. J., 13(6), 901-925. https://doi.org/10.12989/sss.2014.13.6.901   DOI
26 Agrawal, A.K., Kawaguchi, A. and Chen, Z. (2010), "Deterioration rates of typical bridge elements in New York", J. Bridge Eng., 15(4), 419-429. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000123   DOI
27 Lee, B.Y., Kim, Y.Y., Yi, S.T. and Kim, J.K. (2013), "Automated image processing technique for detecting and analysing concrete surface cracks", Struct. Infrastruct. Eng., 9(6), 567-577. https://doi.org/10.1080/15732479.2011.593891   DOI
28 Levenberg, K. (1944), "A method for the solution of certain nonlinear problems in least squares", Quarterly Appl. Mathe., 2(2), 164-168.   DOI
29 Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P. and Zitnick, C.L. (2001), "Microsoft COCO: Common Objects in Context", Proceedings of the European Conference on Computer Vision, Springer, Cham.
30 Marquardt, D.W. (1963), "An algorithm for least-squares estimation of nonlinear parameters", J. Soc. Indust. Appl. Mathe., 11(2), 431-441. https://doi.org/10.1137/0111030   DOI
31 Matheron, G. (1975), "Random Sets and Integral Geometry", 91(4), 1301-1328.
32 Besuner, P.M. (1976), "Residual life estimates for structures with partial thickness cracks", Proceedings of Mechanics of Crack Growth, ASTM International.
33 Aliakbar, M., Qidwai, U., Jahanshahi, M.R., Masri, S. and Shen, W.M. (2016), "Progressive image stitching algorithm for vision based automated inspection", Proceedings of 2016 International Conference on Machine Learning and Cybernetics (ICMLC), 1, 337-343.
34 Aliakbar, M., Qidwai, U. and Jahanshahi, M.R. (2019), "An evaluation of image-based structural health monitoring using integrated unmanned aerial vehicle platform", Struct. Control Health Monitor., 26(1), e2276. https://doi.org/10.1002/stc.2276   DOI
35 Bay, H., Tuytelaars, T. and Van Gool, L. (2006), "Surf: Speeded Up Robust Features", Proceedings of the European Conference on Computer Vision, Springer, Berlin, Heidelberg.
36 Boller, C., Starke, P., Dobmann, G., Kuo, C.M. and Kuo, C.H. (2015), "Approaching the assessment of ageing bridge infrastructure", Smart Struct. Syst., Int. J., 15(3), 593-608. https://doi.org/10.12989/sss.2015.15.3.593   DOI
37 Bottou, L. (2010), "Large-scale machine learning with stochastic gradient descent", Proceedings of COMPSTAT'2010, Physica-Verlag HD.
38 Cha, Y.J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastruct. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263   DOI
39 Mohanty, S., Chattopadhyay. A. and Peralta, P. (2010), "Adaptive residual useful life estimation of a structural hotspot", J. Intel. Mater. Syst. Struct., 21(3), 321-335. https://doi.org/10.1177/1045389X09357972   DOI
40 Minkowski, H. (1989), "Volumen und oberflache", Proceedings of Ausgewahlte Arbeiten zur Zahlentheorie und zur Geometrie, Springer, Vienna, Austria.
41 Myung, H., Wang, Y., Kang, S.C. and Chen, X. (2014), "Survey on robotics and automation technologies for civil infrastructure", University of Canterbury, Mechanical Engineering.
42 Otsu, N. (1979), "A threshold selection method from gray-level histograms", IEEE Transactions on Systems, Man, and Cybernetics, 9(1), 62-66.   DOI
43 Shan, B., Zheng, S. and Ou, J. (2016), "A stereovision-based crack width detection approach for concrete surface assessment", KSCE J. Civil Eng., 20(2), 803-812. https://doi.org/10.1007/s12205-015-0461-6   DOI
44 Ren, S., He, K., Girshick, R. and Sun, J. (2015), "Faster R-CNN: Towards real-time object detection with region proposal networks", Proceedings of Advances in Neural Information Processing Systems.
45 Salembier, P. (1990), "Comparison of some morphological segmentation algorithms based on contrast enhancement", Proceedings of 5. European Signal Processing Conference.
46 Spencer Jr., B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Eng., 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030   DOI
47 Van Uitert, R. and Bitter, I. (2007), "Subvoxel precise skeletons of volumetric data based on fast marching methods", Medical Physics, 34(2), 627-638. https://doi.org/10.1118/1.2409238   DOI
48 Szegedy, C., Ioffe, S., Vanhoucke, V. and Alemi, A.A. (2001a), "Inception-v4, Inception-Resnet and the impact of residual connections on learning", Proceedings of the 31st AAAI Conference on Artificial Intelligence.
49 Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z. (2001b), "Rethinking the inception architecture for computer vision", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
50 Tanathong, S. and Lee, I. (2013), "Translation-based KLT tracker under severe camera rotation using GPS/INS data", IEEE Geosci. Remote Sensing Lett., 11(1), 64-68. https://doi.org/10.1109/LGRS.2013.2245858   DOI
51 Wang, X., Balakrishnan, N. and Guo, B. (2014), "Residual life estimation based on a generalized Wiener degradation process", Reliabil. Eng. Syst. Safety, 124, 13-23. https://doi.org/10.1016/j.ress.2013.11.011   DOI
52 Cruse, T.A. and Besuner, P.M. (1975), "Residual life prediction for surface cracks in complex structural details", J. Aircraft, 12(4), 369-375. https://doi.org/10.2514/3.44458   DOI
53 Cha, Y.J., Choi, W., Suh, G., Mahmoudkhani, S. and Buyukozturk, O. (2018), "Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types", Comput.-Aided Civil Infrastruct. Eng., 33(9), 731-747. https://doi.org/10.1111/mice.12334   DOI
54 Chen, F.C. and Jahanshahi, M.R. (2018), "NB-CNN: deep learning-based crack detection using convolutional neural network and Naive Bayes data fusion", IEEE Transact. Indust. Electron., 65(5), 4392-4400. https://doi.org/10.1109/TIE.2017.2764844   DOI
55 Choi, J., Yeum, C.M., Dyke, S.J. and Jahanshahi, M.R (2018), "Computer-aided approach for rapid post-event visual evaluation of a building facade", Multidisciplinary Digital Publishing Institute, 18(9), 3017. https://doi.org/10.3390/s18093017
56 Dhawan, S.K., Bindal, A., Bhalla, S. and Bhattacharjee, B. (2019), "Expected residual service life of reinforced concrete structures from current strength considerations", Adv. Struct. Eng., 22(7), 1631-1643. https://doi.org/10.1177/1369433218818001   DOI
57 Duan, Y., Chen, Q., Zhang, H., Yun, C.B., Wu, S. and Zhu, Q. (2019), "CNN-based damage identification method of tied-arch bridge using spatial-spectral information", Smart Struct. Syst., Int. J., 23(5), 507-520. https://doi.org/10.12989/sss.2019.23.5.507
58 Esser, B. and Huston, D.R. (2005), "Versatile robotic platform for structural health monitoring and surveillance", Smart Struct. Syst., Int. J., 1(4), 325-338. https://doi.org/10.12989/sss.2005.1.4.325   DOI
59 Fischler, M.A. and Bolles, R.C. (1981), "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography", Communicat. ACM, 24(6), 381-395. https://doi.org/10.1145/358669.358692   DOI