Browse > Article
http://dx.doi.org/10.12989/sss.2022.30.6.571

Deep reinforcement learning for optimal life-cycle management of deteriorating regional bridges using double-deep Q-networks  

Xiaoming, Lei (Department of Civil and Environmental Engineering, The Hong Kong Polytechnic University)
You, Dong (Department of Civil and Environmental Engineering, The Hong Kong Polytechnic University)
Publication Information
Smart Structures and Systems / v.30, no.6, 2022 , pp. 571-582 More about this Journal
Abstract
Optimal life-cycle management is a challenging issue for deteriorating regional bridges. Due to the complexity of regional bridge structural conditions and a large number of inspection and maintenance actions, decision-makers generally choose traditional passive management strategies. They are less efficiency and cost-effectiveness. This paper suggests a deep reinforcement learning framework employing double-deep Q-networks (DDQNs) to improve the life-cycle management of deteriorating regional bridges to tackle these problems. It could produce optimal maintenance plans considering restrictions to maximize maintenance cost-effectiveness to the greatest extent possible. DDQNs method could handle the problem of the overestimation of Q-values in the Nature DQNs. This study also identifies regional bridge deterioration characteristics and the consequence of scheduled maintenance from years of inspection data. To validate the proposed method, a case study containing hundreds of bridges is used to develop optimal life-cycle management strategies. The optimization solutions recommend fewer replacement actions and prefer preventative repair actions when bridges are damaged or are expected to be damaged. By employing the optimal life-cycle regional maintenance strategies, the conditions of bridges can be controlled to a good level. Compared to the nature DQNs, DDQNs offer an optimized scheme containing fewer low-condition bridges and a more costeffective life-cycle management plan.
Keywords
condition assessment; deteriorating structures; life-cycle management; regional bridges; reinforcement learning;
Citations & Related Records
Times Cited By KSCI : 10  (Citation Analysis)
연도 인용수 순위
1 Lydon, D., Taylor, S.E., Lydon, M., del Rincon, J.M. and Hester, D. (2019), "Development and testing of a composite system for bridge health monitoring utilising computer vision and deep learning", Smart Struct. Syst., Int. J., 24(6), 723-732. https://doi.org/10.12989/sss.2019.24.6.723   DOI
2 Memarzadeh, M. and Pozzi, M. (2019), "Model-free reinforcement learning with model-based safe exploration: Optimizing adaptive recovery process of infrastructure systems", Struct. Safety, 80, 46-55. https://doi.org/10.1016/j.strusafe.2019.04.003   DOI
3 Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G. and Petersen, S. (2015), "Human-level control through deep reinforcement learning", Nature, 518(7540), 529-533. https://doi.org/10.1038/nature14236   DOI
4 Mondal, T.G. and Jahanshahi, M.R. (2020), "Autonomous vision-based damage chronology for spatiotemporal condition assessment of civil infrastructure using unmanned aerial vehicle", Smart Struct. Syst., Int. J., 25(6), 733-749. https://doi.org/10.12989/sss.2020.25.6.733   DOI
5 Oh, B.K., Glisic, B. and Park, H.S. (2021), "Convolutional neural network-based damage detection method for building structures", Smart Struct. Syst., Int. J., 27(6), 903-916. https://doi.org/10.12989/sss.2021.27.6.903   DOI
6 Ti, Z., Deng, X.W. and Yang, H. (2020), "Wake modeling of wind turbines using machine learning", Appl. Energy, 257, p. 114025. https://doi.org/10.1016/j.apenergy.2019.114025   DOI
7 Ti, Z., Deng, X.W. and Zhang, M. (2021), "Artificial Neural Networks based wake model for power prediction of wind farm", Renewable Energy, 172, 618-631. https://doi.org/10.1016/j.renene.2021.03.030   DOI
8 Van Hasselt, H., Guez, A. and Silver, D. (2016), "Deep reinforcement learning with double q-learning", Proceedings of the 13th AAAI Conference on Artificial Intelligence, Vol. 30, No. 1, pp. 2094-2100, Phoenix. AZ, USA, February. https://doi.org/10.1609/aaai.v30i1.10295   DOI
9 Wang, Z., Dong, Y. and Jin, W. (2021), "Life-cycle cost analysis of deteriorating civil infrastructures incorporating social sustainability", J. Infrastr. Syst., 27(3), p. 04021013. https://doi.org/10.1061/(asce)is.1943-555x.0000607   DOI
10 Wei, S., Bao, Y. and Li, H. (2020), "Optimal policy for structure maintenance: A deep reinforcement learning framework", Struct. Safety, 83, p. 101906. https://doi.org/10.1016/j.strusafe.2019.101906   DOI
11 Xia, H.W., Ni, Y.Q., Wong, K.Y. and Ko, J.M. (2012), "Reliability-based condition assessment of in-service bridges using mixture distribution models", Comput. Struct., 106, 204-213. https://doi.org/10.1016/j.compstruc.2012.05.003   DOI
12 Xia, Y., Lei, X., Wang, P., Liu, G. and Sun, L. (2020), "Long-term performance monitoring and assessment of concrete beam bridges using neutral axis indicator", Struct. Control Health Monitor., 27(12), p. e2637. https://doi.org/10.1002/stc.2637   DOI
13 Xia, Y., Lei, X., Wang, P. and Sun, L. (2021), "Artificial intelligence based structural assessment for regional short-and medium-span concrete beam bridges with inspection information", Remote Sens., 13(18), p. 3687. https://doi.org/10.3390/rs13183687   DOI
14 Xia, Y., Lei, X., Wang, P. and Sun, L. (2022), "A data-driven approach for regional bridge condition assessment using inspection reports", Struct. Control Health Monitor., 29(4), p. e2915. https://doi.org/10.1002/stc.2915   DOI
15 Yang, D.Y. (2022), "Adaptive Risk-Based Life-Cycle Management for Large-Scale Structures Using Deep Reinforcement Learning and Surrogate Modeling", J. Eng. Mech., 148(1), p. 04021126. https://doi.org/10.1061/(asce)em.1943-7889.0002028   DOI
16 Yang, X.M., Yi, T.H., Qu, C.X., Li, H.N. and Liu, H. (2020), "Continuous tracking of bridge modal parameters based on subspace correlations", Struct. Control Health Monitor., 27(10), p. e2615. https://doi.org/10.1002/stc.2615   DOI
17 Yang, S., Deng, X., Ti, Z., Yan, B. and Yang, Q. (2022), "Cooperative yaw control of wind farm using a double-layer machine learning framework", Renew. Energy, 193, 519-537. https://doi.org/10.1016/j.renene.2022.04.104   DOI
18 Ye, X.W., Jin, T. and Yun, C.B. (2019), "A review on deep learning-based structural health monitoring of civil infrastructures", Smart Struct. Syst., Int. J., 24(5), 567-585. https://doi.org/10.12989/sss.2019.24.5.567   DOI
19 Yi, T.H., Zhang, J., Qu, C.X. and Li, H.N. (2021), "Damage detection for decks of concrete girder bridges using the frequency obtained from an actively excited vehicle", Smart Struct. Syst., Int. J., 27(1), 101-114. https://doi.org/10.12989/sss.2021.27.1.101   DOI
20 Zhang, N. and Si, W. (2020), "Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks", Reliabil. Eng. Syst. Safety, 203, p. 107094. https://doi.org/10.1016/j.ress.2020.107094   DOI
21 Zhang, Y., Kim, C.W., Zhang, L., Bai, Y., Yang, H., Xu, X. and Zhang, Z. (2020), "Long term structural health monitoring for old deteriorated bridges: a copula-ARMA approach", Smart Struct. Syst., Int. J., 25(3), 285-299. https://doi.org/10.12989/sss.2020.25.3.285   DOI
22 Zheng, X., Yang, D.H., Yi, T.H. and Li, H.N. (2021), "Bridge influence surface identification method considering the spatial effect of vehicle load", Struct. Control Health Monitor., 28(8), p. e2769. https://doi.org/10.1002/stc.2769   DOI
23 Zhu, J. and Wang, Y. (2021), "Feature Selection and Deep Learning for Deterioration Prediction of the Bridges", J. Perform. Constr. Facil., 35(6), p. 04021078. https://doi.org/10.1061/(ASCE)CF.1943-5509.0001653   DOI
24 Andriotis, C.P. and Papakonstantinou, K.G. (2021), "Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints", Reliabil. Eng. Syst. Safety, 212, P. 107551. https://doi.org/10.1016/j.ress.2021.107551   DOI
25 Arulkumaran, K., Deisenroth, M.P., Brundage, M. and Bharath, A.A. (2017), "Deep reinforcement learning: A brief survey", IEEE Signal Process. Magaz., 34(6), 26-38. https://doi.org/10.1109/msp.2017.2743240   DOI
26 Bocchini, P. and Frangopol, D.M. (2011), "Probabilistic bridge network life-cycle connectivity assessment and optimization", In: Applications of Statistics and Probability in Civil Engineering, pp. 160-161.
27 Bocchini, P., Saydam, D. and Frangopol, D.M. (2013), "Efficient, accurate, and simple Markov chain model for the life-cycle analysis of bridge groups", Struct. Safety, 40, 51-64. https://doi.org/10.1016/j.strusafe.2012.09.004   DOI
28 Chalabi, Z. and Lorenc, T. (2013), "Using agent-based models to inform evaluation of complex interventions: Examples from the built environment", Prevent. Medic., 57(5), 434-435. https://doi.org/10.1016/j.ypmed.2013.07.013   DOI
29 Chen, S.R., Cai, C.S. and Levitan, M. (2007), "Understand and improve dynamic performance of transportation system - a case study of Luling Bridge", Eng. Struct., 29(6), 1043-1051. https://doi.org/10.1016/j.engstruct.2006.07.019   DOI
30 Cheng, M. and Frangopol, D.M. (2021a), "A decision-making framework for load rating planning of aging bridges using deep reinforcement learning", J. Comput. Civil Eng., 35(6), p. 04021024. https://doi.org/10.1061/(asce)cp.1943-5487.0000991   DOI
31 Cheng, M. and Frangopol, D.M. (2021b), "Optimal load rating-based inspection planning of corroded steel girders using Markov decision process", Probabil. Eng. Mech., 66, p. 103160. https://doi.org/10.1016/j.probengmech.2021.103160   DOI
32 Cheng, M.Y., Chiu, Y.F., Chiu, C.K., Prayogo, D., Wu, Y.W., Hsu, Z.L. and Lin, C.H. (2019), "Risk-based maintenance strategy for deteriorating bridges using a hybrid computational intelligence technique: a case study", Struct. Infrastr. Eng., 15(3), 334-350. https://doi.org/10.1080/15732479.2018.1547767   DOI
33 Dong, Y., Frangopol, D.M. and Saydam, D. (2014), "Sustainability of highway bridge networks under seismic hazard", J. Earthq. Eng., 18(1), 41-66. https://doi.org/10.1080/13632469.2013.841600   DOI
34 Fan, W., Chen, Y., Li, J., Sun, Y., Feng, J., Hassanin, H. and Sareh, P. (2021), "Machine learning applied to the design and inspection of reinforced concrete bridges: Resilient methods and emerging applications", Structures, 33, 3954-3963. https://doi.org/10.1016/j.istruc.2021.06.110   DOI
35 Fiorillo, G. and Nassif, H. (2020), "Improving the conversion accuracy between bridge element conditions and NBI ratings using deep convolutional neural networks", Struct. Infrastr. Eng., 16(12), 1669-1682. https://doi.org/10.1080/15732479.2020.1725065   DOI
36 Frangopol, D.M. and Bocchini, P. (2012), "Bridge network performance, maintenance and optimisation under uncertainty: accomplishments and challenges", Struct. Infrastr. Eng., 8(4), 341-356. https://doi.org/10.1080/15732479.2011.563089   DOI
37 Frangopol, D.M., Dong, Y. and Sabatino, S. (2017), "Bridge life-cycle performance and cost: analysis, prediction, optimisation and decision-making", Struct. Infrastr. Eng., 13(10), 1239-1257. https://doi.org/10.1080/15732479.2016.1267772   DOI
38 Ge, B. and Kim, S. (2021a), "Determination of appropriate updating parameters for effective life-cycle management of deteriorating structures under uncertainty", Struct. Infrastr. Eng., 17(9), 1284-1298. https://doi.org/10.1080/15732479.2020.1809466   DOI
39 Ge, B. and Kim, S. (2021b), "Probabilistic service life prediction updating with inspection information for RC structures subjected to coupled corrosion and fatigue", Eng. Struct., 238, p. 112260. https://doi.org/10.1016/j.engstruct.2021.112260   DOI
40 Guan, X., Xiang, Z., Bao, Y. and Li, H. (2022), "Structural dominant failure modes searching method based on deep reinforcement learning", Reliabil. Eng. Syst. Safety, 219, p. 108258. https://doi.org/10.1016/j.ress.2021.108258   DOI
41 Hadjidemetriou, G.M., Herrera, M. and Parlikad, A.K. (2022), "Condition and criticality-based predictive maintenance prioritisation for networks of bridges", Struct. Infrastr. Eng., 18(8), 1207-1221. https://doi.org/10.1080/15732479.2021.1897146   DOI
42 Huynh, T.C., Nguyen, T.T., Kim, J.T., Ta, Q.B., Ho, D.D. and Phan, T.T.V. (2021), "Deep learning-based functional assessment of piezoelectric-based smart interface under various degradations", Smart Struct. Syst., Int. J., 28(1), 69-87. https://doi.org/10.12989/sss.2021.28.1.069   DOI
43 Jung, H.J., Lee, J.H., Yoon, S.S. and Kim, I.H. (2019), "Bridge Inspection and condition assessment using Unmanned Aerial Vehicles (UAVs): Major challenges and solutions from a practical perspective", Smart Struct. Syst., Int. J., 24(5), 669-681. https://doi.org/10.12989/sss.2019.24.5.669   DOI
44 Kabir, G., Sadiq, R. and Tesfamariam, S. (2014), "A review of multi-criteria decision-making methods for infrastructure management", Struct. Infrastr. Eng., 10(9), 1176-1210. https://doi.org/10.1080/15732479.2013.795978   DOI
45 Law, S.S. and Li, J. (2010), "Updating the reliability of a concrete bridge structure based on condition assessment with uncertainties", Eng. Struct., 32(1), 286-296. https://doi.org/10.1016/j.engstruct.2009.09.015   DOI
46 Lee, D.H. and Koh, B.H. (2021), "An image-based deep learning network technique for structural health monitoring", Smart Struct. Syst., Int. J., 28(6), 799-810. https://doi.org/10.12989/sss.2021.28.6.799   DOI
47 Lee, J.H., Yoon, S., Kim, B., Gwon, G.H., Kim, I.H. and Jung, H.J. (2021), "A new image-quality evaluating and enhancing methodology for bridge inspection using an unmanned aerial vehicle", Smart Struct. Syst., Int. J., 27(2), 209-226. https://doi.org/10.12989/sss.2021.27.2.209   DOI
48 Lei, X., Sun, L. and Xia, Y. (2020), "Lost data reconstruction for structural health monitoring using deep convolutional generative adversarial networks", Struct. Health Monitor., 20(4), 2069-2087. https://doi.org/10.1177/1475921720959226   DOI
49 Lei, X., Sun, L. and Xia, Y. (2021a), "Lost data reconstruction for structural health monitoring using deep convolutional generative adversarial networks", Struct. Health Monitor. - Int. J., 20(4), 2069-2087. https://doi.org/10.1177/1475921720959226   DOI
50 Lei, X., Sun, L. and Xia, Y. (2021b), "Seismic fragility assessment and maintenance management on regional bridges using bayesian multi-parameter estimation", Bull. Earthq. Eng., 19(15), 6693-6717. https://doi.org/10.1007/s10518-021-01072-6   DOI
51 Lei, X., Xia, Y., Dong, Y. and Sun, L. (2022a), "Multi-level timevariant vulnerability assessment of deteriorating bridge networks with structural condition records", Eng. Struct., 266, p. 114581. https://doi.org/10.1016/j.engstruct.2022.114581   DOI
52 Lei, X., Xia, Y., Komarizadehasl, S. and Sun, L. (2022b), "Condition level deteriorations modeling of RC beam bridges with U-Net convolutional neural networks", Structures, 42, 333-342. http://https://doi.org/10.1016/j.istruc.2022.06.013   DOI
53 Li, S., Ou, J., Wang, J., Gao, X. and Yang, C. (2019), "Level 2 safety evaluation of concrete-filled steel tubular arch bridges incorporating structural health monitoring and inspection information based on China bridge standards", Struct. Control Health Monitor., 26(3), p. e2303. https://doi.org/10.1002/stc.2303   DOI
54 Liu, H. and Zhang, Y. (2020), "Bridge condition rating data modeling using deep learning algorithm", Struct. Infrastr. Eng., 16(10), 1447-1460. https://doi.org/10.1080/15732479.2020.1712610   DOI
55 Liu, K., Zhai, C. and Dong, Y. (2021), "Optimal restoration schedules of transportation network considering resilience", Struct. Infrastr. Eng., 17(8), 1141-1154. https://doi.org/10.1080/15732479.2020.1801764   DOI