Nuclear Engineering and Technology

Korean Nuclear Society (한국원자력학회)
권호 표지
DOI QR코드

DOI QR Code

Flexible operation and maintenance optimization of aging cyber-physical energy systems by deep reinforcement learning

  • Received : 2023.02.10
  • Accepted : 2023.11.30
  • Published : 2024.04.25
Download PDF
⟨ Previous Next ⟩

Abstract

Cyber-Physical Energy Systems (CPESs) integrate cyber and hardware components to ensure a reliable and safe physical power production and supply. Renewable Energy Sources (RESs) add uncertainty to energy demand that can be dealt with flexible operation (e.g., load-following) of CPES; at the same time, scenarios that could result in severe consequences due to both component stochastic failures and aging of the cyber system of CPES (commonly overlooked) must be accounted for Operation & Maintenance (O&M) planning. In this paper, we make use of Deep Reinforcement Learning (DRL) to search for the optimal O&M strategy that, not only considers the actual system hardware components health conditions and their Remaining Useful Life (RUL), but also the possible accident scenarios caused by the failures and the aging of the hardware and the cyber components, respectively. The novelty of the work lies in embedding the cyber aging model into the CPES model of production planning and failure process; this model is used to help the RL agent, trained with Proximal Policy Optimization (PPO) and Imitation Learning (IL), finding the proper rejuvenation timing for the cyber system accounting for the uncertainty of the cyber system aging process. An application is provided, with regards to the Advanced Lead-cooled Fast Reactor European Demonstrator (ALFRED).

Keywords

References

  1. E. Zio, The future of risk assessment, Reliab. Eng. Syst. Saf. (2018), https://doi.org/10.1016/j.ress.2018.04.020. 
  2. L. Pinciroli, P. Baraldi, G. Ballabio, M. Compare, E. Zio, Optimization of the operation and maintenance of renewable energy systems by deep reinforcement learning, Renew. Energy 183 (2022) 752-763. 
  3. E. Gursel, B. Reddy, A. Khojandi, M. Madadi, J.B. Coble, V. Agarwal, V. Yadav, R. L. Boring, Using artificial intelligence to detect human errors in nuclear power plants: a case in operation and maintenance, Nucl. Eng. Technol. (2022), https://doi.org/10.1016/j.net.2022.10.032. 
  4. E. Zio, Prognostics and Health Management (PHM): where are we and where do we (need to) go in theory and practice, Reliab. Eng. Syst. Saf. 218 (2022), 108119. 
  5. P. Baraldi, F. Mangili, E. Zio, Investigation of uncertainty treatment capability of model-based and data-driven prognostic methods using simulated data, Reliab. Eng. Syst. Saf. 112 (2013) 94-108. 
  6. F. Di Maio, P. Baraldi, E. Zio, R. Seraoui, Fault detection in nuclear power plants components by a combination of statistical methods, IEEE Trans. Reliab. 62 (2013) 833-845. 
  7. L.M. Elshenawy, M.A. Halawa, T.A. Mahmoud, H.A. Awad, M.I. Abdo, Unsupervised machine learning techniques for fault detection and diagnosis in nuclear power plants, Prog. Nucl. Energy 142 (2021), 103990. 
  8. G. Qian, J. Liu, Development of deep reinforcement learning-based fault diagnosis method for rotating machinery in nuclear power plants, Prog. Nucl. Energy 152 (2022), 104401. 
  9. Z. Welz, J. Coble, B. Upadhyaya, W. Hines, Maintenance-based prognostics of nuclear plant equipment for long-term operation, Nucl. Eng. Technol. 49 (2017) 914-919, https://doi.org/10.1016/j.net.2017.06.001. 
  10. M. Compare, P. Baraldi, E. Zio, Challenges to IoT-enabled predictive maintenance for industry 4.0, IEEE Internet Things J. 7 (2019) 4585-4597. 
  11. H. Peng, Y. Wang, X. Zhang, Q. Hu, B. Xu, Optimization of preventive maintenance of nuclear safety-class DCS based on reliability modeling, Nucl. Eng. Technol. 54 (2022) 3595-3603, https://doi.org/10.1016/j.net.2022.05.011. 
  12. H.A. Gohel, H. Upadhyay, L. Lagos, K. Cooper, A. Sanzetenea, Predictive maintenance architecture development for nuclear infrastructure using machine learning, Nucl. Eng. Technol. 52 (2020) 1436-1442, https://doi.org/10.1016/j.net.2019.12.029. 
  13. T. Jiejuan, M. Dingyuan, X. Dazhi, A genetic algorithm solution for a nuclear power plant risk-cost maintenance model, Nucl. Eng. Des. 229 (2004) 81-89. 
  14. A.W. Labib, M.N. Yuniarto, Maintenance strategies for changeable manufacturing, in: Changeable and Reconfigurable Manufacturing Systems, Springer, 2009, pp. 337-351. 
  15. L. Pinciroli, P. Baraldi, G. Ballabio, C. Compare, E. Zio, Deep reinforcement learning for optimizing operation and maintenance of energy systems equipped with phm capabilities, in: Proceedings of the Proceedings of the 30th European Safety and Reliability Conference and the 15th Probabilistic Safety Assessment and Management Conference, 2020. 
  16. M.T. Kartal, A. Samour, T.S. Adebayo, S.K. Depren, Do nuclear energy and renewable energy surge environmental quality in the United States? New insights from novel bootstrap Fourier Granger causality in quantiles approach, Prog. Nucl. Energy 155 (2023), 104509. 
  17. G. Chen, M. Li, Y. Zou, H. Xu, Analysis of load-following operation characteristics of liquid fuel molten salt reactor, Prog. Nucl. Energy 150 (2022), 104308. 
  18. B. Tjahjono, C. Esplugues, E. Ares, G. Pelaez, What does industry 4.0 mean to supply chain? Procedia Manuf. 13 (2017) 1175-1182. 
  19. Z. Hao, F. Di Maio, L. Pinciroli, E. Zio, Optimal prescriptive maintenance of nuclear power plants by deep reinforcement learning, in: Proceedings of the Proceedings of the 32nd European Safety and Reliability Conference, 2022. 
  20. V. Holmgren, General-purpose Maintenance Planning Using Deep Reinforcement Learning and Monte Carlo Tree Search, 2019. 
  21. M. Grottke, R. Matias, K.S. Trivedi, The fundamentals of software aging, in: Proceedings of the 2008 IEEE International Conference on Software Reliability Engineering Workshops (ISSRE Wksp), Ieee, 2008, pp. 1-6. 
  22. K.S. Trivedi, K. Vaidyanathan, K. Goseva-Popstojanova, Modeling and analysis of software aging and rejuvenation, in: Proceedings of the Proceedings 33rd Annual Simulation Symposium (SS 2000), IEEE, 2000, pp. 270-279. 
  23. W. Wang, A. Cammi, F. Di Maio, S. Lorenzi, E. Zio, A Monte Carlo-based exploration framework for identifying components vulnerable to cyber threats in nuclear power plants, Reliab. Eng. Syst. Saf. 175 (2018) 24-37. 
  24. Z. Hao, F. Di Maio, E. Zio, A multi-state model of the aging process of cyber-physical systems, in: Proceedings of the 30th European Safety and Reliability Conference, ESREL 2020 and 15th Probabilistic Safety Assessment and Management Conference, PSAM15 2020, Research Publishing, Singapore, 2020, pp. 2241-2248. 
  25. Y. Huang, C. Kintala, N. Kolettis, N.D. Fulton, Software rejuvenation: analysis, module and applications, in: Proceedings of the Twenty-Fifth International Symposium on Fault-Tolerant Computing. Digest of Papers, IEEE, 1995, pp. 381-390. 
  26. D. Cotroneo, R. Natella, R. Pietrantuono, S. Russo, A survey of software aging and rejuvenation studies, ACM J. Emerg. Technol. Comput. Syst. 10 (2014) 1-34. 
  27. R.S. Sutton, A.G. Barto, Reinforcement Learning: an Introduction, MIT press, 2018. ISBN 0262352702. 
  28. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization Algorithms, 2017 arXiv Prepr. arXiv1707.06347. 
  29. J. Ho, J.K. Gupta, S. Ermon, Model-free imitation learning with policy optimization, in: Proceedings of the 33rd International Conference on Machine Learning, vol. 6, ICML 2016, 2016, pp. 4036-4046. 
  30. Z. Hao, F. Di Maio, E. Zio, Dynamic reliability assessment of cyber-physical energy systems (CPEs) by GTST-MLD, in: Proceedings of the 2021 5th International Conference on System Reliability and Safety (ICSRS), IEEE, 2021, pp. 98-102. 
  31. Z. Hao, F. Di Maio, E. Zio, Modelling the Aging Process of a Cyber Physical System, 2019. 
  32. Z. Hao, F. Di Maio, E. Zio, Multi-state reliability assessment model of base-load cyber-physical energy systems (CPES) during flexible operation considering the aging of cyber components, Energies 14 (2021) 3241. 
  33. R. Ponciroli, A. Bigoni, A. Cammi, S. Lorenzi, L. Luzzi, Object-oriented modelling and simulation for the ALFRED dynamics, Prog. Nucl. Energy 71 (2014) 15-29. 
  34. K.J. Astrom, B. Wittenmark, Computer-controlled Systems: Theory and Design, Courier Corporation, 2013. ISBN 0486284042. 
  35. X. Du, Y. Qi, D. Hou, Y. Chen, X. Zhong, Modeling and performance analysis of software rejuvenation policies for multiple degradation systems, in: Proceedings of the 2009 33rd Annual IEEE International Computer Software and Applications Conference, vol. 1, IEEE, 2009, pp. 240-245. 
  36. Y.-J. Lin, J.-M. Yang, R.-Y. Wang, Y.-X. Yang, Research on common cause fault evaluation model of RTS based on β-factor method, in: Proceedings of the International Symposium on Software Reliability, Industrial Safety, Cyber Security and Physical Protection for Nuclear Power Plant, Springer, 2022, pp. 590-599. 
  37. Z.-G. Wu, J. Zhu, X.-B. Yu, Reliability analysis of tripping solenoid valve power system based on dynamic fault tree and sequential Monte Carlo, in: Proceedings of the International Symposium on Software Reliability, Industrial Safety, Cyber Security and Physical Protection for Nuclear Power Plant, Springer, 2022, pp. 148-158. 
  38. N. Vanvuchelen, J. Gijsbrechts, R. Boute, Use of proximal policy optimization for the joint replenishment problem, Comput. Ind. 119 (2020), 103239, https://doi.org/10.1016/j.compind.2020.103239. 
  39. S. Ross, J.A. Bagnell, Efficient reductions for imitation learning, J. Mach. Learn. Res. 9 (2010) 661-668. 
  40. Z. Hao, F. Di Maio, E. Zio, Optimal prescriptive maintenance of nuclear power plants by deep reinforcement learning, in: Proceedings of the 32nd European Safety and Reliability Conference, ESREL, 2022, p. 2022. 
  41. G. Terol, Porous Media Approach in CFD Thermohydraulic Simulation of Nuclear Generation-IV Lead-Cooled Fast Reactor ALFRED, 2021. 
  42. F. Di Maio, R. Mascherona, E. Zio, Risk analysis of cyber-physical systems by GTSTMLD, IEEE Syst. J. 14 (2019) 1333-1340. 
  43. S. Zhang, M. Du, J. Tong, Y.-F. Li, Multi-objective optimization of maintenance program in multi-unit nuclear power plant sites, Reliab. Eng. Syst. Saf. 188 (2019) 532-548. 
  44. S. Martorell, A. Sanchez, S. Carlos, V. Serradell, Simultaneous and multi-criteria optimization of TS requirements and maintenance at NPPs, Ann. Nucl. Energy 29 (2002) 147-168. 
  45. H. Ludwig, T. Salnikova, A. Stockman, U. Waas, Load cycling capabilities of German nuclear power plants (NPP), VGB PowerTech 91 (2011) 38-44. 
  46. O. Eungse, L. Kangdae, Y. Sungok, Evaluation of commercial digital control systems for NPP I&C system upgrades, in: Proceedings of the Transactions of the, Korean Nuclear Society Spring Meeting, 2007. 
  47. International Atomic Energy Agency Non-baseload Operations in Nuclear Power Plants: Load Following and Frequency Control Modes of Flexible Operation, IAEA, 2018. ISBN 9201108168. 
  48. R.T. Rockafellar, S. Uryasev, Conditional value-at-risk for general loss distributions, J. Bank. Finance 26 (2002) 1443-1471.