Nuclear Engineering and Technology

Korean Nuclear Society (한국원자력학회)
권호 표지
DOI QR코드

DOI QR Code

PESA: Prioritized experience replay for parallel hybrid evolutionary and swarm algorithms - Application to nuclear fuel

  • Radaideh, Majdi I. (Department of Nuclear Science and Engineering, Massachusetts Institute of Technology) ;
  • Shirvan, Koroush (Department of Nuclear Science and Engineering, Massachusetts Institute of Technology)
  • Received : 2022.03.09
  • Accepted : 2022.05.01
  • Published : 2022.10.25
Download PDF
⟨ Previous Next ⟩

Abstract

We propose a new approach called PESA (Prioritized replay Evolutionary and Swarm Algorithms) combining prioritized replay of reinforcement learning with hybrid evolutionary algorithms. PESA hybridizes different evolutionary and swarm algorithms such as particle swarm optimization, evolution strategies, simulated annealing, and differential evolution, with a modular approach to account for other algorithms. PESA hybridizes three algorithms by storing their solutions in a shared replay memory, then applying prioritized replay to redistribute data between the integral algorithms in frequent form based on their fitness and priority values, which significantly enhances sample diversity and algorithm exploration. Additionally, greedy replay is used implicitly to improve PESA exploitation close to the end of evolution. PESA features in balancing exploration and exploitation during search and the parallel computing result in an agnostic excellent performance over a wide range of experiments and problems presented in this work. PESA also shows very good scalability with number of processors in solving an expensive problem of optimizing nuclear fuel in nuclear power plants. PESA's competitive performance and modularity over all experiments allow it to join the family of evolutionary algorithms as a new hybrid algorithm; unleashing the power of parallel computing for expensive optimization.

Keywords

References

  1. A. Abraham, H. Guo, H. Liu, Swarm intelligence: foundations, perspectives and applications, in: Swarm Intelligent Systems, Springer, 2006, pp. 3-25.
  2. E. Sanchez, G. Squillero, A. Tonda, Industrial Applications of Evolutionary Algorithms.
  3. A.A. Freitas, A review of evolutionary algorithms for e-commerce, in: E-Commerce and Intelligent Methods, Springer, 2002, pp. 159-179.
  4. C. Grosan, A. Abraham, Hybrid evolutionary algorithms: methodologies, architectures, and reviews, in: Hybrid Evolutionary Algorithms, Springer, 2007, pp. 1-17.
  5. Y.-T. Kao, E. Zahara, A hybrid genetic algorithm and particle swarm optimization for multimodal functions, Appl. Soft Comput. 8 (2) (2008) 849-857. https://doi.org/10.1016/j.asoc.2007.07.002
  6. P.-H. Chen, S.M. Shahandashti, Hybrid of genetic algorithm and simulated annealing for multiple project scheduling with multiple resource constraints, Autom. ConStruct. 18 (4) (2009) 434-443. https://doi.org/10.1016/j.autcon.2008.10.007
  7. P.C. Ma, F. Tao, Y.L. Liu, L. Zhang, H.X. Lu, Z. Ding, A hybrid particle swarm optimization and simulated annealing algorithm for job-shop scheduling, in: 2014 IEEE International Conference on Automation Science and Engineering (CASE), IEEE, 2014, pp. 125-130.
  8. Q. Shen, W.-M. Shi, W. Kong, Hybrid particle swarm optimization and tabu search approach for selecting genes for tumor classification using gene expression data, Comput. Biol. Chem. 32 (1) (2008) 53-60. https://doi.org/10.1016/j.compbiolchem.2007.10.001
  9. A. Jamasb, S.-H. Motavalli-Anbaran, K. Ghasemi, A novel hybrid algorithm of particle swarm optimization and evolution strategies for geophysical nonlinear inverse problems, Pure Appl. Geophys. 176 (4) (2019) 1601-1613. https://doi.org/10.1007/s00024-018-2059-7
  10. T. Shankar, S. Shanmugavel, A. Rajesh, Hybrid hsa and pso algorithm for energy efficient cluster head selection in wireless sensor networks, Swarm Evol. Comput. 30 (2016) 1-10. https://doi.org/10.1016/j.swevo.2016.03.003
  11. J. Sun, H. Zhang, A. Zhou, Q. Zhang, K. Zhang, A new learning-based adaptive multi-objective evolutionary algorithm, Swarm Evol. Comput. 44 (2019) 304-319. https://doi.org/10.1016/j.swevo.2018.04.009
  12. Y. Zhang, X. Liu, F. Bao, J. Chi, C. Zhang, P. Liu, Particle Swarm Optimization with Adaptive Learning Strategy, Knowledge-Based Systems, 2020, 105789.
  13. A. Assad, K. Deep, A hybrid harmony search and simulated annealing algorithm for continuous optimization, Inf. Sci. 450 (2018) 246e266.
  14. A. Saad, S.A. Khan, A. Mahmood, A multi-objective evolutionary artificial bee colony algorithm for optimizing network topology design, Swarm Evol. Comput. 38 (2018) 187e201.
  15. A. Kaur, K. Kumar, A reinforcement learning based evolutionary multiobjective optimization algorithm for spectrum allocation in cognitive radio networks, Phys. Commun. 43 (2020), 101196. https://doi.org/10.1016/j.phycom.2020.101196
  16. H. Samma, C.P. Lim, J.M. Saleh, A new reinforcement learning-based memetic particle swarm optimizer, Appl. Soft Comput. 43 (2016) 276-297. https://doi.org/10.1016/j.asoc.2016.01.006
  17. M.I. Radaideh, K. Shirvan, Rule-based reinforcement learning methodology to inform evolutionary algorithms for constrained optimization of engineering applications, Knowl. Base Syst. 217 (2021), 106836. https://doi.org/10.1016/j.knosys.2021.106836
  18. M.M. Drugan, Reinforcement learning versus evolutionary computation: a survey on hybrid algorithms, Swarm Evol. Comput. 44 (2019) 228-246. https://doi.org/10.1016/j.swevo.2018.03.011
  19. C.M. del Campo, J. Francois, H. Lopez, Axial: a system for boiling water reactor fuel assembly axial optimization using genetic algorithms, Ann. Nucl. Energy 28 (16) (2001) 1667-1682. https://doi.org/10.1016/S0306-4549(01)00002-0
  20. J. Francois, C. Martin-del Campo, R. Francois, L. Morales, A practical optimization procedure for radial bwr fuel lattice design using tabu search with a multiobjective function, Ann. Nucl. Energy 30 (12) (2003) 1213-1229. https://doi.org/10.1016/S0306-4549(03)00055-0
  21. T. Rogers, J. Ragusa, S. Schultz, R.S. Clair, Optimization of pwr fuel assembly radial enrichment and burnable poison location based on adaptive simulated annealing, Nucl. Eng. Des. 239 (6) (2009) 1019-1029. https://doi.org/10.1016/j.nucengdes.2009.02.005
  22. A. Charles, G. Parks, Application of differential evolution algorithms to multiobjective optimization problems in mixed-oxide fuel assembly design, Ann. Nucl. Energy 127 (2019) 165-177. https://doi.org/10.1016/j.anucene.2018.12.002
  23. F. Khoshahval, A. Zolfaghari, H. Minuchehr, M. Abbasi, A new hybrid method for multi-objective fuel management optimization using parallel pso-sa, Prog. Nucl. Energy 76 (2014) 112-121. https://doi.org/10.1016/j.pnucene.2014.05.014
  24. A. Zameer, S.M. Mirza, N.M. Mirza, Core loading pattern optimization of a typical two-loop 300 mwe pwr using simulated annealing (sa), novel crossover genetic algorithms (ga) and hybrid ga (sa) schemes, Ann. Nucl. Energy 65 (2014) 122-131. https://doi.org/10.1016/j.anucene.2013.10.024
  25. O. Safarzadeh, A. Zolfaghari, M. Zangian, O. Noori-kalkhoran, Pattern optimization of pwr reactor using hybrid parallel artificial bee colony, Ann. Nucl. Energy 63 (2014) 295-301. https://doi.org/10.1016/j.anucene.2013.08.011
  26. E. Schlunz, P.M. Bokov, J.H. van Vuuren, Multiobjective in-core nuclear fuel management optimisation by means of a hyperheuristic, Swarm Evol. Comput. 42 (2018) 58-76. https://doi.org/10.1016/j.swevo.2018.02.019
  27. G.T. Parks, Multiobjective pressurized water reactor reload core design by nondominated genetic algorithm search, Nucl. Sci. Eng. 124 (1) (1996) 178-187. https://doi.org/10.13182/NSE96-A24233
  28. V. Mnih, K. Kavukcuoglu, D. Silver, A.A. Rusu, J. Veness, M.G. Bellemare, A. Graves, M. Riedmiller, A.K. Fidjeland, G. Ostrovski, et al., Human-level control through deep reinforcement learning, Nature 518 (7540) (2015) 529-533. https://doi.org/10.1038/nature14236
  29. T. Schaul, J. Quan, I. Antonoglou, D. Silver, Prioritized Experience Replay, arXiv preprint arXiv:1511.05952 (2015).
  30. D.W. Corne, J.D. Knowles, M.J. Oates, The pareto envelope-based selection algorithm for multiobjective optimization, in: International Conference on Parallel Problem Solving from Nature, Springer, 2000, pp. 839-848.
  31. J. Kennedy, R. Eberhart, Particle swarm optimization, in: Proceedings of ICNN'95-International Conference on Neural Networks vol. 4, IEEE, 1995, pp. 1942-1948.
  32. H.-G. Beyer, H.-P. Schwefel, Evolution strategiesea comprehensive introduction, Nat. Comput. 1 (1) (2002) 3-52. https://doi.org/10.1023/A:1015059928466
  33. S. Kirkpatrick, C.D. Gelatt, M.P. Vecchi, Optimization by simulated annealing, Science 220 (4598) (1983) 671-680. https://doi.org/10.1126/science.220.4598.671
  34. R. Li, M.T. Emmerich, J. Eggermont, T. Back, M. Schutz, J. Dijkstra, J.H. Reiber, Mixed integer evolution strategies for parameter optimization, Evol. Comput. 21 (1) (2013) 29-64. https://doi.org/10.1162/EVCO_a_00059
  35. M. Clerc, J. Kennedy, The particle swarm-explosion, stability, and convergence in a multidimensional complex space, IEEE Trans. Evol. Comput. 6 (1) (2002) 58-73. https://doi.org/10.1109/4235.985692
  36. Y. Shi, R. Eberhart, A modified particle swarm optimizer, in: 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No. 98TH8360), IEEE, 1998, pp. 69-73.
  37. M.S. Arumugam, M. Rao, On the performance of the particle swarm optimization algorithm with various inertia weight variants for computing optimal control of a class of hybrid systems, Discrete Dynam Nat. Soc. 2006 (2006), 079295.
  38. J. Kennedy, R.C. Eberhart, A discrete binary version of the particle swarm algorithm, in: 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation vol. 5, IEEE, 1997, pp. 4104-4108.
  39. L.V. Santana-Quintero, A.A. Montano, C.A.C. Coello, A review of techniques for handling expensive functions in evolutionary multi-objective optimization, in: Computational Intelligence in Expensive Optimization Problems, Springer, 2010, pp. 29-59.
  40. Z. Zhou, Y.S. Ong, P.B. Nair, Hierarchical surrogate-assisted evolutionary optimization framework, in: Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No. 04TH8753) vol. 2, IEEE, 2004, pp. 1586-1593.
  41. M.A. El-Beltagy, A.J. Keane, Evolutionary optimization for computationally expensive problems using Gaussian processes, in: Proc. Int. Conf. On Artificial Intelligence vol. 1, Citeseer, 2001, pp. 708-714.
  42. L. Willmes, T. Back, Y. Jin, B. Sendhoff, Comparing neural networks and kriging for fitness approximation in evolutionary optimization, in: The 2003 Congress on Evolutionary Computation vol. 1, IEEE, 2003, pp. 663-670.
  43. N.H. Awad, M.Z. Ali, P.N. Suganthan, J.J. Liang, B.Y. Qu, Problem Definitions and Evaluation Criteria for the Cec 2017 Special Session and Competition on Single Objective Real-Parameter Numerical Optimization, Nanyang Technological University, Singapore, 2017. Tech. rep.
  44. M. Edenius, K. Ekberg, B. H. Forssen, D. Knott, Casmo-4, a Fuel Assembly Burnup Program, User's Manual, Studsvik-SOA-9501, Studsvik of America, Inc.
  45. M.I. Radaideh, B. Forget, K. Shirvan, Large-scale design optimisation of boiling water reactor bundles with neuroevolution, Ann. Nucl. Energy 160 (2021), 108355. https://doi.org/10.1016/j.anucene.2021.108355
  46. S. Challita, F. Zalila, C. Gourdin, P. Merle, A precise model for google cloud platform, in: 2018 IEEE International Conference on Cloud Engineering (IC2E), IEEE, 2018, pp. 177-183.
  47. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization Algorithms, 2017 arXiv preprint arXiv:1707.06347.
  48. A. Hill, A. Raffin, M. Ernestus, A. Gleave, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, et al., Stable Baselines, GitHub Repository.
  49. M.I. Radaideh, K. Du, P. Seurin, D. Seyler, X. Gu, H. Wang, K. Shirvan, NEORL: NeuroEvolution Optimization with Reinforcement Learning, arXiv preprint, 2021. arXiv:2112.07057.