Browse > Article
http://dx.doi.org/10.12673/jant.2022.26.4.226

A slide reinforcement learning for the consensus of a multi-agents system  

Yang, Janghoon (Department of AI Software Engineering, Seoul Media Institute of Technology)
Publication Information
Journal of Advanced Navigation Technology / v.26, no.4, 2022 , pp. 226-234 More about this Journal
Abstract
With advances in autonomous vehicles and networked control, there is a growing interest in the consensus control of a multi-agents system to control multi-agents with distributed control beyond the control of a single agent. Since consensus control is a distributed control, it is bound to have delay in a practical system. In addition, it is often difficult to have a very accurate mathematical model for a system. Even though a reinforcement learning (RL) method was developed to deal with these issues, it often experiences slow convergence in the presence of large uncertainties. Thus, we propose a slide RL which combines the sliding mode control with RL to be robust to the uncertainties. The structure of a sliding mode control is introduced to the action in RL while an auxiliary sliding variable is included in the state information. Numerical simulation results show that the slide RL provides comparable performance to the model-based consensus control in the presence of unknown time-varying delay and disturbance while outperforming existing state-of-the-art RL-based consensus algorithms.
Keywords
Consensus; Delay; Multi-agents system; Reinforcement Learning; Sliding mode control;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Q. Zhang, Y. Niu, L. Wang, L. Shen, and H. Zhu, "Average consensus seeking of high-order continuous-time multi-agent systems with multiple time-varying communication delays," Int. J. Control Autom. Syst., Vol. 9, pp. 12090-1218, Dec. 2011.
2 Y.-J. Sun, G.-L. Zhang, and J. Zeng, "Consensus analysis for a class of heterogeneous multiagent systems with time delay based on frequency domain method," Math. Problems Eng., Vol. 2014, pp. 1-7, Sep. 2014.
3 D. Zhang, L. Liu, and G. Feng, "Consensus of Heterogeneous Linear Multiagent Systems Subject to Aperiodic Sampled-Data and DoS Attack," IEEE Transactions on Cybernetics, Vol. 49, No. 4, pp. 1501 - 1511, April 2019.   DOI
4 O. Vinyals, I. Babuschkin, W. M. Czarnecki, et al., "Grandmaster level in StarCraft II using multi-agent reinforcement learning," Nature, Vol. 575, pp. 350-354, 2019.   DOI
5 S. Gu, E. Holly, T. Lillicrap and S. Levine, "Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates," in Proceeding of IEEE International Conference on Robotics and Automation (ICRA), Singapore, pp. 3389-3396, July 2017.
6 H. Zhang, H. Jiang, Y. Luo and G. Xiao, "Data-Driven Optimal Consensus Control for Discrete-Time Multi-Agent Systems With Unknown Dynamics Using Reinforcement Learning Method," IEEE Transactions on Industrial Electronics, Vol. 64, No. 5, pp. 4091-4100, May 2017.   DOI
7 J. Li, L. Ji, and H. Li, "Optimal consensus control for unknown second-order multi-agent systems: Using model-free reinforcement learning method." Applied Mathematics and Computation, Vol. 410, pp. 1-15, Dec. 2021.
8 J. Yang, "Deep Learning-Based Consensus Control of a Multi-Agents System with Unknown Time-varying Delay," Electronics, Vol. 11, No. 8, pp. 1-15, Apr. 2022.
9 W. Dong, C. Wang, J. Li and J. Wang, "Graphical Minimax Game and On-Policy Reinforcement Learning for Consensus of Leaderless Multi-Agent Systems," in Proceeding of the 16th International Conference on Control & Automation (ICCA), Singapore, pp. 606-611, Oct. 2020.
10 J. Yang, "Reinforcement Learning for the Consensus of Multi-agents with Unknown Time Varying Delays," Journal of Digital Contents Society, Vol. 23, No. 7, pp. 1,277 -1,287, July 2022.
11 M. Li, X. Gao, Y. Wen, J. Si and H. H. Huang, "Offline Policy Iteration Based Reinforcement Learning Controller for Online Robotic Knee Prosthesis Parameter Tuning," in Proceeding of the International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, pp. 2831-2837, May 2019.
12 J. B. Kim, H. K. Lim, C. M. Kim, M. S. Kim, Y. G. Hong and Y. H. Han, "Imitation Reinforcement Learning-Based Remote Rotary Inverted Pendulum Control in OpenFlow Network," IEEE Access, Vol. 7, pp. 36682-36690, 2019.   DOI
13 A. Khalil, and J. Wang, "Stability and Time Delay Tolerance Analysis Approach for Networked Control Systems," Mathematical Problems in Engineering, Vol. 2015, pp.1-9, 2015.
14 Y. Liu, T. Li, Q. Shan, R. Yu, Y. Wu, C.L.P. Chen, "Online optimal consensus control of unknown linear multi-agent systems via time-based adaptive dynamic programming," Neurocomputing, Vol. 404, pp. 137-144, Sept. 2020.   DOI
15 M. Colombino, D. Gross and F. Dorfler, "Global phase and voltage synchronization for power inverters: A decentralized consensus-inspired approach," in Proceeding of the 56th Annual Conference on Decision and Control (CDC), Melbourne, VIC, Australia, pp. 5690-5695, Dec. 2017.
16 H. Septanto, B. Riyanto-Trilaksono, A. Syaichu-Rohman and R. Eko-Poetro, "Consensus-based controllers for spacecraft attitude alignment: Simulation results," in Proceeding of the 2nd International Conference on Instrumentation, Communications, Information Technology, and Biomedical Engineering, Bandung, Indonesia, pp. 52-57, Dec. 2011.
17 Z. Zhang, Z. Li, and Y. Gao, "Consensus reaching for group decision making with multi-granular unbalanced linguistic information: A bounded confidence and minimum adjustment-based approach," Information Fusion, Vol. 74, pp. 96-110, Oct. 2021.   DOI
18 R. Olfati-Saber and R. M. Murray, "Consensus problems in networks of agents with switching topology and time-delays," IEEE Trans. Autom. Control, Vol. 49, No. 9, pp. 1520-1533, Sep. 2004.   DOI
19 J. Yang, "A Consensus Control for a Multi-Agent System With Unknown Time-Varying Communication Delays," IEEE Access, Vol. 9, pp. 55844-55852, 2021.   DOI
20 X. Wang, and H. Su, "Completely model-free RL-based consensus of continuous-time multi-agent systems," Applied Mathematics and Computation, Vol. 382, pp. 1-11, Oct. 2020.
21 J. Zhang, H. Zhang and T. Feng, "Distributed Optimal Consensus Control for Nonlinear Multiagent System With Unknown Dynamic," IEEE Transactions on Neural Networks and Learning Systems, Vol. 29, No. 8, pp. 3339-3348, Aug. 2018.   DOI
22 R. Olfati-Saber, J. A. Fax and R. M. Murray, "Consensus and Cooperation in Networked Multi-Agent Systems," Proceedings of the IEEE, Vol. 95, No. 1, pp. 215-233, Jan. 2007.   DOI
23 V. Trianni, D. De Simone, A. Reina and A. Baronchelli, "Emergence of Consensus in a Multi-Robot Network: From Abstract Models to Empirical Validation," IEEE Robotics and Automation Letters, Vol. 1, No. 1, pp. 348-353, Jan. 2016.   DOI
24 B. Kim and H. Ahn, "Distributed Coordination and Control for a Freeway Traffic Network Using Consensus Algorithms," IEEE Systems Journal, Vol. 10, No. 1, pp. 162-168, March 2016.   DOI
25 A. T. Chin Loon and M. N. Mahyuddin, "Network server load balancing using consensus-based control algorithm," in Proceeding of the IEEE Industrial Electronics and Applications Conference (IEACon), Kota Kinabalu, Malaysia, pp. 291-296, Nov. 2016.
26 Y. Cao and W. Ren, "Optimal Linear-Consensus Algorithms: An LQR Perspective," IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), Vol. 40, No. 3, pp. 819-830, June 2010.   DOI
27 D. H. Nguyen, "A sub-optimal consensus design for multi-agent systems based on hierarchical LQR," Automatica, Vol. 55, pp. 88-94, May 2015.   DOI