The Journal of Korea Robotics Society (로봇학회논문지)

Korea Robotics Society (한국로봇학회)
권호 표지
DOI QR코드

DOI QR Code

Online Adaptation of Control Parameters with Safe Exploration by Control Barrier Function

제어 장벽함수를 이용한 안전한 행동 영역 탐색과 제어 매개변수의 실시간 적응

  • Kim, Suyeong (Dept. of Mechanical Engineering, Ulsan National Institute of Science and Technology) ;
  • Son, Hungsun (Dept. of Mechanical Engineering, Ulsan National Institute of Science and Technology)
  • Received : 2022.01.03
  • Accepted : 2022.02.16
  • Published : 2022.02.28
Download PDF
⟨ Previous Next ⟩

Abstract

One of the most fundamental challenges when designing controllers for dynamic systems is the adjustment of controller parameters. Usually the system model is used to get the initial controller, but eventually the controller parameters must be manually adjusted in the real system to achieve the best performance. To avoid this manual tuning step, data-driven methods such as machine learning were used. Recently, reinforcement learning became one alternative of this problem to be considered as an agent learns policies in large state space with trial-and-error Markov Decision Process (MDP) which is widely used in the field of robotics. However, on initial training step, as an agent tries to explore to the new state space with random action and acts directly on the controller parameters in real systems, MDP can lead the system safety-critical system failures. Therefore, the issue of 'safe exploration' became important. In this paper we meet 'safe exploration' condition with Control Barrier Function (CBF) which converts direct constraints on the state space to the implicit constraint of the control inputs. Given an initial low-performance controller, it automatically optimizes the parameters of the control law while ensuring safety by the CBF so that the agent can learn how to predict and control unknown and often stochastic environments. Simulation results on a quadrotor UAV indicate that the proposed method can safely optimize controller parameters quickly and automatically.

Keywords

Acknowledgement

This project was partially funded by Development of Drone System for Ship and Marine Mission (2.200021.01) of Institute of Civil-Military Technology Cooperation, and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2020R1F1A1075857), respectively

References

  1. F. Berkenkamp, A. P. Schoellig, and A. Krause, "Safe and automatic controller tuning with Gaussian processes," Workshop on Machine Learning in Planning and Control of Robot Motion, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015, [Online], https://www.dynsyslab.org/wp-content/papercite-data/pdf/berkenkamp-icra16.pdf.
  2. N. O. Lambert, D. S. Drew, J. Yaconelli, R. Calandra, S. Levine, and K. S. J. Pister, "Low level control of a quadrotor with deep model-based reinforcement learning," arXiv:1901.03737v2 [cs.RO], 2019, [Online], https://arxiv.org/pdf/1901.03737.pdf.
  3. F. Berkenkamp, M. Turchetta, A. P. Schoellig, and A. Krause, "Safe model-based reinforcement learning with stability guarantees," IEEE Transactions on Automatic Control, vol. 64, no. 7, Jul., 2017, DOI: 10.1109/TAC.2018.2876389.
  4. A.Y. Zomaya, "Reinforcement Learning to Adaptive Control of Nonlinear Systems," IEEE Transactions on Systems, Man, and Cybernetics, vol. 24, no. 2, Feb., 1994, DOI: 10.1109/21.281435.
  5. X.-S. Wang, Y.-H. Cheng, and W. Sun, "A proposal of adaptive PID controller based on reinforcement learning," Journal of China Univ. Mining and Technology, vol. 17, no. 1, 2007, [Online], http://www.paper.edu.cn/scholar/showpdf/MUT2MNzINTD0cx2h.
  6. M. N. Howell and M. C. Best, "On-line PID tuning for engine idle-speed control using continuous action reinforcement learning automata," Control Engineering Practice, vol. 8, no. 2, Feb., 2000, DOI: 10.1016/S0967-0661(99)00141-0.
  7. Z. S. Jin, H. C. Li, and H. M. Gao, "An intelligent weld control strategy based on reinforcement learning approach," The International Journal of Advanced Manufacturing Technology, Feb., 2019, DOI: 10.1007/s00170-018-2864-2.
  8. M. Sedighizadeh and A. Rezazadeh, "Adaptive PID Controller based on Reinforcement Learning for Wind Turbine Control," World Academy of Science, Engineering and Technology, 2008, DOI: 10.5281/zenodo.1057789.
  9. J. Achiam, D. Held, A. Tamar, and P. Abbeel, "Constrained policy optimization," arXiv:1705.10528v1 [cs.LG], 2017, [Online], https://arxiv.org/pdf/1705.10528.pdf.
  10. S. Gangapurwala, A. Mitchell, and I. Havoutis, "Guided constrained policy optimization for dynamic quadrupedal robot locomotion," IEEE Robotics and Automation Letters, vol. 5, no. 2, Apr., 2020, DOI: 10.1109/LRA.2020.2979656.
  11. F. Berkenkamp, A. P. Schoellig, and A. Krause, "Safe controller optimization for quadrotors with Gaussian processes," 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 2016, DOI: 10.1109/ICRA.2016.7487170.
  12. Y. Sui, A. Gotovos, J. W. Burdick, and A. Krause, "Safe exploration for optimization with Gaussian processes," 32nd International Conference on Machine Learning, 2015, [Online], http://proceedings.mlr.press/v37/sui15.pdf.
  13. A. K. Akametalu, J. F. Fisac, J. H. Gillula, S. Kaynama, M. N. Zeilinger, and C. J. Tomlin, "Reachability-based safe learning with Gaussian processes," 53rd IEEE Conference on Decision and Control, Los Angeles, CA, USA, 2014, DOI: 10.1109/CDC.2014.7039601.
  14. A. Aswani, H. Gonzalez, S. S. Sastry, and C. Tomlin, "Provably safe and robust learning-based model predictive control," Automatica, vol. 49, no. 5, May, 2013, DOI: 10.1016/j.automatica.2013.02.003.
  15. T. M. Moldovan and P. Abbeel, "Safe exploration in Markov decision processes," arXiv:1205.4810v3 [cs.LG], 2012, [Online], https://arxiv.org/pdf/1205.4810.pdf.
  16. A. M. Lyapunov, The General Problem of the Stability of Motion, Taylor and Francis Ltd, London, UK, 1992, [Online], https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.910.9566&rep=rep1&type=pdf.
  17. A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada, "Control barrier function based quadratic programs for safety critical systems," IEEE Transactions on Automatic Control, vol. 62, no. 8, Aug., 2017, DOI: 10.1109/TAC.2016.2638961.
  18. K. Galloway, K. Sreenath, A. D. Ames, and J. W. Grizzle, "Torque saturation in bipedal robotic walking through control Lyapunov function-based quadratic programs," IEEE Access, vol. 3, pp. 323-332, 2015, DOI: 10.1109/ACCESS.2015.2419630.
  19. A. D. Ames and M. Powell, "Towards the unification of locomotion and manipulation through control Lyapunov functions and quadratic programs," Control of Cyber-Physical Systems, vol. 449, 2013, DOI: 10.1007/978-3-319-01159-2_12.
  20. B. J. Morris, M. J. Powell, and A. D. Ames, "Continuity and smoothness properties of nonlinear optimization-based feedback controllers," 2015 54th IEEE Conference on Decision and Control (CDC), 2015, DOI: 10.1109/CDC.2015.7402101.
  21. A. Vahidi and A. Eskandarian, "Research advances in intelligent collision avoidance and adaptive cruise control," IEEE Trans. Intell. Transp. Syst., vol. 4, no. 3, pp. 143-153, Sep. 2003. DOI: 10.1109/TITS.2003.821292.
  22. S. Li, K. Li, R. Rajamani, and J. Wang, "Model predictive multiobjective vehicular adaptive cruise control," IEEE Transactions on Control Systems Technology, vol. 19, no. 3, pp. 556-566, 2011, DOI: 10.1109/TCST.2010.2049203.
  23. G. J. L. Naus, J. Ploeg, M. J. G. Van de Molengraft, W. P. M. H. Heemels, and M. Steinbuch, "Design and implementation of parameterized adaptive cruise control: An explicit model predictive control approach," Control Engineering Practice, vol. 18, no. 8, pp. 882-892, Aug., 2010, DOI: 10.1016/j.conengprac.2010.03.012.
  24. P. A. Ioannou and C. C. Chien, "Autonomous intelligent cruise control," IEEE Transactions on Vehicular Technology, vol. 42, no. 4, pp. 657-672, Nov., 1993, DOI: 10.1109/25.260745.
  25. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv:1707.06347v2 [cs.LG], 2017, [Online], https://arxiv.org/pdf/1707.06347.pdf.
  26. J.-M. Kai, G. Allibert, M.-D. Hua, and T. Hamel, "Nonlinear feedback control of quadrotors exploiting first-order drag effects," IFAC-PapersOnLine, Jul., 2017, DOI: 10.1016/j.ifacol. 2017.08.1267.
  27. E. D. Sontag, "A Lyapunov-like stabilization of asymptotic controllability," SIAM Journal of Control and Optimization, vol. 21, no. 3, 1983, DOI: 10.1137/0321028.
  28. E. Squires, P. Pierpaoli, and M. Egerstedt, "Constructive barrier certificates with applications to fixed-wing aircraft collision avoidance," 2018 IEEE Conference on Control Technology and Applications (CCTA), Aug., 2018, DOI: 10.1109/CCTA.2018.8511342.
  29. P. Auer. "Using confidence bounds for exploitation-exploration trade-offs," The Journal of Machine Learning Research, vol. 3, pp. 397-422, 2002, [Online], https://www.jmlr.org/papers/volume3/auer02a/auer02a.pdf.