Journal of the Korea Institute of Military Science and Technology (한국군사과학기술학회지)

The Korea Institute of Military Science and Technology (한국군사과학기술학회)
권호 표지
DOI QR코드

DOI QR Code

Reinforcement Learning based on Deep Deterministic Policy Gradient for Roll Control of Underwater Vehicle

수중운동체의 롤 제어를 위한 Deep Deterministic Policy Gradient 기반 강화학습

  • Kim, Su Yong (Maritime Technology Research Institute, Agency for Defense Development) ;
  • Hwang, Yeon Geol (Maritime Technology Research Institute, Agency for Defense Development) ;
  • Moon, Sung Woong (Maritime Technology Research Institute, Agency for Defense Development)
  • 김수용 (국방과학연구소 해양기술연구원) ;
  • 황연걸 (국방과학연구소 해양기술연구원) ;
  • 문성웅 (국방과학연구소 해양기술연구원)
  • Received : 2021.03.29
  • Accepted : 2021.07.16
  • Published : 2021.10.05
Download PDF
⟨ Previous Next ⟩

Abstract

The existing underwater vehicle controller design is applied by linearizing the nonlinear dynamics model to a specific motion section. Since the linear controller has unstable control performance in a transient state, various studies have been conducted to overcome this problem. Recently, there have been studies to improve the control performance in the transient state by using reinforcement learning. Reinforcement learning can be largely divided into value-based reinforcement learning and policy-based reinforcement learning. In this paper, we propose the roll controller of underwater vehicle based on Deep Deterministic Policy Gradient(DDPG) that learns the control policy and can show stable control performance in various situations and environments. The performance of the proposed DDPG based roll controller was verified through simulation and compared with the existing PID and DQN with Normalized Advantage Functions based roll controllers.

Keywords

References

  1. J. Heo, et. al., "Technology Development of Unmanned Underwater Vehicles (UUVs)," Journal of Computer and Communications, Vol. 5, No. 7, pp. 28-35, 2017. https://doi.org/10.4236/jcc.2017.57003
  2. K. Y. Jung, et. al., "Autopilot Design of an Autonomous Underwater Vehicle using Robust Control," Transaction on Control Automation, and Systems Engineering, Vol. 4, No. 4, pp. 264-269, 2002.
  3. J.-Y. Park, et. al., "Depth Controller Design for Submerged Body Moving near Free Surface Based on Adaptive Control," Journal of Ocean Engineering and Technology, Vol. 29, No. 3, pp. 270-282, 2015. https://doi.org/10.5574/KSOE.2015.29.3.270
  4. S. Y. Kim, et. al., "Neural Network for a roll Control of the Underwater Vehicle," KIMST Annual Conference Proceedings, pp. 14-15, 2018.
  5. S. Y. Kim, et. al., "Reinforcement Learning for a Roll Control of the Unmanned Underwater Vehicle," Naval Ship Technology & Weapon Systems Seminar Proceedings, pp. 474-477, 2019.
  6. H.-J. Chae, et. al., "Time-varying Proportional Navigation Guidance using Deep Reinforcement Learning," Journal of the Korea Institute of Military Science and Technology, Vol. 23, No. 4, pp. 399-406, 2020. https://doi.org/10.9766/KIMST.2020.23.4.399
  7. V. Mnih, et. al., "Playing Atari with Deep Reinforcement Learning," In NIPS Deep Learning Workshop, 2013.
  8. H. v. Hasselt, et. al., "Deep Reinforcement Learning with Double Q-learning," AAAI, Vol. 16, 2016.
  9. Z. Wang, et. al., "Dueling Network Architectures for Deep Reinforcement Learning," Proceedings of The 33rd International Conference on Machine Learning, 2016.
  10. S. Y. Kim, et. al., "The roll control of Unmanned Underwater Vehicle using Double Deep-Q Network Reinforcement Learning," KIMST Annual Conference Proceedings, pp. 1601-1620, 2020.
  11. B. Lee, "Roll control of Underwater Vehicle based Reinforcement Learning using Advantage ActorCritic," Journal of the Korea Institute of Military Science and Technology, Vol. 24, No. 1, pp. 123-132, 2021. https://doi.org/10.9766/KIMST.2021.24.1.123
  12. R. S. Sutton, and A. G. Barto, "Reinforcement Learning: An Introduction," The MIT Press, pp. 328-333, 2018.
  13. W. W. Lee, et. al., "Reinforcement Learning with Python and Keras," Wikibook, pp. 225-277, 2020.
  14. Silver, David, et al. "Deterministic Policy Gradient Algorithms," Proceedings of the 31st International Conference on Machine Learning(ICML-14), 2014.
  15. T. P. Lillicrap, et al. "Continuous Control with Deep Reinforcement Learning," International Conference on Learning Representations(ICLR), 2016.
  16. S. Gu, et. al., "Continuous Deep Q-Learning with Model-based Acceleration," International Journal of Precision Engineering and Manufacturing, Vol. 14, No. 12, pp. 2067-2073, 2013. https://doi.org/10.1007/s12541-013-0280-y
  17. H. J. Cho, et. al., "A Two-Stage Initial Alignment Technique for Underwater Vehicles Dropped from a Mother Ship," International Journal of Precision Engineering and Manufacturing, Vol. 14, No. 12, pp. 2067-2073, 2013. https://doi.org/10.1007/s12541-013-0280-y
  18. A. Nair, et. al., "Massively Parallel Methods for Deep Reinforcement Learning," In ICML Deep Learning Workshop, 2015.
  19. B. C. Kuo, "Automatic Control Systems," Prentice Hall, 1994.