Browse > Article
http://dx.doi.org/10.9766/KIMST.2021.24.5.558

Reinforcement Learning based on Deep Deterministic Policy Gradient for Roll Control of Underwater Vehicle  

Kim, Su Yong (Maritime Technology Research Institute, Agency for Defense Development)
Hwang, Yeon Geol (Maritime Technology Research Institute, Agency for Defense Development)
Moon, Sung Woong (Maritime Technology Research Institute, Agency for Defense Development)
Publication Information
Journal of the Korea Institute of Military Science and Technology / v.24, no.5, 2021 , pp. 558-568 More about this Journal
Abstract
The existing underwater vehicle controller design is applied by linearizing the nonlinear dynamics model to a specific motion section. Since the linear controller has unstable control performance in a transient state, various studies have been conducted to overcome this problem. Recently, there have been studies to improve the control performance in the transient state by using reinforcement learning. Reinforcement learning can be largely divided into value-based reinforcement learning and policy-based reinforcement learning. In this paper, we propose the roll controller of underwater vehicle based on Deep Deterministic Policy Gradient(DDPG) that learns the control policy and can show stable control performance in various situations and environments. The performance of the proposed DDPG based roll controller was verified through simulation and compared with the existing PID and DQN with Normalized Advantage Functions based roll controllers.
Keywords
Underwater Vehicle; Roll Control; Actor-Critic; Deep Deterministic Policy Gradient;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 H. v. Hasselt, et. al., "Deep Reinforcement Learning with Double Q-learning," AAAI, Vol. 16, 2016.
2 T. P. Lillicrap, et al. "Continuous Control with Deep Reinforcement Learning," International Conference on Learning Representations(ICLR), 2016.
3 B. Lee, "Roll control of Underwater Vehicle based Reinforcement Learning using Advantage ActorCritic," Journal of the Korea Institute of Military Science and Technology, Vol. 24, No. 1, pp. 123-132, 2021.   DOI
4 Silver, David, et al. "Deterministic Policy Gradient Algorithms," Proceedings of the 31st International Conference on Machine Learning(ICML-14), 2014.
5 S. Y. Kim, et. al., "Neural Network for a roll Control of the Underwater Vehicle," KIMST Annual Conference Proceedings, pp. 14-15, 2018.
6 J. Heo, et. al., "Technology Development of Unmanned Underwater Vehicles (UUVs)," Journal of Computer and Communications, Vol. 5, No. 7, pp. 28-35, 2017.   DOI
7 K. Y. Jung, et. al., "Autopilot Design of an Autonomous Underwater Vehicle using Robust Control," Transaction on Control Automation, and Systems Engineering, Vol. 4, No. 4, pp. 264-269, 2002.
8 J.-Y. Park, et. al., "Depth Controller Design for Submerged Body Moving near Free Surface Based on Adaptive Control," Journal of Ocean Engineering and Technology, Vol. 29, No. 3, pp. 270-282, 2015.   DOI
9 S. Y. Kim, et. al., "Reinforcement Learning for a Roll Control of the Unmanned Underwater Vehicle," Naval Ship Technology & Weapon Systems Seminar Proceedings, pp. 474-477, 2019.
10 H.-J. Chae, et. al., "Time-varying Proportional Navigation Guidance using Deep Reinforcement Learning," Journal of the Korea Institute of Military Science and Technology, Vol. 23, No. 4, pp. 399-406, 2020.   DOI
11 V. Mnih, et. al., "Playing Atari with Deep Reinforcement Learning," In NIPS Deep Learning Workshop, 2013.
12 W. W. Lee, et. al., "Reinforcement Learning with Python and Keras," Wikibook, pp. 225-277, 2020.
13 Z. Wang, et. al., "Dueling Network Architectures for Deep Reinforcement Learning," Proceedings of The 33rd International Conference on Machine Learning, 2016.
14 S. Y. Kim, et. al., "The roll control of Unmanned Underwater Vehicle using Double Deep-Q Network Reinforcement Learning," KIMST Annual Conference Proceedings, pp. 1601-1620, 2020.
15 R. S. Sutton, and A. G. Barto, "Reinforcement Learning: An Introduction," The MIT Press, pp. 328-333, 2018.
16 B. C. Kuo, "Automatic Control Systems," Prentice Hall, 1994.
17 H. J. Cho, et. al., "A Two-Stage Initial Alignment Technique for Underwater Vehicles Dropped from a Mother Ship," International Journal of Precision Engineering and Manufacturing, Vol. 14, No. 12, pp. 2067-2073, 2013.   DOI
18 S. Gu, et. al., "Continuous Deep Q-Learning with Model-based Acceleration," International Journal of Precision Engineering and Manufacturing, Vol. 14, No. 12, pp. 2067-2073, 2013.   DOI
19 A. Nair, et. al., "Massively Parallel Methods for Deep Reinforcement Learning," In ICML Deep Learning Workshop, 2015.