Journal of the Society of Naval Architects of Korea (대한조선학회논문집)

The Society of Naval Architects of Korea (대한조선학회)
권호 표지
DOI QR코드

DOI QR Code

Determination of Ship Collision Avoidance Path using Deep Deterministic Policy Gradient Algorithm

심층 결정론적 정책 경사법을 이용한 선박 충돌 회피 경로 결정

  • Kim, Dong-Ham (Department of Naval Architecture and Ocean Systems Engineering, Graduate School, Korea Maritime and Ocean University) ;
  • Lee, Sung-Uk (Division of Naval Architecture and Ocean Systems Engineering, Korea Maritime and Ocean University) ;
  • Nam, Jong-Ho (Division of Naval Architecture and Ocean Systems Engineering, Korea Maritime and Ocean University) ;
  • Furukawa, Yoshitaka (Department of Marine Systems Engineering, Kyushu University,)
  • 김동함 (한국해양대학교 조선해양시스템공학과) ;
  • 이성욱 (한국해양대학교 조선해양시스템공학부) ;
  • 남종호 (한국해양대학교 조선해양시스템공학부) ;
  • 요시타카 후루카와 (큐슈대학교 마린시스템공학과)
  • Received : 2018.07.16
  • Accepted : 2018.10.19
  • Published : 2019.02.20
Download PDF
⟨ Previous Next ⟩

Abstract

The stability, reliability and efficiency of a smart ship are important issues as the interest in an autonomous ship has recently been high. An automatic collision avoidance system is an essential function of an autonomous ship. This system detects the possibility of collision and automatically takes avoidance actions in consideration of economy and safety. In order to construct an automatic collision avoidance system using reinforcement learning, in this work, the sequential decision problem of ship collision is mathematically formulated through a Markov Decision Process (MDP). A reinforcement learning environment is constructed based on the ship maneuvering equations, and then the three key components (state, action, and reward) of MDP are defined. The state uses parameters of the relationship between own-ship and target-ship, the action is the vertical distance away from the target course, and the reward is defined as a function considering safety and economics. In order to solve the sequential decision problem, the Deep Deterministic Policy Gradient (DDPG) algorithm which can express continuous action space and search an optimal action policy is utilized. The collision avoidance system is then tested assuming the $90^{\circ}$intersection encounter situation and yields a satisfactory result.

Keywords

References

  1. Bergstra, J. & Bengio, Y., 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13, pp.281-305.
  2. Furukawa, Y., Kijima, K. & Ibaragi, H., 2004. Development of automatic course modification system using fuzzy inference. International Federation of Automatic Control Proceedings, 37(10), pp.77-82.
  3. Jeong, J.S., 2015. Korean e-Navigation goal and government plan. Telecommunications Technology Association Journal, 159, pp.20-27.
  4. Kijima, K. & Furukawa, Y., 2002. Development of collision avoidance algorithm using fuzzy inference. Proceedings of ISOPE Pacific/Asia Offshore Mechanics Symposium, pp.123-130.
  5. Kijima, K. & Furukawa, Y., 2003. Automatic collision avoidance system using the concept of blocking area. International Federation of Automatic Control Proceedings, 36(21), pp.223-228.
  6. Kijima, K. & Nakiri, Y., 2003. On the practical prediction method for ship manoeuvring characteristics. Transaction of the West-Japan Society of Naval Architects, 105, pp.21-31.
  7. Kim, D.J. & Kwak, S.Y., 2011. Evaluation of human factors in ship accidents in the domestic sea. Journal of the Ergonomics Society of Korea, 30(1), pp.87-98. https://doi.org/10.5143/JESK.2011.30.1.87
  8. Korean Maritime Safety Tribunal, 2017, Current situation of causes of maritime accidents by type of accident [online] Available at: https://www.kmst.go.kr/kmst/statistics/annualReport/selectAnnualReportList.do [Accessed 21 May 2018].
  9. Kose, K., Hirono, K., Sugano, K. & Sato, I., 1998. A new collision-avoidance-supporting-system and its application to coastal-cargo-ship "SHOYO MARU". IFAC Proceeding, 31, 263-268.
  10. Lee, H.J. & Rhee, K.P., 2001. Development of collision avoidance system by using expert system and search algorithm. International Shipbuilding Progress, 48, pp.197-212.
  11. Lee, W.W., Yang, H.R., Kim, K.W., Lee, Y.M. & Lee, U.R., 2017. Reinforcement Learning with Python and Keras. Wikibook.
  12. Li, Y., 2017. Deep Reinforcement Learning: An Overview. arXiv preprint arXiv:1701.07274.
  13. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D. & Wierstra, D., 2016. Continuous control with deep reinforcement learning. International Conference on Learning Representations, 1509.02971.
  14. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. & Riedmiller, M., 2013. Playing Atari with Deep Reinforcement Learning. Neural Information Processing Systems, Lake Tahoe, USA, 9 December 2013.
  15. Mori, S., 1995. Note of Ship Form Design(24). FUNE-NO-KAGAKU, 48, pp.40-49.
  16. Ota, D., Masuyama, T., Furukawa, Y. & Ibaragi, H., 2016. Development of automatic collision avoidance system for ships using reinforcement learning. Proceedings of 7th PAAMES and AMEC2016, Hong Kong, 13-14 October 2016.
  17. Shim, W.S., Park, J.W. & Lim, Y.K., 2010. The study on the trend of international standards and the domestic plan to cope with e-navigation. Journal of the Korea Institute of Information and Communication Engineering, 14(5), pp.1057-1063. https://doi.org/10.6109/jkiice.2010.14.5.1057
  18. Silver, D., Huang, A., Maddison, C., Guez, A., Sifre, L., Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Graepel, T., Lillicrap T., Leach, M., Kavukcuoglu, K. & Hassabis, D., 2016. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587), pp.484-489. https://doi.org/10.1038/nature16961
  19. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D. & Riedmiller, M., 2014. Deterministic policy gradient algorithms. International Conference on Machine Learning, 32, pp.387-395.
  20. Son, N.S., Furukawa, Y., Kim, S.Y. & Kijima, K., 2009. Study on the collision avoidance algorithm against multiple traffic ships using changeable action space searching method. Journal of the Korean Society for Marine Environmental Engineering, 12(1), pp.15-22.
  21. Van, S.H., 2007. Planning research for development of core technologies for smart ship. KORDI Report No. UCPM0147A-42-7.