Journal of Navigation and Port Research (한국항해항만학회지)

Korean Institute of Navigation and Port Research (한국항해항만학회)
권호 표지
DOI QR코드

DOI QR Code

A Study on Ship Route Generation with Deep Q Network and Route Following Control

  • Min-Kyu Kim (Korea Ocean Satellite Center, Korea Institute of Ocean Science & Technology) ;
  • Hyeong-Tak Lee (Korea Ocean Satellite Center, Korea Institute of Ocean Science & Technology)
  • Received : 2023.04.10
  • Accepted : 2023.04.18
  • Published : 2023.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

Ships need to ensure safety during their navigation, which makes route determination highly important. It must be accompanied by a route following controller that can accurately follow the route. This study proposes a method for automatically generating the ship route based on deep reinforcement learning algorithm and following it using a route following controller. To generate a ship route, under keel clearance was applied to secure the ship's safety and navigation chart information was used to apply ship navigation related regulations. For the experiment, a target ship with a draft of 8.23 m was designated. The target route in this study was to depart from Busan port and arrive at the pilot boarding place of the Ulsan port. As a route following controller, a velocity type fuzzy P ID controller that could compensate for the limitation of a linear controller was applied. As a result of using the deep Q network, a route with a total distance of 62.22 km and 81 waypoints was generated. To simplify the route, the Douglas-Peucker algorithm was introduced to reduce the total distance to 55.67 m and the number of way points to 3. After that, an experiment was conducted to follow the path generated by the target ship. Experiment results revealed that the velocity type fuzzy P ID controller had less overshoot and fast settling time. In addition, it had the advantage of reducing the energy loss of the ship because the change in rudder angle was smooth. This study can be used as a basic study of route automatic generation. It suggests a method of combining ship route generation with the route following control.

Keywords

Acknowledgement

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean Government (Ministry of Science and ICT) (No. 2022R1C1C2010897).

References

  1. Chen, C., Chen, X. Q., Ma, F., Zeng, X. J. and Wang, J. (2019). "A knowledge-free path planning approach for smart ships based on reinforcement learning", Ocean Engineering, Vol. 189, 106299. https://doi.org/10.1016/j.oceaneng.2019.106299
  2. Davidson, K. S. M. and Schiff, L. I.(1946), Turning and course keeping qualities transactions of SNAME.
  3. Guo, S., Zhang, X., Du, Y., Zheng, Y. and Cao, Z.(2021). "Path planning of coastal ships based on optimized DQN reward function", Journal of Marine Science and Engineering, Vol. 9, No. 2, 210.
  4. He, S., Tan, S., Xu, F. and Wang, P.(1993), "Fuzzy self-tuning of PID controllers", Fuzzy sets and systems, Vol. 56, pp. 37-46. https://doi.org/10.1016/0165-0114(93)90183-I
  5. Kim, C., Park, K. W., Kim, T. S. and Lee, M. K.(2009), "Study on model-based control for the roll motion of an underwater robot", Journal of navigation and port research, Vol. 33, No. 5, pp. 323-330. (in Korean) https://doi.org/10.5394/KINPR.2009.33.5.323
  6. Kim, D., Lee, C. and Yim, J.(2023). "An Evaluation Technique for the Path-following Control Performance of Autonomous Surface Ships", Journal of Navigation and Port Research, Vol. 47, No. 1, pp. 10-17. (in Korean) https://doi.org/10.5394/KINPR.2023.47.1.10
  7. Kim, H. T. and Yang, Y. H.(2019), "A review of human element issues of remote operators on maritime autonomous surface ships", Journal of navigation and port research, Vol. 43, No. 6, pp. 395-402. (in Korean) https://doi.org/10.5394/KINPR.2019.43.6.395
  8. Kim, J. H. and Oh, S. J.(2000), "A fuzzy PID controller for nonlinear and uncertain systems", Soft computing, Vol. 4, pp. 312-129.
  9. Kim, M. K., Kim, J. H. and Choi, J. K.(2018), "A fuzzy PID autopilot and path following using fuzzy disturbance estimator for a ship", Journal of the Korean society of marine engineering, Vol. 42, No. 1, pp. 49-57. (in Korean) https://doi.org/10.5916/jkosme.2018.42.1.49
  10. Kim, S., Jeon, H. and Y. K.(2021), "Simplified rule-based strategy control for electric propulsion system", Journal of advanced marine engineering and technology, Vol. 45, No. 6, pp. 380-385. https://doi.org/10.5916/jamet.2021.45.6.380
  11. Kwak, S. and Choi, B. J.(2018), "Defuzzification scheme and its numerical example for fuzzy logic based control system", Journal of korean institute of intelligent system, Vol. 28, No. 4, pp. 350-354. https://doi.org/10.5391/JKIIS.2018.28.4.350
  12. Lee, H. T., Choi, H. M., Lee, J. S., Yang, H. and Cho, I. S.(2022a). "Generation of Ship's Passage Plan Using Data-Driven Shortest Path Algorithms", IEEE Access, Vol. 10, pp. 126217-126231. https://doi.org/10.1109/ACCESS.2022.3225571
  13. Lee, J. S., Lee, H. T. and Cho, I. S.(2022b). "Maritime traffic route detection framework based on statistical density analysis from AIS data using a clustering algorithm", IEEE Access, Vol. 10, pp. 23355-23366. https://doi.org/10.1109/ACCESS.2022.3154363
  14. Lee, K. J., Lee, C. M. and Y, H. K.(2004), "Modeling and simulation of a ship with anti-rolling devices in waves", Journal of navigation and port research, Vol. 28, No. 4, pp. 347-352. (in Korean) https://doi.org/10.5394/KINPR.2004.28.5.347
  15. Lee, W., Yoo, W., Choi. G. H., Ham, S. H. and Kim, T. W.(2019), "Determination of optimal route in coastal sea considering sea state and under keel clearance", Journal of the society of naval architects of korea, Vol. 56, No. 6, pp. 480-487. (in Korean) https://doi.org/10.3744/SNAK.2019.56.6.480
  16. Lie, C., Mao, Q., Chu, X. and Xie, S.(2019), "A-star algorithm considering water current, traffic separation and berthing for vessel path planning", Applied science, Vol. 9, No. 6.
  17. Logan, C., Jeff, C., Yisong, Y. and John, P. O.(2021), "Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments", Neuron, Vol. 109, No. 4, pp. 724-738. https://doi.org/10.1016/j.neuron.2020.11.021
  18. Nam, S. K. and Yoo, W. S.(1994), "Fuzzy PID control with accelerated reasoning for DC servo motors", Engineering application of artificial intelligent, Vol. 7, pp. 559-569. https://doi.org/10.1016/0952-1976(94)90034-5
  19. Novac, V., Rusu, E., Gasparotti, C. and Stavarache, G. (2020). "Ship routing in the black sea based on Dijkstra algorithm", In Proc. SGEM Int. Multidisciplinary Sci. GeoConf. EXPO, September 2020, pp. 301-308.
  20. Orsic, J. P., Vettor, R., Faltinsen, O. M. and Soares, C. G.(2016), "The influence of route choice and operating conditions on fuel consumption and CO2 emission of ships", Journal of marine science and technology, Vol. 20, pp. 434-457. https://doi.org/10.1007/s00773-015-0367-5
  21. Ozturk, U. G., Akdag, M. and Ayabakan, T.(2022), "A review of path planning algorithms in maritime autonomous surface ships: navigation safety perspective", Ocean Engineering, Vol. 251.
  22. Roh, M. I.(2013), "Determination of an economical shipping route considering the effects of sea state for lower fuel consumption", International journal of naval architecture and ocean engineering, Vol. 5, No. 2, pp. 246-262. https://doi.org/10.2478/IJNAOE-2013-0130
  23. Roland, S. B.(1992), "An intelligent integrated ship guidance system", IFAC proceedings volumes, Vol. 25, No. 3, pp. 199-208. https://doi.org/10.1016/S1474-6670(17)50292-X
  24. Wang, H., Mao, W. and Eriksson.(2019), "A three-dimensional Dijkstra's algorithm for multi-objective ship voyage optimization", Ocean engineering, Vol. 186.
  25. Zhao, L. and Shi, G.(2018), "A method for simplifying ship trajectory based on improves Douglas-peucker algorithms", Ocean engineering, Vol. 166, pp. 37-46. https://doi.org/10.1016/j.oceaneng.2018.08.005