• Journal of the Korea Institute of Military Science and Technology
• /
• v.24 no.1
• /
• pp.123-132
• /
• 2021

In order for the underwater vehicle to perform various tasks, it is important to control the depth, course, and roll of the underwater vehicle. To design such a controller, it is necessary to construct a dynamic model of the underwater vehicle and select the appropriate hydrodynamic coefficients. For the controller design, since the dynamic model is linearized assuming a limited operating range, the control performance in the steady state is well satisfied, but the control performance in the transient state may be unstable. In this paper, in order to overcome the problems of the existing controller design, we propose a A2C(Advantage Actor-Critic) based roll controller for underwater vehicle with stable learning performance in a continuous space among reinforcement learning methods that can be learned through rewards for actions. The performance of the proposed A2C based roll controller is verified through simulation and compared with PID and Dueling DDQN based roll controllers.

• International Journal of Internet, Broadcasting and Communication
• /
• v.16 no.3
• /
• pp.205-211
• /
• 2024

Effective warehouse management requires advanced resource planning to optimize profits and space. Robots offer a promising solution, but their effectiveness relies on embedded artificial intelligence. Multi-agent reinforcement learning (MARL) enhances robot intelligence in these environments. This study explores various MARL algorithms using the Multi-Robot Warehouse Environment (RWARE) to determine their suitability for warehouse resource planning. Our findings show that cooperative MARL is essential for effective warehouse management. IA2C outperforms MAA2C and VDA2C on smaller maps, while VDA2C excels on larger maps. IA2C's decentralized approach, focusing on cooperation over collaboration, allows for higher reward collection in smaller environments. However, as map size increases, reward collection decreases due to the need for extensive exploration. This study highlights the importance of selecting the appropriate MARL algorithm based on the specific warehouse environment's requirements and scale.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2019.05a
• /
• pp.493-495
• /
• 2019

제어 분야의 가장 기초적인 시스템인 Rotary Inverted Pendulum 을 제어하기 위하여, 본 논문에서는 강화학습에서 Deep Q-Network 과 함께 대표적인 알고리즘으로 알려진 Asynchronous Advantage Actor-Critic 을 활용하여 다중 디바이스 제어를 설계한다. Deep Q-Network 알고리즘을 활용한 기존 연구와 동일한 방식으로 실 세계의 물리 에이전트와 가상 환경을 맵핑시키며, 스위치를 통하여 로컬 에이전트와 글로벌 네트워크 간 통신을 구성한다. 본 논문에서는 분산 Asynchronous Advantage Actor-Critic 을 이용함으로써 실 세계의 다중 에이전트 제어를 위한 강화 학습의 활용 가능성을 조명한다.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2022.05a
• /
• pp.45-47
• /
• 2022

Mobile edge computing (MEC), which enables mobile terminals to offload computational tasks to a server located at the user's edge, is considered an effective way to reduce the heavy computational burden and achieve efficient computational offloading. In this paper, we study a multi-user MEC system in which multiple user devices (UEs) can offload computation to the MEC server via a wireless channel. To solve the resource allocation and task offloading problem, we take the total cost of latency and energy consumption of all UEs as our optimization objective. To minimize the total cost of the considered MEC system, we propose an DRL-based method to solve the resource allocation problem in wireless MEC. Specifically, we propose a Asynchronous Advantage Actor-Critic (A3C)-based scheme. Asynchronous Advantage Actor-Critic (A3C) is applied to this framework and compared with DQN, and Double Q-Learning simulation results show that this scheme significantly reduces the total cost compared to other resource allocation schemes

• International Journal of Internet, Broadcasting and Communication
• /
• v.16 no.3
• /
• pp.192-198
• /
• 2024

This paper investigates the application of multi-agent deep reinforcement learning in the fighting game Samurai Shodown using Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C) algorithms. Initially, agents are trained separately for 200,000 timesteps using Convolutional Neural Network (CNN) and Multi-Layer Perceptron (MLP) with LSTM networks. PPO demonstrates superior performance early on with stable policy updates, while A2C shows better adaptation and higher rewards over extended training periods, culminating in A2C outperforming PPO after 1,000,000 timesteps. These findings highlight PPO's effectiveness for short-term training and A2C's advantages in long-term learning scenarios, emphasizing the importance of algorithm selection based on training duration and task complexity. The code can be found in this link https://github.com/Lexer04/Samurai-Shodown-with-Reinforcement-Learning-PPO.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.14 no.11
• /
• pp.4268-4289
• /
• 2020

Advanced traffic signal timing method plays very important role in reducing road congestion and air pollution. Reinforcement learning is considered as superior approach to build traffic light timing scheme by many recent studies. It fulfills real adaptive control by the means of taking real-time traffic information as state, and adjusting traffic light scheme as action. However, existing works behave inefficient in complex intersections and they are lack of feasibility because most of them adopt traffic light scheme whose phase sequence is flexible. To address these issues, a novel adaptive traffic signal timing scheme is proposed. It's based on actor-critic reinforcement learning algorithm, and advanced techniques proximal policy optimization and generalized advantage estimation are integrated. In particular, a new kind of reward function and a simplified form of state representation are carefully defined, and they facilitate to improve the learning efficiency and reduce the computational complexity, respectively. Meanwhile, a fixed phase sequence signal scheme is derived, and constraint on the variations of successive phase durations is introduced, which enhances its feasibility and robustness in field applications. The proposed scheme is verified through field-data-based experiments in both medium and high traffic density scenarios. Simulation results exhibit remarkable improvement in traffic performance as well as the learning efficiency comparing with the existing reinforcement learning-based methods such as 3DQN and DDQN.

• The Journal of the Convergence on Culture Technology
• /
• v.7 no.4
• /
• pp.807-814
• /
• 2021

As economic activities decrease, and the stock market decline due to COVID-19, many people are jumping into stock investment as an alternative source of income. As people's interest increases, many stock price analysis studies are underway to earn more profits. Due to the variance observed in the stock markets, it is necessary to analyze each stock independently and consistently. To solve this problem, we designed and implemented models and services that analyze stock prices using a reinforcement learning technique called Asynchronous Advantage Actor-Critic(A3C). Stock market data reflected external factors such as government bonds and KOSPI (Korea Composite Stock Price Index) as well as stock prices. Our proposed work provides a web service with a visual representation of predictions of stocks and stock information through which directions are given to investors to make safe investments without analyzing domestic and foreign stock market trends.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2019.05a
• /
• pp.545-547
• /
• 2019

Policy Gradient 방식의 학습은 최근 강화학습 분야에서 많이 연구되고 있는 주제로, 본 논문에서는 강화학습을 적용시킬 수 있는 OpenAi Gym 의 'CartPole-v0' 와 'Pendulum-v0' 환경에서 Policy Gradient 방식의 Asynchronous Advantage Actor-Critic (A3C) 알고리즘과 Proximal Policy Optimization (PPO) 알고리즘의 학습 성능을 비교 분석한 결과를 제시한다. 딥러닝 모델 등 두 알고리즘이 동일하게 지닐 수 있는 조건들은 가능한 동일하게 맞추면서 Episode 진행에 따른 Score 변화 과정을 실험하였다. 본 실험을 통해서 두 가지 서로 다른 환경에서 PPO 가 A3C 보다 더 나은 성능을 보임을 확인하였다.

• KIPS Transactions on Computer and Communication Systems
• /
• v.8 no.1
• /
• pp.17-28
• /
• 2019

In the financial investment management strategy, the distributed investment selecting and combining various financial assets is called portfolio management theory. In recent years, the blockchain based financial assets, such as cryptocurrencies, have been traded on several well-known exchanges, and an efficient portfolio management approach is required in order for investors to steadily raise their return on investment in cryptocurrencies. On the other hand, deep learning has shown remarkable results in various fields, and research on application of deep reinforcement learning algorithm to portfolio management has begun. In this paper, we propose an efficient financial portfolio investment management method based on Asynchronous Advantage Actor-Critic (A3C), which is a representative asynchronous reinforcement learning algorithm. In addition, since the conventional cross-entropy function can not be applied to portfolio management, we propose a proper method where the existing cross-entropy is modified to fit the portfolio investment method. Finally, we compare the proposed A3C model with the existing reinforcement learning based cryptography portfolio investment algorithm, and prove that the performance of the proposed A3C model is better than the existing one.

• The Journal of the Convergence on Culture Technology
• /
• v.8 no.5
• /
• pp.697-703
• /
• 2022

Due to the recent economic downturn caused by Covid-19 and the unstable international situation, many investors are choosing the derivatives market as a means of investment. However, the derivatives market has a greater risk than the stock market, and research on the market of market participants is insufficient. Recently, with the development of artificial intelligence, machine learning has been widely used in the derivatives market. In this paper, reinforcement learning, one of the machine learning techniques, is applied to analyze the scalping technique that trades futures in minutes. The data set consists of 21 attributes using the closing price, moving average line, and Bollinger band indicators of 1 minute and 3 minute data for 6 months by selecting 4 products among futures products traded at trading firm. In the experiment, DNN artificial neural network model and three reinforcement learning algorithms, namely, DQN (Deep Q-Network), A2C (Advantage Actor Critic), and A3C (Asynchronous A2C) were used, and they were trained and verified through learning data set and test data set. For scalping, the agent chooses one of the actions of buying and selling, and the ratio of the portfolio value according to the action result is rewarded. Experiment results show that the energy sector products such as Heating Oil and Crude Oil yield relatively high cumulative returns compared to the index sector products such as Mini Russell 2000 and Hang Seng Index.