Journal of the Korea Academia-Industrial cooperation Society (한국산학기술학회논문지)

The Korea Academia-Industrial cooperation Society (한국산학기술학회)
권호 표지
DOI QR코드

DOI QR Code

Trading Strategies Using Reinforcement Learning

강화학습을 이용한 트레이딩 전략

  • Cho, Hyunmin (Department of Management Engineering, Sangmyung University) ;
  • Shin, Hyun Joon (Department of Management Engineering, Sangmyung University)
  • 조현민 (상명대학교 경영공학과) ;
  • 신현준 (상명대학교 경영공학과)
  • Received : 2020.10.15
  • Accepted : 2021.01.08
  • Published : 2021.01.31
Download PDF
⟨ Previous Next ⟩

Abstract

With the recent developments in computer technology, there has been an increasing interest in the field of machine learning. This also has led to a significant increase in real business cases of machine learning theory in various sectors. In finance, it has been a major challenge to predict the future value of financial products. Since the 1980s, the finance industry has relied on technical and fundamental analysis for this prediction. For future value prediction models using machine learning, model design is of paramount importance to respond to market variables. Therefore, this paper quantitatively predicts the stock price movements of individual stocks listed on the KOSPI market using machine learning techniques; specifically, the reinforcement learning model. The DQN and A2C algorithms proposed by Google Deep Mind in 2013 are used for the reinforcement learning and they are applied to the stock trading strategies. In addition, through experiments, an input value to increase the cumulative profit is selected and its superiority is verified by comparison with comparative algorithms.

최근 컴퓨터 기술이 발전하면서 기계학습 분야에 관한 관심이 높아지고 있고 다양한 분야에 기계학습 이론을 적용하는 사례가 크게 증가하고 있다. 특히 금융 분야에서는 금융 상품의 미래 가치를 예측하는 것이 난제인데 80년대부터 지금까지 기술적 및 기본적 분석에 의존하고 있다. 기계학습을 이용한 미래 가치 예측 모형들은 다양한 잠재적 시장변수에 대응하기 위한 모형 설계가 무엇보다 중요하다. 따라서 본 논문은 기계학습의 하나인 강화학습 모형을 이용해 KOSPI 시장에 상장되어 있는 개별 종목들의 주가 움직임을 정량적으로 판단하여 이를 주식매매 전략에 적용한다. 강화학습 모형은 2013년 구글 딥마인드에서 제안한 DQN와 A2C 알고리즘을 이용하여 KOSPI에 상장된 14개 업종별 종목들의 과거 약 13년 동안의 시계열 주가에 기반한 데이터세트를 각각 입력 및 테스트 데이터로 사용한다. 데이터세트는 8개의 주가 관련 속성들과 시장을 대표하는 2개의 속성으로 구성하였고 취할 수 있는 행동은 매입, 매도, 유지 중 하나이다. 실험 결과 매매전략의 평균 연 환산수익률 측면에서 DQN과 A2C이 대안 알고리즘들보다 우수하였다.

Keywords

References

  1. Banz, R. W., "The Relationship between Return and Market Value of Common Stocks", Journal of Financial Economics, Vol.9, No.1, pp.3-18, 1981. DOI: http://dx.doi.org/10.1016/0304-405X(81)90018-0
  2. DeBondt, W. and R. Thaler, "Does the Stock Market Overreact?", Journal of Finance, Vol.40, No.3 pp.793-805, 1985. DOI: https://doi.org/10.1111/j.1540-6261.1985.tb05004.x
  3. Han, Y., K. Yang, and G. Zhou, "A New Anomaly: The Cross-Sectional Profitability of Technical Analysis," Journal of Financial and Quantitative Analysis, Vol.48, No.5, pp.1433-1461, 2013. DOI: http://dx.doi.org/10.2139/ssrn.1656460
  4. Zhu, Y. and G. Zhou, "Technical Analysis: An Asset Allocation Perspective on the Use of Moving Averages," Journal of Financial Economics, Vol.92, No.3, pp.519-544, 2009. DOI: http://dx.doi.org/10.2139/ssrn.1656460
  5. Benjamin Graham, David L. Dodd, Security Analysis, p.258, Natl Book Network, 2003, pp.77-120.
  6. Fama, E. F. and K. R. French, "Common Risk Factors in the Returns on Bonds and Stocks", Journal of Financial Economics, Vol.33, No.1, pp.3-56, 1993. DOI: http://dx.doi.org/10.1016/0304-405X(93)90023-5
  7. Jae Pil Ryu, Chang Hoon Hahn, and Hyun Joon Shin, "Portfolio Construction Strategy for IT Companies Using DEA Method", The Journal of Information Technology and Architecture, Vol.14, No.2, pp.139-146, 2017. https://www.earticle.net/Article/A345705
  8. H. G. Shong, Multimodal Reinforcement Learning based Stock Trading System combined with CNN and LSTM, Master's thesis, Kwangwoon University of Science and Technology, Seoul, Korea, pp.8-15, 2018.
  9. Jae Pil Ryu, Hyun Joon Shin, "Portfolio Selection Strategy Using Deep Learning", The Journal of Information Technology and Architecture, Vol.15, No.1, pp.43-50, 2018. https://www.earticle.net/Article/A345667 https://doi.org/10.22865/JITA.2018.15.1.43
  10. P. R. Burrell and B. O. Folarin, "The impact of neural networks in finance", Neural Computing & Applications, Vol.6, pp.193-200, 1997. http://link.springer.com/article/10.1007/BF01501506
  11. M. S. LEE and H. C. Ahn, "A Time Series Graph based Convolutional Neural Network Model for Effective Input Variable Pattern Learning : Application to the Prediction of Stock Market", Journal of Intelligence and Information Systems, Vol.24, No.1, pp.167-181, 2018. http://dbpia.co.kr/journal/articleDetail?nodeId=NODE07408509 https://doi.org/10.13088/jiis.2018.24.1.167
  12. I. T. Joo and S. H. Choi, "Stock Prediction Model based on Bidirectional LSTM Recurrent Neural Network", Journal of Korea Institute of Information, Electronics, and Communication Technology, Vol.11, No.2, pp.204-208, 2018. http://dbpia.co.kr/Journal/articleDetail?nodeId=NODE07424401 https://doi.org/10.17661/JKIIECT.2018.11.2.204
  13. Garcia-Galicia, Mauricio, Alin A. Carsteanu, and Julio B. Clempner, "Continuous-time reinforcement learning approach for portfolio management with time penalization", Expert Systems with Applications, Vol.129, No.2, pp.27-36, 2019. DOI: http://dx.doi.org/10.1016/j.eswa.2019.03.055
  14. Angelos Kanas, "Non-linear forecasts of stock returns", Journal of Forecasting, Vol.22, No.4, pp.299-315, 2003. DOI: http://dx.doi.org/10.1002/for.858
  15. Yoon, Y., Swales G., "Predicting stock price performance: a neural network approach", Proceedings of the Twenty-Fourth Annual Hawaii International Conference on System Sciences, IEEE, HI, USA, pp.156-162, August 2002 https://ieeexplore.ieee.org/abstract/document/184055
  16. Wong, L. K., Leung, F. H. F. and Tam P. K. S, "An Improved Lyapunov Function Based Stability Analysis Method for Fuzzy Logic Control Systems", Electronics Letters, Vol.36, pp.1085-1086, 2000. DOI: http://dx.doi.org/10.1109/FUZZY.2000.838698
  17. Jae Yeon Park, Jae Pil Ryu and Hyun Joon Shin, "Predicting KOSPI Stock Index using Machine Learning Algorithms with Technical Indicators", The Journal of Information Technology and Architecture, Vol.13, No.2, pp.331-340, 2016. https://www.earticle.net/Article/A346152
  18. Hamid, Shaikh A. and Iqbal, Zahid, "Using neural networks for forecasting volatility of S&P 500 Index futures prices", Journal of Business Research, Elsevier, Vol.57, No.10, pp.1116-1125, 2004. http://ideas.repec.org/a/eee/jbrese/v57y2004i10p1116-1125.html https://doi.org/10.1016/S0148-2963(03)00043-2
  19. Hadavandi, E., H. Shavandi, and A. Ghanbari, "Integration of genetic fuzzy systems and artificial neural networks for stock price forecasting", Knowledge-Based Systems, Vol.23, No.8, pp.800-808, 2010. DOI: https://doi.org/10.1016/j.knosys.2010.05.004
  20. Ping Feng Pai and Chih Sheng Lin, "A hybrid ARIMA and support vector machines model in stock price forecasting", Omega, Vol.33, No.6, pp.497-505, 2005. DOI: https://doi.org/10.1016/j.knosys.2010.05.004
  21. Zhiqiang, Guo, Wang Huaiqing, and Liu Quan, "Financial time series forecasting using LPP and SVM optimized by PSO", Soft Computing, Vol.17, No.5, pp.805-818, 2013. DOI: http://dx.doi.org/10.1007/s00500-012-0953-y
  22. Volodymyr Mnih, "Playing Atari with Deep Reinforcement Learning", DeepMind Technologies, United States, pp.2-4, 2013. https://arxiv.org/pdf/1312.5602.pdf
  23. Nando de Freitas, Reinforcement learning : OXFORD University p.28, 2019, pp.20, http://www.cs.ox.ac.uk/people/nando.defreitas/machinelearning/lecture12.pdf
  24. David Silver, Lecture7:Policy Gradient, p.41, 2020, pp.10-3, www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf