Journal of the Korea Society for Simulation (한국시뮬레이션학회논문지)

The Korea Society for Simulation (한국시뮬레이션학회)
권호 표지
DOI QR코드

DOI QR Code

Earthwork Planning via Reinforcement Learning with Heterogeneous Construction Equipment

강화학습을 이용한 이종 장비 토목 공정 계획

  • 지민기 (한국과학기술원 산업 및 시스템공학과) ;
  • 박준건 (한국과학기술원 산업 및 시스템공학과) ;
  • 김도형 (한국과학기술원 산업 및 시스템공학과) ;
  • 정요한 (한국과학기술원 산업 및 시스템공학과) ;
  • 박진규 (한국과학기술원 산업 및 시스템공학과) ;
  • 문일철
  • Received : 2017.07.12
  • Accepted : 2017.12.14
  • Published : 2018.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

Earthwork planning is one of the critical issues in a construction process management. For the construction process management, there are some different approaches such as optimizing construction with either mathematical methodologies or heuristics with simulations. This paper propose a simulated earthwork scenario and an optimal path for the simulation using a reinforcement learning. For reinforcement learning, we use two different Markov decision process, or MDP, formulations with interacting excavator agent and truck agent, sequenced learning, and independent learning. The simulation result shows that two different formulations can reach the optimal planning for a simulated earthwork scenario. This planning could be a basis for an automatic construction management.

토목 공정 계획은 건설 공정 관리에서 중요한 과제 중 하나이다. 수학적 방법론에 기반을 둔 최적화 기법, 휴리스틱에 기반을 둔 최적화 기법 그리고 행위자 기반의 시뮬레이션 등의 방법론이 건설 공정 관리를 위해 적용되어왔다. 본 연구에서는 가상의 토목 공정 환경을 개발하고, 가상의 토목 공정 환경에서 강화학습을 이용한 시뮬레이션을 통해 토목 공정의 최적 경로를 찾는 방법을 제안하였다. 강화학습에 있어 본 연구에서는 상호작용 하며 서로 다른 행동을 하는 굴삭기와 트럭 에이전트들 에 대해 순차적 학습과 독립적 학습에 기반을 둔 두 가지의 Markov decision process (MDP)를 사용하였다. 가상의 토목 공정 환경에서 두 가지 방법 모두 최적에 가까운 토목 공정 계획을 만들어 낼 수 있음을 시뮬레이션 결과에 따라 알 수 있었으며, 이 계획은 건설 자동화의 기초가 될 수 있을 것이다.

Keywords

References

  1. Angelou, Plamen., "Construction Scheduling, Cost Optimization, and Management: A New Model Based on Neurocomputing and Object Technologies.", Engineering Construction and Architectural Management 8.3: 233-234, 2001.
  2. Busoniu, Lucian, Robert Babuska, and Bart De Schutter. "A comprehensive survey of multiagent reinforcement learning.", IEEE Transactions on Systems, Man, And Cybernetics-Part C: Applications and Reviews, 38 (2), 2008.
  3. Chan, Jonathan Cheung-Wai, Kwok-Ping Chan, and Anthony Gar-On Yeh. "Detecting the nature of change in an urban environment: A comparison of machine learning algorithms." Photogrammetric Engineering and Remote Sensing 67.2: 213-226, 2001.
  4. Cheng, Min-Yuan, et al., "Estimate at completion for construction projects using evolutionary support vector machine inference model.", Automation in Construction 19.5: 619-629, 2010. https://doi.org/10.1016/j.autcon.2010.02.008
  5. D. Chang and R. Carr, "Resque: A resource oriented simulation system for multiple resource constrained processes," in Proceedings of the PMI Seminar/Symposium, pp. 4-19, 1987.
  6. Easa, Said M., "Resource leveling in construction by optimization.", Journal of construction engineering and management 115.2: 302-316, 1989. https://doi.org/10.1061/(ASCE)0733-9364(1989)115:2(302)
  7. Elazouni, Ashraf M.,et al., "Estimating resource requirements at conceptual design stage using neural networks.", Journal of Computing in Civil Engineering 11.4: 217-223, 1997. https://doi.org/10.1061/(ASCE)0887-3801(1997)11:4(217)
  8. Halpin, Daniel W., "CYCLONE-method for modeling job site processes.", Journal of the construction division 103.ASCE 13234 Proceeding, 1977.
  9. J. F. Lluch, "Analysis of construction operations using microcomputers," Ph.D. dissertation, Georgia Institute of Technology, 1981.
  10. J. Kim, M. Fischer, C. Kam, A. Kiremidjian, and S. U. C. . E. E. Department, Framework for Dynamic Generation and Evaluation of Excavation Schedules for Hard Rock Tunnels in Preconstruction and Construction, 2016.
  11. Lee Cheol Kyu, Kim Sung-Keun, Sung Young-Jun, "A Study on 2D-Based Earthwork Planning Methods", Korean Society of Civil Engineering, 349-357, 2003.
  12. Liu, Liang, Scott A. Burns, and Chung-Wei Feng., "Construction time-cost trade-off analysis using LP/IP hybrid method.", Journal of Construction Engineering and Management, 121.4: 446-454, 1995. https://doi.org/10.1061/(ASCE)0733-9364(1995)121:4(446)
  13. Lim, So-Young, Kim, Sung-Keun, Ahn, Seo-Hyun, "Development of a Soil Distribution Method and Equipment Operation Models Using Worker''s Heuristics", Journal of the Korean Society of Civil Engineers, Vol.36, No.3, pp.551-564, 2016. https://doi.org/10.12652/Ksce.2016.36.3.0551
  14. Lu, Pengzhen, Shengyong Chen, and Yujun Zheng. "Artificial intelligence in civil engineering." Mathematical Problems in Engineering 2012.
  15. M. Lauer and M. Riedmiller, "An algorithm for distributed reinforcement learning in cooperative multi-agent systems," in In Proceedings of the Seventeenth International Conference on Machine Learning. Citeseer, 2000.
  16. Park, J.G., M.G. Ji, D.H. Kim, Y.H. Jung, H.S. Lee, J.G. Park, and I.C. Moon, "Optimize earthwork Planning using reinforcement learning",, 2172-2179. spring joint meeting of Korean Institute of Industrial Engineers, 2017.
  17. P. G. Ioannou and C. P. G. Ioannou, "Um-cyclone discrete event simulation system users guide," 1990.
  18. Q. Zhang, E. Durfee, S. Singh, A. Chen, and S. Witwicki, "Commitment semantics for sequential decision making under reward uncertainty," in Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI, pp. 3315-3323, 2016.
  19. R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press Cambridge, vol. 1, no. 1, 1998.
  20. S. Abu-Samra, A. H. El Hakea, C. Khalil, O. Hosny, and A. Elhakeem, "Optimum planning for multi-project earthmoving operations.", International Symposium on Automation and Robotics in Construction and Mining, 348-357, 2013.
  21. S. J. Witwicki and E. H. Durfee, "Influence-based policy abstraction for weakly-coupled dec-pomdps." in ICAPS, pp. 185-192, 2010.
  22. Watkins, Christopher JCH, and Peter Dayan., "Qlearning.", Machine learning 8.3-4: 279-292, 1992. https://doi.org/10.1023/A:1022676722315