DOI QR코드

DOI QR Code

Reinforcement Learning of Bipedal Walking with Musculoskeletal Models and Reference Motions

근골격 모델과 참조 모션을 이용한 이족보행 강화학습

  • Jiwoong Jeon (Dept. of Computer and Software, Hanyang University) ;
  • Taesoo Kwon (Dept. of Computer and Software, Hanyang University)
  • 전지웅 (한양대학교 일반대학원 컴퓨터 소프트웨어학과) ;
  • 권태수 (한양대학교 일반대학원 컴퓨터 소프트웨어학과)
  • Received : 2022.11.23
  • Accepted : 2023.01.27
  • Published : 2023.03.01

Abstract

In this paper, we introduce a method to obtain high-quality results at a low cost for simulating musculoskeletal characters based on data from the reference motion through motion capture on two-legged walking through reinforcement learning. We reset the motion data of the reference motion to allow the character model to perform, and then train the corresponding motion to be learned through reinforcement learning. We combine motion imitation of the reference model with minimal metabolic energy for the muscles to learn to allow the musculoskeletal model to perform two-legged walking in the desired direction. In this way, the musculoskeletal model can learn at a lower cost than conventional manually designed controllers and perform high-quality bipedal walking.

본 논문은 강화학습을 통해 이족보행에 대한 모션 캡처를 통해 참조 모션의 데이터들을 기반으로 근골격 캐릭터의 시뮬레이션을 적은 비용으로 높은 품질의 결과를 얻을 방법을 소개한다. 우리는 참조 모션 데이터를 캐릭터 모델이 수행할 수 있게끔 재설정을 한 후, 강화학습을 통해 해당 모션을 학습하도록 훈련시킨다. 참조 모션 모방과 근육에 대한 최소한의 메타볼릭 에너지를 결합하여 원하는 방향으로 근골격 모델이 이족보행을 수행하게끔 학습한다. 이러한 방법으로 근골격 모델은 기존의 수동으로 설계된 컨트롤러보다 적은 비용으로 학습할 수 있으며 높은 품질의 이족보행을 수행할 수 있게 된다.

Keywords

References

  1. Yin, K., Loken, K., And Panne, M. V. D. 2007. SIMBICON: simple biped locomotion control. ACM Trans. Graph. 26, 3. 
  2. Geyer H, Herr H. A muscle-reflex model that encodes principles of legged mechanics produces human walking dynamics and muscle activities. IEEE Trans Neural Syst Rehabil Eng. 2010;18(3):263-273.  https://doi.org/10.1109/TNSRE.2010.2047592
  3. Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018. Deepmimic:Example-guided Deep Reinforcement Learning of Physics-based Character Skills. ACM Trans. Graph. 37, 4, Article 143 (July 2018), 14 pages. 
  4. Jack M. Wang, Samuel R. Hamner, Scott L. Delp, and Vladlen Koltun. 2012. Optimizing Locomotion Controllers Using Biologically-based Actuators and Objectives. ACM Trans. Graph. 31, 4, Article 25 (July 2012), 11 pages. 
  5. Seunghwan S. Lee, Ri Yu, Jungnam Park, Mridul Aanjaneya, Eftychios Sifakis, and Jehee Lee. 2018. Dexterous Manipulation and Control with Volumetric Muscles. ACM Trans. Graph. 37, 4, Article 57 (July 2018), 13 pages. 
  6. Yoonsang Y.S. Lee, Moon Seok Park, Taesoo Kwon, and Jehee Lee. 2014. Locomotion Control for Many-muscle Humanoids. ACM Trans. Graph. 33, 6, Article 218 (Nov. 2014), 11 pages 
  7. Thomas Geijtenbeek, Antonie J. van den Bogert, Ben J.H. van Basten, and Arjan Egges. 2010. Evaluating the Physical Realism of Character Animations Using Musculoskeletal Models. In Third International Conference in Motion in Games, 2010. (Lecture Notes in Computer Science). 
  8. Jungnam Park, Sehee Min, Philsik Chang, Jaedong Lee, Moonseok Park, Jehee Lee. 2022. Generative GaitNet. ACM Trans. Graph. (SIGGRAPH 2022) 22, 1-9.