Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Real-Time Joint Animation Production and Expression System using Deep Learning Model and Kinect Camera

딥러닝 모델과 Kinect 카메라를 이용한 실시간 관절 애니메이션 제작 및 표출 시스템 구축에 관한 연구

  • Kim, Sang-Joon (Dept. of Information Technology and Media Engineering, The graduate School of Nano IT Design Fusion, Seoul National University of Science and Technology) ;
  • Lee, Yu-Jin (Dept. of Media IT Engineering, The Graduate School, Seoul National University of Science and Technology) ;
  • Park, Goo-man (Dept. of Media IT Engineering, The Graduate School, Seoul National University of Science and Technology)
  • 김상준 (서울과학기술대학교 정보통신미디어공학전공) ;
  • 이유진 (서울과학기술대학교 미디어IT공학과) ;
  • 박구만 (서울과학기술대학교 미디어IT공학과)
  • Received : 2021.04.12
  • Accepted : 2021.05.14
  • Published : 2021.05.30
Download PDF
⟨ Previous Next ⟩

Abstract

As the distribution of 3D content such as augmented reality and virtual reality increases, the importance of real-time computer animation technology is increasing. However, the computer animation process consists mostly of manual or marker-attaching motion capture, which requires a very long time for experienced professionals to obtain realistic images. To solve these problems, animation production systems and algorithms based on deep learning model and sensors have recently emerged. Thus, in this paper, we study four methods of implementing natural human movement in deep learning model and kinect camera-based animation production systems. Each method is chosen considering its environmental characteristics and accuracy. The first method uses a Kinect camera. The second method uses a Kinect camera and a calibration algorithm. The third method uses deep learning model. The fourth method uses deep learning model and kinect. Experiments with the proposed method showed that the fourth method of deep learning model and using the Kinect simultaneously showed the best results compared to other methods.

증강현실과 가상현실 같은 3차원 콘텐츠 보급이 증가함에 따라 실시간 컴퓨터 애니메이션 기술의 중요성이 높아지고 있다. 하지만 컴퓨터 애니메이션 제작 과정은 대부분 수작업 혹은 마커를 부착하는 모션캡쳐 방식으로 이루어져 있다. 때문에 사실적인 영상을 얻기 위해서는 숙련된 전문가에게도 매우 오랜 시간이 필요하다. 이러한 문제점을 해결하기 위해 최근에는 딥러닝 모델과 센서를 기반으로 하는 애니메이션 제작 시스템과 알고리즘이 나오고 있다. 이에 본 논문에서는 딥러닝과 Kinect 카메라 기반 FBX 형식의 애니메이션 제작 시스템에서 자연스러운 인체 움직임을 구현하는 4가지 방법에 대해 연구했다. 각 방법은 환경적 특성과 정확도를 고려하여 선택된다. 첫 번째 방법은 Kinect 카메라를 사용한다. 두 번째 방법은 Kinect 카메라와 보정 알고리즘을 사용한다. 세 번째 방법은 딥러닝 모델을 사용한다. 네 번째 방법은 딥러닝 모델과 Kinect를 사용한다. 제안 방법을 오차와 처리 속도를 실험한 결과, 네 번째 딥러닝 모델과 Kinect를 동시에 사용하는 방법이 다른 방법에 비해 가장 좋은 결과를 보였다.

Keywords

Acknowledgement

This work was supported by Institute for Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) in 2021(No. 2017-0-00217, Development of Immersive Signage Based on Variable Transparency and Multiple Layers).

References

  1. Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh, "OpenPose: realtime multi-person 3D poseestimation using Part Affinity Fields", In arXiv preprintarXiv:1812.08008, 2018.
  2. R. A. Guler, N. Neverova, and I. Kokkinos. Densepose: Dense human pose estimation in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  3. Bruno Artacho, Andreas Savakis, "UniPose: Unified Human Pose Estimation in Single Images and Videos" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 7035-7044
  4. Moon, Gyeongsik, Ju Yong Chang, and Kyoung Mu Lee. "V2v-posenet: Voxel-to-voxel prediction network for accurate 3d hand and human pose estimation from a single depth map." Proceedings of the IEEE conference on computer vision and pattern Recognition. 2018.
  5. Sun, X., Xiao, B., Liang, S., Wei, Y.: Integral human pose regression. arXiv preprint arXiv:1711.08229 (2017)
  6. [Internet] Microsoft Kinect v3 https://azure.microsoft.com/ko-kr/services/kinect-dk/
  7. [Internet] ASUS Xtion PRO LIVe https://www.asus.com/kr/3DSensor/Xtion_PRO_LIVE/
  8. [Internet] ASUS Xtion2 https://www.asus.com/kr/3D-Sensor/Xtion-2/
  9. [Internet] LEAP Motion https://developer.leapmotion.com/#101
  10. [Internet] "OpenMMD" https://github.com/peterljq/OpenMMD
  11. D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin,M. Shafiei, H.-P. Seidel, W. Xu, D. Casas, and C. Theobalt, "Vnect: Real-time 3d human pose estimation with a singlergb camera", In ACM Transactions on Graphics, volume 36, 2017.
  12. [Internet] "Instructions"http://marcojrfurtado.github.io/KinectAnimationStudio/usage.html
  13. [Internet] "Avateering with kinect v2 - Joint Orientations" https://peted.azurewebsites.net/avateering-with-kinect-v2-joint-orientations/
  14. [Internet] "Brekel Body v2" https://brekel.com/brekel-pro-body-v2/
  15. J. Martinez, R. Hossain, J. Romero, and J. J. Little. A simple yet effective baseline for 3d human pose estimation. In IEEE International Conference on Computer Vision, ICCV, 2017.
  16. Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 3D Vision (3DV), 2016 Fourth International Conference on, pages 239-248. IEEE, 2016.
  17. Jun hee Kim, Sae-Woung Yoo and Kyung-Won Min, Microsoft Kinect-based Indoor Building Information Model Acquisition., Computational Structural Engineering Institute of Korea 31(4), 207-214.
  18. Sang-Joon Kim, "Design and Implementation of Authoring Tool for Dynamic Projection Mapping Content," Degree thesis (Bachelor's degree), Seoul Media Graduate School: New Media Studies Department 2019.2
  19. A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages1653-1660. IEEE, 2014.
  20. [Internet] Leeds Sports Pose(LSP) https://sam.johnson.io/research/lsp.html
  21. [Internet] MPII Human Pose http://human-pose.mpi-inf.mpg.de/
  22. [Internet] MS COCO https://cocodataset.org/#home
  23. [Internet] AI Challenger http://dataju.cn/Dataju/web/datasetInstanceDetail/440
  24. [Internet] Human3.6M http://vision.imar.ro/human3.6m/description.php
  25. [Internet] CMU Panoptic http://domedb.perception.cs.cmu.edu/
  26. Alexander Toshev, Christian Szegedy "DeepPose: Human Pose Estimation via Deep Neural Networks" arXiv preprint arXiv: 1312.4659, 2013
  27. S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh. Convolutional pose machines. In CVPR, 2016
  28. Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu. Crowdpose: Efficient crowded scenes pose estimation and a new benchmark. arXiv preprint arXiv:1812.00324, 2018
  29. J. heon jeong, S. joon kim, M. suk Yoon and G. man Park "Body Segment Length and Joint Motion Range Restriction for Joint Errors Correction in FBX Type Motion Capture Animation based on Kinect Camera ", JBE Vol. 25, No. 3, May 2020