Browse > Article
http://dx.doi.org/10.5909/JBE.2021.26.3.269

Real-Time Joint Animation Production and Expression System using Deep Learning Model and Kinect Camera  

Kim, Sang-Joon (Dept. of Information Technology and Media Engineering, The graduate School of Nano IT Design Fusion, Seoul National University of Science and Technology)
Lee, Yu-Jin (Dept. of Media IT Engineering, The Graduate School, Seoul National University of Science and Technology)
Park, Goo-man (Dept. of Media IT Engineering, The Graduate School, Seoul National University of Science and Technology)
Publication Information
Journal of Broadcast Engineering / v.26, no.3, 2021 , pp. 269-282 More about this Journal
Abstract
As the distribution of 3D content such as augmented reality and virtual reality increases, the importance of real-time computer animation technology is increasing. However, the computer animation process consists mostly of manual or marker-attaching motion capture, which requires a very long time for experienced professionals to obtain realistic images. To solve these problems, animation production systems and algorithms based on deep learning model and sensors have recently emerged. Thus, in this paper, we study four methods of implementing natural human movement in deep learning model and kinect camera-based animation production systems. Each method is chosen considering its environmental characteristics and accuracy. The first method uses a Kinect camera. The second method uses a Kinect camera and a calibration algorithm. The third method uses deep learning model. The fourth method uses deep learning model and kinect. Experiments with the proposed method showed that the fourth method of deep learning model and using the Kinect simultaneously showed the best results compared to other methods.
Keywords
Kinect; deep learning; 3D animation; skeleton; human pose estimation;
Citations & Related Records
연도 인용수 순위
  • Reference
1 [Internet] Microsoft Kinect v3 https://azure.microsoft.com/ko-kr/services/kinect-dk/
2 D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin,M. Shafiei, H.-P. Seidel, W. Xu, D. Casas, and C. Theobalt, "Vnect: Real-time 3d human pose estimation with a singlergb camera", In ACM Transactions on Graphics, volume 36, 2017.
3 Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh, "OpenPose: realtime multi-person 3D poseestimation using Part Affinity Fields", In arXiv preprintarXiv:1812.08008, 2018.
4 R. A. Guler, N. Neverova, and I. Kokkinos. Densepose: Dense human pose estimation in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
5 Bruno Artacho, Andreas Savakis, "UniPose: Unified Human Pose Estimation in Single Images and Videos" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 7035-7044
6 Moon, Gyeongsik, Ju Yong Chang, and Kyoung Mu Lee. "V2v-posenet: Voxel-to-voxel prediction network for accurate 3d hand and human pose estimation from a single depth map." Proceedings of the IEEE conference on computer vision and pattern Recognition. 2018.
7 Sun, X., Xiao, B., Liang, S., Wei, Y.: Integral human pose regression. arXiv preprint arXiv:1711.08229 (2017)
8 [Internet] ASUS Xtion PRO LIVe https://www.asus.com/kr/3DSensor/Xtion_PRO_LIVE/
9 [Internet] ASUS Xtion2 https://www.asus.com/kr/3D-Sensor/Xtion-2/
10 [Internet] LEAP Motion https://developer.leapmotion.com/#101
11 [Internet] "Instructions"http://marcojrfurtado.github.io/KinectAnimationStudio/usage.html
12 [Internet] "Avateering with kinect v2 - Joint Orientations" https://peted.azurewebsites.net/avateering-with-kinect-v2-joint-orientations/
13 [Internet] "OpenMMD" https://github.com/peterljq/OpenMMD
14 Sang-Joon Kim, "Design and Implementation of Authoring Tool for Dynamic Projection Mapping Content," Degree thesis (Bachelor's degree), Seoul Media Graduate School: New Media Studies Department 2019.2
15 J. Martinez, R. Hossain, J. Romero, and J. J. Little. A simple yet effective baseline for 3d human pose estimation. In IEEE International Conference on Computer Vision, ICCV, 2017.
16 Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 3D Vision (3DV), 2016 Fourth International Conference on, pages 239-248. IEEE, 2016.
17 Jun hee Kim, Sae-Woung Yoo and Kyung-Won Min, Microsoft Kinect-based Indoor Building Information Model Acquisition., Computational Structural Engineering Institute of Korea 31(4), 207-214.
18 A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages1653-1660. IEEE, 2014.
19 [Internet] MPII Human Pose http://human-pose.mpi-inf.mpg.de/
20 [Internet] "Brekel Body v2" https://brekel.com/brekel-pro-body-v2/
21 [Internet] Human3.6M http://vision.imar.ro/human3.6m/description.php
22 [Internet] AI Challenger http://dataju.cn/Dataju/web/datasetInstanceDetail/440
23 Alexander Toshev, Christian Szegedy "DeepPose: Human Pose Estimation via Deep Neural Networks" arXiv preprint arXiv: 1312.4659, 2013
24 [Internet] Leeds Sports Pose(LSP) https://sam.johnson.io/research/lsp.html
25 [Internet] MS COCO https://cocodataset.org/#home
26 J. heon jeong, S. joon kim, M. suk Yoon and G. man Park "Body Segment Length and Joint Motion Range Restriction for Joint Errors Correction in FBX Type Motion Capture Animation based on Kinect Camera ", JBE Vol. 25, No. 3, May 2020
27 [Internet] CMU Panoptic http://domedb.perception.cs.cmu.edu/
28 S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh. Convolutional pose machines. In CVPR, 2016
29 Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu. Crowdpose: Efficient crowded scenes pose estimation and a new benchmark. arXiv preprint arXiv:1812.00324, 2018