Browse > Article
http://dx.doi.org/10.9708/jksci.2020.25.05.019

Generating Augmented Lifting Player using Pose Tracking  

Choi, Jong-In (Dept. of Digital Media, Seoul Women's University)
Kim, Jong-Hyun (Dept. of Software Application, Kangnam University)
Publication Information
Journal of the Korea Society of Computer and Information / v.25, no.5, 2020 , pp. 19-26 More about this Journal
Abstract
This paper proposes a framework for creating acrobatic scenes such as soccer ball lifting using various users' videos. The proposed method can generate a desired result within a few seconds using a general video of user recorded with a mobile phone. The framework of this paper is largely divided into three parts. The first is to analyze the posture by receiving the user's video. To do this, the user can calculate the pose of the user by analyzing the video using a deep learning technique, and track the movement of a selected body part. The second is to analyze the movement trajectory of the selected body part and calculate the location and time of hitting the object. Finally, the trajectory of the object is generated using the analyzed hitting information. Then, a natural object lifting scenes synchronized with the input user's video can be generated. Physical-based optimization was used to generate a realistic moving object. Using the method of this paper, we can produce various augmented reality applications.
Keywords
Augmented Reality; Artificial Intelligence; Deep Neural Networks; Posture Tracking; Video Synthesis;
Citations & Related Records
연도 인용수 순위
  • Reference
1 C. Wojtan, P. Mucha, and G. Turk, "Keyframe control of complex particle systems using the adjoint method," pp. 15-23, 01 2006. DOI:10.1145/1218064.1218067.
2 J. Barbi and J. Popovi, "Real-time control of physically based simulations using gentle forces," ACM Transactions on Graphics, vol. 27, 12 2008. doi: 10.1145/1409060.1409116.
3 S. Jain and C. Liu, "Interactive synthesis of human-object interaction," pp. 47-53, 01 2009. doi: 10.1145/1599470.1599476.
4 J. I. Choi, S. J. Kang, C. H. Kim, and J. Lee, "Virtual ball player," The Visual Computer, vol. 31, 05 2015. doi: 10.1007/s00371-015-1116-9.
5 J. Chemin and J. Lee. "A physics-based juggling simulation using reinforcement learning," In Proceedings of the 11th Annual International Conference on Motion, Interaction, and Games (MIG '18), Article 3, 1-7. DOI: 10.1145/3274247.3274516.
6 S. Hong, D. Han, K. Cho, J. S. Shin, and J. Noh. 2019. Physics-based full-body soccer motion control for dribbling and shooting. ACM Trans. Graph. vol 38, no 4, Article 74 (July 2019), pp. 1-12. DOI:10.1145/3306346.3322963.
7 A. Treuille, A. McNamara, Z. Popovic, and J. Stam, "Keyframe control of smoke simulations," ACM Trans. Graph., vol. 22, pp. 716-723, 07 2003. doi: 10.1145/882262.882337.   DOI
8 S. Johnson and M. Everingham, "Clustered pose and nonlinear appearance models for human pose estimation," pp. 1-11, 01 2010. doi:10.5244/C.24.12.
9 Z. Cao, G. Hidalgo, T. Simon, S. Wei, and Y. Sheikh, "Openpose: Realtime multi-person 2d pose estimation using part affinity fields," CoRR, vol. abs/1812.08008, 2018. arXiv:1611.08050.
10 M. Andriluka, S. Roth, and B. Schiele, "Pictorial structures revisited: People detection and articulated pose estimation," pp. 1014 - 1021, 07 2009. doi: 10.1109/CVPR.2009.5206754.
11 A. Bulat and G. Tzimiropoulos, "Human pose estimation via convolutional part heatmap regression," vol. 9911, 10 2016. doi:10.1007/978-3-319-46478-7_44.
12 U. Iqbal and J. Gall, "Multi-person pose estimation with local joint-toperson associations," vol. 9914, 10 2016. doi: 10.1007/978-3-319-48881-344.
13 V. Ramakrishna, D. Munoz, M. Hebert, J. Bagnell, and Y. Sheikh, "Pose machines: Articulated pose estimation via inference machines," Conference Paper, Proceedings of European Conference on Computer Vision. pp. 33-47, 09 2014. doi:10.1007/978-3-319-10605-2_3.
14 M. Sun and S. Savarese, "Articulated part-based model for joint object detection and pose estimation," pp. 723-730, 11 2011. DOI:10.1109/ICCV.2011.6126309.
15 A. Jain, "Articulated people detection and pose estimation: Reshaping the future," pp. 3178-3185, 06 2012. DOI: 10.1109/CVPR.2012.6248052.
16 G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, and K. Murphy, "Towards accurate multi-person pose estimation in the wild," pp. 3711-3719, 07 2017. doi: 10.1109/CVPR.2017.395.
17 D. Holden, T. Komura, and J. Saito, "Phase-functioned neural networks for character control," ACM Transactions on Graphics, vol. 36, pp. 1-13, 07 2017. doi: 10.1145/3072959.3073663.
18 X. Peng, G. Berseth, K. Yin, and M. Panne, "Deeploco: dynamic locomotion skills using hierarchical deep reinforcement learning," ACM Transactions on Graphics, vol. 36, pp. 1-13, 07 2017. doi: 10.1145/3072959.3073602.
19 D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin, M. Shafiei Rezvani Nezhad, H.-P. Seidel, W. Xu, D. Casas, and C. Theobalt, "Vnect: Real-time 3d human pose estimation with a single rgb camera," ACM Transactions on Graphics, vol. 36, 05 2017. doi:10.1145/3072959.3073596
20 D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, M. Elgharib, P. Fua, H. Seidel, H. Rhodin, G. Pons-Moll, and C. Theobalt, "Xnect: Real-time multiperson 3d human pose estimation with a single RGB camera," CoRR, vol. abs/1907.00837, 2019. arXiv:1907.00837.
21 E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, and B. Schiele, "Deepercut: A deeper, stronger, and faster multi-person pose estimation model," 05 2016. arXiv: 1605.03170.
22 A. Toshev and C. Szegedy, "Deeppose: Human pose estimation via deep neural networks," Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 12 2013. DOI: 10.1109/CVPR.2014.214.
23 W. Ouyang, X. Chu, and X. Wang, "Multi-source deep learning for human pose estimation," pp. 2337-2344, 06 2014. doi: 10.1109/CVPR.2014.299.
24 J. Tompson, A. Jain, Y. Lecun, and C. Bregler, "Joint training of a convolutional network and a graphical model for human pose estimation," 06 2014. arXiv:1406.2984.
25 A. Newell, K. Yang, and J. Deng, "Stacked hourglass networks for human pose estimation," vol. 9912, pp. 483-499, 10 2016. doi: 10.1007/978-3-319-46484-829.
26 L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. Gehler, and B. Schiele, "Deepcut: Joint subset partition and labeling for multi person pose estimation," pp. 4929-4937, 06 2016. arXiv:1511.06645.
27 K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," vol. 7, 12 2015. DOI: 10.1109/CVPR.2016.90.
28 C. Twigg and D. James, "L.: Backward steps in rigid body simulation," ACM Trans. Graph., vol. 27, 08 2008. doi: 10.1145/1399504.1360624.
29 J. Popovic, S. Seitz, M. Erdmann, Z. Popovic, and A. Witkin, "Interactive manipulation of rigid body simulations," Proceedings of the ACM SIGGRAPH Conference on Computer Graphics, 10 2001. doi:10.1145/344779.344880.
30 R. Fattal and D. Lischinski, "Target-driven smoke animation," ACM Transaction on Graphics, vol. 23, 06 2004. doi:10.1145/1015706.1015743.