Browse > Article
http://dx.doi.org/10.5909/JBE.2020.25.4.545

Implementation of Interactive Media Content Production Framework based on Gesture Recognition  

Koh, You-jin (Department of Newmedia, Seoul Media Institute of Technology)
Kim, Tae-Won (Department of Newmedia, Seoul Media Institute of Technology)
Kim, Yong-Goo (Department of Newmedia, Seoul Media Institute of Technology)
Choi, Yoo-Joo (Department of Newmedia, Seoul Media Institute of Technology)
Publication Information
Journal of Broadcast Engineering / v.25, no.4, 2020 , pp. 545-559 More about this Journal
Abstract
In this paper, we propose a content creation framework that enables users without programming experience to easily create interactive media content that responds to user gestures. In the proposed framework, users define the gestures they use and the media effects that respond to them by numbers, and link them in a text-based configuration file. In the proposed framework, the interactive media content that responds to the user's gesture is linked with the dynamic projection mapping module to track the user's location and project the media effects onto the user. To reduce the processing speed and memory burden of the gesture recognition, the user's movement is expressed as a gray scale motion history image. We designed a convolutional neural network model for gesture recognition using motion history images as input data. The number of network layers and hyperparameters of the convolutional neural network model were determined through experiments that recognize five gestures, and applied to the proposed framework. In the gesture recognition experiment, we obtained a recognition accuracy of 97.96% and a processing speed of 12.04 FPS. In the experiment connected with the three media effects, we confirmed that the intended media effect was appropriately displayed in real-time according to the user's gesture.
Keywords
gesture recognition; interactive media; dynamic projection mapping; deep learning; convolutional neural network;
Citations & Related Records
Times Cited By KSCI : 4  (Citation Analysis)
연도 인용수 순위
1 KOCCA, Trends in stage production of large events and performances in the United States, Content Industry Trend of USA, Vol. 21, 2018
2 bot&dolly - Box, https://youtu.be/lX6JcybgDFo (accessed Sep. 24, 2013)
3 Ishikawa group Lab - DynaFlash v2 and Post Reality, https://youtu.be/QDppJ9NWtaE (accessed Mar. 5, 2018)
4 connected colors / real-time face tracking and 3d projection mapping, https://youtu.be/nMvFwC3bo_E (accessed Mar. 14, 2016)
5 緣 ' NMARA Interdisciplinary Art Performance Works, https://youtu.be/0oa7kVbVxsA (accessed Dec. 16, 2019)
6 Virtual Reality Interactive Sandbox (Contour Line), https://youtu.be/PyJfIdNbtv4 (accessed Feb. 4, 2015)
7 J. Park, W. Kim, "A Study of the Value and Utility of Projection Mapping in the Contents of Dance Performances-Focused on the Work :"Our Karma"-", Journal of Korean Dance, Vol.46, pp.9-28, August 2018.
8 Projection Mapping Contents of Media Art - LG CNS Blog, https://blog.lgcns.com/1149 (accessed July. 14, 2016)
9 S. Kim, Y. Koh and Y. Choi, "Design and Implementation of Immersive Media System Based on Dynamic Projection Mapping and Gesture Recognition" KIPS Transactions on Software and Data Engineering, Vol.9, No.3, pp.109-122, January 2020. https://doi.org/10.3745/KTSDE.2020.9.3.109   DOI
10 H. Yang, "Deep-learning and Gesture recognition" Broadcasting and Media Magazine, Vol. 22, No.1, pp.67-74, January 2017.
11 J. Lee, Y. Kim, D. Kim, "Realtime Projection Mapping on Flexible Dynamic Objects" Proceeding Of HCI Korea Conference, Gangwondo, Korea, pp. 187-190, 2014.
12 Interactive wall and Floor Projection, https://youtu.be/AZA6X3mPdtg (accessed May. 7, 2013)
13 B. Lee, D. Oh, T. Kim, "3D Virtual Reality Game with Deep Learning-based Hand Gesture Recognition." Journal of the Korea Computer Graphics Society, Vol. 24, No. 5, pp.41-48, December 2018. https://doi.org/10.15701/kcgs.2018.24.5.41   DOI
14 Maryam Asadi-Aghbolaghi, Albert Clapes, Marco Bellantonio, "A survey on deep learning based approaches for action and gesture recognition in image sequences." Proceeding of IEEE International Conference on Automatic Face & Gesture Recognition, Washington, DC, pp. 476-483, 2017, https://doi.org/10.1109/FG.2017.150 (accessed June.1, 2017)