Browse > Article
http://dx.doi.org/10.9717/kmms.2019.22.4.424

Real-Time Hand Gesture Recognition Based on Deep Learning  

Kim, Gyu-Min (School of Electronic & Information Eng., Korea Aerospace University)
Baek, Joong-Hwan (School of Electronic & Information Eng., Korea Aerospace University)
Publication Information
Journal of Korea Multimedia Society / v.22, no.4, 2019 , pp. 424-431 More about this Journal
Abstract
In this paper, we propose a real-time hand gesture recognition algorithm to eliminate the inconvenience of using hand controllers in VR applications. The user's 3D hand coordinate information is detected by leap motion sensor and then the coordinates are generated into two dimensional image. We classify hand gestures in real-time by learning the imaged 3D hand coordinate information through SSD(Single Shot multibox Detector) model which is one of CNN(Convolutional Neural Networks) models. We propose to use all 3 channels rather than only one channel. A sliding window technique is also proposed to recognize the gesture in real time when the user actually makes a gesture. An experiment was conducted to measure the recognition rate and learning performance of the proposed model. Our proposed model showed 99.88% recognition accuracy and showed higher usability than the existing algorithm.
Keywords
Leap Motion; Deep Learning; VR; Gesture Recognition;
Citations & Related Records
Times Cited By KSCI : 3  (Citation Analysis)
연도 인용수 순위
1 J.J. Chae, J.H. Lim, H.S. Kim, and J.J. Lee "Study on Real-time Gesture Recognition Based on Convolutional Neural Network for Game Applications," Journal of Korea Multimedia Society, Vol. 20, No. 5, pp. 835-843, 2017.   DOI
2 M.J. Kim, J. Heo, J.H. Kim, S.Y. Park, and J. Chang, “Development and Evaluation of Leapmotion-based Game Interface Considering Intuitive Hand Gestures,” Journal of the Korean Society for Computer Game, Vol. 27, No. 4, pp. 69-75, 2014.
3 J. Gu and D. Shin, “An Empirical Evaluation of the Representative Hand Gesture of Task from User Perspective,” Archives of Design Research, Vol. 28, No. 1, pp. 133-145, 2015.
4 I. Sin, D. Cheon, and H. Park, "Implementing Leap-motion-based Interface for Enhancing the Realism of Shooter Games," Journal of The Human Computer Interaction Society of Korea, Vol. 22, No. 1, pp. 67-74, 2017.
5 H. Ju, M. Jo, S. In, G. Jo, and J. Min, "Development of Baseball Game Using Leap Motion Controllers," Korean Institute of Information Scientists and Engineers Transactions on Computing Practices, Vol. 21, No. 5, pp. 343-350, 2015.
6 M. Go, G. Lee, C. Kim, J. An, and I. Kim, "An Implementation of User Interface Using Vision-based Gesture Recognition," Korean Institute of Information Scientists and Engineers, Vol. 35, No.1, pp. 507-511, 2008.
7 J. Jeong, J. Kim, D. Kim, S. Gwon, S. Ju, W. O et. al., “A Hand Gesture Interface for Controlling 3D e-Books,” Journal of The Korean Society for Computer Game, Vol. 25, No. 4, pp. 119-127, 2012.
8 J. Suarez and R.R. Murphy, "Hand Gesture Recognition with Depth Images: A Review," Proceeding of 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, pp. 411-417, 2012.
9 H.K. Lee and J.H. Kim, “An HMM-based Threshold Model Approach for Gesture Recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 21, No. 10, pp. 961-973, 1999.   DOI
10 C. Cortes and V. Vapnik, “Support-vector Networks,” Machine Learning, Vol. 20, No. 3, pp. 273-297, 1995.   DOI
11 L. Breiman, "Random Forests," Machine Learning, Vol. 45, pp. 5-32, 2001.   DOI
12 J. Chae, J. Im, and J. Lee, “Gesture Classification Based on k-Nearest Neighbors Algorithm for Game Interface,” Journal of Korea Multimedia Society, Vol. 19, No. 5, pp. 874-880, 2016.   DOI
13 G. Marin, F. Dominio, and P. Zanuttigh, "Hand Gesture Recognition with Leap Motion and Kinect Devices," Proceeding of 2014 IEEE International Conference on Image Processing, pp. 1565-1569, 2014.
14 D. Song, D. Kim, and C. Lee, “The Chinese Characters Learning Contents Based on Gesture Recognition Using HMM Algorithim,” Journal of Korea Multimedia Society, Vol. 15, No. 8, pp. 1067-1074, 2012.   DOI
15 R. McCartney, J. Yuan, and H.P. Bischof, "Gesture Recognition with the Leap Motion Controller," Proceeding of International Conference on Image Processing, Computer Vision, and Pattern Recognition, pp. 3-9, 2015.