Browse > Article
http://dx.doi.org/10.15701/kcgs.2018.24.5.41

3D Virtual Reality Game with Deep Learning-based Hand Gesture Recognition  

Lee, Byeong-Hee (Department of Computer Engineering, Seokyeong University)
Oh, Dong-Han (Department of Computer Engineering, Seokyeong University)
Kim, Tae-Young (Department of Computer Engineering, Seokyeong University)
Publication Information
Journal of the Korea Computer Graphics Society / v.24, no.5, 2018 , pp. 41-48 More about this Journal
Abstract
The most natural way to increase immersion and provide free interaction in a virtual environment is to provide a gesture interface using the user's hand. However, most studies about hand gesture recognition require specialized sensors or equipment, or show low recognition rates. This paper proposes a three-dimensional DenseNet Convolutional Neural Network that enables recognition of hand gestures with no sensors or equipment other than an RGB camera for hand gesture input and introduces a virtual reality game based on it. Experimental results on 4 static hand gestures and 6 dynamic hand gestures showed that they could be used as real-time user interfaces for virtual reality games with an average recognition rate of 94.2% at 50ms. Results of this research can be used as a hand gesture interface not only for games but also for education, medicine, and shopping.
Keywords
Hand Gesture Recognition; Deep Learning; Convolutional Neural Network; DenseNet; Virtual Reality; Game;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 윤종원, 민준기, 조성배, "몰입형 가상현실의 착용식 사용자 인터페이스를 위한 Mixture-of- Experts 기반 제스처 인식," 한국HCI학회 논문지, 6(1), pp. 1-8, 2011. 5.
2 나민영, 유휘종, 김태영, "스마트 디바이스 제어를 위한 비전 기반 실시간 손 포즈 및 제스처 인식방법," 한국차세대컴퓨팅학회 논문지, 8(4), pp.27-34, 2012. 8.
3 이새봄, 정일홍, "키넥트를 사용한 NUI 설계 및 구현," 한국디지털콘텐츠학회 논문지, 15(4), pp. 473-480, 2014. 8.
4 고택균, 윤민호, 김태영, "HMM과 MCSVM 기반 손 제스처 인터페이스 연구," 한국차세대컴퓨팅학회 논문지, 14(1), pp. 57-64, 2018. 2.
5 김민재, 허정만, 김진형, 박소영, 장준호, "직관적인 손동작을 고려한 립모션 기반 게임 인터페이스의 개발 및 평가," 한국컴퓨터게임학회 논문지, 27(4), pp. 69-75, 2014. 12.
6 문현철, 양안나, 김재곤, "웨어러블 응용을 위한 CNN 기반 손 제스처 인식," 방송공학회 논문지, 23(2), pp. 246-252, 2018. 3.
7 A. Sinha, C. Choi and K. Ramani, "DeepHand: Robust hand pose estimation by completing a matrix imputed with deep features," The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4150-4158, 2016.
8 P. Molchanov, S. Gupta, K. Kim and J. Kauts, "Hand gesture recognition with 3D convolutional neural networks," The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1-7, 2015.
9 G. Huang, Z. Liu, K. Q. Weinberger and L. van der Maaten, "Densely connected convolutional networks," The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 3-11, 2017.
10 C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D.Erhan, V. Vanhoucke and A. Rabinovich, "Going deeper with convolutions," The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015.
11 K. He, X. Zhang, S. Ren and J. Sun, "Deep residual learning for image recognition," The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
12 M. Lin, Q. Chen and S. Yan, "Network In Network," arXiv preprint arXiv:1312.4400, 2013.
13 Diederik P. Kingma, Jimmy Lei Ba, "Adam: A method for stochastic optimization," International Conference on Learning Representations (ICLR), pp. 1-15, 2015.
14 박경범, 이재열, "가상현실 환경에서 3D 가상객체 조작을 위한 인터페이스와 인터랙션 비교 연구," 한국CDE학회 논문집, 21(1), pp. 20-30, 2016. 3.