Browse > Article
http://dx.doi.org/10.4218/etrij.2018-0066

Human-like sign-language learning method using deep learning  

Ji, Yangho (Department of Electrical Engineering, Kwangwoon University)
Kim, Sunmok (Department of Electrical Engineering, Kwangwoon University)
Kim, Young-Joo (Logistics System Research Team, Korea Railroad Research Institute)
Lee, Ki-Baek (Department of Electrical Engineering, Kwangwoon University)
Publication Information
ETRI Journal / v.40, no.4, 2018 , pp. 435-445 More about this Journal
Abstract
This paper proposes a human-like sign-language learning method that uses a deep-learning technique. Inspired by the fact that humans can learn sign language from just a set of pictures in a book, in the proposed method, the input data are pre-processed into an image. In addition, the network is partially pre-trained to imitate the preliminarily obtained knowledge of humans. The learning process is implemented with a well-known network, that is, a convolutional neural network. Twelve sign actions are learned in 10 situations, and can be recognized with an accuracy of 99% in scenarios with low-cost equipment and limited data. The results show that the system is highly practical, as well as accurate and robust.
Keywords
CNN; deep learning; sign language;
Citations & Related Records
연도 인용수 순위
  • Reference
1 J. Singha et al., Recognition of Indian sign language in live video, arXiv: 1306-1301, 2013.
2 B. Garcia et al., Real-time American sign language recognition with convolutional neural networks, Convolutional Neural Netw. for Vis. Recogn. (2016), 225-232.
3 O. Kang, Sign language (The most valuable language in the world) , Light and Fragrance, 2001.
4 J.-Y. Lee et al., A real-time hand gesture recognition technique and its application to music display system, J. Autom. Contr. Eng. 4 (2016), no. 2, 177-180.   DOI
5 D. Weinland et al., Free viewpoint action recognition using motion history volumes, Comput. Vis. Image Underst. 104, (2006) 249-257.   DOI
6 O. Russakovsky et al., Imagenet large scale visual recognition challenge, Int. J. of Comput. Vis. 115 (2015), 211-252.   DOI
7 K. Simonyan et al., Very deep convolutional networks for largescale image recognition, arXiv preprint arXiv: 1409.1556, 2014.
8 D. Kingma et al., A method for stochastic optimization, arXiv preprint arXiv: 1412.6980, 2014.
9 S. Ioffe et al., Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv: 1502.03167, 2015.
10 K. He et al., Deep residual learning for image recognition, Proc. IEEE Conf. Comput. Vis. Pattern Recogn., Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778.
11 A. Agarwal et al., Sign language recognition using Microsoft Kinect, IEEE Int. Conf. Contemp. Comput., Noida, India, Aug. 8-10, 2013, pp. 181-185.
12 O. Koller et al., Deep learning of mouth shapes for sign language, Proc. IEEE Int. Conf. Comput. Vis. Workshops, Santiago. Chile, Dec. 7-13, 2015, pp. 477-483.
13 D. Wu et al., Deep dynamic neural networks for multimodal gesture segmentation and recognition, IEEE Trans. Pattern Anal. Mach. Intell. 38 (2016), no. 8, 1583-1597.   DOI
14 J. Huang et al., Sign language recognition using 3d convolutional neural networks, IEEE Int. Conf. Multimed. Expo, Turin, Italy, June29-July 3, 2015, pp. 1-6.
15 D. Wu et al., Deep dynamic neural networks for gesture segmentation and recognition, Workshop Eur. Conf. Comput. Vis. 38 (2014), no. 8, 1583-1597.
16 C. Xiujuan et al., Sign language recognition and translation with Kinect, IEEE Int. Conf. on AFGR 655 (2013).
17 I. Lim et al., Sign-language recognition through gesture & movement analysis (SIGMA), DLSU Res. Congress 2, Manila, Philippine, Mar. 2-4, 2015, p. HCT-I-011.
18 L. Pigou et al., Sign language recognition using convolutional neural networks, Workshop Eur. Conf. Comput. Vis., Zurich, Swiss, Sept. 6-12, 2014, pp. 572-578.
19 H. Cooper et al., Sign language recognition using sub-units, J. Mach. Learn. Res. 13 (2017), no. 1, 2205-2231.
20 H. Liu et al., Gesture recognition for human-robot collaboration: A review, Int. J. of Ind. Ergon. 2017.
21 O. Koller et al., Deep Sign: Hybrid CNN-HMM for continuous sign language recognition, Proc. Br. Mach. Vis. Conf., York, UK, Sept. 19-22, 2016.
22 Y. L. Gweth et al., Enhanced continuous sign language recognition using PCA and neural network features, IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn. Workshops, Providence, RI, USA, June 16-21, 2012, pp. 55-60.
23 N. Neverova et al., Multi-scale deep learning for gesture detection and localization, Workshop Eur. Conf. Comput. Vis., Zurich, Swiss, Sept. 6-12, 2014, pp. 1-17.
24 O. Koller et al., Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers, Comput. Vis. Image Underst. 141 (2015), 108-125.   DOI
25 M. Oliveira et al., A comparison between end-to-end approaches and feature extraction based approaches for sign language recognition, Int. Conf. on Image and Vis. Comput. New Zealand, Christchurch, New Zealand, Dec. 4-6, 2017, pp. 1-5.
26 J. Forster et al., Improving continuous: Speech recognition techniques and system design, Workshop Speech Lang. Process. Assist. Technol., Grenoble, France, Aug. 21-22, 2013, pp. 41-46.
27 S. Jain et al., Indian sign language gesture recognition, 2015.
28 A. K. Sahoo et al., Sign language recognition: state of the art, ARPN J. Eng. Applicat. Sci. 9 (2014), 116-134.