ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지
DOI QR코드

DOI QR Code

Human-like sign-language learning method using deep learning

  • Ji, Yangho (Department of Electrical Engineering, Kwangwoon University) ;
  • Kim, Sunmok (Department of Electrical Engineering, Kwangwoon University) ;
  • Kim, Young-Joo (Logistics System Research Team, Korea Railroad Research Institute) ;
  • Lee, Ki-Baek (Department of Electrical Engineering, Kwangwoon University)
  • Received : 2018.01.30
  • Accepted : 2018.05.14
  • Published : 2018.08.07
Download PDF
⟨ Previous Next ⟩

Abstract

This paper proposes a human-like sign-language learning method that uses a deep-learning technique. Inspired by the fact that humans can learn sign language from just a set of pictures in a book, in the proposed method, the input data are pre-processed into an image. In addition, the network is partially pre-trained to imitate the preliminarily obtained knowledge of humans. The learning process is implemented with a well-known network, that is, a convolutional neural network. Twelve sign actions are learned in 10 situations, and can be recognized with an accuracy of 99% in scenarios with low-cost equipment and limited data. The results show that the system is highly practical, as well as accurate and robust.

Keywords

References

  1. H. Liu et al., Gesture recognition for human-robot collaboration: A review, Int. J. of Ind. Ergon. 2017.
  2. H. Cooper et al., Sign language recognition using sub-units, J. Mach. Learn. Res. 13 (2017), no. 1, 2205-2231.
  3. D. Wu et al., Deep dynamic neural networks for multimodal gesture segmentation and recognition, IEEE Trans. Pattern Anal. Mach. Intell. 38 (2016), no. 8, 1583-1597. https://doi.org/10.1109/TPAMI.2016.2537340
  4. J. Huang et al., Sign language recognition using 3d convolutional neural networks, IEEE Int. Conf. Multimed. Expo, Turin, Italy, June29-July 3, 2015, pp. 1-6.
  5. D. Wu et al., Deep dynamic neural networks for gesture segmentation and recognition, Workshop Eur. Conf. Comput. Vis. 38 (2014), no. 8, 1583-1597.
  6. C. Xiujuan et al., Sign language recognition and translation with Kinect, IEEE Int. Conf. on AFGR 655 (2013).
  7. A. Agarwal et al., Sign language recognition using Microsoft Kinect, IEEE Int. Conf. Contemp. Comput., Noida, India, Aug. 8-10, 2013, pp. 181-185.
  8. I. Lim et al., Sign-language recognition through gesture & movement analysis (SIGMA), DLSU Res. Congress 2, Manila, Philippine, Mar. 2-4, 2015, p. HCT-I-011.
  9. L. Pigou et al., Sign language recognition using convolutional neural networks, Workshop Eur. Conf. Comput. Vis., Zurich, Swiss, Sept. 6-12, 2014, pp. 572-578.
  10. O. Koller et al., Deep Sign: Hybrid CNN-HMM for continuous sign language recognition, Proc. Br. Mach. Vis. Conf., York, UK, Sept. 19-22, 2016.
  11. N. Neverova et al., Multi-scale deep learning for gesture detection and localization, Workshop Eur. Conf. Comput. Vis., Zurich, Swiss, Sept. 6-12, 2014, pp. 1-17.
  12. O. Koller et al., Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers, Comput. Vis. Image Underst. 141 (2015), 108-125. https://doi.org/10.1016/j.cviu.2015.09.013
  13. M. Oliveira et al., A comparison between end-to-end approaches and feature extraction based approaches for sign language recognition, Int. Conf. on Image and Vis. Comput. New Zealand, Christchurch, New Zealand, Dec. 4-6, 2017, pp. 1-5.
  14. Y. L. Gweth et al., Enhanced continuous sign language recognition using PCA and neural network features, IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn. Workshops, Providence, RI, USA, June 16-21, 2012, pp. 55-60.
  15. J. Forster et al., Improving continuous: Speech recognition techniques and system design, Workshop Speech Lang. Process. Assist. Technol., Grenoble, France, Aug. 21-22, 2013, pp. 41-46.
  16. S. Jain et al., Indian sign language gesture recognition, 2015.
  17. A. K. Sahoo et al., Sign language recognition: state of the art, ARPN J. Eng. Applicat. Sci. 9 (2014), 116-134.
  18. J. Singha et al., Recognition of Indian sign language in live video, arXiv: 1306-1301, 2013.
  19. B. Garcia et al., Real-time American sign language recognition with convolutional neural networks, Convolutional Neural Netw. for Vis. Recogn. (2016), 225-232.
  20. O. Kang, Sign language (The most valuable language in the world) , Light and Fragrance, 2001.
  21. J.-Y. Lee et al., A real-time hand gesture recognition technique and its application to music display system, J. Autom. Contr. Eng. 4 (2016), no. 2, 177-180. https://doi.org/10.12720/joace.4.2.177-180
  22. O. Koller et al., Deep learning of mouth shapes for sign language, Proc. IEEE Int. Conf. Comput. Vis. Workshops, Santiago. Chile, Dec. 7-13, 2015, pp. 477-483.
  23. D. Weinland et al., Free viewpoint action recognition using motion history volumes, Comput. Vis. Image Underst. 104, (2006) 249-257. https://doi.org/10.1016/j.cviu.2006.07.013
  24. O. Russakovsky et al., Imagenet large scale visual recognition challenge, Int. J. of Comput. Vis. 115 (2015), 211-252. https://doi.org/10.1007/s11263-015-0816-y
  25. K. Simonyan et al., Very deep convolutional networks for largescale image recognition, arXiv preprint arXiv: 1409.1556, 2014.
  26. D. Kingma et al., A method for stochastic optimization, arXiv preprint arXiv: 1412.6980, 2014.
  27. S. Ioffe et al., Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv: 1502.03167, 2015.
  28. K. He et al., Deep residual learning for image recognition, Proc. IEEE Conf. Comput. Vis. Pattern Recogn., Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778.

Cited by

  1. Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning vol.19, pp.7, 2018, https://doi.org/10.3390/s19071716
  2. Deep Forest-Based Monocular Visual Sign Language Recognition vol.9, pp.9, 2018, https://doi.org/10.3390/app9091945
  3. OpenCL-Darknet: implementation and optimization of OpenCL-based deep learning object detection framework vol.24, pp.4, 2021, https://doi.org/10.1007/s11280-020-00778-y