Browse > Article

Automatic Coarticulation Detection for Continuous Sign Language Recognition  

Yang, Hee-Deok (조선대학교 컴퓨터공학과)
Lee, Seong-Whan (고려대학교 컴퓨터.통신공학과)
Publication Information
Journal of KIISE:Software and Applications / v.36, no.1, 2009 , pp. 82-91 More about this Journal
Abstract
Sign language spotting is the task of detecting and recognizing the signs in a signed utterance. The difficulty of sign language spotting is that the occurrences of signs vary in both motion and shape. Moreover, the signs appear within a continuous gesture stream, interspersed with transitional movements between signs in a vocabulary and non-sign patterns(which include out-of-vocabulary signs, epentheses, and other movements that do not correspond to signs). In this paper, a novel method for designing a threshold model in a conditional random field(CRF) model is proposed. The proposed model performs an adaptive threshold for distinguishing between signs in the vocabulary and non-sign patterns. A hand appearance-based sign verification method, a short-sign detector, and a subsign reasoning method are included to further improve sign language spotting accuracy. Experimental results show that the proposed method can detect signs from continuous data with an 88% spotting rate and can recognize signs from isolated data with a 94% recognition rate, versus 74% and 90% respectively for CRFs without a threshold model, short-sign detector, subsign reasoning, and hand appearance-based sign verification.
Keywords
Sign language recognition; sign language spotting; conditional random field; threshold model;
Citations & Related Records
연도 인용수 순위
  • Reference
1 H.-K. Lee and J.-H. Kim, 'An HMM-based Threshold Model Approach for Gesture Recognition,' IEEE Trans. on Pattern Analysis and Machine Recognition, Vol. 21, No. 10, pp. 961-973, 1999   DOI   ScienceOn
2 J. Alon, V. Athitsos, and S. Sclaroff, 'Accurate And Efficient Gesture Spotting via Pruning and Subgesture Reasoning,' Proc. of ICCV-HCI, Beijing, China, pp. 199-207, Oct. 2005
3 C.W. Ong and S. Ranganath, 'Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning,' IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 27, No. 6, pp. 873-891, 2005   DOI   ScienceOn
4 E.-J. Holden, G. Lee, and R. Owens, 'Australian Sign Language Recognition,' Machine Vision and Applications, Vol. 1, No. 5, pp. 312-320, 2005   DOI
5 H. M. Wallach, 'Conditional Random Fields: An Introduction,' Technical Report MS-CIS-04-21, University of Pennsylvania, 2004
6 T. Kudo, CRF++: Yet Another CRF Toolkit, 2005, http://chasen.org/taku/software/CRF++/
7 W.C. Stokoe, Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf, Studies in Linguistics: Occasional Papers 8, Linstok Press, 1960
8 C. Vogler and D. Metaxas, 'A Framework for Recognizing the Simultaneous Aspects of American Sign Language,' Computer Vision and Image Understanding, Vol. 81, No. 3, pp. 358-384, 2001   DOI   ScienceOn
9 A. Braffort, 'Argo: An Architecture for Sign Language Recognition and Interpretation,' Proc. of Int. Gesture Workshop on Progress in Gestural Interaction, London, UK, pp. 17-30, 1996
10 M. Yang, N. Ahuja, and M. Tabb, 'Extraction of 2D Motion Trajectories and Its Application to Hand Gesture Recognition,' IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 24, No. 8, pp. 1061-1074, 2002   DOI   ScienceOn
11 L.-P. Morency, A. Quattoni, and T. Darrell, 'Latent-dynamic Discriminative Models for Continuous Gesture Recognition,' Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, Minneapolis, USA, 2007, pp. 1-8, http://sourceforge. net/projects/crf
12 A. McCallum, D. Freitag, and F. Pereira, 'Maximum Entropy Markov Models for Information Extraction and Segmentation,' Proc. of Int. Conf. on Machine Learning, Standford, USA, pp. 591-598, 2000
13 S. Wang, A. Quattoni, L.P. Morency, D. Demirdjian, and T. Darrell, 'Hidden Conditional Random Fields for Gesture Recognition,' Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, New York, USA, pp. 1521-1527, Jun. 2006   DOI
14 R.D. Yang and S. Sarkar, 'Detecting Coarticulation in Sign Language Using Conditional Random Fields,' Proc. of Int. Conf. on Pattern Recognition, Hong Kong, China, pp. 108-112, Aug. 2006   DOI
15 W. Gao, G. Fang, D. Zhao, and Y. Chen, 'Transition Movement Models for Large Vocabulary Continuous Sign Language Recognition,' Proc. of Int. Conf. on Automatic Face and Gesture Recognition, Seoul, Korea, pp. 553-558, May 2004   DOI
16 R. Bowden, D. Windridge, T. Kadir, A. Zisserman, and M. Brady, 'A Linguistic Feature Vector for the Visual Interpretation of Sign Language,' Proc. of European Conference on Computer Vision, Plague, Czech Republic, pp. 390-401, 2004
17 T. Starner, J. Weaver, and A. Pentland, 'Real- Time American Sign Language Recognition Using Desk and Wearable Computer Based Video,' IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 20, No. 12, pp. 1371-1375, 1998   DOI   ScienceOn
18 R. Kasturi and R. Jain, Computer Vision: Principles, IEEE Computer Society Press, 1991
19 C.-C. Chang and C.-J. Lin, LIBSVM: A Library for Support Vector Machine, 2001, http://www. csie.ntu.edu.tw/cjlin/libsvmtools/
20 H.-D. Yang, A.-Y. Park, and S.-W. Lee, 'Gesture Spotting and Recognition for Human-Robot Interaction,' IEEE Trans. on Robotics, Vol. 23, No. 2, pp. 256-270, 2007   DOI   ScienceOn
21 R.D. Yang, S. Sarkar, and B. Loeding, 'Enhanced Level Building Algorithm for the Movement Epenthesis Problem in Sign Language Recognition,' Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, Minnesota, USA, pp. 1-8, Aug. 2007   DOI   ScienceOn
22 J. Lafferty, A. McCallum, and F. Pereira, 'Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data,' Proc. of Int. Conf. on Machine Learning, Williamstown, USA, pp. 282-289, Jun. 2001