ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지
DOI QR코드

DOI QR Code

Selection of features and hidden Markov model parameters for English word recognition from Leap Motion air-writing trajectories

  • Deval Verma (School of Computer Science & Engineering, Bennett University) ;
  • Himanshu Agarwal (Department of Mathematics, Jaypee Institute of Information Technology) ;
  • Amrish Kumar Aggarwal (Department of Mathematics, Jaypee Institute of Information Technology)
  • Received : 2022.08.13
  • Accepted : 2022.12.26
  • Published : 2024.04.20
Download PDF
⟨ Previous Next ⟩

Abstract

Air-writing recognition is relevant in areas such as natural human-computer interaction, augmented reality, and virtual reality. A trajectory is the most natural way to represent air writing. We analyze the recognition accuracy of words written in air considering five features, namely, writing direction, curvature, trajectory, orthocenter, and ellipsoid, as well as different parameters of a hidden Markov model classifier. Experiments were performed on two representative datasets, whose sample trajectories were collected using a Leap Motion Controller from a fingertip performing air writing. Dataset D1 contains 840 English words from 21 classes, and dataset D2 contains 1600 English words from 40 classes. A genetic algorithm was combined with a hidden Markov model classifier to obtain the best subset of features. Combination ftrajectory, orthocenter, writing direction, curvatureg provided the best feature set, achieving recognition accuracies on datasets D1 and D2 of 98.81% and 83.58%, respectively.

Keywords

Acknowledgement

We thank Dr. Partha Pratim Roy and Dr. Rajkumar Saini from the Department of Computer Science and Engineering, Indian Institute of Technology (IIT), Roorkee, India, for their help with the data collection and valuable suggestions. The authors also acknowledge the research support provided by the Jaypee Institute of Information Technology, Noida, India.

References

  1. G. D'Amico, A. Del Bimbo, F. Dini, L. Landucci, and N. Torpei, Natural human-computer interaction, Multimedia interaction and intelligent user interfaces, Springer, 2010, pp. 85-106.
  2. M. Alam, K.-C. Kwon, M. Y. Abbass, S. M. Imtiaz, and N. Kim, Trajectory-based air-writing recognition using deep neural network and depth sensor, Sensors 20 (2020), no. 2, 376.
  3. D. H. Kim, H. I. Choi, and J. H. Kim, 3D space handwriting recognition with ligature model, International symposium on ubiquitious computing systems, Springer, 2006, pp. 41-56.
  4. M. Chen, G. AlRegib, and B.-H. Juang, Air-writing recognition part i: Modeling and recognition of characters, words, and connecting motions, IEEE Trans. Human-Mach. Syst. 46 (2015), no. 3, 403-413.
  5. P. P. Roy, A. K. Bhunia, A. Das, P. Dey, and U. Pal, HMMbased Indic handwritten word recognition using zone segmentation, Pattern Recog. 60 (2016), 1057-1075. https://doi.org/10.1016/j.patcog.2016.04.012
  6. R. Plamondon and S. N. Srihari, Online and off-line handwriting recognition: A comprehensive survey, IEEE Trans. Pattern Anal. Mach. Intell. 22 (2000), no. 1, 63-84.
  7. O. Patsadu, C. Nukoolkit, and B. Watanapa, Human gesture recognition using Kinect camera, (Ninth International Conference on Computer Science and Software Engineering, Bangkok, Thailand), 2012, pp. 28-32.
  8. S. Ghosh, S. Ghosh, P. Kumar, E. Scheme, and P. P. Roy, A novel spatio-temporal siamese network for 3D signature recognition, Pattern Recogn. Lett. 144 (2021), 13-20. https://doi.org/10.1016/j.patrec.2021.01.012
  9. E. Guerra-Segura, A. Ortega-Perez, and C. M. Travieso, In-air signature verification system using leap motion, Expert Syst. Appl. 165 (2021), 113797.
  10. J. Han, L. Shao, D. Xu, and J. Shotton, Enhanced computer vision with microsoft Kinect sensor: A review, IEEE Trans. Cybern. 43 (2013), no. 5, 1318-1334.
  11. F. Weichert, D. Bachmann, B. Rudak, and D. Fisseler, Analysis of the accuracy and robustness of the leap motion controller, Sensors 13 (2013), no. 5, 6380-6393.
  12. S. Patil, D. Kim, S. Park, and Y. Chai, Handwriting recognition in free space using WIMU-based hand motion analysis, J. Sensors 2016 (2016). https://doi.org/10.1155/2016/3692876
  13. M. Arsalan and A. Santra, Character recognition in air-writing based on network of radars for human-machine interface, IEEE Sensors J. 19 (2019), no. 19, 8855-8864.
  14. S. Mukherjee, S. A. Ahmed, D. P. Dogra, S. Kar, and P. P. Roy, Fingertip detection and tracking for recognition of air-writing in videos, Expert Syst. Appl. 136 (2019), 217-229. https://doi.org/10.1016/j.eswa.2019.06.034
  15. S. Mohammadi and R. Maleki, Air-writing recognition system for Persian numbers with a novel classifier, The Visual Comput. 36 (2020), no. 5, 1001-1015.
  16. T. Guzsvinecz, V. Szucs, and C. Sik-Lanyi, Suitability of the Kinect sensor and leap motion controller: A literature review, Sensors 19 (2019), no. 5, 1072.
  17. T. Yanay and E. Shmueli, Air-writing recognition using smartbands, Pervasive Mob. Comput. 66 (2020), 101183.
  18. H.-S. Yoon, J. Soh, Y. J. Bae, and H. S. Yang, Hand gesture recognition using combined features of location, angle and velocity, Pattern Recog. 34 (2001), no. 7, 1491-1501.
  19. Z. Feng, S. Xu, X. Zhang, L. Jin, Z. Ye, and W. Yang, Real-time fingertip tracking and detection using Kinect depth sensor for a new writing-in-the air system, (Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, Wuhan, China), 2012, pp. 70-74.
  20. X. Zhang, Z. Ye, L. Jin, Z. Feng, and S. Xu, A new writing experience: Finger writing in the air using a Kinect sensor, IEEE MultiMedia 20 (2013), no. 4, 85-93.
  21. T. Murata and J. Shin, Hand gesture and character recognition based on Kinect sensor, Int. J. Distrib. Sensor Netw. 10 (2014), no. 7, 278460.
  22. R. Aggarwal, S. Swetha, A. M. Namboodiri, J. Sivaswamy, and C. V. Jawahar, Online handwriting recognition using depth sensors, (13th International Conference on Document Analysis and Recognition, Tunis, Tunisia), 2015, pp. 1061-1065.
  23. C. Wang, C.-Y. Su, and C.-L. Lin, A novel recognition system for digits writing in the air using coordinated path ordering, (International Conference on Intelligent Informatics and Biomedical Sciences, Okinawa, Japan), 2015, pp. 244-249.
  24. S. Vikram, L. Li, and S. Russell, Handwriting and gestures in the air, recognizing on the fly, Proceedings of the CHI, Paris, france,13, 2013, pp. 1179-1184.
  25. N. Xu, W. Wang, and X. Qu, On-line sample generation for inair written chinese character recognition based on leap motion controller, (Pacific Rim Conference on Multimedia, Gwangju, Republic of Korea), 2015, pp. 171-180.
  26. C. Agarwal, D. P. Dogra, R. Saini, and P. P. Roy, Segmentation and recognition of text written in 3D using leap motion interface, 3rd IAPR Asian Conference on Pattern Recognition, Kuala Lumpur, Malaysia, 2015, pp. 539-543.
  27. P. Kumar, R. Saini, P. P. Roy, and D. P. Dogra, Study of text segmentation and recognition using leap motion sensor, IEEE Sensors J. 17 (2017), no. 5, 1293-1301.
  28. P. Kumar, R. Saini, P. P. Roy, and D. P. Dogra, 3D text segmentation and recognition using leap motion, Multimedia Tools Appl. 76 (2017), no. 15, 16491-16510.
  29. S. Misra, J. Singha, and R. H. Laskar, Vision-based hand gesture recognition of alphabets, numbers, arithmetic operators and ascii characters in order to develop a virtual text-entry interface system, Neural Comput. Appl. 29 (2018), no. 8, 117-135.
  30. J. K. Oh, S.-J. Cho, W.-C. Bang, W. Chang, E. Choi, J. Yang, J. Cho, and D. Y. Kim, Inertial sensor based recognition of 3-d character gestures with an ensemble classifiers, (Ninth international workshop on frontiers in handwriting recognition, Kokubunji, Japan), 2004, pp. 112-117.
  31. J.-S. Wang, Y.-L. Hsu, and C.-L. Chu, Online handwriting recognition using an accelerometer-based pen device, 2nd international conference on advances in computer science and engineering (cse 2013). Atlantis Press, 2013.
  32. C. Amma, M. Georgi, and T. Schultz, Airwriting: A wearable handwriting recognition system, Pers. Ubiquitous Comput. 18 (2014), no. 1, 191-203.
  33. D. Keysers, T. Deselaers, H. A. Rowley, L.-L. Wang, and V. Carbune, Multi-language online handwriting recognition, IEEE Trans. Pattern Anal. Machine Intell. 39 (2016), no. 6, 1180-1194.
  34. S. Jaeger, S. Manke, J. Reichert, and A. Waibel, Online handwriting recognition: The NPen++ recognizer, Int. J. Document Anal. Recogn. 3 (2001), no. 3, 169-180.
  35. P. P. Roy, P. Kumar, S. Patidar, and R. Saini, 3D word spotting using leap motion sensor, Multimedia Tools Appl. 80 (2021), no. 8, 11671-11689.
  36. L. R. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition, Proc. IEEE 77 (1989), no. 2, 257-286.
  37. I. Guyon, P. Albrecht, Y. Le Cun, J. Denker, and W. Hubbard, Design of a neural network character recognizer for a touch terminal, Pattern Recogn. 24 (1991), no. 2, 105-119.
  38. V. Ghods, E. Kabir, and F. Razzazi, Decision fusion of horizontal and vertical trajectories for recognition of online Farsi subwords, Eng. Appl. Artif. Intell. 26 (2013), no. 1, 544-550.
  39. F. Remondino, 3D reconstruction of static human body with a digital camera, 2003. Videometrics VII, 5013.
  40. A. Malyugina, K. Igudesman, and D. Chickrin, Least-squares fitting of a three-dimensional ellipsoid to noisy data, Appl. Math. Sci. 8 (2014), no. 149, 7409-7421.
  41. A. Bharath and S. Madhvanath, HMM-based lexicon-driven and lexicon-free word recognition for online handwritten Indic scripts, IEEE Trans. Pattern Anal. Mach. Intell. 34 (2012), no. 4, 670-682.
  42. J. O. Rawlings, S. G. Pantula, and D. A. Dickey, Applied regression analysis: A research tool, Springer Science & Business Media, 2001.
  43. G. M. Furnival and R. W. Wilson, Regressions by leaps and bounds, Technometrics 16 (1974), no. 4, 499-511.
  44. J. D. Foley, F. D. Van, A. Van Dam, S. K. Feiner, J. F. Hughes, J. Hughes, and E. Angel, Computer graphics: Principles and practice, 3rd ed., Addison-Wesley Professional, 1996.
  45. G. Calafiore, Approximation of n-dimensional data using spherical and ellipsoidal primitives, IEEE Trans. Syst., Man, Cybern.-Part A: Syst. Humans 32 (2002), no. 2, 269-278.
  46. J. Yu, S. R. Kulkarni, and H. V. Poor, Robust ellipse and spheroid fitting, Pattern Recogn. Lett. 33 (2012), no. 5, 492-499.
  47. D. A. Turner, I. J. Anderson, J. C. Mason, and M. G. Cox, An algorithm for fitting an ellipsoid to data, National Physical Laboratory, UK, 1999.
  48. G. Taubin, Estimation of planar curves, surfaces, and nonplanar space curves defined by implicit equations with applications to edge and range image segmentation, IEEE Trans. Pattern Anal. Mach. Intell. 11 (1991), 1115-1138. https://doi.org/10.1109/34.103273
  49. D. Dimov and I. Azmanov, Experimental specifics of using HMM in isolated word speech recognition, 2005.
  50. S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, and D. Povey, The HTK book, Cambridge Univers. Eng. Dep. 3 (2002), 175.
  51. H. Vafaie and K. A. De Jong, Genetic algorithms as a tool for feature selection in machine learning, (Proceedings Fourth International Conference on Tools with Artificial Intelligence, Arlington, VA, USA), 1992, pp. 200-203.
  52. J. Huang, Y. Cai, and X. Xu, A hybrid genetic algorithm for feature selection wrapper based on mutual information, Pattern Recogn. Lett. 28 (2007), no. 13, 1825-1844.
  53. L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamaria, M. A. Fadhel, M. Al-Amidie, and L. Farhan, Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions, J. Big Data 8 (2021), no. 1, 1-74.
  54. S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Comput. 9 (1997), no. 8, 1735-1780.
  55. M. Schuster and K. K. Paliwal, Bidirectional recurrent neural networks, IEEE Trans. Signal Process. 45 (1997), no. 11, 2673-2681.
  56. S. Mukherjee, S. A. Ahmed, D. P. Dogra, S. Kar, and P. P. Roy, Fingertip detection and tracking for recognition of air-writing in videos, Expert Syst. Appl. 136 (2019), 217-229. https://doi.org/10.1016/j.eswa.2019.06.034
  57. B. Graham, Sparse arrays of signatures for online character recognition, arXiv preprint, 2013. https://doi.org/10.48550/arXiv.1308.0371