Browse > Article
http://dx.doi.org/10.15701/kcgs.2021.27.5.81

Design of Handwriting-based Text Interface for Support of Mobile Platform Education Contents  

Cho, Yunsik (Department of Computer Engineering, Graduate School, Hansung University)
Cho, Sae-Hong (Division of Computer Engineering, Hansung University)
Kim, Jinmo (Division of Computer Engineering, Hansung University)
Publication Information
Journal of the Korea Computer Graphics Society / v.27, no.5, 2021 , pp. 81-89 More about this Journal
Abstract
This study proposes a text interface for support of language-based educational contents in a mobile platform environment. The proposed interface utilizes deep learning as an input structure to write words through handwriting. Based on GUI (Graphical User Interface) using buttons and menus of mobile platform contents and input methods such as screen touch, click, and drag, we design a text interface that can directly input and process handwriting from the user. It uses the EMNIST (Extended Modified National Institute of Standards and Technology database) dataset and a trained CNN (Convolutional Neural Network) to classify and combine alphabetic texts to complete words. Finally, we conduct experiments to analyze the learning support effect of the interface proposed by directly producing English word education contents and to compare satisfaction. We compared the ability to learn English words presented by users who have experienced the existing keypad-type interface and the proposed handwriting-based text interface in the same educational environment, and we analyzed the overall satisfaction in the process of writing words by manipulating the interface.
Keywords
text interface; EMNIST; CNN; deep learning; mobile; English education contents;
Citations & Related Records
연도 인용수 순위
  • Reference
1 A. van Dam, "Post-wimp user interfaces," Commun. ACM, vol. 40, no. 2, p. 63-67, feb 1997. [Online]. Available: https://doi.org/10.1145/253671.253708   DOI
2 H.-Y. Huang, C.-W. Ning, P.-Y. Wang, J.-H. Cheng, and L.-P. Cheng, "Haptic-go-round: A surrounding platform for encounter-type haptics in virtual reality experiences," in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ser. CHI '20. New York, NY, USA: Association for Computing Machinery, 25-30 April 2020, p. 1-10. [Online]. Available: https://doi.org/10.1145/3313831.3376476   DOI
3 J.-H. Sung and T. Y. Kim, "Leap motion-based playing yut through hand gesture interaction," The Journal of Korean Institute of Next Generation Computing, vol. 13, no. 1, pp. 74-81, 2017.
4 Y. Cho, J. Kang, J. Jeon, J. Park, M. Kim, and J. Kim, "X-person asymmetric interaction in virtual and augmented realities," Computer Animation and Virtual Worlds, vol. 32, no. 5, p. e1985, 2021. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/cav.1985   DOI
5 Y. Cho and J. Kim, "Production of mobile english language teaching application based on text interface using deep learning," Electronics, vol. 10, no. 15, 2021. [Online]. Available: https://www.mdpi.com/2079-9292/10/15/1809
6 R. Jayakanthan, "Application of computer games in the field of education," Electronic Library, vol. 20, no. 2, pp. 98-102, 2002. [Online]. Available: https://www.learntechlib.org/p/95865   DOI
7 V. S. Zirawaga, A. I. Olusanya, and T. Maduku, "Gaming in education: Using games as a support tool to teach history," Journal of Education and Practice, vol. 8, no. 15, pp. 55-64, 2017.
8 D. Aminatun and L. Oktaviani, "Memrise: Promoting students' autonomous learning skill through language learning application," Metathesis: Journal of English Language, Literature, and Teaching, vol. 3, no. 2, pp. 214-223, 2019.   DOI
9 M. Fan, A. N. Antle, and J. L. Warren, "Augmented reality for early language learning: A systematic review of augmented reality application design, instructional strategies, and evaluation outcomes," Journal of Educational Computing Research, vol. 58, no. 6, pp. 1059-1100, 2020. [Online]. Available: https://doi.org/10.1177/0735633120927489   DOI
10 J. Jun, J. Lim, and S. Kim, "Investigating the educational effectiveness of an online educational game," The journal of Educational Studies, vol. 39, no. 1, pp. 85-105, 2008.
11 K. V. Nesbitt and I. Hoskens, "Multi-sensory game interface improves player satisfaction but not performance," in Proceedings of the Ninth Conference on Australasian User Interface - Volume 76, ser. AUIC '08. AUS: Australian Computer Society, Inc., 2008, p. 13-18.
12 N. Sidorakis, G. A. Koulieris, and K. Mania, "Binocular eye-tracking for the control of a 3d immersive multimedia user interface," in 2015 IEEE 1st Workshop on Everyday Virtual Reality (WEVR), 23 March 2015, pp. 15-18.
13 I. Choi, E. Ofek, H. Benko, M. Sinclair, and C. Holz, "Claw: A multifunctional handheld haptic controller for grasping, touching, and triggering in virtual reality," in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ser. CHI '18. New York, NY, USA: ACM, 21-26 April 2018, pp. 654:1-654:13. [Online]. Available: http://doi.acm.org/10.1145/3173574.3174228   DOI
14 A. Baldominos, Y. Saez, and P. Isasi, "A survey of handwritten character recognition with mnist and emnist," Applied Sciences, vol. 9, no. 15, 2019. [Online]. Available: https://www.mdpi.com/2076-3417/9/15/3169
15 S. Hong, G. Na, Y. Cho, and J. Kim, "A study on movement interface in mobile virtual reality," Journal of the Korea Computer Graphics Society, vol. 27, no. 3, pp. 55-63, 2021.
16 J.-H. Kim, "Interaction technique in smoke simulations using mouth-wind on mobile devices," Journal of the Korea Computer Graphics Society, vol. 24, no. 4, pp. 21-27, 2018.   DOI
17 J. Kim, "A study on comparative experiment of hand-based interface in immersive virtua reality," Journal of the Korea Computer Graphics Society, vol. 25, no. 2, pp. 1-9, 2019.
18 J. Kim, "Vivr: Presence of immersive interaction for visual impairment virtual reality," IEEE Access, vol. 8, pp. 196 151-196 159, 2020.   DOI
19 G. Cohen, S. Afshar, J. Tapson, and A. van Schaik, "Emnist: Extending mnist to handwritten letters," in 2017 International Joint Conference on Neural Networks (IJCNN), 14-19 May 2017, pp. 2921-2926.
20 S. Yang and M. Zhang, "Application of brain neural network in personalized english education system," International Journal of Emerging Technologies in Learning (iJET), vol. 13, no. 10, pp. 15-22, 2018. [Online]. Available: https://online-journals.org/index.php/i-jet/article/view/9488
21 A. Lund, "Measuring usability with the use questionnaire," Usability Interface, vol. 8, no. 2, pp. 3-6, 01 2001.
22 S. S. Mor, S. Solanki, S. Gupta, S. Dhingra, M. Jain, and R. Saxena, "Handwritten text recognition: With deep learning and android," International Journal of Engineering and Advanced Technology (IJEAT), vol. 8, no. 3S, pp. 819-825, 2019. [Online]. Available: https://www.ijeat.org/wpcontent/uploads/papers/v8i3S/C11730283S19.pdf   DOI
23 UnityTechnologies, "Unity engine," Unity Technologies, 2019. [Online]. Available: https://unity3d.com/
24 GoogleBrain, "Tensorflow," Google Brain, 2021. [Online]. Available: https://www.tensorflow.org/
25 Shneiderman, "Direct manipulation: A step beyond programming languages," Computer, vol. 16, no. 8, pp. 57-69, 1983.   DOI
26 Y. Chen, D. Zhou, Y. Wang, and J. Yu, "Application of augmented reality for early childhood english teaching," in 2017 International Symposium on Educational Technology (ISET), 2017, pp. 111-115.
27 M. Kim, J. Lee, C. Jeon, and J. Kim, "A study on interaction of gaze-based user interface in mobile virtual reality environment," Journal of the Korea Computer Graphics Society, vol. 23, no. 3, pp. 39-46, 2017. [Online]. Available: http://dx.doi.org/10.15701/kcgs.2017.23.3.39   DOI
28 M. Kim, J. Kim, K. Jeong, and C. Kim, "Grasping vr: Presence of pseudo-haptic interface based portable hand grip system in immersive virtual reality," International Journal of Human-Computer Interaction, vol. 36, no. 7, pp. 685-698, 2020.   DOI
29 S. E. Lindley, J. Le Couteur, and N. L. Berthouze, "Stirring up experience through movement in game play: Effects on engagement and social behaviour," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI '08. New York, NY, USA: Association for Computing Machinery, 2008, p. 511-514. [Online]. Available: https://doi.org/10.1145/1357054.1357136   DOI
30 T. Sweeney, "Foundational principles & technologies for the metaverse," in ACM SIGGRAPH 2019 Talks, ser. SIGGRAPH '19. New York, NY, USA: Association for Computing Machinery, 28 July-1 August 2019. [Online]. Available: https://doi.org/10.1145/3306307.3339844   DOI
31 B. D. Song, H.-J. Kim, H.-J. Jeong, and Y. J. Choi, "Development of interactive media player for kiosk with user motion detection," The Journal of the Korea Contents Association, vol. 19, no. 11, pp. 270-277, 2019.   DOI
32 T. Kang, M. Chae, E. Seo, M. Kim, and J. Kim, "Deep-handsvr: Hand interface using deep learning in immersive virtual reality," Electronics, vol. 9, no. 11, 2020. [Online]. Available: https://www.mdpi.com/2079-9292/9/11/1863
33 남현숙, "2020년 국외 디지털콘텐츠 시장조사 및 동향 심층분석," 소프트웨어정책연구소 연구보고서, vol. RE-107, pp. 1-378, 2021.
34 M. Lu, "Effectiveness of vocabulary learning via mobile phone," Journal of Computer Assisted Learning, vol. 24, no. 6, pp. 515-525, 2008. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1365-2729.2008.00289.x   DOI