Browse > Article
http://dx.doi.org/10.5143/JESK.2006.25.2.135

Design of Parallel Input Pattern and Synchronization Method for Multimodal Interaction  

Im, Mi-Jeong (아주대학교)
Park, Beom (아주대학교)
Publication Information
Journal of the Ergonomics Society of Korea / v.25, no.2, 2006 , pp. 135-146 More about this Journal
Abstract
Multimodal interfaces are recognition-based technologies that interpret and encode hand gestures, eye-gaze, movement pattern, speech, physical location and other natural human behaviors. Modality is the type of communication channel used for interaction. It also covers the way an idea is expressed or perceived, or the manner in which an action is performed. Multimodal Interfaces are the technologies that constitute multimodal interaction processes which occur consciously or unconsciously while communicating between human and computer. So input/output forms of multimodal interfaces assume different aspects from existing ones. Moreover, different people show different cognitive styles and individual preferences play a role in the selection of one input mode over another. Therefore to develop an effective design of multimodal user interfaces, input/output structure need to be formulated through the research of human cognition. This paper analyzes the characteristics of each human modality and suggests combination types of modalities, dual-coding for formulating multimodal interaction. Then it designs multimodal language and input synchronization method according to the granularity of input synchronization. To effectively guide the development of next-generation multimodal interfaces, substantially cognitive modeling will be needed to understand the temporal and semantic relations between different modalities, their joint functionality, and their overall potential for supporting computation in different forms. This paper is expected that it can show multimodal interface designers how to organize and integrate human input modalities while interacting with multimodal interfaces.
Keywords
Multimodal interaction; Human-computer interaction; Human modality I/O design;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Cohen, P. R. and Johnston, M., 'QuickSet: Multimodal Interaction for Simulation Set-up and Control,' in Proceedings of the fifth conference on Applied natural language processing, 20-24, 1997
2 Adam Cheter, Luc Julia, 'MVIEW: Multimodal Tools for the Video Analyst,' in Proceeding of IUI98, 55-62, 1998
3 Somsak Walairacht, '4 + 4 fingers manipulating virtual objects in mixedreality environment,' Presence: Teleoperators and Virtual Environments, 11, 2002
4 Buchmann, S., Violich, M. and Billinghurst, A., Cockburn. 'FingARtips: gesture based direct manipulation in Augmented Reality,' In Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and SouthEast Asia (Graphite 2004), ACM Press, 212-221, 2004
5 Dizio, P., Proprioceptive Adaptation and Aftereffects. In Handbook of Virtual Environments, 751-771, 2002
6 Hauptmann, A. G. and McAvinney, P., 'Gesture with speech for graphics manipulation,' Int. J. Man-Machine Studies, 38, 231-249, 1993   DOI   ScienceOn
7 Oviatt, S., DeAngeli, A. and Kuhn, K., 'Integration and synchronization of input modes during multimodal human-computer interaction,' in Proc. Conf. human Factors in Computing Systems(CHI'97), Atlanta, GA, 415-422, 1997
8 Cohen, P. R., Darlymple, M., Pereira, F. C. N., Sullivan, J. W., Gargan, Jr. R. A., Schlossberg, J. L. and Tyler, S. W., 'Synergic use of direct manipulation and natural language,' in Proc. Conf. human Factors in Computing Systems(CHI '89), Austin, TX, 227-233, 1989
9 존 R.앤더슨 著., 李永愛 譯., 認知心理學 (Cognitive psychology and its implications), 乙酉文化社, 88-91, 1987
10 Sharma, R., 'Toward multimodal human-computer interface,' in Proc. IEEE, 86(5), 853-869, 1998
11 Alan Wexelblat, 'An approach to natural gesture in virtual environments,' ACM Transactions on Computer-Human Interaction(TOCHI), 2, 179-200, 1995   DOI   ScienceOn
12 Klatzky, R. and Lederman, S., Touch., In Handbook of Psychology, l.4, 147-176, 2003
13 Andrea Corradini, Richard M. Wesson, Philip R. Cohen, 'A Map-based System Using Speech and 3D Gestures for Pervasive computing,' in Proceedings of the 4th IEEE International Conference on Multimodal Interface, IEEE Computer Society, 191, 2002
14 Oviatt, S. L., Multimodal Interfaces, Handbook of Human-Computer Interface, Ed. By J.Jacko & A.Sears, Lawrence Erlbaum: New Jersey, 2002
15 Koons, D. B., Sparrell, C. J. and Thorisson, K. R., 'Integrating simultaneous input from speech, gaze, and hand gestures. M.Maybury(Ed.),' Intelligent Multimodal Interfaces, 257-276, Menlo Park, CA: MIT, 1993
16 Qiaohui Zhang, Atsumi Imamiya, Kentaro Go, Xiaoyang Mao, 'Gaze and Speech Multimodal Interface,' Int. Conf. on Distributed Computing Systems Workshops(ICDCSW'04), 208-214
17 Oviatt, S. L., Mutual disambiguation of recognition errors in a multimodal Architecture, in Proc. CHI'99 Human Factors in Computing Systems Conf., Pittsburgh, PA, 576-583, 1999
18 Oviatt, S. L., 'User-Centered Modeling and Evaluation of Multimodal Interfaces,' in Proc. of the IEEE, 91(9), 1457-1468, 2003
19 http://www.w3.org/TR/2003/NOTE-mmi-reqs-20030108/
20 http://www.w3.org/TR/mmi-framework/
21 Jennifer L. Leopold and allen L. Ambler, 'Keyboardless Visual Programming Using Voice, Handwriting, and Gesture', in Proc. of the 1997 IEEE Symposium on Visual Languages(VL '97), 28-35, 1997
22 Anthony G. Greenwald, 'A Reminder about procedures needed to reliably produce perfect timesharing: Comment on Lien, McCann, Ruthruff, and Proctor,' in Journal of Experimental Psychology: Human Perception and Performance, 31(1), 221-225, 2005   DOI   ScienceOn