Browse > Article
http://dx.doi.org/10.3745/JIPS.02.0051

A Survey of Multimodal Systems and Techniques for Motor Learning  

Tadayon, Ramin (Center for Cognitive Ubiquitous Computing, School of Computing, Informatics, and Decision Systems Engineering, Arizona State University)
McDaniel, Troy (Center for Cognitive Ubiquitous Computing, School of Computing, Informatics, and Decision Systems Engineering, Arizona State University)
Panchanathan, Sethuraman (Center for Cognitive Ubiquitous Computing, School of Computing, Informatics, and Decision Systems Engineering, Arizona State University)
Publication Information
Journal of Information Processing Systems / v.13, no.1, 2017 , pp. 8-25 More about this Journal
Abstract
This survey paper explores the application of multimodal feedback in automated systems for motor learning. In this paper, we review the findings shown in recent studies in this field using rehabilitation and various motor training scenarios as context. We discuss popular feedback delivery and sensing mechanisms for motion capture and processing in terms of requirements, benefits, and limitations. The selection of modalities is presented via our having reviewed the best-practice approaches for each modality relative to motor task complexity with example implementations in recent work. We summarize the advantages and disadvantages of several approaches for integrating modalities in terms of fusion and frequency of feedback during motor tasks. Finally, we review the limitations of perceptual bandwidth and provide an evaluation of the information transfer for each modality.
Keywords
Augmented Motor Learning and Training; Multimodal Systems and Feedback; Rehabilitative Technologies;
Citations & Related Records
연도 인용수 순위
  • Reference
1 H. M. Huang, U. Rauch, and S. S. Liaw, "Investigating learners attitudes toward virtual reality learning environments: based on a constructivist approach," Computers & Education, vol. 55, no. 3, pp. 1171-1182, 2010.   DOI
2 M. K. Holden, "Virtual environments for motor rehabilitation: review," CyberPsychology & Behavior, vol. 8, no. 3, pp. 187-211, 2005.   DOI
3 B. Dalgarno and M. J. Lee, "What are the learning affordances of 3-D virtual environments?," British Journal of Educational Technology, vol. 41, no. 1, pp. 10-32, 2010.   DOI
4 M. R. Mouawad, C. G. Doust, M. D. Max, and P. A. McNulty, "Wii-based movement therapy to promote improved upper extremity function post-stroke: a pilot study," Journal of Rehabilitation Medicine, vol. 43, no. 6, pp. 527-533, 2011.   DOI
5 M. S. Cameirao, S. B. I. Badia, L. Zimmerli, E. Duarte Oller, and P. F. M. J. Verschure, "The rehabilitation gaming system: a virtual reality based system for the evaluation and rehabilitation of motor deficits," in Proceedings of the 6th 2007 International Workshop on Virtual Rehabilitation, Venice, Italy, 2007, pp. 29-33.
6 G. Saposnik and M. Levin, "Virtual reality in stroke rehabilitation a meta-analysis and implications for clinicians," Stroke, vol. 42, no. 5, pp. 1380-1386, 2011.   DOI
7 D. Jack, R. Boian, A. S. Merians, M. Tremaine, G. C. Burdea, S. V. Adamovich, M. Recce, and H. Poizner, "Virtual reality-enhanced stroke rehabilitation," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 9, no. 3, pp. 308-318, 2001.   DOI
8 A. S. Merians, D. Jack, R. Boian, M. Tremaine, G. C. Burdea, S. V. Adamovich, M. Recce, and H. Poizner, "Virtual reality-augmented rehabilitation for patients following stroke," Physical Therapy, vol. 82, no. 9, pp. 898-915, 2002.
9 K. LaBelle, "Evaluation of Kinect joint tracking for clinical and in-home stroke rehabilitation tools," Ph.D. dissertation, University of Notre Dame, IN, 2011.
10 B. Lange, C. Y. Chang, E. Suma, B. Newman, A. S. Rizzo and M. Bolas, "Development and evaluation of low cost game-based balance rehabilitation tool using the Microsoft Kinect sensor," in Proceedings of 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, 2011, pp. 1831-1834.
11 Y. J. Chang, S. F. Chen, and J. D. Huang, "A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities," Research in Developmental Disabilities, vol. 32, no. 6, pp. 2566-2570, 2011.   DOI
12 C. Y. Chang, B. Lange, M. Zhang, S. Koenig, P. Requejo, N. Somboon, A. A. Sawchuk, and A. A. Rizzo, "Towards pervasive physical rehabilitation using Microsoft Kinect," in Proceedings of 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops, San Diego, CA, 2012, pp. 159-162.
13 J. E. Deutsch, M. Borbely, J. Filler, K. Huhn, and P. Guarrera-Bowlby, "Use of a low-cost, commercially available gaming console (Wii) for rehabilitation of an adolescent with cerebral palsy," Physical Therapy, vol. 88, no. 10, pp. 1196-1207, 2008.   DOI
14 S. Brosnan, "The potential of Wii-rehabilitation for persons recovering from acute stroke," Physical Disabilities Special Interest Section Quarterly, vol. 32, no. 1, pp. 1-3, 2009.
15 A. Hartveld and J. R. Hegarty, "Augmented feedback and physiotherapy practice," Physiotherapy, vol. 82, no. 8, pp. 480-490, 1996.   DOI
16 H. S. Vitense, J. A. Jacko, and V. K. Emery, "Multimodal feedback: an assessment of performance and mental workload," Ergonomics, vol. 46, no. 1-3, pp. 68-87, 2003.   DOI
17 S. Holland, R. L. Wright, A. Wing, T. Crevoisier, O. Hodl, and M. Canelli, "A pilot study using tactile cueing for gait rehabilitation following stroke," in Proceedings of 2nd International Workshop on ICTs for Improving Patients Rehabilitation Research Techniques, Oldenburg, Germany, 2014, pp. 222-233.
18 R. Sigrist, G. Rauter, R. Riener, and P. Wolf, "Terminal feedback outperforms concurrent visual, auditory, and haptic feedback in learning a complex rowing-type task," Journal of Motor Behavior, vol. 45, no. 6, pp. 455-472, 2013.   DOI
19 H. B. Helbig and M. O. Ernst, "Visual-haptic cue weighting is independent of modality-specific attention," Journal of Vision, vol. 8, no. 1, pp. 21-21, 2008.
20 S. Oviatt, R. Coulston, and R. Lunsford, "When do we interact multimodally?: cognitive load and multimodal communication patterns," in Proceedings of 6th International Conference on Multimodal Interfaces, State College, PA, 2004, pp. 129-136.
21 C. Schonauer, K. Fukushi, A. Olwal, H. Kaufmann, and R. Raskar, "Multimodal motion guidance: techniques for adaptive and dynamic feedback," in Proceedings of 14th ACM International Conference on Multimodal Interaction, Santa Monica, CA, 2012, pp. 133-140.
22 B. Xiao, R. Lunsford, R. Coulston, M. Wesson, and S. Oviatt, "Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences," in Proceedings of 5th International Conference on Multimodal Interfaces, Vancouver, Canada, 2003, pp. 265-272.
23 M. Sun, X. Ren, and X. Cao, "Effects of multimodal error feedback on human performance in steering tasks," Information and Media Technologies, vol. 6, no. 1, pp. 193-201, 2011.
24 A. Rizzo and G. J. Kim, "A SWOT analysis of the field of virtual reality rehabilitation and therapy," Presence, vol. 14, no. 2, pp. 119-146, 2005.   DOI
25 M. S. Cameirao, S. B. I. Badia, E. Duarte Oller, and P. F. M. J. Verschure, "The rehabilitation gaming system: a review," Studies in Health Technology and Informatics, vol. 145, pp. 65-83, 2009.
26 A. Hayes, P. Dukes, and L. F. Hodges, "A virtual environment for post-stroke motor rehabilitation," 2012 [Online]. Available: https://www.cs.clemson.edu/nsfreu/documents/stroke_rehab_abstract.pdf.
27 M. Mihelj, D. Novak, M. Milavec, J. Ziherl, A. Olensek, and M. Munih, "Virtual rehabilitation environment using principles of intrinsic motivation and game design," Presence, vol. 21, no. 1, pp. 1-15, 2012.   DOI
28 K. Laver, S. George, S. Thomas, J. E. Deutsch, and M. Crotty, "Virtual reality for stroke rehabilitation," Stroke, vol. 43, no. 2, pp. e20-e21, 2012.   DOI
29 C. D. Wickens, "Multiple resources and performance prediction," Theoretical Issues in Ergonomics Science, vol. 3, no. 2, pp. 159-177, 2002.   DOI
30 B. Bongers and S. Smith, "Interactivating rehabilitation through active multimodal feedback and guidance," in Smart Health Applications and Services: Developments and Practices. Hershey, PA: IGI-Global, 2011, pp. 236-260.
31 J. L. Burke, M. S. Prewett, A. A. Gray, L. Yang, F. R. Stilson, M. D. Coovert, L. R. Elliot, and E. Redden, "Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis," in Proceedings of 8th International Conference on Multimodal Interfaces, Banff, Canada, 2006, pp. 108-117.
32 M. A. Guadagnoli and T. D. Lee, "Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning," Journal of Motor Behavior, vol. 36, no. 2, pp. 212-224, 2004.   DOI
33 F. Chen, N. Ruiz, E. Choi, J. Epps, M. A. Khawaja, R. Taib, B. Yin, and Y. Wang, "Multimodal behavior and interaction as indicators of cognitive load," ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 2, no. 4, article no. 22, 2012.
34 C. H. Anderson, D. C. van Essen, and B. A. Olshausen, "Directed visual attention and the dynamic control of information flow," in Neurobiology of Attention. Boston, MA: Elsevier Academic Press, 2005, pp. 11-17.
35 B. F. Miller and C. B. Keane, Encyclopedia and Dictionary of Medicine, Nursing, and Allied Health. Philadelphia, PA: W.B. Saunders, 1983.
36 M. S. Sanders and E. J. McCormick, Human Factors in Engineering and Design, 6th ed. New York, NY: McGraw-Hill, 1987.
37 D. C. Ruspini, K. Kolarov, and O. Khatib, "The haptic display of complex graphical environments," in Proceedings of 24th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, 1997, pp. 345-352.
38 Y. M. Aung and A. Al-Jumaily, "AR based upper limb rehabilitation system," in Proceedings of 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), Rome, 2012, pp. 213-218.
39 B. Lange, S. M. Flynn, and A. A. Rizzo, "Game-based telerehabilitation," European Journal of Physical and Rehabilitation Medicine, vol. 45, no. 1, pp. 143-151, 2009.
40 C. Kirner and T. G. Kirner, "Development of an interactive artifact for cognitive rehabilitation based on augmented reality," in Proceedings of 2011 IEEE International Conference on Virtual Rehabilitation (ICVR), Zurich, Switzerland, 2011, pp. 1-7.
41 J. Parker, S. Mawson, G. Mountain, N. Nasr, and H. Zheng, "Stroke patients utilisation of extrinsic feedback from computer-based technology in the home: a multiple case study realistic evaluation," BMC Medical Informatics and Decision Making, vol. 14, article no. 46, pp. 1-13, 2014.   DOI
42 L. Beursgens, A. Timmermans, and P. Markopoulos, "Playful arm hand training after stroke," in Proceedings of 2012 ACM Annual Conference on Human Factors in Computing Systems Extended Abstracts (CHI'12), Austin, TX, 2012, pp. 2399-2404.
43 R. Sigrist, G. Rauter, L. Marchal-Crespo, R. Riener, and P. Wolf, "Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning," Experimental Brain Research, vol. 233, no. 3, pp. 909-925, 2015.   DOI
44 J. Parker, S. Mawson, G. Mountain, N. Nasr, R. Davies, and H. Zheng, "The provision of feedback through computer-based technology to promote self-managed post-stroke rehabilitation in the home," Disability and Rehabilitation: Assistive Technology, vol. 9, no. 6, pp. 529-538, 2014.   DOI
45 J. Parker, G. Mountain, and J. Hammerton, "A review of the evidence underpinning the use of visual and auditory feedback for computer technology in post-stroke upper-limb rehabilitation," Disability and Rehabilitation: Assistive Technology, vol. 6, no. 6, pp. 465-472, 2011.   DOI
46 M. Ma and K. Bechkoum, "Serious games for movement therapy after stroke," in Proceedings of 2008 IEEE International Conference on Systems, Man and Cybernetics, Singapore, 2008, pp. 1872-1877.
47 R. G. Carson and J. S. Kelso, "Governing coordination: behavioural principles and neural correlates," Experimental Brain Research, vol. 154, no. 3, pp. 267-274, 2004.   DOI
48 J. C. Perry, H. Zabaleta, A. Belloso, C. Rodriguez-de Pablo, F. I. Cavallaro, and T. Keller, "ArmAssist: development of a functional prototype for at-home telerehabilitation of post-stroke arm impairment," in Proceedings of 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), Rome, 2012, pp. 1561-1566.
49 H. Kazemi, R. E. Kearney, and T. E. Milner, "Using a robotic interface and haptic feedback to improve grip coordination of hand function following stroke: case study," in Proceedings of 2013 IEEE International Conference on Virtual Rehabilitation (ICVR), Philadelphia, PA, 2013, pp. 196-197.
50 C. Rodriguez-de-Pablo, J. C. Perry, F. I. Cavallaro, H. Zabaleta, and T. Keller, "Development of computer games for assessment and training in post-stroke arm telerehabilitation," in Proceedings of 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA, 2012, pp. 4571-4574.
51 H. Z. Tan, C. M. Reed, and N. I. Durlach, "Optimum information transfer rates for communication through haptic and other sensory modalities," IEEE Transactions on Haptics, vol. 3, no. 2, pp. 98-108, 2010.   DOI
52 R. Sigrist, G. Rauter, R. Riener, and P. Wolf, "Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review," Psychonomic Bulletin & Review, vol. 20, no. 1, pp. 21-53, 2013.   DOI
53 A. Godbout and J. E. Boyd, "Corrective sonic feedback for speed skating: a case study," in Proceedings of 16th International Conference on Auditory Display, Washington, DC, 2010, pp. 23-30.
54 F. van Wijck, D. Knox, C. Dodds, G. Cassidy, G. Alexander, and R. MacDonald, "Making music after stroke: using musical activities to enhance arm function," Annals of the New York Academy of Sciences, vol. 1252, no. 1, pp. 305-311, 2012.   DOI
55 M. O. Belardinelli, S. Federici, F. Delogu, and M. Palmiero, "Sonification of spatial information: audio-tactile exploration strategies by normal and blind subjects," in Proceedings of 13th International Conference on Universal Access in Human-Computer Interaction, San Diego, CA, 2009, pp. 557-563.
56 S. J. Lederman and R. L. Klatzky, "Haptic perception: a tutorial," Attention, Perception, & Psychophysics, vol. 71, no. 7, pp. 1439-1459, 2009.   DOI
57 M. Slater, "How colorful was your day? Why questionnaires cannot assess presence in virtual environments," Presence, vol. 13, no. 4, pp. 484-493, 2004.   DOI
58 E. A. Alluisi, P. F. Muller, and P. M. Fitts, "An information analysis of verbal and motor responses in a forced-paced serial task," Journal of Experimental Psychology, vol. 53, no. 3, pp. 153-158, 1957.   DOI
59 H. Z. Tan, N. I. Durlach, C. M. Reed, and W. M. Rabinowitz, "Information transmission with a multifinger tactual display," Perception & Psychophysics, vol. 61, no. 6, pp. 993-1008, 1999.   DOI
60 M. J. Johnson, M. Trickey, E. Brauer, and X. Feng, "TheraDrive: a new stroke therapy concept for home-based, computer-assisted motivating rehabilitation," in Proceedings of the 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Francisco, CA, 2004, pp. 4844-4847.
61 R. Ruparel, M. J. Johnson, E. Strachota, J. McGuire, and G. Tchekanov, "Evaluation of the TheraDrive system for robot/computer assisted motivating rehabilitation after stroke," in Proceedings of 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN, 2009, pp. 811-814.
62 M. H. Milot, L. Marchal-Crespo, C. S. Green, S. C. Cramer, and D. J. Reinkensmeyer, "Comparison of error-amplification and haptic guidance training techniques for learning of a timing-based motor task by healthy individuals," Experimental Brain Research, vol. 201, no. 2, pp. 119-131, 2010.   DOI
63 I. Wallis, T. Ingalls, T. Rikakis, L. Olsen, Y. Chen, W. Xu, and H. Sundaram, "Real-time sonification of movement for an immersive stroke rehabilitation environment," in Proceedings of 13th International Conference on Auditory Display, Montreal, Canada, 2007, pp. 26-29.
64 R. Ronsse, V. Puttemans, J. P. Coxon, D. J. Goble, J. Wagemans, N. Wenderoth, and S. P. Swinnen, "Motor learning with augmented feedback: modality-dependent behavioral and neural consequences," Cerebral Cortex, vol. 21, no. 6, pp. 1283-1294, 2011.   DOI
65 A. Alamri, R. Iglesias, M. Eid, A. El Saddik, S. Shirmohammadi, and E. Lemaire, "Haptic exercises for measuring improvement of post-stroke rehabilitation patients," in Proceedings of 2007 IEEE International Workshop on Medical Measurement and Applications, Warsaw, Poland, 2007, pp. 1-6.
66 R. Garris, R. Ahlers, and J. E. Driskell, "Games, motivation, and learning: a research and practice model," Simulation & Gaming, vol. 33, no. 4, pp. 441-467, 2002.   DOI
67 G. Alankus, R. Proffitt, C. Kelleher, and J. Engsberg, "Stroke therapy through motion-based games: a case study," in Proceedings of 12th International ACM SIGACCESS Conference on Computers and Accessibility, Orlando, FL, 2010, pp. 219-226.
68 A. Jacobs, A. Timmermans, M. Michielsen, M. Vander Plaetse, and P. Markopoulos, "CONTRAST: gamification of arm-hand training for stroke survivors," in Proceedings of the 31st Annual CHI Conference on Human Factors in Computing Systems Presence Abstract (CHI'13), Paris, 2013, pp. 415-420.
69 G. Alankus and C. Kelleher, "Reducing compensatory motions in video games for stroke rehabilitation," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'12), Austin, TX, 2012, pp. 2049-2058.
70 K. Salen and E. Zimmerman, "Game design and meaningful play," in Handbook of Computer Game Studies. Cambridge, MA: MIT Press, 2005, pp. 59-79.
71 J. W. Burke, M. D. J. McNeill, D. K. Charles, P. J. Morrow, J. H. Crosbie, and S. M. McDonough, "Optimising engagement for stroke rehabilitation using serious games," The Visual Computer, vol. 25, no. 12, pp. 1085-1099, 2009.   DOI
72 M. C. Rhoads, G. B. Da Matta, N. Larson, and S. Pulos, "A meta-analysis of visual feedback for motor learning," Athletic Insight, vol. 6, no. 1, pp. 17-33, 2014.
73 M. Mihelj, T. Nef, and R. Riener, "A novel paradigm for patient-cooperative control of upper-limb rehabilitation robots," Advanced Robotics, vol. 21, no. 8, pp. 843-867, 2007.   DOI
74 D. Feygin, M. Keehner, and R. Tendick, "Haptic guidance: experimental evaluation of a haptic training method for a perceptual motor skill," in Proceedings of 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS2002), Orlando, FL, 2002, pp. 40-47.
75 G. Grindlay, "Haptic guidance benefits musical motor learning," in Proceedings of 2008 Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Reno, NE, 2008, pp. 397-404.
76 J. Wiemeyer and A. Kliem, "Serious games in prevention and rehabilitation: a new panacea for elderly people?," European Review of Aging and Physical Activity, vol. 9, no. 1, pp. 41-50, 2012.   DOI
77 D. Johnson and J. Wiles, "Effective affective user interface design in games," Ergonomics, vol. 46, no. 13-14, pp. 1332-1345, 2003.   DOI
78 S. Moya, S. Grau, D. Tost, R. Campeny and M. Ruiz, "Animation of 3D Avatars for rehabilitation of the upper limbs," in Proceedings of 2011 3rd International Conference on Games and Virtual Worlds for Serious Applications, Athens, Greece, 2011, pp. 168-171.
79 Z. Ruttkay and H. van Welbergen, "Elbows higher! Performing, observing and correcting exercises by a virtual trainer," in Proceedings of 8th International Conference on Intelligent Virtual Agents, Tokyo, Japan, 2008, pp. 409-416.
80 J. T. Ramsburg, K. Bharadwaj, S. Conner, N. Nandakumar, and A. Simes, "Augmented motor skills learning," 2015 [Online]. Available: https://www.researchgate.net/publication/281035957_Augmented_Motor_Skills_Learning.
81 K. Ogomori, M. Nagamachi, K. Ishihara, S. Ishihara, and M. Kohchi, "Requirements for a cognitive training game for elderly or disabled people," in Proceedings of 2011 International Conference on Biometrics and Kansei Engineering, Takamatsu, Japan, 2011, pp. 150-154.
82 E. Flores, G. Tobon, E. Cavallaro, F. I. Cavallaro, J. C. Perry, and T. Keller, "Improving patient motivation in game development for motor deficit rehabilitation," in Proceedings of 2008 International Conference on Advances in Computer Entertainment Technology, Yokohama, Japan, 2008, pp. 381-384.
83 P. Rego, P. M. Moreira, and L. P. Reis, "Serious games for rehabilitation: a survey and a classification towards a taxonomy," in Proceedings of 5th Iberian Conference on Information Systems and Technologies, Santiago de Compostela, Spain, 2010, pp. 1-6.
84 G. Alankus, A. Lazar, M. May, and C. Kelleher, "Towards customizable games for stroke rehabilitation," in Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI'10), Atlanta, GA, 2010, pp. 2113-2122.
85 J. L. Gonzalez Sanchez, N. Padilla Zea, and F. Gutierrez, "From usability to playability: introduction to playercentred video game development process," in Proceedings of 1st International Conference on Human Centered Design, San Diego, CA, 2009, pp. 65-74.
86 J. W. Burke, M. McNeill, D. Charles, P. Morrow, J. Crosbie, and S. McDonough, "Serious games for upper limb rehabilitation following stroke," in Proceedings of 2009 Conference in Games and Virtual Worlds for Serious Applications, Coventry, UK, 2009, pp. 103-110.
87 H. Bunt, R. J. Beun, and T. Borghuis, Multimodal Human-Computer Communication: Systems, Techniques, and Experiments. Heidelberg: Springer, 1998.
88 H. T. Jung, T. Takahashi, Y. K. Choe, J. Baird, T. Foster, and R. A. Grupen, "Towards extended virtual presence of the therapist in stroke rehabilitation," in Proceedings of 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR), Seattle, WA, 2013, pp. 1-6.
89 D. L. Eaves, G. Breslin, and P. van Schaik, "The short-term effects of real-time virtual reality feedback on motor learning in dance," Presence, vol. 20, no. 1, pp. 62-77, 2011.   DOI
90 A. Jaume-i-Capo, P. Martinez-Bueso, B. Moya-Alcover, and J. Varona, "Improving vision-based motor rehabilitation interactive systems for users with disabilities using mirror feedback," The Scientific World Journal, vol. 2014, article no. 964576, 2014.
91 F. Quek, D. McNeill, R. Bryll, S. Duncan, X. F. Ma, C. Kirbas, K. E. McCullough, and R. Ansari, "Multimodal human discourse: gesture and speech," ACM Transactions on Computer-Human Interaction (TOCHI), vol. 9, no. 3, pp. 171-193, 2002.   DOI
92 B. Paras, "Game, motivation, and effective learning: an integrated model for educational game design," in Proceedings of 2005 Digital Games Research Association's 2nd International Conference: Changing Views: Worlds in Play, Vancouver, Canada, 2005.
93 R. Tadayon, T. McDaniel, M. Goldberg, P. M. Robles-Franco, J. Zia, M. Laff, M. Geng, and S. Panchanathan, "Interactive motor learning with the autonomous training assistant: a case study," in Proceedings of 17th International Conference on Human-Computer Interaction, Los Angeles, CA, 2015, pp. 495-506.
94 M. Johnston, P. R. Cohen, D. McGee, S. L. Oviatt, J. A. Pittman, and I. Smith, "Unification-based multimodal integration," in Proceedings of 8th Conference on European Chapter of the Association for Computational Linguistics, Madrid, Spain, 1997, pp. 281-288.
95 A. Gouaich, N. Hocine, L. Van Dokkum, and D. Mottet, "Digital-pheromone based difficulty adaptation in post-stroke therapeutic games," in Proceedings of 2nd ACM SIGHIT International Health Informatics Symposium, Miami, FL, 2012, pp. 5-12.
96 J. Chen, "Flow in games (and everything else)," Communications of the ACM, vol. 50, no. 4, pp. 31-34, 2007.   DOI
97 B. Kim, H. Park, and Y. Baek, "Not just fun, but serious strategies: using meta-cognitive strategies in game-based learning," Computers & Education, vol. 52, no. 4, pp. 800-810, 2009.   DOI
98 E. Hudlicka, "Affective game engines: motivation and requirements," in Proceedings of 4th International Conference on Foundations of Digital Games (FDG'09), Orlando, FL, 2009, pp. 299-306.
99 K. Tanaka, J. R. Parker, G. Baradoy, D. Sheehan, J. R. Holash, and L. Katz, "A comparison of exergaming interfaces for use in rehabilitation programs and research," Loading..., vol. 6, no. 9, pp. 69-81, 2012.
100 G. Saposnik, R. Teasell, M. Mamdani, J. Hall, W. McIlroy, D. Cheung, K. E. Thorpe, L. G. Cohen, and M. Bayley, "Effectiveness of virtual reality using wii gaming technology in stroke rehabilitation a pilot randomized clinical trial and proof of principle," Stroke, vol. 41, no. 7, pp. 1477-1484, 2010.   DOI
101 Y. Song, L. P. Morency, and R. Davis, "Multimodal human behavior analysis: learning correlation and interaction across modalities," in Proceedings of 14th ACM International Conference on Multimodal Interaction, Santa Monica, CA, 2012, pp. 27-30.
102 L. Wu, S. L. Oviatt, and P. R. Cohen, "Multimodal integration-a statistical view," IEEE Transactions on Multimedia, vol. 1, no. 4, pp. 334-341, 1999.   DOI
103 J. Y. Chai, P. Hong, and M. X. Zhou, "A probabilistic approach to reference resolution in multimodal user interfaces," in Proceedings of 9th International Conference on Intelligent User Interfaces, Funchal, Portugal, 2004, pp. 70-77.
104 H. Mendonca, J. Y. L. Lawson, O. Vybornova, B. Macq, and J. Vanderdonckt, "A fusion framework for multimodal interactive applications," in Proceedings of 2009 International Conference on Multimodal Interfaces, Cambridge, MA, 2009, pp. 161-168.
105 L. Nigay and J. Coutaz, "A design space for multimodal systems: concurrent processing and data fusion," in Proceedings of the INTERACT'93 and CHI'93 Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands, 1993, pp. 172-178.
106 R. S. Schaefer, "Auditory rhythmic cueing in movement rehabilitation: findings and possible mechanisms," Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 369, article no. 20130402, pp. 1-9, 2014.