Browse > Article
http://dx.doi.org/10.7471/ikeee.2018.22.2.362

Implementation of Mutual Conversion System between Body Movement and Visual·Auditory Information  

Bae, Myung-Jin (Dept. of Convergence IT Engineering, Kyungnam University)
Kim, Sung-Ill (Dept. of Electronic Engineering, Kyungnam University)
Publication Information
Journal of IKEEE / v.22, no.2, 2018 , pp. 362-368 More about this Journal
Abstract
This paper has implemented a mutual conversion system that mutually converts between body motion signals and both visual and auditory signals. The present study is based on intentional synesthesia that can be perceived by learning. The Euler's angle was used in body movements as the output of a wearable armband(Myo). As a muscle sense, roll, pitch and yaw signals were used in this study. As visual and auditory signals, MIDI(Musical Instrument Digital Interface) signals and HSI(Hue, Saturation, Intensity) color model were used respectively. The method of mutual conversion between body motion signals and both visual and auditory signals made it easy to infer by applying one-to-one correspondence. Simulation results showed that input motion signals were compared with output simulation ones using ROS(Root Operation System) and Gazebo which is a 3D simulation tool, to enable the mutual conversion between body motion information and both visual and auditory information.
Keywords
Visualization; Sonification; Mutual Conversion; Human Movements; ROS; MIDI; HSI;
Citations & Related Records
Times Cited By KSCI : 3  (Citation Analysis)
연도 인용수 순위
1 K. Jun. "Synesthetic Expression of Sounds and Colors," Master's Thesis of Ewha Womans University, 2004.
2 Y. Kim, "A Study on the Factors That Create Synesthesia," Doctor's Thesis of Hong-Ik Universisy, 2005.
3 M. Bae, S. Kim, "Implementation of Muscular Sense into both Color and Sound Conversion System based on Wearable Device," Journal of Korea Multimedia Society, Vol. 19, No. 3, pp.642-649, 2016.DOI: 10.9717/kmms.2016.19.3.642   DOI
4 Suslick, S Kenneth. "Synesthesia in science and technology: more than making the unseen visible," Current opinion in chemical biology Vol. 16, No. 5, pp. 557-563, 2012.DOI:10.1016/j.cbpa.2012.10.030   DOI
5 J. Jang, "The Study on Visual Language for Visualizing Taste," Master's Thesis of Yonsei University, 2017.
6 S. Cho, "Developing Digital Contents Using Multimedia Tehnology", Digital Contents Society, Vol. 1, No. 1, pp. 103-110, 2000.
7 J. Baek, "Media Art Musical Sonification based on Cityscope," Master's Thesis of Chung-Ang University, 2015.
8 S. Kim, "Conversion of Image into Sound Based on HSI Histogram," The Journal of the Acoustical Society of Korea, Vol. 30, No. 3, pp. 142-148, 2011.DOI:10.7776/ASK.2011.30.3.142   DOI
9 D. Bor, N. Rothen, D. Schwartzman, S. Clayton, A. Seth, "Adults Can Be Trained to Acquire Synesthetic Experiences," Scientific Reports 4, No. 7089, 2014.DOI:10.1038/srep07089   DOI
10 Witthoft, Nathan, and Jonathan Winawer. "Learning, memory, and synesthesia." Psychological science, Vol. 24, No. 3, pp. 258-265, 2013.DOI:10.1177/0956797612452573   DOI
11 I. Zamora, N. G. Lopez, V. M. Vilches, and A. H. Cordero, "Extending the OpenAI Gym for robotics: a toolkit for reinforcement learning using ROS and Gazebo," arXiv:1608.05742v2 [cs.RO], Feb. 2017.
12 M. Bae, S. Kim, "Implementation of the Visualization and Sonification System of Human Movements based on Intentional Synesthesia," Journal of Korean institute of intelligent systems, Vol. 28, No. 1, pp. 83-90, 2018.DOI:10.5391/JKIIS.2018.28.1.83   DOI
13 J. Messias, R. Ventura, P. Lima, and J. Sequeira, "ROS in the MOnarCH Project: A Case Study in Networked Robot Systems," Robot Operating System (ROS) Springer International Publishing, pp.375-395, 2016.DOI:10.1007/978-3-319-26054-9_14   DOI
14 K. Nathan and A. Howard, "Design and use paradigms for gazebo, an open-source multi-robot simulator," Proc. Intelligent Robots and Systems, vol3, 2004.DOI:10.1109/IROS.2004.1389727   DOI
15 T. Kenta, A. Toshinori el al. "Simulation Environment for Mobile Robots Testing Using ROS Gazebo," System Theory, Control and Computing(ICSTTCC) IEEE, pp.96-101, 2016.DOI:10.1109/ICSTCC.2016.7790647   DOI