Journal of IKEEE (전기전자학회논문지)

Institute of Korean Electrical and Electronics Engineers (한국전기전자학회)
권호 표지
DOI QR코드

DOI QR Code

Implementation of Mutual Conversion System between Body Movement and Visual·Auditory Information

신체 움직임-시·청각 정보 상호변환 시스템의 구현

  • Bae, Myung-Jin (Dept. of Convergence IT Engineering, Kyungnam University) ;
  • Kim, Sung-Ill (Dept. of Electronic Engineering, Kyungnam University)
  • Received : 2018.06.12
  • Accepted : 2018.06.20
  • Published : 2018.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

This paper has implemented a mutual conversion system that mutually converts between body motion signals and both visual and auditory signals. The present study is based on intentional synesthesia that can be perceived by learning. The Euler's angle was used in body movements as the output of a wearable armband(Myo). As a muscle sense, roll, pitch and yaw signals were used in this study. As visual and auditory signals, MIDI(Musical Instrument Digital Interface) signals and HSI(Hue, Saturation, Intensity) color model were used respectively. The method of mutual conversion between body motion signals and both visual and auditory signals made it easy to infer by applying one-to-one correspondence. Simulation results showed that input motion signals were compared with output simulation ones using ROS(Root Operation System) and Gazebo which is a 3D simulation tool, to enable the mutual conversion between body motion information and both visual and auditory information.

본 논문은 학습으로 공감각 현상을 지각할 수 있는 의도적인 공감각을 기반으로 신체의 움직임에서 시각과 청각정보로의 변환 및 역변환 시스템을 구현하였다. 신체의 움직임은 웨어러블 암밴드인 Myo의 출력인 오일러 각을 사용하였고, 근감각 정보로서 롤(Roll), 피치(Pitch), 요(Yaw) 신호를 사용하였다. 또한, 시각과 청각 정보로서 미디(MIDI, Musical Instrument Digital Interface)신호와 HSI 컬러 모델을 사용하였다. 근감각 신호와 시 청각 신호 사이의 상호변환 방법은 일대일 대응 관계를 적용함으로써 직관적으로 쉽게 유추할 수 있도록 하였다. 시뮬레이션 결과에서 신체의 움직임 정보와 시 청각 정보의 상호변환이 가능함을 ROS(Root Operation System)와 3D 시뮬레이션 툴인 Gazebo를 사용하여 입력과 출력을 비교하였고 변환 오차가 작음을 확인하였다.

Keywords

References

  1. K. Jun. "Synesthetic Expression of Sounds and Colors," Master's Thesis of Ewha Womans University, 2004.
  2. Y. Kim, "A Study on the Factors That Create Synesthesia," Doctor's Thesis of Hong-Ik Universisy, 2005.
  3. M. Bae, S. Kim, "Implementation of Muscular Sense into both Color and Sound Conversion System based on Wearable Device," Journal of Korea Multimedia Society, Vol. 19, No. 3, pp.642-649, 2016.DOI: 10.9717/kmms.2016.19.3.642
  4. Suslick, S Kenneth. "Synesthesia in science and technology: more than making the unseen visible," Current opinion in chemical biology Vol. 16, No. 5, pp. 557-563, 2012.DOI:10.1016/j.cbpa.2012.10.030
  5. J. Jang, "The Study on Visual Language for Visualizing Taste," Master's Thesis of Yonsei University, 2017.
  6. S. Cho, "Developing Digital Contents Using Multimedia Tehnology", Digital Contents Society, Vol. 1, No. 1, pp. 103-110, 2000.
  7. J. Baek, "Media Art Musical Sonification based on Cityscope," Master's Thesis of Chung-Ang University, 2015.
  8. S. Kim, "Conversion of Image into Sound Based on HSI Histogram," The Journal of the Acoustical Society of Korea, Vol. 30, No. 3, pp. 142-148, 2011.DOI:10.7776/ASK.2011.30.3.142
  9. D. Bor, N. Rothen, D. Schwartzman, S. Clayton, A. Seth, "Adults Can Be Trained to Acquire Synesthetic Experiences," Scientific Reports 4, No. 7089, 2014.DOI:10.1038/srep07089
  10. Witthoft, Nathan, and Jonathan Winawer. "Learning, memory, and synesthesia." Psychological science, Vol. 24, No. 3, pp. 258-265, 2013.DOI:10.1177/0956797612452573
  11. M. Bae, S. Kim, "Implementation of the Visualization and Sonification System of Human Movements based on Intentional Synesthesia," Journal of Korean institute of intelligent systems, Vol. 28, No. 1, pp. 83-90, 2018.DOI:10.5391/JKIIS.2018.28.1.83
  12. J. Messias, R. Ventura, P. Lima, and J. Sequeira, "ROS in the MOnarCH Project: A Case Study in Networked Robot Systems," Robot Operating System (ROS) Springer International Publishing, pp.375-395, 2016.DOI:10.1007/978-3-319-26054-9_14
  13. K. Nathan and A. Howard, "Design and use paradigms for gazebo, an open-source multi-robot simulator," Proc. Intelligent Robots and Systems, vol3, 2004.DOI:10.1109/IROS.2004.1389727
  14. I. Zamora, N. G. Lopez, V. M. Vilches, and A. H. Cordero, "Extending the OpenAI Gym for robotics: a toolkit for reinforcement learning using ROS and Gazebo," arXiv:1608.05742v2 [cs.RO], Feb. 2017.
  15. T. Kenta, A. Toshinori el al. "Simulation Environment for Mobile Robots Testing Using ROS Gazebo," System Theory, Control and Computing(ICSTTCC) IEEE, pp.96-101, 2016.DOI:10.1109/ICSTCC.2016.7790647