Browse > Article
http://dx.doi.org/10.5391/JKIIS.2014.24.5.462

The System of Converting Muscular Sense into both Color and Sound based on the Synesthetic Perception  

Bae, Myung-Jin (Department of Electronic Engineering, Kyungnam University)
Kim, Sung-Ill (Department of Electronic Engineering, Kyungnam University)
Publication Information
Journal of the Korean Institute of Intelligent Systems / v.24, no.5, 2014 , pp. 462-469 More about this Journal
Abstract
As a basic study on both engineering applications and representation methods of synesthesia, this paper aims at building basic system which converts a muscular sense into both visual and auditory elements. As for the building method, data of the muscular sense can be acquired through roll and pitch signals which are calculated from both three-axis acceleration sensor and the two-axis gyro sensor. The roll and pitch signals are then converted into both visual and auditory information as outputs. The roll signals are converted into both intensity elements of the HSI color model and octaves as one of auditory elements. In addition, the pitch signals are converted into both hue elements of the HSI color model and scales as another one of auditory elements. Each of the extracted elements of the HSI color model is converted into each of the three elements of the RGB color model respectively, so that the real-time output color signals can be obtained. Octaves and scales are also converted and synthesized into MIDI signals, so that the real-time sound signals can be obtained as anther one of output signals. In experiments, the results revealed that normal color and sound output signals were successfully obtained from roll and pitch values that represent muscular senses or physical movements, depending on the conversion relationship based on the similarity between color and sound.
Keywords
Muscular Sense; Acceleration Sensor; Gyro Sensor; Synesthetic Perception; RGB Color model; HSI Color model;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 G-Y Jun, Synesthetic expression of sounds and colors, Ewha Womans University, Master's thesis, 2004.
2 Suslick K.S., "Synesthesia in science and technology: more than making the unseen visible", Current opinion in chemical biology, vol. 16 no. 5/6, pp. 557-563, 2012.   DOI
3 Hubbard E.M, RamachandranV.S "Neurocognitive Mechanisms of Synesthesia", Neuron, vol. 48, no. 3, pp. 509-520, 2005.   DOI   ScienceOn
4 Nikolinakos D., Georgiadou A., "A case of color-taste synesthesia", Neurocase, vol. 19, no. 3, pp. 282-294, 2013.   DOI
5 Goller Aviva I, Otten Leun J, "Seeing sounds and hearing colors: an event-related potential study of auditory-visual synesthesia.", Journal of cognitive neuroscience, vol. 21, no. 10, pp. 1869-1881, 2009.   DOI
6 Fernay L., Reby D., Ward J., "Visualized voices: A case study of audio-visual synesthesia", Neurocase, vol. 18, no. 1, pp. 50-56, 2012.   DOI
7 J-H Han, Color harmony scheme using the synesthetical characteristic of music, Yonsei University, Master's thesis, 2004.
8 J-H Kim, Upper limb position sense of the athletes in different sport events, Korea National University of Education, Master's thesis, 2004.
9 Milosevic B., Naldi R., Farella E, "Design and validation of an attitude and heading reference system for an aerial robot prototype", American Control Conference(ACC) , pp. 1720-1725, 2012.
10 Martin P., Salaun E., "Design and implementation of a low-cost observer-based attitude and heading reference system", Control engineering practice, vol. 18, no. 7, pp. 712-722, 2010.   DOI   ScienceOn
11 O-C Hur, Attitude estimation of the moving bodies using the MEMS sensor , Sungkyunkwan University, Master's thesis, 2010.
12 S-G Huang, Image Processing Programming by Visual C++(Korean), Hanbit Media, 2009.
13 G-H Kim, J-G Beak, Sound-color harmonization, Imress, 2003.
14 S-I Kim, J-S Jung, "A Basic Study on the System of Converting Color Image into Sound", Journal of the Korean Institute of Intelligent Systems, vol. 20, no. 2, pp. 251-256, 2010.   과학기술학회마을   DOI
15 G-W Kang, A Basic Study on the Conversion of Sound into Color Image Using the Pitch Signal , Kyungnam University, Master's thesis, 2012.
16 H-G Lee, Study on Effective Orchestration in Computer Music by Application of MIDI , Jeju National University, Master's thesis, 1999.
17 MyARS, Available: http://www.withrobot.com, [Accessed: July 02, 2014]
18 Information and Download on Praat, Available: http://www.praat.org, [Accessed: July 02, 2014]