Journal of the Korea Convergence Society (한국융합학회논문지)

Korea Convergence Society (한국융합학회)
권호 표지
DOI QR코드

DOI QR Code

Convergence evaluation method using multisensory and matching painting and music using deep learning based on imaginary soundscape

Imaginary Soundscape 기반의 딥러닝을 활용한 회화와 음악의 매칭 및 다중 감각을 이용한 융합적 평가 방법

  • Jeong, Hayoung (Dept. of Human ICT Convergence, Sungkyunkwan University) ;
  • Kim, Youngjun (Dept. of Electrical and Computer Engineering, Sungkyunkwan University) ;
  • Cho, Jundong (Dept. of Human ICT Convergence, Sungkyunkwan University)
  • 정하영 (성균관대학교 휴먼ICT융합학과) ;
  • 김영준 (성균관대학교 전자전기컴퓨터공학과) ;
  • 조준동 (성균관대학교 휴먼ICT융합학과)
  • Received : 2020.10.06
  • Accepted : 2020.11.20
  • Published : 2020.11.28
Download PDF
⟨ Previous Next ⟩

Abstract

In this study, we introduced the technique of matching classical music using deep learning to design soundscape that can help the viewer appreciate painting and proposed an evaluation index to evaluate how well matching painting and music. The evaluation index was conducted with suitability evaluation through the Likeard 5-point scale and evaluation in a multimodal aspect. The suitability evaluation score of the 13 test participants for the deep learning based best match between painting and music was 3.74/5.0 and band the average cosine similarity of the multimodal evaluation of 13 participants was 0.79. We expect multimodal evaluation to be an evaluation index that can measure a new user experience. In addition, this study aims to improve the experience of multisensory artworks by proposing the interaction between visual and auditory. The proposed matching of painting and music method can be used in multisensory artwork exhibition and furthermore it will increase the accessibility of visually impaired people to appreciate artworks.

본 연구에서는 회화 감상에 도움이 되는 사운드스케이프를 구성하기 위해 딥러닝 기술을 활용하여 클래식 음악을 매칭하는 기술을 소개하고 회화와 음악 매칭이 얼마나 잘 되었는지에 대해 평가할 수 있는 평가 지표를 제안한다. 평가 지표는 리커드 5점 척도를 통한 적합도 평가와 멀티모달 측면의 평가로 진행하였다. 회화와 음악 매칭에 대해 13명의 실험 참가자의 적합도 평가의 점수는 3.74/5.0 이었고, 또한 13명의 실험 참가자의 멀티모달 평가에서 회화와 음악 매칭의 코사인 유사도의 평균은 0.79였다. 멀티모달적 평가는 새로운 사용자 경험을 측정할 수 있는 평가 지표가 될 것으로 기대된다. 또한 본 연구를 통해 시각과 청각의 인터랙션을 제안함으로써 다중감각 예술작품 경험을 향상시키고자 하였다. 본 연구에서 제안된 회화와 음악 매칭이 다중감각 예술작품 전시에서 활용되며 더 나아가 이는 시각 장애인들의 예술작품 감상에 대한 접근성을 높일 수 있을 것이라 기대한다.

Keywords

References

  1. L. Cavazos Quero, J. Iranzo Bartolome, S. Lee, E. Han, S. Kim & J. Cho. (2018). An Interactive Multimodal Guide to Improve Art Accessibility for Blind People. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 346-348). DOI: 10.1145/3234695.3241033
  2. J. Iranzo Bartolome, L. Cavazos Quero, S. Kim, M. Y. Um & J. Cho. (2019, March). Exploring Art with a Voice Controlled Multimodal Guide for Blind People. In Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction (pp. 383-390). DOI: 10.1145/3294109.3300994
  3. J. D. Cho et al. (2019). Color Information Transfer Multi-modal Interface Concept Design for People with Visually Impairment to Appreciate Works of Art - Focused on the Case of "Blind-Touch", a Reproduction Art for Blind -. Design Works, 2(2), 44-58. https://doi.org/10.15187/dw.2019.10.2.2.44
  4. Multisensory Artworks exhibition (2019). Human ICT Convergence, (professor: Jun Dong Cho) BlindTouch (Multisensory Painting Platform for the Blind) Exhibition; Exhibition Place: Siloam Center for the Blind S-Gallery.
  5. D. B. Faustino, S. Gabriele, R. Ibrahim, A. L Theus & A. Girouard. (2017, October). SensArt demo: A multisensory prototype for engaging with visual art. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (pp. 462-465). DOI: 10.1145/3132272.3132290
  6. S. Wang. (2020). Museum as a Sensory Space: A Discussion of Communication Effect of Multi-Senses in Taizhou Museum. Sustainability, 12(7), 3061. DOI: 10.3390/su12073061
  7. R. Murray Schafer. (1977). The Soundscape : Our Sonic Environment and the Tuning of the World. Rochester, Vermont : Destiny Books.
  8. R. E. Cytowic. (2002). Synesthesia: A union of the senses. Cambridge : MIT press.
  9. Y. G. Jeon. (2004). (A)Study on the Sound Uses to Maximize Visual Images in Digital Media. Masters dissertation. Hansung University, Seoul.
  10. T. Baumgartner M. Esslen & L. Jancke. (2006). From emotion perception to emotion experience: Emotions evoked by pictures and classical music. International journal of psychophysiology, 60(1), 34-43. DOI: 10.1016/j.ijpsycho.2005.04.007
  11. C. T. Vi, D. Ablart, E. Gatti, C. Velasco & M. Obrist. (2017). Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition. International Journal of Human-Computer Studies, 108, 1-14. DOI: 10.1016/j.ijhcs.2017.06.004
  12. https://www.bunkerdelumieres.com
  13. K. W. Guk. (2019). Examples of applications by AI technology and industry sectors. Weekly Technical Trends, 20, 15-27.
  14. M. Muller-Eberstein & N. van Noord. (2019). Translating Visual Art into Music. In Proceedings of the IEEE International Conference on Computer Vision Workshops. DOI: 10.1109/ICCVW.2019.00378
  15. Y. Kajihara, S. Ozono & N. Tokui. (2017). Imaginary Soundscape : Cross-Modal Approach to Generate Pseudo Sound Environments. NIPS Workshop.
  16. Y. Aytar, C. Vondrick & A. Torralba. (2016). Soundnet: Learning sound representations from unlabeled video. In Advances in neural information processing systems (pp. 892-900).
  17. A. Sharghi, J. S. Laurel & B. Gong. (2017). Query-focused video summarization: Dataset, evaluation, and a memory network based approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4788-4797). DOI: 10.1109/CVPR.2017.229
  18. A. Howard et al. (2019). Searching for mobilenetv3. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1314-1324).
  19. S. J. Shin. (1999). classification of adjectives. A Collection of Korean Language and Literature Studies at Sookmyung Women's University, 6, 19-40.
  20. H. W. Jung & K. Nah. (2007). A Study on the Meaning of Sensibility and Vocabulary System for Sensibility Evaluation. Journal of the Ergonomics Society of Korea, 26(3), 17-25. DOI: 10.5143/JESK.2007.26.3.01
  21. R. Baeza-Yates & B. Ribeiro-Neto. (1999). Modern information retrieval (Vol. 463). New York: ACM press.