Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Brain Correlates of Emotion for XR Auditory Content

XR 음향 콘텐츠 활용을 위한 감성-뇌연결성 분석 연구

  • Park, Sangin (Industry-Academy Cooperation Team, Hanyang University) ;
  • Kim, Jonghwa (Department of Artificial Intelligence Research, Hansol Inticube) ;
  • Park, Soon Yong (Department of Smart City Research, Seoul Institute of Technology) ;
  • Mun, Sungchul (Department of Industrial Engineering, Jeonju University)
  • 박상인 (한양대학교 산학협력단) ;
  • 김종화 (한솔인티큐브 인공지능융합연구팀) ;
  • 박순용 (서울기술연구원 스마트도시연구실) ;
  • 문성철 (전주대학교 산업공학과)
  • Received : 2022.07.19
  • Accepted : 2022.09.19
  • Published : 2022.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

In this study, we reviewed and discussed whether auditory stimuli with short length can evoke emotion-related neurological responses. The findings implicate that if personalized sound tracks are provided to XR users based on machine learning or probability network models, user experiences in XR environment can be enhanced. We also investigated that the arousal-relaxed factor evoked by short auditory sound can make distinct patterns in functional connectivity characterized from background EEG signals. We found that coherence in the right hemisphere increases in sound-evoked arousal state, and vice versa in relaxed state. Our findings can be practically utilized in developing XR sound bio-feedback system which can provide preference sound to users for highly immersive XR experiences.

본 논문은 XR 콘텐츠나 인터페이스 환경에서 활용할 수 있는 음향 자극의 종류를 고찰하고, 청각 자극 기반의 감성 유발이 뇌과학적으로 실효성을 가지는지에 대해 논의하였다. 외부 청각자극, 감성변화 및 뇌연결성의 상관관계 규명에 초점을 맞추어, XR 환경에서 사용자 경험을 제고하기 위한 기계학습 기반 개인 맞춤형 사운드 트랙 제공 서비스 개발이 필요하다는 시사점을 도출하였다. 또한, 짧은 음향자극으로 감성을 유발할 수 있는지를 테스트하여 청각자극에 의해 유발된 각성상태에서 우측 전두엽이나 전두엽, 두정엽, 후두엽 네트워크에서 뇌의 기능적 연결성이 강화되고 이완시에는 상반된 패턴을 보이는 것을 확인하였다. 본 연구에서 도출된 결과는 보다 입체적인 XR 상호작용 경험을 제시하고 사용자의 XR 인지수용성을 제고하여, 현장에서 실질적으로 적용될 수 있는 초실감 XR 사운드 바이오피드백 시스템 개발에 활용될 수 있을 것이다.

Keywords

Acknowledgement

This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea gover- nment(MSIT) (No. 2022-0-00137, XR User Interaction Evaluation and Employing Its Technology, 50%) and the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. NRF-2022R1C1C1010458, Cross Bio-sensing System for the Future XR Interface, 50%).

References

  1. Apple Glasses: here's everything we know so far, TechRadar News, https://www.techradar.com/news/apple-glasses(accessed Jul. 03, 2022).
  2. Apple Quietly Released One of The Most Impressive AR Room-mapping Tools, Road to VR - Virtual Reality News, https://www.roadtovr.com/apple-roomplan-ar-mapping (accessed Jul. 03, 2022).
  3. H. Huang and C.-F. Lee, "Factors affecting usability of 3D model learning in a virtual reality environment," Interactive Learning Environments, Vol.30, No.5, pp. 848-861, 2022, (accessed Jul. 8, 2022). doi: https://doi.org/10.1080/10494820.2019.1691605
  4. Y. Xing et al., "Web XR User Interface Research: Design 3D Layout Framework in Static Websites," Applied Sciences, Vol.12, No.11, pp. 5600, 2022, (accessed Jul. 8, 2022). doi: https://doi.org/10.3390/app12115600
  5. DeepMotion Animate 3D, https://www.deepmotion.com (accessed Jul. 7, 2022).
  6. OpenBCI and Varjo Partner to Bring Neurotechnology to Spatial Computing, https://varjo.com/company-news/openbci-and-varjo-partnerto-bring-neurotechnology-to-spatial-computing (accessed Jul. 7, 2022).
  7. S. Mun, E. S. Kim, and M. C. Park, "Effect of Mental Fatigue Caused by Mobile 3d Viewing on Selective Attention: An Erp Study," International Journal of Psychophysiology, Vol.94, No.3, pp.373-381, 2014, (accessed Jul. 7, 2022). doi: https://doi.org/10.1016/j.ijpsycho.2014.08.1389
  8. V. Putkinen et al., "Decoding Music-Evoked Emotions in the Auditory and Motor Cortex," Cerebral Cortex, Vol.31, No.5, pp.2549-2560, 2020, (accessed Jul. 7, 2022). doi: https://doi.org/10.1093/cercor/bhaa373
  9. L. Taruffi, S. Skouras, C. Pehrs, and S. Koelsch, "Trait Empathy Shpaes Neural Responses Toward Sad Music," Cognitive, Affective, & Behavioral Neuroscience, Vol.21, No.1, pp.231-241, 2021, (accessed Jul. 8, 2022). doi: http://doi.org/10.3758/s13415-020-00861-x
  10. L. Taruffi et al., "Effects of Sad and Happy Music on Mind-Wandering and the Default Mode Network," Scientific Reports, Vol.7, pp. 14396, 2017, (accessed Jul. 8, 2022). doi: https://doi.org/10.1038/s41598-017-14849-0
  11. S. Koelsch et al., "Neocortical Substrates of Feelings Evoked with Music in the ACC, Insula, and Somatosensory Cortex," Scientific Reports, Vol.11, No. 1, pp. 10119, 2021, (accessed Jul. 8, 2022). doi: https://doi.org/10.1038/s41598-021-89405-y
  12. B. Lee and W. Son, "Evaluation of Human Factors in Extended Reality Content," Information and Communications Magazine, The Korean Institute of Commucations and Information Sciences, Sempember2021 (accessed Jul. 8, 2022).
  13. G. Marion, G. M. D. Liberto, and S. A. Shamma, "The Music of Silence: Part I: Responses to Musical Imagery Encode Melodic Expectations and Acoustics," The Journal of Neuroscience, Vol.41, No. 35, pp. 7435-7448, 2021, (accessed Jul. 8, 2022). doi: https://doi.org/10.1523/JNEUROSCI.0183-21.2021
  14. S. Sangnark et al., "Revealing Preference in Popular Music Through Familiarity and Brain Response," IEEE Sensors Journal, Vol.21, No. 13, pp. 14931-14940, 2021, (accessed Jul. 8, 2022). doi: https://doi.org/10.1109/JSEN.2021.3073040
  15. E. Fox, "Perspectives from affective science on understanding the nature of emotion," Brain and Neuroscience Advances, Vol.2, pp. 1-8, 2018, (accessed Sep. 1, 2022). doi: https://doi.org/10.1177/2398212818812628
  16. H. Engelbregt et al., "Effects of binaural and monaural beat stimulation on attention and EEG," Experimental Brain Research, Vol.239, pp. 2781-2791, 2021, (accessed Sep. 1, 2022). doi: https://doi.org/10.1007/s00221-021-06155-z
  17. S. Park and M. Whang, "Infrared camera-based non-contact measurement of brain activity from pupillary rhythms," Frontiers in Physiology, Vol.9, pp. 1400, 2018, (accessed Sep. 1, 2022). doi: https://doi.org/10.3389/fphys.2018.01400
  18. L. A. Schmidt and L. J., "Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions," Cognition & Emotion, Vol.15, No. 4, pp. 487-500, 2001, (accessed Sep. 1, 2022). doi: https://doi.org/10.1080/02699930126048
  19. J. H. Kim et al., "A research on EEG coherence variation by relaxation," Korean Journal of the science of Emotion & sensibility, Vol.13, No. 1, pp. 121-128, 2010 (accessed Jul. 8, 2022).
  20. S. Mun, et al., "SSVEP and ERP Measurement of Cognitive Fatigue Caused by Stereoscopic 3d," Neuroscience Letters, Vol.525, No.2, pp.89-94, 2012, (accessed Jul. 8, 2022). doi: https://doi.org/10.1016/j.neulet.2012.07.049
  21. S. Mun, M.-C. Park, and S. Yano, "Performance Comparison of a SSVEP BCI Task by Individual Stereoscopic 3d Susceptibility," International Journal of Human-Computer Interaction," Vol.29, No.12, pp.789-797, 2013, (accessed Jul. 8, 2022). doi: https://doi.org/10.1080/10447318.2013.765289
  22. S. Mun, E.-S. Kim, and M.-C. Park, "Effect of mental fatigue caused by mobile 3D viewing on selective attention: an ERP study," International Journal of Psychophysiology," Vol.94, No.3, pp.373-381, 2014, (accessed Jul. 8, 2022). doi: https://doi.org/10.1016/j.ijpsycho.2014.08.1389
  23. S. Mun and M.-C. Park, "Affective Three-Dimensional Brain-Computer Interface Created Using a Prism Array-Based Display," Optical Engineering, Vol.53, No.12, pp.123105, 2014, (accessed Jul. 8, 2022). doi: https://doi.org/10.1117/1.OE.53.12.123105
  24. S. Mun, M.-C. Park, and S. Yano, "Evaluation of Viewing Experiences Induced by a Curved Three-Dimensional Display," Optical Engineering, Vol.54, No.10, pp.103104, 2015, (accessed Jul. 8, 2022). doi: https://doi.org/10.1117/1.OE.54.10.103104
  25. S. Mun, et al., "Effects of Mental Workload on Involuntary Attention: A Somatosensory ERP Study," Neuropsychologia, Vol.106, pp.7-20, 2017, (accessed Jul. 8, 2022). doi: https://doi.org/10.1016/j.neuropsychologia.2017.08.021
  26. S. Park, L. Kim, J. Ha, and S. Mun, "Infrared Webcam-Based Non-Contact Measurement of Event-Related Potentials from Event-Related Pupillary Responses: An Approach Focused on Mental Workload," Journal of Computational Design and Engineering, Vol.9, No. 4, pp.1327-1347, 2022, (accessed Jul. 8, 2022). doi: https://doi.org/10.1093/jcde/qwac059