• Science of Emotion and Sensibility
• /
• v.23 no.1
• /
• pp.41-56
• /
• 2020

This study suggests multisensory interaction techniques to help visually impaired people appreciate and understand artworks through non-visual senses such as tactile, auditory, and olfactory senses. A user study was conducted on the basis of a qualitative interview about the experience of appreciating artwork through the multisensory interaction system to visually impaired people so as to verify the development of the interaction techniques. The user test shows that the multisensory interactions in artwork generally not only help them appreciate and understand it but also give them satisfaction through artwork appreciation. However, it also indicates that some multisensory interactions caused the visually impaired people confusion and could not be perceived during the appreciation. On the basis of these outcomes, implications in this study are as follows. This study has contributed to providing specific development guidelines and indicators of non-visual multisensory interactions as a technical alternative to improve accessibility to cultural and artistic activities for the visually impaired. Furthermore, this study is expected to contribute to building a technical background, which can provide comprehensive sensory experiences with not only blind people but also non-blind people such as children and the elderly through universal interaction techniques beyond existing visual-oriented fragmentary experience.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.14 no.2
• /
• pp.737-743
• /
• 2013

This study was to investigate the effects of balance and trunk repositioning sense through multisensorial training using visual cue deprivation in subacute stroke patients. Multisensorial training group practiced after visual deprivation, control group practiced in the same training except visual deprivation with thirty subjects. Both groups were conducted training programs on sixty minutes a day, five days a week during four weeks. The subjects were evaluated by static balance, dynamic balance and trunk repositioning sense in the pre-posttest. There was significant improvement by multisensorial training that static balance, dynamic balance and the trunk repositioning sense between pretest and post test (p<.05). There were significantly differences in the amount of change of dynamic balance, trunk repositioning sense between the two groups(p<.05). Through this study, multisensorial training suggest that is effective in the improvement of dynamic balance and trunk repositioning sense.

• Science of Emotion and Sensibility
• /
• v.21 no.4
• /
• pp.77-90
• /
• 2018

Behavioral studies of emotion recognition in autism spectrum disorders (ASD) have yielded mixed results. Most of the studies focused on emotion recognition abilities with regard to ASD using stimuli with unisensory modality, making it difficult to determine difficulties in real life emotion perception in ASD. Herein, we review the recent behavioral and cognitive neuroscience studies on emotion recognition functions in ASD, including both unisensory and multisensory emotional information, to elucidate the possible difficulties in emotion recognition in ASD. In our study, we discovered that people with ASD have problems in the process of integrating emotional information during emotion recognition tasks. The following four points are discussed: (1) The restrictions of previous studies, (2) deficits in emotion recognition in ASD especially in recognizing multisensory information, (3) possible compensation mechanisms for emotion recognition in ASD, and (4) the possible roles of attention and language functions in emotion recognition in ASD. The compensatory mechanisms proposed herein for ASD with regard to emotion recognition abilities could contribute to a therapeutic approach for improving emotion recognition functions in ASD.

• 한국HCI학회:학술대회논문집
• /
• 2007.02a
• /
• pp.500-506
• /
• 2007

얼굴에 대한 시각적 인지는 오랫동안 인간에게 중요한 문제로 인식되어 왔다. 수 세기 동안 이루어져 왔던 미용 화장과 성형, 치아 교정 등의 다양한 연구는 사람의 얼굴을 감성적 측면에서 어떻게 하면 아름답게 만들 수 있는 가에 초점을 두었다. 본 논문에서는 휴먼 입출력 인터페이스로서 햅틱 장치와 스테레오 디스플레이를 혼합한 다중 감각 인터랙션 기반의 감성적인 3차원 얼굴 메이크업 시뮬레이션 프레임워크를 개발한다. 본 연구는 3차원 스캐너 장비로부터 사용자의 얼굴 모델을 추출하고, 그 데이터를 이용하여 자연스럽고 직관적인 얼굴 메이크업 시뮬레이션을 수행하는 것을 목표로 하고 있다. 이를 위하여 본 연구에서는 surface elements 표현 기반의 3차원 얼굴 필터링 방법과 얼굴 메이크업을 지원하는 페인팅 방법을 개발한다. 우선 사용자의 얼굴 모델을 3차원 스캐너로 획득한 후, 전처리 얼굴 필터링을 수행하여 조명, 그리고 사용자 얼굴 피부 상태에 기인하는 에러 및 속성들을 보정하고 피부 톤을 사용자가 선호하는 색으로 변경한다. 최종적으로 사용자는 햅틱 및 스테레오 디스플레이 장치를 이용하여 두 개의 레이어로 구성된 페인팅 표면 모델에 메이크업을 수행한다. 본 연구에서 적용한 surface elements 표현 기반의 그래픽 렌더링은 일반적인 메쉬 기반 페인팅의 문제점인 텍스쳐 왜곡 현상을 완화하고, 3차원 스캐너 장치에 기인하는 표면 에러를 보정한다. 그리고 전처리 얼굴 필터링과 메이크업 페인팅 방법은 사용자 중심의 감성적인 3차원 얼굴을 재구성하도록 한다. 결과적으로 본 연구에서 개발한 이러한 기술들이 다중 감각 인터페이스 기반의 메이크업 시뮬레이터의 기본 프레임워크가 되어, 차후 메이크업이나 코디네이션 분야 등의 디지털 콘텐츠 산업에서 활용될 수 있음을 확인하였다.

• Journal of the Korea Convergence Society
• /
• v.11 no.11
• /
• pp.175-182
• /
• 2020

In this study, we introduced the technique of matching classical music using deep learning to design soundscape that can help the viewer appreciate painting and proposed an evaluation index to evaluate how well matching painting and music. The evaluation index was conducted with suitability evaluation through the Likeard 5-point scale and evaluation in a multimodal aspect. The suitability evaluation score of the 13 test participants for the deep learning based best match between painting and music was 3.74/5.0 and band the average cosine similarity of the multimodal evaluation of 13 participants was 0.79. We expect multimodal evaluation to be an evaluation index that can measure a new user experience. In addition, this study aims to improve the experience of multisensory artworks by proposing the interaction between visual and auditory. The proposed matching of painting and music method can be used in multisensory artwork exhibition and furthermore it will increase the accessibility of visually impaired people to appreciate artworks.

• Science of Emotion and Sensibility
• /
• v.20 no.3
• /
• pp.49-60
• /
• 2017

Previous research found that merely touching an object can create psychological ownership and the endowment effect. It was also found that just imagining touching an object without actually touching the object can make the same effect on psychological ownership. Prior research on haptic imagery examined the effect of haptic imagery induced by direct instruction of imaging on psychological ownership. We investigate a new method which can induce the haptic imagery in a more natural way than direct instruction of imaging. We manipulated imagery conditions such as visual-haptic congruence multimodal cue, visual-haptic incongruent multimodal cue, direct instruction condition and control condition, and examined the effects on imagery vividness, feeling of physical control, perceived ownership, and purchase intention. We conducted the experiment on 140 undergraduate students and our results showed that visual-haptic congruence multimodal cue condition is more effective than direct instruction of haptic imagery while visual-haptic incongruence multimodal cue condition is not effective. Our study extends prior haptic imagery research by making important marketing implications for online retailing.

• Proceedings of the Korean Society for Emotion and Sensibility Conference
• /
• 2000.11a
• /
• pp.185-188
• /
• 2000

현재의 3차원 시각 정보 제시기는 Scene Graph 기반의 3차원 그래픽스 API에 기반을 두고 있다. 본 연구에서는 시각 정보 제시에 더불어 다양한 감각을 제시하기 위한 통합 제시 능력을 가지는 시스템으로 시각화 시스템을 확장하였다. 이 확장된 시스템은 다양한 감각 정보를 제시하기 위한 추가적인 정보를 가지는 확장된 Scene Graph을 가지고 운영되며, 통합된 Script System으로 각 감각 제시 시스템들간의 동기화 및 통합 관리와 제시 시스템의 확장에 유리한 장점을 가진다.

• Journal of Digital Convergence
• /
• v.20 no.4
• /
• pp.575-584
• /
• 2022

The purpose of this study is to derive interaction types that appear variously in immersive contents through John Dewey's empirical theory, and to explore how the derived types are delivered in real cases. For this reason, two cases of the National Museum of Korea 'Immersive Digital Gallery' and 'Arte Museum Jeju' were analyzed through interaction types derived from the empirical theory. The interaction types derived based on the experience theory are the elements of 'multi-sensory', 'simultaneous experience' and 'sensory expansion'. In both cases, these types appear connected rather than grafted one by one in common. In one direction, 'multi-sensory' leads to 'sensory expansion', and in two directions, 'simultaneous experience' leads to 'sensory expansion'. As such, the core types of communication between technology and humans are not delivered one by one, but a cycle of interaction is formed in multiple ways. Therefore, it can be seen that the interaction type of immersive contents is expanded step by step by fusion of various senses and experiences in various fields, rather than a 1:1 partial delivery method. Based on this, it will be necessary to study how types are expanded and how viewers are affected when interaction is implemented in immersive contents in the future.

• Journal of Broadcast Engineering
• /
• v.21 no.3
• /
• pp.330-340
• /
• 2016

This paper explains about the structure of multiple sensorial media application format (ISO/IEC 23000-17), which is newly standardized as a project of MPEG-A. This format facilitates effective storage, playing, and management of media with multiple sensorial effects. The ISO base media file format from MPEG-4 Part 12 and sensory effect metadata (SEM) from MPEG-V Part 3 are used to composed the multiple sensorial media application format. In this paper, a fragmentation method to break a SEM XML document into valid SEM samples is presented. Several binarization methods to compress the SEM samples are compared and evaluated as well. The compression ratio and processing time using the MPEG-V binary representation and the Binary MPEG format for XML (BiM) are superior to the gzip compression.

• Proceedings of the Korean Society for Emotion and Sensibility Conference
• /
• 1999.11a
• /
• pp.92-96
• /
• 1999

다중 감각 실험 환경을 구축시에 시청각 뿐 아니라 열감각, 촉각, 후각등 다중 감각등의 제시가 필요하다. 이러한 실험 환경은 작업의 부하를 분산하고, 모듈 개발 방법을 도입하기 위해서 분산 시스템으로 구축되는데 이에 추가적으로 실시간 동작 속성을 가지는 프로토콜이 필요하다. 본 연구는 이러한 실시간 동작 속성을 가지며 개별 모듈의 용이한 개발의 지원과 고장시의 대책으로서 임의의 모듈의 고장시의 대처 방안과 제어 모듈의 프로그램 용이성을 제공하는 프로토콜을 설계하였다. 다양한 시뮬레이션 재구성 능력을 제공할 수 있는 필드의 동적 바인딩을 지원하며, 동일 모듈의 연결로 새로운 모듈을 합성할 수 있는 능력을 가질 수 있는 분산 실험 시스템을 설계하였다.