• Science of Emotion and Sensibility
• /
• v.23 no.1
• /
• pp.41-56
• /
• 2020

This study suggests multisensory interaction techniques to help visually impaired people appreciate and understand artworks through non-visual senses such as tactile, auditory, and olfactory senses. A user study was conducted on the basis of a qualitative interview about the experience of appreciating artwork through the multisensory interaction system to visually impaired people so as to verify the development of the interaction techniques. The user test shows that the multisensory interactions in artwork generally not only help them appreciate and understand it but also give them satisfaction through artwork appreciation. However, it also indicates that some multisensory interactions caused the visually impaired people confusion and could not be perceived during the appreciation. On the basis of these outcomes, implications in this study are as follows. This study has contributed to providing specific development guidelines and indicators of non-visual multisensory interactions as a technical alternative to improve accessibility to cultural and artistic activities for the visually impaired. Furthermore, this study is expected to contribute to building a technical background, which can provide comprehensive sensory experiences with not only blind people but also non-blind people such as children and the elderly through universal interaction techniques beyond existing visual-oriented fragmentary experience.

• Journal of the Korea Convergence Society
• /
• v.11 no.11
• /
• pp.175-182
• /
• 2020

In this study, we introduced the technique of matching classical music using deep learning to design soundscape that can help the viewer appreciate painting and proposed an evaluation index to evaluate how well matching painting and music. The evaluation index was conducted with suitability evaluation through the Likeard 5-point scale and evaluation in a multimodal aspect. The suitability evaluation score of the 13 test participants for the deep learning based best match between painting and music was 3.74/5.0 and band the average cosine similarity of the multimodal evaluation of 13 participants was 0.79. We expect multimodal evaluation to be an evaluation index that can measure a new user experience. In addition, this study aims to improve the experience of multisensory artworks by proposing the interaction between visual and auditory. The proposed matching of painting and music method can be used in multisensory artwork exhibition and furthermore it will increase the accessibility of visually impaired people to appreciate artworks.

• Journal of Korean Academy of Nursing
• /
• v.41 no.4
• /
• pp.528-538
• /
• 2011

Purpose: The purpose of the study was to examine the effects of multisensory stimulation (MSS) using familiarity on persons with dementia (PWDs) residing in nursing homes in Korea. Methods: A nonequivalent control group with pre and posttests was used. Fifty one PWDs were included if they: 1) were over 65 yr old, 2) were diagnosed with dementia, 3) had no visual or speech impairments, 4) were able to communicate, and 5) had spent more than one month in a nursing home. The experimental group (n=25) received a 55 min MSS program twice a week for 10 weeks. The outcome variables included were cognition, activities of daily living, grip strength, depression, wandering, and aggressive behaviors. Repeated ANOVA was used for data analysis. Results: There were no significant differences in demographics or the main variables at pretest. Cognition, depression, wandering, and aggressive behaviors were significant over time between the two groups. Grip strength was only significant when accounting for interaction between group and time. Conclusion: An intervention of MSS using familiarity was marginally effective in improving cognition, depression, wandering, and aggression. Future study is suggested with a larger sample and longer treatment to retest the effects of MSS.

• Science of Emotion and Sensibility
• /
• v.24 no.4
• /
• pp.139-148
• /
• 2021

This study aimed to verify how a multisensory storytelling-based activity-based intervention affects social interaction in children with cerebral palsy. As a quasi-experimental investigation, this study used a single-blind, two-group pre-post test design. This study comprised 24 children aged 7 to 8 y who had been diagnosed with spastic cerebral palsy and were classified as having GMFCS stages I to III. Twelve children were randomly assigned to experimental and control groups, with neither the children nor their guardians knowing which group they were placed in. The group program comprised 16 sessions of 60 min each, twice a week for eight weeks. The experimental group engaged in an activity-centered intervention centered on multisensory storytelling, whereas the control group engaged in structured physical activity. The activities were assessed using the peer relations skills scale to determine the extent to which social interaction had changed prior to and during the child's intervention. The SPSS 25.0 for Windows (IBM Corp, USA) application was used to analyze the data, and the significance level (α) for statistical verification was set to 0.05. Furthermore, the Wilcoxon Signed-Rank and Mann-Whitney U tests were used to assess the differences in social interaction between the experimental and control groups. Significant differences were observed in the total of the peer relationship skill scale and cooperation and empathy areas of the subtest in the intragroup change of the peer relationship skill scale between the experimental and control groups. However, the experimental group demonstrated a significant difference in the initiative area, whereas the control group demonstrated no significant difference. A significant difference was observed in the amount of change between the two groups in the initiative area and total of the subtest of peer relationship skills but no significant difference in the collaboration and empathy areas. We gave a multisensory storytelling-based activity-based intervention based on multisensory storytelling to children with cerebral palsy and saw a significant improvement in peer relationship skills. It may be proposed as an effective intervention strategy for children with cerebral palsy who struggle with social contact.

• Journal of the Korea Society of Computer and Information
• /
• v.24 no.11
• /
• pp.61-69
• /
• 2019

In this paper, we propose a multisensory(i.e., visual-auditory-tactile) based AR system for the education of cultural heritage. The proposed system provides a multisensory interaction by designing a user to experience with a 3D printed artifact which is mapped by a virtual 3D content of digital heritage. Compared with the existing systems of cultural heritage education based on augmented reality(AR) technology, this system focused on not only providing learning experience via a sense of visual and auditory, but also a sense of tactile. Furthermore, since this systems mainly provided the direct interactions using a 3D printed model, it gives a higher degree of realism than existing system that use touch or click motions on a 2D display of mobile phones and tablets. According to a result of user testing, we concluded that the proposed system delivered the excellent presence and learning flow to users. Particularly, from the usability evaluation, a 3D printed target artifact which is similar in shape to original heritage artifact, achieved the highest scores among the various tested targets.

• The Journal of Korean Academy of Sensory Integration
• /
• v.8 no.1
• /
• pp.41-49
• /
• 2010

Objective : Multisensory integration (MSI) is the essential process to use diverse sensory information for cognitive task or execution of motor action. Especially, visual and somatosensory integration is critical for motor behavior and coordination. This study was designed to explain spatial and temporal characteristics of visual and somatosensory integration by neurophysiological research method that identifies the time course and brain location of the SI process. Methods : Electroencephalography (EEG) and event-related potential (ERP) is used in this study in order to observe neural activities when integrating visual and tactile input. We calculate the linear summation (SUM) of visual-related potentials (VEPs) and somatosensory-related potentials (SEPs), and compared the SUM with simultaneously presented visual-tactile ERPs(SIM) Results : There were significant differences between the SIM and SUM in later time epochs (about 200-300ms) at contralateral somatosensory areas (C4) and occipital cortices (O1&O2). The amplitude of the SIM was mathematically larger than the summed signals, implying that the integration made some extra neural activities. Conclusion : This study provides some empirical neural evidence of that multisensory integration is more powerful than just combing two unisensory inputs in the brain and ERP data reveals neural signature relating to multisensory integrative process. Since this study is preliminary pilot study, larger population and criteria are needed for level of the significance. Further study is recommended to consider issues including effect of internally-driven attention and laterality of interaction to make the evidence by this study solid.

• Journal of the Korean Institute of Navigation
• /
• v.24 no.3
• /
• pp.133-140
• /
• 2000

The paper deals with the implementation of prototype Ship Simulator system (VRSS) by Virtual Reality techniques with Head Mounted Display (HMD) device. The prototype VRSS was consists of PC-based human sensors, object oriented operating system. In addition, two kinds of databases arranged from Head Related Transfer Functions and 3D object models were used to create 3D sea sound, and to construct virtual world, respectively. Using the prototype system, we carried out some simulation tests for the overtaking situation to prevent collisions at sea, and discussed on the usability of the system. As results from simulations, the prototype VRSS can provide multisensory and interactive display environment. The results gave rise to the user interaction with 3D objects that give realistic reproduction of navigational environments under a given scenario. Thus, we found that the prototype VRSS should be one of the next-generation ship simulation system.

• 한국HCI학회:학술대회논문집
• /
• 2008.02b
• /
• pp.552-557
• /
• 2008

'Digilog Book' dealing with the combination of each advantage of analog sensibility on texts and digital sensation on virtual reality is one of the noble applications of U-book. Based on contents from books, enhancing 3D immersive display enable readers to enjoy various experiences with multisensory feedback provided. Also, 'Digilog Book' provides digitally created stories that would project different experiences and environments interacting users' choices on the interface of the book. This study focuses on developing users friendly implementation tools providing more interaction between digitally generated stories and digitally generated interface so that interaction between virtual reality and actual realty is expanded through simulating contexts and extents. From the application scenario 'Digilog Book' would perform as a storytelling tool with developing subordinated elements such as context structure, screen structure, definition of writing function, writing interface, writing system structure, etc. Therefore, this study would become a significance to direct and indicate the development of writing techniques on virtual reality back to life through digitally augmented reality.

• Journal of the Korea Computer Graphics Society
• /
• v.28 no.3
• /
• pp.67-77
• /
• 2022

The sense of touch by natural wind is one of the most common feels that every person experiences in daily life. However, it has been rarely studied how natural wind can be reproduced in a VR environment and whether the multisensory contents equipped with artificial winds do improve human emotion or not. To address these issues, we first propose a wind reproduction VR system guided by video and wind capture data and also study the effect of the system on positive affect. We collected wind direction and speed data together with a 360-degree video on a yacht. These pieces of data were used to produce a multisensory VR environment by our wind reproduction VR system. 19 college students participated in the experiments, where the Korean version of Positive and Negative Affect Schedule (K-PANAS) was introduced to measure their emotions. Through the K-PANAS, we found that 'inspired' and 'active' emotions increase significantly after experiencing the yacht VR contents with artificial wind. Our experimental results also show that another emotion, 'interested', is most notably affected depending on the presence of the wind. The presented system can be effectively used in various VR applications such as interactive media and experiential contents.

• The Journal of the Korea Contents Association
• /
• v.21 no.9
• /
• pp.75-87
• /
• 2021

The purpose of this study is to derive the characteristics of interaction design for each stage of content composition so that it can be referred to in the planning and production of virtual reality safety education contents. It was confirmed that each of the following interaction design features was found in the three configuration steps: acquisition of situation response procedure knowledge, accident situation experiential learning, and content confirmation and evaluation. First, it was revealed that the quality of experience was controlled by increasing the fidelity of behaviors and reducing general and repetitive behaviors in order to emphasize the educational content-related experiences in the learner experience stage. Second, in order for learners to easily recognize main interaction objects in order to acquire information on safe behavior procedures in unfamiliar environments, use of spatial UI or signifiers using arrows or symbols, posts that specifically instruct actions, and multisensory signals Therefore, it was found to be important to emphasize essential actions in a way that lowers the degree of freedom of user experience, and the proportion of non-realistic interactions for cognitive interactions was found to increase. Lastly, in the confirmation and evaluation stage of the experience, it is important to use the meta UI to alleviate negative experiences such as physical damage after experiencing a safety accident situation,