• Journal of Internet Computing and Services
• /
• v.13 no.5
• /
• pp.9-19
• /
• 2012

Virtual human used as HCI in digital contents expresses his various emotions across modalities like facial expression and body posture. However, few studies considered combinations of such nonverbal multimodal in emotion perception. Computational engine models have to consider how a combination of nonverbal modal like facial expression and body posture will be perceived by users to implement emotional virtual human, This paper proposes the impacts of nonverbal multimodal in design of emotion expressed virtual human. First, the relative impacts are analysed between different modals by exploring emotion recognition of modalities for virtual human. Then, experiment evaluates the contribution of the facial and postural congruent expressions to recognize basic emotion categories, as well as the valence and activation dimensions. Measurements are carried out to the impact of incongruent expressions of multimodal on the recognition of superposed emotions which are known to be frequent in everyday life. Experimental results show that the congruence of facial and postural expression of virtual human facilitates perception of emotion categories and categorical recognition is influenced by the facial expression modality, furthermore, postural modality are preferred to establish a judgement about level of activation dimension. These results will be used to implementation of animation engine system and behavior syncronization for emotion expressed virtual human.

• The Journal of the Acoustical Society of Korea
• /
• v.42 no.5
• /
• pp.429-440
• /
• 2023

This study explores recent research trends and potential applications of ultrasound optical imaging-based multimodal technology. Ultrasound imaging has been widely utilized in medical diagnostics due to its real-time capability and relative safety. However, the drawback of low resolution in ultrasound imaging has prompted active research on multimodal imaging techniques that combine ultrasound with other imaging modalities to enhance diagnostic accuracy. In particular, ultrasound optical imaging-based multimodal technology enables the utilization of each modality's advantages while compensating for their limitations, offering a means to improve the accuracy of the diagnosis. Various forms of multimodal imaging techniques have been proposed, including the fusion of optical coherence tomography, photoacoustic, fluorescence, fluorescence lifetime, and spectral technology with ultrasound. This study investigates recent research trends in ultrasound optical imaging-based multimodal technology, and its potential applications are demonstrated in the biomedical field. The ultrasound optical imaging-based multimodal technology provides insights into the progress of integrating ultrasound and optical technologies, laying the foundation for novel approaches to enhance diagnostic accuracy in the biomedical domain.

• The Journal of the Institute of Internet, Broadcasting and Communication
• /
• v.14 no.4
• /
• pp.157-163
• /
• 2014

User's sensitivity is recognized as a very important parameter for communication between company, government and personnel. Especially in many studies, researchers use voice tone, voice speed, facial expression, moving direction and speed of body, and gestures to recognize the sensitivity. Multiplex modality is more precise than single modality however it has limited recognition rate and overload of data processing according to multi-sensing also an excellent algorithm is needed to deduce the sensing value. That is as each modality has different concept and property, errors might be happened to convert the human sensibility to standard values. To deal with this matter, the sensibility expression modality is needed to be extracted using technologies like analyzing of relational network, understanding of context and digital filter from multiplex modality. In specific situation to recognize the sensibility if the priority modality and other surrounding modalities are processed to implicit values, a robust system can be composed in comparison to the consuming of computer resource. As a result of this paper, it is proposed how to assign the weight of multiplex modality using implicit data.

• Journal of the Korea Society of Computer and Information
• /
• v.20 no.1
• /
• pp.143-151
• /
• 2015

Biometric technology has been proposed as a new means to replace conventional PIN or password because it is hard to be lost and has the low possibility of illegal use. However, unlike a PIN, password, and personal information there is no way to modify the exposure if it is exposed and used illegally. Therefore, the existing single modality with single biometrics is critical when it expose. However in this paper, we use a multi-modality and multi-biometrics to authenticate between users and TTP or between users and financial institutions. Thereby, we propose a more reliable method and compared this paper with existed methods about security and performance in this paper.

• The KIPS Transactions:PartA
• /
• v.11A no.5
• /
• pp.391-400
• /
• 2004

Within multimodal medical image registration techniques, which correlate different images and Provide integrated information, surface registration methods generally minimize the surface distance between two modalities. However, the features of two modalities acquired from one subject are similar. So, it can improve the accuracy of registration result to match two images based on optimization of both surface distance and shape feature. This research proposes a registration method which optimizes surface distance and surface curvature of two brain modalities. The registration process has two steps. First, surface information is extracted from the reference images and the test images. Next, the optimization process is performed. In the former step, the surface boundaries of regions of interest are extracted from the two modalities. And for the boundary of reference volume image, distance map and curvature map are generated. In the optimization step, a transformation minimizing both surface distance and surface curvature difference is determined by a cost function referring to the distance map and curvature map. The applying of the result transformation makes test volume be registered to reference volume. The suggested cost function makes possible a more robust and accurate registration result than that of the cost function using the surface distance only. Also, this research provides an efficient means for image analysis through volume visualization of the registration result.

• Proceedings of the Korean Society of Broadcast Engineers Conference
• /
• 2017.06a
• /
• pp.102-104
• /
• 2017

최근 Convolutional neural network (CNN)을 도입하여, SAR 영상의 목표물 인식 알고리즘이 높은 성능을 보여주었다. SAR 영상은 4 종류의 polarization 정보로 구성되어있다. 기계와 신호처리의 비용으로 인하여 일부 데이터는 적은 수의 polarization 정보를 가지고 있다. 따라서 우리는 SAR 영상 data 를 멀티모달 데이터로 해석하였다. 그리고 우리는 이러한 멀티모달 데이터에 잘 작동할 수 있는 콘볼루션 신경망을 제안하였다. 우리는 데이터가 포함하는 모달의 수에 반비례 하도록 scale factor 구성하고 이를 입력 크기조절에 사용하였다. 입력의 크기를 조절하여, 네트워크는 특징맵의 크기를 모달의 수와 상관없이 일정하게 유지할 수 있었다. 또한 제안하는 입력 크기조절 방법은 네트워크의 dead filter 의 수를 감소 시켰고, 이는 네트워크가 자신의 capacity 를 잘 활용한다는 것을 의미한다. 또 제안된 네트워크는 특징맵을 구성할 때 다양한 모달을 활용하였고, 이는 네트워크가 모달간의 상관관계를 학습했다는 것을 의미한다. 그 결과, 제안된 네트워크의 성능은 입력 크기조절이 없는 일반적인 네트워크보다 높은 성능을 보여주었다. 또한 우리는 전이학습의 개념을 이용하여 네트워크를 모달의 수가 많은 데이터부터 차례대로 학습시켰다. 전이학습을 통하여 네트워크가 학습되었을 때, 제안된 네트워크는 특정 모달의 조합 경우만을 위해 학습된 네트워크보다 높은 성능을 보여준다.

• Journal of the Korea Convergence Society
• /
• v.11 no.11
• /
• pp.175-182
• /
• 2020

In this study, we introduced the technique of matching classical music using deep learning to design soundscape that can help the viewer appreciate painting and proposed an evaluation index to evaluate how well matching painting and music. The evaluation index was conducted with suitability evaluation through the Likeard 5-point scale and evaluation in a multimodal aspect. The suitability evaluation score of the 13 test participants for the deep learning based best match between painting and music was 3.74/5.0 and band the average cosine similarity of the multimodal evaluation of 13 participants was 0.79. We expect multimodal evaluation to be an evaluation index that can measure a new user experience. In addition, this study aims to improve the experience of multisensory artworks by proposing the interaction between visual and auditory. The proposed matching of painting and music method can be used in multisensory artwork exhibition and furthermore it will increase the accessibility of visually impaired people to appreciate artworks.

• Journal of KIISE:Software and Applications
• /
• v.30 no.7_8
• /
• pp.719-726
• /
• 2003

This paper proposes a multimodal registration method for segmentation of the Visible Human color images, in which color characteristics of bones are very similar to those of its surrounding fat areas. Bones are initially segmented in CT images, and then registered into color images to lineate their boundaries in the color images. For the segmentation of bones in CT images, a thresholding method is developed. The registration method registers boundaries of bodies in CT and color images using a cross-correlation approach, in which the boundaries of bodies are extracted by thresholding segmentation methods. The proposed method has been applied to segmentation of bones in a head and legs whose boundary is ambiguous due to surrounding fat areas with similar color characteristics, and produced promising results.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2005.05a
• /
• pp.1633-1636
• /
• 2005

의료영상정합은 환자의 해부학적 정보와 기능적 정보를 혼합함으로써 기능이상부위의 해부학적 위치를 판별하기 위한 목적으로 널리 이용되고 있다. 그러나 실제적으로 여러 종류의 환자영상 취득이 어렵거나 해부학적 영상정보가 손실되는 경우가 적지 않다. 따라서 표준 정상인 해부학적 영상과 환자 기능영상을 정합함으로써 보다 객관적인 환자 기능이상부위 분석이 요구된다. 이는 다중개체, 다중모달리티간 영상정합으로 기존의 표면정보 또는 복셀정보 기반 방법으로는 한계가 있다. 따라서 본 연구에서는 두 대상영상 표면 뿐 아니라 내부 볼륨까지 대응시킬 수 있는 표면정보와 복셀정보를 혼합 적용한 기법을 제안한다.

• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 2001.10a
• /
• pp.729-732
• /
• 2001

Individual authentication system that take advantage of multimodal information is very efficient method that can take advantage of method of speech recognition, face recognition, electron signature etc. and protect important information from much dangers that exits on communication network whole as skill that construct security system. This paper deal product connected with hardware from internet space based on public key sign and electron signature description embodied system. Maintenance of public security is explaining that commercial transaction system implementation that is considered is possible as applying individual authentication.