• Journal of Internet Computing and Services
• /
• v.13 no.5
• /
• pp.9-19
• /
• 2012

Virtual human used as HCI in digital contents expresses his various emotions across modalities like facial expression and body posture. However, few studies considered combinations of such nonverbal multimodal in emotion perception. Computational engine models have to consider how a combination of nonverbal modal like facial expression and body posture will be perceived by users to implement emotional virtual human, This paper proposes the impacts of nonverbal multimodal in design of emotion expressed virtual human. First, the relative impacts are analysed between different modals by exploring emotion recognition of modalities for virtual human. Then, experiment evaluates the contribution of the facial and postural congruent expressions to recognize basic emotion categories, as well as the valence and activation dimensions. Measurements are carried out to the impact of incongruent expressions of multimodal on the recognition of superposed emotions which are known to be frequent in everyday life. Experimental results show that the congruence of facial and postural expression of virtual human facilitates perception of emotion categories and categorical recognition is influenced by the facial expression modality, furthermore, postural modality are preferred to establish a judgement about level of activation dimension. These results will be used to implementation of animation engine system and behavior syncronization for emotion expressed virtual human.

• Smart Media Journal
• /
• v.12 no.9
• /
• pp.81-88
• /
• 2023

Recently, communication through online is increasing due to the spread of non-face-to-face services due to COVID-19. In non-face-to-face situations, the other person's opinions and emotions are recognized through modalities such as text, speech, and images. Currently, research on multimodal emotion recognition that combines various modalities is actively underway. Among them, emotion recognition using speech data is attracting attention as a means of understanding emotions through sound and language information, but most of the time, emotions are recognized using a single speech feature value. However, because a variety of emotions exist in a complex manner in a conversation, a method for recognizing multiple emotions is needed. Therefore, in this paper, we propose a multi-emotion regression model that extracts feature vectors after preprocessing speech data to recognize complex, inherent emotions and takes into account the passage of time.

• The Journal of the Acoustical Society of Korea
• /
• v.40 no.5
• /
• pp.515-522
• /
• 2021

It is hard to prepare sufficient training data for speech emotion recognition due to the difficulty of emotion labeling. In this paper, we apply transfer learning with large-scale training data for speech recognition on a transformer-based model to improve the performance of speech emotion recognition. In addition, we propose a method to utilize context information without decoding by multi-task learning with speech recognition. According to the speech emotion recognition experiments using the IEMOCAP dataset, our model achieves a weighted accuracy of 70.6 % and an unweighted accuracy of 71.6 %, which shows that the proposed method is effective in improving the performance of speech emotion recognition.

• Journal of the Korean Institute of Intelligent Systems
• /
• v.14 no.2
• /
• pp.150-155
• /
• 2004

In this paper, an emotion recognition method using speech signal is presented. Six basic human emotions including happiness, sadness, anger, surprise, fear and dislike are investigated. The proposed recognizer have each codebook constructed by using the wavelet transform for the emotional state. Here, we first verify the emotional state at each filterbank and then the final recognition is obtained from a multi-decision method scheme. The database consists of 360 emotional utterances from twenty person who talk a sentence three times for six emotional states. The proposed method showed more 5% improvement of the recognition rate than previous works.

• Smart Media Journal
• /
• v.11 no.8
• /
• pp.65-72
• /
• 2022

Due to COVID-19, the importance of non-face-to-face counseling is increasing as the face-to-face counseling method has progressed to non-face-to-face counseling. The advantage of non-face-to-face counseling is that it can be consulted online anytime, anywhere and is safe from COVID-19. However, it is difficult to understand the client's mind because it is difficult to communicate with non-verbal expressions. Therefore, it is important to recognize emotions by accurately analyzing text and voice in order to understand the client's mind well during non-face-to-face counseling. Therefore, in this paper, text data is vectorized using FastText after separating consonants, and voice data is vectorized by extracting features using Log Mel Spectrogram and MFCC respectively. We propose a multi-emotion recognition model that recognizes five emotions using vectorized data using an LSTM model. Multi-emotion recognition is calculated using RMSE. As a result of the experiment, the RMSE of the proposed model was 0.2174, which was the lowest error compared to the model using text and voice data, respectively.

• Proceedings of the Korean Institute of Intelligent Systems Conference
• /
• 2006.11a
• /
• pp.131-134
• /
• 2006

본 논문에서는 가정용 로봇 및 서비스 로봇과 같은 이동로봇에서 사용자의 감정을 인식하는 방법중 한가지인 얼굴영상을 이용한 감정인식 방법을 제안한다. 얼굴영상인식을 위하여 얼굴의 여러 가지 특징(눈썹, 눈, 코, 입)의 움직임 및 위치를 이용하며, 이동로봇에서 움직이는 사용자를 인식하기 위한 움직임 추적 알고리즘을 구현하고, 획득된 사용자의 영상에서 얼굴영역 검출 알고리즘을 사용하여 얼굴 영역을 제외한 손과 배경 영상의 피부색은 제거한다. 검출된 얼굴영역의 거리에 따른 영상 확대 및 축소, 얼굴 각도에 따른 영상 회전변환 등의 정규화 작업을 거친 후 이동 로봇에서는 항상 고정된 크기의 얼굴 영상을 획득 할 수 있도록 한다. 또한 기존의 특징점 추출이나 히스토그램을 이용한 감정인식 방법을 혼합하여 인간의 감성 인식 시스템을 모방한 로봇에서의 감정인식을 수행한다. 본 논문에서는 이러한 다중 특징점 추출 방식을 통하여 이동로봇에서의 얼굴 영상을 이용한 사용자의 감정인식 시스템을 제안한다.

• Journal of Digital Contents Society
• /
• v.19 no.7
• /
• pp.1349-1355
• /
• 2018

Recently, a variety of contents services over the internet are becoming popular, among which MCN(Multi Channel Network) platform services have become popular with the generalization of smart phones. The MCN platform is based on streaming, and various factors are added to improve the service. Among them, augmented reality sticker service using face recognition is widely used. In this paper, we implemented the MCN platform that masks the augmented reality sticker on the face through facial emotion recognition in order to further increase the interest factor. We analyzed seven facial emotions using deep learning technology for facial emotion recognition, and applied the emotional sticker to the face based on it. To implement the proposed MCN platform, emotional stickers were applied to the clients and various servers that can stream the servers were designed.

• The Journal of the Convergence on Culture Technology
• /
• v.10 no.1
• /
• pp.617-623
• /
• 2024

Identifying emotions through speech poses a significant challenge due to the complex relationship between language and emotions. Our paper aims to take on this challenge by employing feature engineering to identify emotions in speech through a multimodal classification task involving both speech and text data. We evaluated two classifiers-Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM)-both integrated with a BERT-based pre-trained model. Our assessment covers various performance metrics (accuracy, F-score, precision, and recall) across different experimental setups). The findings highlight the impressive proficiency of two models in accurately discerning emotions from both text and speech data.

• Journal of KIISE:Software and Applications
• /
• v.31 no.6
• /
• pp.799-807
• /
• 2004

In this paper, we deal with an emotion recognition method using facial images and speech signal. Six basic human emotions including happiness, sadness, anger, surprise, fear and dislike are investigated. Emotion recognition using the facial expression is performed by using a multi-resolution analysis based on the discrete wavelet transform. And then, the feature vectors are extracted from the linear discriminant analysis method. On the other hand, the emotion recognition from speech signal method has a structure of performing the recognition algorithm independently for each wavelet subband and then the final recognition is obtained from a multi-decision making scheme.

• Proceedings of the Korean Institute of Intelligent Systems Conference
• /
• 2007.04a
• /
• pp.401-404
• /
• 2007

본 논문은 거울 투영을 이용하여 2D의 감정인식 데이터베이스를 3D에 적용 가능하다는 것을 증명한다. 또한, 감정 확률을 이용하여 퍼지 모델링을 기반으로한 얼굴표정을 생성하고, 표정을 움직이는 3가지 기본 움직임에 대한 퍼지이론을 적용하여 얼굴표현함수를 제안한다. 제안된 방법은 거울 투영을 통한 다중 이미지를 이용하여 2D에서 사용되는 감정인식에 대한 특징벡터를 3D에 적용한다. 이로 인해, 2D의 모델링 대상이 되는 실제 모델의 기본감정에 대한 비선형적인 얼굴표정을 퍼지를 기반으로 모델링한다. 그리고 얼굴표정을 표현하는데 기본 감정 6가지인 행복, 슬픔, 혐오, 화남, 놀람, 무서움으로 표현되며 기본 감정의 확률에 대해서 각 감정의 평균값을 사용하고, 6가지 감정 확률을 이용하여 동적 얼굴표정을 생성한다. 제안된 방법을 3D 인간형 아바타에 적용하여 실제 모델의 표정 벡터와 비교 분석한다.