• 대한임베디드공학회논문지
• /
• 제17권2호
• /
• pp.93-100
• /
• 2022

In this paper, we propose a machine learning-based method for supporting resource management of IoT software platforms in a multi-modal sensing scenario. We assume that an IoT device installed with a oneM2M-compatible software platform is connected with various sensors such as PIR, sound, dust, ambient light, ultrasonic, accelerometer, through different embedded system interfaces such as general purpose input output (GPIO), I2C, SPI, USB. Based on a collected dataset including CPU usage and user-defined priority, a machine learning model is trained to estimate the level of nice value required to adjust according to the resource usage patterns. The proposed method is validated by comparing with a rule-based control strategy, showing its practical capability in a multi-modal sensing scenario of IoT devices.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• 제17권1호
• /
• pp.51-73
• /
• 2023

Social media play a significant role in communicating information across the globe, connecting with loved ones, getting the news, communicating ideas, etc. However, a group of people uses social media to spread fake information, which has a bad impact on society. Therefore, minimizing fake news and its detection are the two primary challenges that need to be addressed. This paper presents a multi-modal deep learning technique to address the above challenges. The proposed modal can use and process visual and textual features. Therefore, it has the ability to detect fake information from visual and textual data. We used EfficientNetB0 and a sentence transformer, respectively, for detecting counterfeit images and for textural learning. Feature embedding is performed at individual channels, whilst fusion is done at the last classification layer. The late fusion is applied intentionally to mitigate the noisy data that are generated by multi-modalities. Extensive experiments are conducted, and performance is evaluated against state-of-the-art methods. Three real-world benchmark datasets, such as MediaEval (Twitter), Weibo, and Fakeddit, are used for experimentation. Result reveals that the proposed modal outperformed the state-of-the-art methods and achieved an accuracy of 86.48%, 82.50%, and 88.80%, respectively, for MediaEval (Twitter), Weibo, and Fakeddit datasets.

• 정보과학회 컴퓨팅의 실제 논문지
• /
• 제23권2호
• /
• pp.104-109
• /
• 2017

본고에서는 다중 웨어러블 센서 데이터로부터 사용자의 일상 생활 행동 패턴을 분석할 수 있는 새로운 기계학습 모델을 제안한다. 제안하는 모델은 다중 웨어러블 센서 데이터를 효과적으로 학습하기 위하여 사람이 다중 센서 정보를 처리하는 방법을 적용한 새로운 신경망 모델이다. 제안하는 Gated multi-modal neural netoworks는 계층적 신경망 구조를 가지고 있으며 Gate 모듈을 통해 각 센서 데이터를 선택적으로 결합하여 처리하는 특징을 가진다. 실험을 위해 다중 웨어러블 장치를 착용하고 일상 생활 중 한 가지인 레스토랑에서의 행동 센서 데이터를 수집하였다. 실험 결과로서, 제시하는 모델을 이용하여 실제 웨어러블 센서 데이터를 분석하였을 때 분류 정확도가 비교적 정확하고 빠르게 처리할 수 있음을 확인하였다. 또한 모델의 중간 계층에서의 노드의 활성화 패턴 분석을 통해 자동으로 일상생활 패턴을 추출할 수 있고 이를 이용하여 지식 스키마를 생성할 수 있음을 확인하였다.

• 한국멀티미디어학회논문지
• /
• 제23권2호
• /
• pp.146-154
• /
• 2020

Negative emotion causes stress and lack of attention concentration. The classification of negative emotion is important to recognize risk factors. To classify emotion status, various methods such as questionnaires and interview are used and it could be changed by personal thinking. To solve the problem, we acquire multi modal bio-signals such as electrocardiogram (ECG), skin temperature (ST), galvanic skin response (GSR) and extract features. The neural network (NN), the deep neural network (DNN), and the deep belief network (DBN) is designed using the multi modal bio-signals to analyze emotion status. As a result, the DBN based on features extracted from ECG, ST and GSR shows the highest accuracy (93.8%). It is 5.7% higher than compared to the NN and 1.4% higher than compared to the DNN. It shows 12.2% higher accuracy than using only single bio-signal (GSR). The multi modal bio-signal acquisition and the deep learning classifier play an important role to classify emotion.

• 제어로봇시스템학회논문지
• /
• 제20권6호
• /
• pp.619-624
• /
• 2014

This paper introduces a robot-based learning system which is designed to teach multiplication to children. In addition to a small humanoid and a smart device delivering educational content, we employ a type of mixed-initiative operation which provides enhanced multi-modal cognition to the r-learning system through human intervention. To investigate major factors that influence people's intention to use the r-learning system and to see how the multi-modality affects the connections, we performed a user study based on TAM (Technology Acceptance Model). The results support the fact that the quality of the system and the natural interaction are key factors for the r-learning system to be used, and they also reveal very interesting implications related to the human behaviors.

• 방송공학회논문지
• /
• 제23권3호
• /
• pp.351-360
• /
• 2018

인간 감정 인식은 컴퓨터 비전 및 인공 지능 영역에서 지속적인 관심을 받는 연구 주제이다. 본 논문에서는 wild 환경에서 이미지, 얼굴 특징점 및 음성신호로 구성된 multi-modal 신호를 기반으로 여러 신경망을 통해 인간의 감정을 분류하는 방법을 제안한다. 제안 방법은 다음과 같은 특징을 갖는다. 첫째, multi task learning과 비디오의 시공간 특성을 이용한 준 감독 학습을 사용함으로써 영상 기반 네트워크의 학습 성능을 크게 향상시켰다. 둘째, 얼굴의 1 차원 랜드 마크 정보를 2 차원 영상으로 변환하는 모델을 새로 제안하였고, 이를 바탕으로 한 CNN-LSTM 네트워크를 제안하여 감정 인식을 향상시켰다. 셋째, 특정 감정에 오디오 신호가 매우 효과적이라는 관측을 기반으로 특정 감정에 robust한 오디오 심층 학습 메커니즘을 제안한다. 마지막으로 소위 적응적 감정 융합 (emotion adaptive fusion)을 적용하여 여러 네트워크의 시너지 효과를 극대화한다. 제안 네트워크는 기존의 지도 학습과 반 지도학습 네트워크를 적절히 융합하여 감정 분류 성능을 향상시켰다. EmotiW2017 대회에서 주어진 테스트 셋에 대한 5번째 시도에서, 제안 방법은 57.12 %의 분류 정확도를 달성하였다.

• 한국컴퓨터정보학회:학술대회논문집
• /
• 한국컴퓨터정보학회 2022년도 제66차 하계학술대회논문집 30권2호
• /
• pp.99-100
• /
• 2022

본 논문에서는 유방암 초음파 이미지를 학습한 multi-modal 구조를 이용하여 유방암을 판별하는 인공지능을 제안한다. 학습된 인공지능은 유방암을 판별과 동시에, 설명 가능한 인공지능 기법과 ROI를 함께 사용하여 종양의 위치를 나타내준다. 시각적으로 판단 근거를 제시하기 때문에 인공지능의 판단 신뢰도는 더 높아진다.

• 한국정보처리학회:학술대회논문집
• /
• 한국정보처리학회 2023년도 추계학술발표대회
• /
• pp.526-527
• /
• 2023

Speech emotion recognition(SER) is one of the interesting topics in the machine learning field. By developing multi-modal speech emotion recognition system, we can get numerous benefits. This paper explain about fusing BERT as the text recognizer and CNN as the speech recognizer to built a multi-modal SER system.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• 제18권3호
• /
• pp.670-684
• /
• 2024

3D Cross-Modal Retrieval (3DCMR) is a task that retrieves 3D objects regardless of modalities, such as images, meshes, and point clouds. One of the most prominent methods used for 3DCMR is the Cross-Modal Center Loss Function (CLF) which applies the conventional center loss strategy for 3D cross-modal search and retrieval. Since CLF is based on center loss, the center features in CLF are also susceptible to subtle changes in hyperparameters and external inferences. For instance, performance degradation is observed when the batch size is too small. Furthermore, the Mean Squared Error (MSE) used in CLF is unable to adapt to changes in batch size and is vulnerable to data variations that occur during actual inference due to the use of simple Euclidean distance between multi-modal features. To address the problems that arise from small batch training, we propose a Noisy Center Loss (NCL) method to estimate the optimal center features. In addition, we apply the simple Siamese representation learning method (SimSiam) during optimal center feature estimation to compare projected features, making the proposed method robust to changes in batch size and variations in data. As a result, the proposed approach demonstrates improved performance in ModelNet40 dataset compared to the conventional methods.

• 한국멀티미디어학회논문지
• /
• 제23권8호
• /
• pp.1006-1018
• /
• 2020

Dataset is getting more import to develop a learning based algorithm. Quality of the algorithm definitely depends on dataset. So we introduce new dataset over 200 thousands images which are fully labeled multi modal sensor data. Proposed dataset was designed and constructed for researchers who want to develop detection, tracking, and action classification in outdoor environment for surveillance scenarios. The dataset includes various images and multi modal sensor data under different weather and lighting condition. Therefor, we hope it will be very helpful to develop more robust algorithm for systems equipped with difference kinds of sensors in outdoor application. Case studies with the proposed dataset are also discussed in this paper.