• The Transactions of the Korea Information Processing Society
• /
• v.13 no.4
• /
• pp.174-180
• /
• 2024

In this paper, we explore an emotion classification method through multimodal learning utilizing wav2vec 2.0 and KcELECTRA models. It is known that multimodal learning, which leverages both speech and text data, can significantly enhance emotion classification performance compared to methods that solely rely on speech data. Our study conducts a comparative analysis of BERT and its derivative models, known for their superior performance in the field of natural language processing, to select the optimal model for effective feature extraction from text data for use as the text processing model. The results confirm that the KcELECTRA model exhibits outstanding performance in emotion classification tasks. Furthermore, experiments using datasets made available by AI-Hub demonstrate that the inclusion of text data enables achieving superior performance with less data than when using speech data alone. The experiments show that the use of the KcELECTRA model achieved the highest accuracy of 96.57%. This indicates that multimodal learning can offer meaningful performance improvements in complex natural language processing tasks such as emotion classification.

• International Journal of Fuzzy Logic and Intelligent Systems
• /
• v.5 no.2
• /
• pp.157-163
• /
• 2005

In this paper, we present the multimodal system based on the fusion of two user-friendly biometric modalities: Iris and Face. In order to reach robust identification and verification we are going to combine two different biometric features. we specifically apply 2-D discrete wavelet transform to extract the feature sets of low dimensionality from iris and face. And then to obtain Reduced Joint Feature Vector(RJFV) from these feature sets, Direct Linear Discriminant Analysis (DLDA) is used in our multimodal system. In addition, the Synergetic Neural Network(SNN) is used to obtain matching score of the preprocessed data. This system can operate in two modes: to identify a particular person or to verify a person's claimed identity. Our results for both cases show that the proposed method leads to a reliable person authentication system.

• Journal of the Korean Chemical Society
• /
• v.61 no.4
• /
• pp.211-217
• /
• 2017

The purpose of this study was to examine UK students' use of multimodal representations in science. Students were asked to explain their understandings of the scientific concept and presentation of the multimodal representations in a science Assessing Pupils' Progress (APP) task. Participants of this study were fifty-four Year 7 students taught by the same teacher. Students from one class (27 students) were assigned to the experimental group, and then they received instruction encouraging the using of multimodal representations as evidences to support students' claims. One class (27 students) was assigned to the control group and they received instruction with traditional teaching methods. Both groups performed an APP task for assessment. The samples of APP assessments produced by students both from the experimental and control groups were analyzed using an analysis framework of multimodal representations, embeddedness in evidence and understanding of scientific concepts. Data analysis indicated that the students in the experimental group performed better than that of the control group on embeddedness of multimodal representations in the APP task. In addition, there was a significant difference between the two groups in the evaluation of understand of the scientific concepts.

• Journal of Animal Science and Technology
• /
• v.63 no.6
• /
• pp.1443-1452
• /
• 2021

This study aimed to investigate the effects of the multimodal group intervention that combined animal-assisted therapy (AAT) and integrated play therapy (IPT) on depression and self-esteem in undergraduate university students. The subjects were 40 students attending animal-related and social welfare departments of universities located in a metropolitan city. The multimodal intervention program was conducted for eight sessions (twice a week, 60 min each) in the experimental group. Data analysis was conducted using the independent sample t-test, ANCOVA, and paired sample t-test for pre- and post-test. Therefore, the multimodal intervention program applying AAT and IPT showed positive effects on depression and self-esteem in university students. Based on these results, this study proposed the operation of multidisciplinary education and practical and policy utilization methods to reduce depression among university students and help improve their self-esteem.

• Journal of the Korean Chemical Society
• /
• v.56 no.4
• /
• pp.500-508
• /
• 2012

The purpose of this study was to investigate the impact of multimodal representation-based lesson on embeddedness of multimodal representation in high school students' writing. The participants in this study were two groups of second-year science-track students (74 students) at an academic high school in a metropolitan city. One group (41 students) was assigned to the experimental group, the other group (33 students) was assigned to the comparative group. Data analysis showed that the students of the experimental group were better at utilizing and embedding multimodal representations. Thus, the conclusion was drawn that multimodal representation-based lesson had an effect on high school students' embeddedness of multimodal representation.

• Proceedings of the Korean Institute of Navigation and Port Research Conference
• /
• 2004.04a
• /
• pp.421-426
• /
• 2004

Increasing of the multinational corporation brought into the international multimodal Increasing of the multinational corporation brought into the international multimodal transport on the logistics environment. In case of Europe which have a great infrastructure, they are tried to develope a second of the silk road constantly. This paper emphasized the importance of international multimodal transport and proposed the model for northeast multimodal transport. For this research, we analyzed the multimodal transport system in Europe and north corridor of TAR. We are expecting economic effect of the route is including republic of korea and developed a model for connecting with sea, air and road. Actually, this research can not be enough data of numerical value for proving this effectiveness. but we developed and proposed a specific route of multimodal transport that was never suggested. Consequently, we established basic ground for comparing each transport route in the future research.

• International Journal of Fuzzy Logic and Intelligent Systems
• /
• v.6 no.4
• /
• pp.304-313
• /
• 2006

This paper describes a novel way of applying capacitive sensing technology to a mobile user interface. The key idea is to use grip-pattern, which is naturally produced when a user tries to use the mobile device, as a clue to determine an application to be launched. To this end, a capacitive touch sensing system is carefully designed and installed underneath the housing of the mobile device to capture the information of the user's grip-pattern. The captured data is then recognized by dedicated recognition algorithms. The feasibility of the proposed user interface system is thoroughly evaluated with various recognition tests.

• ETRI Journal
• /
• v.44 no.3
• /
• pp.426-437
• /
• 2022

To understand the multilateral characteristics of human behavior and physiological markers related to physical, emotional, and environmental states, extensive lifelog data collection in a real-world environment is essential. Here, we propose a data collection method using multimodal mobile sensing and present a long-term dataset from 22 subjects and 616 days of experimental sessions. The dataset contains over 10 000 hours of data, including physiological, data such as photoplethysmography, electrodermal activity, and skin temperature in addition to the multivariate behavioral data. Furthermore, it consists of 10 372 user labels with emotional states and 590 days of sleep quality data. To demonstrate feasibility, human activity recognition was applied on the sensor data using a convolutional neural network-based deep learning model with 92.78% recognition accuracy. From the activity recognition result, we extracted the daily behavior pattern and discovered five representative models by applying spectral clustering. This demonstrates that the dataset contributed toward understanding human behavior using multimodal data accumulated throughout daily lives under natural conditions.

• The Journal of the Korea institute of electronic communication sciences
• /
• v.14 no.1
• /
• pp.69-80
• /
• 2019

This paper proposes a multimodal sensor platform which supports plug and play (PnP) technology. PnP technology automatically recognizes a connected sensor module and an application program easily controls a sensor. To verify a multimodal platform for PnP technology, we build up a firmware and have the experiment on a sensor system. When a sensor module is connected to the platform, a firmware recognizes the sensor module and reads sensor data. As a result, it provides PnP technology to simply plug sensors without any software configuration. Measured sensor raw data suffer from various distortions such as gain, offset, and non-linearity errors. Therefore, we introduce a polynomial calculation to compensate for sensor distortions. To find the optimal coefficients for sensor calibration, we apply a genetic algorithm which reduces the calibration time. It achieves reasonable performance using only a few data points with reducing 97% error in the worst case. The platform supports various protocols for multimodal sensors, i.e., UART, I2C, I2S, SPI, and GPIO.

• Proceedings of the KSR Conference
• /
• 2011.10a
• /
• pp.607-613
• /
• 2011

This paper reviews search methods of multiple reasonable paths to implement multimodal public transportation network of Seoul. Such a large scale multimodal public transportation network as Seoul, the computation time of path finding algorithm is a key and the result of path should reflect route choice behavior of public transportation passengers. Search method of alternative path is divided by removing path method and deviation path method. It analyzes previous researches based on the complexity of algorithm for large-scale network. Applying path finding algorithm in public transportation network, transfer and loop constraints must be included to be able to reflect real behavior. It constructs the generalized cost function based on the smart card data to reflect travel behavior of public transportation. To validate the availability of algorithm, experiments conducted with Seoul metropolitan public multimodal transportation network consisted with 22,109 nodes and 215,859 links by using the deviation path method, suitable for large-scale network.