• Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography
• /
• v.38 no.6
• /
• pp.635-644
• /
• 2020

Object recognition, detection and instance segmentation based on DL (Deep Learning) have being used in various practices, and mainly optical images are used as training data for DL models. The major objective of this paper is object segmentation and building detection by utilizing multimodal datasets as well as optical images for training Detectron2 model that is one of the improved R-CNN (Region-based Convolutional Neural Network). For the implementation, infrared aerial images, LiDAR data, and edges from the images, and Haralick features, that are representing statistical texture information, from LiDAR (Light Detection And Ranging) data were generated. The performance of the DL models depends on not only on the amount and characteristics of the training data, but also on the fusion method especially for the multimodal data. The results of segmenting objects and detecting buildings by applying hybrid fusion - which is a mixed method of early fusion and late fusion - results in a 32.65% improvement in building detection rate compared to training by optical image only. The experiments demonstrated complementary effect of the training multimodal data having unique characteristics and fusion strategy.

• International Journal of Computer Science & Network Security
• /
• v.22 no.7
• /
• pp.268-282
• /
• 2022

One of the most important technologies today is augmented reality technology, it allows users to experience the real world using virtual objects that are combined with the real world. This technology is interesting and has become applied in many sectors such as the shopping and medicine, also it has been included in the sector of education. In the field of education, AR technology has become widely used due to its effectiveness. It has many benefits, such as arousing students' interest in learning imaginative concepts that are difficult to understand. On the other hand, studies have proven that collaborative between students increases learning opportunities by exchanging information, and this is known as Collaborative Learning. The use of multimodal creates a distinctive and interesting experience, especially for students, as it increases the interaction of users with the technologies. The research aims at developing collaborative framework for developing achievement of 6th graders through designing a framework that integrated a collaborative framework with a multimodal input "hand-gesture and touch", considering the development of an effective, fun and easy to use framework with a multimodal interaction in AR technology that was applied to reformulate the genetics and traits lesson from the science textbook for the 6th grade, the first semester, the second lesson, in an interactive manner by creating a video based on the science teachers' consultations and a puzzle game in which the game images were inserted. As well, the framework adopted the cooperative between students to solve the questions. The finding showed a significant difference between post-test and pre-test of the experimental group on the mean scores of the science course at the level of remembering, understanding, and applying. Which indicates the success of the framework, in addition to the fact that 43 students preferred to use the framework over traditional education.

• Journal of Chest Surgery
• /
• v.55 no.5
• /
• pp.405-412
• /
• 2022

Background: Malignant pleural mesothelioma (MPM) is an aggressive pleural malignancy, and despite all multimodal treatment modalities, the 5-year overall survival rate of patients with MPM is less than 20%. In the present study, we aimed to analyze the surgical and prognostic outcomes of patients with MPM who received multimodal treatment. Methods: In this retrospective, single-center study, the records of patients who underwent surgery for MPM between January 2010 and December 2020 at our department were reviewed retrospectively. Results: Sixty-four patients were included in the study, of whom 23 (35.9%) were women and 41 (64.1%) were men. Extrapleural pneumonectomy, pleurectomy/decortication, and extended pleurectomy/decortication procedures were performed in 34.4%, 45.3%, and 20.3% of patients, respectively. The median survival of patients was 21 months, and the 5-year survival rate was 20.2%. Advanced tumor stage (hazard ratio [HR], 1.8; p=0.04), right-sided extrapleural pneumonectomy (HR, 3.1; p=0.02), lymph node metastasis (HR, 1.8; p=0.04), and incomplete multimodal therapy (HR, 1.9; p=0.03) were poor prognostic factors. There was no significant survival difference according to surgical type or histopathological subtype. Conclusion: Multimodal therapy can offer an acceptable survival rate in patients with MPM. Despite its poor reputation in the literature, the survival rate after extrapleural pneumonectomy, especially left-sided, was not as poor as might be expected.

• The Transactions of the Korea Information Processing Society
• /
• v.13 no.4
• /
• pp.174-180
• /
• 2024

In this paper, we explore an emotion classification method through multimodal learning utilizing wav2vec 2.0 and KcELECTRA models. It is known that multimodal learning, which leverages both speech and text data, can significantly enhance emotion classification performance compared to methods that solely rely on speech data. Our study conducts a comparative analysis of BERT and its derivative models, known for their superior performance in the field of natural language processing, to select the optimal model for effective feature extraction from text data for use as the text processing model. The results confirm that the KcELECTRA model exhibits outstanding performance in emotion classification tasks. Furthermore, experiments using datasets made available by AI-Hub demonstrate that the inclusion of text data enables achieving superior performance with less data than when using speech data alone. The experiments show that the use of the KcELECTRA model achieved the highest accuracy of 96.57%. This indicates that multimodal learning can offer meaningful performance improvements in complex natural language processing tasks such as emotion classification.

• Journal of Navigation and Port Research
• /
• v.39 no.5
• /
• pp.409-415
• /
• 2015

According to the globalization of world economy on distribution and sales, logistics and transportation parts are playing an important role. Especially, they have to decide what is the key factor of route choice model and how to choose the right transport route in multimodal transport system. By considering the key factors in rote choice model for freight forwarders between Mongolia and Korea, this paper propose 4 main factors: Cost, Delivery time, Freight and Logistics service with 13 sub factors. The importance of factors is surveyed base on AHP through interview with freight forwarders. In results, the empirical insights about current status of Mongolian forwarders are provided with different factors between transportation modes. Expecially, the Time factor is a role factor to choose transport route for air transportation forwarders.

• Proceedings of the KIEE Conference
• /
• 2008.04c
• /
• pp.44-46
• /
• 2008

This paper Proposes a new couple particle swarm optimization (CPSO) for multimodal functions. In this method, main particles are generated uniformly using Faure-sequences, and move accordingly to cognition only model. If any main particle detects the movement direction which has local optimum, this particle would create a new particle beside itself and make a couple. After that, all couples move accordingly to conventional particle swarm optimization (PSO) model. If these couples tend toward the same local optimum, only the best couple would be kept and the others would be eliminated. We had applied this method to some analytic multimodal functions and successfully locate all local optima.

• Proceedings of the Korean Information Science Society Conference
• /
• 2011.06b
• /
• pp.231-234
• /
• 2011

Audiovisual-based human emotion recognition can be considered a good approach for multimodal humancomputer interaction. However, the optimal multimodal information fusion remains challenges. In order to overcome the limitations and bring robustness to the interface, we propose a framework of automatic human emotion recognition system from speech and face display. In this paper, we develop a new approach for fusing information in model-level based on the relationship between speech and face expression to detect automatic temporal segments and perform multimodal information fusion.

• Journal of Institute of Control, Robotics and Systems
• /
• v.6 no.12
• /
• pp.1061-1069
• /
• 2000

In this paper, a new genetic algorithm is proposed for solving multimodal function optimization problems that are not easily solved by conventional genetic algorithm(GA)s. This algorithm finds one of local optima first and another optima at the next iteration. By repeating this process, we can locate all the local solutions instead of one local solution as in conventional GAs. To avoid converging to the same optimum again, we devise a new genetic operator, called a Mendel operator which simulates the Mendels genetic law. The proposed algorithm remembers the optima obtained so far, compels individuals to move away from them, and finds a new optimum.

• Smart Structures and Systems
• /
• v.13 no.3
• /
• pp.435-451
• /
• 2014

In this paper, we investigate the vibration control of multimodal structures and present an efficient control law that requires less energy supply than active strategies. This strategy is called modal global semi-active control and is designed to work as effectively as the active control and consume less power which represents its major limitation. The proposed law is based on an energetic management of the optimal law such that the controller follows this latter only if there is sufficient energy which will be extracted directly from the system vibrations itself. The control algorithm is presented and validated for a cantilever beam structure subjected to external perturbations. Comparisons between the proposed law performances and those obtained by independent modal space control (IMSC) and semi-active control schemes are offered.

• Proceedings of the KSPS conference
• /
• 2007.05a
• /
• pp.29-32
• /
• 2007

This paper describes a multimodal dialog system that uses Hidden Information State (HIS) method to manage the human-machine dialog. HIS dialog manager is a variation of classic partially observable Markov decision process (POMDP), which provides one of the stochastic dialog modeling frameworks. Because dialog modeling using conventional POMDP requires very large size of state space, it has been hard to apply POMDP to the real domain of dialog system. In HIS dialog manager, system groups the belief states to reduce the size of state space, so that HIS dialog manager can be used in real world domain of dialog system. We adapted this HIS method to Smart-home domain multimodal dialog system.