• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.16 no.7
• /
• pp.2407-2424
• /
• 2022

Recently, Transformer has made great progress in video retrieval tasks due to its high representation capability. For the structure of a Transformer, the cascaded self-attention modules are capable of capturing long-distance feature dependencies. However, the local feature details are likely to have deteriorated. In addition, increasing the depth of the structure is likely to produce learning bias in the learned features. In this paper, an improved Transformer structure named TransDCS (Transformer with Dynamic Convolution and Shortcut) is proposed. A Multi-head Conv-Self-Attention module is introduced to model the local dependencies and improve the efficiency of local features extraction. Meanwhile, the augmented shortcuts module based on a dual identity matrix is applied to enhance the conduction of input features, and mitigate the learning bias. The proposed model is tested on MSRVTT, LSMDC and Activity-Net benchmarks, and it surpasses all previous solutions for the video-text retrieval task. For example, on the LSMDC benchmark, a gain of about 2.3% MdR and 6.1% MnR is obtained over recently proposed multimodal-based methods.

• The Journal of the Korea Contents Association
• /
• v.22 no.1
• /
• pp.138-147
• /
• 2022

Recently, chatbots have been applied in various fields and have shown good results, and many attempts to use chatbots in shopping mall product recommendation services are being conducted on e-commerce platforms. In this paper, for a conversation system that recommends a fashion that a user wants based on conversation between the user and the system and fashion image information, a transformer model that is currently performing well in various AI fields such as natural language processing, voice recognition, and image recognition. We propose a multimodal-based improved transformer model that is improved to increase the accuracy of recommendation by using dialogue (text) and fashion (image) information together for data preprocessing and data representation. We also propose a method to improve accuracy through data improvement by analyzing the data. The proposed system has a recommendation accuracy score of 0.6563 WKT (Weighted Kendall's tau), which significantly improved the existing system's 0.3372 WKT by 0.3191 WKT or more.

• Journal of Korean Elementary Science Education
• /
• v.42 no.1
• /
• pp.161-177
• /
• 2023

Scientific explanation is composed of various representations such as texts, diagrams, and graphs, and each representation contributes to expanding scientific meaning by connecting similar but slightly different meanings as a 'mode'. Based on this perspective of social semiotics, we aimed to identify the characteristics of meaning formation demonstrated in students' science writing (verbal mode) and science drawing (visual mode) and to discuss the pedagogical use of multimodal representations. To that end, 18 science drawings and 18 scientific writings constructed by science-gifted elementary students on air pressure were collected. The characteristics of the drawn and written explanations were then analyzed from the affordance perspective in social semiotics. In science drawing, students showed a tendency to use the affordance of the visual mode to infer concrete changes from the particle view, such as the movement of air particles, the number of air particles, and the collision of particles. In science writing, students used the affordance of the verbal mode mainly to infer the causal relationship between the concept of air pressure and other related factors at an abstract level. Based on those results, we discuss the educational implications and provide concrete examples of how to use the unique affordances of each form to complement one another.

• Journal of Internet Computing and Services
• /
• v.10 no.5
• /
• pp.165-181
• /
• 2009

This paper presents an effective representation scheme for the shape analysis of the hippocampal structure and a stereoscopic-haptic environment to enhance sense of realism. The parametric model and the 3D skeleton represent various types of hippocampal shapes and they are stored in the Octree data structure. So they can be used for the interactive shape analysis. And the 3D skeleton-based pose normalization allows us to align a position and an orientation of the 3D hippocampal models constructed from multimodal medical imaging data. We also have trained Support Vector Machine (SVM) for classifying between the normal controls and epileptic patients. Results suggest that the presented representation scheme provides various level of shape representation and the SVM can be a useful classifier in analyzing the shape differences between two groups. A stereoscopic-haptic virtual environment combining an auto-stereoscopic display with a force-feedback (or haptic) device takes an advantage of 3D applications for medicine because it improves space and depth perception.

• International Journal of Computer Science & Network Security
• /
• v.21 no.4
• /
• pp.103-114
• /
• 2021

A conceptual model can be used to manage complexity in both the design and implementation phases of the system development life cycle. Such a model requires a firm grasp of the abstract principles on which a system is based, as well as an understanding of the high-level nature of the representation of entities and processes. In this context, models can have distinct architectural characteristics. This paper discusses model multiplicity (e.g., unified modeling language [UML]), model singularity (e.g., object-process methodology [OPM], thinging machine [TM]), and a heterogeneous model that involves multiplicity and singularity. The basic idea of model multiplicity is that it is not possible to present all views in a single representation, so a number of models are used, with each model representing a different view. The model singularity approach uses only a single unified model that assimilates its subsystems into one system. This paper is concerned with current approaches, especially in software engineering texts, where multimodal UML is introduced as the general-purpose modeling language (i.e., UML is modeling). In such a situation, we suggest raising the issue of multiplicity versus singularity in modeling. This would foster a basic appreciation of the UML advantages and difficulties that may be faced during modeling, especially in the educational setting. Furthermore, we advocate the claim that a multiplicity of views does not necessitate a multiplicity of models. The model singularity approach can represent multiple views (static, behavior) without resorting to a collection of multiple models with various notations. We present an example of such a model where the static representation is developed first. Then, the dynamic view and behavioral representations are built by incorporating a decomposition strategy interleaved with the notion of time.

• Transactions of the Korean Society for Noise and Vibration Engineering
• /
• v.24 no.4
• /
• pp.324-331
• /
• 2014

The sensor configuration of the magnetostrictive guided wave system can be described as a single continuous transducing element which makes it difficult to separate the individual modes from the reflected signal. In this work we develop the mode decomposition technique employing chirplet transform based on the maximum likelihood estimation, which is able to separate the individual modes from dispersive and multimodal waveform measured with the magnetostrictive sensor, and estimate the time-frequency centers and individual energies of the reflection, which would be used to locate and characterize defects. Simulation results on a carbon steel pipe are presented, which show the accurate mode separation and more discernible time-frequency representation could become enabled using the proposed technique.

• International journal of advanced smart convergence
• /
• v.12 no.4
• /
• pp.434-442
• /
• 2023

The relationship between acute kidney injury (AKI) prediction and nephrotoxic drugs, or drugs that adversely affect kidney function, is one that has yet to be explored in the critical care setting. One contributing factor to this gap in research is the limited investigation of drug modalities in the intensive care unit (ICU) context, due to the challenges of processing prescription data into the corresponding drug representations and a lack in the comprehensive understanding of these drug representations. This study addresses this gap by proposing a novel approach that leverages patient prescription data as a modality to improve existing models for AKI prediction. We base our research on Electronic Health Record (EHR) data, extracting the relevant patient prescription information and converting it into the selected drug representation for our research, the extended-connectivity fingerprint (ECFP). Furthermore, we adopt a unique multimodal approach, developing machine learning models and 1D Convolutional Neural Networks (CNN) applied to clinical drug representations, establishing a procedure which has not been used by any previous studies predicting AKI. The findings showcase a notable improvement in AKI prediction through the integration of drug embeddings and other patient cohort features. By using drug features represented as ECFP molecular fingerprints along with common cohort features such as demographics and lab test values, we achieved a considerable improvement in model performance for the AKI prediction task over the baseline model which does not include the drug representations as features, indicating that our distinct approach enhances existing baseline techniques and highlights the relevance of drug data in predicting AKI in the ICU setting.

• The Journal of Society for e-Business Studies
• /
• v.27 no.1
• /
• pp.15-28
• /
• 2022

Due to recent expansion of online market such as clothing, utilizing customer review has become a major marketing measure. User review has been used as a tool of analyzing sentiment of customers. Sentiment analysis can be largely classified with machine learning-based and lexicon-based method. Machine learning-based method is a learning classification model referring review and labels. As research of sentiment analysis has been developed, multi-modal models learned by images and video data in reviews has been studied. Characteristics of words in reviews are differentiated depending on products' and customers' categories. In this paper, sentiment is analyzed via considering review data and metadata of products and users. Gated Recurrent Unit (GRU), Long Short-Term Memory (LSTM), Self Attention-based Multi-head Attention models and Bidirectional Encoder Representation from Transformer (BERT) are used in this study. Same Multi-Layer Perceptron (MLP) model is used upon every products information. This paper suggests a multi-modal sentiment analysis model that simultaneously considers user reviews and product meta-information.

• Journal of The Korean Association For Science Education
• /
• v.35 no.3
• /
• pp.353-361
• /
• 2015

This study focused on the use of peer assessment activities to investigate its the impact on students' argumentation skills in argument-based inquiry. The participants of the study were 106 10th grade students (four classes). Two classes were assigned to the experimental group, and the other two classes were assigned to the comparative group. The experimental group was taught argument-based inquiry through the application of peer assessment activities. The comparative group was taught argument-based inquiry without peer assessments. At the claim and evidence stage, students were asked to evaluate whether peers' claims fit with the evidence and whether peers' explanation of the evidences validity was sufficient. The quality of argumentation used in the students' writing was different in each group. According to the analysis of the summary writing test, the results showed that the experimental group had a significantly higher mean score than the comparative group in argumentation components, including evidence and warrant/backing. In addition, the experimental group used better multimodal representation including explanation of evidence than the comparative group. The findings showed that argument-based inquiry applying peer assessment activities had an effect on the argumentation skills in students' writing.

• Journal of The Korean Association For Science Education
• /
• v.32 no.7
• /
• pp.1099-1108
• /
• 2012

The purpose of this study was to examine the effects of argument structure on students' writing in implementation of argument-based inquiry using the Science Writing Heuristic (SWH) approach. Participants of this study were 108 8th grade students (three classes). Two classes (68 students) were assigned to an experimental group, and the other class (35 students) was assigned to a comparative group. The experimental group was taught argument-based inquiry using the Science Writing Heuristic (SWH) approach, while the comparative group was taught with the traditional teaching strategy. After implementing this program, the two groups were asked to write summaries using structured argumentation in their writing. The result of this study showed that the experimental group used better argument structure and multimodal representation such as pictures, graphs and examples in evidence than the comparative group. The quality of evidence used in the students' writing was different between two groups. Students of the comparative group only listed fragments of science concepts for evidence to support their claims, but students of the experimental group explained science concepts by giving specific examples. The findings show that argument-based inquiry using the SWH approach was effective on argument structure in students' writing.