• Annual Conference on Human and Language Technology
• /
• 2018.10a
• /
• pp.167-171
• /
• 2018

개체명 인식은 문서에서 인명, 지명, 기관명 등의 고유한 의미를 나타내는 단위인 개체명을 추출하고, 추출된 개체명의 범주를 결정하는 작업이다. 최근 개체명 인식과 관련된 연구는 입력 데이터의 앞, 뒤를 고려하기 위한 Bi-RNNs와 출력 데이터 간의 전이 확률을 이용한 CRFs를 결합한 방식을 기반으로 다양한 변형의 심층학습 방법론이 제안되고 있다. 그러나 대부분의 연구는 입력 단위를 단어나 형태소로 사용하고 있으며, 성능 향상을 위해 띄어쓰기 정보, 개체명 사전 자질, 품사 분포 정보 등 다양한 정보를 필요로 한다는 어려움이 있다. 본 논문은 기본적인 학습 말뭉치에서 얻을 수 있는 문자 기반의 입력 정보와 Multi-Head Attention을 추가한 Bi-GRU/CRFs을 이용한 다국어 개체명 인식 방법을 제안한다. 한국어, 일본어, 중국어, 영어에 제안 모델을 적용한 결과 한국어와 일본어에서는 우수한 성능(한국어 $F_1$ 84.84%, 일본어 $F_1$ 89.56%)을 보였다. 영어에서는 $F_1$ 80.83%의 성능을 보였으며, 중국어는 $F_1$ 21.05%로 가장 낮은 성능을 보였다.

• ETRI Journal
• /
• v.45 no.1
• /
• pp.93-104
• /
• 2023

Coreference resolution is a task in discourse analysis that links several headwords used in any document object. We suggest pointer networks-based coreference resolution for Korean using multi-task learning (MTL) with an attention mechanism for a hierarchical structure. As Korean is a head-final language, the head can easily be found. Our model learns the distribution by referring to the same entity position and utilizes a pointer network to conduct coreference resolution depending on the input headword. As the input is a document, the input sequence is very long. Thus, the core idea is to learn the word- and sentence-level distributions in parallel with MTL, while using a shared representation to address the long sequence problem. The suggested technique is used to generate word representations for Korean based on contextual information using pre-trained language models for Korean. In the same experimental conditions, our model performed roughly 1.8% better on CoNLL F1 than previous research without hierarchical structure.

• Annual Conference on Human and Language Technology
• /
• 2021.10a
• /
• pp.540-543
• /
• 2021

본 논문에서는 Multi-Head Attention 대신 Spatial Gating Unit을 사용하는 GMLP[1]에 작은 Attention 신경망을 추가한 모델을 구성하여 뉴스와 위키피디아 데이터로 사전학습을 실시하고 한국어 다운스트림 테스크(감성분석, 개체명 인식)에 적용해 본다. 그 결과, 감성분석에서 Multilingual BERT보다 0.27%높은 Accuracy인 87.70%를 보였으며, 개체명 인식에서는 1.6%높은 85.82%의 F1 Score를 나타내었다. 따라서 GMLP가 기존 Transformer Encoder의 Multi-head Attention[2]없이 SGU와 작은 Attention만으로도 BERT[3]와 견줄만한 성능을 보일 수 있음을 확인할 수 있었다. 또한 BERT와 추론 속도를 비교 실험했을 때 배치사이즈가 20보다 작을 때 BERT보다 1에서 6배 정도 빠르다는 것을 확인할 수 있었다.

• Smart Media Journal
• /
• v.12 no.9
• /
• pp.45-59
• /
• 2023

Weakly supervised object localization (WSOL) is a task of localizing an object in an image using only image-level labels. Previous studies have followed the conventional class activation mapping (CAM) pipeline. However, we reveal the current CAM approach suffers from problems which cause original CAM could not capture the complete defects features. This work utilizes a convolutional neural network (CNN) pretrained on image-level labels to generate class activation maps in a multi-scale manner to highlight discriminative regions. Additionally, a vision transformer (ViT) pretrained was treated to produce multi-head attention maps as an auxiliary detector. By integrating the CNN-based CAMs and attention maps, our approach localizes defective regions without requiring bounding box or pixel-level supervision during training. We evaluate our approach on a dataset of apple images with only image-level labels of defect categories. Experiments demonstrate our proposed method aligns with several Object Detection models performance, hold a promise for improving localization.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.16 no.6
• /
• pp.1833-1848
• /
• 2022

Named entity recognition (NER) is an important basic task in the field of Natural Language Processing (NLP). Recently deep learning approaches by extracting word segmentation or character features have been proved to be effective for Chinese Named Entity Recognition (CNER). However, since this method of extracting features only focuses on extracting some of the features, it lacks textual information mining from multiple perspectives and dimensions, resulting in the model not being able to fully capture semantic features. To tackle this problem, we propose a novel Multi-view Semantic Feature Fusion Model (MSFM). The proposed model mainly consists of two core components, that is, Multi-view Semantic Feature Fusion Embedding Module (MFEM) and Multi-head Self-Attention Mechanism Module (MSAM). Specifically, the MFEM extracts character features, word boundary features, radical features, and pinyin features of Chinese characters. The acquired font shape, font sound, and font meaning features are fused to enhance the semantic information of Chinese characters with different granularities. Moreover, the MSAM is used to capture the dependencies between characters in a multi-dimensional subspace to better understand the semantic features of the context. Extensive experimental results on four benchmark datasets show that our method improves the overall performance of the CNER model.

• Journal of IKEEE
• /
• v.28 no.1
• /
• pp.53-64
• /
• 2024

With the advancement of NLP(Natural Language Processing) models, conversational AI such as ChatGPT is becoming increasingly popular. To enhance processing speed and reduce power consumption, it is important to implement the Transformer algorithm, which forms the basis of the latest natural language processing models, in hardware. In particular, the multi-head attention and feed-forward network, which analyze the relationships between different words in a sentence through matrix multiplication, are the most computationally intensive core algorithms in the Transformer. In this paper, we propose a new variable systolic array based on the number of input words to enhance matrix multiplication speed. Quantization maintains Transformer accuracy, boosting memory efficiency and speed. For evaluation purposes, this paper verifies the clock cycles required in multi-head attention and feed-forward network and compares the performance with other multipliers.

• Annual Conference on Human and Language Technology
• /
• 2021.10a
• /
• pp.274-277
• /
• 2021

의존 구문 분석은 문장 내 의존소와 지배소 사이의 관계를 예측하여 문장 구조를 분석하는 자연어처리 태스크이다. 최근의 딥러닝 기반 의존 구문 분석 연구는 주로 포인터 네트워크를 사용하는 방법으로 연구되고 있다. 포인터 네트워크는 내부적으로 사용하는 attention 기법에 따라 성능이 달라질 수 있다. 따라서 본 논문에서는 포인터 네트워크 모델에 적용되는 attention 기법들을 비교 분석하고, 한국어 의존 구문 분석 모델에 가장 효과적인 attention 기법을 선별한다. KLUE 데이터 셋을 사용한 실험 결과, UAS는 biaffine attention을 사용할 때 95.14%로 가장 높은 성능을 보였으며, LAS는 multi-head attention을 사용했을 때 92.85%로 가장 높은 성능을 보였다.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.17 no.9
• /
• pp.2458-2482
• /
• 2023

Medical image segmentation techniques based on convolution neural networks indulge in feature extraction triggering redundancy of parameters and unsatisfactory target localization, which outcomes in less accurate segmentation results to assist doctors in diagnosis. In this paper, we propose a multi-level semantic-rich encoding-decoding network, which consists of a Pooling-Conv-Former (PCFormer) module and a Cbam-Dilated-Transformer (CDT) module. In the PCFormer module, it is used to tackle the issue of parameter explosion in the conservative transformer and to compensate for the feature loss in the down-sampling process. In the CDT module, the Cbam attention module is adopted to highlight the feature regions by blending the intersection of attention mechanisms implicitly, and the Dilated convolution-Concat (DCC) module is designed as a parallel concatenation of multiple atrous convolution blocks to display the expanded perceptual field explicitly. In addition, MultiHead Attention-DwConv-Transformer (MDTransformer) module is utilized to evidently distinguish the target region from the background region. Extensive experiments on medical image segmentation from Glas, SIIM-ACR, ISIC and LGG demonstrated that our proposed network outperforms existing advanced methods in terms of both objective evaluation and subjective visual performance.

• The Journal of Society for e-Business Studies
• /
• v.27 no.1
• /
• pp.15-28
• /
• 2022

Due to recent expansion of online market such as clothing, utilizing customer review has become a major marketing measure. User review has been used as a tool of analyzing sentiment of customers. Sentiment analysis can be largely classified with machine learning-based and lexicon-based method. Machine learning-based method is a learning classification model referring review and labels. As research of sentiment analysis has been developed, multi-modal models learned by images and video data in reviews has been studied. Characteristics of words in reviews are differentiated depending on products' and customers' categories. In this paper, sentiment is analyzed via considering review data and metadata of products and users. Gated Recurrent Unit (GRU), Long Short-Term Memory (LSTM), Self Attention-based Multi-head Attention models and Bidirectional Encoder Representation from Transformer (BERT) are used in this study. Same Multi-Layer Perceptron (MLP) model is used upon every products information. This paper suggests a multi-modal sentiment analysis model that simultaneously considers user reviews and product meta-information.

• Journal of Broadcast Engineering
• /
• v.27 no.6
• /
• pp.885-896
• /
• 2022

Motor failure in manufacturing plays an important role in future A/S and reliability. Motor failure is detected by measuring sound, current, and vibration. For the data used in this paper, the sound of the car's side mirror motor gear box was used. Motor sound consists of three classes. Sound data is input to the network model through a conversion process through MelSpectrogram. In this paper, various methods were applied, such as data augmentation to improve the performance of classifying fault motors and various methods according to class imbalance were applied resampling, reweighting adjustment, change of loss function and representation learning and classification into two stages. In addition, the curriculum learning method and self-space learning method were compared through a total of five network models such as Bidirectional LSTM Attention, Convolutional Recurrent Neural Network, Multi-Head Attention, Bidirectional Temporal Convolution Network, and Convolution Neural Network, and the optimal configuration was found for motor sound classification.