• ETRI Journal
• /
• 제38권3호
• /
• pp.487-493
• /
• 2016

Two new methods are proposed for an unsupervised adaptation of a language model (LM) with a single sentence for automatic transcription tasks. At the training phase, training documents are clustered by a method known as Latent Dirichlet allocation (LDA), and then a domain-specific LM is trained for each cluster. At the test phase, an adapted LM is presented as a linear mixture of the now trained domain-specific LMs. Unlike previous adaptation methods, the proposed methods fully utilize a trained LDA model for the estimation of weight values, which are then to be assigned to the now trained domain-specific LMs; therefore, the clustering and weight-estimation algorithms of the trained LDA model are reliable. For the continuous speech recognition benchmark tests, the proposed methods outperform other unsupervised LM adaptation methods based on latent semantic analysis, non-negative matrix factorization, and LDA with n-gram counting.

• 한국빅데이터학회지
• /
• 제7권2호
• /
• pp.11-29
• /
• 2022

사전학습 언어모델은 자연어 처리 작업에서 가장 중요하고 많이 활용되는 도구로, 대량의 말뭉치를 대상으로 사전학습이 되어있어 적은 수의 데이터를 이용한 미세조정학습으로도 높은 성능을 기대할 수 있으며, 사전학습된 토크나이저과 딥러닝 모형 등 구현에 필요한 요소들이 함께 배포되기 때문에 자연어 처리 작업에 소요되는 비용과 기간을 크게 단축시켰다. 트랜스포머 변형 모형은 이와 같은 장점을 제공하는 사전학습 언어모델 중에서 최근 가장 많이 사용되고 있는 모형으로, 번역을 비롯하여 문서 요약, 챗봇과 같은 질의 응답, 자연스러운 문장의 생성 및 문서의 분류 등 다양한 자연어 처리 작업에 활용되고 있으며 컴퓨터 비전 분야와 오디오 관련 분야 등 다른 분야에서도 활발하게 활용되고 있다. 본 논문은 연구자들이 보다 쉽게 사전학습 언어모델에 대해 이해하고 자연어 처리 작업에 활용할 수 있도록 하기 위해, 언어모델과 사전학습 언어모델의 정의로부터 시작하여 사전학습 언어모델의 발전과정과 다양한 트랜스포머 변형 모형에 대해 조사하고 정리하였다.

• 한국IT서비스학회지
• /
• 제19권5호
• /
• pp.83-91
• /
• 2020

Language Models such as BERT has been an important factor of deep learning-based natural language processing. Pre-training the transformer-based language models would be computationally expensive since they are consist of deep and broad architecture and layers using an attention mechanism and also require huge amount of data to train. Hence, it became mandatory to do fine-tuning large pre-trained language models which are trained by Google or some companies can afford the resources and cost. There are various techniques for fine tuning the language models and this paper examines three techniques, which are data augmentation, tuning the hyper paramters and partly re-constructing the neural networks. For data augmentation, we use no-answer augmentation and back-translation method. Also, some useful combinations of hyper parameters are observed by conducting a number of experiments. Finally, we have GRU, LSTM networks to boost our model performance with adding those networks to BERT pre-trained model. We do fine-tuning the pre-trained korean-based language model through the methods mentioned above and push the F1 score from baseline up to 89.66. Moreover, some failure attempts give us important lessons and tell us the further direction in a good way.

• International Journal of Computer Science & Network Security
• /
• 제21권6호
• /
• pp.1-6
• /
• 2021

Recently, pre-trained convolutional neural network CNNs have been widely used and applied for medical image classification. These models can utilised in three different ways, for feature extraction, to use the architecture of the pre-trained model and to train some layers while freezing others. In this study, the ResNet18 pre-trained CNNs model is used for feature extraction, followed by the support vector machine for multiple classes to classify medical images from multi-classes, which is used as the main classifier. Our proposed classification method was implemented on Kvasir and PH2 medical image datasets. The overall accuracy was 93.38% and 91.67% for Kvasir and PH2 datasets, respectively. The classification results and performance of our proposed method outperformed some of the related similar methods in this area of study.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• 제16권3호
• /
• pp.771-791
• /
• 2022

Protein-protein interaction (PPI) extraction from original text is important for revealing the molecular mechanism of biological processes. With the rapid growth of biomedical literature, manually extracting PPI has become more time-consuming and laborious. Therefore, the automatic PPI extraction from the raw literature through natural language processing technology has attracted the attention of the majority of researchers. We propose a PPI extraction model based on the large pre-trained language model and adversarial training. It enhances the learning of semantic and syntactic features using BioBERT pre-trained weights, which are built on large-scale domain corpora, and adversarial perturbations are applied to the embedding layer to improve the robustness of the model. Experimental results showed that the proposed model achieved the highest F1 scores (83.93% and 90.31%) on two corpora with large sample sizes, namely, AIMed and BioInfer, respectively, compared with the previous method. It also achieved comparable performance on three corpora with small sample sizes, namely, HPRD50, IEPA, and LLL.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• 제17권9호
• /
• pp.2334-2347
• /
• 2023

Metaverse services generate text data, data of ubiquitous computing, in real-time to analyze user emotions. Analysis of user emotions is an important task in metaverse services. This study aims to classify user sentiments using deep learning and pre-trained language models based on the transformer structure. Previous studies collected data from a single platform, whereas the current study incorporated the review data as "Metaverse" keyword from the YouTube and Google Play Store platforms for general utilization. As a result, the Bidirectional Encoder Representations from Transformers (BERT) and Robustly optimized BERT approach (RoBERTa) models using the soft voting mechanism achieved a highest accuracy of 88.57%. In addition, the area under the curve (AUC) score of the ensemble model comprising RoBERTa, BERT, and A Lite BERT (ALBERT) was 0.9458. The results demonstrate that the ensemble combined with the RoBERTa model exhibits good performance. Therefore, the RoBERTa model can be applied on platforms that provide metaverse services. The findings contribute to the advancement of natural language processing techniques in metaverse services, which are increasingly important in digital platforms and virtual environments. Overall, this study provides empirical evidence that sentiment analysis using deep learning and pre-trained language models is a promising approach to improving user experiences in metaverse services.

• 설비공학논문집
• /
• 제20권3호
• /
• pp.205-212
• /
• 2008

This paper describes the application of an inverse building model to a calibrated forward building model using EnergyPlus program. Typically, inverse models are trained using measured data. However, in this study, an inverse building model was trained using data generated by an EnergyPlus model for an actual office building. The EnergyPlus model was calibrated using field data for the building. A training data set for a month of July was generated from the EnergyPlus model to train the inverse model. Cooling load prediction of the trained inverse model was tested using another data set from the EnergyPlus model for a month of August. Predicted cooling loads showed good agreement with cooling loads from the EnergyPlus model with root-mean square errors of 4.11%. In addition, different control strategies with dynamic cooling setpoint variation were simulated using the inverse model. Peak cooling loads and daily cooling loads were compared for the dynamic simulation.

• 한국컴퓨터정보학회논문지
• /
• 제27권2호
• /
• pp.15-23
• /
• 2022

특허 분야에서 자연어처리(Natural Language Processing) 태스크는 특허문헌의 언어적 특이성으로 문제 해결의 난이도가 높은 과제임에 따라 한국 특허문헌에 최적화된 언어모델의 연구가 시급한 실정이다. 최근 자연어처리 분야에서는 특정 도메인에 특화되게 사전 학습(Pre-trained)한 언어모델을 구축하여 관련 분야의 다양한 태스크에서 성능을 향상시키려는 시도가 지속적으로 이루어지고 있다. 그 중, ELECTRA는 Google이 BERT 이후에 RTD(Replaced Token Detection)라는 새로운 방식을 제안하며 학습 효율성을 높인 사전학습 언어모델이다. 본 연구에서는 대량의 한국 특허문헌 데이터를 사전 학습한 KorPatELECTRA를 제안한다. 또한, 특허 문헌의 특성에 맞게 학습 코퍼스를 정제하고 특허 사용자 사전 및 전용 토크나이저를 적용하여 최적화된 사전 학습을 진행하였다. KorPatELECTRA의 성능 확인을 위해 실제 특허데이터를 활용한 NER(Named Entity Recognition), MRC(Machine Reading Comprehension), 특허문서 분류 태스크를 실험하였고 비교 대상인 범용 모델에 비해 3가지 태스크 모두에서 가장 우수한 성능을 확인하였다.

• Journal of Power Electronics
• /
• 제7권2호
• /
• pp.159-173
• /
• 2007

In this paper, an intelligent sliding-mode position controller (ISMC) for achieving favorable decoupling control and high precision position tracking performance of permanent-magnet synchronous motor (PMSM) servo drives is proposed. The intelligent position controller consists of a sliding-mode position controller (SMC) in the position feed-back loop in addition to an on-line trained fuzzy-neural-network model-following controller (FNNMFC) in the feedforward loop. The intelligent position controller combines the merits of the SMC with robust characteristics and the FNNMFC with on-line learning ability for periodic command tracking of a PMSM servo drive. The theoretical analyses of the sliding-mode position controller are described with a second order switching surface (PID) which is insensitive to parameter uncertainties and external load disturbances. To realize high dynamic performance in disturbance rejection and tracking characteristics, an on-line trained FNNMFC is proposed. The connective weights and membership functions of the FNNMFC are trained on-line according to the model-following error between the outputs of the reference model and the PMSM servo drive system. The FNNMFC generates an adaptive control signal which is added to the SMC output to attain robust model-following characteristics under different operating conditions regardless of parameter uncertainties and load disturbances. A computer simulation is developed to demonstrate the effectiveness of the proposed intelligent sliding mode position controller. The results confirm that the proposed ISMC grants robust performance and precise response to the reference model regardless of load disturbances and PMSM parameter uncertainties.

• Nuclear Engineering and Technology
• /
• 제54권5호
• /
• pp.1825-1834
• /
• 2022

Performing high-fidelity computational fluid dynamics (HF-CFD) to predict the flow and heat transfer state of the coolant in the reactor core is expensive, especially in scenarios that require extensive parameter search, such as uncertainty analysis and design optimization. This work investigated the performance of utilizing a multi-fidelity reduced-order model (MF-ROM) in PWR rod bundles simulation. Firstly, basis vectors and basis vector coefficients of high-fidelity and low-fidelity CFD results are extracted separately by the proper orthogonal decomposition (POD) approach. Secondly, a surrogate model is trained to map the relationship between the extracted coefficients from different fidelity results. In the prediction stage, the coefficients of the low-fidelity data under the new operating conditions are extracted by using the obtained POD basis vectors. Then, the trained surrogate model uses the low-fidelity coefficients to regress the high-fidelity coefficients. The predicted high-fidelity data is reconstructed from the product of extracted basis vectors and the regression coefficients. The effectiveness of the MF-ROM is evaluated on a flow and heat transfer problem in PWR fuel rod bundles. Two data-driven algorithms, the Kriging and artificial neural network (ANN), are trained as surrogate models for the MF-ROM to reconstruct the complex flow and heat transfer field downstream of the mixing vanes. The results show good agreements between the data reconstructed with the trained MF-ROM and the high-fidelity CFD simulation result, while the former only requires to taken the computational burden of low-fidelity simulation. The results also show that the performance of the ANN model is slightly better than the Kriging model when using a high number of POD basis vectors for regression. Moreover, the result presented in this paper demonstrates the suitability of the proposed MF-ROM for high-fidelity fixed value initialization to accelerate complex simulation.