• Title/Summary/Keyword: Natural Language Process

Search Result 246, Processing Time 0.028 seconds

Token-Based Classification and Dataset Construction for Detecting Modified Profanity (변형된 비속어 탐지를 위한 토큰 기반의 분류 및 데이터셋)

  • Sungmin Ko;Youhyun Shin
    • The Transactions of the Korea Information Processing Society
    • /
    • v.13 no.4
    • /
    • pp.181-188
    • /
    • 2024
  • Traditional profanity detection methods have limitations in identifying intentionally altered profanities. This paper introduces a new method based on Named Entity Recognition, a subfield of Natural Language Processing. We developed a profanity detection technique using sequence labeling, for which we constructed a dataset by labeling some profanities in Korean malicious comments and conducted experiments. Additionally, to enhance the model's performance, we augmented the dataset by labeling parts of a Korean hate speech dataset using one of the large language models, ChatGPT, and conducted training. During this process, we confirmed that filtering the dataset created by the large language model by humans alone could improve performance. This suggests that human oversight is still necessary in the dataset augmentation process.

The Theory of Linguistic Semantic Interpretation Rule using Fuzzy Definition (퍼지 논리를 이용한 컴퓨터 언어해석 구현 규칙의 이용법)

  • 진현수
    • Proceedings of the IEEK Conference
    • /
    • 2003.11b
    • /
    • pp.227-230
    • /
    • 2003
  • We can not distinguish semantism of the feature of the current language “big”, “small”, “beautiful”. But we study artificial linguistic interface work and convert natural language to digital binary linguistic theory, we should define the basical conversion process. When we utilize the sum of product fuzzy theory and the visible numerical value, we can establish reasoning rule of input language. Fuzzy theory should be converted to general resulting rule.

  • PDF

An Application of RASA Technology to Design an AI Virtual Assistant: A Case of Learning Finance and Banking Terms in Vietnamese

  • PHAM, Thi My Ni;PHAM, Thi Ngoc Thao;NGUYEN, Ha Phuong Truc;LY, Bao Tuyen;NGUYEN, Truc Linh;LE, Hoanh Su
    • The Journal of Asian Finance, Economics and Business
    • /
    • v.9 no.5
    • /
    • pp.273-283
    • /
    • 2022
  • Banking and finance is a broad term that incorporates a variety of smaller, more specialized subjects such as corporate finance, tax finance, and insurance finance. A virtual assistant that assists users in searching for information about banking and finance terms might be an extremely beneficial tool for users. In this study, we explored the process of searching for information, seeking opportunities, and developing a virtual assistant in the first stages of starting learning and understanding Vietnamese to increase effectiveness and save time, which is also an innovative business practice in Use-case Vietnam. We built the FIBA2020 dataset and proposed a pipeline that used Natural Language Processing (NLP) inclusive of Natural Language Understanding (NLU) algorithms to build chatbot applications. The open-source framework RASA is used to implement the system in our study. We aim to improve our model performance by replacing parts of RASA's default tokenizers with Vietnamese tokenizers and experimenting with various language models. The best accuracy we achieved is 86.48% and 70.04% in the ideal condition and worst condition, respectively. Finally, we put our findings into practice by creating an Android virtual assistant application using the model trained using Whitespace tokenizer and the pre-trained language m-BERT.

Resume Classification System using Natural Language Processing & Machine Learning Techniques

  • Irfan Ali;Nimra;Ghulam Mujtaba;Zahid Hussain Khand;Zafar Ali;Sajid Khan
    • International Journal of Computer Science & Network Security
    • /
    • v.24 no.7
    • /
    • pp.108-117
    • /
    • 2024
  • The selection and recommendation of a suitable job applicant from the pool of thousands of applications are often daunting jobs for an employer. The recommendation and selection process significantly increases the workload of the concerned department of an employer. Thus, Resume Classification System using the Natural Language Processing (NLP) and Machine Learning (ML) techniques could automate this tedious process and ease the job of an employer. Moreover, the automation of this process can significantly expedite and transparent the applicants' selection process with mere human involvement. Nevertheless, various Machine Learning approaches have been proposed to develop Resume Classification Systems. However, this study presents an automated NLP and ML-based system that classifies the Resumes according to job categories with performance guarantees. This study employs various ML algorithms and NLP techniques to measure the accuracy of Resume Classification Systems and proposes a solution with better accuracy and reliability in different settings. To demonstrate the significance of NLP & ML techniques for processing & classification of Resumes, the extracted features were tested on nine machine learning models Support Vector Machine - SVM (Linear, SGD, SVC & NuSVC), Naïve Bayes (Bernoulli, Multinomial & Gaussian), K-Nearest Neighbor (KNN) and Logistic Regression (LR). The Term-Frequency Inverse Document (TF-IDF) feature representation scheme proven suitable for Resume Classification Task. The developed models were evaluated using F-ScoreM, RecallM, PrecissionM, and overall Accuracy. The experimental results indicate that using the One-Vs-Rest-Classification strategy for this multi-class Resume Classification task, the SVM class of Machine Learning algorithms performed better on the study dataset with over 96% overall accuracy. The promising results suggest that NLP & ML techniques employed in this study could be used for the Resume Classification task.

A Process-Centered Knowledge Model for Analysis of Technology Innovation Procedures

  • Chun, Seungsu
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.10 no.3
    • /
    • pp.1442-1453
    • /
    • 2016
  • Now, there are prodigiously expanding worldwide economic networks in the information society, which require their social structural changes through technology innovations. This paper so tries to formally define a process-centered knowledge model to be used to analyze policy-making procedures on technology innovations. The eventual goal of the proposed knowledge model is to apply itself to analyze a topic network based upon composite keywords from a document written in a natural language format during the technology innovation procedures. Knowledge model is created to topic network that compositing driven keyword through text mining from natural language in document. And we show that the way of analyzing knowledge model and automatically generating feature keyword and relation properties into topic networks.

A Study on the Construction of Financial-Specific Language Model Applicable to the Financial Institutions (금융권에 적용 가능한 금융특화언어모델 구축방안에 관한 연구)

  • Jae Kwon Bae
    • Journal of Korea Society of Industrial Information Systems
    • /
    • v.29 no.3
    • /
    • pp.79-87
    • /
    • 2024
  • Recently, the importance of pre-trained language models (PLM) has been emphasized for natural language processing (NLP) such as text classification, sentiment analysis, and question answering. Korean PLM shows high performance in NLP in general-purpose domains, but is weak in domains such as finance, medicine, and law. The main goal of this study is to propose a language model learning process and method to build a financial-specific language model that shows good performance not only in the financial domain but also in general-purpose domains. The five steps of the financial-specific language model are (1) financial data collection and preprocessing, (2) selection of model architecture such as PLM or foundation model, (3) domain data learning and instruction tuning, (4) model verification and evaluation, and (5) model deployment and utilization. Through this, a method for constructing pre-learning data that takes advantage of the characteristics of the financial domain and an efficient LLM training method, adaptive learning and instruction tuning techniques, were presented.

Recognizing Emotional Content of Emails as a byproduct of Natural Language Processing-based Metadata Extraction (이메일에 포함된 감성정보 관련 메타데이터 추출에 관한 연구)

  • Paik, Woo-Jin
    • Journal of the Korean Society for information Management
    • /
    • v.23 no.2
    • /
    • pp.167-183
    • /
    • 2006
  • This paper describes a metadata extraction technique based on natural language processing (NLP) which extracts personalized information from email communications between financial analysts and their clients. Personalized means connecting users with content in a personally meaningful way to create, grow, and retain online relationships. Personalization often results in the creation of user profiles that store individuals' preferences regarding goods or services offered by various e-commerce merchants. We developed an automatic metadata extraction system designed to process textual data such as emails, discussion group postings, or chat group transcriptions. The focus of this paper is the recognition of emotional contents such as mood and urgency, which are embedded in the business communications, as metadata.

Automatic Grading System for Subjective Questions Through Analyzing Question Type (질의문 유형 분석을 통한 서답형 자동 채점 시스템)

  • Kang, Won-Seog
    • The Journal of the Korea Contents Association
    • /
    • v.11 no.2
    • /
    • pp.13-21
    • /
    • 2011
  • It is not easy to develop the system as the subjective-type evaluation has the difficulty in natural language processing. This thesis designs and implements the automatic evaluation system with natural language processing technique. To solve the degradation of general evaluation system, we define the question type and improve the performance of evaluation through the adaptive process for each question type. To evaluate the system, we analyze the correlation between human evaluation and term-based evaluation, and between human evaluation and this system evaluation. We got the better result than term-based evaluation. It needs to expand the question type and improve the adaptive processing technique for each type.

Customized Information Analysis System Using National Defense News Data (국방 기사 데이터를 이용한 맞춤형 정보 분석 시스템)

  • Choi, Jung-Whoan;Lim, Chea-O
    • The Journal of the Korea Contents Association
    • /
    • v.10 no.12
    • /
    • pp.457-465
    • /
    • 2010
  • Customized information analysis system is a software system that can help to extract useful information from non-structured natural language data, process the information to customized form, and provide future forecast and reasoning information. To implement the information analysis system, we need natural language processing technology to analyze natural language, information extraction technology to detect necessary entity and its relationship from text, and data mining technology to discover new and unknown information from extracting data. This paper suggest virtual customized information analysis system processing national defense news data and introduce base technologies for information analysis.

Analyze the Open data for Natural Language Processing of Learning Counseling (학습 상담 내용의 자연어 처리를 위한 오픈 데이터 현황 분석)

  • Kim, Yu-Doo
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2019.05a
    • /
    • pp.500-501
    • /
    • 2019
  • In the $4^{th}$ generation industry, self-directed learning is very important than Injection learning. Therefore many educational institutions has developed method of self-directed learning. In order for self-directed learning to be effective, it is more important for faculty to manage the overall process of learning rather than being directly involved in the student's academic work. Therefore, learning counseling is an important way to effectively carry out self-directed learning. In this paper, we analyze the status of open data for natural language processing that can implement the learning consultation contents so that various applications can be done through natural language processing.

  • PDF