• Title/Summary/Keyword: 사전기반 후처리

Search Result 151, Processing Time 0.027 seconds

Building Sentence Meaning Identification Dataset Based on Social Problem-Solving R&D Reports (사회문제 해결 연구보고서 기반 문장 의미 식별 데이터셋 구축)

  • Hyeonho Shin;Seonki Jeong;Hong-Woo Chun;Lee-Nam Kwon;Jae-Min Lee;Kanghee Park;Sung-Pil Choi
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.12 no.4
    • /
    • pp.159-172
    • /
    • 2023
  • In general, social problem-solving research aims to create important social value by offering meaningful answers to various social pending issues using scientific technologies. Not surprisingly, however, although numerous and extensive research attempts have been made to alleviate the social problems and issues in nation-wide, we still have many important social challenges and works to be done. In order to facilitate the entire process of the social problem-solving research and maximize its efficacy, it is vital to clearly identify and grasp the important and pressing problems to be focused upon. It is understandable for the problem discovery step to be drastically improved if current social issues can be automatically identified from existing R&D resources such as technical reports and articles. This paper introduces a comprehensive dataset which is essential to build a machine learning model for automatically detecting the social problems and solutions in various national research reports. Initially, we collected a total of 700 research reports regarding social problems and issues. Through intensive annotation process, we built totally 24,022 sentences each of which possesses its own category or label closely related to social problem-solving such as problems, purposes, solutions, effects and so on. Furthermore, we implemented four sentence classification models based on various neural language models and conducted a series of performance experiments using our dataset. As a result of the experiment, the model fine-tuned to the KLUE-BERT pre-trained language model showed the best performance with an accuracy of 75.853% and an F1 score of 63.503%.

Building a Korean Sentiment Lexicon Using Collective Intelligence (집단지성을 이용한 한글 감성어 사전 구축)

  • An, Jungkook;Kim, Hee-Woong
    • Journal of Intelligence and Information Systems
    • /
    • v.21 no.2
    • /
    • pp.49-67
    • /
    • 2015
  • Recently, emerging the notion of big data and social media has led us to enter data's big bang. Social networking services are widely used by people around the world, and they have become a part of major communication tools for all ages. Over the last decade, as online social networking sites become increasingly popular, companies tend to focus on advanced social media analysis for their marketing strategies. In addition to social media analysis, companies are mainly concerned about propagating of negative opinions on social networking sites such as Facebook and Twitter, as well as e-commerce sites. The effect of online word of mouth (WOM) such as product rating, product review, and product recommendations is very influential, and negative opinions have significant impact on product sales. This trend has increased researchers' attention to a natural language processing, such as a sentiment analysis. A sentiment analysis, also refers to as an opinion mining, is a process of identifying the polarity of subjective information and has been applied to various research and practical fields. However, there are obstacles lies when Korean language (Hangul) is used in a natural language processing because it is an agglutinative language with rich morphology pose problems. Therefore, there is a lack of Korean natural language processing resources such as a sentiment lexicon, and this has resulted in significant limitations for researchers and practitioners who are considering sentiment analysis. Our study builds a Korean sentiment lexicon with collective intelligence, and provides API (Application Programming Interface) service to open and share a sentiment lexicon data with the public (www.openhangul.com). For the pre-processing, we have created a Korean lexicon database with over 517,178 words and classified them into sentiment and non-sentiment words. In order to classify them, we first identified stop words which often quite likely to play a negative role in sentiment analysis and excluded them from our sentiment scoring. In general, sentiment words are nouns, adjectives, verbs, adverbs as they have sentimental expressions such as positive, neutral, and negative. On the other hands, non-sentiment words are interjection, determiner, numeral, postposition, etc. as they generally have no sentimental expressions. To build a reliable sentiment lexicon, we have adopted a concept of collective intelligence as a model for crowdsourcing. In addition, a concept of folksonomy has been implemented in the process of taxonomy to help collective intelligence. In order to make up for an inherent weakness of folksonomy, we have adopted a majority rule by building a voting system. Participants, as voters were offered three voting options to choose from positivity, negativity, and neutrality, and the voting have been conducted on one of the largest social networking sites for college students in Korea. More than 35,000 votes have been made by college students in Korea, and we keep this voting system open by maintaining the project as a perpetual study. Besides, any change in the sentiment score of words can be an important observation because it enables us to keep track of temporal changes in Korean language as a natural language. Lastly, our study offers a RESTful, JSON based API service through a web platform to make easier support for users such as researchers, companies, and developers. Finally, our study makes important contributions to both research and practice. In terms of research, our Korean sentiment lexicon plays an important role as a resource for Korean natural language processing. In terms of practice, practitioners such as managers and marketers can implement sentiment analysis effectively by using Korean sentiment lexicon we built. Moreover, our study sheds new light on the value of folksonomy by combining collective intelligence, and we also expect to give a new direction and a new start to the development of Korean natural language processing.

Performance Improvement on Similar Texts Searching System for Massive Document Repository (대용량 문서 집합에서 유사문서 탐색 시스템의 성능 개선)

  • Park, Sun-Young;Cho, Hwan-Gue
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2010.04a
    • /
    • pp.413-416
    • /
    • 2010
  • 최근 발생한 수많은 표절 논란으로 인해 많은 유사 문서 탐색 시스템이 개발되어 사용되고 있다. 많은 시스템 중 내용기반 유사문서 탐색 시스템인 DeVAC은 대용량 문서 1:1간의 비교에서 빠른 성능을 보여주지만 수천~수만 개의 문서 집합에 대해서는 적절한 성능을 보여주지 못한다. 이를 해결하기 위해 전역 사전(Global Dictionary)을 이용한 전처리 방법이 고안되어 적용되었다. 이 전처리 방법을 통해 비교해야 할 문서쌍이 줄어들고 전체 시스템의 성능을 향상시킬 수 있다는 것은 밝혀졌으나, 전처리를 위해 발생하는 추가 비용에 대한 계측이 이루어지지 않았을 뿐 아니라 문서 쌍이 얼마나 감소하는지 측정한 실험에서도 언어 처리용 실험적 데이터(말뭉치)에 대한 실험이 대부분을 차지하였기 때문에 실제 데이터에 대해 어떤 성능을 보일지 정확히 예측할 수 없었다. 본 논문에서는 전체 시스템에서 전처리를 위해 필요한 모든 추가 비용을 측정하고, 데이터를 1.5Gb, 6263개의 문서로 이루어진 실존하는 문서 집합으로 구성하여 성능 향상 정도를 측정함으로써 실제 데이터에 대한 전처리 신뢰도를 예측하였다. 실험 결과 전처리 후 찾아낸 유사한 문서 쌍을 전처리를 하지 않을 경우의 80~89.3% 정도로 유지하면서 검사 시간을 기존의 10.8%~15.4% 수준으로 대폭 감소시킬 수 있었다.

A Design on Informal Big Data Topic Extraction System Based on Spark Framework (Spark 프레임워크 기반 비정형 빅데이터 토픽 추출 시스템 설계)

  • Park, Kiejin
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.5 no.11
    • /
    • pp.521-526
    • /
    • 2016
  • As on-line informal text data have massive in its volume and have unstructured characteristics in nature, there are limitations in applying traditional relational data model technologies for data storage and data analysis jobs. Moreover, using dynamically generating massive social data, social user's real-time reaction analysis tasks is hard to accomplish. In the paper, to capture easily the semantics of massive and informal on-line documents with unsupervised learning mechanism, we design and implement automatic topic extraction systems according to the mass of the words that consists a document. The input data set to the proposed system are generated first, using N-gram algorithm to build multiple words to capture the meaning of the sentences precisely, and Hadoop and Spark (In-memory distributed computing framework) are adopted to run topic model. In the experiment phases, TB level input data are processed for data preprocessing and proposed topic extraction steps are applied. We conclude that the proposed system shows good performance in extracting meaningful topics in time as the intermediate results come from main memories directly instead of an HDD reading.

A System for converting natural language queries Into boolean queries for Information Retrieval (정보검색을 위한 자연언어 질의어의 불리언 질의로의 변환)

  • 서광준;최기선;나동열
    • Proceedings of the Acoustical Society of Korea Conference
    • /
    • 1994.06c
    • /
    • pp.258-261
    • /
    • 1994
  • 자연언어 인터페이스는 초보자나 비숙련가의 입장에서는 새로운 시스템의 적응에 있어서 어떤 학습도 필요하지 않다는 장점이 있다. 이 연구에서는 불리언 질의를 처리하는 정보검색 시스템의 자연언어 인터페이스를 구혐하였다. 즉, 한국어 자연언어 질의를 불리언 질의로 변환해주는 시스템이다. 접근 방법은 먼저 자연언어 질의를 구문 해석한 후에, 그 결과인 문자의 의존 구조와 불용어 정보를 사용하여 기본적인 불리언 질의를 만든다음, 시소러스를 이용하여 불리언 질의를 확장한다. 여기에서 사용한 구문 해석 방법은 기존 문법에 기반한 방법이다. 변환 시스템은 SPARC-II 호환기종에서 구현되었으며, 약 5만 단어의 사전을 사용한다. 가공된 120 개의 질의를 대상으로 실험한 결과, 전체 소요시간은 13.5초가 걸렸다. 그리고, 변환된 불리언 연산식중에 110개가 적절하게 변환된 것으로 조사되었다.

  • PDF

COVID-19 Korean Fake News Detection using Named Entity and User Reproliferation Information (개체명 및 사용자 재확산 정보를 이용한 한국어 COVID-19 가짜 뉴스 검출)

  • Park, Chaewon;Kang, Jiwon;Lee, Daeun;Lee, Munyoung;Han, Jinyoung
    • Annual Conference on Human and Language Technology
    • /
    • 2021.10a
    • /
    • pp.85-90
    • /
    • 2021
  • 코로나바이러스감염증-19로 인한 팬데믹 상황이 지속되면서 감염증 정보의 불확실성으로 인해 코로나 관련 루머가 온라인상에서 빠르게 전파되고 있다. 이러한 코로나 관련 가짜 뉴스를 사전에 탐지하기 위해, 본 연구에서는 한국어 코로나 가짜 뉴스 데이터셋을 구축하고, 개체명과 사용자 재확산 정보를 이용한 한국어 가짜 뉴스 탐지 모델을 제안한다. 가짜 뉴스 팩트체킹 언론인 서울대팩트체크센터에서 코로나 관련 루머 및 가짜 뉴스에 대한 검증 기사를 수집한 후, 기사로부터 개체명 추출 모델을 통해 주제 키워드를 추출하고, 이를 이용해 유튜브 상의 사용자 재확산 정보를 수집하여 데이터셋을 구성하였다. BERT 기반의 제안 모델을 다양한 비교군과 비교하였고, 특성 조합에 따른 실험을 통해 각 특성 정보(기사 텍스트, 개체명 데이터, 유튜브 데이터)가 가짜 뉴스 탐지 성능에 미치는 영향을 알아보았다.

  • PDF

PALM for Improving Korean T5: Application to Machine Reading Comprehension & Text Summarization (PALM 기반 한국어 T5 개선: 기계독해 및 텍스트 요약으로의 응용)

  • Park, Eunhwan;Na, Seung-Hoon;Lim, Joon-Ho;Kim, Tae-Hyeong;Choi, Yun-Su;Chang, Du-Seong
    • Annual Conference on Human and Language Technology
    • /
    • 2021.10a
    • /
    • pp.501-504
    • /
    • 2021
  • 최근 언어 모델은 분류, 기계 독해, 생성 등의 태스크에서 성공적인 결과를 보여주고 있다. 본 논문에서는 최근 많은 관심을 받고 있는 인코더-디코더 구조의 언어 모델인 BART, T5 그리고 PALM을 위키피디아 한국어 데이터 집합으로 사전 학습한 후 기계 독해와 문서 생성 요약 태스크에 대하여 미세 조정을 하고 성능 비교를 한다.

  • PDF

The partial matching method for effective recognizing HLA entities (효과적인 HLA개체인식을 위한 부분매칭기법)

  • Chae, Jeong-Min;Jung, Young-Hee;Lee, Tae-Min;Chae, Ji-Eun;Oh, Heung-Bum;Jung, Soon-Young
    • The Journal of Korean Association of Computer Education
    • /
    • v.14 no.2
    • /
    • pp.83-94
    • /
    • 2011
  • In the biomedical domain, the longest matching method is frequently used for recognizing named entity written in the literature. This method uses a dictionary as a resource for named entity recognition. If there exist appropriated dictionary about target domain, the longest matching method has the advantage of being able to recognize the entities of target domain quickly and exactly. However, the longest matching method is difficult to recognize the enumerated named entities, because these entities are frequently expressed as being omitted some words. In order to resolve this problem, we propose the partial matching method using a dictionary. The proposed method makes several candidate entities on the assumption that the ellipses may be included. After that, the method selects the most valid one among candidate entities through the optimization algorithm. We tested the longest and partial matching method about HLA entities: HLA gene, antigen, and allele entities, which are frequently enumerated among biomedical entities. As preparing for named entity recognition, we built two new resource, extended dictionary and tag-based dictionary about HLA entities. And later, we performed the longest and partial matching method using each dictionary. According to our experiment result, the longest matching method was effective in recognizing HLA antigen entities, in which the ellipses are rare, and the partial matching method was effective in recognizing HLA gene and allele entities, in which the ellipses are frequent. Especially, the partial matching method had a high F-score 95.59% about HLA alleles.

  • PDF

Modeling and Digital Predistortion Design of RF Power Amplifier Using Extended Memory Polynomial (확장된 메모리 다항식 모델을 이용한 전력 증폭기 모델링 및 디지털 사전 왜곡기 설계)

  • Lee, Young-Sup;Ku, Hyun-Chul;Kim, Jeong-Hwi;Ryoo, Kyoo-Tae
    • The Journal of Korean Institute of Electromagnetic Engineering and Science
    • /
    • v.19 no.11
    • /
    • pp.1254-1264
    • /
    • 2008
  • This paper suggests an extended memory polynomial model that improves accuracy in modeling memory effects of RF power amplifiers(PAs), and verifies effectiveness of the suggested method. The extended memory polynomial model includes cross-terms that are products of input terms that have different delay values to improve the limited accuracy of basic memory polynomial model that includes the diagonal terms of Volterra kernels. The complexity of the memoryless model, memory polynomial model, and the suggested model are compared. The extended memory polynomial model is represented with a matrix equation, and the Volterra kernels are extracted using least square method. In addition, the structure of digital predistorter and digital signal processing(DSP) algorithm based on the suggested model and indirect learning method are proposed to implement a digital predistortion linearization. To verify the suggested model, the predicted output of the model is compared with the measured output for a 10W GaN HEMT RF PA and 30 W LDMOS RF PA using 2.3 GHz WiBro input signal, and adjacent-channel power ratio(ACPR) performance with the proposed digital predistortion is measured. The proposed model increases model accuracy for the PAs, and improves the linearization performance by reducing ACPR.

Lane Detection in Complex Environment Using Grid-Based Morphology and Directional Edge-link Pairs (복잡한 환경에서 Grid기반 모폴리지와 방향성 에지 연결을 이용한 차선 검출 기법)

  • Lin, Qing;Han, Young-Joon;Hahn, Hern-Soo
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.20 no.6
    • /
    • pp.786-792
    • /
    • 2010
  • This paper presents a real-time lane detection method which can accurately find the lane-mark boundaries in complex road environment. Unlike many existing methods that pay much attention on the post-processing stage to fit lane-mark position among a great deal of outliers, the proposed method aims at removing those outliers as much as possible at feature extraction stage, so that the searching space at post-processing stage can be greatly reduced. To achieve this goal, a grid-based morphology operation is firstly used to generate the regions of interest (ROI) dynamically, in which a directional edge-linking algorithm with directional edge-gap closing is proposed to link edge-pixels into edge-links which lie in the valid directions, these directional edge-links are then grouped into pairs by checking the valid lane-mark width at certain height of the image. Finally, lane-mark colors are checked inside edge-link pairs in the YUV color space, and lane-mark types are estimated employing a Bayesian probability model. Experimental results show that the proposed method is effective in identifying lane-mark edges among heavy clutter edges in complex road environment, and the whole algorithm can achieve an accuracy rate around 92% at an average speed of 10ms/frame at the image size of $320{\times}240$.