• Title/Summary/Keyword: n-gram

Search Result 571, Processing Time 0.05 seconds

Constructive Method for Terminology N-Gram using Wikipedia Document (위키피디아 문서를 이용한 전문용어 N-Gram 구축)

  • Choi, Jun-Ho;Go, Byung-Gyu;Lee, Jun;Kim, Pan-Koo
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2011.04a
    • /
    • pp.297-299
    • /
    • 2011
  • 자연어 처리 분야 중 현재 가장 활용도가 높은 분야는 질의어 추천기능, 단어 자동 완성 기능 등으로 정보검색에서 사용자가 입력한 문자들을 바탕으로 질의어를 완성해주는 것이다. 이러한 기능을 위해서는 문서 내용을 고려한 N-Gram 데이터 구축이 필수적이다. 본 논문에서는 문서 편집기나 검색엔진의 질의어 추천 등에 많이 활용되는 N-Gram 데이터의 전문용어별 구축을 위해 위키피디아 문서를 이용하는 방안을 제시하였다.

Passage Re-ranking Model using N-gram attention between Question and Passage (질문-단락 간 N-gram 주의 집중을 이용한 단락 재순위화 모델)

  • Jang, Youngjin;Kim, Harksoo
    • Annual Conference on Human and Language Technology
    • /
    • 2020.10a
    • /
    • pp.554-558
    • /
    • 2020
  • 최근 사전학습 모델의 발달로 기계독해 시스템 성능이 크게 향상되었다. 하지만 기계독해 시스템은 주어진 단락에서 질문에 대한 정답을 찾기 때문에 단락을 직접 검색해야하는 실제 환경에서의 성능 하락은 불가피하다. 즉, 기계독해 시스템이 오픈 도메인 환경에서 높은 성능을 보이기 위해서는 높은 성능의 검색 모델이 필수적이다. 따라서 본 논문에서는 검색 모델의 성능을 보완해 줄 수 있는 오픈 도메인 기계독해를 위한 단락 재순위화 모델을 제안한다. 제안 모델은 합성곱 신경망을 이용하여 질문과 단락을 구절 단위로 표현했으며, N-gram 구절 사이의 상호 주의 집중을 통해 질문과 단락 사이의 관계를 효과적으로 표현했다. KorQuAD를 기반으로한 실험에서 제안모델은 MRR@10 기준 93.0%, Top@1 Precision 기준 89.4%의 높은 성능을 보였다.

  • PDF

N- gram Adaptation Using Information Retrieval and Dynamic Interpolation Coefficient (정보검색 기법과 동적 보간 계수를 이용한 N-gram 언어모델의 적응)

  • Choi Joon Ki;Oh Yung-Hwan
    • MALSORI
    • /
    • no.56
    • /
    • pp.207-223
    • /
    • 2005
  • The goal of language model adaptation is to improve the background language model with a relatively small adaptation corpus. This study presents a language model adaptation technique where additional text data for the adaptation do not exist. We propose the information retrieval (IR) technique with N-gram language modeling to collect the adaptation corpus from baseline text data. We also propose to use a dynamic language model interpolation coefficient to combine the background language model and the adapted language model. The interpolation coefficient is estimated from the word hypotheses obtained by segmenting the input speech data reserved for held-out validation data. This allows the final adapted model to improve the performance of the background model consistently The proposed approach reduces the word error rate by $13.6\%$ relative to baseline 4-gram for two-hour broadcast news speech recognition.

  • PDF

A Modified Binary n-gram Algorithm for the postprocessing of the Automatic Document Reading (자동문서판독 후처리를 위한 수정된 n-gram 알고리즘)

  • Kim, Il-Hwoe;Ryoo, Keun-Ho;Lee, Cheol-Hee
    • Proceedings of the KIEE Conference
    • /
    • 1987.07b
    • /
    • pp.1352-1355
    • /
    • 1987
  • This Paper proposed the modified binary n-gram algorithm for the contextual post processing system in English sentence. Backward gram was used to correct the first position error in a word. It is not requires additional storage but more times of comparison it allows interactive correction routine. Experiments were implemented using PASCAL language on a micro computer, IBM PC/XT. This algorithm improves the correction rate around $4{\sim}5%$ on a limited experimental environments.

  • PDF

Image Retrieval Using Meanvalue Color N$\times$M-grams and GLCM (평균값 Color N$\times$M-grams와 GLCM을 이용한 영산 검색)

  • 안세정;정성환
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2000.10b
    • /
    • pp.464-466
    • /
    • 2000
  • 오늘날 영상정보의 보편화로 효율적인 영상 검색 기술이 요구되고 있다. 최근 발표된 Color N$\times$M-grams 기반의 영상 검색 방법은 그 중의 하나이다. 그러나 이 방법은 영상의 특징을 추출한 벡터 Bin의 수가 많아서 검색을 위한 공간과 시간을 많이 필요로 하는 문제점을 가지고 있다. 이러한 문제를 보완하기 위해 본 연구에서는 영상의 국부성을 이용하여 Color N$\times$M-grams의 단점인 공간과 시간의 비효율성을 개선하고, GLCM의 결합으로 검색 효율을 향상시키는 연구를 수행하였다. WWW의 Color Draw Photo Album에 분류되어 있는 영상들과 미국의 코넬대학의 연구에 사용된 330개의 Benchmark 영상을 가지고 실험한 결과, 기존의 Color N$\times$M-grams에 비해 약 10배의 공간효율개선과 약 2배의 시간효율개선을 얻을 수 있었고, 검색율과 정확성공율에 있어서 각각 25%, 63% 향상되었다.

  • PDF

Synthesis and antimickrobial activity of benzofuran-carboxamide derivatives

  • Hishmat, O.H.;Nasef, A.M.;El-Naem, Sh.I.A.;Shalaby, A.M.
    • Archives of Pharmacal Research
    • /
    • v.12 no.4
    • /
    • pp.259-262
    • /
    • 1989
  • The reaction of the sodium salts of 4-methoxy and 4, 7-dimethoxy 6-hydroxy benzofuran-5-carboxylic acid with ethyl chloroformate yields the corresponding dicarbethoxy derivatives. The N-substituted amides were obtained by treating the latter compounds with amines. The corresponding hydrazides were synthesized by the reaction of hydrazine hydrate on the dicarbethoxy derivatives which spontaneously cyclized to 5-substituted-2, 3- dihydro-1, 3, 4, -oxadiazol-2-one. Also the reaction with phenyl hydrazine has been studied. The dicarbethoxy derivatives and N-substituted amides were tested against Gram positive and Gram negative bacteria in vitro. Most of the compounds posses moderate or slight activity against Gram positive bacteria.

  • PDF

Comparison Between Optimal Features of Korean and Chinese for Text Classification (한중 자동 문서분류를 위한 최적 자질어 비교)

  • Ren, Mei-Ying;Kang, Sinjae
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.25 no.4
    • /
    • pp.386-391
    • /
    • 2015
  • This paper proposed the optimal attributes for text classification based on Korean and Chinese linguistic features. The experiments committed to discover which is the best feature among n-grams which is known as language independent, morphemes that have language dependency and some other feature sets consisted with n-grams and morphemes showed best results. This paper used SVM classifier and Internet news for text classification. As a result, bi-gram was the best feature in Korean text categorization with the highest F1-Measure of 87.07%, and for Chinese document classification, 'uni-gram+noun+verb+adjective+idiom', which is the combined feature set, showed the best performance with the highest F1-Measure of 82.79%.

A Study on Negation Handling and Term Weighting Schemes and Their Effects on Mood-based Text Classification (감정 기반 블로그 문서 분류를 위한 부정어 처리 및 단어 가중치 적용 기법의 효과에 대한 연구)

  • Jung, Yu-Chul;Choi, Yoon-Jung;Myaeng, Sung-Hyon
    • Korean Journal of Cognitive Science
    • /
    • v.19 no.4
    • /
    • pp.477-497
    • /
    • 2008
  • Mood classification of blog text is an interesting problem, with a potential for a variety of services involving the Web. This paper introduces an approach to mood classification enhancements through the normalized negation n-grams which contain mood clues and corpus-specific term weighting(CSTW). We've done experiments on blog texts with two different classification methods: Enhanced Mood Flow Analysis(EMFA) and Support Vector Machine based Mood Classification(SVMMC). It proves that the normalized negation n-gram method is quite effective in dealing with negations and gave gradual improvements in mood classification with EMF A. From the selection of CSTW, we noticed that the appropriate weighting scheme is important for supporting adequate levels of mood classification performance because it outperforms the result of TF*IDF and TF.

  • PDF

N-gram based Language Model for the QWERTY Keyboard Input Errors in a Touch Screen Environment (터치스크린 환경에서 쿼티 자판 오타 교정을 위한 n-gram 언어 모델)

  • Ong, Yoon Gee;Kang, Seung Shik
    • Smart Media Journal
    • /
    • v.7 no.2
    • /
    • pp.54-59
    • /
    • 2018
  • With the increasing use of touch-enabled mobile devices such as smartphones and tablet PCs, the works are done on desktop computers and smartphones, and tablet PCs perform laptops. However, due to the nature of smart devices that require portability, QWERTY keyboard is densely arranged in a small screen. This is the cause of different typographical errors when using the mechanical QWERTY keyboard. Unlike the mechanical QWERTY keyboard, which has enough space for each button, QWERTY keyboard on the touch screen often has a small area assigned to each button, so that it is often the case that the surrounding buttons are input rather than the button the user intends to press. In this paper, we propose a method to automatically correct the input errors of the QWERTY keyboard in the touch screen environment by using the n-gram language model using the word unigram and the bigram probability.

Text Mining Analysis Technique on ECDIS Accident Report (텍스트 마이닝 기법을 활용한 ECDIS 사고보고서 분석)

  • Lee, Jeong-Seok;Lee, Bo-Kyeong;Cho, Ik-Soon
    • Journal of the Korean Society of Marine Environment & Safety
    • /
    • v.25 no.4
    • /
    • pp.405-412
    • /
    • 2019
  • SOLAS requires that ECDIS be installed on ships of more than 500 gross tonnage engaged in international navigation until the first inspection arriving after July 1, 2018. Several accidents related to the use of ECDIS have occurred with its installation as a new major navigation instrument. The 12 incident reports issued by MAIB, BSU, BEAmer, DMAIB, and DSB were analyzed, and the cause of accident was determined to be related to the operation of the navigator and the ECDIS system. The text was analyzed using the R-program to quantitatively analyze words related to the cause of the accident. We used text mining techniques such as Wordcloud, Wordnetwork and Wordweight to represent the importance of words according to their frequency of derivation. Wordcloud uses the N-gram model as a way of expressing the frequency of used words in cloud form. As a result of the uni-gram analysis of the N-gram model, ECDIS words were obtained the most, and the bi-gram analysis results showed that the word "Safety Contour" was used most frequently. Based on the bi-gram analysis, the causative words are classified into the officer and the ECDIS system, and the related words are represented by Wordnetwork. Finally, the related words with the of icer and the ECDIS system were composed of word corpus, and Wordweight was applied to analyze the change in corpus frequency by year. As a result of analyzing the tendency of corpus variation with the trend line graph, more recently, the corpus of the officer has decreased, and conversely, the corpus of the ECDIS system is gradually increasing.