• Title/Summary/Keyword: Text Similarity

Search Result 277, Processing Time 0.034 seconds

Mining Parallel Text from the Web based on Sentence Alignment

  • Li, Bo;Liu, Juan;Zhu, Huili
    • Proceedings of the Korean Society for Language and Information Conference
    • /
    • 2007.11a
    • /
    • pp.285-292
    • /
    • 2007
  • The parallel corpus is an important resource in the research field of data-driven natural language processing, but there are only a few parallel corpora publicly available nowadays, mostly due to the high labor force needed to construct this kind of resource. A novel strategy is brought out to automatically fetch parallel text from the web in this paper, which may help to solve the problem of the lack of parallel corpora with high quality. The system we develop first downloads the web pages from certain hosts. Then candidate parallel page pairs are prepared from the page set based on the outer features of the web pages. The candidate page pairs are evaluated in the last step in which the sentences in the candidate web page pairs are extracted and aligned first, and then the similarity of the two web pages is evaluate based on the similarities of the aligned sentences. The experiments towards a multilingual web site show the satisfactory performance of the system.

  • PDF

Language Model Adaptation for Conversational Speech Recognition (대화체 연속음성 인식을 위한 언어모델 적응)

  • Park Young-Hee;Chung Minhwa
    • Proceedings of the KSPS conference
    • /
    • 2003.05a
    • /
    • pp.83-86
    • /
    • 2003
  • This paper presents our style-based language model adaptation for Korean conversational speech recognition. Korean conversational speech is observed various characteristics of content and style such as filled pauses, word omission, and contraction as compared with the written text corpora. For style-based language model adaptation, we report two approaches. Our approaches focus on improving the estimation of domain-dependent n-gram models by relevance weighting out-of-domain text data, where style is represented by n-gram based tf*idf similarity. In addition to relevance weighting, we use disfluencies as predictor to the neighboring words. The best result reduces 6.5% word error rate absolutely and shows that n-gram based relevance weighting reflects style difference greatly and disfluencies are good predictor.

  • PDF

A Novel Text to Image Conversion Method Using Word2Vec and Generative Adversarial Networks

  • LIU, XINRUI;Joe, Inwhee
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2019.05a
    • /
    • pp.401-403
    • /
    • 2019
  • In this paper, we propose a generative adversarial networks (GAN) based text-to-image generating method. In many natural language processing tasks, which word expressions are determined by their term frequency -inverse document frequency scores. Word2Vec is a type of neural network model that, in the case of an unlabeled corpus, produces a vector that expresses semantics for words in the corpus and an image is generated by GAN training according to the obtained vector. Thanks to the understanding of the word we can generate higher and more realistic images. Our GAN structure is based on deep convolution neural networks and pixel recurrent neural networks. Comparing the generated image with the real image, we get about 88% similarity on the Oxford-102 flowers dataset.

Text Similarity Decision System by Term Selection Method (용어 선별 기법에 의한 유사 문서 판별 시스템)

  • 장성호;강승식
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2003.04c
    • /
    • pp.534-536
    • /
    • 2003
  • 대부분의 정보 검색 시스템은 문서 내어서 추출된 모든 용어를 이용해서 문서간 유사도 계산이나 문서 분류, 문서 클러스터링 등에 활용한다. 그러나 실질적으로 문서 내외 모든 용어를 추출해야만 이러한 정보 검색 시스템을 활용할 수 있는 것은 아니며, 오히려 용어 빈도수 같은 가중치가 낮은 용어를 용어 추출에서 제외시킴으로써 모든 용어 추출로 인해서 발생하는 시간과 공간을 많이 소비하는 문제를 해결할 수 있다. 또한 정확하고 자동적인 문서 분류를 위한 문서 클러스터링보다 유사 문서 검색의 활용은 검색효율의 증가를 가져 올 수 있다. 본 논문에서는 유사 문서 판별 시스템을 이용해 용어 추출의 효율성을 실험하였으며, 모든 용어를 추출한 경우보다 중요 용어만 추출한 경우에 더 좋은 성능을 보였다.

  • PDF

Improved PageRank Algorithm Using Similarity Information of Documents (문서간의 유사도를 이용한 개선된 PageRank 알고리즘)

  • 이경희;김민구;박승규
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2003.10a
    • /
    • pp.169-171
    • /
    • 2003
  • 웹에서의 검색 방법에는 크게 Text-Based 기법과 Link-Based 기법이 있다. 본 논문은 그 중에서 Link-Based 기법의 하나인 PageRank 알고리즘에 대해 연구 하고자 한다. 이 PageRank 알고리즘은 각 페이지의 중요성을 수치로 계산하는 방법이다. 하지만 이 알고리즘에서는 페이지에서 페이지로 링크를 따라갈 확률의 값을 일정하게 주어서 모든 페이지의 값을 획일적으로 계산하였기 때문에 각 페이지의 검색 효율성에 문제가 있다고 판단하여, 이를 해결하고자 본 논문은 페이지사이의 유사도를 측정하여 유사도에 따라 링크를 따라가는 확률 값인 Damping factor값을 다르게 부여하여 검색의 효율성을 높였다. 이를 위하여 두 가지 방법의 실험을 통하여 구현, 증명하였다.

  • PDF

A Comparative Analysis about Various Editions of Donguibogam (판본별 교감을 통한 『동의보감』의 정본화)

  • Lee, Jeong-Hyeon;Oh, Junho
    • The Journal of Korean Medical History
    • /
    • v.31 no.1
    • /
    • pp.57-70
    • /
    • 2018
  • Much research has already been done on Donguibogam. However, comparison of specific characters was not done because researchers found it difficult to compare different editions of the text in one place. Recently, important editions have been published on the Internet, making comparison possible. In this paper, researchers compare eight editions Donguibogam, including the original edition published in 1613 and seven other editions corrected by the Naeuiwon (Joseon Dynasty National Medical Center). The comparison results were summarized and tabulated. The results of the comparison are analyzed and presented in this article as a chart. The result of comparing the characters and the analyzed graph were in agreement. The authors propose that all written and electronic publications of Donguibogam should refer to other editions implied, quoted or referenced within the text and including with proper citations, and reference the original and first edition. Inadequate referencing will pollute future knowledge of this foundational text of Traditional Korean Medicine and may result in perpetration of mis-information. Based on accumulated knowledge and study of historical Korean Medicine texts, the Namsan edition made a mistake in the editing process. The year of publication of Gabsul-yoengyoeng-gegan Edition needs to be studied again and corrections made where appropriate.

A Study on the Database Structure for Utilizing Classical Literature Knowledge (고문헌 지식활용을 위한 DB구조에 관한 고찰)

  • Woo, Dong-Hyun;Kim, Ki-Wook;Lee, Byung-Wook
    • The Journal of Korean Medical History
    • /
    • v.33 no.2
    • /
    • pp.89-104
    • /
    • 2020
  • The purpose of this research is to build a database structure that can be useful for evidence-based medical practices by constructing the knowledge related to oriental medicine in the classical literature knowledge in a form that can utilize new forms of information technology. As a method, "database" is used as a keyword to search published studies in the field of oriental medicine, research is conducted on classic literature knowledge, and studies describing the contents of the data structure are found and analyzed. In conclusion, the original text DB for the preservation of the original texts and the presentation of the supporting texts should include 'Contents Text', 'Tree Structure', 'Herbal Structure', 'Medicine Manufacture', and 'Disease Structure' tables. In order to search, calculate, and automatically extract expressions written in the original text of the old literature, the tool DB should include 'Unit List', 'Capacity Notation List', 'CUI', 'LUI', and 'SUI' tables. In addition, In order to manage integrated knowledge such as herbal, medicine, acupuncture, disease, and literature, and to implement a search function such as comparison of similarity of control composition, the knowledge DB must contain 'dose-controlled medicine name', 'dose-controlled medicine composition', 'relational knowledge', 'knowledge structure', and 'computational knowledge' tables.

A Generation-based Text Steganography by Maintaining Consistency of Probability Distribution

  • Yang, Boya;Peng, Wanli;Xue, Yiming;Zhong, Ping
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.15 no.11
    • /
    • pp.4184-4202
    • /
    • 2021
  • Text steganography combined with natural language generation has become increasingly popular. The existing methods usually embed secret information in the generated word by controlling the sampling in the process of text generation. A candidate pool will be constructed by greedy strategy, and only the words with high probability will be encoded, which damages the statistical law of the texts and seriously affects the security of steganography. In order to reduce the influence of the candidate pool on the statistical imperceptibility of steganography, we propose a steganography method based on a new sampling strategy. Instead of just consisting of words with high probability, we select words with relatively small difference from the actual sample of the language model to build a candidate pool, thus keeping consistency with the probability distribution of the language model. What's more, we encode the candidate words according to their probability similarity with the target word, which can further maintain the probability distribution. Experimental results show that the proposed method can outperform the state-of-the-art steganographic methods in terms of security performance.

Linguistic and Stylistic Markers of Influence in the Essayistic Text: A Linguophilosophic Aspect

  • Kolkutina, Viktoriia;Orekhova, Larysa;Gremaliuk, Tetiana;Borysenko, Natalia;Fedorova, Inna;Cheban, Oksana
    • International Journal of Computer Science & Network Security
    • /
    • v.22 no.5
    • /
    • pp.163-167
    • /
    • 2022
  • The article explores linguo-stylistic influence markers in essayistic texts. The novelty of this investigation is provided by its perspective. Essayism is looked at as a style of thinking and writing and studied as a holistic philosophical and cultural phenomenon, as a revalent form of comprehension of reality that features non-lasting author's judgements and enhancement of the author's voice in the text. Based on the texts by V. Rosanov, G.K. Chesterton, and D. Dontsov, the remarkable English, Russian, and Ukrainian essay-writers of the first party of the 20th century, the article tracks the typical ontological-and-existentialist correlation at the content, stylistic, and semantic levels. It is observed in terms of the ideas presented in the texts of these publicists and the lexicostylistic markers of the influence on the reader that enable these ideas to implement. The explored poetic syntax, key lexemes, dialogueness, intonational melodics, specific language, free associations, aphoristic nature, verbalization of emotions and feeling in the psycholinguistic form of their expression, stress, heroic elevation, metaphors and evaluative linguistic units in the ontological-and-existentialist aspects contribute to extremely delicate and demanding nature of the essayistic style. They create a "lacework" of unpredictable properties, intellectual illumination, unexpected similarity, metaphorical freshness, sudden discoveries, unmotivated unities.

Media coverage of the conflicts over the 4th Industrial Revolution in the Republic of Korea from 2016 to 2020: a text-mining approach

  • Yang, Jiseong;Kim, Byungjun;Lee, Wonjae
    • Asian Journal of Innovation and Policy
    • /
    • v.11 no.2
    • /
    • pp.202-221
    • /
    • 2022
  • The media has depicted an abrupt socio-technological change in the Republic of Korea with the 4th Industrial Revolution. Because technologies cannot realize their potential without social acceptance, studying conflicts incurred by such a change is imperative. However, little literature has focused on conflicts caused by technologies. Therefore, the current study investigated media coverage regarding conflicts related to the 4th Industrial Revolution from 2016 to 2020 in the Republic of Korea, applying text-mining techniques. We found that the overall amount and coverage pattern conforms to the issue attention cycle. Also, the three major topics ("SMEs & Startups," "Mobility Conflict," and "Human & Technology") indicate quarrels between conflicting social entities. Moreover, the temporal change in media coverage implies the political use of the term rather than technological. However, we also found the media's deliberative discussion on the socio-technological impact. This study is significant because we expanded the discussion on media coverage of technologies to the realm of social conflicts. Furthermore, we explored the news articles of the recent five years with a text-mining approach that enhanced the objectivity of the research.