• Title/Summary/Keyword: Text-based Similarity Measure

Search Result 27, Processing Time 0.017 seconds

Phonetic Similarity Meausre for the Korean Transliterations of Foreign Words (외국어 음차 표기의 음성적 유사도 비교 알고리즘)

  • Gang, Byeong-Ju;Lee, Jae-Seong;Choe, Gi-Seon
    • Journal of KIISE:Software and Applications
    • /
    • v.26 no.10
    • /
    • pp.1237-1246
    • /
    • 1999
  • 최근 모든 분야에서 외국과의 교류가 증대됨에 따라서 한국어 문서에는 점점 더 많은 외국어 음차 표기가 사용되는 경향이 있다. 하지만 같은 외국어에 대한 음차 표기에 개인차가 심하여 이들 음차 표기를 포함한 문서들에 대한 검색을 어렵게 만드는 원인이 되고 있다. 한 가지 해결 방법은 색인 시에 같은 외국어에서 온 음차 표기들을 등가부류로 묶어서 색인해 놓았다가 질의 시에 확장하는 방법이다. 본 논문에서는 외국어 음차 표기들의 등가부류를 만드는데 필요한 음차 표기의 음성적 유사도 비교 알고리즘인 Kodex를 제안한다. Kodex 방법은 기존의 스트링 비교 방법인 비음성적 방법에 비해 음차 표기들을 등가부류로 클러스터링하는데 있어 더 나은 성능을 보이면서도, 계산이 간단하여 훨씬 효율적으로 구현될 수 있는 장점이 있다.Abstract With the advent of digital communication technologies, as Koreans communicate with foreigners more frequently, more foreign word transliterations are being used in Korean documents more than ever before. The transliterations of foreign words are very various among individuals. This makes text retrieval tasks about these documents very difficult. In this paper we propose a new method, called Kodex, of measuring the phonetic similarity among foreign word transliterations. Kodex can be used to generate the equivalence classes of the transliterations while indexing and conflate the equivalent transliterations at the querying stage. We show that Kodex gives higher precision at the similar recall level and is more efficient in computation than non-phonetic methods based on string similarity measure.

The proposition of cosine net confidence in association rule mining (연관 규칙 마이닝에서의 코사인 순수 신뢰도의 제안)

  • Park, Hee Chang
    • Journal of the Korean Data and Information Science Society
    • /
    • v.25 no.1
    • /
    • pp.97-106
    • /
    • 2014
  • The development of big data technology was to more accurately predict diversified contemporary society and to more efficiently operate it, and to enable impossible technique in the past. This technology can be utilized in various fields such as the social science, economics, politics, cultural sector, and science technology at the national level. It is a prerequisite to find valuable information by data mining techniques in order to analyze big data. Data mining techniques associated with big data involve text mining, opinion mining, cluster analysis, association rule mining, and so on. The most widely used data mining technique is to explore association rules. This technique has been used to find the relationship between each set of items based on the association thresholds such as support, confidence, lift, similarity measures, etc.This paper proposed cosine net confidence as association thresholds, and checked the conditions of interestingness measure proposed by Piatetsky-Shapiro, and examined various characteristics. The comparative studies with basic confidence and cosine similarity, and cosine net confidence were shown by numerical example. The results showed that cosine net confidence are better than basic confidence and cosine similarity because of the relevant direction.

An Incremental Web Document Clustering Based on the Transitive Closure Tree (이행적 폐쇄트리를 기반으로 한 점증적 웹 문서 클러스터링)

  • Youn Sung-Dae;Ko Suc-Bum
    • Journal of Korea Multimedia Society
    • /
    • v.9 no.1
    • /
    • pp.1-10
    • /
    • 2006
  • In document clustering methods, the k-means algorithm and the Hierarchical Alglomerative Clustering(HAC) are often used. The k-means algorithm has the advantage of a processing time and HAC has also the advantage of a precision of classification. But both methods have mutual drawbacks, a slow processing time and a low quality of classification for the k-means algorithm and the HAC, respectively. Also both methods have the serious problem which is to compute a document similarity whenever new document is inserted into a cluster. A main property of web resource is to accumulate an information by adding new documents frequently. Therefore, we propose a new method of transitive closure tree based on the HAC method which can improve a processing time for a document clustering, and also propose a superior incremental clustering method for an insertion of a new document and a deletion of a document contained in a cluster. The proposed method is compared with those existing algorithms on the basis of a pre챠sion, a recall, a F-Measure, and a processing time and we present the experimental results.

  • PDF

Rearranged DCT Feature Analysis Based on Corner Patches for CBIR (contents based image retrieval) (CBIR을 위한 코너패치 기반 재배열 DCT특징 분석)

  • Lee, Jimin;Park, Jongan;An, Youngeun;Oh, Sangeon
    • The Transactions of The Korean Institute of Electrical Engineers
    • /
    • v.65 no.12
    • /
    • pp.2270-2277
    • /
    • 2016
  • In modern society, creation and distribution of multimedia contents is being actively conducted. These multimedia information have come out the enormous amount daily, the amount of data is also large enough it can't be compared with past text information. Since it has been increased for a need of the method to efficiently store multimedia information and to easily search the information, various methods associated therewith have been actively studied. In particular, image search methods for finding what you want from the video database or multiple sequential images, have attracted attention as a new field of image processing. Image retrieval method to be implemented in this paper, utilizes the attribute of corner patches based on the corner points of the object, for providing a new method of efficient and robust image search. After detecting the edge of the object within the image, the straight lines using a Hough transformation is extracted. A corner patches is formed by defining the extracted intersection of the straight line as a corner point. After configuring the feature vectors with patches rearranged, the similarity between images in the database is measured. Finally, for an accurate comparison between the proposed algorithm and existing algorithms, the recall precision rate, which has been widely used in content-based image retrieval was used to measure the performance evaluation. For the image used in the experiment, it was confirmed that the image is detected more accurately in the proposed method than the conventional image retrieval methods.

A Comparative Analysis of Content-based Music Retrieval Systems (내용기반 음악검색 시스템의 비교 분석)

  • Ro, Jung-Soon
    • Journal of the Korean Society for information Management
    • /
    • v.30 no.3
    • /
    • pp.23-48
    • /
    • 2013
  • This study compared and analyzed 15 CBMR (Content-based Music Retrieval) systems accessible on the web in terms of DB size and type, query type, access point, input and output type, and search functions, with reviewing features of music information and techniques used for transforming or transcribing of music sources, extracting and segmenting melodies, extracting and indexing features of music, and matching algorithms for CBMR systems. Application of text information retrieval techniques such as inverted indexing, N-gram indexing, Boolean search, truncation, keyword and phrase search, normalization, filtering, browsing, exact matching, similarity measure using edit distance, sorting, etc. to enhancing the CBMR; effort for increasing DB size and usability; and problems in extracting melodies, deleting stop notes in queries, and using solfege as pitch information were found as the results of analysis.

A New Similarity Measure for Improving Ranking in QA Systems (질의응답시스템 응답순위 개선을 위한 새로운 유사도 계산방법)

  • Kim Myung-Gwan;Park Young-Tack
    • Journal of KIISE:Computing Practices and Letters
    • /
    • v.10 no.6
    • /
    • pp.529-536
    • /
    • 2004
  • The main idea of this paper is to combine position information in sentence and query type classification to make the documents ranking to query more accessible. First, the use of conceptual graphs for the representation of document contents In information retrieval is discussed. The method is based on well-known strategies of text comparison, such as Dice Coefficient, with position-based weighted term. Second, we introduce a method for learning query type classification that improves the ability to retrieve answers to questions from Question Answering system. Proposed methods employ naive bayes classification in machine learning fields. And, we used a collection of approximately 30,000 question-answer pairs for training, obtained from Frequently Asked Question(FAQ) files on various subjects. The evaluation on a set of queries from international TREC-9 question answering track shows that the method with machine learning outperforms the underline other systems in TREC-9 (0.29 for mean reciprocal rank and 55.1% for precision).

Research on Designing Korean Emotional Dictionary using Intelligent Natural Language Crawling System in SNS (SNS대상의 지능형 자연어 수집, 처리 시스템 구현을 통한 한국형 감성사전 구축에 관한 연구)

  • Lee, Jong-Hwa
    • The Journal of Information Systems
    • /
    • v.29 no.3
    • /
    • pp.237-251
    • /
    • 2020
  • Purpose The research was studied the hierarchical Hangul emotion index by organizing all the emotions which SNS users are thinking. As a preliminary study by the researcher, the English-based Plutchick (1980)'s emotional standard was reinterpreted in Korean, and a hashtag with implicit meaning on SNS was studied. To build a multidimensional emotion dictionary and classify three-dimensional emotions, an emotion seed was selected for the composition of seven emotion sets, and an emotion word dictionary was constructed by collecting SNS hashtags derived from each emotion seed. We also want to explore the priority of each Hangul emotion index. Design/methodology/approach In the process of transforming the matrix through the vector process of words constituting the sentence, weights were extracted using TF-IDF (Term Frequency Inverse Document Frequency), and the dimension reduction technique of the matrix in the emotion set was NMF (Nonnegative Matrix Factorization) algorithm. The emotional dimension was solved by using the characteristic value of the emotional word. The cosine distance algorithm was used to measure the distance between vectors by measuring the similarity of emotion words in the emotion set. Findings Customer needs analysis is a force to read changes in emotions, and Korean emotion word research is the customer's needs. In addition, the ranking of the emotion words within the emotion set will be a special criterion for reading the depth of the emotion. The sentiment index study of this research believes that by providing companies with effective information for emotional marketing, new business opportunities will be expanded and valued. In addition, if the emotion dictionary is eventually connected to the emotional DNA of the product, it will be possible to define the "emotional DNA", which is a set of emotions that the product should have.