• Journal of Software Assessment and Valuation
• /
• v.14 no.2
• /
• pp.35-44
• /
• 2018

Tools for detecting cross-language clones usually compare abstract-syntax-tree representations of source code, which lacks scalability. In order to compare large source code to a practical level, we need a similarity checking technique that works on a token level basis. In this paper, we define common tokens that represent all tokens commonly used in programming languages of different paradigms. Each source code of different language is then transformed into the list of common tokens that are compared. Experimental results using exEyes show that our proposed method using common tokens is effective in detecting cross-language clones.

• Annual Conference on Human and Language Technology
• /
• 2020.10a
• /
• pp.441-445
• /
• 2020

토큰화(Tokenization)는 사람이 작성한 자연어 문장을 기계가 잘 이해할 수 있도록 최소 단위인 토큰으로 분리하는 작업을 말하여, 이러한 토큰화는 자연어처리 전반적인 태스크들의 전처리에 필수적으로 사용되고 있다. 최근 자연어처리 분야에서 높은 성능을 보이며, 다양한 딥러닝 모델에 많이 활용되고 있는 SentencePiece 토큰화는 여러 단어에서 공통적으로 출현하는 부분단어들을 기준으로, BPE 알고리즘을 이용하여 문장을 압축 표현하는 토큰화 방법이다. 본 논문에서는 한국어 기반 특허 문헌의 초록 자연어 데이터를 기반으로 SentencePiece를 비롯한 여러 토큰화 방법에 대하여 소개하며, 해당 방법을 응용한 기계번역 (Neural Machine Translation) 태스크를 수행하고, 토큰화 방법별 비교 평가를 통해 특허 분야 자연어 데이터에 최적화된 토큰화 방법을 제안한다. 그리고 본 논문에서 제안한 방법을 사용하여 특허 초록 한-영 기계번역 태스크에서 성능이 향상됨을 보였다.

• KIPS Transactions on Software and Data Engineering
• /
• v.2 no.4
• /
• pp.267-272
• /
• 2013

As an important operation with many applications such as data cleaning and duplicate detection, the similarity join is a challenging issue, which finds all pairs of records whose similarities are above a given threshold in a dataset. We propose a new algorithm that uses the prefix filtering principle as strong constraints on generation of candidate pairs for fast similarity joins. The candidate pair is generated only when the current prefix token of a probing record shares one prefix token of an indexing record within the constrained prefix tokens by the principle. This generation method needs not to compute an upper bound of the overlap between two records, which results in reduction of execution time. Experimental results show that our algorithm significantly outperforms the previous prefix filtering-based algorithms on real datasets.

• Journal of the Korea Convergence Society
• /
• v.13 no.3
• /
• pp.43-54
• /
• 2022

Simultaneous translation is online decoding to translates with only subsentence. The goal of simultaneous translation research is to improve translation performance against delay. For this reason, most studies find trade-off performance between delays. We studied the experiments of the fixed policy-based simultaneous translation in Korean. Our experiments suggest that Korean tokenization causes many fragments, resulting in delay compared to other languages. We suggest follow-up studies such as n-gram tokenization to solve the problems.

• KIPS Transactions on Software and Data Engineering
• /
• v.4 no.2
• /
• pp.71-76
• /
• 2015

In similarity join processing, a general technique employs a generation-verification framework, which includes two phases: the first phase generates a set of candidate pairs from a collection of records; and the second phase verifies each candidate pair by computing real similarity. In order to reduce the number of candidate pairs in the verification phase, the median of one record of each candidate pair is used as a filter in this paper to test whether the other record can has the proper number of overlapped tokens. We propose a similarity join algorithm with the median filter, and show that the proposed algorithm has better performance in execution time than recent algorithms without the filter through extensive experiments on real-world datasets.

• The Korean Journal of Archival Studies
• /
• no.70
• /
• pp.85-117
• /
• 2021

The purpose of this study is to investigate and analyze the current status of unit tasks, unit task operation, and record management problems used by local governments, and to present improvement measures using text-based big data technology based on the implications derived from the process. Local governments are in a serious state of record management operation due to errors in preservation period due to misclassification of unit tasks, inability to identify types of overcommon and institutional affairs, errors in unit tasks, errors in name, referenceable standards, and tools. However, the number of unit tasks is about 720,000, which cannot be effectively controlled due to excessive quantities, and thus strict and controllable tools and standards are needed. In order to solve these problems, this study developed a system that applies text-based analysis tools such as corpus and tokenization technology during big data analysis, and applied them to the names and construction terms constituting the record management standard. These unit task operation support tools are expected to contribute significantly to record management tasks as they can support standard operability such as uniform preservation period, identification of delegated office records, control of duplicate and similar unit task creation, and common tasks. Therefore, if the big data analysis methodology can be linked to BRM and RMS in the future, it is expected that the quality of the record management standard work will increase.