• Journal of KIISE:Software and Applications
• /
• v.33 no.12
• /
• pp.1073-1089
• /
• 2006

The goal of Natural Language Processing(NLP) is to make a computer understand a natural language and to deliver the meanings of natural language to humans. Word sense Disambiguation(WSD is a very important technology to achieve the goal of NLP. In this paper, we describe a technology for automatic homonyms disambiguation using both Mutual Information(MI) and a Sense-Tagged Compound Noun Dictionary. Previous research work using word definitions in dictionary suffered from the problem of data sparseness because of the use of exact word matching. Our work overcomes this problem by using MI which is an association measure between words. To reflect language features, the rate of word-pairs with MI values, sense frequency and site of word definitions are used as weights in our system. We constructed a Sense-Tagged Compound Noun Dictionary for high frequency compound nouns and used it to resolve homonym sense disambiguation. Experimental data for testing and evaluating our system is constructed from QA(Question Answering) test data which consisted of about 200 query sentences and answer paragraphs. We performed 4 types of experiments. In case of being used only MI, the result of experiment showed a precision of 65.06%. When we used the weighted values, we achieved a precision of 85.35% and when we used the Sense-Tagged Compound Noun Dictionary, we achieved a precision of 88.82%, respectively.

• Journal of KIISE:Software and Applications
• /
• v.32 no.5
• /
• pp.406-415
• /
• 2005

Prosperity of Internet led to the web containing huge number of documents. Thus increasing importance is given to the web information retrieval technology that can provide users with documents that contain the right information they want. This paper proposes several techniques that are effective for the improvement of web information retrieval. Similarity between a document and the query is a major source of information exploited by conventional systems. However, we suggest a technique to make use of similarity between a sentence and the query. We introduce a technique to compute the approximate score of the sentence-query similarity even without a mature technology of natural language processing. It was shown that the amount of computation for this task is linear to the number of documents in the total collection, which implies that practical systems can make use of this technique. The next important technique proposed in this paper is to use stratification of documents in re-ranking the documents to output. It was shown that it can lead to significant improvement in performance. We furthermore showed that using hyper links, anchor texts, and titles can result in enhancement of performance. To justify the proposed techniques we developed a large scale web information retrieval system and used it for experiments.

• Journal of the Korea Institute of Information Security & Cryptology
• /
• v.30 no.4
• /
• pp.669-677
• /
• 2020

As the number of Internet users exploded, attacks on the web increased. In addition, the attack patterns have been diversified to bypass existing defense techniques. Traditional web firewalls are difficult to detect attacks of unknown patterns.Therefore, the method of detecting abnormal behavior by artificial intelligence has been studied as an alternative. Specifically, attempts have been made to apply natural language processing techniques because the type of script or query being exploited consists of text. However, because there are many unknown words in scripts and queries, natural language processing requires a different approach. In this paper, we propose a new classification model which uses byte pair encoding (BPE) technology to learn the embedding vector, that is often used for web attack payloads, and uses an attention mechanism-based Bi-GRU neural network to extract a set of tokens that learn their order and importance. For major web attacks such as SQL injection, cross-site scripting, and command injection attacks, the accuracy of the proposed classification method is about 0.9990 and its accuracy outperforms the model suggested in the previous study.

• The Korean Journal of Applied Statistics
• /
• v.32 no.5
• /
• pp.763-782
• /
• 2019

This paper proposes a probabilistic document ranking model incorporating term dependencies. Document ranking is a fundamental information retrieval task. The task is to sort documents in a collection according to the relevance to the user query (Qin et al., Information Retrieval Journal, 13, 346-374, 2010). A probabilistic model is a model for computing the conditional probability of the relevance of each document given query. Most of the widely used models assume the term independence because it is challenging to compute the joint probabilities of multiple terms. Words in natural language texts are obviously highly correlated. In this paper, we assume a multinomial distribution model to calculate the relevance probability of a document by considering the dependency structure of words, and propose an information retrieval model to rank a document by estimating the probability with the maximum entropy method. The results of the ranking simulation experiment in various multinomial situations show better retrieval results than a model that assumes the independence of words. The results of document ranking experiments using real-world datasets LETOR OHSUMED also show better retrieval results.

• Journal of the Institute of Convergence Signal Processing
• /
• v.20 no.1
• /
• pp.31-36
• /
• 2019

A chatbot is an important area of mobile service, which understands informal data of a user as a conversational form and provides a customized service information for user. However, there is still a lack of a service way to fully understand the user's natural language typed query dialogue. Therefore, in this paper, we extract meaningful words, such a region, a food category, and a restaurant name from user's dialogue sentences for recommending a restaurant. and by comparing the extracted words against the contents of the knowledge database that is built from the hashtag for recommending a restaurant in SNS, and provides user target information having statistically much the word-similarity. In order to evaluate the performance of the restaurant recommendation chatbot system implemented in this paper, we measured the accessibility of various user query information by constructing a web-based mobile environment. As a results by comparing a previous similar system, our chabot is reduced by 37.2% and 73.3% with respect to the touch-count and the cutaway-count respectively.

• Journal of Digital Contents Society
• /
• v.19 no.5
• /
• pp.927-933
• /
• 2018

One of the factors that have the greatest influence on the error of the question answering system that finds and provides answers to natural language questions is the step of searching for documents or passages that contain correct answers. In order to improve the retrieval performance, it is necessary to understand the characteristics of documents and passages containing correct answers. This paper experimentally analyzes how many question words appear in the correct answer documents, how the location of the question word is distributed, and how the topic of the question and the correct answer document are similar using the corpus composed of the question, the documents with correct answer, and the documents without correct answer. This study explains the causes of previous search research results for question answer system and discusses the necessary elements of effective search step.

• Journal of the Korean Society for Library and Information Science
• /
• v.17
• /
• pp.49-83
• /
• 1989

A Full text retrieval system is a natural language document retrieval system in which the full text of all documents in a collection is stored on a computer so that every word in every sentence of every document can be located by the machine. This kind of IR System is recently becoming rapidly available online in the field of legal, newspaper, journal and reference book indexing. Increased research interest has been in this field. In this paper, research on full text databases and retrieval systems are reviewed, directions for research in this field are speculated, questions in the field that need answering are considered, and variables affecting online full text retrieval and various role that variables play in a research study are described. Two obvious research questions in full text retrieval have been how full text retrieval performs and how to improve the retrieval performance of full text databases. Research to improve the retrieval performance has been incorporated with ranking or weighting algorithms based on word occurrences, combined menu-driven and query-driven systems, and improvement of computer architectures and record structure for databases. Recent increase in the number of full text databases with various sizes, forms and subject matters, and recent development in computer architecture artificial intelligence, and videodisc technology promise new direction of its research and scholarly growth. Studies on the interrelationship between every elements of the full text retrieval situation and the relationship between each elements and retrieval performance may give a professional view in theory and practice of full text retrieval.

• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 2004.05b
• /
• pp.552-556
• /
• 2004

This paper surveys the design of the adaptive information filtering agents to retrieve the useful information within a large scale database. As the information retrieval through the Internet is generalized, it is necessary to extract the useful information satisfied the user's request condition to reduce the seeking time. For the first, this module is designed by the Rough reduct to generate the reduced minimal knowledge database considered the users natural query language in a large scale knowledge database, and also it is executed the soft computing by the fuzzy composite processing to operate the uncertain value of the reduced schema domain.

• Journal of KIISE:Software and Applications
• /
• v.33 no.8
• /
• pp.703-717
• /
• 2006

The Semantic Web has the objective of developing universal media in which machine-processable semantic information can be represented and shared, and it is therefore important to distribute ontologies that represent this kind of semantic information to the Web and make them available to multiple parties. However, the current ontology authoring tools are not operating on the Web, which makes it difficult to distribute ontologies directly to the Web and to create and edit them collaboratively with other people. This paper proposes a framework that facilitates the ontology construction and sharing, realizing easy distribution of ontologies to the Web. Wiki is one of the frameworks for collaborative construction and sharing of knowledge on the Web, and Wiki contents consist of natural language texts and simple markup language for visualization. For better collaboration in creating and sharing ontologies, this paper suggests the Semantic Wiki that embodies the Semantic Web features to the existing Wiki system. The Semantic Wiki framework facilitates the collaboration in ontology co-authoring and sharing for people, and at the same time, makes it possible for the agent software to easily manage the ontology information. Eventually, the Semantic Wiki system accomplishes various tasks including the semantic view, the semantic navigation, and the semantic query.

• Journal of the Korea Society of Computer and Information
• /
• v.19 no.9
• /
• pp.103-115
• /
• 2014

In this paper, we propose UniBAS(Unified Bug Analysis System) to provide a unified repository model by integrating the extracted data from the heterogeneous bug tracking systems. The UniBAS reduces the cost and complexity of the MSR(Mining Software Repositories) research process and enables the researchers to focus on their logics rather than the tedious and repeated works such as extracting repositories, processing data and building analysis models. Additionally, the system not only extracts the data but also automatically generates database tables, views and stored procedures which are required for the researchers to perform query-based analysis easily. It can also generate various types of exported files for utilizing external analysis tools or managing research data. A case study of detecting duplicate bug reports from the Firfox project of the Mozilla site has been performed based on the UniBAS in order to evaluate the usefulness of the system. The results of the experiments with various algorithms of natural language processing and flexible querying to the automatically extracted data also showed the effectiveness of the proposed system.