• Title/Summary/Keyword: TF-IDF

Search Result 332, Processing Time 0.027 seconds

Souce Code Identification Using Deep Neural Network (심층신경망을 이용한 소스 코드 원작자 식별)

  • Rhim, Jisu;Abuhmed, Tamer
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.8 no.9
    • /
    • pp.373-378
    • /
    • 2019
  • Since many programming sources are open online, problems with reckless plagiarism and copyrights are occurring. Among them, source codes produced by repeated authors may have unique fingerprints due to their programming characteristics. This paper identifies each author by learning from a Google Code Jam program source using deep neural network. In this case, the original creator's source is to be vectored using a pre-processing instrument such as predictive-based vector or frequency-based approach, TF-IDF, etc. and to identify the original program source by learning by using a deep neural network. In addition a language-independent learning system was constructed using a pre-processing machine and compared with other existing learning methods. Among them, models using TF-IDF and in-depth neural networks were found to perform better than those using other pre-processing or other learning methods.

Topic Modeling Analysis of Franchise Research Trends Using LDA Algorithm (LDA 알고리즘을 이용한 프랜차이즈 연구 동향에 대한 토픽모델링 분석)

  • YANG, Hoe-Chang
    • The Korean Journal of Franchise Management
    • /
    • v.12 no.4
    • /
    • pp.13-23
    • /
    • 2021
  • Purpose: This study aimed to derive clues for the franchise industry to overcome difficulties such as various legal regulations and social responsibility demands and to continuously develop by analyzing the research trends related to franchises published in Korea. Research design, data and methodology: As a result of searching for 'franchise' in ScienceON, abstracts were collected from papers published in domestic academic journals from 1994 to June 2021. Keywords were extracted from the abstracts of 1,110 valid papers, and after preprocessing, keyword analysis, TF-IDF analysis, and topic modeling using LDA algorithm, along with trend analysis of the top 20 words in TF-IDF by year group was carried out using the R-package. Results: As a result of keyword analysis, it was found that businesses and brands were the subjects of research related to franchises, and interest in service and satisfaction was considerable, and food and coffee were prominently studied as industries. As a result of TF-IDF calculation, it was found that brand, satisfaction, franchisor, and coffee were ranked at the top. As a result of LDA-based topic modeling, a total of 12 topics including "growth strategy" were derived and visualized with LDAvis. On the other hand, the areas of Topic 1 (growth strategy) and Topic 9 (organizational culture), Topic 4 (consumption experience) and Topic 6 (contribution and loyalty), Topic 7 (brand image) and Topic 10 (commercial area) overlap significantly. Finally, the trend analysis results for the top 20 keywords with high TF-IDF showed that 10 keywords such as quality, brand, food, and trust would be more utilized overall. Conclusions: Through the results of this study, the direction of interest in the franchise industry was confirmed, and it was found that it was necessary to find a clue for continuous growth through research in more diverse fields. And it was also considered an important finding to suggest a technique that can supplement the problems of topic trend analysis. Therefore, the results of this study show that researchers will gain significant insights from the perspectives related to the selection of research topics, and practitioners from the perspectives related to future franchise changes.

A Study on Research Paper Classification Using Keyword Clustering (키워드 군집화를 이용한 연구 논문 분류에 관한 연구)

  • Lee, Yun-Soo;Pheaktra, They;Lee, JongHyuk;Gil, Joon-Min
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.7 no.12
    • /
    • pp.477-484
    • /
    • 2018
  • Due to the advancement of computer and information technologies, numerous papers have been published. As new research fields continue to be created, users have a lot of trouble finding and categorizing their interesting papers. In order to alleviate users' this difficulty, this paper presents a method of grouping similar papers and clustering them. The presented method extracts primary keywords from the abstracts of each paper by using TF-IDF. Based on TF-IDF values extracted using K-means clustering algorithm, our method clusters papers to the ones that have similar contents. To demonstrate the practicality of the proposed method, we use paper data in FGCS journal as actual data. Based on these data, we derive the number of clusters using Elbow scheme and show clustering performance using Silhouette scheme.

Analysis of whether the feeling of relative deprivation is shown in the comments of the Luxury Howl YouTube video - Focusing on modern sentiment analysis using TF-IDF, Word2vec, LDA and LSTM - (명품 하울 유튜브 영상 댓글에 나타난 상대적 박탈감 여부와 특징 분석 - TF-IDF, Word2vec, LDA, LSTM을 이용한 현대인의 감정 분석을 중심으로 -)

  • Choi, Jung Min;Oh, Hayoung
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.25 no.3
    • /
    • pp.355-360
    • /
    • 2021
  • Recently Youtube has been more popular. As many studies show the comparative deprivation of the Social Medeia, this study looks into whether the comparative deprivation is expressed on the YouTube comments. It focuses on the Luxury Haul contents, videos about huge amounts of luxurious products, of which Youtubers'economic feature are demonstrative. The comments of the videos are analyzed with LDA TF-IDF and Word2Vec. Additionally, the comments were classified into positive and negative groups by the LSTM model as well. As a result of the study, even though many comments turned out positive, the negative keywords were indicated related to comparative deprivation. Also it was found that the viewers compared themselves with Youtubers. In particular, some YouTubers are more criticized if they are younger or does not seem to afford the luxurious products themselves. This study suggests that the users express the comparative deprivation on YouTube as well like on the other Social Media.

A Study on Improving Precision Rate in Security Events Using Cyber Attack Dictionary and TF-IDF (공격키워드 사전 및 TF-IDF를 적용한 침입탐지 정탐률 향상 연구)

  • Jongkwan Kim;Myongsoo Kim
    • Convergence Security Journal
    • /
    • v.22 no.2
    • /
    • pp.9-19
    • /
    • 2022
  • As the expansion of digital transformation, we are more exposed to the threat of cyber attacks, and many institution or company is operating a signature-based intrusion prevention system at the forefront of the network to prevent the inflow of attacks. However, in order to provide appropriate services to the related ICT system, strict blocking rules cannot be applied, causing many false events and lowering operational efficiency. Therefore, many research projects using artificial intelligence are being performed to improve attack detection accuracy. Most researches were performed using a specific research data set which cannot be seen in real network, so it was impossible to use in the actual system. In this paper, we propose a technique for classifying major attack keywords in the security event log collected from the actual system, assigning a weight to each key keyword, and then performing a similarity check using TF-IDF to determine whether an actual attack has occurred.

An Exploratory Study on Survey Data Categorization using DDI metadata (메타데이터를 활용한 조사자료의 문서범주화에 관한 연구)

  • Park, Ja-Hyun;Song, Min
    • Proceedings of the Korean Society for Information Management Conference
    • /
    • 2012.08a
    • /
    • pp.73-76
    • /
    • 2012
  • 본 연구는 DDI 메타데이터를 활용하여 귀납적 학습모델(supervised learning model)의 문서범주화 실험을 수행함으로써 조사자료의 체계적이고 효율적인 분류작업을 설계하는데 그 목적이 있다. 구체적으로 조사자료의 DDI 메타데이터를 대상으로 단순 TF 가중치, TF-IDF 가중치, Okapi TF 가중치에 따른 나이브 베이즈(Naive Bayes), kNN(k nearest neighbor), 결정트리(Decision tree) 분류기의 성능비교 실험을 하였다. 그 결과, 나이브 베이즈가 가장 좋은 성능을 보였으며, 단순 TF 가중치와 TF-IDF 가중치는 나이브 베이즈, kNN, 결정트리 분류기에서 동일한 성능을 보였으나, Okapi TF 가중치의 경우 나이브 베이즈에서 가장 좋은 성능을 보였다.

  • PDF

Comparison of Term-Weighting Schemes for Environmental Big Data Analysis (환경 빅데이터 이슈 분석을 위한 용어 가중치 기법 비교)

  • Kim, JungJin;Jeong, Hanseok
    • Proceedings of the Korea Water Resources Association Conference
    • /
    • 2021.06a
    • /
    • pp.236-236
    • /
    • 2021
  • 최근 텍스트와 같은 비정형 데이터의 생성 속도가 급격하게 증가함에 따라, 이를 분석하기 위한 기술들의 필요성이 커지고 있다. 텍스트 마이닝은 자연어 처리기술을 사용하여 비정형 텍스트를 정형화하고, 문서에서 가치있는 정보를 획득할 수 있는 기법 중 하나이다. 텍스트 마이닝 기법은 일반적으로 각각의 분서별로 특정 용어의 사용 빈도를 나타내는 문서-용어 빈도행렬을 사용하여 용어의 중요도를 나타내고, 다양한 연구 분야에서 이를 활용하고 있다. 하지만, 문서-용어 빈도 행렬에서 나타내는 용어들의 빈도들은 문서들의 차별성과 그에 따른 용어들의 중요도를 나타내기 어렵기때문에, 용어 가중치를 적용하여 문서가 가지고 있는 특징을 분류하는 방법이 필수적이다. 다양한 용어 가중치를 적용하는 방법들이 개발되어 적용되고 있지만, 환경 분야에서는 용어 가중치 기법 적용에 따른 효율성 평가 연구가 미비한 상황이다. 또한, 환경 이슈 분석의 경우 단순히 문서들에 특징을 파악하고 주어진 문서들을 분류하기보다, 시간적 분포도에 따른 각 문서의 특징을 반영하는 것도 상대적으로 중요하다. 따라서, 본 연구에서는 텍스트 마이닝을 이용하여 2015-2020년의 서울지역 환경뉴스 데이터를 사용하여 환경 이슈 분석에 적합한 용어 가중치 기법들을 비교분석하였다. 용어 가중치 기법으로는 TF-IDF (Term frequency-inverse document frquency), BM25, TF-IGM (TF-inverse gravity moment), TF-IDF-ICSDF (TF-IDF-inverse classs space density frequency)를 적용하였다. 본 연구를 통해 환경문서 및 개체 분류에 대한 최적화된 용어 가중치 기법을 제시하고, 서울지역의 환경 이슈와 관련된 핵심어 추출정보를 제공하고자 한다.

  • PDF

Exploring Information Ethics Issues based on Text Mining using Big Data from Web of Science (Web of Science 빅데이터를 활용한 텍스트 마이닝 기반의 정보윤리 이슈 탐색)

  • Kim, Han Sung
    • The Journal of Korean Association of Computer Education
    • /
    • v.22 no.3
    • /
    • pp.67-78
    • /
    • 2019
  • The purpose of this study is to explore information ethics issues based on academic big data from Web of Science (WoS) and to provide implications for information ethics education in informatics subject. To this end, 318 published papers from WoS related to information ethics were text mined. Specifically, this paper analyzed the frequency of key-words(TF, DF, TF-IDF), information ethics issues using topic modeling, and frequency of appearances by year for each issue. This paper used 'tm', 'topicmodel' package of R for text mining. The main results are as follows. First, this paper confirmed that the words 'digital', 'student', 'software', and 'privacy' were the main key-words through TF-IDF. Second, the topic modeling analysis showed 8 issues such as 'Professional value', 'Cyber-bullying', 'AI and Social Impact' et al., and the proportion of 'Professional value' and 'Cyber-bullying' was relatively high. This study discussed the implications for information ethics education in Korea based on the results of this analysis.

Rating and Comments Mining Using TF-IDF and SO-PMI for Improved Priority Ratings

  • Kim, Jinah;Moon, Nammee
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.13 no.11
    • /
    • pp.5321-5334
    • /
    • 2019
  • Data mining technology is frequently used in identifying the intention of users over a variety of information contexts. Since relevant terms are mainly hidden in text data, it is necessary to extract them. Quantification is required in order to interpret user preference in association with other structured data. This paper proposes rating and comments mining to identify user priority and obtain improved ratings. Structured data (location and rating) and unstructured data (comments) are collected and priority is derived by analyzing statistics and employing TF-IDF. In addition, the improved ratings are generated by applying priority categories based on materialized ratings through Sentiment-Oriented Point-wise Mutual Information (SO-PMI)-based emotion analysis. In this paper, an experiment was carried out by collecting ratings and comments on "place" and by applying them. We confirmed that the proposed mining method is 1.2 times better than the conventional methods that do not reflect priorities and that the performance is improved to almost 2 times when the number to be predicted is small.

Weighting Methods and their Evaluations for Compound Nouns in Korean Text Retrieval (한국어 정보검색에서의 복합명사 가중치 부여 방법 및 평가)

  • Kim, Ji-Young;Sung, Hyon-Myaeng
    • Annual Conference on Human and Language Technology
    • /
    • 2001.10d
    • /
    • pp.157-162
    • /
    • 2001
  • 한국어의 경우 띄어쓰기의 자유로움과 명사들이 비교적 자유롭게 결합하여 새로운 복합명사(compound noun)를 형성한다. 따라서, 정보검색에서 복합명사를 적절하게 처리하게 되면 검색 효율을 향상시킬 수 있다. 본 논문에서는 질의에 포함된 단일명사, 복합명사, 그리고 복합명사를 이루는 구성명사의 적절한 가중치 부여 방법에 대하여 기술한다. 일반적인 tf*idf가중치 방법은 문서 내 빈도수(tf)만을 강조하여 문서 내 발생빈도가 낮은 복합명사의 경우 낮은 가중치를 갖는다. 반대로, 역문헌 빈도수(idf)로 인해 복합명사가 단일명사보다 높은 가중치를 갖게 되면 단일명사의 가중치를 지나치게 떨어뜨려 검색 성능을 저하시킨다. 이런 문제를 해결하기 위해서 복합명사의 통계적인 특성을 고려하고, 복합명사를 이루는 구성명사의 적절한 가중치 사용과 tf*idf 변화 범위에 따른 파라메터를 이용하였다. 결과적으로 본 논문에서는 질의 색인어의 종류에 따라 가중치를 달리 부여함으로써 검색 성능을 향상시킬 수 있는 가중치 부여 방법을 제시하고 검증 실험을 통해 유효성을 제시했다는 점에서 그 의의가 있다고 하겠다.

  • PDF