• Title/Summary/Keyword: 데이터 희소성 문제

Search Result 57, Processing Time 0.027 seconds

Applying Different Similarity Measures based on Jaccard Index in Collaborative Filtering

  • Lee, Soojung
    • Journal of the Korea Society of Computer and Information
    • /
    • v.26 no.5
    • /
    • pp.47-53
    • /
    • 2021
  • Sparse ratings data hinder reliable similarity computation between users, which degrades the performance of memory-based collaborative filtering techniques for recommender systems. Many works in the literature have been developed for solving this data sparsity problem, where the most simple and representative ones are the methods of utilizing Jaccard index. This index reflects the number of commonly rated items between two users and is mostly integrated into traditional similarity measures to compute similarity more accurately between the users. However, such integration is very straightforward with no consideration of the degree of data sparsity. This study suggests a novel idea of applying different similarity measures depending on the numeric value of Jaccard index between two users. Performance experiments are conducted to obtain optimal values of the parameters used by the proposed method and evaluate it in comparison with other relevant methods. As a result, the proposed demonstrates the best and comparable performance in prediction and recommendation accuracies.

A Collaborative Recommendation Based on Neural Networks Using the Clustering (클러스터링을 이용한 신경망 기반 협력적 추천)

  • 김은주;류정우;김명원
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2002.10d
    • /
    • pp.343-345
    • /
    • 2002
  • 개인화를 위한 협력적 추천의 대표적인 방법인 최근접 이웃 방법은 적용이 쉽지만, 사용자의 선호도 정보가 적을 경우 회소성(sparsity)문제와 사용자 수가 많은 경우 수행 속도가 느려지는 범위성(Scalability)문제 그리고 사용자간의 가중치가 결여되었다는 점에서 추천의 정확성이 떨어진다. 신경망 기반 추천은 자료의 유형에 상관없이 데이터의 처리가 용이하고, 사용자간의 가중치를 학습할 수 있으며, 내용 정보, 인구통계학적 정보 등을 입력 노드에 추가함으로써 희소성 문제를 해결할 수 있으나. 범위성 문제는 존재한다. 따라서 본 논문에서는 최근접 이웃 방법으로 클러스터링 한 유사한 사용자 또는 항목들을 고려한 신경망 기반 추천 방법을 제안하여 범위성 문제를 최소화시킴으로써 추천의 성능을 향상시키고 있다. 제안한 추천 방법의 타당성을 보이기 위해 EachMovie데이터를 이용하여 기존 신경망 추천과 비교 실험하여 성능을 분석한다.

  • PDF

Sparse Document Data Clustering Using Factor Score and Self Organizing Maps (인자점수와 자기조직화지도를 이용한 희소한 문서데이터의 군집화)

  • Jun, Sung-Hae
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.22 no.2
    • /
    • pp.205-211
    • /
    • 2012
  • The retrieved documents have to be transformed into proper data structure for the clustering algorithms of statistics and machine learning. A popular data structure for document clustering is document-term matrix. This matrix has the occurred frequency value of a term in each document. There is a sparsity problem in this matrix because most frequencies of the matrix are 0 values. This problem affects the clustering performance. The sparseness of document-term matrix decreases the performance of clustering result. So, this research uses the factor score by factor analysis to solve the sparsity problem in document clustering. The document-term matrix is transformed to document-factor score matrix using factor scores in this paper. Also, the document-factor score matrix is used as input data for document clustering. To compare the clustering performances between document-term matrix and document-factor score matrix, this research applies two typed matrices to self organizing map (SOM) clustering.

A Music Recommendation System by Using Graph-based Collaborative Filtering (그래프 기반 협동적 여과를 이용한 음악 추천 시스템)

  • Kim, Hyung-Il;Lee, Jin-Seok;Lee, Jeong-Hyun;Cho, Chin-Kwna;Kim, Kyoung-Sup;Kim, Jun-Tae
    • Annual Conference of KIPS
    • /
    • 2006.11a
    • /
    • pp.51-54
    • /
    • 2006
  • 본 논문에서는 각 사용자들의 취향에 맞는 음악을 추천하는 개인화된 음악 추천 시스템을 소개한다. 추천 시스템이란 사용자의 선호도를 분석하고 아이템들에 대한 사용자의 선호도를 예측하여 영화, 음악, 기사, 책, 웹 페이지 등과 같은 아이템들을 추천하는 시스템을 말한다. 추천 시스템들에서 가장 많이 사용하고 있는 협동적 추천 방식은 선호도 데이터를 기반으로 유사한 사용자들을 찾고, 유사 사용자들의 선호도를 기반으로 예측을 수행하는 것으로서, 여러 장점들이 있으나 희소성(sparsity) 문제와 확장성(scalability) 문제에 대해 취약점을 가지고 있다. 아이템들의 전체 수에 비해 매우 적은 수의 아이템 선호도 데이터만 존재한다면 사용자들의 유사도를 계산하기가 어려우며, 또한 사용자의 수가 늘어날수록 유사도 계산에 걸리는 시간이 급격하게 늘어남으로써 수백만 사용자가 있는 웹 사이트 등에서 실시간 추천을 수행하기 어렵다. 본 논문에서 소개하는 음악 추천 시스템은 이러한 문제점들을 해결하기 위해 그래프 기반 협동적 여과 기법을 사용한다. 그래프 기반 협동적 여과 기법은 기존의 협동적 여과 기법들과 달리 아이템들 사이의 연관관계를 그래프 모델로 표현하고 저장함으로써 묵시적인 선호도 정보들을 누적하여 희소성 문제를 해결하고, 추천 아이템을 선정하는데 필요한 계산 시간을 크게 단축하여 대규모 데이터에서 실시간 추천을 가능하게 한다는 장점이 있다.

  • PDF

Expanding Korean/English Parallel Corpora using Back-translation for Neural Machine Translation (신경망 기반 기계 번역을 위한 역-번역을 이용한 한영 병렬 코퍼스 확장)

  • Xu, Guanghao;Ko, Youngjoong;Seo, Jungyun
    • Annual Conference on Human and Language Technology
    • /
    • 2018.10a
    • /
    • pp.470-473
    • /
    • 2018
  • 최근 제안된 순환 신경망 기반 Encoder-Decoder 모델은 기계번역에서 좋은 성능을 보인다. 하지만 이는 대량의 병렬 코퍼스를 전제로 하며 병렬 코퍼스가 소량일 경우 데이터 희소성 문제가 발생하며 번역의 품질은 다소 제한적이다. 본 논문에서는 기계번역의 이러한 문제를 해결하기 위하여 단일-언어(Monolingual) 데이터를 학습과정에 사용하였다. 즉, 역-번역(Back-translation)을 이용하여 단일-언어 데이터를 가상 병렬(Pseudo Parallel) 데이터로 변환하는 방식으로 기존 병렬 코퍼스를 확장하여 번역 모델을 학습시켰다. 역-번역 방법을 이용하여 영-한 번역 실험을 수행한 결과 +0.48 BLEU 점수의 성능 향상을 보였다.

  • PDF

A Research on the Regulations and Perception of Interactive Game in Data Broadcasting: Special Emphasis on the TV-Betting Game (데이터방송 인터랙티브 게임 규제 및 이용자 인식에 관한 연구: 승부게임을 중심으로)

  • Byun, Dong-Hyun;Jung, Moon-Ryul;Bae, Hong-Seob
    • Korean journal of communication and information
    • /
    • v.35
    • /
    • pp.250-291
    • /
    • 2006
  • This study examines the regulatory issues and introduction problems of TV-betting data broadcasts in Korea by in-depth interview with a panel group. TV-betting data broadcast services of card games and horse racing games are widely in use in Europe and other parts of the world. In order to carry out the study, a demo program of TV-betting data broadcast in the OCAP(OpenCableTM Application Platform Specification) system environment, which is the data broadcasting standard for digital cable broadcasts in Korea was exposed to the panel group and then they were interviewed after watching and using the program. The results could be summarized as below. First of all, while TV-betting data broadcasts have many elements of entertainment, the respondents thought that it would be difficult to introduce TV-betting in data broadcasts as in overseas countries largely due to social factors. In addition, in order to introduce TV-betting data broadcasts, they suggested that excessive speculativeness must be suppressed through a series of regulatory system devices, such as by guaranteeing credibility of the media based on safe security systems for transactions, scheduling programs with effective time constraints to prevent the games from running too frequently, limiting the betting values, and by prohibiting access to games through set-top boxes of other data broadcast subscribers. The general consensus was that TV-betting could be considered for gradual introduction within the governmental laws and regulations that would minimize its ill effects. Therefore, the government should formulate long-term regulations and policies for data broadcasts. Once the groundwork is laid for safe introduction of TV-betting on data broadcasts within the boundary of laws and regulations, interactive TV games are expected to be introduced in Korea not only for added functionality of entertainment but also for far-ranging development of data broadcast and new media industries.

  • PDF

Conditional Generative Adversarial Network based Collaborative Filtering Recommendation System (Conditional Generative Adversarial Network(CGAN) 기반 협업 필터링 추천 시스템)

  • Kang, Soyi;Shin, Kyung-shik
    • Journal of Intelligence and Information Systems
    • /
    • v.27 no.3
    • /
    • pp.157-173
    • /
    • 2021
  • With the development of information technology, the amount of available information increases daily. However, having access to so much information makes it difficult for users to easily find the information they seek. Users want a visualized system that reduces information retrieval and learning time, saving them from personally reading and judging all available information. As a result, recommendation systems are an increasingly important technologies that are essential to the business. Collaborative filtering is used in various fields with excellent performance because recommendations are made based on similar user interests and preferences. However, limitations do exist. Sparsity occurs when user-item preference information is insufficient, and is the main limitation of collaborative filtering. The evaluation value of the user item matrix may be distorted by the data depending on the popularity of the product, or there may be new users who have not yet evaluated the value. The lack of historical data to identify consumer preferences is referred to as data sparsity, and various methods have been studied to address these problems. However, most attempts to solve the sparsity problem are not optimal because they can only be applied when additional data such as users' personal information, social networks, or characteristics of items are included. Another problem is that real-world score data are mostly biased to high scores, resulting in severe imbalances. One cause of this imbalance distribution is the purchasing bias, in which only users with high product ratings purchase products, so those with low ratings are less likely to purchase products and thus do not leave negative product reviews. Due to these characteristics, unlike most users' actual preferences, reviews by users who purchase products are more likely to be positive. Therefore, the actual rating data is over-learned in many classes with high incidence due to its biased characteristics, distorting the market. Applying collaborative filtering to these imbalanced data leads to poor recommendation performance due to excessive learning of biased classes. Traditional oversampling techniques to address this problem are likely to cause overfitting because they repeat the same data, which acts as noise in learning, reducing recommendation performance. In addition, pre-processing methods for most existing data imbalance problems are designed and used for binary classes. Binary class imbalance techniques are difficult to apply to multi-class problems because they cannot model multi-class problems, such as objects at cross-class boundaries or objects overlapping multiple classes. To solve this problem, research has been conducted to convert and apply multi-class problems to binary class problems. However, simplification of multi-class problems can cause potential classification errors when combined with the results of classifiers learned from other sub-problems, resulting in loss of important information about relationships beyond the selected items. Therefore, it is necessary to develop more effective methods to address multi-class imbalance problems. We propose a collaborative filtering model using CGAN to generate realistic virtual data to populate the empty user-item matrix. Conditional vector y identify distributions for minority classes and generate data reflecting their characteristics. Collaborative filtering then maximizes the performance of the recommendation system via hyperparameter tuning. This process should improve the accuracy of the model by addressing the sparsity problem of collaborative filtering implementations while mitigating data imbalances arising from real data. Our model has superior recommendation performance over existing oversampling techniques and existing real-world data with data sparsity. SMOTE, Borderline SMOTE, SVM-SMOTE, ADASYN, and GAN were used as comparative models and we demonstrate the highest prediction accuracy on the RMSE and MAE evaluation scales. Through this study, oversampling based on deep learning will be able to further refine the performance of recommendation systems using actual data and be used to build business recommendation systems.

Performance Analysis of Similarity Reflecting Jaccard Index for Solving Data Sparsity in Collaborative Filtering (협력필터링의 데이터 희소성 해결을 위한 자카드 지수 반영의 유사도 성능 분석)

  • Lee, Soojung
    • The Journal of Korean Association of Computer Education
    • /
    • v.19 no.4
    • /
    • pp.59-66
    • /
    • 2016
  • It has been studied to reflect the number of co-rated items for solving data sparsity problem in collaborative filtering systems. A well-known method of Jaccard index allowed performance improvement, when combined with previous similarity measures. However, the degree of performance improvement when combined with existing similarity measures in various data environments are seldom analyzed, which is the objective of this study. Jaccard index as a sole similarity measure yielded much higher prediction quality than traditional measures and very high recommendation quality in a sparse dataset. In general, previous similarity measures combined with Jaccard index improved performance regardless of dataset characteristics. Especially, cosine similarity achieved the highest improvement in sparse datasets, while similarity of Mean Squared Difference degraded prediction quality in denser sets. Therefore, one needs to consider characteristics of data environment and similarity measures before combining Jaccard index for similarity use.

Toward Preventing Cold-start Problem: Basis Recommendation System (콜드스타트 문제 완화를 위한 기저속성 추출 기반 추천시스템 제안)

  • Jungseob Lee;Hyeonseok Moon;Chanjun Park;Myunghoon Kang;Seungjun Lee;Sungmin Ahn;Jeongbae Park;Heuiseok Lim
    • Annual Conference on Human and Language Technology
    • /
    • 2022.10a
    • /
    • pp.427-430
    • /
    • 2022
  • 추천시스템에서 콜드스타트 문제를 해결하기 위해 다양한 연구들이 진행되고 있다. 하지만, 대부분의 연구는 아직도 사용자 기반의 히스토리 데이터셋을 반드시 필요로 하여, 콜드스타트 문제를 완벽히 해결하지 못하고 있다. 이에 본 논문은 콜드스타트 문제를 완화할 수 있는 기저속성 기반의 추천시스템을 제안한다. 제안하는 방법론을 검증하기 위해, 직접 수집한 한국어 영화 리뷰 데이터셋을 기반으로 성능을 검증하였으며, 평가 결과 제안한 방법론이 키워드와 사용자의 리뷰 점수를 효과적으로 반영한 추천시스템임을 확인할 수 있었고, 데이터 희소성 및 콜드스타트 문제를 완화하여 기존의 텍스트 기반 랭킹 시스템의 성능을 압도하는 것을 확인하였다. 더 나아가 제안된 기저속성 추천시스템은 추론 시에 GPU 컴퓨팅 자원을 요구하지 않기에 서비스 측면에서도 많은 이점이 있음을 확인하였다.

  • PDF

Jaccard Index Reflecting Time-Context for User-based Collaborative Filtering

  • Soojung Lee
    • Journal of the Korea Society of Computer and Information
    • /
    • v.28 no.10
    • /
    • pp.163-170
    • /
    • 2023
  • The user-based collaborative filtering technique, one of the implementation methods of the recommendation system, recommends the preferred items of neighboring users based on the calculations of neighboring users with similar rating histories. However, it fundamentally has a data scarcity problem in which the quality of recommendations is significantly reduced when there is little common rating history. To solve this problem, many existing studies have proposed various methods of combining Jaccard index with a similarity measure. In this study, we introduce a time-aware concept to Jaccard index and propose a method of weighting common items with different weights depending on the rating time. As a result of conducting experiments using various performance metrics and time intervals, it is confirmed that the proposed method showed the best performance compared to the original Jaccard index at most metrics, and that the optimal time interval differs depending on the type of performance metric.