• Title/Summary/Keyword: Data sparsity

Search Result 174, Processing Time 0.025 seconds

Extended High Dimensional Clustering using Iterative Two Dimensional Projection Filtering (반복적 2차원 프로젝션 필터링을 이용한 확장 고차원 클러스터링)

  • Lee, Hye-Myeong;Park, Yeong-Bae
    • The KIPS Transactions:PartD
    • /
    • v.8D no.5
    • /
    • pp.573-580
    • /
    • 2001
  • The large amounts of high dimensional data contains a significant amount of noises by it own sparsity, which adds difficulties in high dimensional clustering. The CLIP is developed as a clustering algorithm to support characteristics of the high dimensional data. The CLIP is based on the incremental one dimensional projection on each axis and find product sets of the dimensional clusters. These product sets contain not only all high dimensional clusters but also they may contain noises. In this paper, we propose extended CLIP algorithm which refines the product sets that contain cluster. We remove high dimensional noises by applying two dimensional projections iteratively on the already found product sets by CLIP. To evaluate the performance of extended algorithm, we demonstrate its effectiveness through a series of experiments on synthetic data sets.

  • PDF

Development of a Personalized Recommendation Procedure Based on Data Mining Techniques for Internet Shopping Malls (인터넷 쇼핑몰을 위한 데이터마이닝 기반 개인별 상품추천방법론의 개발)

  • Kim, Jae-Kyeong;Ahn, Do-Hyun;Cho, Yoon-Ho
    • Journal of Intelligence and Information Systems
    • /
    • v.9 no.3
    • /
    • pp.177-191
    • /
    • 2003
  • Recommender systems are a personalized information filtering technology to help customers find the products they would like to purchase. Collaborative filtering is the most successful recommendation technology. Web usage mining and clustering analysis are widely used in the recommendation field. In this paper, we propose several hybrid collaborative filtering-based recommender procedures to address the effect of web usage mining and cluster analysis. Through the experiment with real e-commerce data, it is found that collaborative filtering using web log data can perform recommendation tasks effectively, but using cluster analysis can perform efficiently.

  • PDF

A Study on Measurement Selection Algorithm for Power System State Estimation under the consideration of Dummy Buses (DUMMY 모선을 고려한 상태추정 측정점선정 알고리즘에 관한 연구)

  • Lee, Tae-Shik;Moon, Young-Hyun;Ham, W.K.;Kwon, T.W.
    • Proceedings of the KIEE Conference
    • /
    • 1991.07a
    • /
    • pp.406-410
    • /
    • 1991
  • This paper presents an improved algorithm of optimal measurement system design with a reliability evaluation method for large power system. The proposed algorithm is developed to consider the dummy bus and to achieve highest accuracy of the state estimator as well with the limited Investment cost. The dummy bus in the power system is impossible to install measurement meter, while real and reactive power measurement values are exactly zero. Thus, the effect on these dummy bus measurements is considered in the proposed algorithm. On the other hand, P/C model is developed by taking advantage of the matrix sparsity. The improved program is successfully tested for KEPCO system with PSS/E lineflow calculated data package.

  • PDF

웹마이닝과 상품계층도를 이용한 협업필터링 기반 개인별 상품추천시스템

  • An, Do-Hyeon;Kim, Jae-Gyeong;Jo, Yun-Ho
    • Proceedings of the Korean Operations and Management Science Society Conference
    • /
    • 2004.05a
    • /
    • pp.510-514
    • /
    • 2004
  • Recommender systems are a personalized information filtering technology to help customers find the products they would like to purchase. Collaborative filtering is known to be the most successful recommendation technology, but its widespread use has exposed some problems such as sparsity and scalability in the e-business environment. In this paper, we propose a recommendation methodology based on Web usage mining and product taxonomy to enhance the recommendation quality and the system performance of original CF-based recommender systems. Web usage mining populates the rating database by tracking customers' shopping behaviors on the Web, so leading to better quality recommendations. The product taxonomy is used to improve the performance of searching for nearest neighbors through dimensionality reduction of the rating database. Several experiments on real e-commerce data show that the proposed methodology provides higher quality recommendations and better performance than original collaborative filtering methodology.

  • PDF

Performance Improvement of a Recommendation System using Stepwise Collaborative Filtering (단계적 협업필터링을 이용한 추천시스템의 성능 향상)

  • Lee, Jae-Sik;Park, Seok-Du
    • Proceedings of the Korea Inteligent Information System Society Conference
    • /
    • 2007.05a
    • /
    • pp.218-225
    • /
    • 2007
  • Recommendation system is one way of implementing personalized service. The collaborative filtering is one of the major techniques that have been employed for recommendation systems. It has proven its effectiveness in the recommendation systems for such domain as motion picture or music. However, it has some limitations, i.e., sparsity and scalability. In this research, as one way of overcoming such limitations, we proposed the stepwise collaborative filtering method. To show the practicality of our proposed method, we designed and implemented a movie recommendation system which we shall call Step_CF, and its performance was evaluated using MovieLens data. The performance of Step_CF was better than that of Basic_CF that was implemented using the original collaborative filtering method.

  • PDF

Using Fuzzy Rating Information for Collaborative Filtering-based Recommender Systems

  • Lee, Soojung
    • International journal of advanced smart convergence
    • /
    • v.9 no.3
    • /
    • pp.42-48
    • /
    • 2020
  • These days people are overwhelmed by information on the Internet thus searching for useful information becomes burdensome, often failing to acquire some in a reasonable time. Recommender systems are indispensable to fulfill such user needs through many practical commercial sites. This study proposes a novel similarity measure for user-based collaborative filtering which is a most popular technique for recommender systems. Compared to existing similarity measures, the main advantages of the suggested measure are that it takes all the ratings given by users into account for computing similarity, thus relieving the inherent data sparsity problem and that it reflects the uncertainty or vagueness of user ratings through fuzzy logic. Performance of the proposed measure is examined by conducting extensive experiments. It is found that it demonstrates superiority over previous relevant measures in terms of major quality metrics.

Quantile Regression with Non-Convex Penalty on High-Dimensions

  • Choi, Ho-Sik;Kim, Yong-Dai;Han, Sang-Tae;Kang, Hyun-Cheol
    • Communications for Statistical Applications and Methods
    • /
    • v.16 no.1
    • /
    • pp.209-215
    • /
    • 2009
  • In regression problem, the SCAD estimator proposed by Fan and Li (2001), has many desirable property such as continuity, sparsity and unbiasedness. In this paper, we extend SCAD penalized regression framework to quantile regression and hence, we propose new SCAD penalized quantile estimator on high-dimensions and also present an efficient algorithm. From the simulation and real data set, the proposed estimator performs better than quantile regression estimator with $L_1$ norm.

Collaborative Recommendations using Adjusted Product Hierarchy : Methodology and Evaluation (재구성된 제품 계층도를 이용한 협업 추천 방법론 및 그 평가)

  • Cho, Yoon-Ho;Park, Su-Kyung;Ahn, Do-Hyun;Kim, Jae-Kyeong
    • Journal of the Korean Operations Research and Management Science Society
    • /
    • v.29 no.2
    • /
    • pp.59-75
    • /
    • 2004
  • Recommendation is a personalized information filtering technology to help customers find which products they would like to purchase. Collaborative filtering works by matching customer preferences to other customers in making recommendations. But collaborative filtering based recommendations have two major limitations, sparsity and scalability. To overcome these problems we suggest using adjusted product hierarchy, grain. This methodology focuses on dimensionality reduction and uses a marketer's specific knowledge or experience to improve recommendation quality. The qualify of recommendations using each grain is compared with others by several experimentations. Experiments present that the usage of a grain holds the promise of allowing CF-based recommendations to scale to large data sets and at the same time produces better recommendations. In addition. our methodology is proved to save the computation time by 3∼4 times compared with collaborative filtering.

압축센싱 기반의 무선통신 시스템

  • Reu, Na-Tan;Sin, Yo-An
    • The Magazine of the IEIE
    • /
    • v.38 no.1
    • /
    • pp.56-67
    • /
    • 2011
  • As a result of quickly growing data, a digital transmission system is required to deal with the challenge of acquiring signals at a very high sampling rate, Fortunately, the CS (Compressed Sensing or Compressive Sensing) theory, a new concept based on theoretical results of signal reconstruction, can be employed to exploit the sparsity of the received signals. Then, they can be adequately reconstructed from a set of their random projections, leading to dramatic reduction in the sampling rate and in the use of ADC (Analog-to-Digital Converter) resources. The goal of this article is provide an overview of the basic CS theory and to survey some important compressed sensing applications in wireless communications.

  • PDF

Improving Performance of Jaccard Coefficient for Collaborative Filtering

  • Lee, Soojung
    • Journal of the Korea Society of Computer and Information
    • /
    • v.21 no.11
    • /
    • pp.121-126
    • /
    • 2016
  • In recommender systems based on collaborative filtering, measuring similarity is very critical for determining the range of recommenders. Data sparsity problem is fundamental in collaborative filtering systems, which is partly solved by Jaccard coefficient combined with traditional similarity measures. This study proposes a new coefficient for improving performance of Jaccard coefficient by compensating for its drawbacks. We conducted experiments using datasets of various characteristics for performance analysis. As a result of comparison between the proposed and the similarity metric of Pearson correlation widely used up to date, it is found that the two metrics yielded competitive performance on a dense dataset while the proposed showed much better performance on a sparser dataset. Also, the result of comparing the proposed with Jaccard coefficient showed that the proposed yielded far better performance as the dataset is denser. Overall, the proposed coefficient demonstrated the best prediction and recommendation performance among the experimented metrics.