• Title/Summary/Keyword: Data sparsity

Search Result 174, Processing Time 0.029 seconds

Optimization of the Similarity Measure for User-based Collaborative Filtering Systems (사용자 기반의 협력필터링 시스템을 위한 유사도 측정의 최적화)

  • Lee, Soojung
    • The Journal of Korean Association of Computer Education
    • /
    • v.19 no.1
    • /
    • pp.111-118
    • /
    • 2016
  • Measuring similarity in collaborative filtering-based recommender systems greatly affects system performance. This is because items are recommended from other similar users. In order to overcome the biggest problem of traditional similarity measures, i.e., data sparsity problem, this study suggests a new similarity measure that is the optimal combination of previous similarity and the value reflecting the number of co-rated items. We conducted experiments with various conditions to evaluate performance of the proposed measure. As a result, the proposed measure yielded much better performance than previous ones in terms of prediction qualities, specifically the maximum of about 7% improvement over the traditional Pearson correlation and about 4% over the cosine similarity.

DiLO: Direct light detection and ranging odometry based on spherical range images for autonomous driving

  • Han, Seung-Jun;Kang, Jungyu;Min, Kyoung-Wook;Choi, Jungdan
    • ETRI Journal
    • /
    • v.43 no.4
    • /
    • pp.603-616
    • /
    • 2021
  • Over the last few years, autonomous vehicles have progressed very rapidly. The odometry technique that estimates displacement from consecutive sensor inputs is an essential technique for autonomous driving. In this article, we propose a fast, robust, and accurate odometry technique. The proposed technique is light detection and ranging (LiDAR)-based direct odometry, which uses a spherical range image (SRI) that projects a three-dimensional point cloud onto a two-dimensional spherical image plane. Direct odometry is developed in a vision-based method, and a fast execution speed can be expected. However, applying LiDAR data is difficult because of the sparsity. To solve this problem, we propose an SRI generation method and mathematical analysis, two key point sampling methods using SRI to increase precision and robustness, and a fast optimization method. The proposed technique was tested with the KITTI dataset and real environments. Evaluation results yielded a translation error of 0.69%, a rotation error of 0.0031°/m in the KITTI training dataset, and an execution time of 17 ms. The results demonstrated high precision comparable with state-of-the-art and remarkably higher speed than conventional techniques.

A Hybrid Recommendation System based on Fuzzy C-Means Clustering and Supervised Learning

  • Duan, Li;Wang, Weiping;Han, Baijing
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.15 no.7
    • /
    • pp.2399-2413
    • /
    • 2021
  • A recommendation system is an information filter tool, which uses the ratings and reviews of users to generate a personalized recommendation service for users. However, the cold-start problem of users and items is still a major research hotspot on service recommendations. To address this challenge, this paper proposes a high-efficient hybrid recommendation system based on Fuzzy C-Means (FCM) clustering and supervised learning models. The proposed recommendation method includes two aspects: on the one hand, FCM clustering technique has been applied to the item-based collaborative filtering framework to solve the cold start problem; on the other hand, the content information is integrated into the collaborative filtering. The algorithm constructs the user and item membership degree feature vector, and adopts the data representation form of the scoring matrix to the supervised learning algorithm, as well as by combining the subjective membership degree feature vector and the objective membership degree feature vector in a linear combination, the prediction accuracy is significantly improved on the public datasets with different sparsity. The efficiency of the proposed system is illustrated by conducting several experiments on MovieLens dataset.

Confidence intervals for the COVID-19 neutralizing antibody retention rate in the Korean population

  • Apio, Catherine;Kamruzzaman, Md.;Park, Taesung
    • Genomics & Informatics
    • /
    • v.18 no.3
    • /
    • pp.31.1-31.8
    • /
    • 2020
  • The coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has become a global pandemic. No specific therapeutic agents or vaccines for COVID-19 are available, though several antiviral drugs, are under investigation as treatment agents for COVID-19. The use of convalescent plasma transfusion that contain neutralizing antibodies for COVID-19 has become the major focus. This requires mass screening of populations for these antibodies. While several countries started reporting population based antibody rate, its simple point estimate may be misinterpreted without proper estimation of standard error and confidence intervals. In this paper, we review the importance of antibody studies and present the 95% confidence intervals COVID-19 antibody rate for the Korean population using two recently performed antibody tests in Korea. Due to the sparsity of data, the estimation of confidence interval is a big challenge. Thus, we consider several confidence intervals using Asymptotic, Exact and Bayesian estimation methods. In this article, we found that the Wald method gives the narrowest interval among all Asymptotic methods whereas mid p-value gives the narrowest among all Exact methods and Jeffrey's method gives the narrowest from Bayesian method. The most conservative 95% confidence interval estimation shows that as of 00:00 on September 15, 2020, at least 32,602 people were infected but not confirmed in Korea.

Clustering-based Collaborative Filtering Using Genetic Algorithms (유전자 알고리즘을 이용한 클러스터링 기반 협력필터링)

  • Lee, Soojung
    • Journal of Creative Information Culture
    • /
    • v.4 no.3
    • /
    • pp.221-230
    • /
    • 2018
  • Collaborative filtering technique is a major method of recommender systems and has been successfully implemented and serviced in real commercial online systems. However, this technique has several inherent drawbacks, such as data sparsity, cold-start, and scalability problem. Clustering-based collaborative filtering has been studied in order to handle scalability problem. This study suggests a collaborative filtering system which utilizes genetic algorithms to improve shortcomings of K-means algorithm, one of the widely used clustering techniques. Moreover, different from the previous studies that have targeted for optimized clustering results, the proposed method targets the optimization of performance of the collaborative filtering system using the clustering results, which practically can enhance the system performance.

Resolving the 'Gray sheep' Problem Using Social Network Analysis (SNA) in Collaborative Filtering (CF) Recommender Systems (소셜 네트워크 분석 기법을 활용한 협업필터링의 특이취향 사용자(Gray Sheep) 문제 해결)

  • Kim, Minsung;Im, Il
    • Journal of Intelligence and Information Systems
    • /
    • v.20 no.2
    • /
    • pp.137-148
    • /
    • 2014
  • Recommender system has become one of the most important technologies in e-commerce in these days. The ultimate reason to shop online, for many consumers, is to reduce the efforts for information search and purchase. Recommender system is a key technology to serve these needs. Many of the past studies about recommender systems have been devoted to developing and improving recommendation algorithms and collaborative filtering (CF) is known to be the most successful one. Despite its success, however, CF has several shortcomings such as cold-start, sparsity, gray sheep problems. In order to be able to generate recommendations, ordinary CF algorithms require evaluations or preference information directly from users. For new users who do not have any evaluations or preference information, therefore, CF cannot come up with recommendations (Cold-star problem). As the numbers of products and customers increase, the scale of the data increases exponentially and most of the data cells are empty. This sparse dataset makes computation for recommendation extremely hard (Sparsity problem). Since CF is based on the assumption that there are groups of users sharing common preferences or tastes, CF becomes inaccurate if there are many users with rare and unique tastes (Gray sheep problem). This study proposes a new algorithm that utilizes Social Network Analysis (SNA) techniques to resolve the gray sheep problem. We utilize 'degree centrality' in SNA to identify users with unique preferences (gray sheep). Degree centrality in SNA refers to the number of direct links to and from a node. In a network of users who are connected through common preferences or tastes, those with unique tastes have fewer links to other users (nodes) and they are isolated from other users. Therefore, gray sheep can be identified by calculating degree centrality of each node. We divide the dataset into two, gray sheep and others, based on the degree centrality of the users. Then, different similarity measures and recommendation methods are applied to these two datasets. More detail algorithm is as follows: Step 1: Convert the initial data which is a two-mode network (user to item) into an one-mode network (user to user). Step 2: Calculate degree centrality of each node and separate those nodes having degree centrality values lower than the pre-set threshold. The threshold value is determined by simulations such that the accuracy of CF for the remaining dataset is maximized. Step 3: Ordinary CF algorithm is applied to the remaining dataset. Step 4: Since the separated dataset consist of users with unique tastes, an ordinary CF algorithm cannot generate recommendations for them. A 'popular item' method is used to generate recommendations for these users. The F measures of the two datasets are weighted by the numbers of nodes and summed to be used as the final performance metric. In order to test performance improvement by this new algorithm, an empirical study was conducted using a publically available dataset - the MovieLens data by GroupLens research team. We used 100,000 evaluations by 943 users on 1,682 movies. The proposed algorithm was compared with an ordinary CF algorithm utilizing 'Best-N-neighbors' and 'Cosine' similarity method. The empirical results show that F measure was improved about 11% on average when the proposed algorithm was used

    . Past studies to improve CF performance typically used additional information other than users' evaluations such as demographic data. Some studies applied SNA techniques as a new similarity metric. This study is novel in that it used SNA to separate dataset. This study shows that performance of CF can be improved, without any additional information, when SNA techniques are used as proposed. This study has several theoretical and practical implications. This study empirically shows that the characteristics of dataset can affect the performance of CF recommender systems. This helps researchers understand factors affecting performance of CF. This study also opens a door for future studies in the area of applying SNA to CF to analyze characteristics of dataset. In practice, this study provides guidelines to improve performance of CF recommender systems with a simple modification.

  • User and Item based Collaborative Filtering Using Classification Property Naive Bayesian (분류 속성과 Naive Bayesian을 이용한 사용자와 아이템 기반의 협력적 필터링)

    • Kim, Jong-Hun;Kim, Yong-Jip;Rim, Kee-Wook;Lee, Jung-Hyun;Chung, Kyung-Yong
      • The Journal of the Korea Contents Association
      • /
      • v.7 no.11
      • /
      • pp.23-33
      • /
      • 2007
    • The collaborative filtering has used the nearest neighborhood method based on the preference and the similarity using the Pearson correlation coefficient. Therefore, it does not reflect content of the items and has the problems of the sparsity and scalability as well. the item-based collaborative filtering has been practically used to improve these defects, but it still does not reflect attributes of the item. In this paper, we propose the user and item based collaborative filtering using the classification property and Naive Bayesian to supplement the defects in the existing recommendation system. The proposed method complexity refers to the item similarity based on explicit data and the user similarity based on implicit data for handing the sparse problem. It applies to the Naive Bayesian to the result of reference. Also, it can enhance the accuracy as computation of the item similarity reflects on the correlative rank among the classification property to reflect attributes.

    Personal Information Protection Recommendation System using Deep Learning in POI (POI 에서 딥러닝을 이용한 개인정보 보호 추천 시스템)

    • Peng, Sony;Park, Doo-Soon;Kim, Daeyoung;Yang, Yixuan;Lee, HyeJung;Siet, Sophort
      • Proceedings of the Korea Information Processing Society Conference
      • /
      • 2022.11a
      • /
      • pp.377-379
      • /
      • 2022
    • POI refers to the point of Interest in Location-Based Social Networks (LBSNs). With the rapid development of mobile devices, GPS, and the Web (web2.0 and 3.0), LBSNs have attracted many users to share their information, physical location (real-time location), and interesting places. The tremendous demand of the user in LBSNs leads the recommendation systems (RSs) to become more widespread attention. Recommendation systems assist users in discovering interesting local attractions or facilities and help social network service (SNS) providers based on user locations. Therefore, it plays a vital role in LBSNs, namely POI recommendation system. In the machine learning model, most of the training data are stored in the centralized data storage, so information that belongs to the user will store in the centralized storage, and users may face privacy issues. Moreover, sharing the information may have safety concerns because of uploading or sharing their real-time location with others through social network media. According to the privacy concern issue, the paper proposes a recommendation model to prevent user privacy and eliminate traditional RS problems such as cold-start and data sparsity.

    Improvement of Bandwidth Efficiency for High Transmission Capacity of Contents Streaming Data using Compressive Sensing Technique (컨텐츠 스트리밍 데이터의 전송효율 증대를 위한 압축센싱기반 전송채널 대역폭 절감기술 연구)

    • Jung, Eui-Suk;Lee, Yong-Tae;Han, Sang-Kook
      • Journal of the Korea Academia-Industrial cooperation Society
      • /
      • v.16 no.3
      • /
      • pp.2141-2145
      • /
      • 2015
    • A new broadcasting signal transmission, which can save its channel bandwidth using compressive sensing(CS), is proposed in this paper. A new compression technique, which uses two dimensional discrete wavelet transform technique, is proposed to get high sparsity of multimedia image. A L1 minimization technique based on orthogonal matching pursuit is also introduced in order to reconstruct the compressed multimedia image. The CS enables us to save the channel bandwidth of wired and wireless broadcasting signal because various transmitted data are compressed using it. A $256{\times}256$ gray-scale image with compression rato of 20 %, which is sampled by 10 Gs/s, was transmitted to an optical receiver through 20-km optical transmission and then was reconstructed successfully using L1 minimization (bit error rate of $10^{-12}$ at the received optical power of -12.2 dB).

    Analyzing the Research Fronts of Women's Studies in Korea Using Citation Image Makers Profiling (인용 이미지 구축자 프로파일링을 이용한 국내 여성학 분야 연구 전선 분석)

    • Kim, Jo-Ah;Lee, Jae Yun
      • Journal of the Korean Society for information Management
      • /
      • v.33 no.2
      • /
      • pp.201-225
      • /
      • 2016
    • A new technique for revealing the research fronts of a interdisciplinary discipline has been developed. Citation image makers profiling (CIMP) determines the relationships between research papers with the title words of the citing documents. We adapted this new technique to analyze the research fronts and hot topics in women's studies of Korea. By Korean Citation Index (KCI) data in 2015, we selected 148 papers cited more than 9 times as the core documents of women's studies. Analysis of intellectual structure using citation image makers profiling was performed with the 148 core documents and those citing papers. Document co-citation analysis was hindered by citation data sparsity, while CIMP method successfully revealed the structure of research fronts of Korean women's studies including 2 divisions and 6 subdivisions. The CIMP method suggested in this study has good potential to discover the characteristics of research fronts of interdisciplinary research domains.


    (34141) Korea Institute of Science and Technology Information, 245, Daehak-ro, Yuseong-gu, Daejeon
    Copyright (C) KISTI. All Rights Reserved.