• 제목/요약/키워드: data reduction

검색결과 6,192건 처리시간 0.037초

Data Reduction Method in Massive Data Sets

  • Namo, Gecynth Torre;Yun, Hong-Won
    • Journal of information and communication convergence engineering
    • /
    • 제7권1호
    • /
    • pp.35-40
    • /
    • 2009
  • Many researchers strive to research on ways on how to improve the performance of RFID system and many papers were written to solve one of the major drawbacks of potent technology related with data management. As RFID system captures billions of data, problems arising from dirty data and large volume of data causes uproar in the RFID community those researchers are finding ways on how to address this issue. Especially, effective data management is important to manage large volume of data. Data reduction techniques in attempts to address the issues on data are also presented in this paper. This paper introduces readers to a new data reduction algorithm that might be an alternative to reduce data in RFID Systems. A process on how to extract data from the reduced database is also presented. Performance study is conducted to analyze the new data reduction algorithm. Our performance analysis shows the utility and feasibility of our categorization reduction algorithms.

A Classification Method Using Data Reduction

  • Uhm, Daiho;Jun, Sung-Hae;Lee, Seung-Joo
    • International Journal of Fuzzy Logic and Intelligent Systems
    • /
    • 제12권1호
    • /
    • pp.1-5
    • /
    • 2012
  • Data reduction has been used widely in data mining for convenient analysis. Principal component analysis (PCA) and factor analysis (FA) methods are popular techniques. The PCA and FA reduce the number of variables to avoid the curse of dimensionality. The curse of dimensionality is to increase the computing time exponentially in proportion to the number of variables. So, many methods have been published for dimension reduction. Also, data augmentation is another approach to analyze data efficiently. Support vector machine (SVM) algorithm is a representative technique for dimension augmentation. The SVM maps original data to a feature space with high dimension to get the optimal decision plane. Both data reduction and augmentation have been used to solve diverse problems in data analysis. In this paper, we compare the strengths and weaknesses of dimension reduction and augmentation for classification and propose a classification method using data reduction for classification. We will carry out experiments for comparative studies to verify the performance of this research.

DEVELOPMENT STATUS OF THE DOTIFS DATA SIMULATOR AND THE REDUCTION PACKAGE

  • CHUNG, HAEUN;RAMAPRAKASH, A.N.;PARK, CHANGBOM
    • 천문학논총
    • /
    • 제30권2호
    • /
    • pp.675-677
    • /
    • 2015
  • A data simulator and reduction package for the Devasthal Optical Telescope Integral Field Spectrograph (DOTIFS) has been developed. Since data reduction for the Integral Field Spectrograph (IFS) requires complicated procedures due to the complex nature of the integral spectrograph, common reduction procedures are usually not directly applicable for such an instrument. Therefore, the development of an optimized package for the DOTIFS is required. The data simulator observes artificial object and simulates CCD images for the instrument considering various effects; e.g. atmosphere, sky background, transmission, spectrograph optics aberration, and detector noise. The data reduction package has been developed based on the outcomes from the DOTIFS data simulator. The reduction package includes the entire processes for the reduction; pre-processing, flat-fielding, and sky subtraction. It generates 3D data cubes as a final product, which users can use for science directly.

Data Visualization using Linear and Non-linear Dimensionality Reduction Methods

  • Kim, Junsuk;Youn, Joosang
    • 한국컴퓨터정보학회논문지
    • /
    • 제23권12호
    • /
    • pp.21-26
    • /
    • 2018
  • As the large amount of data can be efficiently stored, the methods extracting meaningful features from big data has become important. Especially, the techniques of converting high- to low-dimensional data are crucial for the 'Data visualization'. In this study, principal component analysis (PCA; linear dimensionality reduction technique) and Isomap (non-linear dimensionality reduction technique) are introduced and applied to neural big data obtained by the functional magnetic resonance imaging (fMRI). First, we investigate how much the physical properties of stimuli are maintained after the dimensionality reduction processes. We moreover compared the amount of residual variance to quantitatively compare the amount of information that was not explained. As result, the dimensionality reduction using Isomap contains more information than the principal component analysis. Our results demonstrate that it is necessary to consider not only linear but also nonlinear characteristics in the big data analysis.

Applications of response dimension reduction in large p-small n problems

  • Minjee Kim;Jae Keun Yoo
    • Communications for Statistical Applications and Methods
    • /
    • 제31권2호
    • /
    • pp.191-202
    • /
    • 2024
  • The goal of this paper is to show how multivariate regression analysis with high-dimensional responses is facilitated by the response dimension reduction. Multivariate regression, characterized by multi-dimensional response variables, is increasingly prevalent across diverse fields such as repeated measures, longitudinal studies, and functional data analysis. One of the key challenges in analyzing such data is managing the response dimensions, which can complicate the analysis due to an exponential increase in the number of parameters. Although response dimension reduction methods are developed, there is no practically useful illustration for various types of data such as so-called large p-small n data. This paper aims to fill this gap by showcasing how response dimension reduction can enhance the analysis of high-dimensional response data, thereby providing significant assistance to statistical practitioners and contributing to advancements in multiple scientific domains.

A FAST REDUCTION METHOD OF SURVEY DATA IN RADIO ASTRONOMY

  • LEE YOUNGUNG
    • 천문학회지
    • /
    • 제34권1호
    • /
    • pp.1-8
    • /
    • 2001
  • We present a fast reduction method of survey data obtained using a single-dish radio telescope. Along with a brief review of classical method, a new method of identification and elimination of negative and positive bad channels are introduced using cloud identification code and several IRAF (Image Reduction and Analysis Facility) tasks relating statistics. Removing of several ripple patterns using Fourier Transform is also discussed. It is found that BACKGROUND task within IRAF is very efficient for fitting and subtraction of base-line with varying functions. Cloud identification method along with the possibility of its application for analysis of cloud structure is described, and future data reduction method is discussed.

  • PDF

개념 변동 고차원 스트리밍 데이터에 대한 차원 감소 방법 (Dimension Reduction Methods on High Dimensional Streaming Data with Concept Drift)

  • 박정희
    • 정보처리학회논문지:소프트웨어 및 데이터공학
    • /
    • 제5권8호
    • /
    • pp.361-368
    • /
    • 2016
  • 고차원데이터에 대한 차원 감소 기법들은 많이 연구되어져 온 반면, 개념 변동을 가진 고차원 스트리밍 데이터에서 적용할 수 있는 차원 감소 기법에 대한 연구는 제한적이다. 이 논문에서는 스트리밍 데이터에서 적용할 수 있는 점층적 차원 감소 기법들을 살펴보고, 개념 변동 고차원 스트리밍 데이터에 대해 분류 성능을 향상시킬 수 있도록 차원 감소를 효과적으로 적용하는 방법을 제안한다.