• Title/Summary/Keyword: redundant processing

Search Result 202, Processing Time 0.026 seconds

An Efficient Angular Space Partitioning Based Skyline Query Processing Using Sampling-Based Pruning (데이터 샘플링 기반 프루닝 기법을 도입한 효율적인 각도 기반 공간 분할 병렬 스카이라인 질의 처리 기법)

  • Choi, Woosung;Kim, Minseok;Diana, Gromyko;Chung, Jaehwa;Jung, Soonyong
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.6 no.1
    • /
    • pp.1-8
    • /
    • 2017
  • Given a multi-dimensional dataset of tuples, a skyline query returns a subset of tuples which are not 'dominated' by any other tuples. Skyline query is very useful in Big data analysis since it filters out uninteresting items. Much interest was devoted to the MapReduce-based parallel processing of skyline queries in large-scale distributed environment. There are three requirements to improve parallelism in MapReduced-based algorithms: (1) workload should be well balanced (2) avoid redundant computations (3) Optimize network communication cost. In this paper, we introduce MR-SEAP (MapReduce sample Skyline object Equality Angular Partitioning), an efficient angular space partitioning based skyline query processing using sampling-based pruning, which satisfies requirements above. We conduct an extensive experiment to evaluate MR-SEAP.

Convolutional Neural Network with Expert Knowledge for Hyperspectral Remote Sensing Imagery Classification

  • Wu, Chunming;Wang, Meng;Gao, Lang;Song, Weijing;Tian, Tian;Choo, Kim-Kwang Raymond
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.13 no.8
    • /
    • pp.3917-3941
    • /
    • 2019
  • The recent interest in artificial intelligence and machine learning has partly contributed to an interest in the use of such approaches for hyperspectral remote sensing (HRS) imagery classification, as evidenced by the increasing number of deep framework with deep convolutional neural networks (CNN) structures proposed in the literature. In these approaches, the assumption of obtaining high quality deep features by using CNN is not always easy and efficient because of the complex data distribution and the limited sample size. In this paper, conventional handcrafted learning-based multi features based on expert knowledge are introduced as the input of a special designed CNN to improve the pixel description and classification performance of HRS imagery. The introduction of these handcrafted features can reduce the complexity of the original HRS data and reduce the sample requirements by eliminating redundant information and improving the starting point of deep feature training. It also provides some concise and effective features that are not readily available from direct training with CNN. Evaluations using three public HRS datasets demonstrate the utility of our proposed method in HRS classification.

An Efficient Graph Algorithm Processing Scheme using GPUs with Limited Memory (제한된 메모리를 가진 GPU를 이용한 효율적인 그래프 알고리즘 처리 기법)

  • Song, Sang-ho;Lee, Hyeon-byeong;Choi, Do-jin;Lim, Jong-tae;Bok, Kyoung-soo;Yoo, Jae-soo
    • The Journal of the Korea Contents Association
    • /
    • v.22 no.8
    • /
    • pp.81-93
    • /
    • 2022
  • Recently, research on processing a large-capacity graph using GPUs has been conducting. In order to process a large-capacity graph in a GPU with limited memory, the graph must be divided into subgraphs and then processed by scheduling subgraphs. In this paper, we propose an efficient graph algorithm processing scheme in GPU environments with limited memory and performance evaluation. The proposed scheme consists of a graph differential subgraph scheduling method and a graph segmentation method. The bulk graph segmentation method determines how a large-capacity graph can be segmented into subgraphs so that it can be processed efficiently by the GPU. The differential subgraph scheduling method schedule subgraphs processed by GPUs to reduce redundant transmission of the repeatedly used data between HOST-GPUs. It shows the superiority of the proposed scheme by performing various performance evaluations.

Efficient Processing of Aggregate Queries in Wireless Sensor Networks (무선 센서 네트워크에서 효율적인 집계 질의 처리)

  • Kim, Joung-Joon;Shin, In-Su;Lee, Ki-Young;Han, Ki-Joon
    • Spatial Information Research
    • /
    • v.19 no.3
    • /
    • pp.95-106
    • /
    • 2011
  • Recently as efficient processing of aggregate queries for fetching desired data from sensors has been recognized as a crucial part, in-network aggregate query processing techniques are studied intensively in wireless sensor networks. Existing representative in-network aggregate query processing techniques propose routing algorithms and data structures for processing aggregate queries. However, these aggregate query processing techniques have problems such as high energy consumption in sensor nodes, low accuracy of query processing results, and long query processing time. In order to solve these problems and to enhance the efficiency of aggregate query processing in wireless sensor networks, this paper proposes Bucket-based Parallel Aggregation(BPA). BPA divides a query region into several cells according to the distribution of sensor nodes and builds a Quad-tree, and then processes aggregate queries in parallel for each cell region according to routing. And it sends data in duplicate by removing redundant data, which, in turn, enhances the accuracy of query processing results. Also, BPA uses a bucket-based data structure in aggregate query processing, and divides and conquers the bucket data structure adaptively according to the number of data in the bucket. In addition, BPA compresses data in order to reduce the size of data in the bucket and performs data transmission filtering when each sensor node sends data. Finally, in this paper, we prove its superiority through various experiments using sensor data.

Subsequence Matching Under Time Warping in Time-Series Databases : Observation, Optimization, and Performance Results (시계열 데이터베이스에서 타임 워핑 하의 서브시퀀스 매칭 : 관찰, 최적화, 성능 결과)

  • Kim Man-Soon;Kim Sang-Wook
    • The KIPS Transactions:PartD
    • /
    • v.11D no.7 s.96
    • /
    • pp.1385-1398
    • /
    • 2004
  • This paper discusses an effective processing of subsequence matching under time warping in time-series databases. Time warping is a trans-formation that enables finding of sequences with similar patterns even when they are of different lengths. Through a preliminary experiment, we first point out that the performance bottleneck of Naive-Scan, a basic method for processing of subsequence matching under time warping, is on the CPU processing step. Then, we propose a novel method that optimizes the CPU processing step of Naive-Scan. The proposed method maximizes the CPU performance by eliminating all the redundant calculations occurring in computing the time warping distance between the query sequence and data subsequences. We formally prove the proposed method does not incur false dismissals and also is the optimal one for processing Naive-Scan. Also, we discuss the we discuss to apply the proposed method to the post-processing step of LB-Scan and ST-Filter, the previous methods for processing of subsequence matching under time warping. Then, we quantitatively verify the performance improvement ef-fects obtained by the proposed method via extensive experiments. The result shows that the performance of all the three previous methods im-proves by employing the proposed method. Especially, Naive-Scan, which is known to show the worst performance, performs much better than LB-Scan as well as ST-Filter in all cases when it employs the proposed method for CPU processing. This result is so meaningful in that the performance inversion among Nive- Scan, LB-Scan, and ST-Filter has occurred by optimizing the CPU processing step, which is their perform-ance bottleneck.

A Study on Enhancement of Digital Image Performance Using Dual Tree Wavelet Transformation in Non-separable Image Processing (비분리 영상처리에서 이중 트리 웨이브렛 변환을 사용한 디지털 영상 성능 개선에 관한 연구)

  • Lim, Joong-Hee;Jee, Inn-Ho
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.12 no.1
    • /
    • pp.65-74
    • /
    • 2012
  • In this paper, we explore the application of 2-D dual-tree discrete wavelet transform (DDWT), which is a directional and redundant transform, for image coding. DDWT introduces limited redundancy and allows the transform to provide approximate shift invariance and directionally selective filters while preserving the usual properties of perfect reconstruction and computational efficiency with good well-balanced frequency responses. Also, quincunx lattice yields a non separable 2D-wavelet transform, which is also symmetric in both horizontal and vertical direction. And non-separable wavelet transformation can generate sub-images of multiple degrees rotated versions. The proposed 2-D non-separable DDWT can provide efficient approximation for directional features of images schemes, such as edges and contours in images that are not aligned with the horizontal or vertical direction. Finally, non-separable image processing using DDWT services good performance.

A Study on Data Pre-filtering Methods for Fault Diagnosis (시스템 결함원인분석을 위한 데이터 로그 전처리 기법 연구)

  • Lee, Yang-Ji;Kim, Duck-Young;Hwang, Min-Soon;Cheong, Young-Soo
    • Korean Journal of Computational Design and Engineering
    • /
    • v.17 no.2
    • /
    • pp.97-110
    • /
    • 2012
  • High performance sensors and modern data logging technology with real-time telemetry facilitate system fault diagnosis in a very precise manner. Fault detection, isolation and identification in fault diagnosis systems are typical steps to analyze the root cause of failures. This systematic failure analysis provides not only useful clues to rectify the abnormal behaviors of a system, but also key information to redesign the current system for retrofit. The main barriers to effective failure analysis are: (i) the gathered data (event) logs are too large in general, and further (ii) they usually contain noise and redundant data that make precise analysis difficult. This paper therefore applies suitable pre-processing techniques to data reduction and feature extraction, and then converts the reduced data log into a new format of event sequence information. Finally the event sequence information is decoded to investigate the correlation between specific event patterns and various system faults. The efficiency of the developed pre-filtering procedure is examined with a terminal box data log of a marine diesel engine.

Modal parameter identification with compressed samples by sparse decomposition using the free vibration function as dictionary

  • Kang, Jie;Duan, Zhongdong
    • Smart Structures and Systems
    • /
    • v.25 no.2
    • /
    • pp.123-133
    • /
    • 2020
  • Compressive sensing (CS) is a newly developed data acquisition and processing technique that takes advantage of the sparse structure in signals. Normally signals in their primitive space or format are reconstructed from their compressed measurements for further treatments, such as modal analysis for vibration data. This approach causes problems such as leakage, loss of fidelity, etc., and the computation of reconstruction itself is costly as well. Therefore, it is appealing to directly work on the compressed data without prior reconstruction of the original data. In this paper, a direct approach for modal analysis of damped systems is proposed by decomposing the compressed measurements with an appropriate dictionary. The damped free vibration function is adopted to form atoms in the dictionary for the following sparse decomposition. Compared with the normally used Fourier bases, the damped free vibration function spans a space with both the frequency and damping as the control variables. In order to efficiently search the enormous two-dimension dictionary with frequency and damping as variables, a two-step strategy is implemented combined with the Orthogonal Matching Pursuit (OMP) to determine the optimal atom in the dictionary, which greatly reduces the computation of the sparse decomposition. The performance of the proposed method is demonstrated by a numerical and an experimental example, and advantages of the method are revealed by comparison with another such kind method using POD technique.

A Study on the Design of Network System for Defense Integrated Data Center Using NFV/SDN (NFV/SDN을 활용한 군(軍) 데이터센터 네트워크 체계 설계에 관한 연구)

  • Chae, Woong;Kwon, Taewook
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.9 no.2
    • /
    • pp.31-36
    • /
    • 2020
  • The creation of the Defense Integrated Data Center(DIDC) has resulted in a reduction in manpower, operating costs, efficient and effective management of resources. However, it is difficult to effectively collect and manage the data of a large number of battlefields coming from equipments such as drones, robots, and IoT added to the fourth industrial revolution and the future battlefield. Therefore, we will propose the design of DIDC network system using NFV and SDN, which are emerging as the core technologies of 5G, a mobile communication technology. After analyzing the data sheet of each equipment, it is considered that by integrating the redundant functions, energy efficiency, resource utilization and effective network management will be possible.

A Study on the Hierarchical Representation of Images: An Efficient Representation of Quadtrees BF Linear Quadtree (화상의 구조적 표현에 관한 연구- 4진트리의 효율적인 표현법:BF선형 4진트)

  • Kim, Min-Hwan;Han, Sang-Ho;Hwang, Hee-Yeung
    • The Transactions of the Korean Institute of Electrical Engineers
    • /
    • v.37 no.7
    • /
    • pp.498-509
    • /
    • 1988
  • A BF(breadth-first) linear quadtree as a new data structure for image data is suggested, which enables us to compress the image data efficiently and to make operations of the compressed data easily. It is a list of path names for black nodes as the linear quadtree is. The path name for each black node of a BF linear quadtree is represented as a sequence of path codes from the root node to itself, whereas that of linear quadtree as a sequence of path codes from the root node to itself and fill characters for cut-offed path from it to any n-level node which corresponds to a pixel of an image. The BF linear quadtree provides a more efficent compression ratio than the linear quadtree does, because the former does not require redundant characters, fill characters, for the cut-offed paths. Several operations for image processing can be also implemented on this hierarchical structure efficiently, because it is composed of only the black nodes ad the linear quadtree is . In this paper, algorithms for several operations on the BF linear quadtree are defined and analyzed. Experimental results for forur image data are also given and discussed.