• Title/Summary/Keyword: Stream Query Processing

Search Result 124, Processing Time 0.031 seconds

Query Indexing Technique for Processing Stream Data (스트림 데이터 처리를 위한 질의 색인 기법)

  • Lee, Dong-Gyu;Chung, Jae-Du;Lee, Yang-Koo;Jung, Young-Jin;Ryu, Keun-Ho
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2006.11a
    • /
    • pp.381-384
    • /
    • 2006
  • 센서 네트워크 환경에서 스트림 데이터를 모니터링 하기 위해서는 스트림 데이터에 대한 연속적인 질의들을 효과적으로 처리하는 것이 필요하다. 이러한 연속적인 질의를 빠르게 검색하고 처리하기 위하여 낮은 저장 비용과 빠른 탐색 성능을 가진 질의 색인 기법이 많이 활용되고 있다. 기존 연구들은 사전에 삽입될 Interval 을 알고 트리를 구성하므로 동적인 삽입, 삭제가 불가능하거나 삽입된 Interval 수와 Interval 의 범위에 따라 높은 저장 비용이나 상대적으로 느린 탐색 속도를 보인다. 따라서 이 논문에서는 연속적인 질의 처리를 효율적으로 하기 위하여 Hashed Multiple Lists 를 제안한다. 제안된 기법은 빠른 선형 탐색 성능과 낮은 저장 비용을 요구하며 삽입, 삭제가 용이하고 다양한 범위를 표현할 수 있는 장점이 있다. 제안된 색인 기법은 센서 네트워크를 응용한 시스템과 상황 인식 시스템 등에서 연속적인 질의를 처리하는데 활용할 수 있다.

  • PDF

Efficient Processing of an Aggregate Query Stream in MapReduce (맵리듀스에서 집계 질의 스트림의 효율적인 처리 기법)

  • Choi, Hyunjean;Lee, Ki Yong
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2013.11a
    • /
    • pp.1207-1210
    • /
    • 2013
  • 최근 들어 맵리듀스는 빅데이터 처리의 표준 기술로 자리잡고 있다. 빅데이터 분석에 널리 쓰이는 질의 중 하나는 집계(aggregate) 질의이다. 본 논문에서는 서로 다른 집계 질의가 계속적으로 요청되는 환경에서, 맵리듀스를 사용하여 이들 질의를 효율적으로 처리하는 방법을 제안한다. 제안 방법은 여러 집계 질의를 하나의 효율적인 맵리듀스 잡(job)으로 묶어 일괄 처리함으로써, 단순 방법에 비해 시간당 처리되는 질의 수를 크게 증가시킨다. 성능 평가를 통해, 제안 방법은 단순 방법에 비해 처리 성능을 크게 향상시킴을 확인하였다.

Fast Twig Query Processing for Streaming XML Data (스트리밍 XML 데이터에 대한 빠른 트윅 질의 처리 기법)

  • Ryu, Byung-Gul;Park, Sang-Hyun;Ha, Jong-Woo;Lee, SangKeun
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2010.11a
    • /
    • pp.65-68
    • /
    • 2010
  • 스트리밍 XML 데이터는 고정된 저장소에 유지되지 않고 사용자 측으로 계속적으로 데이터가 전송된다는 특성을 지닌다. 이러한 스트리밍 XML에 대한 질의 처리를 위해서는 효과적인 메모리 관리와 빠른 질의 처리 성능이 요구된다. 최근 최소한의 메모리 사용으로 효과적으로 트윅 질의를 처리하기 위한 기법인 StreamTX가 제안되었으나 반복적인 질의 처리 알고리즘 호출로 인해 불필요한 질의 처리시간이 발생한다. 따라서, 본 논문에서는 이러한 불필요한 질의 처리 시간을 줄이기 위해 실시간으로 질의와 무관한 노드를 제거하여 보다 효과적인 질의 처리를 수행 기법을 제안한다. 제안된 기법은 기존 연구와 유사한 메모리 사용량을 가지면서도 빠른 질의 처리 속도를 가짐을 성능평가를 통해 검증한다.

Linear Resource Sharing Method for Query Optimization of Sliding Window Aggregates in Multiple Continuous Queries (다중 연속질의에서 슬라이딩 윈도우 집계질의 최적화를 위한 선형 자원공유 기법)

  • Baek, Seong-Ha;You, Byeong-Seob;Cho, Sook-Kyoung;Bae, Hae-Young
    • Journal of KIISE:Databases
    • /
    • v.33 no.6
    • /
    • pp.563-577
    • /
    • 2006
  • A stream processor uses resource sharing method for efficient of limited resource in multiple continuous queries. The previous methods process aggregate queries to consist the level structure. So insert operation needs to reconstruct cost of the level structure. Also a search operation needs to search cost of aggregation information in each size of sliding windows. Therefore this paper uses linear structure for optimization of sliding window aggregations. The method comprises of making decision, generation and deletion of panes in sequence. The decision phase determines optimum pane size for holding accurate aggregate information. The generation phase stores aggregate information of data per pane from stream buffer. At the deletion phase, panes are deleted that are no longer used. The proposed method uses resources less than the method where level structures were used as data structures as it uses linear data format. The input cost of aggregate information is saved by calculating only pane size of data though numerous stream data is arrived, and the search cost of aggregate information is also saved by linear searching though those sliding window size is different each other. In experiment, the proposed method has low usage of memory and the speed of query processing is increased.

The XP-table: Runtime-efficient Region-based Structure for Collective Evaluation of Multiple Continuous XPath Queries (The XP-table: 다중 연속 XPath 질의의 집단 처리를 위한 실행시간 효율적인 영역 기반 구조체)

  • Lee, Hyun-Ho;Lee, Won-Suk
    • Journal of KIISE:Databases
    • /
    • v.35 no.4
    • /
    • pp.307-318
    • /
    • 2008
  • One of the primary issues confronting XML message brokers is the difficulty associated with processing a large set of continuous XPath queries over incoming XML seams. This paper proposes a novel system designed to present an effective solution to this problem. The proposed system transforms multiple XPath queries before their run-time into a new region-based data structure, called an XP-table, by sharing their common constraints. An XP-table is matched with a stream relation (SR) transformed from a target XML stream by a SAX parser. This arrangement is intended to minimize the runtime workload of continuous query processing. Also, system performance is estimated and verified through a variety of experiments, including comparisons with previous approaches such as YFilter and LazyDFA. The proposed system is practically linear- scalable and stable for evaluating a set of XPath queries in a continuous and timely fashion.

Content-Based Video Search Using Eigen Component Analysis and Intensity Component Flow (고유성분 분석과 휘도성분 흐름 특성을 이용한 내용기반 비디오 검색)

  • 전대홍;강대성
    • Journal of the Institute of Convergence Signal Processing
    • /
    • v.3 no.3
    • /
    • pp.47-53
    • /
    • 2002
  • In this paper, we proposed a content-based video search method using the eigen value of key frame and intensity component. We divided the video stream into shot units to extract key frame representing each shot, and get the intensity distribution of the shot from the database generated by using ECA(Eigen Component Analysis). The generated codebook, their index value for each key frame, and the intensity values were used for database. The query image is utilized to find video stream that has the most similar frame by using the euclidean distance measure among the codewords in the codebook. The experimental results showed that the proposed algorithm is superior to any other methols in the search outcome since it makes use of eigen value and intensity elements, and reduces the processing time etc.

  • PDF

Query Optimization for Keyword Search on Relational Data Stream (관계형 데이터 스트림에서 키워드 검색을 위한 질의 최적화)

  • Jin-Ho Hwang;Hak Soo Kim;Jhong-Jin Kim;Seung Mi Lee;Jin Hyun Son
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2008.11a
    • /
    • pp.360-363
    • /
    • 2008
  • 최근 관계형 데이터 스트림에서 키워드 검색에 관한 연구가 진행되고 있다. 키워드 검색을 통해 사용자는 시스템의 복잡한 내부 데이터 스키마나 질의언어에 대한 지식이 없이도 데이터 스트림에서 정보 검색이 가능하다. 하지만, 빈번하고 동적으로 변화하는 특성을 지닌 데이터 스트림에서 수행되는 연속 질의 처리를 위해서 보다 효과적인 질의 최적화 방안이 요구된다. 따라서, 우리는 본 논문을 통해 계층적 클러스터링을 이용하여 중간결과 공유의 최대화를 통한 질의 최적화를 방안을 제안한다.

A Query Index for Processing Continuous Queries over RFID Tag Data (RFID 태그 데이타의 연속질의 처리를 위한 질의 색인)

  • Seok, Su-Wook;Park, Jae-Kwan;Hong, Bong-Hee
    • Journal of KIISE:Databases
    • /
    • v.34 no.2
    • /
    • pp.166-178
    • /
    • 2007
  • The ALE specification of EPCglobal is leading the development of RFID standards, includes the Event Cycle Specification (ECSpec) describing how long a cycle is, how to filter RFID tag data and which reader is interested in. The ECSpec is a specification for filtering and collecting RFID tag data. It is registered to a middleware for long time and is evaluated to return results satisfying the requirements included in it. Thus, it is quite similar to the continuous query. It can be transformed into a continuous query as its predicate in WHERE clause is characterized by the long interval. Long intervals cause problems deteriorating insertion and search performance of existing query indices. In this paper, we propose a TLC-index as a new query index structure for long interval data. The TLC-index has hybrid structure that uses the cell construct of CQI-index with the virtual construct of VCR-index for partitioning long intervals. The TLC-index can reduce the storage cost and improve the insertion performance through decomposing long intervals into one or more cell constructs that have long size. It can also improve the search performance through decomposing short intervals into one or more virtual constructs that have short size enough to fit into those intervals.

SENSOR DATA MINING TECHNIQUES AND MIDDLEWARE STRUCTURE FOR USN ENVIRONMENT

  • Jin, Cheng-Hao;Lee, Yong-Mi;Kim, Hi-Seok;Pok, Gou-Chol;Ryu, Keun-Ho
    • Proceedings of the KSRS Conference
    • /
    • 2007.10a
    • /
    • pp.353-356
    • /
    • 2007
  • With advances in sensor technology, current researches on the pertinent techniques are actively directed toward the way which enables the USN computing service. For many applications using sensor networks, the incoming data are by nature characterized as high-speed, continuous, real-time and infinite. Due to such uniqueness of sensor data characteristics, for some instances a finite-sized buffer may not accommodate the entire incoming data, which leads to inevitable loss of data, and requirement for fast processing makes it impossible to conduct a thorough investigation of data. In addition to the potential problem of loss of data, incoming data in its raw form may exhibit high degree of complexity which evades simple query or alerting services for capturing and extracting useful information. Furthermore, as traditional mining techniques are developed to handle fixed, static historical data, they are not useful and directly applicable for analyzing the sensor data. In this paper, (1) describe how three mining techniques (sensor data outlier analysis, sensor pattern analysis, and sensor data prediction analysis) are appropriate for the USN middleware structure, with their application to the stream data in ocean environment. (2) Another proposal is a middleware structure based on USN environment adaptive to above mining techniques. This middleware structure includes sensor nodes, sensor network common interface, sensor data processor, sensor query processor, database, sensor data mining engine, user interface and so on.

  • PDF

In-memory Compression Scheme Based on Incremental Frequent Patterns for Graph Streams (그래프 스트림 처리를 위한 점진적 빈발 패턴 기반 인-메모리 압축 기법)

  • Lee, Hyeon-Byeong;Shin, Bo-Kyoung;Bok, Kyoung-Soo;Yoo, Jae-Soo
    • The Journal of the Korea Contents Association
    • /
    • v.22 no.1
    • /
    • pp.35-46
    • /
    • 2022
  • Recently, with the development of network technologies, as IoT and social network service applications have been actively used, a lot of graph stream data is being generated. In this paper, we propose a graph compression scheme that considers the stream graph environment by applying graph mining to the existing compression technique, which has been focused on compression rate and runtime. In this paper, we proposed Incremental frequent pattern based compression technique for graph streams. Since the proposed scheme keeps only the latest reference patterns, it increases the storage utilization and improves the query processing time. In order to show the superiority of the proposed scheme, various performance evaluations are performed in terms of compression rate and processing time compared to the existing method. The proposed scheme is faster than existing similar scheme when the number of duplicated data is large.