• Title/Summary/Keyword: stream data processing

Search Result 447, Processing Time 0.027 seconds

Load Shedding via Predicting the Frequency of Tuple for Efficient Analsis over Data Streams (효율적 데이터 스트림 분석을 위한 발생빈도 예측 기법을 이용한 과부하 처리)

  • Chang, Joong-Hyuk
    • The KIPS Transactions:PartD
    • /
    • v.13D no.6 s.109
    • /
    • pp.755-764
    • /
    • 2006
  • In recent, data streams are generated in various application fields such as a ubiquitous computing and a sensor network, and various algorithms are actively proposed for processing data streams efficiently. They mainly focus on the restriction of their memory usage and minimization of their processing time per data element. However, in the algorithms, if data elements of a data stream are generated in a rapid rate for a time unit, some of the data elements cannot be processed in real time. Therefore, an efficient load shedding technique is required to process data streams effcientlv. For this purpose, a load shedding technique over a data stream is proposed in this paper, which is based on the predicting technique of the frequency of data element considering its current frequency. In the proposed technique, considering the change of the data stream, its threshold for tuple alive is controlled adaptively. It can help to prevent unnecessary load shedding.

A method of event data stream processing for ALE Middleware (ALE 미들웨어를 위한 이벤트 데이터 처리 방법)

  • Noh, Young-Sik;Lee, Dong-Cheol;Byun, Yung-Cheol
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.12 no.9
    • /
    • pp.1554-1563
    • /
    • 2008
  • As the interests on RFID technologies increase, a lot of research activities on RFID middleware systems to handle the data acquired by RFID readers are going on actively. Meanwhile, even though various kinds of RFID middleware methodologies and related techniques have been proposed, the common data type which is dealt with in those systems is an EPC code, mainly. Also, there are few researches of the implementation of collecting the stream data queued from RFID readers endlessly and without blocking, classifying the data into some groups according to usage, and sending the resulting data to specific applications. In this paper, we propose the method of data handling in RFID middleware to efficiently process an EPC event stream data using detail filtering, checking of data modification, creation of data set to transfer, data grouping, and various kinds of RFID data format transform. Our method is based on a de facto international standard interface defined in the ALE middleware specification by EPCglobal, and application and service users can directly set various kinds of conditions to handle the stream data.

A Multi-Query Optimizing Method for Data Stream Similar Queries on Sliding Window (슬라이딩 윈도에서의 데이터 스팀데이터 유사 질의 처리를 위한 다중질의 최적화 기법)

  • Liangbo Li;Yan Li;Song-Sun Shin;Dong-Wook Lee;Weon-Il Chung;Hae-Young Bae
    • Annual Conference of KIPS
    • /
    • 2008.11a
    • /
    • pp.413-416
    • /
    • 2008
  • In the presence of multiple continuous queries, multi-query optimizing is a new challenge to process multiple stream data in real-time. So, in this paper, we proposed an approach to optimize multi-query of sliding window on network traffic data streams and do some comparisons to traditional queries without optimizing. We also detail some method of scheduling on different data streams, while different scheduling made different results. We test the results on variety of multi-query processing schedule, and proofed the proposed method is effectively optimized the data stream similar multi-queries.

Video Stream Smoothing Using Multistreams (멀티스트림을 이용한 비디오 스트림의 평활화)

  • 강경원;문광석
    • Journal of the Institute of Convergence Signal Processing
    • /
    • v.3 no.1
    • /
    • pp.21-26
    • /
    • 2002
  • Video stream invoke a variety of traffic with the structure of compression algorithm and image complexity. Thus, it is difficult to allocate the resource on the both sides of sender and receiver, and playout on the Internet such as a packet switched network. Thus, in this paper we proposed video stream smoothing using multistream for the effective transmission of video stream. This method specifies the type of LDU(logical data unit) according to the type of original stream, and then makes a large number of streams as a fixed size, and transfers them. So, the proposed method can reduce the buffering time which occurs during the process of the smoothing and prefetch be robust to the jitter on network, as well. Consequently, it has the effective transmission characteristics of fully utilizing the clients bandwidth.

  • PDF

Partition-based Operator Sharing Scheme for Spatio-temporal Data Stream Processing (시공간 데이터 스트림 처리를 위한 영역 기반의 연산자 공유 기법)

  • Chung, Weon-Il;Kim, Young-Ki
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.11 no.12
    • /
    • pp.5042-5048
    • /
    • 2010
  • In ubiquitous environments, many continuous query processing techniques make use of operator network and sharing methods on continuous data stream generated from various sensors. Since similar continuous queries with the location information intensively occur in specific regions, we suggest a new operator sharing method based on grid partition for the spatial continuous query processing for location-based applications. Due to the proposed method shares moving objects by the given grid cell without sharing spatial operators individually, our approach can not only share spatial operators including similar conditions, but also increase the query processing performance and the utilization of memory by reducing the frequency of use of spatial operators.

Development of a Spatial DSMS for Efficient Real-Time Processing of Spatial Sensor Data (공간 센서 데이타의 효율적인 실시간 처리를 위한공간 DSMS의 개발)

  • Kang, Hong-Koo;Park, Chi-Min;Hong, Dong-Suk;Han, Ki-Joon
    • Journal of Korea Spatial Information System Society
    • /
    • v.9 no.1
    • /
    • pp.45-57
    • /
    • 2007
  • Recently, the development of sensor devices has accelerated researches on advanced technologies like Wireless Sensor Networks. Moreover, spatial sensors using GPS lead to the era of the Ubiquitous Computing Environment which generally uses spatial information and non-spatial information together. In this new era, a real-time processing system for spatial sensor data is essential. In this reason, new data processing systems called DSMS(Data Stream Management System) are being developed by many researchers. However, since most of them do not support geometry types and spatial functions to process spatial sensor data, they are not suitable for the Ubiquitous Computing Environment. For these reasons, in this paper, we designed and implemented a spatial DSMS by extending STREAM which stands for STanford stREam datA Manager, to solve these problems. We added geometry types and spatial functions to STREAM in order to process spatial sensor data efficiently. In addition, we implemented a Spatial Object Manager to manage shared spatial objects within the system. Especially, we implemented the Simple Features Specification for SQL of OGC for interoperability and applied algorithms in GEOS to our system.

  • PDF

Data Source Management using weight table in u-GIS DSMS

  • Kim, Sang-Ki;Baek, Sung-Ha;Lee, Dong-Wook;Chung, Warn-Il;Kim, Gyoung-Bae;Bae, Hae-Young
    • Journal of Korea Spatial Information System Society
    • /
    • v.11 no.2
    • /
    • pp.27-33
    • /
    • 2009
  • The emergences of GeoSensor and researches about GIS have promoted many researches of u-GIS. The disaster application coupled in the u-GIS can apply to monitor accident area and to prevent spread of accident. The application needs the u-GIS DSMS technique to acquire, to process GeoSensor data and to integrate them with GIS data. The u-GIS DSMS must process big and large-volume data stream such as spatial data and multimedia data. Due to the feature of the data stream, in u-GIS DSMS, query processing can be delayed. Moreover, as increasing the input rate of data in the area generating events, the network traffic is increased. To solve this problem, in this paper we describe TRIGGER ACTION clause in CQ on the u-GIS DSMS environment and proposes data source management. Data source weight table controls GES information and incoming data rate. It controls incoming data rate as increasing weight at GES of disaster area. Consequently, it can contribute query processing rate and accuracy

  • PDF

Cost Analysis of Window Memory Relocation for Data Stream Processing (데이터 스트림 처리를 위한 윈도우 메모리 재배치의 비용 분석)

  • Lee, Sang-Don
    • The Journal of the Korea Contents Association
    • /
    • v.8 no.4
    • /
    • pp.48-54
    • /
    • 2008
  • This paper analyzes cost tradeoffs between memory usage and computation for window-based operators in data stream environments. It identifies generic operator network constructs, and sets up a cost model for the estimation of the expected memory reduction and the computation overheads when window memory relocations are applied to each operator network construct. This cost model helps to identify the utility of window memory relocations. It also helps to apply window memory relocation to improve a query execution plan to save memory usage. The proposed approach contributes to expand the scope of query processing and optimization in data stream environments. It also provides a basis to develop a cost estimation model for the query optimization using window memory relocations.

Spatio-temporal Query Processing Systems for Ubiquitous Environments

  • Kim, Jeong Joon;Kang, Jeong Jin;Rothwell, Edward J.;Lee, Ki Young
    • International Journal of Internet, Broadcasting and Communication
    • /
    • v.5 no.2
    • /
    • pp.1-4
    • /
    • 2013
  • With the recent development of the ubiquitous computing technology, there are increasing interest and research in technologies such as sensors and RFID related to information recognition and location positioning in various ubiquitous fields. Especially, RTLS (Real-Time Locating Services) dealing with spatio-temporal data is emerging as a promising technology. For these reasons, the ISO/IEC published RTLS standard specification for compatibility and interoperability in RTLS. Therefore, in this paper, we designed and implemented Spatio-temporal Query Processing Systems for efficiently managing and searching the incoming Spatio-temporal data stream of moving objects. Spatio-temporal Query Processing Systems's spatio-temporal middleware maintains interoperability among heterogeneous devices and guarantees data integrity in query processing through real time processing of unceasing spatio-temporal data streams and two way synchronization of spatio-temporal DBMSs. Web Server uses the SOAP(Simple Object Access Protocol) message between client and server for interoperability and translates client's SOAP message into CQL(Continuous Query Language) of the spatio-temporal middleware.

Efficient Processing of Multidimensional Sensor stream Data in Digital Marine Vessel (디지털 선박 내 다차원 센서 스트림 데이터의 효율적인 처리)

  • Song, Byoung-Ho;Park, Kyung-Woo;Lee, Jin-Seok;Lee, Keong-Hyo;Jung, Min-A;Lee, Sung-Ro
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.35 no.5B
    • /
    • pp.794-800
    • /
    • 2010
  • It is necessary to accurate and efficient management for measured digital data from various sensors in digital marine vessel. It is not efficient that sensor network process input stream data of mass storage stored in database the same time. In this paper, We propose to improve the processing performance of multidimensional stream data continuous incoming from multiple sensor. We propose that we arrange some sensors (temperature, humidity, lighting, voice) and process query based on sliding window for efficient input stream and found multiple query plan to Mjoin method and we reduce stored data using SVM algorithm. We automatically delete that it isn't necessary to the data from the database and we used to ship diagnosis system for available data. As a result, we obtained to efficient result about 18.3% reduction rate of database using 35,912 data sets.