• Title/Summary/Keyword: data streams

Search Result 821, Processing Time 0.023 seconds

Load Shedding for Temporal Queries over Data Streams

  • Al-Kateb, Mohammed;Lee, Byung-Suk
    • Journal of Computing Science and Engineering
    • /
    • v.5 no.4
    • /
    • pp.294-304
    • /
    • 2011
  • Enhancing continuous queries over data streams with temporal functions and predicates enriches the expressive power of those queries. While traditional continuous queries retrieve only the values of attributes, temporal continuous queries retrieve the valid time intervals of those values as well. Correctly evaluating such queries requires the coalescing of adjacent timestamps for value-equivalent tuples prior to evaluating temporal functions and predicates. For many stream applications, the available computing resources may be too limited to produce exact query results. These limitations are commonly addressed through load shedding and produce approximated query results. There have been many load shedding mechanisms proposed so far, but for temporal continuous queries, the presence of coalescing makes theses existing methods unsuitable. In this paper, we propose a new accuracy metric and load shedding algorithm that are suitable for temporal query processing when memory is insufficient. The accuracy metric uses a combination of the Jaccard coefficient to measure the accuracy of attribute values and $\mathcal{PQI}$ interval orders to measure the accuracy of the valid time intervals in the approximate query result. The algorithm employs a greedy strategy combining two objectives reflecting the two accuracy metrics (i.e., value and interval). In the performance study, the proposed greedy algorithm outperforms a conventional random load shedding algorithm by up to an order of magnitude in its achieved accuracy.

A New File System for Multimedia Data Stream (멀티미디어 데이터 스트림을 위한 파일 시스템의 설계 및 구현)

  • Lee, Minsuk;Song, Jin-Seok
    • IEMEK Journal of Embedded Systems and Applications
    • /
    • v.1 no.2
    • /
    • pp.90-103
    • /
    • 2006
  • There are many file systems in various operating systems. Those are usually designed for server environments, where the common cases are usually 'multiple active users', 'great many small files' And they assume a big main memory to be used as buffer cache. So the existing file systems are not suitable for resource hungry embedded systems that process multimedia data streams. In this study, we designed and implemented a new file system which efficiently stores and retrieves multimedia data steams. The proposed file system has a very simple disk layout, which guarantees a quick disk initialization and file system recovery. And we introduced a new indexing-scheme, called the time-based indexing scheme, with the file system. With the indexing scheme, the file system maintains the relation between time and the location for all the multimedia streams. The scheme is useful in searching and playing the compressed multimedia streams by locating exact frame position with given time, resulting in reduction of CPU processing and power consumption. The proposed file system and its APIs utilizing the time-based indexing schemes were implemented firstly on a Linux environment, though it is operating system independent. In the performance evaluation on a real DVR system, which measured the execution time of multi-threaded reading and writing, we found the proposed file system is maximum 38.7% faster than EXT2 file system.

  • PDF

An Experimental Delay Analysis Based on M/G/1-Vacation Queues for Local Audio/Video Streams

  • Kim, Doo-Hyun;Lee, Kyung-Hee;Kung, Sang-Hwan;Kim, Jin-Hyung
    • ETRI Journal
    • /
    • v.19 no.4
    • /
    • pp.344-362
    • /
    • 1997
  • The delay which is one of the quality of service parameters is considered to be a crucial factor for the effective usage of real-time audio and video streams in interactive multimedia collaborations. Among the various causes of the delay, we focus in this paper on the local delay concerned with the schemes which handle continuous inflow of encoded data from constant or variable bit-rate audio and video encoders. We introduce two kinds of implementation approaches, pull model and push model. While the pull model periodically pumps out the incoming data from the system buffer, the push model receives events from the device drivers. From our experiments based on Windows NT 3.51, it is shown that the push model outperforms the other for both constant and variable bit-rate streams in terms of the local delay, when the system suffers reasonable loads. We interpret this experimental data with M/G/1 multiple vacation queuing theories, and show that it is consistent with the queuing theoretic interpretations.

  • PDF

Multi-level Load Shedding Scheme to Increase Spatial Data Stream Query Accuracy (공간 데이터 스트림 질의 정확도 향상을 위한 다단계 부하제한 기법)

  • Jeong, Weonil
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.16 no.12
    • /
    • pp.8370-8377
    • /
    • 2015
  • In spatial data stream management systems, it is needed appropriate load shedding algorithm because real-time input spatial data streams could exceed the limitation of main memory. However previous researches, lack regard for input ratio and spatial utilization rates of spatial data streams, or the characteristics of data source which generates data streams with spatial information efficiently, can lead to decrease the performance and accuracy of spatial data stream query. Therefore, multi-level load shedding scheme for spatial data stream management systems is proposed to increase the spatial query performance and accuracy. This proposed scheme limits overloads in relation to the input rate and the characteristics of data source first, and then, if needed, query data representing low query participation probability based on spatial utilizations are dropped relatively. Our experiments show that the proposed method could decrease load shedding frequency for previous researches by more than 11% despite query results accuracy and query performance are superior at 0.04% and 3%.

Dynamic Subspace Clustering for Online Data Streams (온라인 데이터 스트림에서의 동적 부분 공간 클러스터링 기법)

  • Park, Nam Hun
    • Journal of Digital Convergence
    • /
    • v.20 no.2
    • /
    • pp.217-223
    • /
    • 2022
  • Subspace clustering for online data streams requires a large amount of memory resources as all subsets of data dimensions must be examined. In order to track the continuous change of clusters for a data stream in a finite memory space, in this paper, we propose a grid-based subspace clustering algorithm that effectively uses memory resources. Given an n-dimensional data stream, the distribution information of data items in data space is monitored by a grid-cell list. When the frequency of data items in the grid-cell list of the first level is high and it becomes a unit grid-cell, the grid-cell list of the next level is created as a child node in order to find clusters of all possible subspaces from the grid-cell. In this way, a maximum n-level grid-cell subspace tree is constructed, and a k-dimensional subspace cluster can be found at the kth level of the subspace grid-cell tree. Through experiments, it was confirmed that the proposed method uses computing resources more efficiently by expanding only the dense space while maintaining the same accuracy as the existing method.

RSP-DS: Real Time Sequential Patterns Analysis in Data Streams (RSP-DS: 데이터 스트림에서의 실시간 순차 패턴 분석)

  • Shin Jae-Jyn;Kim Ho-Seok;Kim Kyoung-Bae;Bae Hae-Young
    • Journal of Korea Multimedia Society
    • /
    • v.9 no.9
    • /
    • pp.1118-1130
    • /
    • 2006
  • Existed pattern analysis algorithms in data streams environment have researched performance improvement and effective memory usage. But when new data streams come, existed pattern analysis algorithms have to analyze patterns again and have to generate pattern tree again. This approach needs many calculations in real situation that needs real time pattern analysis. This paper proposes a method that continuously analyzes patterns of incoming data streams in real time. This method analyzes patterns fast, and thereafter obtains real time patterns by updating previously analyzed patterns. The incoming data streams are divided into several sequences based on time based window. Informations of the sequences are inputted into a hash table. When the number of the sequences are over predefined bound, patterns are analyzed from the hash table. The patterns form a pattern tree, and later created new patterns update the pattern tree. In this way, real time patterns are always maintained in the pattern tree. During pattern analysis, suffixes of both new pattern and existed pattern in the tree can be same. Then a pointer is created from the new pattern to the existed pattern. This method reduce calculation time during duplicated pattern analysis. And old patterns in the tree are deleted easily by FIFO method. The advantage of our algorithm is proved by performance comparison with existed method, MILE, in a condition that pattern is changed continuously. And we look around performance variation by changing several variable in the algorithm.

  • PDF

Dynamic Load Management Method for Spatial Data Stream Processing on MapReduce Online Frameworks (맵리듀스 온라인 프레임워크에서 공간 데이터 스트림 처리를 위한 동적 부하 관리 기법)

  • Jeong, Weonil
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.19 no.8
    • /
    • pp.535-544
    • /
    • 2018
  • As the spread of mobile devices equipped with various sensors and high-quality wireless network communications functionsexpands, the amount of spatio-temporal data generated from mobile devices in various service fields is rapidly increasing. In conventional research into processing a large amount of real-time spatio-temporal streams, it is very difficult to apply a Hadoop-based spatial big data system, designed to be a batch processing platform, to a real-time service for spatio-temporal data streams. This paper extends the MapReduce online framework to support real-time query processing for continuous-input, spatio-temporal data streams, and proposes a load management method to distribute overloads for efficient query processing. The proposed scheme shows a dynamic load balancing method for the nodes based on the inflow rate and the load factor of the input data based on the space partition. Experiments show that it is possible to support efficient query processing by distributing the spatial data stream in the corresponding area to the shared resources when load management in a specific area is required.

Design of a Format Converter from MPEG-4 Over MPEG-2 TS to MP4 (MPEG-4 Over MPEG-2 TS로부터 MP4 파일로의 포맷 변환기 설계)

  • 최재영;정제창
    • Journal of Broadcast Engineering
    • /
    • v.5 no.2
    • /
    • pp.176-187
    • /
    • 2000
  • MPEG-4 is a digital bit stream format and associated protocols for representing multimedia content consisting of natural and synthetic audio, video and object data. This paper describes an application where multiple audio/visual data stream are combined in MPEG-4 and transported via MPTG-2 transport streams(TS). Also, this paper describes how to convert MPEG-4 Over MPEG-2 TS bit streams into MP4 file which Is designed to contain the media information of an MPEG-4 presentation in a flexible, extensible format. MPEG-4 is presented in the form of audio-visual objects that are arranged into an audio-visual scene by means of a scene descriptor and is composed of the audio-visual objects by means of an object descriptor. These descriptor streams are not defined MPEG-2 TS. So. this paper focuses on handling of these descriptors and parsing TS streams to get MPEG-4 data. The MPEG-4 Over MPEG-2 TS to MP4 format converter is implemented in the demonstrated systems.

  • PDF

A Spatial Data Stream Processing System for Spatial Context Analysis in Real-time (실시간 공간 상황 분석을 위한 공간 데이터 스트림 처리 시스템)

  • Kwon, O-Je;Kim, Jae-Hun;Li, Ki-Joune
    • Spatial Information Research
    • /
    • v.18 no.1
    • /
    • pp.69-76
    • /
    • 2010
  • Spatial data streams from sensors are useful in context-awareness for many types of applications. However, an important gap is found between spatial data stream management in real-time and complex computation for spatial context-awareness, and this brings about serious difficulty to integrate spatial data stream processing and context-awareness. In this paper, we present a system called SCONSTREAM(Spatial CONtext STREAm Management) that we have developed to resolve the gap between spatial data stream and context-awareness. The key approach of our system is to filter off unnecessary spatial data streams and convert them to the spatial context streams, which are smaller and more suitable to be processed by the context-awareness module than raw data from sensors. By experimentation, We show that SCONSTREAM resolves the functional gap between spatial stream processing and spatial context-awareness module.

New Interactive TV Service Model based on the MPEG-4 System

  • Kim, Jongho;Jechang Jeong
    • Proceedings of the IEEK Conference
    • /
    • 2002.07a
    • /
    • pp.125-128
    • /
    • 2002
  • In this paper, a new interactive TV service model is proposed. The MPEG-4 system is specified for composing and managing various object streams including user interactions. The data broadcasting model supporting user interactions is designed using MPEG-4 system in our proposal. We evaluate possibility of proposed service model using simulation player. This player supports MPEG-2 TS which contains MPEG-2 video and AC-3 audio streams as a main service and MPEC-4 system data as interactive services as well as user specific EPG information, and XML data, etc as supplemetary services. The player also supports a multi-channel environment. The synchronization between audio and visual data is achieved by DTS and PTS in TS.

  • PDF