• Title/Summary/Keyword: stream data processing

Search Result 447, Processing Time 0.024 seconds

Adaptive Upstream Backup Scheme based on Throughput Rate in Distributed Spatial Data Stream System (분산 공간 데이터 스트림 시스템에서 연산 처리율 기반의 적응적 업스트림 백업 기법)

  • Jeong, Weonil
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.14 no.10
    • /
    • pp.5156-5161
    • /
    • 2013
  • In distributed spatial data stream processing, processed tuples of downstream nodes are replicated to the upstream node in order to increase the utilization of distributed nodes and to recover the whole system for the case of system failure. However, while the data input rate increases and multiple downstream nodes share the operation result of the upstream node, the data which stores to output queues as a backup can be lost since the deletion operation delay may be occurred by the delay of the tuple processing of upstream node. In this paper, the adaptive upstream backup scheme based on operation throughput in distributed spatial data stream system is proposed. This method can cut down the average load rate of nodes by efficient spatial operation migration as it processes spatial temporal data stream, and it can minimize the data loss by fluid change of backup mode. The experiments show the proposed approach can prevent data loss and can decrease, on average, 20% of CPU utilization by node monitoring.

A Push-Caching and a Transmission Scheme of Continuous Media for NOD Service on the Internet (인테넷상에서 NOD 서비스를 위한 연속미디어 전송 및 푸쉬-캐싱 기법)

  • Park, Seong-Ho;Im, Eun-Ji;Choe, Tae-Uk;Jeong, Gi-Dong
    • The Transactions of the Korea Information Processing Society
    • /
    • v.7 no.6
    • /
    • pp.1766-1777
    • /
    • 2000
  • In multimedia new service on the internet, there are problems such as server overload, network congestion and initial latency. To overcome these problems, we propose a proxy push-caching scheme that stores a portion of continuous media stream or entire stream, and a transmission scheme of NOD continuous media, RTP-RR and RTP-nR to exploit push-caching scheme. With the proposed push-caching scheme, NOD server pushes fixed portion of stream to a proxy when new data is generated, and the cached size of each stream changes dynamically according to the caching utility value of each stream. As a result, the initial latency of client side could be reduced and the amount of data transmitted fro ma proxy server to client could be increased. Moreover, we estimate a caching utility value of each stream using correlation between disk space occupied by the stream and the amount of data stream requested by client. And we applied the caching utility value ot replacement policies. The performance of the proxy push-caching and continuous media transmission schemes proposed were compared with other schemes using simulations. In the simulation, these schemes show better results than other schemes in terms of BHR (Byte Hit Rate), initial latency, the number of replacement and packet loss rate.

  • PDF

An Efficient Scheme to write a Transmission Schedule using Convergence after Interactive Operations in a Stored Video (대화형 연산 후 수렴을 이용한 저장된 비디오의 효율적인 전송 스케줄 작성 방안)

  • Lee, Jae-Hong;Kim, Seung-Hwan
    • The Transactions of the Korea Information Processing Society
    • /
    • v.7 no.7
    • /
    • pp.2050-2059
    • /
    • 2000
  • In a video-on-Demand(VOD) service, a server has to return to he normal playback quickly at a certain new frame position after interactive operations such as jump or last playback. In this paper, we propose an efficient scheme to write a transmission schedule for a playback restart of a video stream at a new frame position after interactive operations. The proposed scheme is based on convergence characteristics, that is transmission schedules with different playback startup frame position in a video stream meet each other at some frame position. The scheme applies a bandwidth smoothing from a new frame position to a convergence position without considering all remaining frames of a video stream. And then the scheme transmits video dta according to the new schedule from the new frame position to the convergence position, and then transmits the remaining video data according to the reference schedule from the convergence position, and then transmits the remaining video data according to the reference schedule from the convergence position to the last frame position. In this paper, we showed that there existed the convergence position corresponding to nay frame position in a video stream through many experiments based on MPEG-1 bit trace data. With the convergence we reduced the computational overhead of a bandwidth smoothing, which was applied to find a new transmission schedule after interactive operations. Also, storage overhead is greatly reduced by storing pre-calculated schedule information up to the convergence position for each I frame position of a video stream with video data off-line. By saving information on a transmission schedule off-line along with the video data and searching the schedule corresponding to the specified restarting frame position, we expect the possibility of normal playback of a video stream with small tolerable playback startup delay.

  • PDF

A Study on the Traffic Flow Analysis Method by Image Processing (화상처리에 의한 교통류 해석방법에 관한 연구)

  • 이종달;이령욱
    • Journal of Korean Society of Transportation
    • /
    • v.12 no.1
    • /
    • pp.97-116
    • /
    • 1994
  • Today advanced traffic management systems are required because of a high increase in traffic demand. Accordingly, the objective of this study is to take advantage of image processing systems and present image processing methods available for collection of the data on traffic characteristics, and then to investigate the possibility of traffic flow analysis by means of comparison and analysis of measured traffic flow. Data were collected at two places of Daegu city and Kyongbu expressway by using VTR. Rear view (down stream) and frontal view (up stream) methods were employed to compare and analyze traffic characteristics including traffic volume, speed, time-headway, time-occupancy, and vehicle-length, by analysis of measured traffic flow and image processing respectively. Judging from the results obtained by this study, image processing techniques are sufficient for the analysis of traffic volume, but a frame grabber equipped with high speed processor is necessary as well, with low level system judged to be sufficient for traffic volume analysis.

  • PDF

TIM: A Trapdoor Hash Function-based Authentication Mechanism for Streaming Applications

  • Seo, Seog Chung;Youn, Taek-Young
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.12 no.6
    • /
    • pp.2922-2945
    • /
    • 2018
  • Achieving efficient authentication is a crucial issue for stream data commonly seen in content delivery, peer-to-peer, and multicast/broadcast networks. Stream authentication mechanisms need to be operated efficiently at both sender-side and receiver-side at the same time because of the properties of stream data such as real-time and delay-sensitivity. Until now, many stream authentication mechanisms have been proposed, but they are not efficient enough to be used in stream applications where the efficiency for sender and receiver sides are required simultaneously since most of them could achieve one of either sender-side and receiver-side efficiency. In this paper, we propose an efficient stream authentication mechanism, so called TIM, by integrating Trapdoor Hash Function and Merkle Hash Tree. Our construction can support efficient streaming data processing at both sender-side and receiver-side at the same time differently from previously proposed other schemes. Through theoretical and experimental analysis, we show that TIM can provide enhanced performance at both sender and receiver sides compared with existing mechanisms. Furthermore, TIM provides an important feature for streaming authentication, the resilience against transmission loss, since each data block can be verified with authentication information contained in itself.

Mining Association Rules in Multidimensional Stream Data (다차원 스트림 데이터의 연관 규칙 탐사 기법)

  • Kim, Dae-In;Park, Joon;Kim, Hong-Ki;Hwang, Bu-Hyun
    • The KIPS Transactions:PartD
    • /
    • v.13D no.6 s.109
    • /
    • pp.765-774
    • /
    • 2006
  • An association rule discovery, a technique to analyze the stored data in databases to discover potential information, has been a popular topic in stream data system. Most of the previous researches are concerned to single stream data. However, this approach may ignore in mining to multidimensional stream data. In this paper, we study the techniques discovering the association rules to multidimensional stream data. And we propose a AR-MS method reflecting the characteristics of stream data since make the summarization information by one data scan and discovering the association rules for significant rare data that appear infrequently in the database but are highly associated with specific event. Also, AR-MS method can discover the maximal frequent item of multidimensional stream data by using the summarization information. Through analysis and experiments, we show that AR-MS method is superior to other previous methods.

A Method for Frequent Itemsets Mining from Data Stream (데이터 스트림 환경에서 효율적인 빈발 항목 집합 탐사 기법)

  • Seo, Bok-Il;Kim, Jae-In;Hwang, Bu-Hyun
    • The KIPS Transactions:PartD
    • /
    • v.19D no.2
    • /
    • pp.139-146
    • /
    • 2012
  • Data Mining is widely used to discover knowledge in many fields. Although there are many methods to discover association rule, most of them are based on frequency-based approaches. Therefore it is not appropriate for stream environment. Because the stream environment has a property that event data are generated continuously. it is expensive to store all data. In this paper, we propose a new method to discover association rules based on stream environment. Our new method is using a variable window for extracting data items. Variable windows have variable size according to the gap of same target event. Our method extracts data using COBJ(Count object) calculation method. FPMDSTN(Frequent pattern Mining over Data Stream using Terminal Node) discovers association rules from the extracted data items. Through experiment, our method is more efficient to apply stream environment than conventional methods.

Energy-efficient Broadcasting of XML Data in Mobile Computing Environments (이동 컴퓨팅 환경에서 XML 데이타의 에너지 효율적인 방송)

  • Kim Chung Soo;Park Chang-Sup;Chung Yon Dohn
    • Journal of KIISE:Databases
    • /
    • v.33 no.1
    • /
    • pp.117-128
    • /
    • 2006
  • In this paper, we propose a streaming method for XML data that supports energy-efficient processing of queries over the stream in mobile clients. We propose new stream organizations for XML data which have different kinds of addresses to related data in a stream. We describe event-driven stream generation algorithms for the proposed stream structures and provide search algorithms for simple XML path queries which leverage the access mechanisms incorporated in the stream. Experimental results show that our approaches can effectively improve the tuning time performance of user queries in a wireless broadcasting environment.

Filtering Method for Analyzing Renewable Energy Stream Data (신재생 에너지 스트림 데이터 분석을 위한 필터링 기법)

  • Jin, Cheng Hao;Li, Xun;Kim, Kyu Ik;Hwang, Mi Yeong;Kim, Sang Yeob;Kim, Kwang Deuk;Ryu, Keun Ho
    • Journal of Convergence Society for SMB
    • /
    • v.1 no.1
    • /
    • pp.39-44
    • /
    • 2011
  • Recently, due to people's incontinent use all over the world, fossil fuels such as coal, oil, and natural gas were nearly to be exhausted and also causes serious environment pollutions. Therefore, there is a strong need to develop solar, wind, hydro, biomass, geothermal to replace fossil fuels to prevent suffering from above problems. Wish advances in sensor technology, such data is collected as a kind of stream data which arrives in an online manner so that it is characterized as high- speed, real-time and unbounded and it requires fast data processing to get the up-to-date results. Therefore, the traditional data processing techniques are not fit to deal with stream data. In this paper, we propose a kalman filter-based algorithm to process renewable stream data.

  • PDF

A Novel Way of Context-Oriented Data Stream Segmentation using Exon-Intron Theory (Exon-Intron이론을 활용한 상황중심 데이터 스트림 분할 방안)

  • Lee, Seung-Hun;Suh, Dong-Hyok
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.16 no.5
    • /
    • pp.799-806
    • /
    • 2021
  • In the IoT environment, event data from sensors is continuously reported over time. Event data obtained in this trend is accumulated indefinitely, so a method for efficient analysis and management of data is required. In this study, a data stream segmentation method was proposed to support the effective selection and utilization of event data from sensors that are continuously reported and received. An identifier for identifying the point at which to start the analysis process was selected. By introducing the role of these identifiers, it is possible to clarify what is being analyzed and to reduce data throughput. The identifier for stream segmentation proposed in this study is a semantic-oriented data stream segmentation method based on the event occurrence of each stream. The existence of identifiers in stream processing can be said to be useful in terms of providing efficiency and reducing its costs in a large-volume continuous data inflow environment.