• Title/Summary/Keyword: stream data processing

Search Result 447, Processing Time 0.026 seconds

A Sliding Window Technique for Open Data Mining over Data Streams (개방 데이터 마이닝에 효율적인 이동 윈도우 기법)

  • Chang Joong-Hyuk;Lee Won-Suk
    • The KIPS Transactions:PartD
    • /
    • v.12D no.3 s.99
    • /
    • pp.335-344
    • /
    • 2005
  • Recently open data mining methods focusing on a data stream that is a massive unbounded sequence of data elements continuously generated at a rapid rate are proposed actively. Knowledge embedded in a data stream is likely to be changed over time. Therefore, identifying the recent change of the knowledge quickly can provide valuable information for the analysis of the data stream. This paper proposes a sliding window technique for finding recently frequent itemsets, which is applied efficiently in open data mining. In the proposed technique, its memory usage is kept in a small space by delayed-insertion and pruning operations, and its mining result can be found in a short time since the data elements within its target range are not traversed repeatedly. Moreover, the proposed technique focused in the recent data elements, so that it can catch out the recent change of the data stream.

Spatial Data Stream Processing Platform for Ubiquitous Application Services (유비쿼터스 응용 서비스를 위한 공간 데이터 스트림 처리 플랫폼)

  • Chung, Weon-Il;Kim, Hwan-Koo
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.11 no.3
    • /
    • pp.906-913
    • /
    • 2010
  • Sensors related to the geographic information are gathering strength as core technologies for various ubiquitous services like u-City project for the new town of the future to provide total information services by the high IT infrastructure. These sensors generate the very large real-time streaming data because these are set up and controlled with wide areas of the geographical distribution. On this, we propose an efficient spatial data stream processing system to support various u-GIS services based on geographic sensors.

Multi-stream Delivery Method of the Video Data Based on SPIHT Wavelet (SPIHT 웨이브릿 기반의 비디오 데이터의 멀티스트림 전송 기법)

  • 강경원;류권열;김기룡;문광석;김문수
    • Journal of the Institute of Convergence Signal Processing
    • /
    • v.3 no.3
    • /
    • pp.14-20
    • /
    • 2002
  • In this paper, we proposed the compression technique of the video data using SPIHT(set partitioning in hierarchical trees) based on wavelet and the multi-stream delivery method for best-effort service as fully utilizing the clients bandwidth over the current Internet. The experiment shows that the proposed method provides about 1.5dB better picture quality without block effects than DCT(discrete consine transform) based coding schemes at the same bit rates because of using the wavelet video coder. In addition, this technique implements the multi-stream transmission based on TCP(transmission control protocol). Thus, it is provided with the best-efforts service which is robust to the network jitter problem, and maximally utilizes the bandwidth of the client's.

  • PDF

PnP Supporting Middleware Framework for Network Based Humanoid (네트워크 기반 휴머노이드에서의 PnP가 가능한 미들웨어 프레임워크)

  • Lee, Ho-Dong;Kim, Dong-Won;Kim, Joo-Hyung;Park, Gwi-Tae
    • The Journal of Korea Robotics Society
    • /
    • v.3 no.3
    • /
    • pp.255-261
    • /
    • 2008
  • This paper describes a network framework that support network based humanoid. The framework utilizes middleware such as CORBA (ACE/TAO) that provides PnP capability for network based humanoid. The network framework transfers data gathered from a network based humanoid to a processing group that is distributed on a network. The data types are video stream, audio stream and control data. Also, the network framework transfers service data produced by the processing group to the network based humanoid. By using this network framework, the network based humanoid can provide high quality of intelligent services to user.

  • PDF

Rate-Controlled Data-Driven Real-Time Stream Processing for an Autonomous Machine (자율 기기를 위한 속도가 제어된 데이터 기반 실시간 스트림 프로세싱)

  • Noh, Soonhyun;Hong, Seongsoo;Kim, Myungsun
    • The Journal of Korea Robotics Society
    • /
    • v.14 no.4
    • /
    • pp.340-347
    • /
    • 2019
  • Due to advances in machine intelligence and increased demands for autonomous machines, the complexity of the underlying software platform is increasing at a rapid pace, overwhelming the developers with implementation details. We attempt to ease the burden that falls onto the developers by creating a graphical programming framework we named Splash. Splash is designed to provide an effective programming abstraction for autonomous machines that require stream processing. It also enables programmers to specify genuine, end-to-end timing constraints, which the Splash framework automatically monitors for violation. By utilizing the timing constraints, Splash provides three key language semantics: timing semantics, in-order delivery semantics, and rate-controlled data-driven stream processing semantics. These three semantics together collectively serve as a conceptual tool that can hide low-level details from programmers, allowing developers to focus on the main logic of their applications. In this paper, we introduce the three-language semantics in detail and explain their function in association with Splash's language constructs. Furthermore, we present the internal workings of the Splash programming framework and validate its effectiveness via a lane keeping assist system.

Comparison between k-means and k-medoids Algorithms for a Group-Feature based Sliding Window Clustering (그룹특징기반 슬라이딩 윈도우 클러스터링에서의 k-means와 k-medoids 비교 평가)

  • Yang, Ju-Yon;Shim, Junho
    • The Journal of Society for e-Business Studies
    • /
    • v.23 no.3
    • /
    • pp.225-237
    • /
    • 2018
  • The demand for processing large data streams is growing rapidly as the generation and processing of large volumes of data become more popular. A variety of large data processing technologies are being developed to suit the increasing demand. One of the technologies that researchers have particularly observed is the data stream clustering with sliding windows. Data stream clustering with sliding windows may create a new set of clusters whenever the window moves. Previous data stream clustering techniques with sliding windows exploit the coresets, also known as group features that summarize the data. In this paper, we present some reformable elements of a group-feature based algorithm, and propose our algorithm that modified the clustering algorithm of the original one. We conduct a performance comparison between two algorithms by using different parameter values. Finally, we provide some guideline for the selective use of those algorithms with regard to the parameter values and their impacts on the performance.

Fast Visualization Technique and Visual Analytics System for Real-time Analyzing Stream Data (실시간 스트림 데이터 분석을 위한 시각화 가속 기술 및 시각적 분석 시스템)

  • Jeong, Seongmin;Yeon, Hanbyul;Jeong, Daekyo;Yoo, Sangbong;Kim, Seokyeon;Jang, Yun
    • Journal of the Korea Computer Graphics Society
    • /
    • v.22 no.4
    • /
    • pp.21-30
    • /
    • 2016
  • Risk management system should be able to support a decision making within a short time to analyze stream data in real time. Many analytical systems consist of CPU computation and disk based database. However, it is more problematic when existing system analyzes stream data in real time. Stream data has various production periods from 1ms to 1 hour, 1day. One sensor generates small data but tens of thousands sensors generate huge amount of data. If hundreds of thousands sensors generate 1GB data per second, CPU based system cannot analyze the data in real time. For this reason, it requires fast processing speed and scalability for analyze stream data. In this paper, we present a fast visualization technique that consists of hybrid database and GPU computation. In order to evaluate our technique, we demonstrate a visual analytics system that analyzes pipeline leak using sensor and tweet data.

Efficient Disk Access Method Using Region Storage Structure in Spatial Continuous Query Processing (공간 연속질의 처리에서 영역 기반의 저장 구조를 이용한 효율적인 디스크 접근 방법)

  • Chung, Weon-Il
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.12 no.5
    • /
    • pp.2383-2389
    • /
    • 2011
  • Ubiquitous applications require hybrid continuous query processing which processes both on-line data stream and spatial data in the disk. In the hybrid continuous spatial query processing, disk access costs for the high-volume spatial data should be minimized. However, previous indexing methods cannot reduce the disk seek time, because it is difficult that the data are stored in contiguity with others. Also, existing methods for the space-filling curve considering data cluster have the problem which does not cluster available data for queries. Therefore, we propose the region storage structure for efficient data access in hybrid continues spatial query processing. This paper shows that there is an obvious improvement of query processing costs through the contiguous data storing method and the group processing for user queries based on the region storage structure.

Design and Implementation of a Distributed Audio/Video Stream Service Framework based on CORBA (CORBA 기반의 분산 오디오/비디오 스트림 서비스 프레임워크의 설계 및 구현)

  • Kim, Jong-Hyeon;No, Yeong-Uk;Jeong, Gi-Dong
    • The KIPS Transactions:PartA
    • /
    • v.9A no.2
    • /
    • pp.207-216
    • /
    • 2002
  • This paper present a design and implementation of a distributed audio, Video stream service framework based on CORBA for efficient processing and control of audio/video stream. We design software components which support processing, control and transmission of audio/video streams as distributed objects. For optimization of stream transmission performance, we separate the transmission path of control data and media data. Distributed objects are defined by IDL and implemented using JAVA. And device dependent facilities like media capturing, playing and communication channels are implemented using JMF (Java Media Framework) components. We show a connection establishment and control procedure of streams communication. And for evaluation, we implement a test system and experiment a system performance. Our experiments show that test system has somewhat longer connection latency time compared to TCP connection establishment, but has optimized media transmission time compared to CORBA IIOP. Also test system show acceptable service quality of media transmission.

Historical Sensor Data Management Using Temporal Information (센서 데이터의 시간 정보를 이용한 이력 정보 관리)

  • Lee, Yang-Koo;Ryu, Keun-Ho
    • Journal of Korea Spatial Information System Society
    • /
    • v.10 no.4
    • /
    • pp.97-102
    • /
    • 2008
  • A wireless sensor network consists of many sensors that collect and transmit physical or environmental conditions at different locations to a server continuously. Many researches mainly focus on processing continuous queries on real-time data stream. However, they do not concern the problem of storing the historical data, which is mandatory to the historical queries. In this paper, we propose two time-based storage methods to store the sensor data stream and reduce the managed tuples without any loss of information, which lead to the improvement of the accuracy of query results.

  • PDF