• Title/Summary/Keyword: Data Stream Processing

Search Result 445, Processing Time 0.035 seconds

Data Structure and Visualization Algorithm in a Post-processing Program (가시화 프로그램에서의 데이터 구조와 가시화 알고리즘)

  • Na J. S.;Kim K. Y.;Kim B. S.
    • 한국전산유체공학회:학술대회논문집
    • /
    • 2003.08a
    • /
    • pp.82-87
    • /
    • 2003
  • Post-processing programs play an important role in the CFD data visualization and analysis. A variety of post-processing softwares have been developed and are being used in the CFD community. Developing a good quality of post-processing program requires dedication and efforts. In this paper an experience obtained through previous studies and developing post-processing programs are introduced which includes data structure and visualization algorithms.

  • PDF

The Design and Implementation of the Real-time Data Stream Server for Continuity of Care Record (실시간 헬스케어 시스템을 위한 데이터 스트림 서버의 설계 및 구현)

  • Wu, Zejun;Li, Yan;Bae, Hae-Young
    • Journal of the Korea Society of Computer and Information
    • /
    • v.16 no.12
    • /
    • pp.71-81
    • /
    • 2011
  • The EMR management services can monitoring the patients' record with any doctors in any hospital by using the internet and smartphones online. To handle the real time, multidimensional, continuous data, database management systems (DBMS) must cope with high insert rates for updates, however the traditional DBMS suffers from processing these kinds of data due to its serious design bottlenecks. So the researchers put forward to Data Stream Management System (DSMS). In this paper we describe the real-time Data Stream Server for Continuity of Care Record (CCR) that including continuos query processor. This system is compiled with DSMS and DBMS in EMR system for processing and monitoring the coming CCR data stream, and also storing the processed result with high-efficiency. The system enables users not only to query stored CCR information from DBMS, but also to execute continue query on real-time CCR Data Stream, and health information can be transferred between different healthcare providers that would reduce medical error. At last, we develop a IPhone mobile application to test the proposed real-time data stream server.

Performance Analysis of Siding Window based Stream High Utility Pattern Mining Methods (슬라이딩 윈도우 기반의 스트림 하이 유틸리티 패턴 마이닝 기법 성능분석)

  • Ryang, Heungmo;Yun, Unil
    • Journal of Internet Computing and Services
    • /
    • v.17 no.6
    • /
    • pp.53-59
    • /
    • 2016
  • Recently, huge stream data have been generated in real time from various applications such as wireless sensor networks, Internet of Things services, and social network services. For this reason, to develop an efficient method have become one of significant issues in order to discover useful information from such data by processing and analyzing them and employing the information for better decision making. Since stream data are generated continuously and rapidly, there is a need to deal with them through the minimum access. In addition, an appropriate method is required to analyze stream data in resource limited environments where fast processing with low power consumption is necessary. To address this issue, the sliding window model has been proposed and researched. Meanwhile, one of data mining techniques for finding meaningful information from huge data, pattern mining extracts such information in pattern forms. Frequency-based traditional pattern mining can process only binary databases and treats items in the databases with the same importance. As a result, frequent pattern mining has a disadvantage that cannot reflect characteristics of real databases although it has played an essential role in the data mining field. From this aspect, high utility pattern mining has suggested for discovering more meaningful information from non-binary databases with the consideration of the characteristics and relative importance of items. General high utility pattern mining methods for static databases, however, are not suitable for handling stream data. To address this issue, sliding window based high utility pattern mining has been proposed for finding significant information from stream data in resource limited environments by considering their characteristics and processing them efficiently. In this paper, we conduct various experiments with datasets for performance evaluation of sliding window based high utility pattern mining algorithms and analyze experimental results, through which we study their characteristics and direction of improvement.

Comparisions of stream activation mechanisms in computer based teleconferencing systems for low delay (지연 축소를 위한 컴퓨터 영상회의 시스템의 시트림 동작 구조 비교)

  • Lee, Gyeong-Hui;Kim, Du-Hyeon;Gang, Min-Gyu;Jeong, Chan-Geun
    • The Transactions of the Korea Information Processing Society
    • /
    • v.4 no.2
    • /
    • pp.363-376
    • /
    • 1997
  • In this paper, we present a hardware architecture and a sofrware architecture for cimputer based teleconferencing systems.And also we analyse stream adtivation mechanisms for them form the viewpoint of delay. MuX that is a multimedia I/O server provides various processing elements for data I/O, synchronization, interleaving and mixing.We describe methods to build teleconferencing systems with the elements and compares the technique using master click with the techniquie using self clock.In the plase of dta input.the technique using self click is berrer than the technique using master clock.When we generate interleved stream from audio and video stream and activate channel objects by periodic audio stream as activation clock, dealy from imput audio stream to imterleved stream is reduced but delay for video stream is not reduced as much as in the case of audio stream.

  • PDF

An Adaptive Grid-based Clustering Algorithm over Multi-dimensional Data Streams (적응적 격자기반 다차원 데이터 스트림 클러스터링 방법)

  • Park, Nam-Hun;Lee, Won-Suk
    • The KIPS Transactions:PartD
    • /
    • v.14D no.7
    • /
    • pp.733-742
    • /
    • 2007
  • A data stream is a massive unbounded sequence of data elements continuously generated at a rapid rate. Due to this reason, memory usage for data stream analysis should be confined finitely although new data elements are continuously generated in a data stream. To satisfy this requirement, data stream processing sacrifices the correctness of its analysis result by allowing some errors. The old distribution statistics are diminished by a predefined decay rate as time goes by, so that the effect of the obsolete information on the current result of clustering can be eliminated without maintaining any data element physically. This paper proposes a grid based clustering algorithm for a data stream. Given a set of initial grid cells, the dense range of a grid cell is recursively partitioned into a smaller cell based on the distribution statistics of data elements by a top down manner until the smallest cell, called a unit cell, is identified. Since only the distribution statistics of data elements are maintained by dynamically partitioned grid cells, the clusters of a data stream can be effectively found without maintaining the data elements physically. Furthermore, the memory usage of the proposed algorithm is adjusted adaptively to the size of confined memory space by flexibly resizing the size of a unit cell. As a result, the confined memory space can be fully utilized to generate the result of clustering as accurately as possible. The proposed algorithm is analyzed by a series of experiments to identify its various characteristics

Processing of Sensor Data Stream for OSGi Frameworks (OSGi를 위한 실시간 센서 데이터스트림 처리 방법)

  • Cha, Ji-Yun;Byun, Yung-Cheol;Lee, Dong-Cheal
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.13 no.5
    • /
    • pp.1014-1021
    • /
    • 2009
  • In an environment of home network where a number of technologies including heterogeneous hardware platforms, networking and protocols, middleware systems, and etc, exist, OSGi provides a platform for deployment and sharing of services managed in hardware and guarantees compatibility among applications. However, only simple control and processing of event data are considered in a home network using OSGi, and the consideration about real time processing of data stream generated by sensors is not enough. Therefore, researches allowing users to effectively develop OSGi applications by using various kinds of sensors generating data streams in the home network environment using OSGi are needed. In this paper, we propose an effective method of processing various types of real time data streams supplied to OSGi applications, including filtering, grouping, and counting, etc.

A novel window strategy for concept drift detection in seasonal time series (계절성 시계열 자료의 concept drift 탐지를 위한 새로운 창 전략)

  • Do Woon Lee;Sumin Bae;Kangsub Kim;Soonhong An
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2023.05a
    • /
    • pp.377-379
    • /
    • 2023
  • Concept drift detection on data stream is the major issue to maintain the performance of the machine learning model. Since the online stream is to be a function of time, the classical statistic methods are hard to apply. In particular case of seasonal time series, a novel window strategy with Fourier analysis however, gives a chance to adapt the classical methods on the series. We explore the KS-test for an adaptation of the periodic time series and show that this strategy handles a complicate time series as an ordinary tabular dataset. We verify that the detection with the strategy takes the second place in time delay and shows the best performance in false alarm rate and detection accuracy comparing to that of arbitrary window sizes.

A study of STB software development for streaming synchronized data processing (스트리밍 동기화 데이터 처리를 위한 단말 소프트웨어 개발에 관한 연구)

  • 신중목;유지상
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.29 no.6A
    • /
    • pp.690-696
    • /
    • 2004
  • Advanced Television Systems Committee (ATSC) -A/90, which is a standard for terrestrial data transmission in Korea, defines synchronized data that has a strong timing association with a separate Program Element. It is classified as synchronized streaming data that is carried in packetized elementary stream (PES) packets or a synchronized non-streaming data that shall be carried in digital storage media command and control (DSM-CC) section. In this paper, we study the design and verification of synchronized streaming data processing algorithm based on ATSC -A/90. We designed a parser and a player for the algorithm development. The received PES packet including synchronized streaming data is parsed in the parser. The parsed synchronized streaming data is synchronized and displayed by player. Finally, we ascertained that STB was working properly with MPEG-2 transport stream (TS) containing synchronized streaming data, as the proposed algorithm is implemented on a set-top box.

Incremental Processing Scheme for Graph Streams Considering Data Reuse (데이터 재사용을 고려한 그래프 스트림의 점진적 처리 기법)

  • Cho, Jungkweon;Han, Jinsu;Kim, Minsoo;Choi, Dojin;Bok, Kyoungsoo;Yoo, Jaesoo
    • The Journal of the Korea Contents Association
    • /
    • v.18 no.1
    • /
    • pp.465-475
    • /
    • 2018
  • Recently, as the use of social media and IoT has increased, large graph streams has been generating and studies on real-time processing for them have been actively carrying out. In this paper we propose a incremental graph stream processing scheme that reuses previous result data when the graph changes continuously. We also propose a cost model to selectively perform incremental processing and static processing. The proposed cost model computes the predicted value of the detection cost and the processing cost of the recalculation area based on the actually processed history and performs the incremental processing when the incremental processing is more profit than the static processing. The proposed incremental processing increases the efficiency by processing only the part that changes when the graph update occurs. Also, by collecting only the previous result data of the changed part and performing the incremental processing, the disk I/O costs are reduced. It is shown through various performance evaluations that the proposed scheme outperforms the existing schemes.

H*-tree/H*-cubing-cubing: Improved Data Cube Structure and Cubing Method for OLAP on Data Stream (H*-tree/H*-cubing: 데이터 스트림의 OLAP를 위한 향상된 데이터 큐브 구조 및 큐빙 기법)

  • Chen, Xiangrui;Li, Yan;Lee, Dong-Wook;Kim, Gyoung-Bae;Bae, Hae-Young
    • The KIPS Transactions:PartD
    • /
    • v.16D no.4
    • /
    • pp.475-486
    • /
    • 2009
  • Data cube plays an important role in multi-dimensional, multi-level data analysis. Meeting on-line analysis requirements of data stream, several cube structures have been proposed for OLAP on data stream, such as stream cube, flowcube, S-cube. Since it is costly to construct data cube and execute ad-hoc OLAP queries, more research works should be done considering efficient data structure, query method and algorithms. Stream cube uses H-cubing to compute selected cuboids and store the computed cells in an H-tree, which form the cuboids along popular-path. However, the H-tree layoutis disorderly and H-cubing method relies too much on popular path.In this paper, first, we propose $H^*$-tree, an improved data structure, which makes the retrieval operation in tree structure more efficient. Second, we propose an improved cubing method, $H^*$-cubing, with respect to computing the cuboids that cannot be retrieved along popular-path when an ad-hoc OLAP query is executed. $H^*$-tree construction and $H^*$-cubing algorithms are given. Performance study turns out that during the construction step, $H^*$-tree outperforms H-tree with a more desirable trade-off between time and memory usage, and $H^*$-cubing is better adapted to ad-hoc OLAP querieswith respect to the factors such as time and memory space.