• Title/Summary/Keyword: stream processing

Search Result 720, Processing Time 0.029 seconds

Load Shedding via Predicting the Frequency of Tuple for Efficient Analsis over Data Streams (효율적 데이터 스트림 분석을 위한 발생빈도 예측 기법을 이용한 과부하 처리)

  • Chang, Joong-Hyuk
    • The KIPS Transactions:PartD
    • /
    • v.13D no.6 s.109
    • /
    • pp.755-764
    • /
    • 2006
  • In recent, data streams are generated in various application fields such as a ubiquitous computing and a sensor network, and various algorithms are actively proposed for processing data streams efficiently. They mainly focus on the restriction of their memory usage and minimization of their processing time per data element. However, in the algorithms, if data elements of a data stream are generated in a rapid rate for a time unit, some of the data elements cannot be processed in real time. Therefore, an efficient load shedding technique is required to process data streams effcientlv. For this purpose, a load shedding technique over a data stream is proposed in this paper, which is based on the predicting technique of the frequency of data element considering its current frequency. In the proposed technique, considering the change of the data stream, its threshold for tuple alive is controlled adaptively. It can help to prevent unnecessary load shedding.

Efficient Query Indexing for Short Interval Query (짧은 구간을 갖는 범위 질의의 효율적인 질의 색인 기법)

  • Kim, Jae-In;Song, Myung-Jin;Han, Dae-Young;Kim, Dae-In;Hwang, Bu-Hyun
    • The KIPS Transactions:PartD
    • /
    • v.16D no.4
    • /
    • pp.507-516
    • /
    • 2009
  • In stream data processing system, generally the interval queries are in advance registered in the system. When a data is input to the system continuously, for realtime processing, a query indexing method is used to quickly search queries. Thus, a main memory-based query index with a small storage cost and a fast search time is needed for searching queries. In this paper, we propose a LVC-based(Limited Virtual Construct-based) query index method using a hashing to meet the both needs. In LVC-based query index, we divide the range of a stream into limited virtual construct, or LVC. We map each interval query to its corresponding LVC and the query ID is stored on each LVC. We have compared with the CEI-based query indexing method through the simulation experiment. When the range of values of input stream is broad and there are many short interval queries, the LVC-based indexing method have shown the performance enhancement for the storage cost and search time.

Hazelcast Vs. Ignite: Opportunities for Java Programmers

  • Maxim, Bartkov;Tetiana, Katkova;S., Kruglyk Vladyslav;G., Murtaziev Ernest;V., Kotova Olha
    • International Journal of Computer Science & Network Security
    • /
    • v.22 no.2
    • /
    • pp.406-412
    • /
    • 2022
  • Storing large amounts of data has always been a big problem from the beginning of computing history. Big Data has made huge advancements in improving business processes by finding the customers' needs using prediction models based on web and social media search. The main purpose of big data stream processing frameworks is to allow programmers to directly query the continuous stream without dealing with the lower-level mechanisms. In other words, programmers write the code to process streams using these runtime libraries (also called Stream Processing Engines). This is achieved by taking large volumes of data and analyzing them using Big Data frameworks. Streaming platforms are an emerging technology that deals with continuous streams of data. There are several streaming platforms of Big Data freely available on the Internet. However, selecting the most appropriate one is not easy for programmers. In this paper, we present a detailed description of two of the state-of-the-art and most popular streaming frameworks: Apache Ignite and Hazelcast. In addition, the performance of these frameworks is compared using selected attributes. Different types of databases are used in common to store the data. To process the data in real-time continuously, data streaming technologies are developed. With the development of today's large-scale distributed applications handling tons of data, these databases are not viable. Consequently, Big Data is introduced to store, process, and analyze data at a fast speed and also to deal with big users and data growth day by day.

Incremental Processing Scheme for Graph Streams Considering Data Reuse (데이터 재사용을 고려한 그래프 스트림의 점진적 처리 기법)

  • Cho, Jungkweon;Han, Jinsu;Kim, Minsoo;Choi, Dojin;Bok, Kyoungsoo;Yoo, Jaesoo
    • The Journal of the Korea Contents Association
    • /
    • v.18 no.1
    • /
    • pp.465-475
    • /
    • 2018
  • Recently, as the use of social media and IoT has increased, large graph streams has been generating and studies on real-time processing for them have been actively carrying out. In this paper we propose a incremental graph stream processing scheme that reuses previous result data when the graph changes continuously. We also propose a cost model to selectively perform incremental processing and static processing. The proposed cost model computes the predicted value of the detection cost and the processing cost of the recalculation area based on the actually processed history and performs the incremental processing when the incremental processing is more profit than the static processing. The proposed incremental processing increases the efficiency by processing only the part that changes when the graph update occurs. Also, by collecting only the previous result data of the changed part and performing the incremental processing, the disk I/O costs are reduced. It is shown through various performance evaluations that the proposed scheme outperforms the existing schemes.

Energy-efficient Broadcasting of XML Data in Mobile Computing Environments (이동 컴퓨팅 환경에서 XML 데이타의 에너지 효율적인 방송)

  • Kim Chung Soo;Park Chang-Sup;Chung Yon Dohn
    • Journal of KIISE:Databases
    • /
    • v.33 no.1
    • /
    • pp.117-128
    • /
    • 2006
  • In this paper, we propose a streaming method for XML data that supports energy-efficient processing of queries over the stream in mobile clients. We propose new stream organizations for XML data which have different kinds of addresses to related data in a stream. We describe event-driven stream generation algorithms for the proposed stream structures and provide search algorithms for simple XML path queries which leverage the access mechanisms incorporated in the stream. Experimental results show that our approaches can effectively improve the tuning time performance of user queries in a wireless broadcasting environment.

Analysis of several VERA benchmark problems with the photon transport capability of STREAM

  • Mai, Nhan Nguyen Trong;Kim, Kyeongwon;Lemaire, Matthieu;Nguyen, Tung Dong Cao;Lee, Woonghee;Lee, Deokjung
    • Nuclear Engineering and Technology
    • /
    • v.54 no.7
    • /
    • pp.2670-2689
    • /
    • 2022
  • STREAM - a lattice transport calculation code with method of characteristics for the purpose of light water reactor analysis - has been developed by the Computational Reactor Physics and Experiment laboratory (CORE) of the Ulsan National Institute of Science and Technology (UNIST). Recently, efforts have been taken to develop a photon module in STREAM to assess photon heating and the influence of gamma photon transport on power distributions, as only neutron transport was considered in previous STREAM versions. A multi-group photon library is produced for STREAM based on the ENDF/B-VII.1 library with the use of the library-processing code NJOY. The developed photon solver for the computation of 2D and 3D distributions of photon flux and energy deposition is based on the method of characteristics like the neutron solver. The photon library and photon module produced and implemented for STREAM are verified on VERA pin and assembly problems by comparison with the Monte Carlo code MCS - also developed at UNIST. A short analysis of the impact of photon transport during depletion and thermal hydraulics feedback is presented for a 2D core also from the VERA benchmark.

Redesign of Stream Cipher Salsa20/8 (스트림 암호 Salsa20/8의 재설계)

  • Kim, Gil-Ho;Kim, Sung-Gi;Cho, Gyeong-Yeon
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.18 no.8
    • /
    • pp.1904-1913
    • /
    • 2014
  • Was develop 256bit output stream cipher of improving for same key reuse prohibition and integrity. The developed stream cipher used Salsa20 round function was implemented to hardware of applying a 5-stage pipeline architecture, such as WSN and DMB for real-time processing can satisfy the speed and security requirements.

Design of the Entropy Processor using the Memory Stream Allocation for the Image Processing (메모리 스트림 할당 기법을 이용한 영상처리용 엔트로피 프로세서 설계)

  • Lee, Seon-Keun;Jeong, Woo-Yeol
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.7 no.5
    • /
    • pp.1017-1026
    • /
    • 2012
  • Due to acceleration of the IT industry and the environment for a variety of media in modern society, such as real-time video images 3D-TV is a very important issue. These high-quality live video is being applied to various fields such as CCTV footage has become an important performance parameters. However, these high quality images, even vulnerable because of shortcomings secure channel or by using various security algorithms attempt to get rid of these disadvantages are underway very active. These shortcomings, this study added extra security technologies to reduce the processing speed image processing itself, but by adding security features to transmit real-time processing and security measures for improving the present.

Data Structure and Visualization Algorithm in a Post-processing Program (가시화 프로그램에서의 데이터 구조와 가시화 알고리즘)

  • Na J. S.;Kim K. Y.;Kim B. S.
    • 한국전산유체공학회:학술대회논문집
    • /
    • 2003.08a
    • /
    • pp.82-87
    • /
    • 2003
  • Post-processing programs play an important role in the CFD data visualization and analysis. A variety of post-processing softwares have been developed and are being used in the CFD community. Developing a good quality of post-processing program requires dedication and efforts. In this paper an experience obtained through previous studies and developing post-processing programs are introduced which includes data structure and visualization algorithms.

  • PDF

Efficient Processing of Multidimensional Sensor stream Data in Digital Marine Vessel (디지털 선박 내 다차원 센서 스트림 데이터의 효율적인 처리)

  • Song, Byoung-Ho;Park, Kyung-Woo;Lee, Jin-Seok;Lee, Keong-Hyo;Jung, Min-A;Lee, Sung-Ro
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.35 no.5B
    • /
    • pp.794-800
    • /
    • 2010
  • It is necessary to accurate and efficient management for measured digital data from various sensors in digital marine vessel. It is not efficient that sensor network process input stream data of mass storage stored in database the same time. In this paper, We propose to improve the processing performance of multidimensional stream data continuous incoming from multiple sensor. We propose that we arrange some sensors (temperature, humidity, lighting, voice) and process query based on sliding window for efficient input stream and found multiple query plan to Mjoin method and we reduce stored data using SVM algorithm. We automatically delete that it isn't necessary to the data from the database and we used to ship diagnosis system for available data. As a result, we obtained to efficient result about 18.3% reduction rate of database using 35,912 data sets.