• Title/Summary/Keyword: Stream Processing System

Search Result 293, Processing Time 0.027 seconds

The Method for Real-time Complex Event Detection of Unstructured Big data (비정형 빅데이터의 실시간 복합 이벤트 탐지를 위한 기법)

  • Lee, Jun Heui;Baek, Sung Ha;Lee, Soon Jo;Bae, Hae Young
    • Spatial Information Research
    • /
    • v.20 no.5
    • /
    • pp.99-109
    • /
    • 2012
  • Recently, due to the growth of social media and spread of smart-phone, the amount of data has considerably increased by full use of SNS (Social Network Service). According to it, the Big Data concept is come up and many researchers are seeking solutions to make the best use of big data. To maximize the creative value of the big data held by many companies, it is required to combine them with existing data. The physical and theoretical storage structures of data sources are so different that a system which can integrate and manage them is needed. In order to process big data, MapReduce is developed as a system which has advantages over processing data fast by distributed processing. However, it is difficult to construct and store a system for all key words. Due to the process of storage and search, it is to some extent difficult to do real-time processing. And it makes extra expenses to process complex event without structure of processing different data. In order to solve this problem, the existing Complex Event Processing System is supposed to be used. When it comes to complex event processing system, it gets data from different sources and combines them with each other to make it possible to do complex event processing that is useful for real-time processing specially in stream data. Nevertheless, unstructured data based on text of SNS and internet articles is managed as text type and there is a need to compare strings every time the query processing should be done. And it results in poor performance. Therefore, we try to make it possible to manage unstructured data and do query process fast in complex event processing system. And we extend the data complex function for giving theoretical schema of string. It is completed by changing the string key word into integer type with filtering which uses keyword set. In addition, by using the Complex Event Processing System and processing stream data at real-time of in-memory, we try to reduce the time of reading the query processing after it is stored in the disk.

Development of a Hash Function and a Stream Cipher and Their Applications to the GSM Security System (해쉬함수와 스트림 암호기의 개발 및 GSM 보안 시스템에의 적용)

  • Kim, Bun-Sik;Shin, In-Chul
    • The Transactions of the Korea Information Processing Society
    • /
    • v.7 no.8
    • /
    • pp.2421-2429
    • /
    • 2000
  • With the advance of wireless communications technology, mobile communications have become more convenient than ever. Nowadays, people can communicate with each other on any place at any time. However, because of the openness of wireless communications, the way to protect the privacy between communicating parties is becoming a very important issue. In this paper, we present a study on the authentication and message encryption algorithm to support roaming service to the GSM network. To propose an authentication and message encryption algorithm applicable to the GSM system, the security architecture of the GSM outlined in the GSM standard is briefly introduced. In the proposed cryptosystems we use a new hash function for user authentication and a stream cipher based on Linear Feedback Shift Register(LFSR) for message encryption and decryption. Moreover, each algorithm is programmed with C language and simulated on IBM-PC system and we analyze the randomness properties of the proposed algorithms by using statistical tests.

  • PDF

Design of Stream Data Processing Operators for Real-time Activity Recognition System (실시간 행위인식 시스템을 위한 스트림데이터 처리기 연산자 설계)

  • Jeong, Jongyoung;Lee, Young-Koo;Han, Yong-koo
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2009.04a
    • /
    • pp.302-305
    • /
    • 2009
  • 본 논문은 오프라인 환경에서 연구되던 행위인식 연구들이 온라인 환경에서 통합되어 동작할 수 있도록 하기 위해 스트림데이터 처리기를 이용한 행위인식 시스템을 제안하였다. 스트림데이터 처리는 멀티 센서 환경에서의 방대한 센서 데이터를 처리하기 위한 기술로서, 행위인식 시스템의 실시간으로 발생하는 데이터를 즉각 처리하여 결과를 추론하여야 한다는 요구사항을 만족시킬 수 있다. 이를 위해 행위인식에 사용되는 필수적인 연산을 정의하여 스트림데이터 처리기에 추가할 수 있도록 행위 인식에 필요한 각종 연산자를 설계하였다.

Real Time Distributed Parallel Processing to Visualize Noise Map with Big Sensor Data and GIS Data for Smart Cities (스마트시티의 빅 센서 데이터와 빅 GIS 데이터를 융합하여 실시간 온라인 소음지도로 시각화하기 위한 분산병렬처리 방법론)

  • Park, Jong-Won;Sim, Ye-Chan;Jung, Hae-Sun;Lee, Yong-Woo
    • Journal of Internet Computing and Services
    • /
    • v.19 no.4
    • /
    • pp.1-6
    • /
    • 2018
  • In smart cities, data from various kinds of sensors are collected and processed to provide smart services to the citizens. Noise information services with noise maps using the collected sensor data from various kinds of ubiquitous sensor networks is one of them. This paper presents a research result which generates three dimensional (3D) noise maps in real-time for smart cities. To make a noise map, we have to converge many informal data which include big image data of geographical Information and massive sensor data. Making such a 3D noise map in real-time requires the processing of the stream data from the ubiquitous sensor networks in real-time and the convergence operation in real-time. They are very challenging works. We developed our own methodology for real-time distributed and parallel processing for it and present it in this paper. Further, we developed our own real-time 3D noise map generation system, with the methodology. The system uses open source softwares for it. Here in this paper, we do introduce one of our systems which uses Apache Storm. We did performance evaluation using the developed system. Cloud computing was used for the performance evaluation experiments. It was confirmed that our system was working properly with good performance and the system can produce the 3D noise maps in real-time. The performance evaluation results are given in this paper, as well.

Access Control of Pay TV Program in Digital Satellite Broadcasting System (디지털 위성방송 시스템에서 유료 TV방송 프로그램 접근제어)

  • Park, Jeong-Hyun;Lee, Sang-Ho
    • The Transactions of the Korea Information Processing Society
    • /
    • v.4 no.12
    • /
    • pp.3123-3132
    • /
    • 1997
  • In this paper, we describe access control system for protection of pay TV program in digital DBS(Direct Broadcast satellite) system. We also propose a possible access control system and operation scenario for scrambling and descrambling which are important in access control system. Transport stream structure and option, entitlement checking message and entitlement management message for access control on digital broadcasting system are described in this paper. Especially, the authentication based on Flat-Shamir and Gulllou-Quisquater schemes required for verification of proper subscriber as access control is oriented to smart card number and subscriber ID(Identity). It has less restriction than scheme oriented to descrambler number.

  • PDF

IMAGE DATA CHAIN ANALYSIS FOR SATELLITE CAMERA ELECTRONIC SYSTEM

  • Park, Jong-Euk;Kong, Jong-Pil;Heo, Haeng-Pal;Kim, Young-Sun;Chang, Young-Jun
    • Proceedings of the KSRS Conference
    • /
    • v.2
    • /
    • pp.791-793
    • /
    • 2006
  • In the satellite camera, the incoming light source is converted to electronic analog signals by the electronic component for example CCD (Charge Coupled Device) detectors. The analog signals are amplified, biased and converted into digital signals (pixel data stream) in the video processor (A/Ds). The outputs of the A/Ds are digitally multiplexed and driven out using differential line drivers (two pairs of wires) for cross strap requirement. The MSC (Multi-Spectral Camera) in the KOMPSAT-2 which is a LEO spacecraft will be used to generate observation imagery data in two main channels. The MSC is to obtain data for high-resolution images by converting incoming light from the earth into digital stream of pixel data. The video data outputs are then MUXd, converted to 8 bit bytes, serialized and transmitted to the NUC (Non-Uniformity Correction) module by the Hotlink data transmitter. In this paper, the video data streams, the video data format, and the image data processing routine for satellite camera are described in terms of satellite camera control hardware. The advanced satellite with very high resolution requires faster and more complex image data chain than this algorithm. So, the effective change of the used image data chain and the fast video data transmission method are discussed in this paper

  • PDF

Implementation and Design of Document Class Editor based on ODA (ODA에 근거한 문서 클래스 에디터 설계 및 구현)

  • 정회경;이수연
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.17 no.12
    • /
    • pp.1412-1422
    • /
    • 1992
  • This paper describes an implementation of the document class editor based on ODA(Open Document Architecture). For processing, we divided document structure into generic logical structure and generic layout structure as ODA standard. Also this editor could edit document profile. Using the utility which was implemented to investigate the composed document by object. we confirmed the document. And we could verify the ODIF stream data of the document. We designed this editor based on DAP level 2 of international functional standard. This system was implemented in environment of the X window system and the Motif as graphical user interface. This document class editor will be used to create real document having specific document structure.

  • PDF

A Study on the Symmetric Hybrid Cryptosystem Design for Adaptation of Network Environment (네트워크 환경에 적용하기 위한 대칭형 혼합형 암호시스템 설계에 관한 연구)

  • Jeong, Woo-Yeol;Lee, Seon-Keun
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.2 no.3
    • /
    • pp.150-156
    • /
    • 2007
  • In this paper, we studied security systems for information security of several systems that use in network environment along with information society. Therefore, we designed symmetry style base mixing style cryptographic system that apply block and stream way to solve problems of complexity and lower processing speed etc. Symmetry style base mixing style cryptographic system including authentication operation holds performance that the processing speed and the calculation amount are more superior than asymmetry style. Result that design system by Synopsys 1999.10 and ALTERA MaxPlus 10.1 and do simulation, mixing style password system that we propose is that information security offers very efficient assistance and performance in necessary field in network environment.

  • PDF

UbiCore : An Effective XML-based RFID Middleware System (UbiCore : XML 기반 RFID 미들웨어 시스템)

  • Lee, Hun-Soon;Choi, Hyun-Hwa;Kim, Byoung-Seob;Lee, Myung-Cheol;Park, Jae-Hong;Lee, Mi-Young;Kim, Myung-Joon;Jin, Sung-Il
    • Journal of KIISE:Databases
    • /
    • v.33 no.6
    • /
    • pp.578-589
    • /
    • 2006
  • Owing to the proliferation of Radio Frequency Identification (RFID) technologies which is being watched as a core technology of ubiquitous computing, applications which offer convenience to people using RFID technologies are more and more increased. To easily develop these applications, a middleware system which acts as a bridge between RFID hardware and application is essential. In this paper, we propose a novel XML-based RFID middleware system called UbiCore (Ubiquitous Core). UbiCore has following features: First, UbiCore employs its own query language called XQueryStream (XQuery for Stream Data) which is originated from XQuery. Second, UbiCore has the preprocessing phase called pre-filtering prior to query evaluation and reuses the intermediate result of previous evaluation to speed up the processing of RFID tag data stream. Third, UbiCore supports query on both continuously generated stream data and archived historical data. And last, UbiCore offers a distinct markup language called Context-driven Service Markup Language (CSML) to easily specify the linking information between context and service.

Finding Weighted Sequential Patterns over Data Streams via a Gap-based Weighting Approach (발생 간격 기반 가중치 부여 기법을 활용한 데이터 스트림에서 가중치 순차패턴 탐색)

  • Chang, Joong-Hyuk
    • Journal of Intelligence and Information Systems
    • /
    • v.16 no.3
    • /
    • pp.55-75
    • /
    • 2010
  • Sequential pattern mining aims to discover interesting sequential patterns in a sequence database, and it is one of the essential data mining tasks widely used in various application fields such as Web access pattern analysis, customer purchase pattern analysis, and DNA sequence analysis. In general sequential pattern mining, only the generation order of data element in a sequence is considered, so that it can easily find simple sequential patterns, but has a limit to find more interesting sequential patterns being widely used in real world applications. One of the essential research topics to compensate the limit is a topic of weighted sequential pattern mining. In weighted sequential pattern mining, not only the generation order of data element but also its weight is considered to get more interesting sequential patterns. In recent, data has been increasingly taking the form of continuous data streams rather than finite stored data sets in various application fields, the database research community has begun focusing its attention on processing over data streams. The data stream is a massive unbounded sequence of data elements continuously generated at a rapid rate. In data stream processing, each data element should be examined at most once to analyze the data stream, and the memory usage for data stream analysis should be restricted finitely although new data elements are continuously generated in a data stream. Moreover, newly generated data elements should be processed as fast as possible to produce the up-to-date analysis result of a data stream, so that it can be instantly utilized upon request. To satisfy these requirements, data stream processing sacrifices the correctness of its analysis result by allowing some error. Considering the changes in the form of data generated in real world application fields, many researches have been actively performed to find various kinds of knowledge embedded in data streams. They mainly focus on efficient mining of frequent itemsets and sequential patterns over data streams, which have been proven to be useful in conventional data mining for a finite data set. In addition, mining algorithms have also been proposed to efficiently reflect the changes of data streams over time into their mining results. However, they have been targeting on finding naively interesting patterns such as frequent patterns and simple sequential patterns, which are found intuitively, taking no interest in mining novel interesting patterns that express the characteristics of target data streams better. Therefore, it can be a valuable research topic in the field of mining data streams to define novel interesting patterns and develop a mining method finding the novel patterns, which will be effectively used to analyze recent data streams. This paper proposes a gap-based weighting approach for a sequential pattern and amining method of weighted sequential patterns over sequence data streams via the weighting approach. A gap-based weight of a sequential pattern can be computed from the gaps of data elements in the sequential pattern without any pre-defined weight information. That is, in the approach, the gaps of data elements in each sequential pattern as well as their generation orders are used to get the weight of the sequential pattern, therefore it can help to get more interesting and useful sequential patterns. Recently most of computer application fields generate data as a form of data streams rather than a finite data set. Considering the change of data, the proposed method is mainly focus on sequence data streams.