• Title/Summary/Keyword: Distributed Stream Processing

Search Result 55, Processing Time 0.022 seconds

A Sliding Window-based Multivariate Stream Data Classification (슬라이딩 윈도우 기반 다변량 스트림 데이타 분류 기법)

  • Seo, Sung-Bo;Kang, Jae-Woo;Nam, Kwang-Woo;Ryu, Keun-Ho
    • Journal of KIISE:Databases
    • /
    • v.33 no.2
    • /
    • pp.163-174
    • /
    • 2006
  • In distributed wireless sensor network, it is difficult to transmit and analyze the entire stream data depending on limited networks, power and processor. Therefore it is suitable to use alternative stream data processing after classifying the continuous stream data. We propose a classification framework for continuous multivariate stream data. The proposed approach works in two steps. In the preprocessing step, it takes input as a sliding window of multivariate stream data and discretizes the data in the window into a string of symbols that characterize the signal changes. In the classification step, it uses a standard text classification algorithm to classify the discretized data in the window. We evaluated both supervised and unsupervised classification algorithms. For supervised, we tested Bayesian classifier and SVM, and for unsupervised, we tested Jaccard, TFIDF Jaro and Jaro Winkler. In our experiments, SVM and TFIDF outperformed other classification methods. In particular, we observed that classification accuracy is improved when the correlation of attributes is also considered along with the n-gram tokens of symbols.

PnP Supporting Middleware Framework for Network Based Humanoid (네트워크 기반 휴머노이드에서의 PnP가 가능한 미들웨어 프레임워크)

  • Lee, Ho-Dong;Kim, Dong-Won;Kim, Joo-Hyung;Park, Gwi-Tae
    • The Journal of Korea Robotics Society
    • /
    • v.3 no.3
    • /
    • pp.255-261
    • /
    • 2008
  • This paper describes a network framework that support network based humanoid. The framework utilizes middleware such as CORBA (ACE/TAO) that provides PnP capability for network based humanoid. The network framework transfers data gathered from a network based humanoid to a processing group that is distributed on a network. The data types are video stream, audio stream and control data. Also, the network framework transfers service data produced by the processing group to the network based humanoid. By using this network framework, the network based humanoid can provide high quality of intelligent services to user.

  • PDF

A Proposal of Event Stream Processing Frameworks applicable to Asynchronous-based Microservice (비동기 기반 마이크로 서비스에 적용 가능한 이벤트 스트림 처리 프레임워크 제안)

  • Park, Sang Il
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.17 no.2
    • /
    • pp.45-50
    • /
    • 2017
  • Micro-service Architecture is a service architecture optimized for large-scale distributed systems such as real-time realistic broadcasting systems, so that are fiercely adopted by Global leading service platform vendors such as Netflix and Twitter due to the merit of horizontal performance scalability enabling the scale-out technique. In addition, micro-service architecture makes it possible to execute image processing and real-time data analysis using an asynchronous-based processing that are difficult to handle in Web API such as REST. In this paper, an event stream processing framework applicable to asynchronous based micro services is proposed in the sense that the accountability of event processing order is not guaranteed in the events such as IoT sensor data analysis or cloud-based image editing because these are the situations where the real-time media editing generates multiple event streams and asynchronous processes in the platform.

Hazelcast Vs. Ignite: Opportunities for Java Programmers

  • Maxim, Bartkov;Tetiana, Katkova;S., Kruglyk Vladyslav;G., Murtaziev Ernest;V., Kotova Olha
    • International Journal of Computer Science & Network Security
    • /
    • v.22 no.2
    • /
    • pp.406-412
    • /
    • 2022
  • Storing large amounts of data has always been a big problem from the beginning of computing history. Big Data has made huge advancements in improving business processes by finding the customers' needs using prediction models based on web and social media search. The main purpose of big data stream processing frameworks is to allow programmers to directly query the continuous stream without dealing with the lower-level mechanisms. In other words, programmers write the code to process streams using these runtime libraries (also called Stream Processing Engines). This is achieved by taking large volumes of data and analyzing them using Big Data frameworks. Streaming platforms are an emerging technology that deals with continuous streams of data. There are several streaming platforms of Big Data freely available on the Internet. However, selecting the most appropriate one is not easy for programmers. In this paper, we present a detailed description of two of the state-of-the-art and most popular streaming frameworks: Apache Ignite and Hazelcast. In addition, the performance of these frameworks is compared using selected attributes. Different types of databases are used in common to store the data. To process the data in real-time continuously, data streaming technologies are developed. With the development of today's large-scale distributed applications handling tons of data, these databases are not viable. Consequently, Big Data is introduced to store, process, and analyze data at a fast speed and also to deal with big users and data growth day by day.

Research Directions for Efficient Query Processing over Sensor Data Streams (센서 데이터 스트림 환경에서 효율적인 질의처리 연구방향)

  • An, Dong-Chan
    • KSCI Review
    • /
    • v.14 no.2
    • /
    • pp.199-204
    • /
    • 2006
  • The sensor network is a wireless network of the sensor nodes which sensing, computation and communication ability. Each sensor nodes create the data items by sensor nodes above one. Like this feature, the sensor network is similar to distributed data base system. The sensor node of the sensor network is restricted from the power and the memory resources is the biggest weak point and is becoming the important research object. In this paper, We try to see efficient sensor data stream management method and efficient query processing method under the restricted sensor network environment.

  • PDF

Distributed Stream Processing System with apache Hadoop for PTAM on Xeon Phi Cluster (PTAM을 위한 제온파이 기반 하둡 분산 스트림 프로세싱 시스템)

  • Seo, Jae Min;Cho, Kyu Nam;Kim, Do Hyung;Jeong, Chang-Sung
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2015.10a
    • /
    • pp.184-186
    • /
    • 2015
  • 본 논문에서는 PTAM을 위한 새로운 분산 스트림 프로세싱 시스템을 제안한다. PTAM은 하나의 시스템에서 동작하도록 설계되었다. 이는 PTAM이 가지고 있는 한계점을 말해주는 부분인데, PTAM은 Bundle Adjustment의 계산 부하가 커지는 경우에 map을 구축하는데 있어 많은 시간과 리소스가 필요하다. 이에 하둡을 통해 계산 부하를 분산하고, PE(Processing Element)를 Xeon phi 시스템을 통해 동작되는 시스템을 제안한다.

Design and Implementation of a Fully Synthesizable Bluetooth Baseband Module Considering IP Reuse

  • Chun, Ik-Jae;Kim, Bo-Gwan
    • Proceedings of the IEEK Conference
    • /
    • 2002.07b
    • /
    • pp.1304-1307
    • /
    • 2002
  • In this paper, we describe the structure and the test results of a Bluetooth baseband module we have developed. The module has a distributed buffer, i.e. FIFO, for data stream. Bus interface of the module is designed on the basis of interface of microprocessor widely used and the external interface is designed to consider chips connected directly. Since the module performs as many hardware efficient tasks as possible, processing load of microprocessor is very small. It can also be controlled either by software or by hardware for flexibility. The fully synthesizable baseband module was fabricated in a $0.25\mu\textrm{m}$ CMOS technology occupying $2.79\times2.8{\textrm{mm}^2}$ area. And an FPGA implementation of this module is tested for file and bit-stream transfers between PCs.

  • PDF

Efficient k-Nearest Neighbor Query Processing Method for a Large Location Data (대용량 위치 데이터에서 효율적인 k-최근접 질의 처리 기법)

  • Choi, Dojin;Lim, Jongtae;Yoo, Seunghun;Bok, Kyoungsoo;Yoo, Jaesoo
    • The Journal of the Korea Contents Association
    • /
    • v.17 no.8
    • /
    • pp.619-630
    • /
    • 2017
  • With the growing popularity of smart devices, various location based services have been providing to users. Recently, some location based social applications that combine social services and location based services have been emerged. The demands of a k-nearest neighbors(k-NN) query which finds k closest locations from a user location are increased in the location based social network services. In this paper, we propose an approximate k-NN query processing method for fast response time in a large number of users environments. The proposed method performs efficient stream processing using big data distributed processing technologies. In this paper, we also propose a modified grid index method for indexing a large amount of location data. The proposed query processing method first retrieves the related cells by considering a user movement. By doing so, it can make an approximate k results set. In order to show the superiority of the proposed method, we conduct various performance evaluations with the existing method.

Litter Processing in Tropical Headwater Streams : Potential Importance of Palm Fruit Fall and Frond Fall

  • Covich, Alan P.
    • The Korean Journal of Ecology
    • /
    • v.23 no.2
    • /
    • pp.113-116
    • /
    • 2000
  • Different phenological patterns of leaf and fruit fall among native and non-native riparian species provide a spatially and temporally heterogeneous series of alternative food resources for detritivores. Relatively little is known about qualitative differences among these different riparian species. Rates of litter inputs, decomposition, and retention for different sources of riparian litter require long-term documentation. Species of freshwater shrimps, crabs. insects. and gastropods are known to consume a wide range of litter inputs but how these dynamic food webs function under changing climatic and land-use conditions is unknown, especially in tropical streams. On-going studies in the Luquillo Experimental Forest. Puerto Rico provide an example of how inputs of fronds and fruits from palms (Prestoea montana) serve as important foods and microhabitat for species of freshwater crabs and shrimp. Native riparian species such as Prestoea montana are commonly distributed in the Luquillo Mountains especially along steep slopes and stream banks. After tropical storms with high winds, the large fronds from these native riparian trees provide important inputs of leaf litter to the stream food web. In some streams, the input of ripe fruit from non-native trees such as Java plum (Syzigium jambos) also provides a major source of detrital food resources, especially during periods when fruit fall from native species of palms may be limited.

  • PDF

An Adaptive Multimedia Synchronization Scheme for Media Stream Delivery in Multimedia Communication (멀티미디어 통신에서 미디어스트림 전송을 위한 적응형 멀티미디어 동기화 기법)

  • Lee, Gi-Sung
    • The KIPS Transactions:PartC
    • /
    • v.9C no.6
    • /
    • pp.953-960
    • /
    • 2002
  • Rel-time application programs have constraints which need to be met between media-data. It is client-leading synchronization that is absorbing variable transmission delay time and that is synchronizing by feedback control and palyout control. It is the important factor for playback rate and QoS if the buffer level is normal or not. This paper, The method of maintenance buffer normal state transmits in multimedia server by appling feedback of filtering function. And synchronization method is processing adaptive playout time for smooth presentation without cut-off while media frame is skip. When audio frame which is master media is in upper threshold buffer level we decrease play out time gradually, low threshold buffer level increase it slowly.