• Proceedings of the Korean Society of Computer Information Conference
• /
• 2020.07a
• /
• pp.545-546
• /
• 2020

스파크는 데이터 분석을 위한 오픈소스 툴이다. 스파크에서는 실시간 데이터 분석을 위하여 스파크 스트리밍이라는 기술을 제공한다. 스파크 스트리밍은 데이터 소스가 분석서버로 데이터 스트림을 전송한다. 이때 전송하는 데이터의 크기가 커질 경우 전송과정에서 지연이 발생할 수 있다. 제안하는 기법은 전송하고자 하는 데이터의 크기가 클 때 허프만 인코딩을 이용하여 데이터를 압축하여 전송시키므로 지연시간을 줄일 수 있다.

• The Korean Journal of Applied Statistics
• /
• v.29 no.6
• /
• pp.1077-1094
• /
• 2016

Apache Spark is a fast and general-purpose cluster computing package. It provides a new abstraction named resilient distributed dataset, which is capable of support for fault tolerance while keeping data in memory. This type of abstraction results in a significant speedup compared to legacy large-scale data framework, MapReduce. In particular, Spark framework is suitable for iterative machine learning applications such as logistic regression and K-means clustering, and interactive data querying. Spark also supports high level libraries for various applications such as machine learning, streaming data processing, database querying and graph data mining thanks to its versatility. In this work, we introduce the concept and programming model of Spark as well as show some implementations of simple statistical computing applications. We also review the machine learning package MLlib, and the R language interface SparkR.

• Journal of KIISE
• /
• v.43 no.4
• /
• pp.470-479
• /
• 2016

As the number of various types of media data such as UCC (User Created Contents) increases, research is actively being carried out in many different fields so as to provide meaningful media services. Amidst these studies, a semantic web-based media classification approach has been proposed; however, it encounters some limitations in video classification because of its underlying ontology derived from meta-information such as video tag and title. In this paper, we define recognized objects in a video and activity that is composed of video objects in a shot, and introduce a reasoning approach based on description logic. We define sequential rules for a sequence of shots in a video and describe how to classify it. For processing the large amount of increasing media data, we utilize Spark streaming, and a distributed in-memory big data processing framework, and describe how to classify media data in parallel. To evaluate the efficiency of the proposed approach, we conducted an experiment using a large amount of media ontology extracted from Youtube videos.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2018.07a
• /
• pp.3-6
• /
• 2018

최근 방대한 양의 스트림 데이터가 생산되면서 이를 실시간으로 처리하기 위한 프레임워크가 등장하였으며, 오픈 소스 영역에서 Spark Streaming이 주목받고 있다. Spark Streaming은 분산 환경에서 성능 향상을 위해 지연 스케줄링을 기반으로 응용을 수행하지만, 장애 발생 시 사용 가능한 태스크 슬롯을 빠르게 할당받지 못할 경우 장애 복구 시간이 지연될 수 있다는 문제점이 있다. 이러한 문제점을 해결하기 위해 본 논문에서는 실행자의 태스크 슬롯 보장을 통해 대기 시간 없이 즉시 할당할 수 있도록 하는 개선 메커니즘을 제안하였고, 실험 결과 장애 복구 응답성이 39.14% 개선됨을 확인하였다.

• Proceedings of the Korean Society of Broadcast Engineers Conference
• /
• 2018.06a
• /
• pp.326-328
• /
• 2018

최근 4K/8K 급 초고품질 콘텐츠의 서비스에 관심이 집중되는 만큼 스트리밍 서비스에 대한 연구도 활발히 이루어지고 있다. 하지만 단일 PC 성능의 한계로 인해 SW 기반 영상 처리에 어려움을 겪고 있다. 본 논문에서는 분산 처리를 통해 실시간 영상 처리가 가능하도록 시스템을 제안한다. 제안한 시스템은 영상 패킷 분석 및 분할, 분산 트랜스코딩, 패킷 통합 단계로 이루어지며 Hadoop 과 Spark 를 이용하여 실시간 분산 처리를 지원한다. 실험 결과는 초고품질 입력 영상($3840{\times}2160@60Hz$, YCbCr 4:2:2, 10-bit)에 대해 평균 74.47fps 의 트랜스코딩 속도를 보인다.

• Journal of KIISE
• /
• v.44 no.1
• /
• pp.84-94
• /
• 2017

Recently, the importance of velocity, one of the characteristics of big data (5V: Volume, Variety, Velocity, Veracity, and Value), has been emphasized in the data processing, which has led to several studies on the real-time stream processing, a technology for quick and accurate processing and analyses of big data. In this paper, we propose a Squall framework using Time-triggered Message-triggered Object (TMO) technology, a model that is widely used for processing real-time big data. Moreover, we provide a description of Squall framework and its operations under a single node. TMO is an object model that supports the non-regular real-time processing method for certain conditions as well as regular periodic processing for certain amount of time. A Squall framework can support the real-time event stream of big data and micro-batch processing with outstanding performances, as compared to Apache storm and Spark Streaming. However, additional development for processing real-time stream under multiple nodes that is common under most frameworks is needed. In conclusion, the advantages of a TMO model can overcome the drawbacks of Apache storm or Spark Streaming in the processing of real-time big data. The TMO model has potential as a useful model in real-time big data processing.

• Journal of KIISE
• /
• v.44 no.5
• /
• pp.460-468
• /
• 2017

In a real-time cloud environment, the data analysis framework plays a pivotal role. Spark Streaming meets most real-time requirements among existing frameworks. However, the framework does not meet the second scale real-time fault recovery requirement. Spark Streaming fault recovery time increases in proportion to the transformation history length called lineage. This is because it recovers the last state data based on the cumulative lineage recorded during normal operation. Therefore, fault recovery time is not bounded within a limited time. In addition, it is impossible to achieve a second-scale fault recovery time because it costs tens of seconds to read initial state data from fault-tolerant storage. In this paper, we propose two techniques to solve the problems mentioned above. We apply the proposed techniques to Spark Streaming 1.6.2. Experimental results show that the fault recovery time is bounded and the average fault recovery time is reduced by up to 41.57%.

• The Journal of Bigdata
• /
• v.1 no.2
• /
• pp.9-19
• /
• 2016

Since ICT convergence became a major issue, German government has carried forward a policy 'Industry 4.0' that triggered ICT convergence with manufacturing. Now this trend gets into our stride. From this facts, we can expect great leap up to quality perfection in low cost. Recently Korean government also enforces policy with 'Manufacturing 3.0' for upgrading Korean manufacturing industry with being accelerated by many related technologies. We, in the paper, developed a custom-made operational big data analysis platform for the implementation of operational intelligence to improve industry capability. Our platform is designed based on spring framework and web. In addition, HDFS and spark architectures helps our system analyze massive data on the field with streamed data processed by process mining algorithm. Extracted knowledge from data will support enhancement of manufacturing performance.