• Journal of Korean Society for Geospatial Information Science
• /
• v.25 no.1
• /
• pp.29-36
• /
• 2017

In this paper, we implement a spatial big data analysis prototype based on Spark which is an in-memory system and compares the performance by the spatial split algorithm on this basis. In cluster computing environments, big data is divided into blocks of a certain size order to balance the computing load of big data. Existing research showed that in the case of the Hadoop based spatial big data system, the split method by spatial is more effective than the general sequential split method. Hadoop based spatial data system stores raw data as it is in spatial-divided blocks. However, in the proposed Spark-based spatial analysis system, there is a difference that spatial data is converted into a memory data structure and stored in a spatial block for search efficiency. Therefore, in this paper, we propose an in-memory spatial big data prototype and a spatial split block storage method. Also, we compare the performance of existing spatial split algorithms in the proposed prototype. We presented an appropriate spatial split strategy with the Spark based big data system. In the experiment, we compared the query execution time of the spatial split algorithm, and confirmed that the BSP algorithm shows the best performance.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2015.10a
• /
• pp.613-616
• /
• 2015

최근 스마트폰 기기의 보급 및 소셜 서비스 산업의 고도화로 인해, 빅데이터가 등장하였다. 한편 빅데이터에서 효율적으로 정보를 분석하는 대표적인 플랫폼으로 하둡이 존재한다. 하둡은 클러스터 환경에 기반한 우수한 확장성, 장애 복구 기능 및 사용자가 기능을 정의할 수 있는 맵리듀스 프레임워크 등을 지원한다. 아울러 하둡은 개인정보나 위치 데이터 등의 민감한 정보를 보호하기 위해 Kerberos를 통한 사용자 인증 기법을 제공하고, HDFS 압축 코덱을 활용한 AES 코덱 기반 데이터 암호화를 지원하고 있다. 그러나 하둡 기반 소프트웨어를 사용하고 있는 국내 기관 및 기업은 국내 ARIA 데이터 암호화를 적용하지 못하고 있다. 이를 해결하기 위해 본 논문에서는 하둡을 기반으로 ARIA 암호화를 지원하는 HDFS 데이터 암호화 기법을 제안한다.

• KIPS Transactions on Software and Data Engineering
• /
• v.6 no.2
• /
• pp.103-116
• /
• 2017

In this paper, we present the design and implementation of a large-scale qualitative spatial reasoner, which can derive new qualitative spatial knowledge representing both topological and directional relationships between two arbitrary spatial objects in efficient way using Aparch Spark SQL. Apache Spark SQL is well known as a distributed parallel programming environment which provides both efficient join operations and query processing functions over a variety of data in Hadoop cluster computer systems. In our spatial reasoner, the overall reasoning process is divided into 6 jobs such as knowledge encoding, inverse reasoning, equal reasoning, transitive reasoning, relation refining, knowledge decoding, and then the execution order over the reasoning jobs is determined in consideration of both logical causal relationships and computational efficiency. The knowledge encoding job reduces the size of knowledge base to reason over by transforming the input knowledge of XML/RDF form into one of more precise form. Repeat of the transitive reasoning job and the relation refining job usually consumes most of computational time and storage for the overall reasoning process. In order to improve the jobs, our reasoner finds out the minimal disjunctive relations for qualitative spatial reasoning, and then, based upon them, it not only reduces the composition table to be used for the transitive reasoning job, but also optimizes the relation refining job. Through experiments using a large-scale benchmarking spatial knowledge base, the proposed reasoner showed high performance and scalability.

• The Journal of Korean Institute of Communications and Information Sciences
• /
• v.40 no.6
• /
• pp.1166-1174
• /
• 2015

There were the attempts to prevent the spread of harmful materials of the agri-food through the record tracking of the products with the bar code, the partial information tracking of the agri-food storage and the delivery vehicle, or the control of the temperature by intuition. However, there were many problems in the attempts because of the insufficient information, the information distortion and the independent information network of each distribution company. As a result, it is difficult to prevent the spread over the life-cycle of the agri-food using the attempts. To solve the problems, we propose the mechanism mainly to do context awareness, predict, and track the harmful materials of agri-food using big data processing.

• The Journal of the Korea Contents Association
• /
• v.13 no.2
• /
• pp.32-43
• /
• 2013

These days, there are more and more IT research institutions which foresee cloud services as the predominant IT service in the near future and there, in fact, are actual cloud services provided by some IT leading vendors. Regardless of physical location of the service and environment of the system, cloud service can provide users with storage services, usage of data and software. On the other hand, cloud service has challenges as well. Even though cloud service has its edge in terms of the extent to which the IT resource can be freely utilized regardless of the confinement of hardware, the availability is another problem to be solved. Hence, this paper is dedicated to tackle the aforementioned issues; prerequisites of cloud computing for distributed file system, open source based Hadoop distributed file system, in-memory database technology and high availability database system. Also the author tries to body out the high availability mass distributed data management architecture in cloud service's perspective using currently used distributed file system in cloud computing market.

• The Journal of the Korea Contents Association
• /
• v.12 no.2
• /
• pp.30-39
• /
• 2012

In these day, some of IT venders already were going to cloud computing market, as well they are going to expand their territory for the cloud computing market through that based on their hardware and software technology, making collaboration between hardware and software vender. Distributed file system is very mainly technology for the cloud computing that must be protect performance and safety for high levels service requests as well data store. This paper introduced distributed file system for cloud computing and how to use this theory such as memory database, Hadoop file system, high availability database system. now In the market, this paper define a very large distributed processing architect as a reference by kind of distributed file systems through using technology in cloud computing market.

• KIPS Transactions on Computer and Communication Systems
• /
• v.8 no.10
• /
• pp.231-238
• /
• 2019

Analyzing next-generation genome sequencing data in a conventional way using single server may take several tens of hours depending on the data size. However, in order to cope with emergency situations where the results need to be known within a few hours, it is required to improve the performance of a single genome analysis. In this paper, we propose a parallelized method for pre-processing genome sequence data which can reduce the analysis time by utilizing the big data technology and the highperformance computing cluster which is connected to the high-speed network and shares the parallel file system. For the reliability of analytical data, we have chosen a strategy to parallelize the existing analytical tools and algorithms to the new environment. Parallelized processing, data distribution, and parallel merging techniques have been developed and performance improvements have been confirmed through experiments.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.19 no.3
• /
• pp.669-678
• /
• 2018

This study developed information technology infrastructures for building a driving environment analysis platform using various big data, such as vehicle sensing data, public data, etc. First, a small platform server with a parallel structure for big data distribution processing was developed with H/W technology. Next, programs for big data collection/storage, processing/analysis, and information visualization were developed with S/W technology. The collection S/W was developed as a collection interface using Kafka, Flume, and Sqoop. The storage S/W was developed to be divided into a Hadoop distributed file system and Cassandra DB according to the utilization of data. Processing S/W was developed for spatial unit matching and time interval interpolation/aggregation of the collected data by applying the grid index method. An analysis S/W was developed as an analytical tool based on the Zeppelin notebook for the application and evaluation of a development algorithm. Finally, Information Visualization S/W was developed as a Web GIS engine program for providing various driving environment information and visualization. As a result of the performance evaluation, the number of executors, the optimal memory capacity, and number of cores for the development server were derived, and the computation performance was superior to that of the other cloud computing.