• The Transactions of the Korean Institute of Electrical Engineers P
• /
• v.64 no.2
• /
• pp.85-91
• /
• 2015

With the increasing popularity of Smart Grid infrastructure, it is much easier to collect energy usage data using AMI (Advanced Measuring Instrument) from residential housing, buildings and factories. Several researches have been done to improve an energy efficiency by analyzing the collected energy usage data. However, it is not easy to store and analyze the energy data using a traditional relational database management system since the data size grows exponentially with an increasing popularity of Smart grid infrastructure. In this paper, we are proposing a Hadoop based Big data system to store and analyze energy usage data. Based on our limited experiments, Hadoop based energy data analysis is three times faster than that of a relational database management system based approach with the current system.

• The Journal of the Korea institute of electronic communication sciences
• /
• v.15 no.1
• /
• pp.107-114
• /
• 2020

This paper studies a method for estimating the scale of a Hadoop cluster to process big data as a cost-effective manner. In the case of medical institutions, demands for cloud-based big data analysis are increasing as medical records can be stored outside the hospital. This paper first analyze the Amazon EMR framework, which is one of the popular cloud-based big data framework. Then, this paper presents a efficiency model for scaling the Hadoop cluster to execute a Mapreduce application more cost-effectively. This paper also analyzes the factors that influence the execution of the Mapreduce application by performing several experiments under various conditions. The cost efficiency of the analysis of the big data can be increased by setting the scale of cluster with the most efficient processing time compared to the operational cost.

• KIISE Transactions on Computing Practices
• /
• v.24 no.1
• /
• pp.24-32
• /
• 2018

This paper proposes a Big Data system for energy Big Data which is aggregated in real-time from industrial and public sources. The constructed Big Data system is based on Hadoop and the Spark framework is simultaneously applied on Big Data processing, which supports in-memory distributed computing. In the paper, we focus on Big Data, in the form of heat energy for district heating, and deal with methodologies for storing, managing, processing and analyzing aggregated Big Data in real-time while considering properties of energy input and output. At present, the Big Data influx is stored and managed in accordance with the designed relational database schema inside the system and the stored Big Data is processed and analyzed as to set objectives. The paper exemplifies a number of heat demand plants, concerned with district heating, as industrial sources of heat energy Big Data gathered in real-time as well as the proposed system.

• KIISE Transactions on Computing Practices
• /
• v.22 no.1
• /
• pp.20-25
• /
• 2016

With the advent of the era of digital big data, the Hadoop platform has become widely used in various fields. However, the Hadoop MapReduce framework suffers from problems related to the increase of the name node's main memory and map tasks for the processing of large number of small files. In addition, a method for running C++-based tasks in the MapReduce framework is required in order to conjugate GPUs supporting hardware-based data parallelism in the MapReduce framework. Therefore, in this paper, we present a face detection system that generates a sequence file for images to process big data for images in the Hadoop platform. The system also deals with tasks for GPU-based face detection in the MapReduce framework using Hadoop Pipes. We demonstrate a performance increase of around 6.8-fold as compared to a single CPU process.

• KIISE Transactions on Computing Practices
• /
• v.22 no.11
• /
• pp.601-613
• /
• 2016

With the increasing need for big data processing, building a shared big data processing platform is important to minimize time and monetary costs. In shared big data processing, multitenancy is a major requirement that needs to be addressed, in order to provide a single isolated personal big data platform for each user, but to share the underlying hardware is shared among users to increase hardware utilization. In this paper, we explore two well-known shared big data processing platform models. One is to use a native Hadoop cluster, and the other is to build a virtual Hadoop cluster for each user. For each model we verified whether it is sufficient to support multi-tenancy. We also present a method to complement unsupported multi-tenancy features in a native Hadoop cluster model. Lastly we built prototype platforms and compared the performance of both models.

• Journal of the Korea Institute of Information and Communication Engineering
• /
• v.19 no.3
• /
• pp.607-612
• /
• 2015

As the amount of vehicle's diagnostics data increases, the actors in automotive ecosystem will encounter difficulties to perform a real time analysis in order to simulate or to design new services according to the data gathered from the connected cars. In this paper, we have conducted a study of a Big Data solution that expresses the essential deep analytics to process and analyze vast quantities of vehicles on board diagnostics data generated by cars. Hadoop and its ecosystems have been deployed to process a large data and delivered useful outcomes that may be used by actors in automotive ecosystem to deliver new services to car owners. As the Intelligent transport system is involved to guarantee safety, reduce rate of crash and injured in the accident due to speed, addressing big data solution based on vehicle diagnostics data is upcoming to monitor real time outcome from it and making collection of data from several connected cars, facilitating reliable processing and easier storage of data collected.

• Journal of Korea Society of Digital Industry and Information Management
• /
• v.13 no.2
• /
• pp.77-85
• /
• 2017

Due to the expansion of accommodation space caused by increase of population along with lifestyle changes, most of people spend their time indoor except for the travel time. Because of this, environmental change of indoor is very important, and it affects people's health and economy in resources. But, most of people don't acknowledge the importance of indoor environment. Thus, monitoring system for sustaining and managing indoor environment systematically is needed, and big data clusters should be used in order to save and manage numerous sensor data collected from many spaces. In this paper, we design a big data cluster for the indoor environment monitoring in order to store the sensor data and monitor unit of the huge building Implementation design big data cluster-based system for the analysis, and a distributed file system and building a Hadoop, HBase for big data processing. Also, various sensor data is saved for collection, and effective indoor environment management and health enhancement through monitoring is expected.

• The Transactions of The Korean Institute of Electrical Engineers
• /
• v.63 no.8
• /
• pp.1111-1115
• /
• 2014

With the increasing interest of a demand side management using a Smart Grid infrastructure, the demand resources and energy usage data management becomes an important factor in energy industry. In addition, with the help of Advanced Measuring Infrastructure(AMI), energy usage data becomes a Big Data System. Therefore, it becomes difficult to store and manage the demand resources big data using a traditional relational database management system. Furthermore, not many researches have been done to analyze the big energy data collected using AMI. In this paper, we are proposing a Hadoop based Big Data system to manage the demand resources energy data and we will also show how the demand side management systems can be used to improve energy efficiency.

• KIPS Transactions on Computer and Communication Systems
• /
• v.5 no.2
• /
• pp.33-40
• /
• 2016

Due to the growth of social network systems (SNS), big data are realized and Hadoop was developed as a distributed platform for analyzing big data. Enterprises analyze data containing users' sensitive information by using Hadoop and utilize them for marketing. Therefore, researches on data encryption have been done to protect the leakage of sensitive data stored in Hadoop. However, the existing researches support only the AES encryption algorithm, the international standard of data encryption. Meanwhile, Korean government choose ARIA algorithm as a standard data encryption one. In this paper, we propose a HDFS data encryption scheme using ARIA algorithms on Hadoop. First, the proposed scheme provide a HDFS block splitting component which performs ARIA encryption and decryption under the distributed computing environment of Hadoop. Second, the proposed scheme also provide a variable-length data processing component which performs encryption and decryption by adding dummy data, in case when the last block of data does not contains 128 bit data. Finally, we show from performance analysis that our proposed scheme can be effectively used for both text string processing applications and science data analysis applications.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2012.07a
• /
• pp.3-6
• /
• 2012

소셜 네트워크와 웹 2.0의 등장은 거대한 데이터 홍수를 초래하였다. 이 와 관련된 다양한 기술들이 연구 개발되고 있으며 특히 동시에 요구되는 data를 처리하기위한 여러 기술이 등장하였다. 본 연구에서는 다양한 BigData 분산처리 기술들중에 가장 각광 받고 있는 Hadoop이라는 기술을 연구 분석할 것이다. 국내에 아직 많은 사용자가 없어 그 존재감이 많이 없다가 요즘 들어 상승하고 있는 추세이며 이러한 Hadoop의 흐름속에 data의 분산과 병렬처리에서 발생되는 문제점을 분석하고 이를 해결할수 있는 모델을 제시하여 새로운 모델의 하둡으로 기본적인 핵심기술인 federation을 쉽게 할 수 있고 향후 이 구조의 기능과 상세모델을 연구하고 구현하여 제안된 연구 구조의 우수성을 입증하고자 한다.