Browse > Article
http://dx.doi.org/10.17661/jkiiect.2019.12.2.127

Study of In-Memory based Hybrid Big Data Processing Scheme for Improve the Big Data Processing Rate  

Lee, Hyeopgeon (Dept. of Data Analysis, Seoul Gangseo Campus of Korea Polytechnic)
Kim, Young-Woon (Dept. of Data Analysis, Seoul Gangseo Campus of Korea Polytechnic)
Kim, Ki-Young (Dept. of Computer Software, Seoil University)
Publication Information
The Journal of Korea Institute of Information, Electronics, and Communication Technology / v.12, no.2, 2019 , pp. 127-134 More about this Journal
Abstract
With the advancement of IT technology, the amount of data generated has been growing exponentially every year. As an alternative to this, research on distributed systems and in-memory based big data processing schemes has been actively underway. The processing power of traditional big data processing schemes enables big data to be processed as fast as the number of nodes and memory capacity increases. However, the increase in the number of nodes inevitably raises the frequency of failures in a big data infrastructure environment, and infrastructure management points and infrastructure operating costs also increase accordingly. In addition, the increase in memory capacity raises infrastructure costs for a node configuration. Therefore, this paper proposes an in-memory-based hybrid big data processing scheme for improve the big data processing rate. The proposed scheme reduces the number of nodes compared to traditional big data processing schemes based on distributed systems by adding a combiner step to a distributed system processing scheme and applying an in-memory based processing technology at that step. It decreases the big data processing time by approximately 22%. In the future, realistic performance evaluation in a big data infrastructure environment consisting of more nodes will be required for practical verification of the proposed scheme.
Keywords
Big Data; Hadoop; Distributed File System; In-Memory Big Data Processing Scheme; MapReduce;
Citations & Related Records
Times Cited By KSCI : 5  (Citation Analysis)
연도 인용수 순위
1 H. Lee, Y. Kim, J. Park and J. Lee, "Map Reduce-Based Partitioner Big Data Analysis Scheme for Processing Rate of Log Analysis," Journal of Korea Institute of Information, Electronics, and Communication Technology, Vol. 11, No. 5, pp. 593-600, 2018   DOI
2 H. Lee, Y. Kim, K. Kim and J. Choi, "Design of GlusterFS Based Big Data Distributed Processing System in Smart Factory," Journal of Korea Institute of Information, Electronics, and Communication Technology, Vol. 11, No. 1, pp. 70-75, 2018   DOI
3 D. Hwang, K. Ko, S. Park and W. Kim, "Development for establishing Big Data-based alley commercial area," Journal of Korea Institute of Information, Electronics, and Communication Technology, Vol. 11, No. 6, pp. 784-792, 2018   DOI
4 H. G. Lee, Y. W. Kim and K. Y. Kim, "Implementation of an Efficient Big Data Collect ion Platform for Smart Manufacturing," Journal of Engineering and Applied Sciences, 12(2Si), pp. 6304-6307, 2018
5 Y. Kwon and I. Kim, "A Study on Anomaly Signal Detection and Management Model using Big Data," The Journal of The Institute of Internet, Broadcasting and Communication, Vol. 16, No. 6, pp. 287-294, 2016   DOI
6 J. Kim, J. Park and S. Chung, "Analysis of Network Log based on Hadoop," The Journal of The Institute of Internet, Broadcasting and Communication, Vol. 17, No. 5, pp. 125-130, 2017   DOI
7 E. Jeong and B. Lee, "A Design of Hadoop Security Protocol using One Time Key based on Hash-chain," Journal of Korea Institute of Information, Electronics, and Communication Technology, Vol. 10, No. 4, pp. 340-349, 2017   DOI