Browse > Article
http://dx.doi.org/10.15207/JKCS.2018.9.10.013

Implement of MapReduce-based Big Data Processing Scheme for Reducing Big Data Processing Delay Time and Store Data  

Lee, Hyeopgeon (Department of Data Analysis, Seoul Gangseo Campus of Korea Polytechnic)
Kim, Young-Woon (Department of Data Analysis, Seoul Gangseo Campus of Korea Polytechnic)
Kim, Ki-Young (Department of Computer Software, Seoil University)
Publication Information
Journal of the Korea Convergence Society / v.9, no.10, 2018 , pp. 13-19 More about this Journal
Abstract
MapReduce, the Hadoop's essential core technology, is most commonly used to process big data based on the Hadoop distributed file system. However, the existing MapReduce-based big data processing techniques have a feature of dividing and storing files in blocks predefined in the Hadoop distributed file system, thus wasting huge infrastructure resources. Therefore, in this paper, we propose an efficient MapReduce-based big data processing scheme. The proposed method enhances the storage efficiency of a big data infrastructure environment by converting and compressing the data to be processed into a data format in advance suitable for processing by MapReduce. In addition, the proposed method solves the problem of the data processing time delay arising from when implementing with focus on the storage efficiency.
Keywords
Big Data; Big Data Process Scheme; Big Data Compress Scheme; Hadoop Distributed File System; MapReduce;
Citations & Related Records
Times Cited By KSCI : 10  (Citation Analysis)
연도 인용수 순위
1 H. G. Lee, Y. W. Kim, K. Y. Kim & J. S. Choi (2018). Design of Splunk Platform based Big Data Analysis System for Objectionable Information Detection, Journal of Korea Institute of Information, Electronics, and Communication Technology, 11(1), 76-81.   DOI
2 S. H. Kim, S. H. Chang & S. W Lee (2017). Consumer Trend Platform Development for Combination Analysis of Structured and Unstructured Big Data, Journal of Digital Convergence, 15(6), 133-143.   DOI
3 C. Y. Lee (2017). A Study on Synchronization Effect of A Multi-dimensional Event Database for Big Data Information Sharing, Journal of Digital Convergence, 15(10), 243-251.   DOI
4 Y. U. Jeong (2015). U-healthcare Service Management Scheme for Big Data of Patient Information, Journal of Convergence for Information Technology, 5(1), 1-6.   DOI
5 J. H. Ku (2017). A Study on the Platform for Big Data Analysis of Manufacturing Process, Journal of Convergence for Information Technology, 7(5), 177-182.   DOI
6 I. H. Joo (2017). Spatial Big Data Query Processing System Supporting SQL-based Query Language in Hadoop, Journal of Korea Institute of Information, Electronics, and Communication Technology, 10(1), 1-8.   DOI
7 Y. J. Baek, W. C. Jeong, S. W. Hong & J. H. Park (2017). A step-by-step service encryption model based on routing pattern in case of IP spoofing attacks on clustering environment, Journal of Korea Institute of Information, Electronics, and Communication Technology, 10(6), 580-586.   DOI
8 E. H. Jeong & B. K. Lee. (2017). A Design of Hadoop Security Protocol using One Time Key based on Hash-chain, Journal of Korea Institute of Information, Electronics, and Communication Technology, 10(4), 340-349.   DOI
9 Y. S. Lee (2015). Authentication Method for Safe Internet of Things Environments, Journal of Korea Institute of Information, Electronics, and Communication Technology, 8(1), 51-58.   DOI
10 J. T. Seong (2017). Analysis of Signal Recovery for Compressed Sensing using Deep Learning Technique, Journal of Korea Institute of Information, Electronics, and Communication Technology, 10(4), 257-267.   DOI
11 H. G. Lee, Y. W. Kim, K. Y. Kim & J. S. Choi. (2018). Design of GlusterFS Based Big Data Distributed Processing System in Smart Factory, Journal of Korea Institute of Information, Electronics, and Communication Technology, 11(1), 70-75.   DOI
12 H. G. Lee, Y. W. Kim & K. Y. Kim (2017), Implementation of an Efficient Big Data Collection Platform for Smart Manufacturing. Journal of Engineering and Applied Sciences, 12(2Si), 6304-6307. DOI: 10.3923/jeasci.2017.6304.6307
13 B. Mahjani, S. Toor, C. Nettelblad & S. Holmgren (2016). A Flexible Computational Framework Using R and Map-Reduce for Permutation Tests of Massive Genetic Analysis of Complex Traits. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 14(2), 381-392. DOI: 10.1109/TCBB.2016.2527639   DOI
14 Y. W. Kim & H. G. Lee (2017). Implementation of Big Data Analysis System to Prevent Illegal Sales in the Cable TV Industry. Journal of Engineering and Applied Sciences, 12(3Si), 6542-6545. DOI: 10.3923/jeasci.2017.6542.6545
15 H. J. Park. (2016). A Study about Performance Evaluation of Various NoSQL Databases, Journal of Korea Institute of Information, Electronics, and Communication Technology, 9(3), 298-305.   DOI