Browse > Article
http://dx.doi.org/10.15207/JKCS.2016.7.5.001

A Study on the Data Collection Methods based Hadoop Distributed Environment  

Jin, Go-Whan (Division of IT Convergence, Woosong University)
Publication Information
Journal of the Korea Convergence Society / v.7, no.5, 2016 , pp. 1-6 More about this Journal
Abstract
Many studies have been carried out for the development of big data utilization and analysis technology recently. There is a tendency that government agencies and companies to introduce a Hadoop of a processing platform for analyzing big data is increasing gradually. Increased interest with respect to the processing and analysis of these big data collection technology of data has become a major issue in parallel to it. However, study of the collection technology as compared to the study of data analysis techniques, it is insignificant situation. Therefore, in this paper, to build on the Hadoop cluster is a big data analysis platform, through the Apache sqoop, stylized from relational databases, to collect the data. In addition, to provide a sensor through the Apache flume, a system to collect on the basis of the data file of the Web application, the non-structured data such as log files to stream. The collection of data through these convergence would be able to utilize as a basic material of big data analysis.
Keywords
Big Data; Hadoop; Apache Sqoop; Apache Flume; Convergence;
Citations & Related Records
Times Cited By KSCI : 9  (Citation Analysis)
연도 인용수 순위
1 M.J. Song, "Big Data is Creating Future Business Map", Hansmedia, 2012.
2 K. S. Noh, S. T. Park. K. H. Park, "Convergence Study on Big Data Competency Reference Model", Journal of Digital Convergence, Vol. 13, No. 3, pp. 55-63, 2015.   DOI
3 S. H. Namn, K. S. Noh, "A Study on the Effective Approaches to Big Data Planning", Journal of Digital Convergence, Vol. 13, No. 1, pp. 227-235, 2015.
4 BigData Monthly, "Big Data in the World," BigData World, Report, Vol. 8, 2015.
5 S. A. Shin, K. E. Kim, "Classification and the Current State of Big Data Technology", National Information Society Agency, Korea Big Data Center, 2013.
6 Y.H. Kang, "Design of a Framework of a System for Handling Streaming Data by Using Apache Flume", Journal of KIIT, Vol. 12, No. 11, pp. 127-132, 2014.
7 U. G. Han, J. H. Ahn, "Load Balancing Method for Improving Performance of Apache Flume Log Aggregator", Proceeding of KIIT, pp. 314-317, 2014.
8 Liu Chen, J.H. Ko, J.M. Yeo, "Analysis of the Influence Factors of Data Loading Performance Using Apache Sqoop", Journal of KIPS, Vol. 4, No. 2, pp. 77-82, 2015.
9 K. C. Choi, J. A. Yoo, "A reviews on the social network analysis using R", Journal of the Korea Convergence Society, Vol. 6, No. 1, pp. 77-83, 2015.   DOI
10 Apache Flume 1.4.0 User Guide, https://flume.apache.org/FlumeUserGuide.html.
11 K. J. Park, "Big Data Eco System(Around the Platform)", Journal of KIIE, ie Magazine, Vol. 19, No. 3, pp. 41-47, 2012.
12 Apache Sqoop, http://sqoop.apache.org
13 Kathleen Ting, Jarek Jarcec Cecho, "Apache Sqoop Cookbook", O'Reilly, 2013.
14 Ognjen V. Jodzic, Dijana R. Vukovic, "The Impact of Cluster Characteristics on HiveQL Query Optimization", in Telecommunications Forum (TELFOR), 21st, 2013.
15 Rinusha Irudeen, Sanjeeva Samaraweera, "Big data solution for Sri Lankan development: A case study from travel and tourism", in Advances in ICT for Emerging Regions, International Conference on, 2013.
16 Nodar Momtselidze, Alex Kuksin "Hadoop Integrating with Oracle Data Warehouse and Data Mining", in Journal of Technical Science and Technologies, Vol.2, No. 1, 2013.
17 Ankit Jain, "Instant Apache Sqoop", Packt Publishing Ltd, 2013.
18 K. H. Lee, D. I. Kim, D. H. Kim, M. Y. Sung, Y. K. Lee, S. Y. Jung, "Implementation of Real-Time Video Transfer System on Android Environment", Journal of th Korea Convergence Society, Vol. 3, No. 1, pp. 1-5, 2012.
19 K.B. Ryu, H.J. Park, "Mobile Web Server Log Analyzer", Proceeding of KSII, Vol. 5, No. 2, pp. 73-76, 2004.
20 O. B. Kwon, K. S. Kim, "The Design and Implementation of Location Information System using Wireless Fidelity in Indoors", Journal of Digital Convergence, Vol. 11, No. 4, pp. 243-249, 2013.   DOI
21 J. T. Kim, B. J. Oh and J. Y. Park, "Standard Trends for the Big Data Technologies", Electronics and Telecommunications Trends 2013, ETRI, pp. 92-99, 2013.
22 Y. S. Jeong, Y. T. Kim, G. C. Park, "Subnet Selection Scheme based on probability to enhance process speed of Big Data", Journal of Digital Convergence, Vol. 13, No. 9, pp. 201-208, 2015.
23 M. G. Song, S. B. Kim, "A Study of improving reliability on prediction model by analyzing method Big data", Journal of Digital Convergence, Vol. 11, No. 6, pp. 103-112, 2013.