Journal of the Korea Convergence Society (한국융합학회논문지)

Korea Convergence Society (한국융합학회)
권호 표지
DOI QR코드

DOI QR Code

A Study on the Data Collection Methods based Hadoop Distributed Environment

하둡 분산 환경 기반의 데이터 수집 기법 연구

  • Jin, Go-Whan (Division of IT Convergence, Woosong University)
  • Received : 2016.09.11
  • Accepted : 2016.10.20
  • Published : 2016.10.31
Download PDF
⟨ Previous Next ⟩

Abstract

Many studies have been carried out for the development of big data utilization and analysis technology recently. There is a tendency that government agencies and companies to introduce a Hadoop of a processing platform for analyzing big data is increasing gradually. Increased interest with respect to the processing and analysis of these big data collection technology of data has become a major issue in parallel to it. However, study of the collection technology as compared to the study of data analysis techniques, it is insignificant situation. Therefore, in this paper, to build on the Hadoop cluster is a big data analysis platform, through the Apache sqoop, stylized from relational databases, to collect the data. In addition, to provide a sensor through the Apache flume, a system to collect on the basis of the data file of the Web application, the non-structured data such as log files to stream. The collection of data through these convergence would be able to utilize as a basic material of big data analysis.

최근 빅데이터 활용과 분석기술의 발전을 위하여 많은 연구가 이루어지고 있고, 빅데이터를 분석하기 위하여 처리 플랫폼인 하둡을 도입하는 정부기관 및 기업이 점차 늘어가고 있는 추세이다. 이러한 빅데이터의 처리와 분석에 대한 관심이 고조되면서 그와 병행하여 데이터의 수집 기술이 주요한 이슈가 되고 있으나, 데이터 분석 기법의 연구에 비하여 수집 기술에 대한 연구는 미미한 상황이다. 이에 본 논문에서는 빅데이터 분석 플랫폼인 하둡을 클러스터로 구축하고 아파치 스쿱을 통하여 관계형 데이터베이스로부터 정형화된 데이터를 수집하고, 아파치 플룸을 통하여 센서 및 웹 애플리케이션의 데이터 파일, 로그 파일과 같은 비정형 데이터를 스트림 기반으로 수집하는 시스템을 제안한다. 이러한 융합을 통한 데이터 수집으로 빅데이터 분석의 기초적인 자료로 활용할 수 있을 것이다.

Keywords

References

  1. O. B. Kwon, K. S. Kim, "The Design and Implementation of Location Information System using Wireless Fidelity in Indoors", Journal of Digital Convergence, Vol. 11, No. 4, pp. 243-249, 2013. https://doi.org/10.14400/JDPM.2013.11.11.243
  2. K. H. Lee, D. I. Kim, D. H. Kim, M. Y. Sung, Y. K. Lee, S. Y. Jung, "Implementation of Real-Time Video Transfer System on Android Environment", Journal of th Korea Convergence Society, Vol. 3, No. 1, pp. 1-5, 2012.
  3. J. T. Kim, B. J. Oh and J. Y. Park, "Standard Trends for the Big Data Technologies", Electronics and Telecommunications Trends 2013, ETRI, pp. 92-99, 2013.
  4. Y. S. Jeong, Y. T. Kim, G. C. Park, "Subnet Selection Scheme based on probability to enhance process speed of Big Data", Journal of Digital Convergence, Vol. 13, No. 9, pp. 201-208, 2015.
  5. M. G. Song, S. B. Kim, "A Study of improving reliability on prediction model by analyzing method Big data", Journal of Digital Convergence, Vol. 11, No. 6, pp. 103-112, 2013.
  6. M.J. Song, "Big Data is Creating Future Business Map", Hansmedia, 2012.
  7. K. S. Noh, S. T. Park. K. H. Park, "Convergence Study on Big Data Competency Reference Model", Journal of Digital Convergence, Vol. 13, No. 3, pp. 55-63, 2015. https://doi.org/10.14400/JDC.2015.13.3.55
  8. S. H. Namn, K. S. Noh, "A Study on the Effective Approaches to Big Data Planning", Journal of Digital Convergence, Vol. 13, No. 1, pp. 227-235, 2015.
  9. BigData Monthly, "Big Data in the World," BigData World, Report, Vol. 8, 2015.
  10. S. A. Shin, K. E. Kim, "Classification and the Current State of Big Data Technology", National Information Society Agency, Korea Big Data Center, 2013.
  11. Y.H. Kang, "Design of a Framework of a System for Handling Streaming Data by Using Apache Flume", Journal of KIIT, Vol. 12, No. 11, pp. 127-132, 2014.
  12. U. G. Han, J. H. Ahn, "Load Balancing Method for Improving Performance of Apache Flume Log Aggregator", Proceeding of KIIT, pp. 314-317, 2014.
  13. Liu Chen, J.H. Ko, J.M. Yeo, "Analysis of the Influence Factors of Data Loading Performance Using Apache Sqoop", Journal of KIPS, Vol. 4, No. 2, pp. 77-82, 2015.
  14. K. C. Choi, J. A. Yoo, "A reviews on the social network analysis using R", Journal of the Korea Convergence Society, Vol. 6, No. 1, pp. 77-83, 2015. https://doi.org/10.15207/JKCS.2015.6.1.077
  15. Apache Flume 1.4.0 User Guide, https://flume.apache.org/FlumeUserGuide.html.
  16. K. J. Park, "Big Data Eco System(Around the Platform)", Journal of KIIE, ie Magazine, Vol. 19, No. 3, pp. 41-47, 2012.
  17. Apache Sqoop, http://sqoop.apache.org
  18. Kathleen Ting, Jarek Jarcec Cecho, "Apache Sqoop Cookbook", O'Reilly, 2013.
  19. Rinusha Irudeen, Sanjeeva Samaraweera, "Big data solution for Sri Lankan development: A case study from travel and tourism", in Advances in ICT for Emerging Regions, International Conference on, 2013.
  20. Nodar Momtselidze, Alex Kuksin "Hadoop Integrating with Oracle Data Warehouse and Data Mining", in Journal of Technical Science and Technologies, Vol.2, No. 1, 2013.
  21. Ankit Jain, "Instant Apache Sqoop", Packt Publishing Ltd, 2013.
  22. Ognjen V. Jodzic, Dijana R. Vukovic, "The Impact of Cluster Characteristics on HiveQL Query Optimization", in Telecommunications Forum (TELFOR), 21st, 2013.
  23. K.B. Ryu, H.J. Park, "Mobile Web Server Log Analyzer", Proceeding of KSII, Vol. 5, No. 2, pp. 73-76, 2004.