Journal of the Korea Academia-Industrial cooperation Society (한국산학기술학회논문지)

The Korea Academia-Industrial cooperation Society (한국산학기술학회)
권호 표지
DOI QR코드

DOI QR Code

Design and Implementation of an Efficient Web Services Data Processing Using Hadoop-Based Big Data Processing Technique

하둡 기반 빅 데이터 기법을 이용한 웹 서비스 데이터 처리 설계 및 구현

  • Kim, Hyun-Joo (Dept. of Electronics&Electrical Engineering, Graduate School Dan-kook University)
  • 김현주 (단국대학교 대학원 전자전기공학부 컴퓨터응용 전공)
  • Received : 2014.10.28
  • Accepted : 2015.01.08
  • Published : 2015.01.31
Download PDF
⟨ Previous Next ⟩

Abstract

Relational databases used by structuralizing data are the most widely used in data management at present. However, in relational databases, service becomes slower as the amount of data increases because of constraints in the reading and writing operations to save or query data. Furthermore, when a new task is added, the database grows and, consequently, requires additional infrastructure, such as parallel configuration of hardware, CPU, memory, and network, to support smooth operation. In this paper, in order to improve the web information services that are slowing down due to increase of data in the relational databases, we implemented a model to extract a large amount of data quickly and safely for users by processing Hadoop Distributed File System (HDFS) files after sending data to HDFSs and unifying and reconstructing the data. We implemented our model in a Web-based civil affairs system that stores image files, which is irregular data processing. Our proposed system's data processing was found to be 0.4 sec faster than that of a relational database system. Thus, we found that it is possible to support Web information services with a Hadoop-based big data processing technique in order to process a large amount of data, as in conventional relational databases. Furthermore, since Hadoop is open source, our model has the advantage of reducing software costs. The proposed system is expected to be used as a model for Web services that provide fast information processing for organizations that require efficient processing of big data because of the increase in the size of conventional relational databases.

데이터를 구조화하여 사용하는 관계형 데이터베이스가 현재까지 데이터 관리에 가장 많이 사용되고 있다. 그러나 관계형 데이터베이스는 데이터가 증가되면 데이터를 저장하거나 조회할 때 읽기, 쓰기 연산 수행에 제약 조건이 발생되어 서비스가 느려지는 현상이 나타난다. 또 새로운 업무가 추가되면 데이터베이스 내 데이터는 증가되고 결국 이를 해결하기 위해 하드웨어의 병렬 구성, CPU, 메모리, 네트워크 등 추가적인 인프라 구성을 필요로 하게 된다. 본 논문에서는 관계형 데이터베이스의 데이터 증가로 느려지는 웹 정보서비스 개선을 위해 기존 관계형 데이터베이스의 데이터를 하둡 HDFS로 전송하고 이를 일원화하여 데이터를 재구성한 후 사용자에게 하둡 데이터 처리로 대량의 데이터를 빠르고 안전하게 추출하는 모델을 구현한다. 본 시스템 적용을 위해 웹 기반 민원시스템과 비정형 데이터 처리인 이미지 파일 저장에 본 제안시스템을 적용하였다. 적용결과 관계형 데이터베이스 시스템보다 제안시스템 데이터 처리가 0.4초 더 빠른 결과를 얻을 수 있었고 기존 관계형 데이터베이스와 같은 대량의 데이터를 처리를 빅 데이터 기법인 하둡 데이터 처리로도 웹 정보서비스를 지원이 가능하였다. 또한 하둡은 오픈소스로 제공되어 소프트웨어 구매 비용을 줄여주는 장점이 있으며 기존 관계형 데이터베이스의 데이터 증가로 효율적인 대용량 데이터 처리를 요구하는 조직에게 도움을 줄 수 있을 것이다.

Keywords

References

  1. Y. J. Song, "Policy Challenges for the Future of Data-Based Country Strategy", NIA , IT Future Strategy No. 3, Apr 2013.
  2. G. S. Hang, "Big Data Platform Strategy: Big Data is Changing Business Platform Future Revolution", Electronic Times, (pp. 83-97, 101-105, 193-203), 22013.
  3. M. R. Choi, "United States․Japan: Big Data R&D Strategies and Country of the Corresponding Problem", Nipa, IT R&D Policy Review, Mar 2013.
  4. Ms Park Presidential Election Camp, "Creative Economy", New World, 2012.
  5. DongA, "[2012 General Election-Big Data Presidential Election Campaign] 'Big Data 'Election Period", Available From: http://news.donga.com/3/all/20120202/43722588/1, Feb, 02, 2012.
  6. Seth Gilbert, Nancy Lynch, "Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services", ACM SIGACT, (pp. 51-59), vol 33 Issue 2, (accessed June, 2002). https://doi.org/10.1145/564585.564601
  7. Anonymous. http://develop.sunshiny.kr/883?category=50, 2013.
  8. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, "The Google File System", 19th ACM Symposium on Operating Systems Principles, Lake George, NY, [Online] Available: http://research.google.com/archive/gfs.html, (accessed Oct, 2003).
  9. Anonymous. "Big Data at the Speed of Business" http://www-01.ibm.com/software/data/bigdata/.
  10. STAMFORD, Conn, http://www.gartner.com/newsroom/id/1731916, June 27, 2011.
  11. McKinsey Global Institute, "Big Data: The next frontier for innovation, competition. and productivity", McKinsey Global Institute, 2011.
  12. PHILIP CARTER, "Big Data Analytics: Future Architectures, Skills and Roadmaps for the CIO", WHITE PAPER, IDC sponsored by SAS, 2011.
  13. Vitaly Friedman, "Data Visualization & Infographics", Graphics, Monday Inspiration, January 14th, Jan, 2008.
  14. Anonymous. "Big Data Era-Hadoop", http://cfic.tistory.com/5, May 25, 2012.
  15. J. Dean, S. Ghemawat "MapReduce: Simplified Data Processing on Large Clusters", Communications of the ACM, vol. 51, No. 1, Jan, 2008.
  16. Colin White, "MapReduce and Data Scientist", BI Research, 2012.
  17. Brian Proffittm, "Cost Analysis of Hadoop and RDBM S…Grenada is a Three-Fold Difference in Operating Costs.", IDG KOREA, Technology Trends, Jan, 12, 2012.