• KSII Transactions on Internet and Information Systems (TIIS)
• /
• 제10권9호
• /
• pp.4063-4086
• /
• 2016

Cloud computing is a robust technology, which facilitate to resolve many parallel distributed computing issues in the modern Big Data environment. Hadoop is an ecosystem, which process large data-sets in distributed computing environment. The HDFS is a filesystem of Hadoop, which process data blocks to the cluster nodes. The data block placement has become a bottleneck to overall performance in a Hadoop cluster. The current placement policy assumes that, all Datanodes have equal computing capacity to process data blocks. This computing capacity includes availability of same storage media and same processing performances of a node. As a result, Hadoop cluster performance gets effected with unbalanced workloads, inefficient storage-tier, network traffic congestion and HDFS integrity issues. This paper proposes a storage-tier-aware Robust Data Placement (RDP) scheme, which systematically resolves unbalanced workloads, reduces network congestion to an optimal state, utilizes storage-tier in a useful manner and minimizes the HDFS integrity issues. The experimental results show that the proposed approach reduced unbalanced workload issue to 72%. Moreover, the presented approach resolve storage-tier compatibility problem to 81% by predicting storage for block jobs and improved overall data block placement by 78% through pre-calculated computing capacity allocations and execution of map files over respective Namenode and Datanodes.

• Structural Monitoring and Maintenance
• /
• 제7권4호
• /
• pp.345-365
• /
• 2020

At present, many machine leaning and data mining methods are used for analyzing and predicting structural response characteristics. However, the platform that combines big data analysis methods with online and offline analysis modules has not been used in actual projects. This work is dedicated to developing a multifunctional Hadoop-Spark big data platform for bridges to monitor and evaluate the serviceability based on structural health monitoring system. It realizes rapid processing, analysis and storage of collected health monitoring data. The platform contains offline computing and online analysis modules, using Hadoop-Spark environment. Hadoop provides the overall framework and storage subsystem for big data platform, while Spark is used for online computing. Finally, the big data Hadoop-Spark platform computational performance is verified through several actual analysis tasks. Experiments show the Hadoop-Spark big data platform has good fault tolerance, scalability and online analysis performance. It can meet the daily analysis requirements of 5s/time for one bridge and 40s/time for 100 bridges.

• Journal of Information Processing Systems
• /
• 제14권4호
• /
• pp.989-1009
• /
• 2018

Rapid advances in science and technology with exponential development of smart mobile devices, workstations, supercomputers, smart gadgets and network servers has been witnessed over the past few years. The sudden increase in the Internet population and manifold growth in internet speeds has occasioned the generation of an enormous amount of data, now termed 'big data'. Given this scenario, storage of data on local servers or a personal computer is an issue, which can be resolved by utilizing cloud computing. At present, there are several cloud computing service providers available to resolve the big data issues. This paper establishes a framework that builds Hadoop clusters on the new single-board computer (SBC) Mobile Raspberry Pi. Moreover, these clusters offer facilities for storage as well as computing. Besides the fact that the regular data centers require large amounts of energy for operation, they also need cooling equipment and occupy prime real estate. However, this energy consumption scenario and the physical space constraints can be solved by employing a Mobile Raspberry Pi with Hadoop clusters that provides a cost-effective, low-power, high-speed solution along with micro-data center support for big data. Hadoop provides the required modules for the distributed processing of big data by deploying map-reduce programming approaches. In this work, the performance of SBC clusters and a single computer were compared. It can be observed from the experimental data that the SBC clusters exemplify superior performance to a single computer, by around 20%. Furthermore, the cluster processing speed for large volumes of data can be enhanced by escalating the number of SBC nodes. Data storage is accomplished by using a Hadoop Distributed File System (HDFS), which offers more flexibility and greater scalability than a single computer system.

• 한국컴퓨터정보학회논문지
• /
• 제20권8호
• /
• pp.77-83
• /
• 2015

The storage and retrieval of multimedia data is becoming increasingly important in many application areas including record management, video(CCTV) management and Internet of Things (IoT). In these applications, the files containing multimedia that need to be stored and managed is tremendous and constantly scaling. In this paper, we propose a technique to retrieve a very large number of files, in multimedia format, using the Hadoop Framework. Our strategy is based on the management of metadata that describes the characteristic of files that are stored in Hadoop Distributed File System (HDFS). The metadata schema is represented in Hbase and looked up using SQL On Hadoop (Hive, Tajo). Both the Hbase, Hive and Tajo are part of the Hadoop Ecosystem. Preliminary experiment on multimedia data files stored in HDFS shows the viability of the proposed strategy.

• 한국정보전자통신기술학회논문지
• /
• 제10권1호
• /
• pp.1-8
• /
• 2017

본 논문에서는 Hadoop에 공간 데이터를 저장하고 SQL 기반 질의언어에 의하여 공간 데이터를 질의할 수 있는 공간 빅데이터 질의처리 시스템을 제시한다. 제안한 시스템은 대용량의 공간 빅데이터를 HDFS 기반의 저장 시스템에 저장하고 공간 데이터 처리기능이 추가확장된 SQL 기반 질의언어로 질의를 할 수 있도록 지원하며 OGC 심플 피처모델 기반의 공간 데이터 표준 데이터타입과 함수를 지원한다. 본 논문에서는 질의언어 파싱, 질의언어 검증, 질의계획 생성, 저장시스템 연동 등 질의처리의 주요 기능 개발을 제시하였다. 제안한 시스템의 성능을 기존 시스템과 비교하였으며, 실험에서는 Hadoop에 저장된 공간 데이터에 대한 영역질의의 질의실행시간에 있어서 비교 시스템 대비 약 58%의 성능향상을 나타냄을 보였다.

• Journal of Information Processing Systems
• /
• 제17권6호
• /
• pp.1035-1043
• /
• 2021

An Internet of Things (IOT) sensor network is an effective solution for monitoring environmental conditions. However, IOT sensor networks generate massive data such that the abilities of massive data storage, processing, and query become technical challenges. To solve the problem, a Hadoop cloud platform is proposed. Using the time and workload genetic algorithm (TWLGA), the data processing platform enables the work of one node to be shared with other nodes, which not only raises efficiency of one single node but also provides the compatibility support to reduce the possible risk of software and hardware. In this experiment, a Hadoop cluster platform with TWLGA scheduling algorithm is developed, and the performance of the platform is tested. The results show that the Hadoop cloud platform is suitable for big data processing requirements of IOT sensor networks.

• Journal of Communications and Networks
• /
• 제17권6호
• /
• pp.592-601
• /
• 2015

Multitenancy has gained growing importance with the development and evolution of cloud computing technology. In a multitenant environment, multiple tenants with different demands can share a variety of computing resources (e.g., CPU, memory, storage, network, and data) within a single system, while each tenant remains logically isolated. This useful multitenancy concept offers highly efficient, and cost-effective systems without wasting computing resources to enterprises requiring similar environments for data processing and management. In this paper, we propose a novel approach supporting multitenancy features for Apache Hadoop, a large scale distributed system commonly used for processing big data. We first analyze the Hadoop framework focusing on "yet another resource negotiator (YARN)", which is responsible for managing resources, application runtime, and access control in the latest version of Hadoop. We then define the problems for supporting multitenancy and formally derive the requirements to solve these problems. Based on these requirements, we design the details of multitenant Hadoop. We also present experimental results to validate the data access control and to evaluate the performance enhancement of multitenant Hadoop.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• 제16권11호
• /
• pp.3619-3637
• /
• 2022

Big data analytics offers endless opportunities for operational enhancement by extracting valuable insights from complex voluminous data. Hadoop is a comprehensive technological suite which offers solutions for the large scale storage and computing needs of Big data. The performance of Hadoop is closely tied with its configuration settings which depends on the cluster capacity and the application profile. Since Hadoop has over 190 configuration parameters, tuning them to gain optimal application performance is a daunting challenge. Our approach is to extract a subset of impactful parameters from which the performance enhancing sub-optimal configuration is then narrowed down. This paper presents a statistical model to analyze the significance of the effect of Hadoop parameters on a variety of performance metrics. Our model decomposes the total observed performance variation and ascribes them to the main parameters, their interaction effects and noise factors. The method clearly segregates impactful parameters from the rest. The configuration setting determined by our methodology has reduced the Job completion time by 22%, resource utilization in terms of memory and CPU by 15% and 12% respectively, the number of killed Maps by 50% and Disk spillage by 23%. The proposed technique can be leveraged to ease the configuration tuning task of any Hadoop cluster despite the differences in the underlying infrastructure and the application running on it.

• 정보처리학회논문지:컴퓨터 및 통신 시스템
• /
• 제2권12호
• /
• pp.569-576
• /
• 2013

최근 많은 관심을 받고 있는 클라우드 컴퓨팅이 해결해야할 가장 큰 문제는 바로 보안이다. 일반적인 클라우드 컴퓨팅 환경에서는 사용자의 데이터 보호를 위해 수많은 분산서버를 이용하여 데이터를 저장한다. 하지만 분산서버에 저장된 데이터를 암호화 과정을 거치지 않고 그대로 저장하게 된다면, 마스터 서버에 저장된 분산파일 위치를 추적하여 그대로 유출이 되는 문제가 발생할 수 있다. 이러한 문제를 방지하기 위해 비밀키를 이용하여 분산된 데이터를 암호화해야할 필요성이 존재한다. 그러나 대용량 데이터의 경우 수십, 수백 개의 조각으로 나누어지게 되는데 분산서버마다 각각의 비밀키를 이용하게 된다면, 관리의 어려움이 존재할 뿐 아니라 분산 서버에 대한 정당한 인증, 암복호화 과정을 수없이 거치게 되어 막대한 오버헤드가 발생하게 된다. 따라서 본 논문에서는 이와 같은 문제점을 해결하기 위해 Hadoop 환경에 적합한 XOR 및 RAID기반의 효율적인 분산 저장 및 복구 기법을 제안하였다.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• 제14권12호
• /
• pp.4648-4663
• /
• 2020

This paper proposes the Lempel-Ziv 4(LZ4) compression accelerator optimized for scale-out servers in data centers. In order to reduce CPU loads caused by compression, we propose an accelerator solution and implement the accelerator on an Field Programmable Gate Array(FPGA) as heterogeneous computing. The LZ4 compression hardware accelerator is a fully pipelined architecture and applies 16 dictionaries to enhance the parallelism for high throughput compressor. Our hardware accelerator is based on the 20-stage pipeline and dictionary architecture, highly customized to LZ4 compression algorithm and parallel hardware implementation. Proposing dictionary architecture allows achieving high throughput by comparing input sequences in multiple dictionaries simultaneously compared to a single dictionary. The experimental results provide the high throughput with intensively optimized in the FPGA. Additionally, we compare our implementation to CPU implementation results of LZ4 to provide insights on FPGA-based data centers. The proposed accelerator achieves the compression throughput of 639MB/s with fine parallelism to be deployed into scale-out servers. This approach enables the low power Intel Atom processor to realize the Hadoop storage along with the compression accelerator.