• KIPS Transactions on Software and Data Engineering
• /
• v.5 no.6
• /
• pp.283-288
• /
• 2016

In recent years, there have been many research efforts on Big Data, and many companies developed a variety of relevant products. Accordingly, we are able to store and analyze a large volume of log data, which have been difficult to be handled in the traditional computing environment. To handle a large volume of log data, which rapidly occur in multiple servers, in this paper we design a new data storage architecture to efficiently analyze those big log data through Apache Hive. We then design and implement anomaly detection methods, which identify abnormal status of servers from log data, based on moving average and 3-sigma techniques. We also show effectiveness of the proposed detection methods by demonstrating that our methods identifies anomalies correctly. These results show that our anomaly detection is an excellent approach for properly detecting anomalies from Hadoop log data.

• Journal of the Institute of Electronics and Information Engineers
• /
• v.50 no.8
• /
• pp.203-211
• /
• 2013

Due to the rapid growth of mobile devices and applications, mobile cloud computing is becoming an important platform in the development of cloud services. However, the mobile cloud computing is facing many challenges in terms of the computing resources and communications. One of them is the performance issue between mobile devices and cloud server. In the paper, we implemented a hadoop-based android mobile application, called mobile TPC-C, and used it for evaluating the performance aspect between mobile devices and cloud server. The mobile TPC-C was implemented based on the existing TPC-C, to make it possible to execute on top of android mobile devices. The performance measurement using mobile TPC-C was executed on various transactions while changing the number of mobile clients. By comparing it to the evaluation on the personal PC, we tried to point out the important aspects affecting the performance improvement between mobile clients and cloud server.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2013.11a
• /
• pp.740-743
• /
• 2013

클라우드 컴퓨팅 환경에서는 사용자의 데이터를 수많은 분산서버를 이용하여 데이터를 암호화하여 저장한다. 구글, 야후 등 글로벌 인터넷 서비스 업체들은 인터넷 서비스 플랫폼의 중요성을 인식하고 자체 연구 개발을 수행, 저가 상용 노드를 기반으로 한 대규모 클러스터 기반의 클라우드 컴퓨팅 플랫폼 기술을 개발 활용하고 있다. 이와 같이 분산 컴퓨팅 환경에서 다양한 데이터 서비스가 가능해지면서 대용량 데이터의 분산관리가 주요 이슈로 떠오르고 있다. 한편, 대용량 데이터의 다양한 이용 형태로부터 악의적인 공격자나 내부 사용자에 의한 보안 취약성 및 프라이버시 침해가 발생할 수 있다. 특히, 하둡에서 데이터 블록의 권한 제어를 위해 사용하는 블록 접근 토큰에도 다양한 보안 취약점이 발생한다. 이러한 보안 취약점을 보완하기 위해 본 논문에서는 비밀분산 기반의 블록 접근 토큰 관리 기법을 제안한다.

• KIISE Transactions on Computing Practices
• /
• v.21 no.10
• /
• pp.651-657
• /
• 2015

The amount of web traffic has increased as a result of the rapid growth of the use of web-based applications. In order to obtain valuable information from web logs, we need to develop systems that can support interactive, flexible, and efficient ways to analyze and handle large amounts of data. In this paper, we present CERES, a log-based, interactive web analytics system for backbone networks. Since CERES focuses on analyzing web log records generated from backbone networks, it is possible to perform a web analysis from the perspective of a network. CERES is designed for deployment in a server cluster using the Hadoop Distributed File System (HDFS) as the underlying storage. We transform and store web log records from backbone networks into relations and then allow users to use a SQL-like language to analyze web log records in a flexible and interactive manner. In particular, we use the data cube technique to enable the efficient statistical analysis of web log. The system provides users a web-based, multi-modal user interface.

• Journal of Digital Contents Society
• /
• v.14 no.4
• /
• pp.513-519
• /
• 2013

In this paper, we provide interfaces according to user application environments and provide tools through web-site that users can create interface to apply a wide range of application environment. HTML5 is used in the creation processing, so users can create various interfaces by dragging mouse and apply it to multimedia, game applications as well as documents by using the ASCII code and key events that are provided in the Android OS. Database of interfaces is stored in HDFS (Hadoop Distributed File System) based on Hadoop for management and users can have their own designed interface or select interfaces through simple login any time. In order to provide interface quickly, HIVE based on Hadoop is used for search and the data is provided in XML file which smart mobile can process quickly.

• Journal of Advanced Navigation Technology
• /
• v.18 no.2
• /
• pp.191-199
• /
• 2014

Recently, researches to facilitate utilization by small and medium business (SMB) of cloud computing and big data paradigm, which is the booming adoption of IT area, has been on the increase. As one of these efforts, in this paper, we design and implement the prototype to tentatively build up Hadoop cluster under private cloud infrastructure environments. Prototype implementation are made on each hardware type such as single board, PC, and server and performance is measured. Also, we present the integrated verification results for the data analysis performance of the analysis software system running on top of realized prototypes by employing ASA (American Standard Association) Dataset. For this, we implement the analysis software system using several open sources such as R, Python, D3, and java and perform a test.

• The Journal of Korea Institute of Information, Electronics, and Communication Technology
• /
• v.10 no.4
• /
• pp.340-349
• /
• 2017

This paper is proposed Hadoop security protocol to protect a reply attack and impersonation attack. The proposed hadoop security protocol is consists of user authentication module, public key based data node authentication module, name node authentication module, and data node authentication module. The user authentication module is issued the temporary access ID from TGS after verifing user's identification on Authentication Server. The public key based data node authentication module generates secret key between name node and data node, and generates OTKL(One-Time Key List) using Hash-chain. The name node authentication module verifies user's identification using user's temporary access ID, and issues DT(Delegation Token) and BAT(Block Access Token) to user. The data node authentication module sends the encrypted data block to user after verifing user's identification using OwerID of BAT. Therefore the proposed hadoop security protocol dose not only prepare the exposure of data node's secret key by using OTKL, timestamp, owerID but also detect the reply attack and impersonation attack. Also, it enhances the data access of data node, and enforces data security by sending the encrypted data.

• Proceedings of the Korean Information Science Society Conference
• /
• 2011.06d
• /
• pp.163-166
• /
• 2011

클라우드 컴퓨팅이란 인터넷 기술을 활용하여 모든 인프라 자원(소프트웨어, 서버, 스토리지, 네트워크 등)을 서비스화(as a Service)하여, 언제, 어디서든, 장치에 독립적으로 네트워크를 통해 사용하고, 사용한 만큼 비용을 지불하는 컴퓨팅으로써, 대표적인 서비스 업체로는 구글과 아마존이 있다. 최근 아파치 재단에서는 구글의 GFS와 동일 또는 유사한 시스템을 만들기 위해 HDFS 오픈소스 프로젝트를 진행하고 있다. HDFS는 빈번한 하드웨어 고장에도 원본 데이터를 복구할 수 있는 가용성을 보장하기 위해 파일 데이터를 블록 단위로 나누어, 다시 datanode에 복제하여 저장한다. 이 기법은 복제가 많아 질수록 가용성은 높아지나 스토리지가 증가한다는 단점을 가지고 있다. 따라서 본 논문에서는 이러한 문제점을 해결하기 위해 행렬의 특성을 이용한 새로운 분산 저장 기법을 제안한다.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2015.10a
• /
• pp.1272-1275
• /
• 2015

최근 관심이 증대되고 있는 빅데이터 분석 및 처리를 위한 병렬분산처리 시스템은 대용량 서버가 필요하고 인프라 구축을 위해 고비용을 지불해야 한다. 이를 해결하기 위해 본 연구에서는 저렴한 라즈베리 파이로 클러스터를 구성하고, 하둡보다 빠른 속도의 처리를 제공하는 아파치 스파크를 분석 솔루션으로 하는 빅데이터 분석 플랫폼을 구축하였다. 구축한 플랫폼이 빅데이터 활용을 위해 적절한 성능을 보이는지 확인하기 위해 텍스트 마이닝을 수행하였고, 분석 결과 유효한 성능을 보였다. 적절한 비용으로 빅데이터 분석이 가능해지면서 중소기업과 개인, 교육 기관에서도 빅데이터 활용이 가능해지면서 활용 분야가 크게 확대될 것으로 보인다.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.19 no.3
• /
• pp.669-678
• /
• 2018

This study developed information technology infrastructures for building a driving environment analysis platform using various big data, such as vehicle sensing data, public data, etc. First, a small platform server with a parallel structure for big data distribution processing was developed with H/W technology. Next, programs for big data collection/storage, processing/analysis, and information visualization were developed with S/W technology. The collection S/W was developed as a collection interface using Kafka, Flume, and Sqoop. The storage S/W was developed to be divided into a Hadoop distributed file system and Cassandra DB according to the utilization of data. Processing S/W was developed for spatial unit matching and time interval interpolation/aggregation of the collected data by applying the grid index method. An analysis S/W was developed as an analytical tool based on the Zeppelin notebook for the application and evaluation of a development algorithm. Finally, Information Visualization S/W was developed as a Web GIS engine program for providing various driving environment information and visualization. As a result of the performance evaluation, the number of executors, the optimal memory capacity, and number of cores for the development server were derived, and the computation performance was superior to that of the other cloud computing.