• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 2013.05a
• /
• pp.681-683
• /
• 2013

And were the prevalence of smartphones is the Big Data era, such as Facebook or Twitter, SNS (Social Network Service) routine is used in the real world. Take advantage of the analysis, and to extract and utilize developed in the Apache Foundation Hadoop (Hadoop) without abandoning the SNS unstructured data here. Hadoop is an open source framework that can handle large amounts of data. Hadoop has been introduced in the domestic corporate and commercial development and Compared to the technology development Hadoop has been pointed out that the lack of security sector. In this paper, we propose a method to enhance the security and vulnerability analysis of security technologies and Hadoop.

• Journal of the Korea Institute of Information Security & Cryptology
• /
• v.23 no.5
• /
• pp.803-813
• /
• 2013

As the developments of smart devices and social network services, the amount of data has been exploding. The world is facing Big data era. For these reasons, the Big data processing technology which is a new technology that can handle such data has attracted much attention. One of the most representative technologies is Hadoop. Hadoop Distributed File System(HDFS) designed to run on commercial Linux server is an open source framework and can store many terabytes of data. The initial version of Hadoop did not consider security because it only focused on efficient Big data processing. As the number of users rapidly increases, a lot of sensitive data including personal information were stored on HDFS. So Hadoop announced a new version that introduces Kerberos and token system in 2009. However, this system is vulnerable to the replay attack, impersonation attack and other attacks. In this paper, we analyze these vulnerabilities of HDFS security and propose a new protocol which complements these vulnerabilities and maintains the performance of Hadoop.

• International Journal of Computer Science & Network Security
• /
• v.23 no.9
• /
• pp.134-140
• /
• 2023

The distributed system is playing a vital role in storing and processing big data and data generation is speedily increasing from various sources every second. Hadoop has a scalable, and efficient distributed system supporting commodity hardware by combining different networks in the topographical locality. Node support in the Hadoop cluster is rapidly increasing in different versions which are facing difficulty to manage clusters. Hadoop does not provide Node management, adding and deletion node futures. Node identification in a cluster completely depends on DHCP servers which managing IP addresses, hostname based on the physical address (MAC) address of each Node. There is a scope to the hacker to theft the data using IP or Hostname and creating a disturbance in a distributed system by adding a malicious node, assigning duplicate IP. This paper proposing novel node management for the distributed system using DNA hiding and generating a unique key using a unique physical address (MAC) of each node and hostname. The proposed mechanism is providing better node management for the Hadoop cluster providing adding and deletion node mechanism by using limited computations and providing better node security from hackers. The main target of this paper is to propose an algorithm to implement Node information hiding in DNA sequences to increase and provide security to the node from hackers.

• Journal of Digital Convergence
• /
• v.12 no.8
• /
• pp.351-359
• /
• 2014

The log data generated by security equipment have been synthetically analyzed on the ESM(Enterprise Security Management) base so far, but due to its limitations of the capacity and processing performance, it is not suited for big data processing. Therefore the another way of technology on the big data platform is necessary. Big Data platform can achieve a large amount of data collection, storage, processing, retrieval, analysis, and visualization by using Hadoop Ecosystem. Currently ESM technology has developed in the way of SIEM (Security Information & Event Management) technology, and to implement security technology in SIEM way, Big Data platform technology is essential that can handle large log data which occurs in the current security devices. In this paper, we have a big data platform Hadoop Ecosystem technology for analyzing the security log for sure how to implement the system model is studied.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.12 no.11
• /
• pp.5252-5268
• /
• 2018

To support a large-scale Hadoop cluster, Hadoop heartbeat messages are designed to deliver the significant messages, including task scheduling and completion messages, via piggybacking to reduce the number of messages received by the NameNode. Although Hadoop is designed and optimized for high-throughput computing via batch processing, the real-time processing of large amounts of data in Hadoop is increasingly important. This paper evaluates Hadoop's performance and costs when the heartbeat period is controlled to support latency sensitive applications. Through an empirical study based on Hadoop 2.0 (YARN) architecture, we improve Hadoop's I/O performance as well as application performance by up to 13 percent compared to the default configuration. We offer a guideline that predicts the performance, costs and limitations of the total system by controlling the heartbeat period using simple equations. We show that Hive performance can be improved by tuning Hadoop's heartbeat periods through extensive experiments.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2016.01a
• /
• pp.305-306
• /
• 2016

현재 스마트 시대에 살고 있는 우리는 매우 복잡하고 거미줄처럼 연결되어 있는 빅 데이터 환경에서 살고 있다. 이런 환경에서는 대용량 데이터를 효율적으로 관리하고 활용하는 것이 개인이나 기업들이 추구하려는 목표이다. 빅 데이터 시대에 데이터의 효율적인 관리와 활용을 위해 다양한 장비에서 수집되고 저장된 대용량 데이터에 대해서 일반적인 데이터 분석을 통한 보안 기술로는 상당한 시간과 자원 낭비가 수반된다. 이를 개선하기 위해 본 논문에서는 하둡을 이용하여 대용량 데이터에 대한 처리 및 분석을 통해 효과적인 보안 시스템을 제안한다.

• Journal of the Korea Institute of Information Security & Cryptology
• /
• v.27 no.5
• /
• pp.1117-1127
• /
• 2017

With the advent of Internet of Things (IoT), the number of devices connected to Internet has rapidly increased, but the security for IoT is still vulnerable. It is difficult to integrate existing security technologies due to generating a large amount of traffic by using different protocols to use various IoT devices according to purposes and to operate in a low power environment. Therefore, in this paper, we propose a normal network behavior profiling method based on big data analysis techniques. The proposed method utilizes a Hadoop/Hive for Big Data analytics and an R for statistical computing. Also we verify the effectiveness of the proposed method through a simulation.

• International Journal of Computer Science & Network Security
• /
• v.22 no.5
• /
• pp.121-126
• /
• 2022

Hadoop and Apache Spark are Apache Software Foundation open source projects, and both of them are premier large data analytic tools. Hadoop has led the big data industry for five years. The processing velocity of the Spark can be significantly different, up to 100 times quicker. However, the amount of data handled varies: Hadoop Map Reduce can process data sets that are far bigger than Spark. This article compares the performance of both spark and map and discusses the advantages and disadvantages of both above-noted technologies.

• Journal of the Korea Institute of Information Security & Cryptology
• /
• v.23 no.5
• /
• pp.831-847
• /
• 2013

The various types of data are being created in large quantities resulting from the spread of social media and the mobile popularization. Many companies want to obtain valuable business information through the analysis of these large data. As a result, it is a trend to integrate the big data technologies into the company work. Especially, Hadoop is regarded as the most representative big data technology due to its terabytes of storage capacity, inexpensive construction cost, and fast data processing speed. However, the authentication token system of Hadoop Distributed File System(HDFS) for the user authentication is currently vulnerable to the replay attack and the datanode hacking attack. This can cause that the company secrets or the personal information of customers on HDFS are exposed. In this paper, we analyze the possible security threats to HDFS when tokens or datanodes are exposed to the attackers. Finally, we propose the secure authentication protocol in HDFS based on hash chain.

• Journal of the Korea Institute of Information and Communication Engineering
• /
• v.19 no.8
• /
• pp.1797-1804
• /
• 2015

Integrated log management system can help to predict the risk of security and contributes to improve the security level of the organization, and leads to prepare an appropriate security policy. In this paper, we have designed and implemented a Hadoop-based log analysis system by using distributed database model which can store large amount of data and reduce analysis time by automating log collecting procedure. In the proposed system, we use the HBase in order to store a large amount of data efficiently in the scale-out fashion and propose an easy data storing scheme for analysing data using a Hadoop-based normal expression, which results in improving data processing speed compared to the existing system.