• KIPS Transactions on Computer and Communication Systems
• /
• v.4 no.4
• /
• pp.141-146
• /
• 2015

Data have been increased enormously due to the development of IT technology such as recent smart equipments, social network services and streaming services. To meet these environments the technologies that can treat mass data have received attention, and the typical one is Hadoop. Hadoop is on the basis of open source, and it has been designed to be used at general purpose computers on the basis of Linux. To initial Hadoop nearly no security was introduced, but as the number of users increased data that need security increased and there appeared new version that introduced Kerberos and Token system in 2009. But in this method there was a problem that only one secret key can be used and access permission to blocks cannot be authenticated to each user, and there were weak points that replay attack and spoofing attack were possible. Hence, to supplement these weak points and to maintain efficiency a protocol on the basis of group key, in which users are authenticated in logical group and then this is reflected to token, is proposed in this paper. The result shows that it has solved the weak points and there is no big overhead in terms of efficiency.

• Journal of Broadcast Engineering
• /
• v.24 no.6
• /
• pp.1122-1133
• /
• 2019

We have designed and implemented a new data processing framework called MOHA(Mtc On HAdoop) which can effectively support Many-Task Computing(MTC) applications in a YARN-based Hadoop platform. MTC applications can be composed of a very large number of computational tasks ranging from hundreds of thousands to millions of tasks, and each MTC application may have different resource usage patterns. Therefore, we have implemented MOHA-TaskExecutor(a pilot-job that executes real MTC application tasks)'s Adaptive Parallel Computability which can adaptively execute multiple tasks simultaneously, in order to improve the parallel computability of a YARN container and the overall system throughput. We have implemented multi-threaded version of TaskExecutor which can "independently and dynamically" adjust the number of concurrently running tasks, and in order to find the optimal number of concurrent tasks, we have employed Hill-Climbing algorithm.

• Journal of the Korea Convergence Society
• /
• v.12 no.7
• /
• pp.45-51
• /
• 2021

In accordance with the rapid non-face-to-face environment and mobile first strategy, the explosive increase and creation of many structured/unstructured data every year demands new decision making and services using big data in all fields. However, there have been few reference cases of using the Hadoop Ecosystem, which uses the rapidly increasing big data every year to collect and load big data into a standard platform that can be applied in a practical environment, and then store and process well-established big data in a relational database. Therefore, in this study, after collecting unstructured data searched by keywords from social network services based on Hadoop 2.0 through three virtual machine servers in the Spring Framework environment, the collected unstructured data is loaded into Hadoop Distributed File System and HBase based on the loaded unstructured data, it was designed and implemented to store standardized big data in a relational database using a morpheme analyzer. In the future, research on clustering and classification and analysis using machine learning using Hive or Mahout for deep data analysis should be continued.

• The Journal of the Korea institute of electronic communication sciences
• /
• v.18 no.2
• /
• pp.361-366
• /
• 2023

Apache Hadoop, a representative framework for distributed processing of big data, has the advantage of increasing cluster size up to thousands of nodes to improve parallel distributed processing performance. However, reducing the size of the cluster is limited to the extent of permanently decommissioning nodes with defects or degraded performance, so there are limitations to operate multiple nodes flexibly in small clusters. In this paper, we discuss the problems that occur when removing nodes from the Hadoop cluster and propose a dynamic down-scaling technique to manage the distributed cluster more flexibly. To do this, we design and implement a modified Hadoop system and interfaces to support dynamic down-scaling of the cluster which supports temporary pause of a node and reconnection of it when necessary, rather than decommissioning the node when removing a node from the Hadoop cluster. We have verified that effective downsizing can be performed without performance degradation based on experimental results.

• Journal of Korea Multimedia Society
• /
• v.23 no.10
• /
• pp.1286-1295
• /
• 2020

It has been increasingly difficult to predict the amounts of Acer mono sap to be collected due to droughts and cold waves caused by recent climate changes with few studies conducted on the prediction of its collection volume. This study thus set out to propose a Big Data prediction system based on meteorological information for the collection of Acer mono sap. The proposed system would analyze collected data and provide managers with a statistical chart of prediction values regarding climate factors to affect the amounts of Acer mono sap to be collected, thus enabling efficient work. It was designed based on Hadoop for data collection, treatment and analysis. The study also analyzed and proposed an optimal prediction model for climate conditions to influence the volume of Acer mono sap to be collected by applying a multiple regression analysis model based on Hadoop and Mahout.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2016.07a
• /
• pp.191-193
• /
• 2016

본 논문에서는 SSD 저장장치를 포함하는 하둡의 Iterative Processing에 대한 성능 분석 결과를 소개한다. 하둡은 맵 리듀스 병렬 프로그래밍 모델을 통해 Batch Processing에 특화된 구조를 가지고 있는 프레임 워크이다. 이는 병렬/분산 환경에서 큰 성능향상을 보장하지만, 반복 작업을 수행하는 Iterative Processing에 대하여는 성능이 낮아지는 문제가 존재하고 있다. 이에 본 논문에서는 점차 낮아지는 가격으로 인해 하둡시스템에 적용 가능성이 타진되는 SSD를 통해 반복 작업의 성능이슈를 해결할 수 있는지 확인하고, SSD를 통한 성능향상의 요소가 존재하는지 알아보고자 실험을 진행하였다. 실험에서는 Batch Processing인 word count와 Iterative Processing인 Page Rank 알고리즘을 MapReduce로 구현하고 데이터 크기에 따른 성능 향상도를 측정하였고, SSD 추가와 같은 하드웨어적인 성능을 통한 하둡의 반복 작업은 큰 효율을 기대하기가 어렵다는 결론을 보였다.

• Annual Conference of KIPS
• /
• 2017.04a
• /
• pp.217-219
• /
• 2017

나날이 증가하는 해킹의 위협에 따라 이를 방어하기 위한 침임 탐지 시스템과 로그 수집 분야에서 많은 연구가 진행되고 있다. 이러한 연구들로 인해 다양한 종류의 침임 탐지 시스템이 생겨났으며, 이는 다양한 종류의 침입 탐지 시스템에서 서로의 단점을 보안할 필요성이 생기게 되었다. 따라서 본 논문에서는 네트워크 기반인 NIDS(Network-based IDS)와 호스트 기반인 HIDS(Host-based IDS)의 장단점을 가진 Hybrid IDS을 구성하기 위해 NIDS와 HIDS의 로그 데이터 통합을 위해 실시간 로그 처리에 특화된 Kafka를 이용하고, 실시간 분석에 Spark Streaming을 이용하여 통합된 로그를 분석하게 되며, 실시간 전송 도중에 발생되는 데이터 유실에 대해 별도로 저장되는 Hadoop의 HDFS에서는 데이터 유실에 대한 보장을 하는 실시간 Hybrid IDS 분석 시스템에 대한 설계를 제안한다.

• Journal of Korea Multimedia Society
• /
• v.20 no.11
• /
• pp.1785-1792
• /
• 2017

Generally, big data contains sensitive information about individuals, and thus directly releasing it for public use may violate existing privacy requirements. Therefore, privacy-preserving data publishing (PPDP) has been actively researched to share big data containing personal information for public use, while protecting the privacy of individuals with minimal data modification. Recently, with increasing demand for big data sharing in various area, there is also a growing interest in the development of software which supports a privacy-preserving data publishing. Thus, in this paper, we develops the system which aims to effectively and efficiently support privacy-preserving data publishing. In particular, the system developed in this paper enables data owners to select the appropriate anonymization level by providing them the information loss matrix. Furthermore, the developed system is able to achieve a high performance in data anonymization by using distributed Hadoop clusters.

• Proceedings of the Korean Information Science Society Conference
• /
• 2011.06d
• /
• pp.252-255
• /
• 2011

인터넷 트래픽의 증가로 인해서 트래픽을 분석, 관리의 필요성이 높아졌다. 인터넷 트래픽의 분석은 크게 패킷 기반의 트래픽 분석과 플로우 기반의 트래픽 분석으로 나눌 수 있다. 증가한 트래픽 양으로 인하여 패킷 기반의 트래픽 분석보다 정확도는 낮지만 빠른 속도를 보장하는 플로우 기반의 트래픽 분석이 널리 이용되고 있다. 하지만 플로우 데이터 분석 서버를 새롭게 구축하거나 기존의 서버를 업그레이드 하는 경우 초기 설치 비용 및 서버 관리 비용이 증가한다. 증가한 비용을 낮추고 트래픽을 분석하는 사용자가 쉽게 사용할 수 있으며 확장 가능하고 고 가용성을 보장하는 Hadoop 기반의 트래픽 분석 시스템을 제안하고자 한다.

• Annual Conference of KIPS
• /
• 2014.04a
• /
• pp.700-703
• /
• 2014

최근 이슈가 되고 있는 하둡(hadoop) 패키지에 접목하여 많은 프로젝트들이 생겨나고 있으며, 이들 중 주요하게 떠오르고 있는 분야가 접근 제어 기술이다. 특히, 인터넷의 발전과 스마트 기기 사용자가 늘어남에 따라 데이터의 양이 증가하여, 데이터의 소유자와 사용자의 필요에 의한 접근 제어 기술이 필요하게 되었다. 본 논문에서는 접근 제어 기술의 필요성을 기반으로 HDFS(Hadoop Distributed File System, 하둡 분산 파일 시스템) 기반의 새로운 데이터 접근 제어 프레임워크를 제안한다. 제안하는 방법은 새로운 메타데이터 저장 모듈과 접근 관리 모듈을 만들어 데이터 접근 제어프레임워크를 구성함으로써, 빅데이터 플랫폼을 사용하는 사용자들을 위한 접근 제어 기능을 제공한다. 제안한 프레임워크는 기존 플랫폼에 추가적인 설치가 필요 없도록 하둡 내부에 설계하여 향후 활용도가 높을 것이라 기대된다.