• Journal of Internet Computing and Services
• /
• v.20 no.5
• /
• pp.49-55
• /
• 2019

Organizations design and operate business process models to achieve their goals efficiently and systematically. With the advancement of IT technology, the number of items that computer systems can participate in and the process becomes huge and complicated. This phenomenon created a more complex and subdivide flow of business process.The process instances that contain workcase and events are larger and have more data. This is an essential resource for process mining and is used directly in model discovery, analysis, and improvement of processes. This event log is getting bigger and broader, which leads to problems such as capacity management and I / O load in management of existing row level program or management through a relational database. In this paper, as the event log becomes big data, we have found the problem of management limit based on the existing original file or relational database. Design and apply schemes to archive and analyze large event logs through Hadoop, an open source distributed file system, and HBase, a NoSQL database system.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2014.07a
• /
• pp.329-332
• /
• 2014

클라우드 컴퓨팅이 활성화 됨에 따라 기존의 파일 시스템과는 다른 대용량 파일 처리에 효율적인 분산파일시스템의 요구가 대두 되었다. 그 중에 하둡 분산 파일 시스템(Hadoop Distribute File System, HDFS)은 기존의 분산파일 시스템과는 달리 가용성과 내고장성을 보장하고, 데이터 접근 패턴을 스트리밍 방식으로 지원하여 대용량 파일을 효율적으로 저장할 수 있다. 이러한 장점 때문에, 클라우드 컴퓨팅의 파일시스템으로 대부분 채택하고 있다. 하지만 실제 HDFS 데이터 집합에서 대용량 파일 보다 소용량 파일이 차지하는 비율이 높으며, 이러한 다수의 소 용량 파일은 데이터 처리에 있어 높은 처리비용을 초래 할 뿐 만 아니라 메모리 성능에 악영향을 끼친다. 하지만 소 용량 파일을 프리패칭 함으로서 이러한 문제점을 해결 할 수 있다. HDFS의 데이터 프리페칭은 기존의 데이터 프리페칭의 기법으로는 적용하기 어려워 HDFS를 위한 데이터 프리패칭 기법을 제안한다.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2014.07a
• /
• pp.241-244
• /
• 2014

본 논문에서는 HDFS(Hadoop Distributed File System)에서 문제되고 있는 복제정책의 개선을 위해 HMM(Hidden Markov Model)을 이용한 동적 데이터 복제본 삭제 전략을 제안한다. HDFS는 대용량 데이터를 효과적으로 처리할 수 있는 분산 파일 시스템으로 높은 Fault-Tolerance를 제공하며, 데이터의 접근에 높은 처리량을 제공하여 대용량 데이터 집합을 갖는 응용 프로그램에 최적화 되어있는 장점을 가지고 있다. 하지만 HDFS 에서의 복제 메커니즘은 시스템의 안정성과 성능을 향상시키지만, 추가 블록 복제본이 많은 디스크 공간을 차지하여 유지보수 비용 또한 증가하게 된다. 본 논문에서는 HMM과 최상의 상태 순서를 찾는 알고리즘인 Viterbi Algorithm을 이용하여 불필요한 데이터 복제본을 탐색하고, 탐색된 복제본의 삭제를 통하여 HDFS의 디스크 공간과 유지보수 비용을 절약 할 수 있는 전략을 제안한다.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2013.05a
• /
• pp.45-48
• /
• 2013

In this paper, we proposed an Indexing Structure System (ISS) based on Apache Hadoop in Wireless Sensor Network (WSN). Nowadays sensors data continuously keep growing that need to control. Data constantly update in order to provide the newest information to users. While data keep growing, data retrieving and storing are face some challenges. So by using the ISS, we can maximize processing quality and minimize data retrieving time. In order to design ISS, Indexing Types have to be defined depend on each sensor type. After identifying, each sensor goes through the Indexing Structure Processing (ISP) in order to be indexed. After ISP, indexed data are streaming and storing in Hadoop Distributed File System (HDFS) across a number of separate machines. Indexed data are split and run by MapReduce tasks. Data are sorted and grouped depend on sensor data object categories. Thus, while users send the requests, all the queries will be filter from sensor data object and managing the task by MapReduce processing framework.

• Journal of the Korea Society of Computer and Information
• /
• v.20 no.8
• /
• pp.1-6
• /
• 2015

The MapReduce Program of Hadoop Distributed File System operates on any unspecified nodes due to distributed-parallel process and block replicate for data stability. Since it is difficult to guarantee the cache locality when a Solid State Drive is used as a cache in hadoop, cache hit-rate is decreased. In this paper, we suggest a method to improve cache hit rate by pre-loading the input data of the MapReduce onto the SSD cache. To perform this method, we estimated the blocks that are used on each node by using capacity scheduler and block metadata. Eventually we could increase the performance of SSD cache by loading the blocks onto SSD cache before the Map Task run.

• Journal of the Korea Institute of Information Security & Cryptology
• /
• v.23 no.5
• /
• pp.831-847
• /
• 2013

The various types of data are being created in large quantities resulting from the spread of social media and the mobile popularization. Many companies want to obtain valuable business information through the analysis of these large data. As a result, it is a trend to integrate the big data technologies into the company work. Especially, Hadoop is regarded as the most representative big data technology due to its terabytes of storage capacity, inexpensive construction cost, and fast data processing speed. However, the authentication token system of Hadoop Distributed File System(HDFS) for the user authentication is currently vulnerable to the replay attack and the datanode hacking attack. This can cause that the company secrets or the personal information of customers on HDFS are exposed. In this paper, we analyze the possible security threats to HDFS when tokens or datanodes are exposed to the attackers. Finally, we propose the secure authentication protocol in HDFS based on hash chain.

• Journal of the Korea Society of Computer and Information
• /
• v.22 no.1
• /
• pp.49-56
• /
• 2017

In this paper we propose the approach for big data integration so as to analyze, visualize and predict the future of the trend of the market, and that is to get the integration data model using the R language which is the future of the statistics and the Hadoop which is a parallel processing for the data. As four approaching methods using R and Hadoop, ff package in R, R and Streaming as Hadoop utility, and Rhipe and RHadoop as R and Hadoop interface packages are used, and the strength and weakness of four methods are described and analyzed, so Rhipe and RHadoop are proposed as a complete set of data integration model. The integration of R, which is popular for processing statistical algorithm and Hadoop contains Distributed File System and resource management platform and can implement the MapReduce programming model gives us a new environment where in R code can be written and deployed in Hadoop without any data movement. This model allows us to predictive analysis with high performance and deep understand over the big data.

• International Journal of Computer Science & Network Security
• /
• v.21 no.7
• /
• pp.359-364
• /
• 2021

In recent years Hadoop usage has been increasing day by day. The need of development of the technology and its specified outcomes are eagerly waiting across globe to adopt speedy access of data. Need of computers and its dependency is increasing day by day. Big data is exponentially growing as the entire world is working in online mode. Large amount of data has been produced which is very difficult to handle and process within a short time. In present situation industries are widely using the Hadoop framework to store, process and produce at the specified time with huge amount of data that has been put on the server. Processing of this huge amount of data having small files & its storage optimization is a big problem. HDFS, Sequence files, HAR, NHAR various techniques have been already proposed. In this paper we have discussed about various existing techniques which are developed for accessing and storing small files efficiently. Out of the various techniques we have specifically tried to implement the HDFS- HAR, NHAR techniques.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.9 no.8
• /
• pp.3182-3202
• /
• 2015

Flexibly expanding the storage capacity required to process a large amount of rapidly increasing unstructured log data is difficult in a conventional computing environment. In addition, implementing a log processing system providing features that categorize and analyze unstructured log data is extremely difficult. To overcome such limitations, we propose and design a MongoDB-based unstructured log processing system (MdbULPS) for collecting, categorizing, and analyzing log data generated from banks. The proposed system includes a Hadoop-based analysis module for reliable parallel-distributed processing of massive log data. Furthermore, because the Hadoop distributed file system (HDFS) stores data by generating replicas of collected log data in block units, the proposed system offers automatic system recovery against system failures and data loss. Finally, by establishing a distributed database using the NoSQL-based MongoDB, the proposed system provides methods of effectively processing unstructured log data. To evaluate the proposed system, we conducted three different performance tests on a local test bed including twelve nodes: comparing our system with a MySQL-based approach, comparing it with an Hbase-based approach, and changing the chunk size option. From the experiments, we found that our system showed better performance in processing unstructured log data.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.6 no.11
• /
• pp.2827-2848
• /
• 2012

Previously, we described a social media cloud computing service environment (SMCCSE). This SMCCSE supports the development of social networking services (SNSs) that include audio, image, and video formats. A social media cloud computing PaaS platform, a core component in a SMCCSE, processes large amounts of social media in a parallel and distributed manner for supporting a reliable SNS. Here, we propose a Hadoop-based multimedia system for image and video transcoding processing, necessary functions of our PaaS platform. Our system consists of two modules, including an image transcoding module and a video transcoding module. We also design and implement the system by using a MapReduce framework running on a Hadoop Distributed File System (HDFS) and the media processing libraries Xuggler and JAI. In this way, our system exponentially reduces the encoding time for transcoding large amounts of image and video files into specific formats depending on user-requested options (such as resolution, bit rate, and frame rate). In order to evaluate system performance, we measure the total image and video transcoding time for image and video data sets, respectively, under various experimental conditions. In addition, we compare the video transcoding performance of our cloud-based approach with that of the traditional frame-level parallel processing-based approach. Based on experiments performed on a 28-node cluster, the proposed Hadoop-based multimedia transcoding system delivers excellent speed and quality.