• KIPS Transactions on Software and Data Engineering
• /
• v.6 no.1
• /
• pp.1-8
• /
• 2017

Given a multi-dimensional dataset of tuples, a skyline query returns a subset of tuples which are not 'dominated' by any other tuples. Skyline query is very useful in Big data analysis since it filters out uninteresting items. Much interest was devoted to the MapReduce-based parallel processing of skyline queries in large-scale distributed environment. There are three requirements to improve parallelism in MapReduced-based algorithms: (1) workload should be well balanced (2) avoid redundant computations (3) Optimize network communication cost. In this paper, we introduce MR-SEAP (MapReduce sample Skyline object Equality Angular Partitioning), an efficient angular space partitioning based skyline query processing using sampling-based pruning, which satisfies requirements above. We conduct an extensive experiment to evaluate MR-SEAP.

• Journal of the Korea Society of Computer and Information
• /
• v.20 no.8
• /
• pp.1-6
• /
• 2015

The MapReduce Program of Hadoop Distributed File System operates on any unspecified nodes due to distributed-parallel process and block replicate for data stability. Since it is difficult to guarantee the cache locality when a Solid State Drive is used as a cache in hadoop, cache hit-rate is decreased. In this paper, we suggest a method to improve cache hit rate by pre-loading the input data of the MapReduce onto the SSD cache. To perform this method, we estimated the blocks that are used on each node by using capacity scheduler and block metadata. Eventually we could increase the performance of SSD cache by loading the blocks onto SSD cache before the Map Task run.

• International Journal of Computer Science & Network Security
• /
• v.22 no.5
• /
• pp.121-126
• /
• 2022

Hadoop and Apache Spark are Apache Software Foundation open source projects, and both of them are premier large data analytic tools. Hadoop has led the big data industry for five years. The processing velocity of the Spark can be significantly different, up to 100 times quicker. However, the amount of data handled varies: Hadoop Map Reduce can process data sets that are far bigger than Spark. This article compares the performance of both spark and map and discusses the advantages and disadvantages of both above-noted technologies.

• Journal of the Korea Institute of Information and Communication Engineering
• /
• v.21 no.6
• /
• pp.1183-1190
• /
• 2017

With the development of network and IT technology, people are searching and purchasing items they want, not bounded by places. Therefore, there are various studies on how to solve the scalability problem due to the rapidly increasing data in the recommendation system. In this paper, we propose an item-based collaborative filtering method using Tag weight and a recommendation technique using MapReduce method, which is a distributed parallel processing method. In order to improve speed and efficiency, the proposed method classifies items into categories in the preprocessing and groups according to the number of nodes. In each distributed node, data is processed by going through Map-Reduce step 4 times. In order to recommend better items to users, item tag weight is used in the similarity calculation. The experiment result indicated that the proposed method has been more enhanced the appropriacy compared to item-based method, and run efficiently on the large amounts of data.

• Journal of Information Technology and Architecture
• /
• v.9 no.4
• /
• pp.455-464
• /
• 2012

Due to the recent data explosion, methods which can meet the requirement of large data analysis has been studying. This paper proposes MRIterativeBUC algorithm which enables efficient computation of large data cube by distributed parallel processing with MapReduce framework. MRIterativeBUC algorithm is developed for efficient iterative operation of the BUC method with MapReduce, and overcomes the limitations about the storage size and processing ability caused by large data cube computation. It employs the idea from the iceberg cube which computes only the interesting aspect of analysts and the distributed parallel process of cube computation by partitioning and sorting. Thus, it reduces data emission so that it can reduce network overload, processing amount on each node, and eventually the cube computation cost. The bottom-up cube computation and iterative algorithm using MapReduce, proposed in this paper, can be expanded in various way, and will make full use of many applications.

• The Journal of Korea Institute of Information, Electronics, and Communication Technology
• /
• v.11 no.5
• /
• pp.593-600
• /
• 2018

Owing to the advancement of Internet and smart devices, access to various media such as social media became easy; thus, a large amount of big data is being produced. Particularly, the companies that provide various Internet services are analyzing the big data by using the MapReduce-based big data analysis techniques to investigate the customer preferences and patterns and strengthen the security. However, with MapReduce, when the big data is analyzed by defining the number of reducer objects generated in the reduce stage as one, the processing rate of big data analysis decreases. Therefore, in this paper, a MapReduce-based split big data analysis method is proposed to improve the log analysis processing rate. The proposed method separates the reducer partitioning stage and the analysis result combining stage and improves the big data processing rate by decreasing the bottleneck phenomenon by generating the number of reducer objects dynamically.

• KIISE Transactions on Computing Practices
• /
• v.21 no.11
• /
• pp.681-688
• /
• 2015

The k nearest neighbor (k-NN) graph construction is an important operation with many web-related applications, including collaborative filtering, similarity search, and many others in data mining and machine learning. Despite its many elegant properties, the brute force k-NN graph construction method has a computational complexity of $O(n^2)$, which is prohibitive for large scale data sets. Thus, (Key, Value)-based distributed framework, MapReduce, is gaining increasingly widespread use in Locality Sensitive Hashing which is efficient for high-dimension and sparse data. Based on the two-stage strategy, we engage the locality sensitive hashing technique to divide users into small subsets, and then calculate similarity between pairs in the small subsets using a brute force method on MapReduce. Specifically, generating a candidate group stage is important since brute-force calculation is performed in the following step. However, existing methods do not prevent large candidate groups. In this paper, we proposed an efficient algorithm for approximate k-NN graph construction by regrouping candidate groups. Experimental results show that our approach is more effective than existing methods in terms of graph accuracy and scan rate.

• Proceedings of the Korean Information Science Society Conference
• /
• 2011.06b
• /
• pp.357-360
• /
• 2011

최근 인터넷과 통신기술, 특히 모바일과 관련된 기술의 급속한 발전으로 소셜 커뮤니케이션 수단으로 대표되는 SNS(Social Networking Service)가 중요한 이슈로 부각되어지고 있다. SNS 서비스 제공시 중요하게 고려되어져야 할 사항은 정확하고 의미 있는 데이터를 통해서 사용자가 원하고 관심 있는 분야의 정보를 어떻게 제공할 것인가에 초점이 맞춰져 있어야 한다. 그러나 최근 폭발적으로 증가되어지고 있는 소셜 데이터 때문에 사용자는 의미 분석이 정확하게 이루어지지 않은 신뢰성이 결여된 소셜 커뮤니케이션 서비스를 제공받고 있다. 이러한 소셜데이터 분석의 문제점을 해결하기 위해서 본 논문에서는 소셜 네트워크 서비스에 필요한 데이터를 수집하고, 클라우드 컴퓨팅 환경에서 수집된 대용량 SNS 데이터의 의미를 분석 할 수 있는 MapReduce 기반의 분석 모듈의 구조를 제안하였다. 제안한 모듈은 의미 분석에 필요한 소셜 데이터를 수집하는 수집 기능과 수집된 소셜데이터의 의미 분석을 수행하는 분석 기능을 포함하고 있다. 수집 기능은 SNS에서 생성되는 텍스트 형태의 데이터를 수집하고 MapReduce를 통해서 데이터를 분석하기 쉽게 적절한 크기로 생성된 파일을 분할한다. 수집된 소셜 데이터의 의미 분석은 기존 TF-IDF 방식에 개선된 Weighted-MINMAX 적용한 알고리즘을 통해서 구현하였다. 개선된 알고리즘은 단어의 중요도를 평가하고, 중요도가 높은 단어로 구성된 의미정보 제공 서비스를 지원한다. 시스템의 성능 평가를 위해서 노드별 데이터 처리시간과 추출 키워드의 정확도를 측정하였다.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2014.01a
• /
• pp.9-13
• /
• 2014

The performance of large scale software applications has been automatically increasing for last few decades under the influence of Moore's law - the number of transistors on a microprocessor roughly doubled every eighteen months. However, on-chip transistors limitations and heating issues led to the emergence of multicore processors. The energy efficient ARM based System-on-Chip (SoC) processors are being considered for future high performance computing systems. In this paper, we present a case study of two widely used parallel programming models i.e. MPI and MapReduce on distributed memory cluster of ARM SoC development boards. The case study application, Black-Scholes option pricing equation, was parallelized and evaluated in terms of power consumption and throughput. The results show that the Hadoop implementation has low instantaneous power consumption that of MPI, but MPI outperforms Hadoop implementation by a factor of 1.46 in terms of total power consumption to execution time ratio.

• KIPS Transactions on Software and Data Engineering
• /
• v.3 no.2
• /
• pp.73-80
• /
• 2014

MapReduce is a widely used programming model for analyzing and processing Big data. Aggregate queries are one of the most common types of queries used for analyzing Big data. In this paper, we propose an efficient method for processing an aggregate query stream, where many concurrent users continuously issue different aggregate queries on the same data. Instead of processing each aggregate query separately, the proposed method processes multiple aggregate queries together in a batch by a single, optimized MapReduce job. As a result, the number of queries processed per unit time increases significantly. Through various experiments, we show that the proposed method improves the performance significantly compared to a naive method.