• Journal of Advanced Navigation Technology
• /
• v.17 no.2
• /
• pp.225-233
• /
• 2013

Recently, Cloud Computing and Big Data has become a buzzword in the field of IT. In this paper, as part of special efforts to support small businesses (SMB) in these situations, we designed the basic prototypes ver. 0.1, 0.2, and 0.5 for Hadoop cluster based on private cloud infrastructure and implemented the part of basic prototypes. And we verified the performances of the basic prototypes using ASA Dataset.

• KIPS Transactions on Software and Data Engineering
• /
• v.5 no.11
• /
• pp.521-526
• /
• 2016

As on-line informal text data have massive in its volume and have unstructured characteristics in nature, there are limitations in applying traditional relational data model technologies for data storage and data analysis jobs. Moreover, using dynamically generating massive social data, social user's real-time reaction analysis tasks is hard to accomplish. In the paper, to capture easily the semantics of massive and informal on-line documents with unsupervised learning mechanism, we design and implement automatic topic extraction systems according to the mass of the words that consists a document. The input data set to the proposed system are generated first, using N-gram algorithm to build multiple words to capture the meaning of the sentences precisely, and Hadoop and Spark (In-memory distributed computing framework) are adopted to run topic model. In the experiment phases, TB level input data are processed for data preprocessing and proposed topic extraction steps are applied. We conclude that the proposed system shows good performance in extracting meaningful topics in time as the intermediate results come from main memories directly instead of an HDD reading.

• KIPS Transactions on Software and Data Engineering
• /
• v.4 no.2
• /
• pp.77-82
• /
• 2015

Big Data technology has been attracted much attention in aspect of fast data processing. Research of practicing Big Data technology is also ongoing to process large-scale structured data much faster in Relatioinal Database(RDB). Although there are lots of studies about measuring analyzing performance, studies about structured data loading performance, prior step of analyzing, is very rare. Thus, in this study, structured data in RDB is tested the performance that loads distributed processing platform Hadoop using Apache sqoop. Also in order to analyze the influence factors of data loading, it is tested repeatedly with different options of data loading and compared with data loading performance among RDB based servers. Although data loading performance of Apache Sqoop in test environment was low, but in large-scale Hadoop cluster environment we can expect much better performance because of getting more hardware resources. It is expected to be based on study improving data loading performance and whole steps of performance analyzing structured data in Hadoop Platform.

• The Journal of the Korea institute of electronic communication sciences
• /
• v.16 no.1
• /
• pp.19-24
• /
• 2021

In this paper, we study performance factor of machine learning in the distributed environment using Apache Spark and presents an efficient distributed processing method through experiments. This work firstly presents performance factor when performing machine learning in a distributed cluster by classifying cluster performance, data size, and configuration of spark engine. In addition, performance study of regression analysis using Spark MLlib running on the Hadoop cluster is performed while changing the configuration of the node and the Spark Executor. As a result of the experiment, it was confirmed that the effective number of executors was affected by the number of data blocks, but depending on the cluster size, the maximum and minimum values were limited by the number of cores and the number of worker nodes, respectively.

• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 2014.10a
• /
• pp.915-916
• /
• 2014

With the upcoming era of internet of things, the study of sensor network has been paid attention. Raspberry pi is a tiny versatile computer system which is able to act as a sensor node in hadoop cluster network. In this paper, we deployed 5 Raspberry pi's to construct an experimental testbed of hadoop sensor network with 5-node map-reduce hadoop software framework. We compared and analyzed the network architecture in terms of efficiency, resource management, and throughput using various parameters. We used a learning machine with support vector machine as test workload. In our experiments, Raspberry pi fulfilled the role of distributed computing sensor node in the sensor network.

• Annual Conference of KIPS
• /
• 2017.11a
• /
• pp.691-693
• /
• 2017

하둡 시스템은 대용량의 데이터를 처리할 수 있는 클러스터 기반 개방형 소프트웨어 프레임워크이다. 이는 하둡 분산 파일시스템(HDFS)과 MapReduce 모델을 활용하여 데이터의 병렬 처리를 지원한다. 본 연구에서는 3D 프린터를 위한 3D 모델 데이터를 G-code로 변환하는 알고리즘을 하둡을 활용하여 구현하였다. 4대의 컴퓨터에 하둡 시스템을 설치한 후 전처리-Map-Shuffling-Reduce의 과정을 거쳐 변환작업이 효율적으로 처리하였음을 보일 수 있었다.

• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 2016.05a
• /
• pp.376-377
• /
• 2016

Performance of Hadoop, which is a distributed data processing framework for big data analysis, is affected by several characteristics of each node in distributed cluster such as processing power and network bandwidth. This paper analyzes previous approaches for heterogeneous hadoop clusters, and presents several requirements for distributed node clustering in small-and-medium sized hospitals by considering computing environments of the hospitals.

• Journal of Korea Society of Digital Industry and Information Management
• /
• v.13 no.2
• /
• pp.77-85
• /
• 2017

Due to the expansion of accommodation space caused by increase of population along with lifestyle changes, most of people spend their time indoor except for the travel time. Because of this, environmental change of indoor is very important, and it affects people's health and economy in resources. But, most of people don't acknowledge the importance of indoor environment. Thus, monitoring system for sustaining and managing indoor environment systematically is needed, and big data clusters should be used in order to save and manage numerous sensor data collected from many spaces. In this paper, we design a big data cluster for the indoor environment monitoring in order to store the sensor data and monitor unit of the huge building Implementation design big data cluster-based system for the analysis, and a distributed file system and building a Hadoop, HBase for big data processing. Also, various sensor data is saved for collection, and effective indoor environment management and health enhancement through monitoring is expected.

• Journal of Advanced Navigation Technology
• /
• v.18 no.5
• /
• pp.468-475
• /
• 2014

The recommend system is getting more difficult real time recommend by lager preference data set, computing power and recommend algorithm. For this reason, recommend system is proceeding actively one's studies toward distribute processing method of large preference data set. This paper studied distribute processing method of large preference data set using hadoop distribute processing platform and mahout machine learning library. The recommend algorithm is used Co-occurrence Matrix similar to item Collaborative Filtering. The Co-occurrence Matrix can do distribute processing by many node of hadoop cluster, and it needs many computation scale but can reduce computation scale by distribute processing. This paper has simplified distribute processing of co-occurrence matrix by changes over from four stage to three stage. As a result, this paper can reduce mapreduce job and can generate recommend file. And it has a fast processing speed, and reduce map output data.

• KIISE Transactions on Computing Practices
• /
• v.22 no.11
• /
• pp.601-613
• /
• 2016

With the increasing need for big data processing, building a shared big data processing platform is important to minimize time and monetary costs. In shared big data processing, multitenancy is a major requirement that needs to be addressed, in order to provide a single isolated personal big data platform for each user, but to share the underlying hardware is shared among users to increase hardware utilization. In this paper, we explore two well-known shared big data processing platform models. One is to use a native Hadoop cluster, and the other is to build a virtual Hadoop cluster for each user. For each model we verified whether it is sufficient to support multi-tenancy. We also present a method to complement unsupported multi-tenancy features in a native Hadoop cluster model. Lastly we built prototype platforms and compared the performance of both models.