• Journal of Digital Convergence
• /
• v.14 no.9
• /
• pp.317-323
• /
• 2016

Big Data is now seen as a major asset in the company's competitiveness, its influence in the future is expected to grow. Companies that recognize the importance are already actively engaged with Big Data in product development and marketing, which are increasingly applied across sectors of society, including politics, sports. However, lack of knowledge of the system implementation and high costs are still a big obstacles to the introduction of Big Data and systems. It is an objective in this study to build a Big Data system, which is based on open source Hadoop and Hive among Big Data systems, utilizing POS sales data of small and medium-sized offline markets. This approach of convergence is expected to improve existing sales systems that have been simply focusing on profit and loss analysis. It will also be able to use it as the basis for the decisions of the executive to enable prediction of the consumption patterns of customer preference and demand in advance.

• KIPS Transactions on Software and Data Engineering
• /
• v.5 no.11
• /
• pp.521-526
• /
• 2016

As on-line informal text data have massive in its volume and have unstructured characteristics in nature, there are limitations in applying traditional relational data model technologies for data storage and data analysis jobs. Moreover, using dynamically generating massive social data, social user's real-time reaction analysis tasks is hard to accomplish. In the paper, to capture easily the semantics of massive and informal on-line documents with unsupervised learning mechanism, we design and implement automatic topic extraction systems according to the mass of the words that consists a document. The input data set to the proposed system are generated first, using N-gram algorithm to build multiple words to capture the meaning of the sentences precisely, and Hadoop and Spark (In-memory distributed computing framework) are adopted to run topic model. In the experiment phases, TB level input data are processed for data preprocessing and proposed topic extraction steps are applied. We conclude that the proposed system shows good performance in extracting meaningful topics in time as the intermediate results come from main memories directly instead of an HDD reading.

• The Journal of the Korea institute of electronic communication sciences
• /
• v.7 no.3
• /
• pp.591-596
• /
• 2012

Recently, the efficient management system for photovoltaic power plants has been required due to the continuously increasing construction of photovoltaic power plants. In this paper, we propose a cloud-based intelligent management system for many photovoltaic power plants. The proposed system stores the measured data of power plants using Hadoop HBase which is a column-oriented database, and processes the calculations of performance, efficiency, and prediction the amount of power generation by parallel processing based on Map-Reduce model. And, Web-based data visualization module allows the administrator to provide information in various forms.

• The Journal of the Korea institute of electronic communication sciences
• /
• v.5 no.6
• /
• pp.672-677
• /
• 2010

Recently, the efficient management system for large-scale sensor data has been required due to the increasing deployment of large-scale sensor networks. In this paper, we propose a cloud-based sensor data management system with low cast, high scalability, and efficiency. Sensor data in sensor networks are transmitted to the cloud through a cloud-gateway. At this point, outlier detection and event processing is performed. Transmitted sensor data are stored in the Hadoop HBase, distributed column-oriented database, and processed in parallel by query processing module designed as the MapReduce model. The proposed system can be work with the application of a variety of platforms, because processed results are provided through REST-based web service.

• Smart Media Journal
• /
• v.6 no.3
• /
• pp.75-82
• /
• 2017

Recently, the marine monitoring technology is actively being studied since the sea is a rich repository of natural resources that is taken notice in the world. In particular, the marine environment data should be collected continuously in order to understand and analyze the marine environment, however the study of automatic monitoring of marine environment in Korea is not enough. In this paper, we proposed the marine environment monitoring system based on open source. The proposed system can be designed as a scale out system using Hadoop based time series database which it can easily process the increasing collection data by a scale out computer resources. It can also be used to analyze marine data by visualizing collected data.

• Journal of KIISE
• /
• v.45 no.3
• /
• pp.280-287
• /
• 2018

Recently, the use of a recommendation system and tensor data analysis, which has high-dimensional data, is increasing, as they allow us to analyze the tensor and extract potential elements and patterns. However, due to the large size and complexity of the tensor, it needs to be decomposed in order to analyze the tensor data. While several tools are used for tensor decomposition such as rTensor, pyTensor, and MATLAB, since such tools run on a single machine, they are unable to handle large data. Also, while distributed tensor decomposition tools based on Hadoop can handle a scalable tensor, its computing speed is too slow. In this paper, we propose S-PARAFAC, which is a tensor decomposition tool based on Apache Spark, in distributed in-memory environments. We converted the PARAFAC algorithm into an Apache Spark version that enables rapid processing of tensor data. We also compared the performance of the Hadoop based tensor tool and S-PARAFAC. The result showed that S-PARAFAC is approximately 4~25 times faster than the Hadoop based tensor tool.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2015.01a
• /
• pp.1-3
• /
• 2015

본 논문에서는 하둡 환경상에서 개발 중인 공간 빅데이터 저장 관리 시스템의 구조를 설명한다. 본 시스템은 공간 센서 및 IoT의 등장으로 대용량화된 공간 데이터로 인한 기존 공간 정보 처리 시스템의 성능적 한계를 극복하기 위한 목적으로 개발 중이다. 본 시스템은 효과적인 대용량 데이터 처리를 위해 현재 활발히 연구되고 있는 빅데이터 처리 기술과 공간 정보 처리 기술을 접목하여, 대용량의 공간 정보를 수집, 저장 관리하는 기능을 제공한다. 또한 효과적인 공간 데이터의 접근을 위해 스크립트 언어 기반의 공간 정보 처리 언어를 제공하고, SQL 형식의 선언적 공간 정보 질의 처리 기능도 제공하기 위해 개발 중에 있다.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2014.07a
• /
• pp.341-342
• /
• 2014

지난 10년간 데이터의 폭발적인 증가로 우리는 빅데이터 시대를 맞이하게 되었다. 특히, 최근 몇 년 사이 소셜 네트워크의 발전으로 인해 발생하는 데이터의 양이 증가하면서, 이를 처리하기 위한 시스템으로 하둡이 등장하였다. 이전에는 저장 및 처리할 수 없었던 대용량 데이터를 오픈소스인 하둡의 등장으로 누구나가 대용량 데이터를 처리할 수 있는 시스템을 운영할 수 있게 된 것이다. 대규모 처리 분석을 위한 소프트웨어 프레임워크인 하둡은 클라우드 컴퓨팅의 대표적인 기술로 널리 사용되고 있다. 하둡은 크게 데이터의 저장을 담당하는 HDFS(Hadoop Distribute File System)와 데이터를 처리하는 맵리듀스로 나뉜다. 본 논문에서는 기존의 MapReduce와 차세대 맵리듀스로 불리는 YARN을 비교 분석하고 맵리듀스의 용도와 효율적인 활용방안을 제시한다.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2014.11a
• /
• pp.137-139
• /
• 2014

Objective: Data exchange, interoperability, and access as a service in healthcare information management systems (HIMS) is the basic need to provision health-services. Data existing in various HIMS not only differ in the basic underlying structure but also in data processing systems. Data interoperability can only be achieved when following a common structure or standard which is shareable such as semantics based structures. We propose web-enabler: A Hadoop MapReduce based distributed approach to transform the existing huge variety data in variety formats to a conformed and flexible ontological format that enables easy access to data, sharing, and providing various healthcare services. Results: For proof of concept, we present a case study of general patient record in conventional system to be enabled for analysis on the web by transforming to semantics based structure. Conclusion: This work achieves transformation of stale as well as future data to be web-enabled and easily available for analytics in healthcare systems.

• Proceedings of the Korean Information Science Society Conference
• /
• 2012.06c
• /
• pp.10-12
• /
• 2012

클라우드를 기반으로 한 대용량 데이터의 처리 및 분석의 요구가 커지면서, 대용량 관계형 데이터에 대한 분산 처리의 수요 또한 증가하고 있다. 본 논문은 HDFS를 사용하는 관계형 저장 시스템에서 대용량 데이터를 효율적으로 처리하기 위해 개발한 BST 기반 색인에 대해 설명한다.