• Journal of Convergence for Information Technology
• /
• v.10 no.10
• /
• pp.1-8
• /
• 2020

The need for storage, management, and retrieval techniques for alternative data has emerged as technologies based on data generated from business activities conducted by enterprises have emerged as the key to business success in recent years. Existing big data platform systems must load a large amount of data generated in real time without delay to process unstructured data, which is an alternative data, and efficiently manage storage space by utilizing a deduplication system of different storages when redundant data occurs. In this paper, we propose a multi-layer distributed data deduplication process system using the similarity of the Cuckoo hashing filter technique considering the characteristics of big data. Similarity between virtual machines is applied as Cuckoo hash, individual storage nodes can improve performance with deduplication efficiency, and multi-layer Cuckoo filter is applied to reduce processing time. Experimental results show that the proposed method shortens the processing time by 8.9% and increases the deduplication rate by 10.3%.

• Journal of Korea Multimedia Society
• /
• v.15 no.12
• /
• pp.1499-1505
• /
• 2012

Recently, the distributed processing system for big data has been actively investigated owing to the development of high speed network and storage technologies. In addition, virtual system that can provide efficient use of system resources through the consolidation of servers has been increasingly recognized. But, when we configure distributed processing system for big data in virtual machine environments, many problems occur. In this paper, we did an experiment on the optimization of I/O bandwidth according to the creation and placement of VMs and tasks with composing Hadoop cluster in virtual environments and evaluated the results of an experiment. These results conducted by this paper will be used in the study on the development of Hadoop Scheduler supporting I/O bandwidth balancing in virtual environments.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2016.10a
• /
• pp.50-53
• /
• 2016

Image processing techniques play an increasingly important role in many aspects of our daily life. For example, it has been shown to improve agricultural productivity in a number of ways such as plant pest detecting or fruit grading. However, massive quantities of images generated in real-time through multi-devices such as remote sensors during monitoring plant growth lead to the challenges of big data. Meanwhile, most current image processing systems are designed for small-scale and local computation, and they do not scale well to handle big data problems with their large requirements for computational resources and storage. In this paper, we have proposed an IPABigData (Image Processing Algorithm BigData) platform which provides algorithms to support large-scale image processing in agriculture based on Hadoop framework. Hadoop provides a parallel computation model MapReduce and Hadoop distributed file system (HDFS) module. It can also handle parallel pipelines, which are frequently used in image processing. In our experiment, we show that our platform outperforms traditional system in a scenario of image segmentation.

• Journal of Computing Science and Engineering
• /
• v.10 no.1
• /
• pp.21-31
• /
• 2016

With the application and development of biomedical techniques such as next-generation sequencing, mass spectrometry, and medical imaging, the amount of biomedical data have been growing explosively. In terms of processing such data, we face the problems surrounding big data, highly intensive computation, and high dimensionality data. Fortunately, cloud computing represents significant advantages of resource allocation, data storage, computation, and sharing and offers a solution to solve big data problems of biomedical research. In order to improve the efficiency of resource management in cloud computing, this paper proposes a clustering method and adopts Radial Basis Function in order to compress comprehensive data sets found in biology and medicine in high quality, and stores these data with resource management in cloud computing. Experiments have validated that with such a data-compression-based resource management in cloud computing, one can store large data sets from biology and medicine in fewer capacities. Furthermore, with reverse operation of the Radial Basis Function, these compressed data can be reconstructed with high accuracy.

• The Journal of the Korea institute of electronic communication sciences
• /
• v.8 no.11
• /
• pp.1719-1724
• /
• 2013

Big data which requires a different approach from existing data processing methods, is unstructured data with a variety of features. The features mean the volume of data, the rate of change of the data, the data with a variety of features. Tweets of twitter in only Korea are more than 5 millions per day. So much cheaper data storage and analysis system due to the increasing demand for information, the value of research is increasing. In this paper, the technology required by the deformation characteristics of the data elements as a technology priority-based word-based recommendation algorithm is proposed.

• Korean Journal of Air-Conditioning and Refrigeration Engineering
• /
• v.17 no.12
• /
• pp.1204-1209
• /
• 2005

In-situ measurement was made to evaluate chiller performance and thermal storage density of an ice thermal storage system. The system belonged to a big hotel and the measurement was conducted during late October. Owing to very small cooling load, the data logging was possible for a single thermal storage cycle. However, operation history of the chiller showed a relatively good spectrum of data for performance evaluation. COP and thermal storage density were calculated. The COP at full load was about 4.07, which was lower than $4.8\~6.4$ of new chillers. The measured storage density was about $10.9RT-h/m^3\;(=152MJ/m^3)$, which also was lower than a criterion of normal performance $(above\;13.0RT-h/m^3\;or\;181MJ/m^3)$. The study result provides technical basis for quantitative ESCO business scenario.

• Journal of Digital Contents Society
• /
• v.17 no.5
• /
• pp.409-416
• /
• 2016

Recently, Big Data issue has become a buzzword and universities, industries and research institutes have been efforts to collect, analyze various data enabled. These things includes accumulated data from the past, even if it is not possible to analysis at this present immediately a which has the potential means. And we are obtained a valuable result from the collected a large amount of data via the semantic analysis. The demand for high-performance storage system that can handle large amounts of data required is increasing around the world. In addition, it must provide a distributed parallel file system that stability to multiple users too perform a variety of analyzes at the same time by connecting a large amount of the accumulated data In this study, we identify the I/O bandwidth of the storage system to be considered, and performance of the metadata in order to provide a file system in stability and propose a method for configuring the optimal environment.

• Journal of Digital Convergence
• /
• v.15 no.4
• /
• pp.13-25
• /
• 2017

Volume in data storage is growing more than ever before. This phenomenon is caused by the participation of governments and firms as well as general users. As for big data, governments and public agencies are likely to play important roles in applications since they can access and operate personal data for public purposes. In this study, the author examined the status and countermeasure of big data from different countries and drew some common grounds. The suggestions are as follows. First of all, securing manpower and technology have to take precedence. In addition, share and development between the government and the private sector are required. And organizations should come up with long-term strategies along with the development of data loading and analysis. In conclusion, the author propose the recognition of the importance of data management, privacy protection and the expansion of field application possibilities for political usage of big data.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.19 no.3
• /
• pp.669-678
• /
• 2018

This study developed information technology infrastructures for building a driving environment analysis platform using various big data, such as vehicle sensing data, public data, etc. First, a small platform server with a parallel structure for big data distribution processing was developed with H/W technology. Next, programs for big data collection/storage, processing/analysis, and information visualization were developed with S/W technology. The collection S/W was developed as a collection interface using Kafka, Flume, and Sqoop. The storage S/W was developed to be divided into a Hadoop distributed file system and Cassandra DB according to the utilization of data. Processing S/W was developed for spatial unit matching and time interval interpolation/aggregation of the collected data by applying the grid index method. An analysis S/W was developed as an analytical tool based on the Zeppelin notebook for the application and evaluation of a development algorithm. Finally, Information Visualization S/W was developed as a Web GIS engine program for providing various driving environment information and visualization. As a result of the performance evaluation, the number of executors, the optimal memory capacity, and number of cores for the development server were derived, and the computation performance was superior to that of the other cloud computing.

• The Journal of the Korea Contents Association
• /
• v.17 no.3
• /
• pp.100-114
• /
• 2017

Most of the researchers establish research directions to conduct the study of new fields by getting advice from experts or through the papers of experts. The existing academic data search services provide paper information by field but do not provide experts by field. Therefore, users should decide experts by field using the searched papers by themselves. In this paper, we design and implement an expert search system by discipline through big data processing based on papers that have been published in the academic societies. The proposed system utilizes distributed big data storage systems to store and manage large papers. We also discriminate experts and analyze data related to the experts by using distributed big data processing technologies. The processed results are provided through web pages when a user searches for experts. The user can get a lot of helps for the research of a particular field since the proposed system recommends the experts of the corresponding research field.