• Title/Summary/Keyword: MapReduce

Search Result 848, Processing Time 0.037 seconds

Turbo MAP Decoding Algorithm based on Radix-4 Method (Radix-4 방식의 터보 MAP 복호 알고리즘)

  • 정지원;성진숙;김명섭;오덕길;고성찬
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.25 no.4A
    • /
    • pp.546-552
    • /
    • 2000
  • The decoding of Turbo-Code relies on the application of a soft input/soft output decoders which can be realized using maximum-a-posteriori(MAP) symbol estimator[l]. Radix-2 MAP decoder can not be used for high speed communications because of a large number of interleaver block size N. This paper proposed a new simple method for radix-4 MAP decoder based on radix-2 MAP decoder in order to reduce the interleave block size. A branch metrics, forward and backward recursive functions are proposed for applying to radix-4 MAP structure with symbol interleaver. Radix-4 MAP decoder shall be illustratively described and its error performance capability shall be compared to conventional radix-2 MAP decoder in AWGN channel.

  • PDF

An Algorithm Application Study on the Updating Digital Maps (수치지도 수시갱신을 위한 알고리즘 적용 연구)

  • Park, Chung;Lee, Ho-Nam;Park, Ki-Suk;Oh, Se-Jung
    • Proceedings of the Korean Society of Surveying, Geodesy, Photogrammetry, and Cartography Conference
    • /
    • 2009.04a
    • /
    • pp.135-143
    • /
    • 2009
  • The purpose of this research is to reduce editors' workload and automation to update(v) Digital Map(n) regarding changed topography. In Digital Map case, it usually uses the way of various survey to update(v). In this case, it costs a great deal and doesn't be efficient backup management. Accordingly, it can reduce editors' the period of the process and be efficiently managed the backup data due to well organized backup data management. We present update plan that used clipping and join for this result by analyzing Boundary of input data.

  • PDF

A Study on the effect of the number of Key to MapRedue performance (Key개수가 MapReduce 성능에 미치는 영향에 관한 연구)

  • Jeong, Seok-Jun;Kim, Jin-Hong;Shin, Dong-Ryeol
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2016.01a
    • /
    • pp.207-209
    • /
    • 2016
  • 정보통신기술의 급속한 발전으로 인해 인터넷은 사회 전 분야를 변화시키고 있고 이를 통해 데이터의 양이 증가하면서 의료, 교육, 경영 등 사회 전 분야에서 빅데이터에 관심이 증가하고 있다. 이에 따라 다양한 빅데이터 오픈소스가 생기고 데이터의 크기에 따라 성능을 비교하는 실험이 진행되었다. 본 논문에서는 데이터의 크기가 아니라 데이터를 분류하는 key의 개수에 따라 성능을 비교하고자 한다.

  • PDF

A Study on the Improving Performance of Massively Small File Using the Reuse JVM in MapReduce (MapReduce에서 Reuse JVM을 이용한 대규모 스몰파일 처리성능 향상 방법에 관한 연구)

  • Choi, Chul Woong;Kim, Jeong In;Kim, Pan Koo
    • Journal of Korea Multimedia Society
    • /
    • v.18 no.9
    • /
    • pp.1098-1104
    • /
    • 2015
  • With the widespread use of smartphones and IoT (Internet of Things), data are being generated on a large scale, and there is increased for the analysis of such data. Hence, distributed processing systems have gained much attention. Hadoop, which is a distributed processing system, saves the metadata of stored files in name nodes; in this case, the main problems are as follows: the memory becomes insufficient; load occurs because of massive small files; scheduling and file processing time increases because of the increased number of small files. In this paper, we propose a solution to address the increase in processing time because of massive small files, and thus improve the processing performance, using the Reuse JVM method provided by Hadoop. Through environment setting, the Reuse JVM method modifies the JVM produced conventionally for every task, so that multiple tasks are reused sequentially in one JVM. As a final outcome, the Reuse JVM method showed the best processing performance when used together with CombineFileInputFormat.

A Parallel HDFS and MapReduce Functions for Emotion Analysis (감성분석을 위한 병렬적 HDFS와 맵리듀스 함수)

  • Back, BongHyun;Ryoo, Yun-Kyoo
    • Journal of the Korea society of information convergence
    • /
    • v.7 no.2
    • /
    • pp.49-57
    • /
    • 2014
  • Recently, opinion mining is introduced to extract useful information from SNS data and to evaluate the true intention of users. Opinion mining are required several efficient techniques to collect and analyze a large amount of SNS data and extract meaningful data from them. Therefore in this paper, we propose a parallel HDFS(Hadoop Distributed File System) and emotion functions based on Mapreduce to extract some emotional information of users from various unstructured big data on social networks. The experiment results have verified that the proposed system and functions perform faster than O(n) for data gathering time and loading time, and maintain stable load balancing for memory and CPU resources.

  • PDF

A MapReduce based Algorithm for Spatial Aggregation of Microblog Data in Spatial Social Analytics (공간 소셜 분석을 위한 마이크로블로그 데이터의 맵리듀스 기반 공간 집계 알고리즘)

  • Cho, Hyun Gu;Yang, Pyoung Woo;Yoo, Ki Hyun;Nam, Kwang Woo
    • Journal of KIISE
    • /
    • v.42 no.6
    • /
    • pp.781-790
    • /
    • 2015
  • In recent times, microblogs have become popular owing to the development of the Internet and mobile environments. Among the various types of microblog data, those containing location data are referred to as spatial social Web objects. General aggregations of such microblog data include data aggregation per user for a single piece of information. This study proposes a spatial aggregation algorithm that combines a general aggregation with spatial data and uses the Geohash and MapReduce operations to perform spatial social analysis, by using microblog data with the characteristics of a spatial social Web object. The proposed algorithm provides the foundation for a meaningful spatial social analysis.

Optimization Driven MapReduce Framework for Indexing and Retrieval of Big Data

  • Abdalla, Hemn Barzan;Ahmed, Awder Mohammed;Al Sibahee, Mustafa A.
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.14 no.5
    • /
    • pp.1886-1908
    • /
    • 2020
  • With the technical advances, the amount of big data is increasing day-by-day such that the traditional software tools face a burden in handling them. Additionally, the presence of the imbalance data in big data is a massive concern to the research industry. In order to assure the effective management of big data and to deal with the imbalanced data, this paper proposes a new indexing algorithm for retrieving big data in the MapReduce framework. In mappers, the data clustering is done based on the Sparse Fuzzy-c-means (Sparse FCM) algorithm. The reducer combines the clusters generated by the mapper and again performs data clustering with the Sparse FCM algorithm. The two-level query matching is performed for determining the requested data. The first level query matching is performed for determining the cluster, and the second level query matching is done for accessing the requested data. The ranking of data is performed using the proposed Monarch chaotic whale optimization algorithm (M-CWOA), which is designed by combining Monarch butterfly optimization (MBO) [22] and chaotic whale optimization algorithm (CWOA) [21]. Here, the Parametric Enabled-Similarity Measure (PESM) is adapted for matching the similarities between two datasets. The proposed M-CWOA outperformed other methods with maximal precision of 0.9237, recall of 0.9371, F1-score of 0.9223, respectively.

Design of a Sentiment Analysis System to Prevent School Violence and Student's Suicide (학교폭력과 자살사고를 예방하기 위한 감성분석 시스템의 설계)

  • Kim, YoungTaek
    • The Journal of Korean Association of Computer Education
    • /
    • v.17 no.6
    • /
    • pp.115-122
    • /
    • 2014
  • One of the problems with current youth generations is increasing rate of violence and suicide in their school lives, and this study aims at the design of a sentiment analysis system to prevent suicide by uising big data process. The main issues of the design are economical implementation, easy and fast processing for the users, so, the open source Hadoop system with MapReduce algorithm is used on the HDFS(Hadoop Distributed File System) for the experimentation. This study uses word count method to do the sentiment analysis with informal data on some sns communications concerning a kinds of violent words, in terms of text mining to avoid some expensive and complex statistical analysis methods.

  • PDF

Effects of Hypervisor on Distributed Big Data Processing in Virtualizated Cluster Environment (가상화 클러스터 환경에서 빅 데이터 분산 처리 성능에 하이퍼바이저가 미치는 영향)

  • Chung, Haejin;Nah, Yunmook
    • KIISE Transactions on Computing Practices
    • /
    • v.22 no.2
    • /
    • pp.89-94
    • /
    • 2016
  • Recently, cluster computing environments have been in a process of change toward virtualized cluster environments. The change of the cluster environment has great impact on the performance of large volume distributed processing. Therefore, many domestic and international IT companies have invested heavily in research on cluster environments. In this paper, we show how the hypervisor affects the performance of distributed processing of a large volume of data. We present a performance comparison of MapReduce processing in two virtualized cluster environments, one built using the Xen hypervisor and the other built using the container-based Docker. Our results show that Docker is faster than Xen.

Standardizing Unstructured Big Data and Visual Interpretation using MapReduce and Correspondence Analysis (맵리듀스와 대응분석을 활용한 비정형 빅 데이터의 정형화와 시각적 해석)

  • Choi, Joseph;Choi, Yong-Seok
    • The Korean Journal of Applied Statistics
    • /
    • v.27 no.2
    • /
    • pp.169-183
    • /
    • 2014
  • Massive and various types of data recorded everywhere are called big data. Therefore, it is important to analyze big data and to nd valuable information. Besides, to standardize unstructured big data is important for the application of statistical methods. In this paper, we will show how to standardize unstructured big data using MapReduce which is a distribution processing system. We also apply simple correspondence analysis and multiple correspondence analysis to nd the relationship and characteristic of direct relationship words for Samsung Electronics and The Korea Economic Daily newspaper as well as Apple Inc.