• Title/Summary/Keyword: 빅데이터플랫폼

Search Result 483, Processing Time 0.027 seconds

Scaling of Hadoop Cluster for Cost-Effective Processing of MapReduce Applications (비용 효율적 맵리듀스 처리를 위한 클러스터 규모 설정)

  • Ryu, Woo-Seok
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.15 no.1
    • /
    • pp.107-114
    • /
    • 2020
  • This paper studies a method for estimating the scale of a Hadoop cluster to process big data as a cost-effective manner. In the case of medical institutions, demands for cloud-based big data analysis are increasing as medical records can be stored outside the hospital. This paper first analyze the Amazon EMR framework, which is one of the popular cloud-based big data framework. Then, this paper presents a efficiency model for scaling the Hadoop cluster to execute a Mapreduce application more cost-effectively. This paper also analyzes the factors that influence the execution of the Mapreduce application by performing several experiments under various conditions. The cost efficiency of the analysis of the big data can be increased by setting the scale of cluster with the most efficient processing time compared to the operational cost.

Evolution of ICT Ecosystem and Mobile Telcos' Counterstrategies (ICT 생태계 변화에 따른 국내 이동통신 사업자의 대응 전략에 대한 연구)

  • Kim, Dong Ju;Kang, Mincheol
    • Journal of Information Technology and Architecture
    • /
    • v.10 no.2
    • /
    • pp.197-209
    • /
    • 2013
  • This study analyzes the nature of consumers and smart phones as well as its limitations that domestic mobile communication companies confront. According to the analysis results, emerging technologies such as 5G communication, pervasive computing, augmented reality, and big data seem to have significant effect on the ICT ecosystem in the near future. Based on the results, this study suggests four counterstrategies for domestic mobile communication companies: big data strategy, preparation of things acting as a main communication agent, new service platform development, and 'total life care service provider' strategy.

Personal Recommendation Service Design Through Big Data Analysis on Science Technology Information Service Platform (과학기술정보 서비스 플랫폼에서의 빅데이터 분석을 통한 개인화 추천서비스 설계)

  • Kim, Dou-Gyun
    • Journal of the Korean BIBLIA Society for library and Information Science
    • /
    • v.28 no.4
    • /
    • pp.501-518
    • /
    • 2017
  • Reducing the time it takes for researchers to acquire knowledge and introduce them into research activities can be regarded as an indispensable factor in improving the productivity of research. The purpose of this research is to cluster the information usage patterns of KOSEN users and to suggest optimization method of personalized recommendation service algorithm for grouped users. Based on user research activities and usage information, after identifying appropriate services and contents, we applied a Spark based big data analysis technology to derive a personal recommendation algorithm. Individual recommendation algorithms can save time to search for user information and can help to find appropriate information.

Design of the Intelligent LBS Service : Using Big Data Distributed Processing System (빅데이터 분산처리 시스템을 활용한 지능형 LBS서비스의 설계)

  • Mun, Chang-Bae;Park, Hyun-Seok
    • The Journal of the Korea Contents Association
    • /
    • v.19 no.2
    • /
    • pp.159-169
    • /
    • 2019
  • Today, the location based service(LBS) is globally developing with the advance of smart phones and IOT devices. The main purpose of this research is to provide users with the most efficient route information, analyzing big data of people with a variety of routes. This system will enable users to have a similar feeling of getting a direct guidance from a person who has often used the route. It is possible because the system server analyzes the route information of people in real time, after composing the distributed processing system on the basis of map information. In the future, the system will be able to amazingly develop with the association of various LBS services, providing users with more precise and safer route information.

SDN based IoT Platform (SDN 기반 IoT 플랫폼)

  • Youn, Byungseong;Cho, Hyungho;Choi, Kyuhwi;Kim, Jigun
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2015.04a
    • /
    • pp.207-209
    • /
    • 2015
  • SDN은 기존 네트워크 장비의 하드웨어와 제어 부분을 분리하는 새로운 네트워크 기술로 중앙 집중 형태를 통해 효율적인 네트워크 관계가 가능하다. IoT 네트워크는 주변의 수많은 장치들을 상호 연결하는 환경으로, 각 장비의 연결과 통신, 발생하는 데이터의 빠른 처리가 요구된다. 본 논문에서는 IoT 네트워크에서 발생하는 빅데이터를 실시간으로 분석 및 처리하고, 최적화된 SDN 컨트롤러를 이용해 IoT 환경에서 효율적인 데이터 처리 및 네트워크 관제가 가능한 SDN 기반 IoT 플랫폼을 제안한다.

Cross Media-Platform Book Recommender System: Based on Book and Movie Ratings (사용자 영화취향을 반영한 크로스미디어 플랫폼 도서 추천 시스템)

  • Kim, Seongseop;Han, Sunwoo;Mok, Ha-Eun;Choi, Hyebong
    • The Journal of the Convergence on Culture Technology
    • /
    • v.7 no.1
    • /
    • pp.582-587
    • /
    • 2021
  • Book recommender system, which suggests book to users according to their book taste and preference effectively improves users' book-reading experience and exposes them to variety of books. Insufficient dataset of book rating records by users degrades the quality of recommendation. In this study, we suggest a book recommendation system that makes use of user's book ratings collaboratively with user's movie ratings where more abundant datasets are available. Through comprehensive experiment, we prove that our methods improve the recommendation quality and effectively recommends more diverse kind of books. In addition, this will be the first attempt for book recommendation system to utilize movie rating data, which is from the media-platform other than books.

Design and Implementation of HDFS data encryption scheme using ARIA algorithms on Hadoop (하둡 상에서 ARIA 알고리즘을 이용한 HDFS 데이터 암호화 기법의 설계 및 구현)

  • Song, Youngho;Shin, YoungSung;Yoon, Min;Jang, Miyoung;Chang, Jae-Woo
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2015.10a
    • /
    • pp.613-616
    • /
    • 2015
  • 최근 스마트폰 기기의 보급 및 소셜 서비스 산업의 고도화로 인해, 빅데이터가 등장하였다. 한편 빅데이터에서 효율적으로 정보를 분석하는 대표적인 플랫폼으로 하둡이 존재한다. 하둡은 클러스터 환경에 기반한 우수한 확장성, 장애 복구 기능 및 사용자가 기능을 정의할 수 있는 맵리듀스 프레임워크 등을 지원한다. 아울러 하둡은 개인정보나 위치 데이터 등의 민감한 정보를 보호하기 위해 Kerberos를 통한 사용자 인증 기법을 제공하고, HDFS 압축 코덱을 활용한 AES 코덱 기반 데이터 암호화를 지원하고 있다. 그러나 하둡 기반 소프트웨어를 사용하고 있는 국내 기관 및 기업은 국내 ARIA 데이터 암호화를 적용하지 못하고 있다. 이를 해결하기 위해 본 논문에서는 하둡을 기반으로 ARIA 암호화를 지원하는 HDFS 데이터 암호화 기법을 제안한다.

A Design of Similar Video Recommendation System using Extracted Words in Big Data Cluster (빅데이터 클러스터에서의 추출된 형태소를 이용한 유사 동영상 추천 시스템 설계)

  • Lee, Hyun-Sup;Kim, Jindeog
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.24 no.2
    • /
    • pp.172-178
    • /
    • 2020
  • In order to recommend contents, the company generally uses collaborative filtering that takes into account both user preferences and video (item) similarities. Such services are primarily intended to facilitate user convenience by leveraging personal preferences such as user search keywords and viewing time. It will also be ranked around the keywords specified in the video. However, there is a limit to analyzing video similarities using limited keywords. In such cases, the problem becomes serious if the specified keyword does not properly reflect the item. In this paper, I would like to propose a system that identifies the characteristics of a video as it is by the system without human intervention, and analyzes and recommends similarities between videos. The proposed system analyzes similarities by taking into account all words (keywords) that have different meanings from training videos, and in such cases, the methods handled by big data clusters are applied because of the large scale of data and operations.

An elastic distributed parallel Hadoop system for bigdata platform and distributed inference engines (동적 분산병렬 하둡시스템 및 분산추론기에 응용한 서버가상화 빅데이터 플랫폼)

  • Song, Dong Ho;Shin, Ji Ae;In, Yean Jin;Lee, Wan Gon;Lee, Kang Se
    • Journal of the Korean Data and Information Science Society
    • /
    • v.26 no.5
    • /
    • pp.1129-1139
    • /
    • 2015
  • Inference process generates additional triples from knowledge represented in RDF triples of semantic web technology. Tens of million of triples as an initial big data and the additionally inferred triples become a knowledge base for applications such as QA(question&answer) system. The inference engine requires more computing resources to process the triples generated while inferencing. The additional computing resources supplied by underlying resource pool in cloud computing can shorten the execution time. This paper addresses an algorithm to allocate the number of computing nodes "elastically" at runtime on Hadoop, depending on the size of knowledge data fed. The model proposed in this paper is composed of the layered architecture: the top layer for applications, the middle layer for distributed parallel inference engine to process the triples, and lower layer for elastic Hadoop and server visualization. System algorithms and test data are analyzed and discussed in this paper. The model hast the benefit that rich legacy Hadoop applications can be run faster on this system without any modification.

SDN에서 데이터 경로 설계에 대한 고려 사항

  • Yun, Hyeon-Sik;Gang, Gyeong-Sun;Kim, Hak-Seo;Park, Hye-Suk
    • Information and Communications Magazine
    • /
    • v.30 no.3
    • /
    • pp.22-28
    • /
    • 2013
  • 오늘날 미래인터넷 기술의 하나로서 SDN이 클라우드 서비스, 모바일 서비스, 스마트 TV, 빅데이터 및 사물지능통신 등의 새로운 서비스를 제공할 수 있는 개방형 플랫폼으로 주목받고 있다. 본고에서는 이러한 SDN 망을 구축함에 있어서 제어 인터페이스, QoS, 멀티캐스트, 이동성 및 네트워크 보안 측면에서 고려해야 할 사항들을 살펴본다. 기존 네트워크의 근본적인 문제점으로 인해 발생한 복잡성 및 성능 저하 요인들이 SDN의 장점을 활용하여 해결될 수 있으며, 이에 본고에서 제시된 고려 사항들이 큰 역할을 할 것이다.