• Title/Summary/Keyword: MapReduce Online

Search Result 10, Processing Time 0.029 seconds

Dynamic Load Management Method for Spatial Data Stream Processing on MapReduce Online Frameworks (맵리듀스 온라인 프레임워크에서 공간 데이터 스트림 처리를 위한 동적 부하 관리 기법)

  • Jeong, Weonil
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.19 no.8
    • /
    • pp.535-544
    • /
    • 2018
  • As the spread of mobile devices equipped with various sensors and high-quality wireless network communications functionsexpands, the amount of spatio-temporal data generated from mobile devices in various service fields is rapidly increasing. In conventional research into processing a large amount of real-time spatio-temporal streams, it is very difficult to apply a Hadoop-based spatial big data system, designed to be a batch processing platform, to a real-time service for spatio-temporal data streams. This paper extends the MapReduce online framework to support real-time query processing for continuous-input, spatio-temporal data streams, and proposes a load management method to distribute overloads for efficient query processing. The proposed scheme shows a dynamic load balancing method for the nodes based on the inflow rate and the load factor of the input data based on the space partition. Experiments show that it is possible to support efficient query processing by distributing the spatial data stream in the corresponding area to the shared resources when load management in a specific area is required.

A Management method of Load Balancing among Game Servers based on Distributed Server System Using Map Balance Server (분산형 서버 구조 기반 Map 밸런스 서버를 이용한 게임 서버 간 부하 관리 방법)

  • Kim, Soon-Gohn;Lee, Nam-Jae;Yang, Seung-Weon
    • Journal of Advanced Navigation Technology
    • /
    • v.15 no.6
    • /
    • pp.1034-1041
    • /
    • 2011
  • Generally, In distributed online game server system, game maps are processed separately by means of dividing into several unit blocks. But the keeping normal distribution of user in game map is very difficult because preferences of game users are not same according to individual user's private character. For this reason, if the huge number of users concentrate on particular region of same game map at once, the game server exceed their threshold so that the system can be getting down. Conversely, the efficiency of system goes down considerably because the game server must perform map processing continuously even under user_empty situation. To solve this problem, in this paper, we propose a Map management method to control relatively normal distribution of users in game maps using Map Balance Server. In suggested model, we can reduce the load of game servers by means of turn off the game map processing temporary when a server is under user_empty situation. we also can maximize server performance by means of redistribution of map processing load among servers.

Framework of Online Shopping Service based on M2M and IoT for Handheld Devices in Cloud Computing (클라우드 컴퓨팅에서 Handheld Devices 기반의 M2M 및 IoT 온라인 쇼핑 서비스 프레임워크)

  • Alsaffar, Aymen Abdullah;Aazam, Mohammad;Park, Jun-Young;Huh, Eui-Nam
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2013.05a
    • /
    • pp.179-182
    • /
    • 2013
  • We develop Framework architecture of Online Shopping Services based on M2M and IoT for Handheld Devices in Cloud Computing. MapReduce model will be used as a method to simplify large scale data processing when user search for purchasing products online which provide efficient, and fast respond time. Therefore, providing user with a enhanced Quality of Experience (QoE) as well as Quality of Service (QoS) when purchasing/searching products Online from big data.

Sparse Feature Convolutional Neural Network with Cluster Max Extraction for Fast Object Classification

  • Kim, Sung Hee;Pae, Dong Sung;Kang, Tae-Koo;Kim, Dong W.;Lim, Myo Taeg
    • Journal of Electrical Engineering and Technology
    • /
    • v.13 no.6
    • /
    • pp.2468-2478
    • /
    • 2018
  • We propose the Sparse Feature Convolutional Neural Network (SFCNN) to reduce the volume of convolutional neural networks (CNNs). Despite the superior classification performance of CNNs, their enormous network volume requires high computational cost and long processing time, making real-time applications such as online-training difficult. We propose an advanced network that reduces the volume of conventional CNNs by producing a region-based sparse feature map. To produce the sparse feature map, two complementary region-based value extraction methods, cluster max extraction and local value extraction, are proposed. Cluster max is selected as the main function based on experimental results. To evaluate SFCNN, we conduct an experiment with two conventional CNNs. The network trains 59 times faster and tests 81 times faster than the VGG network, with a 1.2% loss of accuracy in multi-class classification using the Caltech101 dataset. In vehicle classification using the GTI Vehicle Image Database, the network trains 88 times faster and tests 94 times faster than the conventional CNNs, with a 0.1% loss of accuracy.

The Analysis Framework for User Behavior Model using Massive Transaction Log Data (대규모 로그를 사용한 유저 행동모델 분석 방법론)

  • Lee, Jongseo;Kim, Songkuk
    • The Journal of Bigdata
    • /
    • v.1 no.2
    • /
    • pp.1-8
    • /
    • 2016
  • User activity log includes lots of hidden information, however it is not structured and too massive to process data, so there are lots of parts uncovered yet. Especially, it includes time series data. We can reveal lots of parts using it. But we cannot use log data directly to analyze users' behaviors. In order to analyze user activity model, it needs transformation process through extra framework. Due to these things, we need to figure out user activity model analysis framework first and access to data. In this paper, we suggest a novel framework model in order to analyze user activity model effectively. This model includes MapReduce process for analyzing massive data quickly in the distributed environment and data architecture design for analyzing user activity model. Also we explained data model in detail based on real online service log design. Through this process, we describe which analysis model is fit for specific data model. It raises understanding of processing massive log and designing analysis model.

  • PDF

A Service Platform Development on the GIS-based Analysis and Management of the Fire-fighting Vulnerable Areas (GIS기반 소방취약지 분석관리 서비스 플랫폼 개발)

  • Song, Wanyoung;Cho, Kwanghyun;Cho, Myeongheum;Kim, Seonggon;Kim, Sungjae
    • Journal of the Society of Disaster Information
    • /
    • v.11 no.2
    • /
    • pp.269-278
    • /
    • 2015
  • Developed a service platform technology for automated that analyze the fire-fighting vulnerable area and equipped a map online. As a result, it is possible to provide the information necessary for the vulnerable area of the fire-fighting activity online. If progress on the study of the utilized service, it is possible to reduce the Golden Time of the fire fighting field. In this study, Confirmed the technical viability satisfactory to the management and service expansion improvements to fire-fighting vulnerable area.

Reliability-aware service chaining mapping in NFV-enabled networks

  • Liu, Yicen;Lu, Yu;Qiao, Wenxin;Chen, Xingkai
    • ETRI Journal
    • /
    • v.41 no.2
    • /
    • pp.207-223
    • /
    • 2019
  • Network function virtualization can significantly improve the flexibility and effectiveness of network appliances via a mapping process called service function chaining. However, the failure of any single virtualized network function causes the breakdown of the entire chain, which results in resource wastage, delays, and significant data loss. Redundancy can be used to protect network appliances; however, when failures occur, it may significantly degrade network efficiency. In addition, it is difficult to efficiently map the primary and backups to optimize the management cost and service reliability without violating the capacity, delay, and reliability constraints, which is referred to as the reliability-aware service chaining mapping problem. In this paper, a mixed integer linear programming formulation is provided to address this problem along with a novel online algorithm that adopts the joint protection redundancy model and novel backup selection scheme. The results show that the proposed algorithm can significantly improve the request acceptance ratio and reduce the consumption of physical resources compared to existing backup algorithms.

E-voting Implementation in Egypt

  • Eraky, Ahmed
    • Journal of Contemporary Eastern Asia
    • /
    • v.16 no.1
    • /
    • pp.48-68
    • /
    • 2017
  • Manual elections processes in Egypt have several negative effects; that mainly leads to political corruption due to the lack of transparency. These issues negatively influence citizen's participation in the political life; while electronic voting systems aim to increase efficiency, transparency, and reduce the cost comparing to the manual voting. The main research objectives are, finding the successful factors that positively affects E-voting implementation in Egypt, in addition of finding out the reasons that keep Egyptian government far from applying E-voting, and to come up with the road map that Egyptian government has to take into consideration to successfully implement E-voting systems. The findings of the study suggest that there are seven independent variables affecting e-voting implementation which are; leadership, government willingness, legal framework, technical quality, awareness, citizen's trust in government and IT literacy. Technology-Organization-Environment (TOE) theory was used to provide an analytical framework for the study. A quantitative approach (i.e., survey questionnaire) strategy was used to collect data. A random sampling method was used to select the participants for the survey, whom are targeted voters in Egypt and have access to the internet, since the questionnaire was distributed online and the data is analyzed using regression analysis. Practical implications of this study will lead for more citizen participation in the political life due to the transparency that E-voting system will create, in addition to reduce the political corruption.

Big IoT Healthcare Data Analytics Framework Based on Fog and Cloud Computing

  • Alshammari, Hamoud;El-Ghany, Sameh Abd;Shehab, Abdulaziz
    • Journal of Information Processing Systems
    • /
    • v.16 no.6
    • /
    • pp.1238-1249
    • /
    • 2020
  • Throughout the world, aging populations and doctor shortages have helped drive the increasing demand for smart healthcare systems. Recently, these systems have benefited from the evolution of the Internet of Things (IoT), big data, and machine learning. However, these advances result in the generation of large amounts of data, making healthcare data analysis a major issue. These data have a number of complex properties such as high-dimensionality, irregularity, and sparsity, which makes efficient processing difficult to implement. These challenges are met by big data analytics. In this paper, we propose an innovative analytic framework for big healthcare data that are collected either from IoT wearable devices or from archived patient medical images. The proposed method would efficiently address the data heterogeneity problem using middleware between heterogeneous data sources and MapReduce Hadoop clusters. Furthermore, the proposed framework enables the use of both fog computing and cloud platforms to handle the problems faced through online and offline data processing, data storage, and data classification. Additionally, it guarantees robust and secure knowledge of patient medical data.

Investigating the Performance of Bayesian-based Feature Selection and Classification Approach to Social Media Sentiment Analysis (소셜미디어 감성분석을 위한 베이지안 속성 선택과 분류에 대한 연구)

  • Chang Min Kang;Kyun Sun Eo;Kun Chang Lee
    • Information Systems Review
    • /
    • v.24 no.1
    • /
    • pp.1-19
    • /
    • 2022
  • Social media-based communication has become crucial part of our personal and official lives. Therefore, it is no surprise that social media sentiment analysis has emerged an important way of detecting potential customers' sentiment trends for all kinds of companies. However, social media sentiment analysis suffers from huge number of sentiment features obtained in the process of conducting the sentiment analysis. In this sense, this study proposes a novel method by using Bayesian Network. In this model MBFS (Markov Blanket-based Feature Selection) is used to reduce the number of sentiment features. To show the validity of our proposed model, we utilized online review data from Yelp, a famous social media about restaurant, bars, beauty salons evaluation and recommendation. We used a number of benchmarking feature selection methods like correlation-based feature selection, information gain, and gain ratio. A number of machine learning classifiers were also used for our validation tasks, like TAN, NBN, Sons & Spouses BN (Bayesian Network), Augmented Markov Blanket. Furthermore, we conducted Bayesian Network-based what-if analysis to see how the knowledge map between target node and related explanatory nodes could yield meaningful glimpse into what is going on in sentiments underlying the target dataset.