• Title/Summary/Keyword: Distributed Processing environment

Search Result 591, Processing Time 0.034 seconds

A Study on a Distributed Data Fabric-based Platform in a Multi-Cloud Environment

  • Moon, Seok-Jae;Kang, Seong-Beom;Park, Byung-Joon
    • International Journal of Advanced Culture Technology
    • /
    • v.9 no.3
    • /
    • pp.321-326
    • /
    • 2021
  • In a multi-cloud environment, it is necessary to minimize physical movement for efficient interoperability of distributed source data without building a data warehouse or data lake. And there is a need for a data platform that can easily access data anywhere in a multi-cloud environment. In this paper, we propose a new platform based on data fabric centered on a distributed platform suitable for cloud environments that overcomes the limitations of legacy systems. This platform applies the knowledge graph database technique to the physical linkage of source data for interoperability of distributed data. And by integrating all data into one scalable platform in a multi-cloud environment, it uses the holochain technique so that companies can easily access and move data with security and authority guaranteed regardless of where the data is stored. The knowledge graph database mitigates the problem of heterogeneous conflicts of data interoperability in a decentralized environment, and Holochain accelerates the memory and security processing process on traditional blockchains. In this way, data access and sharing of more distributed data interoperability becomes flexible, and metadata matching flexibility is effectively handled.

Distributed Data Processing for Bigdata Analysis in War Game Simulation Environment (워게임 시뮬레이션 환경에 맞는 빅데이터 분석을 위한 분산처리기술)

  • Bae, Minsu
    • The Journal of Bigdata
    • /
    • v.4 no.2
    • /
    • pp.73-83
    • /
    • 2019
  • Since the emergence of the fourth industrial revolution, data analysis is being conducted in various fields. Distributed data processing has already become essential for the fast processing of large amounts of data. However, in the defense sector, simulation used cannot fully utilize the unstructured data which are prevailing at real environments. In this study, we propose a distributed data processing platform that can be applied to battalion level simulation models to provide visualized data for command decisions during training. 500,000 data points of strategic game were analyzed. Considering the winning factors in the data, variance processing was conducted to analyze the data for the top 10% teams. With the increase in the number of nodes, the model becomes scalable.

  • PDF

Job Classifying method based on Data Traits for Increased Efficiency of Computational Resources in Distributed Environment (분산 환경에서 계산 자원의 효율 증대를 위한 데이터 특성 기반의 작업 분류방법)

  • Moon, Sung-Hwan;Kim, Jae-Kwon;Kim, Tae-Young;Choi, Jeong-Seok;Cho, Kyu-Cheol;Lee, Jong-Sik
    • Journal of the Korea Society for Simulation
    • /
    • v.23 no.4
    • /
    • pp.219-228
    • /
    • 2014
  • Various computational resources in distributed environment are to build a high-performance computing environments through virtualization technology. Recently, there is a growing need for a complicated process due to the improvement of the user-level application, which has led to demand for high-performance computing. The requested job from users is composed of data. And because of each data has own characteristics, the classifier may consider the features of data. In this paper, we propose Job Classifying method based on Data Traits for Increased Efficiency of Computational Resources in Distributed Environment (JCDT). JCDT classifies the job by data traits of the users' request, is expected to improve the job processing time and increase the processing speed of the calculation resources.

Change Impact Analysis for Object-Oriented softwares in the distributed environment (분산 객체지향 소프트웨어를 위한 수정 영향 분석)

  • Kim, Kyung-Hee;Park, Jai-Nyun;Yoon, Yong-Ik
    • The Transactions of the Korea Information Processing Society
    • /
    • v.6 no.5
    • /
    • pp.1280-1290
    • /
    • 1999
  • Applying the change impact analysis to the distributed environment is not straightforward since it is based on the centralized system environment. In this paper, we investigate the change impact analysis of object-oriented shoftwares in the distributed environment. We first, categories the types of changes common in object-oriented software into three sets: data, method, and class level changes. We, then, analyze the impact of each set of changes and represent it in the form of a DPDG. A DPDG is a graph showing relationship of object oriented softwares - with data elements. classes, design documents, servers - in the distributed environment. DPDG searches element of software to retest when the software is changed. Thus, DPDG saves effort of software to retest. In this paper, We propose the table of firewall for retest elements that be discovered by DPDG and implement the table of firewall. The table of firewall is used VIST that we design a software testing tool. The VIST utilizes the minimized firewall, then saves efforts and costs of retesting for distributed object-oriented software.

  • PDF

Rhipe Platform for Big Data Processing and Analysis (빅데이터 처리 및 분석을 위한 Rhipe 플랫폼)

  • Jung, Byung Ho;Shin, Ji Eun;Lim, Dong Hoon
    • The Korean Journal of Applied Statistics
    • /
    • v.27 no.7
    • /
    • pp.1171-1185
    • /
    • 2014
  • Rhipe that integrates R and Hadoop environment, made it possible to process and analyze massive amounts of data using a distributed processing environment. In this paper, we implemented multiple regression analysis using Rhipe with various data sizes of actual data and simulated data. Experimental results for comparing the computing speeds of pseudo-distributed and fully-distributed modes for configuring Hadoop cluster, showed fully-distributed mode was more fast than pseudo-distributed mode and computing speeds of fully-distributed mode were faster as the number of data nodes increases. We also compared the performance of our Rhipe with stats and biglm packages available on bigmemory. The results showed that our Rhipe was more fast than other packages owing to paralleling processing with increasing the number of map tasks as the size of data increases.

The Bigdata Processing Environment Building for the Learning System (학습 시스템을 위한 빅데이터 처리 환경 구축)

  • Kim, Young-Geun;Kim, Seung-Hyun;Jo, Min-Hui;Kim, Won-Jung
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.9 no.7
    • /
    • pp.791-797
    • /
    • 2014
  • In order to create an environment for Apache Hadoop for parallel distributed processing system of Bigdata, by connecting a plurality of computers, or to configure the node, using the configuration of the virtual nodes on a single computer it is necessary to build a cloud fading environment. However, be constructed in practice for education in these systems, there are many constraints in terms of cost and complex system configuration. Therefore, it is possible to be used as training for educational institutions and beginners in the field of Bigdata processing, development of learning systems and inexpensive practical is urgent. Based on the Raspberry Pi board, training and analysis of Big data processing, such as Hadoop and NoSQL is now the design and implementation of a learning system of parallel distributed processing of possible Bigdata in this study. It is expected that Bigdata parallel distributed processing system that has been implemented, and be a useful system for beginners who want to start a Bigdata and education.

DMRUT-MCDS: Discovery Relationships in the Cyber-Physical Integrated Network

  • Lu, Hongliang;Cao, Jiannong;Zhu, Weiping;Jiao, Xianlong;Lv, Shaohe;Wang, Xiaodong
    • Journal of Communications and Networks
    • /
    • v.17 no.6
    • /
    • pp.558-567
    • /
    • 2015
  • In recent years, we have seen a proliferation of mobile-network-enabled smart objects, such as smart-phones and smart-watches, that form a cyber-physical integrated network to connect the cyber and physical worlds through the capabilities of sensing, communicating, and computing. Discovery of the relationship between smart objects is a critical and nontrivial task in cyber-physical integrated network applications. Aiming to find the most stable relationship in the heterogeneous and dynamic cyber-physical network, we propose a distributed and efficient relationship-discovery algorithm, called dynamically maximizing remaining unchanged time with minimum connected dominant set (DMRUT-MCDS) for constructing a backbone with the smallest scale infrastructure. In our proposed algorithm, the impact of the duration of the relationship is considered in order to balance the size and sustain time of the infrastructure. The performance of our algorithm is studied through extensive simulations and the results show that DMRUT-MCDS performs well in different distribution networks.

A Job Allocation Manager for Dynamic Remote Execution of Distributed Jobs in P2P Network (분산처리 작업의 동적 원격실행을 위한 P2P 기반 작업 할당 관리자)

  • Lee, Seung-Ha;Kim, Yang-Woo
    • Journal of Internet Computing and Services
    • /
    • v.7 no.6
    • /
    • pp.87-103
    • /
    • 2006
  • Advances in computer and network technology provide new computing environment that were only possible with supercomputers before. In order to provide the environment, a distributed runtime system has to be provided, but most of the conventional distributed runtime systems lack in providing dynamic and flexible system reconfiguration depending on workload variance, due to a static architecture of fixed master node and slave working nodes. This paper proposes and implements a new model for distributed job allocation and management which is a distributed runtime system is P2P environment for flexible and dynamic system reconfiguration. The implemented systems enables job program transfer and management, remote compile and execution among cooperative developers based on P2P standard protocol Jxta platform. Since it makes dynamic and flexible system reconfiguration possible, the proposed method has some advantages in that it can collect and utilize idle computing resources immediately at a needed time for distributed job processing. Moreover, the implemented system's effectiveness and performance increase are shown by applying and processing the crawler jobs, in a distributed way, for collecting a large amount of data needed for internet search.

  • PDF

Distributed Computing Models for Wireless Sensor Networks (무선 센서 네트워크에서의 분산 컴퓨팅 모델)

  • Park, Chongmyung;Lee, Chungsan;Jo, Youngtae;Jung, Inbum
    • Journal of KIISE
    • /
    • v.41 no.11
    • /
    • pp.958-966
    • /
    • 2014
  • Wireless sensor networks offer a distributed processing environment. Many sensor nodes are deployed in fields that have limited resources such as computing power, network bandwidth, and electric power. The sensor nodes construct their own networks automatically, and the collected data are sent to the sink node. In these traditional wireless sensor networks, network congestion due to packet flooding through the networks shortens the network life time. Clustering or in-network technologies help reduce packet flooding in the networks. Many studies have been focused on saving energy in the sensor nodes because the limited available power leads to an important problem of extending the operation of sensor networks as long as possible. However, we focus on the execution time because clustering and local distributed processing already contribute to saving energy by local decision-making. In this paper, we present a cooperative processing model based on the processing timeline. Our processing model includes validation of the processing, prediction of the total execution time, and determination of the optimal number of processing nodes for distributed processing in wireless sensor networks. The experiments demonstrate the accuracy of the proposed model, and a case study shows that our model can be used for the distributed application.

Rule-Based Cooperation of Distributed EC Systems

  • Lee, Dong-Woo
    • International Journal of Contents
    • /
    • v.5 no.3
    • /
    • pp.79-85
    • /
    • 2009
  • Emergent requests or urgent information among enterprises require their intimate collaboration in B2B EC (electronic commerce). This paper analyzes the needs of intimate cooperation of distributed EC systems in terms of business contracts and presents an active rule-based methodology of close cooperation among EC systems and an active rule component to support it. Since the rule component provides high level rule patterns and event-based immediate processing, system administrators and programmers can easily program and maintain intimate cooperation of distributed EC systems independently to the application logic. The proposed active rule component facilitates HTTP protocol. Its prototype is implemented in B2B EC environment and evaluated using basic trigger facility of a commercial DBMS.