• Title/Summary/Keyword: distributed memory environment

Search Result 80, Processing Time 0.021 seconds

Design and Cost Analysis for a Fault-Tolerant Distributed Shared Memory System

  • Jazi, AL-Harbi Fahad;kim, Kangseok;Kim, Jai-Hoon
    • Journal of Internet Computing and Services
    • /
    • v.17 no.4
    • /
    • pp.1-9
    • /
    • 2016
  • Algorithms implementing distributed shared memory (DSM) were developed for ensuring consistency. The performance of DSM algorithms is dependent on system and usage parameters. However, ensuring these algorithms to tolerate faults is a problem that needs to be researched. In this study, we proposed fault-tolerant scheme for DSM system and analyzed reliability and fault-tolerant overhead. Using our analysis, we can choose a proper algorithm for DSM on error prone environment.

An Efficient Cache Management Scheme for Load Balancing in Distributed Environments with Different Memory Sizes (상이한 메모리 크기를 가지는 분산 환경에서 부하 분산을 위한 캐시 관리 기법)

  • Choi, Kitae;Yoon, Sangwon;Park, Jaeyeol;Lim, Jongtae;Lee, Seokhee;Bok, Kyoungsoo;Yoo, Jaesoo
    • KIISE Transactions on Computing Practices
    • /
    • v.21 no.8
    • /
    • pp.543-548
    • /
    • 2015
  • Recently, volume of data has been growing dramatically along with the growth of social media and digital devices. However, the existing disk-based distributed file systems have limits to their performance of data processing or data access, due to I/O processing costs and bottlenecks. To solve this problem, the caching technique is being used to manage data in the memory. In this paper, we propose a cache management scheme to handle load balancing in a distributed memory environment. The proposed scheme distributes the data according to the memory size, n distributed environments with different memory sizes. If overloaded nodes occur, it redistributes the the access time of the caching data. In order to show the superiority of the proposed scheme, we compare it with an existing distributed cache management scheme through performance evaluation.

Distributed In-Memory Caching Method for ML Workload in Kubernetes (쿠버네티스에서 ML 워크로드를 위한 분산 인-메모리 캐싱 방법)

  • Dong-Hyeon Youn;Seokil Song
    • Journal of Platform Technology
    • /
    • v.11 no.4
    • /
    • pp.71-79
    • /
    • 2023
  • In this paper, we analyze the characteristics of machine learning workloads and, based on them, propose a distributed in-memory caching technique to improve the performance of machine learning workloads. The core of machine learning workload is model training, and model training is a computationally intensive task. Performing machine learning workloads in a Kubernetes-based cloud environment in which the computing framework and storage are separated can effectively allocate resources, but delays can occur because IO must be performed through network communication. In this paper, we propose a distributed in-memory caching technique to improve the performance of machine learning workloads performed in such an environment. In particular, we propose a new method of precaching data required for machine learning workloads into the distributed in-memory cache by considering Kubflow pipelines, a Kubernetes-based machine learning pipeline management tool.

  • PDF

A Distributed Control Architecture for Advanced Testing In Realtime

  • Thoen Bradford K.;Laplace Patrick N.
    • Proceedings of the Earthquake Engineering Society of Korea Conference
    • /
    • 2006.03a
    • /
    • pp.563-570
    • /
    • 2006
  • Distributed control architecture is based on sharing control and data between multiple nodes on a network Communication and task sharing can be distributed between multiple control computers. Although many communication protocols exist, such as TCP/IP and UDP, they do not have the determinism that realtime control demands. Fiber-optic reflective shared memory creates the opportunity for realtime distributed control. This architecture allows control and computational tasks to be divided between multiple systems and operate in a deterministic realtime environment. One such shared memory architecture is based on Curtiss-Wright ScramNET family of fiber-optic reflective memory. MTS has built seismic and structural control software and hardware capable of utilizing ScramNET shared memory, opening up infinite possibilities in research and new capabilities in Hybrid and Model-In-The-Loop control.

  • PDF

Data Replication and Migration Scheme for Load Balancing in Distributed Memory Environments (분산 인-메모리 환경에서 부하 분산을 위한 데이터 복제와 이주 기법)

  • Choi, Kitae;Yoon, Sangwon;Park, Jaeyeol;Lim, Jongtae;Bok, Kyoungsoo;Yoo, Jaesoo
    • KIISE Transactions on Computing Practices
    • /
    • v.22 no.1
    • /
    • pp.44-49
    • /
    • 2016
  • Recently, data has been growing dramatically along with the growth of social media and digital devices. A distributed memory processing system has been used to efficiently process large amounts of data. However, if a load is concentrated in a certain node in distributed environments, a node performance significantly degrades. In this paper, we propose a load balancing scheme to distribute load in a distributed memory environment. The proposed scheme replicates hot data to multiple nodes for managing a node's load and migrates the data by considering the load of the nodes when nodes are added or removed. The client reduces the number of accesses to the central server by directly accessing the data node through the metadata information of the hot data. In order to show the superiority of the proposed scheme, we compare it with the existing load balancing scheme through performance evaluation.

Comparative Analysis of Centralized Vs. Distributed Locality-based Repository over IoT-Enabled Big Data in Smart Grid Environment

  • Siddiqui, Isma Farah;Abbas, Asad;Lee, Scott Uk-Jin
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2017.01a
    • /
    • pp.75-78
    • /
    • 2017
  • This paper compares operational and network analysis of centralized and distributed repository for big data solutions in the IoT enabled Smart Grid environment. The comparative analysis clearly depicts that centralize repository consumes less memory consumption while distributed locality-based repository reduce network complexity issues than centralize repository in state-of-the-art Big Data Solution.

  • PDF

Distributed Incremental Approximate Frequent Itemset Mining Using MapReduce

  • Mohsin Shaikh;Irfan Ali Tunio;Syed Muhammad Shehram Shah;Fareesa Khan Sohu;Abdul Aziz;Ahmad Ali
    • International Journal of Computer Science & Network Security
    • /
    • v.23 no.5
    • /
    • pp.207-211
    • /
    • 2023
  • Traditional methods for datamining typically assume that the data is small, centralized, memory resident and static. But this assumption is no longer acceptable, because datasets are growing very fast hence becoming huge from time to time. There is fast growing need to manage data with efficient mining algorithms. In such a scenario it is inevitable to carry out data mining in a distributed environment and Frequent Itemset Mining (FIM) is no exception. Thus, the need of an efficient incremental mining algorithm arises. We propose the Distributed Incremental Approximate Frequent Itemset Mining (DIAFIM) which is an incremental FIM algorithm and works on the distributed parallel MapReduce environment. The key contribution of this research is devising an incremental mining algorithm that works on the distributed parallel MapReduce environment.

Parallel Computing Environment for R with on Supercomputer Systems (빅데이터 분석을 위한 슈퍼컴퓨터 환경에서 R의 병렬처리)

  • Lee, Sang Yeol;Won, Joong Ho
    • Journal of the Korean Operations Research and Management Science Society
    • /
    • v.39 no.4
    • /
    • pp.19-31
    • /
    • 2014
  • We study parallel processing techniques for the R programming language of high performance computing technology. In this study, we used massively parallel computing system which has 25,408 cpu cores. We conducted a performance evaluation of a distributed memory system using MPI and of a the shared memory system using OpenMP. Our findings are summarized as follows. First, For some particular algorithms, parallel processing is about 150 times faster than serial processing in R. Second, the distributed memory system gets faster as the number of nodes increases while shared memory system is limited in the improvement of performance, due to the limit of the number of cpus in a single system.

A Study on a Distributed Data Fabric-based Platform in a Multi-Cloud Environment

  • Moon, Seok-Jae;Kang, Seong-Beom;Park, Byung-Joon
    • International Journal of Advanced Culture Technology
    • /
    • v.9 no.3
    • /
    • pp.321-326
    • /
    • 2021
  • In a multi-cloud environment, it is necessary to minimize physical movement for efficient interoperability of distributed source data without building a data warehouse or data lake. And there is a need for a data platform that can easily access data anywhere in a multi-cloud environment. In this paper, we propose a new platform based on data fabric centered on a distributed platform suitable for cloud environments that overcomes the limitations of legacy systems. This platform applies the knowledge graph database technique to the physical linkage of source data for interoperability of distributed data. And by integrating all data into one scalable platform in a multi-cloud environment, it uses the holochain technique so that companies can easily access and move data with security and authority guaranteed regardless of where the data is stored. The knowledge graph database mitigates the problem of heterogeneous conflicts of data interoperability in a decentralized environment, and Holochain accelerates the memory and security processing process on traditional blockchains. In this way, data access and sharing of more distributed data interoperability becomes flexible, and metadata matching flexibility is effectively handled.

Web-Based Organizational Memory Acquisition by Using a Fuzzy Cognitive Map (퍼지인식도를 이용한 웹기반 조직지식획득에 관한 연구)

  • 이건창
    • Journal of Intelligence and Information Systems
    • /
    • v.5 no.2
    • /
    • pp.79-97
    • /
    • 1999
  • Knowledge management (KM) is emerging as a robust management mechanism with which an organization can remain highly intelligent and competitive in a turbulent market. Organization knowledge is at the heart of KM success. As a vehicle of acquiring organizational knowledge in a distributed decision-making environment, we applied a fuzzy cognitive map (FMM) technique and proved its effectiveness in a distributed knowledge management environment. Our approach was applied to the financial statement analysis problem, yielding a robust result.

  • PDF