• Title/Summary/Keyword: Heterogeneous distributed storage

Search Result 19, Processing Time 0.031 seconds

Optimal Heterogeneous Distributed Storage Regenerating Code at Minimum Remote-Repair Bandwidth Regenerating Point

  • Xu, Jian;Cao, Yewen;Wang, Deqiang;Wu, Changlei;Yang, Guang
    • ETRI Journal
    • /
    • v.38 no.3
    • /
    • pp.529-539
    • /
    • 2016
  • Recently, a product-matrix (PM) framework was proposed to construct optimal regenerating codes for homogeneous distributed storage systems (DSSs). In this paper, we propose an extended PM (EPM) framework for coding of heterogeneous DSSs having different repair bandwidths but identical storage capacities. Based on the EPM framework, an explicit construction of minimum remote-repair bandwidth regenerating (MRBR) codes is presented for a specific heterogeneous DSS, where two geographically different datacenters with associated storage nodes are deployed. The data reconstruction and regeneration properties of the MRBR code are proved strictly. For the purpose of demonstration, an example implementation of MRBR code is provided. The presented MRBR code is the first optimal strict-regenerating code for heterogeneous DSSs. In addition, our proposed EPM framework can be applied to homogeneous systems also.

Adaptive-and-Resolvable Fractional Repetition Codes Based on Hypergraph

  • Tiantian Wang;Jing Wang;Haipeng Wang;Jie Meng;Chunlei Yu;Shuxia Wang
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.17 no.4
    • /
    • pp.1182-1199
    • /
    • 2023
  • Fractional repetition (FR) codes can achieve exact uncoded repair for multiple failed nodes, with lower computational complexity and bandwidth overhead, and effectively improve repair performance in distributed storage systems (DSS). The actual distributed storage system is dynamic, that is, the parameters such as node storage overhead and number of storage nodes will change randomly and dynamically. Considering that traditional FR codes cannot be flexibly applied to dynamic distributed storage systems, a new construction scheme of adaptive-and-resolvable FR codes based on hypergraph coloring is proposed in this paper. Specifically, the linear uniform regular hypergraph can be constructed based on the heuristic algorithm of hypergraph coloring proposed in this paper. Then edges and vertices in hypergraph correspond to nodes and coded packets of FR codes respectively, further, FR codes is constructed. According to hypergraph coloring, the FR codes can achieve rapid repair for multiple failed nodes. Further, FR codes based on hypergraph coloring can be generalized to heterogeneous distributed storage systems. Compared with Reed-Solomon (RS) codes, simple regenerating codes (SRC) and locally repairable codes (LRC), adaptive-and-resolvable FR codes have significant advantages over repair locality, repair bandwidth overhead, computational complexity and time overhead during repairing failed nodes.

Energy and Performance-Efficient Dynamic Load Distribution for Mobile Heterogeneous Storage Devices (에너지 및 성능 효율적인 이종 모바일 저장 장치용 동적 부하 분산)

  • Kim, Young-Jin;Kim, Ji-Hong
    • Journal of the Korea Society of Computer and Information
    • /
    • v.14 no.4
    • /
    • pp.9-17
    • /
    • 2009
  • In this paper, we propose a dynamic load distribution technique at the operating system level in mobile storage systems with a heterogeneous storage pair of a small form-factor and disk and a flash memory, which aims at saving energy consumption as well as enhancing I/O performance. Our proposed technique takes a combinatory approach of file placement and buffer cache management techniques to find how the load can be distributed in an energy and performance-aware way for a heterogeneous mobile storage air of a hard disk and a flash memory. We demonstrate that the proposed technique provides better experimental results with heterogeneous mobile storage devices compared with the existing techniques through extensive simulations.

Issues in Next Generation Streaming Server Design

  • Won, Youjip
    • Proceedings of the Korean Society of Broadcast Engineers Conference
    • /
    • 2001.11a
    • /
    • pp.335-354
    • /
    • 2001
  • .Next Generation Multimedia Streaming Technology Massive Scale Support $\rightarrow$ Clustered Solution Adaptive to Heterogeneous Network daptive to Heterogeneous Terminal Capability Presentation Technique .SMART Server Architecture .HERMES File System .Clustered Solution . High Speed Storage Interconnect .' Content Partitioning . Load Management . Support for Heterogeniety . Adaptive End to End Streaming Transport: Unicast vs. Multicast '. Scalable Encoding

  • PDF

RDP: A storage-tier-aware Robust Data Placement strategy for Hadoop in a Cloud-based Heterogeneous Environment

  • Muhammad Faseeh Qureshi, Nawab;Shin, Dong Ryeol
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.10 no.9
    • /
    • pp.4063-4086
    • /
    • 2016
  • Cloud computing is a robust technology, which facilitate to resolve many parallel distributed computing issues in the modern Big Data environment. Hadoop is an ecosystem, which process large data-sets in distributed computing environment. The HDFS is a filesystem of Hadoop, which process data blocks to the cluster nodes. The data block placement has become a bottleneck to overall performance in a Hadoop cluster. The current placement policy assumes that, all Datanodes have equal computing capacity to process data blocks. This computing capacity includes availability of same storage media and same processing performances of a node. As a result, Hadoop cluster performance gets effected with unbalanced workloads, inefficient storage-tier, network traffic congestion and HDFS integrity issues. This paper proposes a storage-tier-aware Robust Data Placement (RDP) scheme, which systematically resolves unbalanced workloads, reduces network congestion to an optimal state, utilizes storage-tier in a useful manner and minimizes the HDFS integrity issues. The experimental results show that the proposed approach reduced unbalanced workload issue to 72%. Moreover, the presented approach resolve storage-tier compatibility problem to 81% by predicting storage for block jobs and improved overall data block placement by 78% through pre-calculated computing capacity allocations and execution of map files over respective Namenode and Datanodes.

A Study on the Design and Implementation of the Lightweight Object Model Supporting Distributed Trader (분산 트레이더를 지원하는 경량 (lightweight) 객체 모델 설계 및 구현 방안 연구)

  • Jin, Myeong-Suk;Song, Byeong-Gwon
    • The Transactions of the Korea Information Processing Society
    • /
    • v.7 no.4
    • /
    • pp.1050-1061
    • /
    • 2000
  • This paper presents a new object model, LOM(Lightweight Object Model) and an implementation method for the distributed trader in heterogeneous distributed computing environment including mobile network. Trader is third party object that enables clients to find suitable servers, which provide the most appropriate services to client in distributed environment including dynamic reconfiguration of services and servers. Trading service requires simpler and more specific object model than genetic object models which provide richer multimedia data types and semantic characteristics with complex data structures. LOM supports a new reference attribute type instead of the relationship, inheritance and composite attribute types of the general object oriented models and so LOM has simple data structures. Also in LOM, the modelling step includes specifying of the information about users and the access right to objects for security in the mobile environment and development of the distributed storage for trading service. Also, we propose and implementation method of the distributed trader, which integrates the LOM-information object model and the OMG (object Management Group) computational object model.

  • PDF

Robust and Auditable Secure Data Access Control in Clouds

  • KARPAGADEEPA.S;VIJAYAKUMAR.P
    • International Journal of Computer Science & Network Security
    • /
    • v.24 no.5
    • /
    • pp.95-102
    • /
    • 2024
  • In distributed computing, accessible encryption strategy over Auditable data is a hot research field. Be that as it may, most existing system on encoded look and auditable over outsourced cloud information and disregard customized seek goal. Distributed storage space get to manage is imperative for the security of given information, where information security is executed just for the encoded content. It is a smaller amount secure in light of the fact that the Intruder has been endeavored to separate the scrambled records or Information. To determine this issue we have actualize (CBC) figure piece fastening. It is tied in with adding XOR each plaintext piece to the figure content square that was already delivered. We propose a novel heterogeneous structure to evaluate the issue of single-point execution bottleneck and give a more proficient access control plot with a reviewing component. In the interim, in our plan, a CA (Central Authority) is acquainted with create mystery keys for authenticity confirmed clients. Not at all like other multi specialist get to control plots, each of the experts in our plan deals with the entire trait set independently. Keywords: Cloud storage, Access control, Auditing, CBC.

Asymmetric data storage management scheme to ensure the safety of big data in multi-cloud environments based on deep learning (딥러닝 기반의 다중 클라우드 환경에서 빅 데이터의 안전성을 보장하기 위한 비대칭 데이터 저장 관리 기법)

  • Jeong, Yoon-Su
    • Journal of Digital Convergence
    • /
    • v.19 no.3
    • /
    • pp.211-216
    • /
    • 2021
  • Information from various heterogeneous devices is steadily increasing in distributed cloud environments. This is because high-speed network speeds and high-capacity multimedia data are being used. However, research is still underway on how to minimize information errors in big data sent and received by heterogeneous devices. In this paper, we propose a deep learning-based asymmetric storage management technique for minimizing bandwidth and data errors in networks generated by information sent and received in cloud environments. The proposed technique applies deep learning techniques to optimize the load balance after asymmetric hash of the big data information generated by each device. The proposed technique is characterized by allowing errors in big data collected from each device, while also ensuring the connectivity of big data by grouping big data into groups of clusters of dogs. In particular, the proposed technique minimizes information errors when storing and managing big data asymmetrically because it used a loss function that extracted similar values between big data as seeds.

Study on the Job Execution Time of Mobile Cloud Computing (모바일 클라우드 컴퓨팅의 작업 실행 시간에 대한 연구)

  • Jung, Sung Min;Kim, Tae Kyung
    • Journal of Korea Society of Digital Industry and Information Management
    • /
    • v.8 no.1
    • /
    • pp.99-105
    • /
    • 2012
  • Given the numbers of smartphones, tablets and other mobile devices shipped every day, more and more users are relying on the cloud as the main driver for satisfying their computing needs, whether it is data storage, applications or infrastructure. Mobile cloud computing is simply cloud computing in which at least some of the devices involved are mobile. Each node is owned by a different user and is likely to be mobile. Using mobile hardware for cloud computing has advantages over using traditional hardware. These advantage include computational access to multimedia and sensor data without the need for large network transfer, more efficient access to data stored on other mobile devices and distributed ownership and maintenance of hardware. It is important to predict job execution time in mobile cloud computing because there are many mobile nodes with different capabilities. This paper analyzes the job execution time for mobile cloud computing in terms of network environment and heterogeneous mobile nodes using a mathematical model.

An Efficient Method for Determining Work Process Number of Each Node on Computation Grid (계산 그리드 상에서 각 노드의 작업 프로세스 수를 결정하기 위한 효율적인 방법)

  • Kim Young-Hak;Cho Soo-Hyun
    • The Journal of the Korea Contents Association
    • /
    • v.5 no.1
    • /
    • pp.189-199
    • /
    • 2005
  • The grid computing is a technique to solve big problems such as a field of scientific technique by sharing the computing power and a big storage space of the numerous computers on the distributed network. The environment of the grid computing is composed with the WAN which has a different performance and a heterogeneous network condition. Therefore, it is more important to reflect heterogeneous performance elements to calculation work. In this paper, we propose an efficient method that decides work process number of each node by considering a network state information. The network state information considers the latency, the bandwidth and latency-bandwidth mixture information. First, using information which was measured, we compute the performance ratio and decide work process number of each node. Finally, RSL file was created automatically based on work process number which was decided, and then accomplishes a work. The network performance information is collected by the NWS. According to experimental results, the method which was considered of network performance information is improved respectively 23%, 31%, and 57%, compared to the methods of existing in a viewpoint of work amount, work process number, and node number.

  • PDF