• Title/Summary/Keyword: Distributed File Systems

Search Result 123, Processing Time 0.026 seconds

Implementation and Performance Measuring of Erasure Coding of Distributed File System (분산 파일시스템의 소거 코딩 구현 및 성능 비교)

  • Kim, Cheiyol;Kim, Youngchul;Kim, Dongoh;Kim, Hongyeon;Kim, Youngkyun;Seo, Daewha
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.41 no.11
    • /
    • pp.1515-1527
    • /
    • 2016
  • With the growth of big data, machine learning, and cloud computing, the importance of storage that can store large amounts of unstructured data is growing recently. So the commodity hardware based distributed file systems such as MAHA-FS, GlusterFS, and Ceph file system have received a lot of attention because of their scale-out and low-cost property. For the data fault tolerance, most of these file systems uses replication in the beginning. But as storage size is growing to tens or hundreds of petabytes, the low space efficiency of the replication has been considered as a problem. This paper applied erasure coding data fault tolerance policy to MAHA-FS for high space efficiency and introduces VDelta technique to solve data consistency problem. In this paper, we compares the performance of two file systems, MAHA-FS and GlusterFS. They have different IO processing architecture, the former is server centric and the latter is client centric architecture. We found the erasure coding performance of MAHA-FS is better than GlusterFS.

Design and implementation of a Shared-Concurrent File System in distributed UNIX environment (분산 UNIX 환경에서 Shared-Concurrent File System의 설계 및 구현)

  • Jang, Si-Ung;Jeong, Gi-Dong
    • The Transactions of the Korea Information Processing Society
    • /
    • v.3 no.3
    • /
    • pp.617-630
    • /
    • 1996
  • In this paper, a shared-concurrent file system (S-CFS) is designed and implemented using conventional disks as disk arrays on a Workstation Cluster which can be used as a small-scale server. Since it is implemented on UNIX operating systems, S_CFS is not only portable and flexible but also efficient in resource usage because it does not require additional I/O nodes. The result of the research shows that on small-scale systems with enough disks, the performance of the concurrent file system on transaction processing applications is bounded by the bottleneck of CPUs computing powers while the performance of the concurrent file system on massive data I/Os is bounded by the time required to copy data between buffers. The concurrent file system,which has been implemented on a Workstation Cluster with 8 disks,shows a throughput of 388 tps in case of transaction processing applications and can provide the bandwidth of 15.8 Mbytes/sec in case of massive data processing applications. Moreover,the concurrent file system has been dsigned to enhance the throughput of applications requirring high performance I/O by controlling the paralleism of the concurrent file system on user's side.

  • PDF

A Design and its Implementation of Java based Management System for Distributed File Systems (Distributed file System을 위한 Java 기반 Management System설계 및 구현)

  • 김태형;정규식
    • Proceedings of the IEEK Conference
    • /
    • 2002.06c
    • /
    • pp.149-152
    • /
    • 2002
  • 최근 인터넷 사용자가 늘어나면서 클러스터링 기반의 대형 서버가 등장하게 되었고 또한 분산파일시스템(DFS)에 대한 관심이 커지게 되었다. 기존 분산파일 시스템들은 파일시스템 측면의 많은 기능들이 제공되고 뛰어난 성능을 발휘하지만 사용자 인터페이스 및 관리 측면에서 본다면 미흡한 점이 많다. 본 논문에서는 분산파일 시스템의 종류와 플랫폼에 상관없이, 사용하기 쉬운 DFS Management System을 설계하고 구축한다. 전체 시스템 구조를 파악 할 수 있도록 GUI 환경을 제공하며, 모듈화/계층화 구조로 설계하는데 기존 DFS와 연동하기 위한 하부 통신모듈이 포함된다. 본 논문에서는 Java기반으로 DFS Management System을 구축하였으며, Coda DFS가 설치된 클러스터링 서버와 연동하여 시험하였다.

  • PDF

Trickle Write-Back Scheme for Cache Management in Mobile Computing Environments (?이동 컴퓨팅 환경에서 캐쉬 관리를 위한 TWB 기법)

  • Kim, Moon-Jeong;Eom, Young-Ik
    • Journal of KIISE:Computer Systems and Theory
    • /
    • v.27 no.1
    • /
    • pp.89-100
    • /
    • 2000
  • Recently, studies on the mobile computing environments that enable mobile hosts to move while retaining its network connection are in progress. In these mobile computing environments, one of the necessary components is the distributed file system supporting mobile hosts, and there are several issues for the design and implementation of the shared file system. Among these issues, there are problems caused by network traffic on limited bandwidth of wireless media. Also, there are consistency maintenance issues that are caused by update-conflicts on the shared files in the distributed file system. In this paper, we propose TWB(Trickle Write-Back) scheme that utilizes weak connectivity for cache management of mobile clients. This scheme focuses on saving bandwidth, reducing waste of disk space, and reducing risks caused by disconnection. For such goals, this scheme lets mobile clients write back intermediate states periodically or on demand while delaying unnecessary write-backs. Meanwhile, this scheme is based on the existing distributed file system architecture and provides transparency.

  • PDF

A study on a collective behavior of interacting simple robots

  • Sugawara, Ken;Sano, Masaki;Watanabe, Toshinori
    • 제어로봇시스템학회:학술대회논문집
    • /
    • 2002.10a
    • /
    • pp.93.6-93
    • /
    • 2002
  • Many livings form groups that we consider as collective systems. Their collective behaviors are good model for the development of useful distributed systems. In this paper, We discussed the group of motile elements that is described by simple model. The dynamics of each element is described by simple kinematics, but the group shows various types of motions. In addition, we found out the formation of the group changes by modifying rc which is an optimum distance between each element. This modification shows us to observe close-packed structure, face-centered lattice, sim pie lattice and double file.

  • PDF

Reliable & Sealable Multicast Communication in Real Time Collaborative Systems

  • Patel, Jayesh-M;Shamsul Sahibuddin
    • Proceedings of the IEEK Conference
    • /
    • 2002.07c
    • /
    • pp.1752-1755
    • /
    • 2002
  • The world wide web (WWW) already accounts f3r more Internee network traffic than any other application, including il and simple file transfer. It is also a collaborative technology in a weak sense of the word - it allows people to share information. Synchronous collaboration is where an interactive activity is simultaneous and in teal-time. Computer based real time collaborative systems like shared whiteboards. collaborative editor etc. are only beginning to emerge recently. These applications invoking more than two users exchanging information, require Multicast communication. Multicast communication is a transmission mode that is now supported by a variety of local and wide area networks. Multicasting enables multiparty communication across a wide area to sparsely distributed groups by minimizing the network load. Multicasting itself is one of the key technologies in the nut generation of the Internet This paper describes the technical issues from the aspect of multicast communication and its reliability in synchronous collaborative application.

  • PDF

A study on high availability of the linux clustering web server (리눅스 클러스터링 웹 서버의 고가용성에 대한 연구)

  • 박지현;이상문;홍태화;김학배
    • 제어로봇시스템학회:학술대회논문집
    • /
    • 2000.10a
    • /
    • pp.88-88
    • /
    • 2000
  • As more and more critical commercial applications move on the Internet, providing highly available servers becomes increasingly important. One of the advantages of a clustered system is that it has hardware and software redundancy. High availability can be provided by detecting node or daemon failure and reconfiguring the system appropriately so that the workload can be taken over bi the remaining nodes in the cluster. This paper presents how to provide the guaranteeing high availability of clustering web server. The load balancer becomes a single failure point of the whole system. In order to prevent the failure of the load balancer, we setup a backup server using heartbeat, fake, mon, and checkpointing fault-tolerance method. For high availability of file servers in the cluster, we setup coda file system. Coda is a advanced network fault-tolerance distributed file system.

  • PDF

A Content-based Load Balancing Algorithm for Metadata Servers in Cluster File System (클러스터 파일 시스템의 메타데이터 서버를 위한 내용 기반 부하 분산 알고리즘)

  • Jang Jun-Ho;Han Sae-Young;Park Sung-Yong
    • The KIPS Transactions:PartA
    • /
    • v.13A no.4 s.101
    • /
    • pp.323-334
    • /
    • 2006
  • A metadata service is one of the important factors to affect the performance of cluster file systems. We propose a content-based load balancing algorithm that dynamically distributes client requests to appropriate metadata servers based on the types of metadata operations. By replicating metadatas and logging update messages in each server, rather than moving metadatas across servers, we significantly reduced the response time and evenly distributed client's requests among metadata servers.

Efficient Cooperative Caching Algorithm for Distributed File Systems (분산 파일시스템을 위한 효율적인 협력캐쉬 알고리즘)

  • 박새미;이석재;유재수
    • Proceedings of the Korea Contents Association Conference
    • /
    • 2003.11a
    • /
    • pp.234-244
    • /
    • 2003
  • In distributed file-systems, cooperative caching algorithm which owns the data cached at each node jointly is used to reduce an expense of disk access. Cooperative caching algorithm is the method that increases a cache hit-ratio and decrease a disk access as it holds the cache information of distributed systems in common and makes cache larger virtually. Recently, several cooperative caching algorithms decrease the message costs by using approximate information of the cache and increase the cache hit-ratio by using local and global cache fields dynamically. And they have an advantage that increases the whole field hit-ratio by sending a replaced block to the idel node on cache replacement in order to maintain the replaced block in the cache field. However the wrong approximate information deteriorates the performance, the concistency maintenance goes to great expense to exchange messeges and the cost that manages Age-information of each node to choose the idle node increases. In this thesis, we propose a cooperative cache algorithm that maintains correct cache information, minimizes the maintance cost for consistency and the management cost for cache Age-information. Also, we show the superiority of our algorithm through the performance evaluation.

  • PDF

IT-based Technology An Efficient Global Buffer Management ,algorithm for SAN Environments (SAN 환경을 위한 효율적인 전역버퍼 관리 알고리즘)

  • 이석재;박새미;송석일;유재수;이장선
    • The Journal of the Korea Contents Association
    • /
    • v.4 no.3
    • /
    • pp.71-80
    • /
    • 2004
  • In distributed file-systems, cooperative caching algorithm which owns the data cached at each node jointly is used to reduce an expense of disk access. Cooperative caching algorithm is the method that increases a cache hit-ratio and decrease a disk access as it holds the cache information of distributed systems in common and makes cache larger virtually. Recently, several cooperative caching algorithms decrease the message costs by using approximate information of the cache and increase the cache hit-ratio by using local and global cache fields dynamically. And they have an advantage that increases the whole field hit-ratio by sending a replaced buffer to the idle node on buffers replacement in order to maintain the replaced cache in the cache field. However the wrong approximate information deteriorates the performance, the consistency maintenance goes to great expense to exchange messages and the cost that manages Age-information of each node to choose the idle node increases. In this thesis, we propose a cooperative cache algorithm that maintains correct cache information, minimizes the maintenance cost for consistency and the management cost for buffer Age-information. Also, we show the superiority of our algorithm through the performance evaluation.

  • PDF