• Title/Summary/Keyword: distributed file system

Search Result 251, Processing Time 0.025 seconds

Design and Implementation of a Metadata Structure for Large-Scale Shared-Disk File System (대용량 공유디스크 파일 시스템에 적합한 메타 데이타 구조의 설계 및 구현)

  • 이용주;김경배;신범주
    • Journal of KIISE:Computer Systems and Theory
    • /
    • v.30 no.1
    • /
    • pp.33-49
    • /
    • 2003
  • Recently, there have been large storage demands for manipulating multimedia data. To solve the tremendous storage demands, one of the major researches is the SAN(Storage Area Network) that provides the local file requests directly from shared-disk storage and also eliminates the server bottlenecks to performance and availability. SAN also improve the network latency and bandwidth through new channel interface like FC(Fibre Channel). But to manipulate the efficient storage network like SAN, traditional local file system and distributed file system are not adaptable and also are lack of researches in terms of a metadata structure for large-scale inode object such as file and directory. In this paper, we describe the architecture and design issues of our shared-disk file system and provide the efficient bitmap for providing the well-formed block allocation in each host, extent-based semi flat structure for storing large-scale file data, and two-phase directory structure of using Extendible Hashing. Also we describe a detailed algorithm for implementing the file system's device driver in Linux Kernel and compare our file system with the general file system like EXT2 and shard disk file system like GFS in terms of file creation, directory creation and I/O rate.

Improving Fault Tolerance for High-capacity Shared Distributed File Systems using the Rotational Lease Under Network Partitioning (대용량 공유 분산 화일 시스템에서 망 분할 시 순환 리스를 사용한 고장 감내성 향상)

  • Tak, Byung-Chul;Chung, Yon-Dohn;Kim, Myoung-Ho
    • Journal of KIISE:Databases
    • /
    • v.32 no.6
    • /
    • pp.616-627
    • /
    • 2005
  • In the shared storage file system, systems can directly access the shared storage device through specialized data-only subnetwork unlike in the network attached file server system. In this shared-storage architecture, data consistency is maintained by some designated set of lock servers which use control network to send and receive the lock information. Furthermore, lease mechanism is introduced to cope with the control network failure. But when the control network is partitioned, participating systems can no longer make progress after the lease term expires until the network recovers. This paper addresses this limitation and proposes a method that allows partitioned systems to make progress under the partition of control network. The proposed method works in a manner that each participating system is rotationally given a predefined lease term periodically. It is also shown that the proposed mechanism always preserves data consistency.

Implementation of a DB-Based Virtual File System for Lightweight IoT Clouds (경량 사물 인터넷 클라우드를 위한 DB 기반 가상 파일 시스템 구현)

  • Lee, Hyung-Bong;Kwon, Ki-Hyeon
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.3 no.10
    • /
    • pp.311-322
    • /
    • 2014
  • IoT(Internet of Things) is a concept of connected internet pursuing direct access to devices or sensors in fused environment of personal, industrial and public area. In IoT environment, it is possible to access realtime data, and the data format and topology of devices are diverse. Also, there are bidirectional communications between users and devices to control actuators in IoT. In this point, IoT is different from the conventional internet in which data are produced by human desktops and gathered in server systems by way of one-sided simple internet communications. For the cloud or portal service of IoT, there should be a file management framework supporting systematic naming service and unified data access interface encompassing the variety of IoT things. This paper implements a DB-based virtual file system maintaining attributes of IoT things in a UNIX-styled file system view. Users who logged in the virtual shell are able to explore IoT things by navigating the virtual file system, and able to access IoT things directly via UNIX-styled file I O APIs. The implemented virtual file system is lightweight and flexible because it maintains only directory structure and descriptors for the distributed IoT things. The result of a test for the virtual shell primitives such as mkdir() or chdir() shows the smooth functionality of the virtual file system, Also, the exploring performance of the file system is better than that of Window file system in case of adopting a simple directory cache mechanism.

Implement of MapReduce-based Big Data Processing Scheme for Reducing Big Data Processing Delay Time and Store Data (빅데이터 처리시간 감소와 저장 효율성이 향상을 위한 맵리듀스 기반 빅데이터 처리 기법 구현)

  • Lee, Hyeopgeon;Kim, Young-Woon;Kim, Ki-Young
    • Journal of the Korea Convergence Society
    • /
    • v.9 no.10
    • /
    • pp.13-19
    • /
    • 2018
  • MapReduce, the Hadoop's essential core technology, is most commonly used to process big data based on the Hadoop distributed file system. However, the existing MapReduce-based big data processing techniques have a feature of dividing and storing files in blocks predefined in the Hadoop distributed file system, thus wasting huge infrastructure resources. Therefore, in this paper, we propose an efficient MapReduce-based big data processing scheme. The proposed method enhances the storage efficiency of a big data infrastructure environment by converting and compressing the data to be processed into a data format in advance suitable for processing by MapReduce. In addition, the proposed method solves the problem of the data processing time delay arising from when implementing with focus on the storage efficiency.

Performance Analysis of Lustre File System using High Performance Storage Devices (고성능 스토리지를 이용한 Lustre 파일 시스템의 성능 분석 연구)

  • Lee, Jaehwan;Koo, Donghun;Park, Kyungmin;Kim, Jiksoo;Hwang, Soonwook
    • KIISE Transactions on Computing Practices
    • /
    • v.22 no.4
    • /
    • pp.163-169
    • /
    • 2016
  • Lustre is a scalable, distributed file system, which is popular in the field of high-performance computing. Recently, the advent of SSD has enabled high-performance storage hardware, but software development requires further improvement. In this paper, we analyzed performance of the Lustre system using SSD via extensive experimentation. We compared performance of Lustre on SSDs and HDDs in terms of file read/write throughputs and metadata access latencies. Our experimental results showed that 1) SSDs improve metadata access performance due to fast random read/write access of SSD characteristics, and 2) SSD are benefited to a greater extent under multiple threads and large numbers of small sized files.

A Study on the Command/File Server for INS (종합정보망(INS)을 위한 command/file server의 연구)

  • 이철수;이화연;황문준
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.12 no.5
    • /
    • pp.439-450
    • /
    • 1987
  • Data, distributed and/or duplicated among multiple computers interconnected by a LAN, need to be managed consistently and accessed transparently by users. A command/file server, named UTP, which meets the preceding needs was developed for supermini-computers which are interconnected by an Ethernet-based LAN and run under UNIX OS.

  • PDF

A Content-based Load Balancing Algorithm for Metadata Servers in Cluster File System (클러스터 파일 시스템의 메타데이터 서버를 위한 내용 기반 부하 분산 알고리즘)

  • Jang Jun-Ho;Han Sae-Young;Park Sung-Yong
    • The KIPS Transactions:PartA
    • /
    • v.13A no.4 s.101
    • /
    • pp.323-334
    • /
    • 2006
  • A metadata service is one of the important factors to affect the performance of cluster file systems. We propose a content-based load balancing algorithm that dynamically distributes client requests to appropriate metadata servers based on the types of metadata operations. By replicating metadatas and logging update messages in each server, rather than moving metadatas across servers, we significantly reduced the response time and evenly distributed client's requests among metadata servers.

Design and Implementation of a Blockchain System for Storing BIM Files in a Distributed Network Environment

  • Seo, Jungwon;Ko, Deokyoon;Park, Sooyong;Kim, Seong-jin;Kim, Bum-Soo;Kim, Do Young
    • Journal of the Korea Society of Computer and Information
    • /
    • v.26 no.12
    • /
    • pp.159-168
    • /
    • 2021
  • Building Information Modeling (BIM) data is a digitized construction design by worldwide construction design stands rules. Some research are being conducted to utilize blockchain for safe sharing and trade of BIM data, but there is no way to store BIM data directly in the blockchain due to the size of BIM data and technical limitation of the blockchain. In this paper, we propose a method of storing BIM data by combining a distributed file system and a blockchain. We propose two network overlays for storing BIM data, and we also propose generating the Level of Detail (LOD)-based merkle tree for efficient verification of BIM data. In addition, this paper proposes a system design for distributed storage of BIM data by using blockchain besu client and IPFS client. Our system design has a result that the processing speed stably increased despite the increase in data size.