• Title/Summary/Keyword: File Storage

Search Result 457, Processing Time 0.028 seconds

The development of the high effective and stoppageless file system for high performance computing (High Performance Computing 환경을 위한 고성능, 무정지 파일시스템 구현)

  • Park, Yeong-Bae;Choe, Seung-Hwan;Lee, Sang-Ho;Kim, Gyeong-Su;Gong, Yong-Jun
    • Proceedings of the Korea Contents Association Conference
    • /
    • 2004.11a
    • /
    • pp.395-401
    • /
    • 2004
  • In the current high network-centralized computing and enterprising environment, it is getting essential to transmit data reliably at very high rates. Until now previous client/server model based NFS(Network File System) or AFS(Andrew's Files System) have met the various demands but from now couldn't satisfy those of the today's scalable high-performance computing environment. Not only performance but data sharing service redundancy have risen as a serious problem. In case of NFS, the locking issue and cache cause file system to reboot and make problem when it is used simply as ip-take over for H/A service. In case of AFS, it provides file sharing redundancy but it is not possible until the storage supporting redundancy and equipments are prepared. Lustre is an open source based cluster file system developed to meet both demands. Lustre consists of three types of subsystems : MDS(Meta-Data Server) which offers the meta-data services, OST(Objec Storage Targets) which provide file I/O, and Lustre Clients which interact with OST and MDS. These subsystems with message exchanging and pursuing scalable high-performance file system service. In this paper, we compare the transmission speed of gigabytes file between Lustre and NFS on the basis of concurrent users and also present the high availability of the file system by removing more than one OST in operation.

  • PDF

A File Clustering Algorithm for Wear-leveling (마모도 평준화를 위한 File Clustering 알고리즘)

  • Lee, Taehwa;Cha, Jaehyuk
    • Journal of Digital Contents Society
    • /
    • v.14 no.1
    • /
    • pp.51-57
    • /
    • 2013
  • Storage device based on Flash Memory have many attractive features such as high performance, low power consumption, shock resistance, and low weight, so they replace HDDs to a certain extent. An Storage device based on Flash Memory has FTL(Flash Translation Layer) which emulate block storage devices like HDDs. A garbage collection, one of major functions of FTL, effects highly on the performance and the lifetime of devices. However, there is no de facto standard for new garbage collection algorithms. To solve this problem, we propose File Clustering Algorithm. File Clustering Algorithm respect to update page from same file at the same time. So, these are clustered to same block. For this mechanism, We propose Page Allocation Policy in FTL and use MIN-MAX GAP to guarantee wear leveling. To verify the algorithm in this paper, we use TPC Benchmark. So, The performance evaluation reveals that the proposed algorithm has comparable result with the existing algorithms(No wear leveling, Hot/Cold) and shows approximately 690% improvement in terms of the wear leveling.

Torus Network Based Distributed Storage System for Massive Multimedia Contents (토러스 연결망 기반의 대용량 멀티미디어용 분산 스토리지 시스템)

  • Kim, Cheiyol;Kim, Dongoh;Kim, Hongyeon;Kim, Youngkyun;Seo, Daewha
    • Journal of Korea Multimedia Society
    • /
    • v.19 no.8
    • /
    • pp.1487-1497
    • /
    • 2016
  • Explosively growing service of digital multimedia data increases the need for highly scalable low-cost storage. This paper proposes the new storage architecture based on torus network which does not need network switch and erasure coding for efficient storage usage for high scalability and efficient disk utilization. The proposed model has to compensate for the disadvantage of long network latency and network processing overhead of torus network. The proposed storage model was compared to two most popular distributed file system, GlusterFS and Ceph distributed file systems through a prototype implementation. The performance of prototype system shows outstanding results than erasure coding policy of two file systems and mostly even better results than replication policy of them.

Priority-Based Hybrid File Storage Management System Using Logical Volume Manager (논리 볼륨 매니저를 이용한 파일 우선순위 기반의 하이브리드 저장장치 관리 시스템)

  • Choi, Hoonha;Kim, Hyeunjee;No, Jaechun
    • Journal of the Institute of Electronics and Information Engineers
    • /
    • v.53 no.12
    • /
    • pp.94-102
    • /
    • 2016
  • Recently, the I/O performance of a single node is rapidly improving due to the advent of high-performance SSD. As a result, the next-generation storage platform based on SSD has received a great deal of attention and such storage platforms are increasingly adopted to commodity servers or data centers that look for the high-bandwidth computation and I/O. However, building all SSD-based storage platform may not be cost-effective because the price per storage capacity is very high as compared to that of HDD. In this paper. we propose a hybrid file management solution, called HyPLVM(Hybrid Priority Logical Volume Manager), which combines the strength of SSD with the desirable aspects of low-price, high-storage capacity HDD. HyPLVM prioritizes the files and directories to be accessed by users, in order to determine the target storage device (SSD/HDD) in which files are allocated, while mitigating the cost of building storage platforms.

Regular File Access of Embedded System Using Flash Memory as a Storage (플래시 메모리를 저장매체로 사용하는 임베디드 시스템에서의 정규파일 접근)

  • 이은주;박현주
    • Journal of Information Technology Applications and Management
    • /
    • v.11 no.1
    • /
    • pp.189-200
    • /
    • 2004
  • Recently Flash Memory which is small and low-powered is widely used as a storage of embedded system, because an embedded system requests portability and a fast response. To resolve a difference of access time between a storage and RAM, Linux is using disk caching which copies a part of file on disk into RAM. It is not also an exception on embedded system. A READ access-time of flash memory is similar to RAMs. So, when a process on an embedded system reads data, it is similar to the time to access cached data in RAM and to access directly data on a flash memory. On the embedded system using limited memory, using a disk cache is that wastes much time and memory spaces to manage it and can not reflects the characteristic of a flash memory. This paper proposes the regular file access of limited using a page cache in the file system based on a flash memory and reflects the characteristic of a flash memory. The proposed algorithm minimizes power consumption because access numbers of the RAM are reduced and doesn't waste a memory space because it accesses directly to a flash memory Therefore, the performance improvement of the system applying the proposed algorithm is expected.

  • PDF

A Study of Method to Restore Deduplicated Files in Windows Server 2012 (윈도우 서버 2012에서 데이터 중복 제거 기능이 적용된 파일의 복원 방법에 관한 연구)

  • Son, Gwancheol;Han, Jaehyeok;Lee, Sangjin
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.27 no.6
    • /
    • pp.1373-1383
    • /
    • 2017
  • Deduplication is a function to effectively manage data and improve the efficiency of storage space. When the deduplication is applied to the system, it makes it possible to efficiently use the storage space by dividing the stored file into chunks and storing only unique chunk. However, the commercial digital forensic tool do not support the file system analysis, and the original file extracted by the tool can not be executed or opened. Therefore, in this paper, we analyze the process of generating chunks of data for a Windows Server 2012 system that can apply deduplication, and the structure of the resulting file(Chunk Storage). We also analyzed the case where chunks that are not covered in the previous study are compressed. Based on these results, we propose the method to collect deduplicated data and reconstruct the original file for digital forensic investigation.

Design of Adaptive Deduplication Algorithm Based on File Type and Size (파일 유형과 크기에 따른 적응형 중복 제거 알고리즘 설계)

  • Hwang, In-Cheol;Kwon, Oh-Young
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.24 no.2
    • /
    • pp.149-157
    • /
    • 2020
  • Today, due to the large amount of data duplication caused by the increase in user data, various deduplication studies have been conducted. However, research on personal storage is relatively poor. Personal storage, unlike high-performance computers, needs to perform deduplication while reducing CPU and memory resource usage. In this paper, we propose an adaptive algorithm that selectively applies fixed size chunking (FSC) and whole file chunking (WFH) according to the file type and size in order to maintain the deduplication rate and reduce the load in personal storage. We propose an algorithm for minimization. The experimental results show that the proposed file system has more than 1.3 times slower at first write operation but less than 3 times reducing in memory usage compare to LessFS and it is 2.5 times faster at rewrite operation.

Performance Analysis of Cloud-Backed File Systems with Various Object Sizes (클라우드 기반 파일 시스템의 오브젝트 크기별 성능 분석)

  • Kim, Jiwon;Lee, Kyungjun;Ryu, Sungtae;Han, wansoo
    • Journal of KIISE
    • /
    • v.43 no.7
    • /
    • pp.744-750
    • /
    • 2016
  • Recent cloud infrastructures provide competitive performances and operation costs for many internet services through pay-per-use model. Particularly, object storages are highlighted, as they have unlimited file holding capacity and allow users to access the stored files anytime and anywhere. Several lines of research are based on cloud-backed file systems, which support traditional POSIX interface rather than RESTful APIs via HTTP. However, these existing file systems handle all files with uniform size backing objects. Consequently, the accesses to cloud object storages are likely to be inefficient. In our research, files are profiled according to characteristics, and appropriate backing unit sizes are determined. We experimentally verify that different backing unit sizes for the object storage improve the performance of cloud-backed file systems. In our comparative experiments with S3QL, our prototype cloud-backed file system shows faster performance by 18.6% on average.

Data Access Frequency based Data Replication Method using Erasure Codes in Cloud Storage System (클라우드 스토리지 시스템에서 데이터 접근빈도와 Erasure Codes를 이용한 데이터 복제 기법)

  • Kim, Ju-Kyeong;Kim, Deok-Hwan
    • Journal of the Institute of Electronics and Information Engineers
    • /
    • v.51 no.2
    • /
    • pp.85-91
    • /
    • 2014
  • Cloud storage system uses a distributed file system for storing and managing data. Traditional distributed file system makes a triplication of data in order to restore data loss in disk failure. However, enforcing data replication method increases storage utilization and causes extra I/O operations during replication process. In this paper, we propose a data replication method using erasure codes in cloud storage system to improve storage space efficiency and I/O performance. In particular, according to data access frequency, the proposed method can reduce the number of data replications but using erasure codes can keep the same data recovery performance. Experimental results show that proposed method improves performance in storage efficiency 40%, read throughput 11%, write throughput 10% better than HDFS does.

A Study of File Replacement Policy in Data Grid Environments (데이터 그리드 환경에서 파일 교체 정책 연구)

  • Park, Hong-Jin
    • The KIPS Transactions:PartA
    • /
    • v.13A no.6 s.103
    • /
    • pp.511-516
    • /
    • 2006
  • The data grid computing provides geographically distributed storage resources to solve computational problems with large-scale data. Unlike cache replacement policies in virtual memory or web-caching replacement, an optimal file replacement policy for data grids is the one of the important problems by the fact that file size is very large. The traditional file replacement policies such as LRU(Least Recently Used) LCB-K(Least Cost Beneficial based on K), EBR(Economic-based cache replacement), LVCT(Least Value-based on Caching Time) have the problem that they have to predict requests or need additional resources to file replacement. To solve theses problems, this paper propose SBR-k(Sized-based replacement-k) that replaces files based on file size. The results of the simulation show that the proposed policy performs better than traditional policies.