• Title/Summary/Keyword: 중복제거

Search Result 552, Processing Time 0.027 seconds

Deduplication Technique for Smartphone Application Update Scenario (스마트폰의 어플리케이션 업데이트 패턴을 고려한 데이터 중복제거 기법 연구)

  • Park, Dae-Jun;Choi, Dong-Soo;Shin, Dong-Kun
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2012.06a
    • /
    • pp.364-366
    • /
    • 2012
  • 스마트폰의 어플리케이션은 어플리케이션 생태계의 발전에 따라 그 수가 많아지고, 업데이트 또한 잦아졌다. 어플리케이션의 업데이트는 낸드 플래시 메모리에 이전 버전을 삭제하고, 새로운 버전의 어플리케이션에 대한 쓰기 명령을 내린다. 따라서 사용자는 낸드 플래시 메모리에서의 상대적으로 느린 쓰기 명령에 의해 스마트폰의 성능의 저하를 느끼고 낸드 플래시 메모리는 반복되는 지우기/쓰기 동작에 의해 수명이 단축된다. 본 논문에서는 업데이트 되는 스마트폰 어플리케이션 데이터가 이전 버전과 큰 차이가 없다는 것에 착안하여 데이터 중복제거를 통해 업데이트 성능을 향상시키고 낸드 플래시 메모리의 수명을 향상시키는 기법을 제안하고 있으며, 실험을 통해서 어플리케이션들에 대한 중복 제거율을 관찰하였다.

LTRE: Lightweight Traffic Redundancy Elimination in Software-Defined Wireless Mesh Networks (소프트웨어 정의 무선 메쉬 네트워크에서의 경량화된 중복 제거 기법)

  • Park, Gwangwoo;Kim, Wontae;Kim, Joonwoo;Pack, Sangheon
    • Journal of KIISE
    • /
    • v.44 no.9
    • /
    • pp.976-985
    • /
    • 2017
  • Wireless mesh network (WMN) is a promising technology for building a cost-effective and easily-deployed wireless networking infrastructure. To efficiently utilize limited radio resources in WMNs, packet transmissions (particularly, redundant packet transmissions) should be carefully managed. We therefore propose a lightweight traffic redundancy elimination (LTRE) scheme to reduce redundant packet transmissions in software-defined wireless mesh networks (SD-WMNs). In LTRE, the controller determines the optimal path of each packet to maximize the amount of traffic reduction. In addition, LTRE employs three novel techniques: 1) machine learning (ML)-based information request, 2) ID-based source routing, and 3) popularity-aware cache update. Simulation results show that LTRE can significantly reduce the traffic overhead by 18.34% to 48.89%.

Optimization Using Partial Redundancy Elimination in SSA Form (SSA Form에서 부분 중복 제거를 이용한 최적화)

  • Kim, Ki-Tae;Yoo, Weon-Hee
    • The KIPS Transactions:PartD
    • /
    • v.14D no.2
    • /
    • pp.217-224
    • /
    • 2007
  • In order to determine the value and type statically. CTOC uses the SSA Form which separates the variable according to assignment. The SSA Form is widely being used as the intermediate expression of the compiler for data flow analysis as well as code optimization. However, the conventional SSA Form is more associated with variables rather than expressions. Accordingly, the redundant expressions are eliminated to optimize expressions of the SSA From. This paper defines the partial redundant expression to obtain a more optimized code and also implements the technique for eliminating such expressions.

Secure and Efficient Client-side Deduplication for Cloud Storage (안전하고 효율적인 클라이언트 사이드 중복 제거 기술)

  • Park, Kyungsu;Eom, Ji Eun;Park, Jeongsu;Lee, Dong Hoon
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.25 no.1
    • /
    • pp.83-94
    • /
    • 2015
  • Deduplication, which is a technique of eliminating redundant data by storing only a single copy of each data, provides clients and a cloud server with efficiency for managing stored data. Since the data is saved in untrusted public cloud server, however, both invasion of data privacy and data loss can be occurred. Over recent years, although many studies have been proposed secure deduplication schemes, there still remains both the security problems causing serious damages and inefficiency. In this paper, we propose secure and efficient client-side deduplication with Key-server based on Bellare et. al's scheme and challenge-response method. Furthermore, we point out potential risks of client-side deduplication and show that our scheme is secure against various attacks and provides high efficiency for uploading big size of data.

Parallel Rabin Fingerprinting on GPGPU for Efficient Data Deduplication (효율적인 데이터 중복제거를 위한 GPGPU 병렬 라빈 핑거프린팅)

  • Ma, Jeonghyeon;Park, Sejin;Park, Chanik
    • Journal of KIISE
    • /
    • v.41 no.9
    • /
    • pp.611-616
    • /
    • 2014
  • Rabin fingerprinting used for chunking requires the largest amount computation time in data deduplication, In this paper, therefore, we proposed parallel Rabin fingerprinting on GPGPU for efficient data deduplication. In addition, for efficient parallelism in Rabin fingerprinting, four issues are considered. Firstly, when dividing input data stream into data sections, we consider the data located near the boundaries between data sections to calculate Rabin fingerprint continuously. Secondly, we consider exploiting the characteristics of Rabin fingerprinting for efficient operation. Thirdly, we consider the chunk boundaries which can be changed compared to sequential Rabin fingerprinting when adapting parallel Rabin fingerprinting. Finally, we consider optimizing GPGPU memory access. Parallel Rabin fingerprinting on GPGPU shows 16 times and 5.3 times better performance compared to sequential Rabin fingerprinting on CPU and compared to parallel Rabin fingerprinting on CPU, respectively. These throughput improvement of Rabin fingerprinting can lead to total performance improvement of data deduplication.

A Study of Method to Restore Deduplicated Files in Windows Server 2012 (윈도우 서버 2012에서 데이터 중복 제거 기능이 적용된 파일의 복원 방법에 관한 연구)

  • Son, Gwancheol;Han, Jaehyeok;Lee, Sangjin
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.27 no.6
    • /
    • pp.1373-1383
    • /
    • 2017
  • Deduplication is a function to effectively manage data and improve the efficiency of storage space. When the deduplication is applied to the system, it makes it possible to efficiently use the storage space by dividing the stored file into chunks and storing only unique chunk. However, the commercial digital forensic tool do not support the file system analysis, and the original file extracted by the tool can not be executed or opened. Therefore, in this paper, we analyze the process of generating chunks of data for a Windows Server 2012 system that can apply deduplication, and the structure of the resulting file(Chunk Storage). We also analyzed the case where chunks that are not covered in the previous study are compressed. Based on these results, we propose the method to collect deduplicated data and reconstruct the original file for digital forensic investigation.

Design of Adaptive Deduplication Algorithm Based on File Type and Size (파일 유형과 크기에 따른 적응형 중복 제거 알고리즘 설계)

  • Hwang, In-Cheol;Kwon, Oh-Young
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.24 no.2
    • /
    • pp.149-157
    • /
    • 2020
  • Today, due to the large amount of data duplication caused by the increase in user data, various deduplication studies have been conducted. However, research on personal storage is relatively poor. Personal storage, unlike high-performance computers, needs to perform deduplication while reducing CPU and memory resource usage. In this paper, we propose an adaptive algorithm that selectively applies fixed size chunking (FSC) and whole file chunking (WFH) according to the file type and size in order to maintain the deduplication rate and reduce the load in personal storage. We propose an algorithm for minimization. The experimental results show that the proposed file system has more than 1.3 times slower at first write operation but less than 3 times reducing in memory usage compare to LessFS and it is 2.5 times faster at rewrite operation.

Indexing method with deduplication for efficient RDF data retrieving (효율적인 RDF 데이터 검색을 위한 중복 제거 색인 방법)

  • Jang, Hyeonggyu;Bang, Sungho;Oh, Sangyoon
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2020.01a
    • /
    • pp.61-62
    • /
    • 2020
  • RDF의 활용이 증가하면서 RDF데이터를 저장하는 방법 또한 많은 연구가 이루어졌다. 그래프 형태인 RDF 데이터를 테이블로 바꿀 때, 동일한 데이터가 중복 저장되어 검색 시 불필요한 연산을 하는 문제점이 발생한다. 본 논문에서는 중복저장 및 불필요한 검색을 줄이기 위해 색인을 주어(S), 목적어(O) 색인과 이들의 중복 값을 별도의 색인을 만들고, 검색 시 중복 값을 확인하여 필요한 색인만 검색하는 기법을 제안한다. 실험에서 본 기법을 사용하여 불필요한 검색을 줄여서 전체적인 검색 시간이 줄어드는 것을 확인하였다.

  • PDF

Design and Implementation of Multiple Filter Distributed Deduplication System Applying Cuckoo Filter Similarity (쿠쿠 필터 유사도를 적용한 다중 필터 분산 중복 제거 시스템 설계 및 구현)

  • Kim, Yeong-A;Kim, Gea-Hee;Kim, Hyun-Ju;Kim, Chang-Geun
    • Journal of Convergence for Information Technology
    • /
    • v.10 no.10
    • /
    • pp.1-8
    • /
    • 2020
  • The need for storage, management, and retrieval techniques for alternative data has emerged as technologies based on data generated from business activities conducted by enterprises have emerged as the key to business success in recent years. Existing big data platform systems must load a large amount of data generated in real time without delay to process unstructured data, which is an alternative data, and efficiently manage storage space by utilizing a deduplication system of different storages when redundant data occurs. In this paper, we propose a multi-layer distributed data deduplication process system using the similarity of the Cuckoo hashing filter technique considering the characteristics of big data. Similarity between virtual machines is applied as Cuckoo hash, individual storage nodes can improve performance with deduplication efficiency, and multi-layer Cuckoo filter is applied to reduce processing time. Experimental results show that the proposed method shortens the processing time by 8.9% and increases the deduplication rate by 10.3%.

Design and Implementation of SANique Smart Vault Backup System for Massive Data Services (대용량 데이터 서비스를 위한 SANique Smart Vault 백업 시스템의 설계 및 구현)

  • Lee, Kyu Woong
    • The Journal of Korean Association of Computer Education
    • /
    • v.17 no.2
    • /
    • pp.97-106
    • /
    • 2014
  • There is a lot of interest in the data storage and backup systems according to increasing the data intensive services and related user's data. The overhead of backup performance in massive storage system is a critical issue because the traditional incremental backup strategies causes the time consuming bottleneck in the SAN environment. The SANique Smart Vault system is a high performance backup solution with data de-duplication technology and it guarantees these requirements. In this paper, we describe the architecture of SANique Smart Vault system and illustrate efficient delta incremental backup method based on journaling files. We also present the record-level data de-duplication method in our proposed backup system. The proposed forever incremental backup and data de-duplication algorithms are analyzed and investigated by performance evaluation of other commercial backup solutions.

  • PDF