• Title/Summary/Keyword: HDFS

Search Result 150, Processing Time 0.026 seconds

Adaptable I/O System based I/O Reduction for Improving the Performance of HDFS

  • Park, Jung Kyu;Kim, Jaeho;Koo, Sungmin;Baek, Seungjae
    • JSTS:Journal of Semiconductor Technology and Science
    • /
    • v.16 no.6
    • /
    • pp.880-888
    • /
    • 2016
  • In this paper, we propose a new HDFS-AIO framework to enhance HDFS with Adaptive I/O System (ADIOS), which supports many different I/O methods and enables applications to select optimal I/O routines for a particular platform without source-code modification and re-compilation. First, we customize ADIOS into a chunk-based storage system so its API semantics can fit the requirement of HDFS easily; then, we utilize Java Native Interface (JNI) to bridge HDFS and the tailored ADIOS. We use different I/O patterns to compare HDFS-AIO and the original HDFS, and the experimental results show the design feasibility and benefits. We also examine the performance of HDFS-AIO using various I/O techniques. There have been many studies that use ADIOS, however our research is expected to help in expanding the function of HDFS.

Design and Implementation of HDFS Data Encryption Scheme Using ARIA Algorithms on Hadoop (하둡 상에서 ARIA 알고리즘을 이용한 HDFS 데이터 암호화 기법의 설계 및 구현)

  • Song, Youngho;Shin, YoungSung;Chang, Jae-Woo
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.5 no.2
    • /
    • pp.33-40
    • /
    • 2016
  • Due to the growth of social network systems (SNS), big data are realized and Hadoop was developed as a distributed platform for analyzing big data. Enterprises analyze data containing users' sensitive information by using Hadoop and utilize them for marketing. Therefore, researches on data encryption have been done to protect the leakage of sensitive data stored in Hadoop. However, the existing researches support only the AES encryption algorithm, the international standard of data encryption. Meanwhile, Korean government choose ARIA algorithm as a standard data encryption one. In this paper, we propose a HDFS data encryption scheme using ARIA algorithms on Hadoop. First, the proposed scheme provide a HDFS block splitting component which performs ARIA encryption and decryption under the distributed computing environment of Hadoop. Second, the proposed scheme also provide a variable-length data processing component which performs encryption and decryption by adding dummy data, in case when the last block of data does not contains 128 bit data. Finally, we show from performance analysis that our proposed scheme can be effectively used for both text string processing applications and science data analysis applications.

A Network Load Sensitive Block Placement Strategy of HDFS

  • Meng, Lingjun;Zhao, Wentao;Zhao, Haohao;Ding, Yang
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.9 no.9
    • /
    • pp.3539-3558
    • /
    • 2015
  • This paper investigates and analyzes the default block placement strategy of HDFS. HDFS is a typical representative distributed file system to stream vast amount of data effectively at high bandwidth to user applications. However, the default HDFS block placement policy assumes that all nodes in the cluster are homogeneous, and places blocks with a simple RoundRobin strategy without considering any nodes' resource characteristics, which decreases self-adaptability of the system. The primary contribution of this paper is the proposition of a network load sensitive block placement strategy. We have implemented our algorithm and justify it through extensive simulations and comparison with similar existing studies. The results indicate that our work not only performs much better in the data distribution but also improves write performance more significantly than the others.

Protective Effects of Bifidobacterium bifidum Culture Supernatants and Intracellular Cell-Free Extracts on Human Dermal Fibroblasts against UV-B Irradiation (인간 진피섬유아세포에서 Bifidobacterium bifidum 배양액 및 추출액의 자외선B에 대한 보호 효능)

  • Gwon, Gi Yeong;Park, Gwi Gun
    • Journal of the Korean Society of Food Science and Nutrition
    • /
    • v.46 no.7
    • /
    • pp.801-808
    • /
    • 2017
  • The present study investigated the protective effects of Bifidobacterium bifidum culture supernatants (BbSC) and intracellular cell-free extracts (BbICFE) on human dermal fibroblasts (HDFs) against ultraviolet-B (UV-B) irradiation. HDFs were treated with UV-B, UV-B+BbCS, and UV-B+BbICFE. Treatment of UV-B-irradiated HDFs with BbCS and BbICFE significantly increased cell viability compared to UV-B-irradiated HDFs. BbCS treatment reduced senescence in HDFs by approximately 40.0%. Moreover, sub-G1 phase was significantly reduced in BbCS- and BbICFE-treated HDFs (3.3% and 4.5%, respectively). The effect of UV-B on oxidative damage of HDFs was measured by dichlorofluorescin diacetate. Fluorescence intensity significantly increased in UV-B-irradiated HDFs. Inhibition of cellular reactive oxygen species in HDFs treated with 0.01% BbCS was the highest at 34.1%. Levels of p21 and p53 protein expression induced by UV-B irradiation were reduced by treatment with BbCS and BbICFE (47.0% and 35.6%, respectively). These results show that BbCS and BbICFE reduce UV-B-induced cellular senescence and apoptosis in HDFs. Thus, BbCS and BbICFE can be used as potential agents for protection of UV-B-induced skin cell damage.

Stereoisomer-specific ginsenoside 20(S)-Rg3 reverses replicative senescence of human diploid fibroblasts via Akt-mTOR-Sirtuin signaling

  • Yang, Kyeong-Eun;Jang, Hyun-Jin;Hwang, In-Hu;Hong, Eun Mi;Lee, Min-Goo;Lee, Soon;Jang, Ik-Soon;Choi, Jong-Soon
    • Journal of Ginseng Research
    • /
    • v.44 no.2
    • /
    • pp.341-349
    • /
    • 2020
  • Background: The replicative senescence of human dermal fibroblasts (HDFs) is accompanied by growth arrest. In our previous study, the treatment of senescent HDFs with Rg3(S) lowered the intrinsic reactive oxygen species (ROS) levels and reversed cellular senescence by inducing peroxiredoxin-3, an antioxidant enzyme. However, the signaling pathways involved in Rg3(S)-induced senescence reversal in HDFs and the relatedness of the stereoisomer Rg3(R) in corresponding signaling pathways are not known yet. Methods: We performed senescence-associated β-galactosidase and cell cycle assays in Rg3(S)-treated senescent HDFs. The levels of ROS, adenosine triphosphate (ATP), and cyclic adenosine monophosphate (cAMP) as well as the mitochondrial DNA copy number, nicotinamide adenine dinucleotide (NAD)+/1,4-dihydronicotinamide adenine dinucleotide (NADH) ratio, and NAD-dependent sirtuins expression were measured and compared among young, old, and Rg3(S)-pretreated old HDFs. Major signaling pathways of phosphatidylinositol 3-kinase/Akt, 5' adenosine monophosphate-activated protein kinase (AMPK), and sirtuin 1/3, including cell cycle regulatory proteins, were examined by immunoblot analysis. Results: Ginsenoside Rg3(S) reversed the replicative senescence of HDFs by restoring the ATP level and NAD+/NADH ratio in downregulated senescent HDFs. Rg3(S) recovered directly the cellular levels of ROS and the NAD+/NADH ratio in young HDFs inactivated by rotenone. Rg3(S) mainly downregulated phosphatidylinositol 3-kinase/Akt through the inhibition of mTOR by cell cycle regulators like p53/p21 in senescent HDFs, whereas Rg3(R) did not alter the corresponding signaling pathways. Rg3(S)-activated sirtuin 3/PGC1α to stimulate mitochondrial biogenesis. Conclusion: Cellular molecular analysis suggests that Rg3(S) specifically reverses the replicative senescence of HDFs by modulating Akt-mTOR-sirtuin signaling to promote the biogenesis of mitochondria.

A Study on the Sharing Criteria Between NGSO/FSS and HDFS (NGSO/FSS와 HDFS간 공유 기준에 관한 연구)

  • 강병수;권태곤;이성팔
    • Proceedings of the IEEK Conference
    • /
    • 2000.11a
    • /
    • pp.465-468
    • /
    • 2000
  • The interference situation for the satellite networks using non-geostationary satellite orbit (NGSO) is more complicated than the situation between geostationary satellite orbit (GSO) networks because of the time varying orbital characteristics of NGSO systems. Most of frequency bands are allocated to fixed-satellite service and liked-service in co-primary basis. In this paper, the sharing criteria between NGSO/FSS systems and HDFS (High Density Fixed-Service) are examined.

  • PDF

A Dynamic Data Replica Deletion Strategy on HDFS using HMM (HMM을 이용한 HDFS 기반 동적 데이터 복제본 삭제 전략)

  • Seo, Young-Ho;Youn, Hee-Yong
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2014.07a
    • /
    • pp.241-244
    • /
    • 2014
  • 본 논문에서는 HDFS(Hadoop Distributed File System)에서 문제되고 있는 복제정책의 개선을 위해 HMM(Hidden Markov Model)을 이용한 동적 데이터 복제본 삭제 전략을 제안한다. HDFS는 대용량 데이터를 효과적으로 처리할 수 있는 분산 파일 시스템으로 높은 Fault-Tolerance를 제공하며, 데이터의 접근에 높은 처리량을 제공하여 대용량 데이터 집합을 갖는 응용 프로그램에 최적화 되어있는 장점을 가지고 있다. 하지만 HDFS 에서의 복제 메커니즘은 시스템의 안정성과 성능을 향상시키지만, 추가 블록 복제본이 많은 디스크 공간을 차지하여 유지보수 비용 또한 증가하게 된다. 본 논문에서는 HMM과 최상의 상태 순서를 찾는 알고리즘인 Viterbi Algorithm을 이용하여 불필요한 데이터 복제본을 탐색하고, 탐색된 복제본의 삭제를 통하여 HDFS의 디스크 공간과 유지보수 비용을 절약 할 수 있는 전략을 제안한다.

  • PDF

LDBAS: Location-aware Data Block Allocation Strategy for HDFS-based Applications in the Cloud

  • Xu, Hua;Liu, Weiqing;Shu, Guansheng;Li, Jing
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.12 no.1
    • /
    • pp.204-226
    • /
    • 2018
  • Big data processing applications have been migrated into cloud gradually, due to the advantages of cloud computing. Hadoop Distributed File System (HDFS) is one of the fundamental support systems for big data processing on MapReduce-like frameworks, such as Hadoop and Spark. Since HDFS is not aware of the co-location of virtual machines in the cloud, the default scheme of block allocation in HDFS does not fit well in the cloud environments behaving in two aspects: data reliability loss and performance degradation. In this paper, we present a novel location-aware data block allocation strategy (LDBAS). LDBAS jointly optimizes data reliability and performance for upper-layer applications by allocating data blocks according to the locations and different processing capacities of virtual nodes in the cloud. We apply LDBAS to two stages of data allocation of HDFS in the cloud (the initial data allocation and data recovery), and design the corresponding algorithms. Finally, we implement LDBAS into an actual Hadoop cluster and evaluate the performance with the benchmark suite BigDataBench. The experimental results show that LDBAS can guarantee the designed data reliability while reducing the job execution time of the I/O-intensive applications in Hadoop by 8.9% on average and up to 11.2% compared with the original Hadoop in the cloud.

Protective Effects of Antioxidant Active Fractions Derived from the Edible Seaweed Hizikia fusiformis in Oxidatively Stressed Human Dermal Fibroblasts (식용 해조류 톳(Hizikia fusiformis) 유래 항산화 활성분획물의 산화적 손상이 유도된 인간피부섬유아세포 보호 효과)

  • Cui, Yong Ri;Kim, Hyun-Soo;Je, Jun-Geon;Wang, Lei;Oh, Jae-Young;Jia, Liu;Jeon, You-Jin
    • Korean Journal of Fisheries and Aquatic Sciences
    • /
    • v.52 no.1
    • /
    • pp.35-42
    • /
    • 2019
  • We investigated the protective effects of antioxidant fractions from a 70% ethanolic extract of Hizikia fusiformis in human dermal fibroblasts (HDFs). Powdered H. fusiformis was extracted with 70% ethanol and then partitioned into three fractions according to polarity using n-hexane (HFH), chloroform (HFC), and ethyl acetate (HFEA). Antioxidant activity was observed in HFEA at 0.66 mg/mL based on the half maximal inhibitory concentration ($IC_{50}$) of 1,1-diphenyl-2-picrylhydrazyl (DPPH), and at 0.24 mg/mL based on alkyl radical scavenging. The protective effects of the HFEA antioxidant fraction against 2,2-azobis-(2-amidinopropane) dihydrochloride (AAPH)-damaged HDFs and the expression of Type I procollagen in HDFs were examined. HFEA caused the proliferation of HDFs with and without AAPH treatment and protected against AAPH damage to HDFs in a dose-dependent manner ($50-200{\mu}g/mL$). This implies that the antioxidant properties of the fractions depended on their proliferative and protective effects. The HFEA antioxidant fraction had significant effects and caused the dose-dependent expression of Type I procollagen, an important anti-wrinkle protein, in HDFs. In conclusion, antioxidant substances in H. fusiformis were found in the ethyl acetate fraction, and the resulting HFEA may have cosmetic applications.

A Parallel HDFS and MapReduce Functions for Emotion Analysis (감성분석을 위한 병렬적 HDFS와 맵리듀스 함수)

  • Back, BongHyun;Ryoo, Yun-Kyoo
    • Journal of the Korea society of information convergence
    • /
    • v.7 no.2
    • /
    • pp.49-57
    • /
    • 2014
  • Recently, opinion mining is introduced to extract useful information from SNS data and to evaluate the true intention of users. Opinion mining are required several efficient techniques to collect and analyze a large amount of SNS data and extract meaningful data from them. Therefore in this paper, we propose a parallel HDFS(Hadoop Distributed File System) and emotion functions based on Mapreduce to extract some emotional information of users from various unstructured big data on social networks. The experiment results have verified that the proposed system and functions perform faster than O(n) for data gathering time and loading time, and maintain stable load balancing for memory and CPU resources.

  • PDF