Browse > Article
http://dx.doi.org/10.3745/KTCCS.2019.8.10.231

Parallelization of Genome Sequence Data Pre-Processing on Big Data and HPC Framework  

Byun, Eun-Kyu (한국과학기술정보연구원)
Kwak, Jae-Hyuck (한국과학기술정보연구원)
Mun, Jihyeob (한국과학기술정보연구원)
Publication Information
KIPS Transactions on Computer and Communication Systems / v.8, no.10, 2019 , pp. 231-238 More about this Journal
Abstract
Analyzing next-generation genome sequencing data in a conventional way using single server may take several tens of hours depending on the data size. However, in order to cope with emergency situations where the results need to be known within a few hours, it is required to improve the performance of a single genome analysis. In this paper, we propose a parallelized method for pre-processing genome sequence data which can reduce the analysis time by utilizing the big data technology and the highperformance computing cluster which is connected to the high-speed network and shares the parallel file system. For the reliability of analytical data, we have chosen a strategy to parallelize the existing analytical tools and algorithms to the new environment. Parallelized processing, data distribution, and parallel merging techniques have been developed and performance improvements have been confirmed through experiments.
Keywords
Genome Sequence Data Preprocessing; NGS; Big Data; Hadoop; HPC; Parallelization;
Citations & Related Records
연도 인용수 순위
  • Reference
1 S. Goodwin, J. D. McPherson, and W. R. McCombie, "Coming of age: ten years of next-generation sequencing technologies," Nature Review Genetics, Vol.17, No.6, pp.333-351, May 2016.   DOI
2 "Sequence Alignment/Map Format Specification", The SAM/BAM Format Specification Working Group [Internet], https://samtools.github.io/hts-specs/SAMv1.pdf
3 An introduction to Next-Generation Sequencing Technology, Illumina, Inc., [Internet], https://www.illumina.com/documents/products/illumina_sequencing_introduction.pdf
4 H. Li and R. Durbin, "Fast and accurate long-read alignment with Burrows-Wheeler transform," Bioinformatics, Vol.26, No.5, pp.589-595, 2010.   DOI
5 Faust, Gregory G., and Ira M. Hall, "SAMBLASTER: Fast Duplicate Marking and Structural Variant Read Extraction," Bioinformatics, Vol.30, No.17, pp.2503-2505, 2014.   DOI
6 H. Li, B. Handsaker, A. Wysoker, T. Fennell, J. Ruan, N. Homer, G. Marth, G. Abecasis, R. Durbin, and 1000 Genome Project Data Processing Subgroup, "The Sequence Alignment/Map format and SAMtools," Bioinformatics, Vol.25, No.16, pp.2078-2079, 2009.   DOI
7 L. Pireddu, S. Leo, and G. Zanetti, "SEAL: a distributed short read mapping and duplicate removal tool," Bioinformatics, Vol. 27, No.15, pp.2159-2160, Aug. 2011.   DOI
8 T. Nguyen, W. Shi, and D. Ruden, "CloudAligner: A fast and full-featured MapReduce based tool for sequence mapping," BMC Res Notes, Vol.4, No.1, pp.171, Jun. 2011.   DOI
9 M. C. Schatz, "CloudBurst: highly sensitive read mapping with MapReduce," Bioinformatics, Vol.25, No.11, pp.1363-1369, Jun. 2009.   DOI
10 J. M. Abuin, J. C. Pichel, T. F. Pena, and J. Amigo, "BigBWA: approaching the Burrows-Wheeler aligner to Big Data technologies," Bioinformatics, Vol.31, No.24, pp.4003-4005, 2015.   DOI
11 D. Decap, J. Reumers, C. Herzeel, P. Costanza, and J. Fostier, "Halvade: Scalable Sequence Analysis with MapReduce," Bioinformatics, Vol.31, No.15, pp.2482-2488, 2015.   DOI
12 E.-K. Byun, J. Lee, S. J. Yu, J.-H. Kwak, and S. Hwang, "Accelerating Genome Sequence Alignment on Hadoop on Lustre Environment," 2017 IEEE 13th International Conference on E-Science, pp.436-437, 2017.
13 Lustre Hadoop Plugin, Seagate [Internet], https://github.com/Seagate/lustrefs