Browse > Article
http://dx.doi.org/10.5303/JKAS.2014.47.3.115

NVST DATA ARCHIVING SYSTEM BASED ON FASTBIT NOSQL DATABASE  

Liu, Ying-Bo (Computer Technology Application Key Lab of Yunnan Province, Kunming University of Science and Technology)
Wang, Feng (Computer Technology Application Key Lab of Yunnan Province, Kunming University of Science and Technology)
Ji, Kai-Fan (Computer Technology Application Key Lab of Yunnan Province, Kunming University of Science and Technology)
Deng, Hui (Computer Technology Application Key Lab of Yunnan Province, Kunming University of Science and Technology)
Dai, Wei (Computer Technology Application Key Lab of Yunnan Province, Kunming University of Science and Technology)
Liang, Bo (Computer Technology Application Key Lab of Yunnan Province, Kunming University of Science and Technology)
Publication Information
Journal of The Korean Astronomical Society / v.47, no.3, 2014 , pp. 115-122 More about this Journal
Abstract
The New Vacuum Solar Telescope (NVST) is a 1-meter vacuum solar telescope that aims to observe the fine structures of active regions on the Sun. The main tasks of the NVST are high resolution imaging and spectral observations, including the measurements of the solar magnetic field. The NVST has been collecting more than 20 million FITS files since it began routine observations in 2012 and produces maximum observational records of 120 thousand files in a day. Given the large amount of files, the effective archiving and retrieval of files becomes a critical and urgent problem. In this study, we implement a new data archiving system for the NVST based on the Fastbit Not Only Structured Query Language (NoSQL) database. Comparing to the relational database (i.e., MySQL; My Structured Query Language), the Fastbit database manifests distinctive advantages on indexing and querying performance. In a large scale database of 40 million records, the multi-field combined query response time of Fastbit database is about 15 times faster and fully meets the requirements of the NVST. Our slestudy brings a new idea for massive astronomical data archiving and would contribute to the design of data management systems for other astronomical telescopes.
Keywords
astronomical databases: miscellaneous; techniques: indexing and querying; techniques: data archiving system;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Wu, K., Madduri, K., & Canon, S. 2010, Multi-Level Bitmap Indexes for Flash Memory Storage, In Proceedings of the Fourteenth International Database Engineering & Applications Symposium, ACM, 114
2 Wu, K. 2005, FastBit: An Efficient Indexing Technology for Accelerating Data-Intensive Science, Journal of Physics: Conference Series, Vol. 16. No. 1, IOP Publishing
3 Wu, K., Otoo, E. J., Shoshani, A., & Nordberg, H. 2001, Notes on Design and Implementation of Compressed Bit Vectors, Technical Report LBNL/PUB-3161, LBNL, Berkeley, CA.
4 Wu, K., Otoo, E. J., & Shoshani, A., 2006, Optimizing Bitmap Indices with Efficient Compression, TODS, 31, 1   DOI   ScienceOn
5 Wu, K., Stockinger, K., & Shoshani, A. 2008, Break-ing the Curse of Cardinality on Bitmap Indexes, Scien-tific and Statistical Database Management (Heidelberg: Springer), 348
6 Chou, J., Wu, K., & Prabhat, P. 2011, FastQuery: A Paral-lel Indexing System for Scientific Data, 2011, IEEE Inter-national Conference on Cluster Computing (CLUSTER), 455
7 Chou, J., Wu, K., Rubel, O., Howison, M., Qiang, J., Austin, B., et al. 2011, Parallel Index and Query for Large Scale Data Analysis, in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, 30
8 O'Neil, P. E. 1989, Model 204 Architecture and Perfor-mance, in HPTS (Heidelberg: Berlin), 39
9 O'Neil, P., & Quass, D. 1997, Improved Query Performance with Variant Indexes, ACM Sigmod Record, 26, 38   DOI   ScienceOn
10 Otoo, E., & Wu, K. 2010, Accelerating Queries on Very Large Datasets, http://crd-legacy.lbl.gov/kewu/ps/LBNL-1677E.pdf
11 Parker, Z., Poe, S., & Vrbsky, S. V. 2013, Comparing NoSQL Mongodb to an SQL db, in Proceedings of the 51st ACM Southeast Conference, 5
12 Parker-Wood, A., Long, D. D., Madden, B. A., Adams, I. F., McThrow, M., & Wildani, A. 2013, Examining Extended and Scientific Metadata for Scalable Index Designs, in Proceedings of the 6th International Systems and Storage Conference, 4
13 Leavitt, N. 2010, Will NoSQL Databases Live up to Their Promise?, Computer, 43, 12
14 Sinha, R. R., Mitra, S., & Winslett, M. 2006, Bitmap In-dexes for Large Scientific Data Sets: A Case Study, in Parallel and Distributed Processing Symposium, 10
15 Strauch, C., Sites, U. L. S., & Kriha, W. 2011, NoSQL Databases, http://www.christof-strauch.de/nosqldbs.pdf
16 Szalay, A. S., Kunszt, P. Z., Thakar, A., Gray, J., Slutz, D., & Brunner, R. J. 2000, Designing and Mining Multi-Terabyte Astronomy Archives: the Sloan Digital Sky Survey, ACM SIGMOD Record, 29, 1   DOI   ScienceOn
17 Liu, Z., Xu, J., Gu, B. Z., Wang, S., You, J. Q., Shen, L. X., et al. 2014, New Vacuum Solar Telescope and Observations with High Resolution, arXiv preprint arXiv:1403.6896
18 Liu, Z., & Xu, J. 2011, 1-meter Near-Infrared Solar Tele-scope. In First Asia-Pacific Solar Physics Meeting ASI Conference Series, 2, 9
19 Jatana, N., Puri, S., Ahuja, M., Kathuria, I., & Gosain, D. 2012, A Survey and Comparison of Relational and Non-Relational Database, IJE, 1, 6   DOI
20 Kunszt, P. Z., Szalay, A. S., Csabai, I., & Thakar, A. R. 2000, The Indexing of the SDSS Science Archive, ADASS, IX, 216, 141
21 Ma, B., Shoshani, A., Sim, A., Wu, K., Byun, Y., Hahm, J., & Shin, M. S. 2012, Efficient AttributeBased Data Access in Astronomy Analysis, High Performance Com-puting, Networking, Storage and Analysis (SCC), 2012 SC Companion, 562
22 Cui, C. Z., Li, W., Yu, C., Xu, Z., Zhao, Y. H., & Yu, J. J. 2008, Search and Location of FITS Data Files, Publications of the National Astronomical Observatories of China, 5, 116
23 Gorski, K. M., Hivon, E., Banday, A. J., Wandelt, B. D., Hansen, F. K., Reinecke, M., & Bartelmann, M. 2005, HEALPix: A Framework for High-Resolution Discretiza-tion and Fast Analysis of Data Distributed on the Sphere, ApJ, 622, 759   DOI
24 Folk, M., Heber, G., Koziol, Q., Pourmal, E., & Robinson, D. 2011, An Overview of the HDF5 Technology Suite and Its Applications, In Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases, 36
25 Fu, B., Fink, E., Gibson, G., & Carbonell, J. 2012, Index-ing and Fast Near-Matching of Billions of Astronomical Objects, in Proceedings of IASDS12
26 Baruffolo, A. 1999, R-Trees for Astronomical Data Indexing, ADASS, VIII, 172, 375
27 Gosink, L., Shalf, J., Stockinger, K., Wu, K., & Bethel, W. 2006, HDF5-FastQuery: Accelerating Complex Queries on HDF Datasets Using Fast Bitmap Indices, 18th Inter-national Conference on Scientific and Statistical Database Management, 149
28 Gray, J., Liu, D. T., Nieto-Santisteban, M., Szalay, A., De-Witt, D. J., & Heber, G. 2005, Scientific Data Manage-ment in the Coming Decade, ACM SIGMOD Record, 34, 34   DOI   ScienceOn
29 Greisen, E. W., & Harten, R. H. 1981, An Extension of FITS for Groups of Small Arrays of Data, A&AS, 44, 371
30 Pesnell, W. D., Thompson, B. J., & Chamberlin, P. C. 2012, The Solar Dynamics Observatory (SDO)(New York: Springer), 3
31 Baba, H., Yasuda, N., Ichikawa, S. I., Yagi, M., Iwamoto, N., Takata, T., Hamabe, M., et al. 2002, Develop-ment of the Subaru-Mitaka-Okayama-Kiso Archive Sys-tem, ADASS, XI, 281, 298