Browse > Article
http://dx.doi.org/10.14191/Atmos.2019.29.1.075

Development of Climate & Environment Data System for Big Data from Climate Model Simulations  

Lee, Jae-Hee (Climate Research Division, National Institute of Meteorological Sciences)
Sung, Hyun Min (Climate Research Division, National Institute of Meteorological Sciences)
Won, Sangho (Mirae Climate)
Lee, Johan (Earth System Research Division, National Institute of Meteorological Sciences)
Byu, Young-Hwa (Climate Research Division, National Institute of Meteorological Sciences)
Publication Information
Atmosphere / v.29, no.1, 2019 , pp. 75-86 More about this Journal
Abstract
In this paper, we introduce a novel Climate & Environment Database System (CEDS). The CEDS is developed by the National Institute of Meteorological Sciences (NIMS) to provide easy and efficient user interfaces and storage management of climate model data, so improves work efficiency. In uploading the data/files, the CEDS provides an option to automatically operate the international standard data conversion (CMORization) and the quality assurance (QA) processes for submission of CMIP6 variable data. This option increases the system performance, removes the user mistakes, and increases the level of reliability as it eliminates user operation for the CMORization and QA processes. The uploaded raw files are saved in a NAS storage and the Cassandra database stores the metadata that will be used for efficient data access and storage management. The Metadata is automatically generated when uploading a file, or by the user inputs. With the Metadata, the CEDS supports effective storage management by categorizing data/files. This effective storage management allows easy and fast data access with a higher level of data reliability when requesting with the simple search words by a novice. Moreover, the CEDS supports parallel and distributed computing for increasing overall system performance and balancing the load. This supports the high level of availability as multiple users can use it at the same time with fast system-response. Additionally, it deduplicates redundant data and reduces storage space.
Keywords
CMIP; database; metadata; big data; quality assurance;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Williams, D. N., K. E. Taylor, L. Cinquini, B. Evans, M. Kawamiya, M. Lautenschlager, B. Lawrence, D. Middleton, and E. Contributors, 2011: The Earth System Grid Federation: Software framework supporting CMIP5 data analysis and dissemination. CLIVAR Exchanges, 56, 40-42.
2 Eyring, V., S. Bony, G. A. Meehl, C. A. Senior, B. Stevens, R. J. Stouffer, and K. E. Taylor, 2016: Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geosci. Model Dev., 9, 1937-1958, doi:10.5194/gmd-9-1937-2016.   DOI
3 Gormley, C., and Z. Tong, 2015: Elasticsearch: The Definitive Guide: A Distributed Real-Time Search and Analytics Engine. O'Reilly Media, 724 pp.
4 Guilyardi, E., and Coauthors, 2011: The CMIP5 model and simulation documentation: a new standard for climate modelling metadata. CLIVAR Exchanges, 56, 42-46.
5 Halili, E. H., 2008: Apache JMeter: A practical beginner's guide to automated testing and performance measurement for your websites. Packt Publishing, 141 pp.
6 IPCC, 2013: Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to The Fifth Assessment Report of The Intergovernmental Panel on Climate Change. Stocker, T. F. et al. Eds., Cambridge University Press, 1535 pp.
7 Lakshman, A., and P. Malik, 2010: Cassandra: A decentralized structured storage system. ACM SIGOPS Operating Systems Review, 44, 35-40, doi:10.1145/1773912.1773922.   DOI
8 Meehl, G. A., C. Covey, B. McAvaney, M. Latif, and R. J. Stouffer, 2005: Overview of the coupled model intercomparison project. B. Am. Meteorol. Soc., 86, 89-93.   DOI
9 Rew, R., and G. Davis, 1990: NetCDF: An interface for scientific data access. IEEE Comput. Graph., 10, 76-82.
10 Taylor, K. E., V. Balaji, S. Hankin, M. Juckes, B. Lawrence, and S. Pascoe, 2012: CMIP5 data reference syntax (DRS) and controlled vocabularies, 16 pp [Available online at https://cmip.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf.].