• 제목/요약/키워드: Data Lake

검색결과 453건 처리시간 0.03초

Abyss Storage Cluster 기반의 DataLake Framework의 설계 (Draft Design of DataLake Framework based on Abyss Storage Cluster)

  • 차병래;박선;신병춘;김종원
    • 스마트미디어저널
    • /
    • 제7권1호
    • /
    • pp.9-15
    • /
    • 2018
  • 기관 또는 조직은 비즈니스 시스템의 규모가 커지면서 이들과 관련된 서로 다른 시스템에서 다양한 대량의 데이터들이 생성되고 있다. 이와 같이 비즈니스 환경에서 서로 다른 시스템에서 데이터를 보다 스마트하게 처리하여 효율성을 높일 수 있는 방법이 필요하다. 이를 위한 가장 기본적인 접근 방법 중 하나는 DataLake와 같이 데이터를 정확하게 설명하고 전체 비즈니스에 대한 가장 중요한 데이터를 나타낼 수 있는 단일 도메인 모델을 만드는 것이다. DataLake의 장점을 구현하기 위해서는 다양하게 요구되어진 기능들을 어떤 구조로, 어떻게 작동 할 것인지에 대한 DataLake의 구성 요소들을 정의하는 게 중요하며, DataLake의 구성 요소들에 의해서 데이터 흐름에 따른 라이프 사이클을 갖게 된다. 또한 데이터 획득 시점에서 DataLake로 유입되는 동안 메타 데이터는 데이터 추적 가능성, 데이터 계보 및 라이프 사이클 전반의 데이터 민감도에 기반 한 보안 측면과 함께 캡처 및 관리되어야 하며, 이러한 이유로 Abyss Storage Cluster 기반의 DataLake Framework를 설계하였다.

Abyss Storage Cluster 기반 DataLake Framework의 Connected Data Architecture 개념 설계 및 검증 (Design and Verification of Connected Data Architecture Concept employing DataLake Framework over Abyss Storage Cluster)

  • 차병래;차윤석;박선;신병춘;김종원
    • 스마트미디어저널
    • /
    • 제7권3호
    • /
    • pp.57-63
    • /
    • 2018
  • 조직과 기업의 비즈니스 시스템의 규모가 커지면서 다양한 대량의 데이터들이 생성되는 비즈니스 환경의 변화와 데이터를 보다 스마트하게 처리하여 효율성을 높일 수 있는 방법으로 DataLake와 같이 단일 도메인 모델이 필요한 상황이다. 특히, 자원의 유한성과 공유 경제에 의한 물리적인 분할된 멀티 사이트의 데이터를 논리적인 단일 도메인 모델을 만드는 것은 컴퓨팅 자원의 효율적 운영 측면에서 매우 중요하다. 기존의 Data Lake 프레임워크의 장점을 기반으로 다양한 응용 영역의 멀티 사이트들을 통합 및 데이터 라이프 사이클을 관리하기 위한 Abyss Storage 기반 DataLake 프레임워크의 Connected Data Architecture 개념 (connected data architecture-concept)과 기능들을 정의하고, Connected Data Architecture 개념을 위한 인터페이스 설계 및 인터페이스(Interface) #2와 #3의 유효성 검증을 수행한다.

영천호에서 남조류 발생과 환경요인의 관련성 연구 (A Study on the Relationship between Cyanobacteria and Environmental Factors in Yeongcheon Lake)

  • 이현미;신라영;이정호;박종근
    • 한국물환경학회지
    • /
    • 제35권4호
    • /
    • pp.352-361
    • /
    • 2019
  • The purpose of this study is to analyze the characteristics and correlations of the Yeongcheon Lake in order to reduce the occurrence of harmful cyanobacteria. In this study, we investigated the water quality and phytoplankton of the lake from May to November in 2017. Correlation and data mining analyses were performed to analyze the relationship between the two factors. The water temperature was lowest at the point where the Yeongcheon Lake inflow occurs at Imha Lake. It was highest at the point where the outflow occurs to Angye Lake. The pH was also highest at the outflow point, but in the case of DO, it was highest at the midpoint between the inflow and outflow. The main cyanobacteria that emerged during the study period were Oscillatorialimosa, Microcysti saeruginosa and Aphanizomenon flos-aquae. As a result of correlation analysis, the water temperature, inflow, COD loading, TOC loading at the inflow point of the Yeongcheon Lake were the items that were related to the harmful cyanobacteria. The data mining analysis indicated that the TP loading and harmful cyanobacteria in the inflow point of the Yeongcheon Lake were influential on the detrimental cyanobacteria in the Yeongcheon Lake outflow point. When the TP loading was less than 39.0 kg/day at the inflow site, it was expected that the amount of harmful cyanobacteria could be maintained below 10,000 cells/mL.

SWMM과 WASP5모형을 이용한 간척지 담수호의 수질거동 특성 조사 (Behavior of Water Quality in Freshwater Lake of Tide Reclaimed Area Using SWMM and WASP5 Models)

  • 김선주;김성준;이석호;이준우
    • 한국농공학회지
    • /
    • 제44권2호
    • /
    • pp.148-160
    • /
    • 2002
  • Lake water quality assessment information is useful to anyone involved in lake management, from lakeshore owners to lake associations. 11 provides lake water quality, which can improve how to manage lake resources and how to measure current conditions. It also provides a knowledge base that can be used to protect and restore lakes. SWMM was applied to simulate the discharge and pollutant loads from Boryeong watershed, and WASP5 was applied to analyze the changes of water quality in Boryeong freshwater lake. In each model, the most suitable parameters were calculated through sensitive analysis and some parameters used default data. Simulated in SWMM and measured discharge showed the accuracy of 88.6%. T-N and T-P exceeds the criteria in the simulation of water quality in Boryeong freshwater lake, and control of pollutant loads in the main stream showed the most effective way. Integrated water quality management system was developed to give convenience in the operation of SWMM and WASP5 and data acquisition.

Business Intelligence를 지원하기 위한 Big Data 기반 Data Lake 플랫폼의 선행 연구 (A Leading Study of Data Lake Platform based on Big Data to support Business Intelligence)

  • 이상범
    • 한국컴퓨터정보학회:학술대회논문집
    • /
    • 한국컴퓨터정보학회 2018년도 제57차 동계학술대회논문집 26권1호
    • /
    • pp.31-34
    • /
    • 2018
  • We live in the digital era, and the characteristics of our customers in the digital era are constantly changing. That's why understanding business requirements and converting them to technical requirements is essential, and you have to understand the data model behind the business layout. Moreover, BI(Business Intelligence) is at the crux of revolutionizing enterprise to minimize losses and maximize profits. In this paper, we have described a leading study about the situation of desk-top BI(software product & programming language) in aspect of front-end side and the Data Lake platform based on Big Data by data modeling in aspect of back-end side to support the business intelligence.

  • PDF

Connected Data Architecture 개념의 확장을 통한 AI 서비스 초안 설계 (Draft Design of AI Services through Concept Extension of Connected Data Architecture)

  • 차병래;박선;오수열;김종원
    • 스마트미디어저널
    • /
    • 제7권4호
    • /
    • pp.30-36
    • /
    • 2018
  • DataLake 프레임워크와 같은 단일 도메인 모델은 비즈니스 시스템의 규모 확장과 대량의 다양한 데이터들이 생성되는 빅데이터 환경, 그리고 데이터를 보다 스마트하게 처리하여 효율성 및 활용도를 높일 수 있는 방법으로 주목 받고 있다. 특히, 논리적인 단일 도메인 모델은 컴퓨팅 리소스의 유한함과 공유 경제에 의한 물리적으로 분할된 멀티 사이트의 데이터 처리를 위한 네트워크, 스토리지, 그리고 컴퓨팅 자원의 효율적 운영이 매우 중요하다. 기존의 Data Lake 프레임워크의 장점들을 기반으로 다양한 영역의 멀티 사이트들을 통합 및 데이터의 라이프 사이클을 관리하기 위한 DataLake 프레임워크의 Connected Data Architecture 개념과 기능들의 확장을 통한 다양한 응용 영역에 활용 가능한 CDA 기반 AI 서비스의 초안 설계 및 시나리오를 제안하고자 한다.

금강호물의 새만금호 도입에 따른 금강호 수질변화 분석 (Estimation of Water Quality of Geumgang Lake by Diversion of Geumgang Lake Flow into Saemangeum Lake)

  • 엄명철;이재면
    • 한국물환경학회지
    • /
    • 제22권6호
    • /
    • pp.1045-1051
    • /
    • 2006
  • Geumgang canal is planned to connect Geumgang lake with Saemangeum lake to accelerate desalinization and dilute polluted water to improve water quality in Saemangeum lake. The purpose of this study is to evaluate the impact of water quality on Geumgang lake by diversion of its lake flow to Saemangeum lake. WASP5 model was used to estimate water quality of Geumgang lake. Model calibration and verification were done for water quality data for 2001 and 2002. Water quality concentrations in Geumgang lake were simulated for 4 scenarios, which were considered whether the Geumgang canal is built or not. As a result of simulations, there was little impact on water quality in Geumgang lake, though a little of the Geumgang lake flow diverted to Saemangeum lake. As the Geumgang canal is planned to divert the Geumgang water flow which were discharged into the sea through sluice gates when canal is not built, it is thought that there will be little change by diversion of water flow.

강우유출오염부하를 고려한 호수수질모델링 (Lake Water Quality Modelling Considering Rainfall-Runoff Pollution Loads)

  • 조재현;강성효
    • 환경영향평가
    • /
    • 제18권2호
    • /
    • pp.59-67
    • /
    • 2009
  • Water quality of the Lake Youngrang in the Sokcho City is eutrophic. Jangcheon is the largest inflow source to the lake. Major pollutant sources are stormwater runoff from resort areas and various land uses in the Jangcheon watershed. A storm sewer on the southern end of the lake is also an important pollution source. In this study, water quality modelling for Lake Youngrang was carried out considering the rainfall-runoff pollution loads from the watershed. The rainfall-runoff curves and the rainfall-runoff pollutant load curves were derived from the rainfall-runoff survey data during the recent 4 years. The rainfall-runoff pollution loads and flow from the Jangcheon watershed and the storm sewer were estimated using the two kinds of curves, and they were used as the flow and the boundary data of the WASP model. With the measured water quality data of the year 2005 and 2006, WASP model was calibrated. Non-point pollution control measures such as wet pond and infiltration trench were considered as the alternative for water quality management of the lake. The predicted water quality were compared with those under the present condition, and the improvement effect of the lake water quality were analyzed.

유역과 담수호를 연계한 담수호 수질관리 시스템 개발 및 적용 (Development and Application of Freshwater Lake Water Quality Management System(ELAQUM) through the Linkage of Watershed and Freshwater Lake)

  • 김선주;김성준;김필식
    • 한국농공학회지
    • /
    • 제44권6호
    • /
    • pp.124-136
    • /
    • 2002
  • A freshwater lake water quality management system(FLAQUM) was developed to help regional manager for the water quality of a rural basin. The integrated user interface system FLAQUM written in Visual Basic, includes three subsystems such as a database management system, basin pollutant loads simulation model using SWMM model and freshwater lake water quality simulation model using WASP5 model. Pollutant load simulation model was applied to simulate the discharge and pollutant loading from the watershed, and freshwater lake water quality model was applied to analyze the changes in water quality with respect to watershed pollutant loads, and this model could be used in planning to control watershed pollutant source for water quality management. Database management system was constructed fur all input and output data processing, and it can be used to analyze statistical characteristics using constructed data. Results are displayed both graph and text for convenience of user. The results of FLAQUM application to Boryeong freshwater lake showed that the lake was in eutrophic condition. The major contribution of pollution comes from tributary No.1 and No.4, which have a large number of livestock farms. Therefore, water quality management must be focused on appropriate management of the livestock farming in the two breanchs.

Hydrological Variability of Lake Chad using Satellite Gravimetry, Altimetry and Global Hydrological Models

  • Buma, Willibroad Gabila;Seo, Jae Young;Lee, Sang-IL
    • 한국수자원학회:학술대회논문집
    • /
    • 한국수자원학회 2015년도 학술발표회
    • /
    • pp.467-467
    • /
    • 2015
  • Sustainable water resource management requires the assessment of hydrological variability in response to climate fluctuations and anthropogenic activities. Determining quantitative estimates of water balance and total basin discharge are of utmost importance to understand the variations within a basin. Hard-to-reach areas with few infrastructures, coupled with lengthy administrative procedures makes in-situ data collection and water management processes very difficult and unreliable. In this study, the hydrological behavior of Lake Chad whose extent, extreme climatic and environmental conditions make it difficult to collect field observations was examined. During a 10 year period [January 2003 to December 2013], dataset from space-borne and global hydrological models observations were analyzed. Terrestial water storage (TWS) data retrieved from Gravity Recovery and Climate Experiment (GRACE), lake level variations from Satellite altimetry, water fluxes and soil moisture from Global Land Data Assimilation System (GLDAS) were used for this study. Furthermore, we combined altimetry lake volume with TWS over the lake drainage basin to estimate groundwater and soil moisture variations. This will be validated with groundwater estimates from WaterGAP Global Hydrology Model (WGHM) outputs. TWS showed similar variation patterns Lake water level as expected. The TWS in the basin area is governed by the lake's surface water. As expected, rainfall from GLDAS precedes GRACE TWS with a phase lag of about 1 month. Estimates of groundwater and soil moisture content volume changes derived by combining altimetric Lake Volume with TWS over the drainage basin are ongoing. Results obtained shall be compared with WaterGap Hydrology Model (WGHM) groundwater estimate outputs.

  • PDF