• Title/Summary/Keyword: Map reduce

Search Result 852, Processing Time 0.03 seconds

The Method for Real-time Complex Event Detection of Unstructured Big data (비정형 빅데이터의 실시간 복합 이벤트 탐지를 위한 기법)

  • Lee, Jun Heui;Baek, Sung Ha;Lee, Soon Jo;Bae, Hae Young
    • Spatial Information Research
    • /
    • v.20 no.5
    • /
    • pp.99-109
    • /
    • 2012
  • Recently, due to the growth of social media and spread of smart-phone, the amount of data has considerably increased by full use of SNS (Social Network Service). According to it, the Big Data concept is come up and many researchers are seeking solutions to make the best use of big data. To maximize the creative value of the big data held by many companies, it is required to combine them with existing data. The physical and theoretical storage structures of data sources are so different that a system which can integrate and manage them is needed. In order to process big data, MapReduce is developed as a system which has advantages over processing data fast by distributed processing. However, it is difficult to construct and store a system for all key words. Due to the process of storage and search, it is to some extent difficult to do real-time processing. And it makes extra expenses to process complex event without structure of processing different data. In order to solve this problem, the existing Complex Event Processing System is supposed to be used. When it comes to complex event processing system, it gets data from different sources and combines them with each other to make it possible to do complex event processing that is useful for real-time processing specially in stream data. Nevertheless, unstructured data based on text of SNS and internet articles is managed as text type and there is a need to compare strings every time the query processing should be done. And it results in poor performance. Therefore, we try to make it possible to manage unstructured data and do query process fast in complex event processing system. And we extend the data complex function for giving theoretical schema of string. It is completed by changing the string key word into integer type with filtering which uses keyword set. In addition, by using the Complex Event Processing System and processing stream data at real-time of in-memory, we try to reduce the time of reading the query processing after it is stored in the disk.

Minimizing Estimation Errors of a Wind Velocity Forecasting Technique That Functions as an Early Warning System in the Agricultural Sector (농업기상재해 조기경보시스템의 풍속 예측 기법 개선 연구)

  • Kim, Soo-ock;Park, Joo-Hyeon;Hwang, Kyu-Hong
    • Korean Journal of Agricultural and Forest Meteorology
    • /
    • v.24 no.2
    • /
    • pp.63-77
    • /
    • 2022
  • Our aim was to reduce estimation errors of a wind velocity model used as an early warning system for weather risk management in the agricultural sector. The Rural Development Administration (RDA) agricultural weather observation network's wind velocity data and its corresponding estimated data from January to December 2020 were used to calculate linear regression equations (Y = aX + b). In each linear regression, the wind estimation error at 87 points and eight time slots per day (00:00, 03:00, 06:00, 09.00, 12.00, 15.00, 18.00, and 21:00) is the dependent variable (Y), while the estimated wind velocity is the independent variable (X). When the correlation coefficient exceeded 0.5, the regression equation was used as the wind velocity correction equation. In contrast, when the correlation coefficient was less than 0.5, the mean error (ME) at the corresponding points and time slots was substituted as the correction value instead of the regression equation. To enable the use of wind velocity model at a national scale, a distribution map with a grid resolution of 250 m was created. This objective was achieved b y performing a spatial interpolation with an inverse distance weighted (IDW) technique using the regression coefficients (a and b), the correlation coefficient (R), and the ME values for the 87 points and eight time slots. Interpolated grid values for 13 weather observation points in rural areas were then extracted. The wind velocity estimation errors for 13 points from January to December 2019 were corrected and compared with the system's values. After correction, the mean ME of the wind velocities reduced from 0.68 m/s to 0.45 m/s, while the mean RMSE reduced from 1.30 m/s to 1.05 m/s. In conclusion, the system's wind velocities were overestimated across all time slots; however, after the correction model was applied, the overestimation reduced in all time slots, except for 15:00. The ME and RMSE improved b y 33% and 19.2%, respectively. In our system, the warning for wind damage risk to crops is driven by the daily maximum wind speed derived from the daily mean wind speed obtained eight times per day. This approach is expected to reduce false alarms within the context of strong wind risk, by reducing the overestimation of wind velocities.

A Study of Collaboration between the Census and GIS for Urban Analysis: Modification of Digital Maps and Establishment of Census Tracts (도시분석을 위한 인구주택센서스와 GIS의 연계활용방안 연구: 수치지도의 보완과 센서스트랙의 결정)

  • Koo, Chamun
    • Journal of the Korean Association of Geographic Information Studies
    • /
    • v.2 no.2
    • /
    • pp.27-44
    • /
    • 1999
  • Digital maps produced in Korea are various in scale and include a lot of geographic and attribute data. In this study, it is argued that, to reduce the production cost and the difficulties for renewal, it is necessary to establish the already nationally drawn 1:5,000 scale digital maps as the base maps and simplify them as much as the TIGER files in the U.S. The comprehensive data included in the digital maps in Korea are mostly land use information, which are supposed to be established separately from the digital maps. The land use information system could be maintained and updated cheaply and frequently at the local government level. In response to common needs, the land use information could be imported to GIS and used for analyses. As technologies and societies changes, the Census questions and methodologies should be changed for better uses. Along with GIS, the Census would be developed and processed more reliably and efficiently. Also, it is recommended for Korean government to develop the Census Tract and Block Group system. Current Eup, Myon, Dong as basic units for Census information may not be useful or effective for micro level urban analyses and public service planning activities because of their large population and land areas. It is recommended that optimum population of a Census Tract be 5,000 and a Block Groups 1,500, and one Census Tract includes 1~9 Block Groups. It is recommend that Census Tract and Block Group boundary lines be decided flexibly in light of population, physical features, socio-economic attributes, and tradition. For urban analyses using GIS, socio-economic census data, city government's information such as parcel data and building permit data, survey data, and satellite image data could also be used. The existence of Census Tracts and Block Groups as well as GIS could help for the data and methods to be useful for urban analyses and public service provisions.

  • PDF

Determination of Flood Reduction Alternatives for responding to climate change in Gyeongan Watershed (기후변화 대응을 위한 경안천 유역의 홍수저감 대안 선정)

  • Han, Daegun;Choi, Changhyun;Kim, Duckhwan;Jung, Jaewon;Kim, Jungwook;Kim, Soo Jun
    • Journal of Wetlands Research
    • /
    • v.18 no.2
    • /
    • pp.154-165
    • /
    • 2016
  • Recently, the frequency of extreme rainfall event has increased due to climate change and impermeable area also has increased due to rapid urbanization. Therefore, we ought to prepare countermeasures for flood reduction to reduce the damage. To consider climate change, the frequency based rainfall was calculated according to the aimed period(reference : 1971~2010, Target period I : 2011~2040, Target period II : 2041~2070, Target period III : 2071~2100) and the flood discharge was also calculated by climate change using HEC-HMS model. Also, the flood elevation was calculated by each alternative through HEC-RAS model, setting 5 sizes of drainage pumps and reservoirs respectively. The flood map was constructed using topographical data and flood elevation, and the economic analysis was conducted for reduction of flood damage using Multi dimension - Flood Damage Analysis, MD-FDA. As a result of the analysis on the flood control effect, a head of drainage pump was reduced by 0.06m up to 0.44m while it was reduced by 0.01m up to 1.86m in the case of a detention pond. The flooded area shrunk by up to 32.64% from 0.3% and inundation depth also dropped. As a result of a comparison of the Benefit/Cost index estimated by the economic analysis, detention pond E in period I and pump D in period II and III were deemed appropriate as an alternative for climate change. The results are expected to be used as good practices when implementing the flood control works considering climate change.

Development of Minimal Processing Technology for Korean Fruit and Vegetables (과실 및 채소의 신선편의 식품화 개발기술에 관한 연구)

  • 김건희
    • Korean journal of food and cookery science
    • /
    • v.16 no.6
    • /
    • pp.577-583
    • /
    • 2000
  • The purpose of this study was to investigate the effectiveness of various quality preservative treatments for extending shelf life and maintaining good quality of minimally processed fruit and vegetables produced in Korea. To determine the suitable treatments for delaying quality deterioration, fresh Asian pears and Chinese cabbages were sliced and treated with various quality preservatives (1% CaCl$_2$, 1% NaCl, 3% sucrose, 1% Ca-lactate, 1% vitamin C, 0.05% chitosan +1% vitamin C, 0.1% Sporix+1% vitamin C, hot water (60$\^{C}$), 0.2% L-cysteine), packed with polyethylene film (60㎛-thick), and stored at 4$\^{C}$/0$\^{C}$ or 20$\^{C}$. Various biological and sensory tests were performed to evaluate the quality changes in minimally processed products. Results indicated that Chinese cabbages treated with 1% CaCl$_2$ at 4, and 1% CaCl$_2$ and 1% NaCl at 20$\^{C}$ were most effective in maintaining the quality and minimizing the biochemical changes during storage. For sliced Asian pears, 0.2% L-cysteine and 1% NaCl treatments were effective to reduce browning, and 1% CaCl$_2$ treatment was the most effective to prevent softening during storage at 20$\^{C}$ and 0$\^{C}$. Modified atmosphere packaging of Pleurotus ostreatus and Lentinus edodes had a significantly different shelf life depending on packaging material, packaging thickness and storage temperature. Sealed packaging with polyethylene film (60㎛-thick) for two kinds of mushrooms maintained a good quality with an extended shelf life by 30-50% at 20$\^{C}$ and by 30-130% at 0$\^{C}$. To minimize the quality deterioration which appeared in the condition of polyethylene film packaging, quality preservatives such as KMnO$_4$ and KHSO$_2$+K$_2$S$_2$O$\_$5/ for SO$_2$ generation were added inside of mushroom packaging. The best condition for maintaining good quality longer was packaging with polyethylene film+SO$_2$ which showed 5080% extended shelf life for both Pleurotus ostreatus and Lentinus edodes at 20$\^{C}$ and 0$\^{C}$.

  • PDF

Assessment of Environmental Conservation Function using Changes of Land Use Area and Surface Temperature in Agricultural Field (용인시의 토지이용면적과 지표면 온도 변화를 이용한 환경보전 기능 변동 계량화)

  • Ko, Byong-Gu;Kang, Kee-Kyung;Hong, Suk-Young;Lee, Deog-Bae;Kim, Min-Kyeong;Seo, Myung-Chul;Kim, Gun-Yeob;Park, Kwang-Lai;Lee, Jung-Taek
    • Korean Journal of Environmental Agriculture
    • /
    • v.28 no.1
    • /
    • pp.1-8
    • /
    • 2009
  • This study was aimed at assess environmental conservation functions by analyzing the change of land use areas in agricultural fields between 1999 and 2006, and comparing land surface temperature distribution between 1994 and 2006 in Yongin city. Land use maps of Yongin city were obtained from soil maps for 1999, Quickbird satellite images(less than 1 m) and parcel map for 2006. The land use area for Yongin city was in the order of forest > paddy field > upland > residence & building in 1999, and forest > residence & building > paddy field > upland in 2006. Decrease of paddy and upland fields reduced 34% and 41% of the capability of agricultural multifunctionality as to environment including flood control, groundwater recharge, and air cooling. Land surface temperature(LST) was derived from Landsat TM thermal infrared band acquired in September of 1994 and 2006 and classified into three grades. The results impplied that green vegetation in agricultural field and forest play an important role to reduce land surface temperature in warm season.

GIS based Development of Module and Algorithm for Automatic Catchment Delineation Using Korean Reach File (GIS 기반의 하천망분석도 집수구역 자동 분할을 위한 알고리듬 및 모듈 개발)

  • PARK, Yong-Gil;KIM, Kye-Hyun;YOO, Jae-Hyun
    • Journal of the Korean Association of Geographic Information Studies
    • /
    • v.20 no.4
    • /
    • pp.126-138
    • /
    • 2017
  • Recently, the national interest in environment is increasing and for dealing with water environment-related issues swiftly and accurately, the demand to facilitate the analysis of water environment data using a GIS is growing. To meet such growing demands, a spatial network data-based stream network analysis map(Korean Reach File; KRF) supporting spatial analysis of water environment data was developed and is being provided. However, there is a difficulty in delineating catchment areas, which are the basis of supplying spatial data including relevant information frequently required by the users such as establishing remediation measures against water pollution accidents. Therefore, in this study, the development of a computer program was made. The development process included steps such as designing a delineation method, and developing an algorithm and modules. DEM(Digital Elevation Model) and FDR(Flow Direction) were used as the major data to automatically delineate catchment areas. The algorithm for the delineation of catchment areas was developed through three stages; catchment area grid extraction, boundary point extraction, and boundary line division. Also, an add-in catchment area delineation module, based on ArcGIS from ESRI, was developed in the consideration of productivity and utility of the program. Using the developed program, the catchment areas were delineated and they were compared to the catchment areas currently used by the government. The results showed that the catchment areas were delineated efficiently using the digital elevation data. Especially, in the regions with clear topographical slopes, they were delineated accurately and swiftly. Although in some regions with flat fields of paddles and downtowns or well-organized drainage facilities, the catchment areas were not segmented accurately, the program definitely reduce the processing time to delineate existing catchment areas. In the future, more efforts should be made to enhance current algorithm to facilitate the use of the higher precision of digital elevation data, and furthermore reducing the calculation time for processing large data volume.

On the Improvement of Precision in Gravity Surveying and Correction, and a Dense Bouguer Anomaly in and Around the Korean Peninsula (한반도 일원의 중력측정 및 보정의 정밀화와 고밀도 부우게이상)

  • Shin, Young-Hong;Yang, Chul-Soo;Ok, Soo-Suk;Choi, Kwang-Sun
    • Journal of the Korean earth science society
    • /
    • v.24 no.3
    • /
    • pp.205-215
    • /
    • 2003
  • A precise and dense Bouguer anomaly is one of the most important data to improve the knowledge of our environment in the aspect of geophysics and physical geodesy. Besides the precise absolute gravity station net, we should consider two parts; one is to improve the precision in gravity measurement and correction of it, and the other is the density of measurement both in number and distribution. For the precise positioning, we have tested how we could use the GPS properly in gravity measurement, and deduced that the GPS measurement for 5 minutes would be effective when we used DGPS with two geodetic GPS receivers and the baseline was shorter than 40km. In this case we should use a precise geoid model such as PNU95. By applying this method, we are able to reduce the cost, time, and number of surveyors, furthermore we also get the benefit of improving in quality. Two kind of computer programs were developed to correct crossover errors and to calculate terrain effects more precisely. The repeated measurements on the same stations in gravity surveying are helpful not only to correct the drifts of spring but also to approach the results statistically by applying network adjustment. So we can find out the blunders of various causes easily and also able to estimate the quality of the measurements. The recent developments in computer technology, digital elevation data, and precise positioning also stimulate us to improve the Bouguer anomaly by more precise terrain correction. The gravity data of various sources, such as land gravity data (by Choi, NGI, etc.), marine gravity data (by NORI), Bouguer anomaly map of North Korea, Japanese gravity data, altimetry satellite data, and EGM96 geopotential model, were collected and processed to get a precise and dense Bouguer anomaly in and around the Korean Peninsula.

Creation of Actual CCTV Surveillance Map Using Point Cloud Acquired by Mobile Mapping System (MMS 점군 데이터를 이용한 CCTV의 실질적 감시영역 추출)

  • Choi, Wonjun;Park, Soyeon;Choi, Yoonjo;Hong, Seunghwan;Kim, Namhoon;Sohn, Hong-Gyoo
    • Korean Journal of Remote Sensing
    • /
    • v.37 no.5_3
    • /
    • pp.1361-1371
    • /
    • 2021
  • Among smart city services, the crime and disaster prevention sector accounted for the highest 24% in 2018. The most important platform for providing real-time situation information is CCTV (Closed-Circuit Television). Therefore, it is essential to create the actual CCTV surveillance coverage to maximize the usability of CCTV. However, the amount of CCTV installed in Korea exceeds one million units, including those operated by the local government, and manual identification of CCTV coverage is a time-consuming and inefficient process. This study proposed a method to efficiently construct CCTV's actual surveillance coverage and reduce the time required for the decision-maker to manage the situation. For this purpose, first, the exterior orientation parameters and focal lengths of the pre-installed CCTV cameras, which are difficult to access, were calculated using the point cloud data of the MMS (Mobile Mapping System), and the FOV (Field of View) was calculated accordingly. Second, using the FOV result calculated in the first step, CCTV's actual surveillance coverage area was constructed with 1 m, 2 m, 3 m, 5 m, and 10 m grid interval considering the occluded regions caused by the buildings. As a result of applying our approach to 5 CCTV images located in Uljin-gun, Gyeongsnagbuk-do the average re-projection error was about 9.31 pixels. The coordinate difference between calculated CCTV and location obtained from MMS was about 1.688 m on average. When the grid length was 3 m, the surveillance coverage calculated through our research matched the actual surveillance obtained from visual inspection with a minimum of 70.21% to a maximum of 93.82%.

Requirement Analysis for Agricultural Meteorology Information Service Systems based on the Fourth Industrial Revolution Technologies (4차 산업혁명 기술에 기반한 농업 기상 정보 시스템의 요구도 분석)

  • Kim, Kwang Soo;Yoo, Byoung Hyun;Hyun, Shinwoo;Kang, DaeGyoon
    • Korean Journal of Agricultural and Forest Meteorology
    • /
    • v.21 no.3
    • /
    • pp.175-186
    • /
    • 2019
  • Efforts have been made to introduce the climate smart agriculture (CSA) for adaptation to future climate conditions, which would require collection and management of site specific meteorological data. The objectives of this study were to identify requirements for construction of agricultural meteorology information service system (AMISS) using technologies that lead to the fourth industrial revolution, e.g., internet of things (IoT), artificial intelligence, and cloud computing. The IoT sensors that require low cost and low operating current would be useful to organize wireless sensor network (WSN) for collection and analysis of weather measurement data, which would help assessment of productivity for an agricultural ecosystem. It would be recommended to extend the spatial extent of the WSN to a rural community, which would benefit a greater number of farms. It is preferred to create the big data for agricultural meteorology in order to produce and evaluate the site specific data in rural areas. The digital climate map can be improved using artificial intelligence such as deep neural networks. Furthermore, cloud computing and fog computing would help reduce costs and enhance the user experience of the AMISS. In addition, it would be advantageous to combine environmental data and farm management data, e.g., price data for the produce of interest. It would also be needed to develop a mobile application whose user interface could meet the needs of stakeholders. These fourth industrial revolution technologies would facilitate the development of the AMISS and wide application of the CSA.