• Title/Summary/Keyword: Very large real-time data

Search Result 133, Processing Time 0.031 seconds

Spatial Data Stream Processing Platform for Ubiquitous Application Services (유비쿼터스 응용 서비스를 위한 공간 데이터 스트림 처리 플랫폼)

  • Chung, Weon-Il;Kim, Hwan-Koo
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.11 no.3
    • /
    • pp.906-913
    • /
    • 2010
  • Sensors related to the geographic information are gathering strength as core technologies for various ubiquitous services like u-City project for the new town of the future to provide total information services by the high IT infrastructure. These sensors generate the very large real-time streaming data because these are set up and controlled with wide areas of the geographical distribution. On this, we propose an efficient spatial data stream processing system to support various u-GIS services based on geographic sensors.

Development of an Automatic Generation Methodology for Digital Elevation Models using a Two-Dimensional Digital Map (수치지형도를 이용한 DEM 자동 생성 기법의 개발)

  • Park, Chan-Soo;Lee, Seong-Kyu;Suh, Yong-Cheol
    • Journal of the Korean Association of Geographic Information Studies
    • /
    • v.10 no.3
    • /
    • pp.113-122
    • /
    • 2007
  • The rapid growth of aerial survey and remote sensing technology has enabled the rapid acquisition of very large amounts of geographic data, which should be analyzed using real-time visualization technology. The level of detail(LOD) algorithm is one of the most important elements for realizing real-time visualization. We chose the triangulated irregular network (TIN) method to generate normalized digital elevation model(DEM) data. First, we generated TIN data using contour lines obtained from a two-dimensional(2D) digital map and created a 2D grid array fitting the size of the area. Then, we generated normalized DEM data by calculating the intersection points between the TIN data and the points on the 2D grid array. We used constrained Delaunay triangulation(CDT) and ray-triangle intersection algorithms to calculate the intersection points between the TIN data and the points on the 2D grid array in each step. In addition, we simulated a three-dimensional(3D) terrain model based on normalized DEM data with real-time visualization using a Microsoft Visual C++ 6.0 program in the DirectX API library and a quad-tree LOD algorithm.

  • PDF

Model selection via Bayesian information criterion for divide-and-conquer penalized quantile regression (베이즈 정보 기준을 활용한 분할-정복 벌점화 분위수 회귀)

  • Kang, Jongkyeong;Han, Seokwon;Bang, Sungwan
    • The Korean Journal of Applied Statistics
    • /
    • v.35 no.2
    • /
    • pp.217-227
    • /
    • 2022
  • Quantile regression is widely used in many fields based on the advantage of providing an efficient tool for examining complex information latent in variables. However, modern large-scale and high-dimensional data makes it very difficult to estimate the quantile regression model due to limitations in terms of computation time and storage space. Divide-and-conquer is a technique that divide the entire data into several sub-datasets that are easy to calculate and then reconstruct the estimates of the entire data using only the summary statistics in each sub-datasets. In this paper, we studied on a variable selection method using Bayes information criteria by applying the divide-and-conquer technique to the penalized quantile regression. When the number of sub-datasets is properly selected, the proposed method is efficient in terms of computational speed, providing consistent results in terms of variable selection as long as classical quantile regression estimates calculated with the entire data. The advantages of the proposed method were confirmed through simulation data and real data analysis.

An Index-Based Approach for Subsequence Matching Under Time Warping in Sequence Databases (시퀀스 데이터베이스에서 타임 워핑을 지원하는 효과적인 인덱스 기반 서브시퀀스 매칭)

  • Park, Sang-Hyeon;Kim, Sang-Uk;Jo, Jun-Seo;Lee, Heon-Gil
    • The KIPS Transactions:PartD
    • /
    • v.9D no.2
    • /
    • pp.173-184
    • /
    • 2002
  • This paper discuss an index-based subsequence matching that supports time warping in large sequence databases. Time warping enables finding sequences with similar patterns even when they are of different lengths. In earlier work, Kim et al. suggested an efficient method for whole matching under time warping. This method constructs a multidimensional index on a set of feature vectors, which are invariant to time warping, from data sequences. For filtering at feature space, it also applies a lower-bound function, which consistently underestimates the time warping distance as well as satisfies the triangular inequality. In this paper, we incorporate the prefix-querying approach based on sliding windows into the earlier approach. For indexing, we extract a feature vector from every subsequence inside a sliding window and construct a multidimensional index using a feature vector as indexing attributes. For query processing, we perform a series of index searches using the feature vectors of qualifying query prefixes. Our approach provides effective and scalable subsequence matching even with a large volume of a database. We also prove that our approach does not incur false dismissal. To verify the superiority of our approach, we perform extensive experiments. The results reveal that our approach achieves significant speedup with real-world S&P 500 stock data and with very large synthetic data.

Intertidal DEM Generation Using Satellite Radar Interferometry (인공위성 레이더 간섭기술을 이용한 조간대 지형도 작성에 관한 연구)

  • Park, Jeong-Won;Choi, Jung-Hyun;Lee, Yoon-Kyung;Won, Joong-Sun
    • Korean Journal of Remote Sensing
    • /
    • v.28 no.1
    • /
    • pp.121-128
    • /
    • 2012
  • High resolution intertidal DEM is a basic material for science research like sedimentation/erosion by ocean current, and is invaluable in a monitoring of environmental changes and practical management of coastal wetland. Since the intertidal zone changes rapidly by the inflow of fluvial debris and tide condition, remote sensing is an effective tool for observing large areas in short time. Although radar interferometry is one of the well-known techniques for generating high resolution DEM, conventional repeat-pass interferometry has difficulty on acquiring enough coherence over tidal flat due to the limited exposure time and the rapid changes in surface condition. In order to overcome these constraints, we tested the feasibility of radar interferometry using Cosmo-SkyMed tandem-like one-day data and ERS-ENVISAT cross tandem data with very short revisit period compared to the conventional repeat pass data. Small temporal baseline combined with long perpendicular baseline allowed high coherence over most of the exposed tidal flat surface in both observations. However the interferometric phases acquired from Cosmo-SkyMed data suffer from atmospheric delay and changes in soil moisture contents. The ERS-ENVISAT pair, on the other hand, provides nice phase which agree well with the real topography, because the atmospheric effect in 30-minute gap is almost same to both images so that they are cancelled out in the interferometric process. Thus, the cross interferometry with very small temporal baseline and large perpendicular baseline is one of the most reliable solutions for the intertidal DEM construction which requires very accurate mapping of the elevation.

Impact of High-Resolution Sea Surface Temperatures on the Simulated Wind Resources in the Southeastern Coast of the Korean Peninsula (고해상도 해수면온도자료가 한반도 남동해안 풍력자원 수치모의에 미치는 영향)

  • Lee, Hwa-Woon;Cha, Yeong-Min;Lee, Soon-Hwan;Kim, Dong-Hyeok
    • Journal of Environmental Science International
    • /
    • v.19 no.2
    • /
    • pp.171-184
    • /
    • 2010
  • Accurate simulation of the meteorological field is very important to assess the wind resources. Some researchers showed that sea surface temperature (SST) plays a leading role on the local meterological simulation. New Generation Sea Surface Temperature (NGSST), Operational Sea Surface Temperature and Sea Ice Analysis (OSTIA), and Real-Time Global Sea Surface Temperature (RTG SST) have different spatial distribution near the coast and OSTIA shows the best accuracy compared with buoy data in the southeastern coast of the Korean Peninsula. Those SST products are used to initialize the Weather Research and Forecasting (WRF) Model for November 13-23 2008. The simulation of OSTIA shows better result in comparison with NGSST and RTG SST. NGSST shows a large difference with OSTIA in horizontal and vertical wind fields during the weak synoptic condition, but wind power density shows a large difference during strong synoptic condition. RTG SST shows the similar patterns but smaller the magnitude and the extent.

Public Satisfaction Analysis of Weather Forecast Service by Using Twitter (Twitter를 활용한 기상예보서비스에 대한 사용자들의 만족도 분석)

  • Lee, Ki-Kwang
    • Journal of Korean Society of Industrial and Systems Engineering
    • /
    • v.41 no.2
    • /
    • pp.9-15
    • /
    • 2018
  • This study is intended to investigate that it is possible to analyze the public awareness and satisfaction of the weather forecast service provided by the Korea Meteorological Administration (KMA) through social media data as a way to overcome limitations of the questionnaire-based survey in the previous research. Sentiment analysis and association rule mining were used for Twitter data containing opinions about the weather forecast service. As a result of sentiment analysis, the frequency of negative opinions was very high, about 75%, relative to positive opinions because of the nature of public services. The detailed analysis shows that a large portion of users are dissatisfied with precipitation forecast and that it is needed to analyze the two kinds of error types of the precipitation forecast, namely, 'False alarm' and 'Miss' in more detail. Therefore, association rule mining was performed on negative tweets for each of these error types. As a result, it was found that a considerable number of complaints occurred when preventive actions were useless because the forecast predicting rain had a 'False alarm' error. In addition, this study found that people's dissatisfaction increased when they experienced inconveniences due to either unpredictable high winds and heavy rains in summer or severe cold in winter, which were missed by weather forecast. This study suggests that the analysis of social media data can provide detailed information about forecast users' opinion in almost real time, which is impossible through survey or interview.

Feasibility study on using crowdsourced smartphones to estimate buildings' natural frequencies during earthquakes

  • Ting-Yu Hsu;Yi-Wen Ke;Yo-Ming Hsieh;Chi-Ting Weng
    • Smart Structures and Systems
    • /
    • v.31 no.2
    • /
    • pp.141-154
    • /
    • 2023
  • After an earthquake, information regarding potential damage to buildings close to the epicenter is very important during the initial emergency response. This study proposes the use of crowdsourced measured acceleration response data collected from smartphones located within buildings to perform system identification of building structures during earthquake excitations, and the feasibility of the proposed approach is studied. The principal advantage of using crowdsourced smartphone data is the potential to determine the condition of millions of buildings without incurring hardware, installation, and long-term maintenance costs. This study's goal is to assess the feasibility of identifying the lowest fundamental natural frequencies of buildings without knowing the orientations and precise locations of the crowds' smartphones in advance. Both input-output and output-only identification methods are used to identify the lowest fundamental natural frequencies of numerical finite element models of a real building structure. The effects of time synchronization and the orientation alignment between nearby smartphones on the identification results are discussed, and the proposed approach's performance is verified using large-scale shake table tests of a scaled steel building. The presented results illustrate the potential of using crowdsourced smartphone data with the proposed approach to identify the lowest fundamental natural frequencies of building structures, information that should be valuable in making emergency response decisions.

Comparison of big data image analysis techniques for user curation (사용자 큐레이션을 위한 빅데이터 영상 분석 기법 비교)

  • Lee, Hyoun-Sup;Kim, Jin-Deog
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2021.05a
    • /
    • pp.563-565
    • /
    • 2021
  • The most important feature of the recently increasing content providing service is that the amount of content increase over time is very large. Accordingly, the importance of user curation is increasing, and various techniques are used to implement it. In this paper, among the techniques for video recommendation, the analysis technique using voice data and subtitles and the video comparison technique based on keyframe extraction are compared with the results of implementing and applying the video content of real big data. In addition, through the comparison result, a video content environment to which each analysis technique can be applied is proposed.

  • PDF

Prediction Performance of Ocean Temperature and Salinity in Global Seasonal Forecast System Version 5 (GloSea5) on ARGO Float Data

  • Jieun Wie;Jae-Young Byon;Byung-Kwon Moon
    • Journal of the Korean earth science society
    • /
    • v.45 no.4
    • /
    • pp.327-337
    • /
    • 2024
  • The ocean is linked to long-term climate variability, but there are very few methods to assess the short-term performance of forecast models. This study analyzes the short-term prediction performance regarding ocean temperature and salinity of the Global Seasonal prediction system version 5 (GloSea5). GloSea5 is a historical climate re-creation (2001-2010) performed on the 1st, 9th, 17th, and 25th of each month. It comprises three ensembles. High-resolution hindcasts from the three ensembles were compared with the Array for Real-Time Geostrophic Oceanography (ARGO) float data for the period 2001-2010. The horizontal position was preprocessed to match the ARGO float data and the vertical layer to the GloSea5 data. The root mean square error (RMSE), Brier Score (BS), and Brier Skill Score (BSS) were calculated for short-term forecast periods with a lead-time of 10 days. The results show that sea surface temperature (SST) has a large RMSE in the western boundary current region in Pacific and Atlantic Oceans and Antarctic Circumpolar Current region, and sea surface salinity (SSS) has significant errors in the tropics with high precipitation, with both variables having the largest errors in the Atlantic. SST and SSS had larger errors during the fall for the NINO3.4 region and during the summer for the East Sea. Computing the BS and BSS for ocean temperature and salinity in the NINO3.4 region revealed that forecast skill decreases with increasing lead-time for SST, but not for SSS. The preprocessing of GloSea5 forecasts to match the ARGO float data applied in this study, and the evaluation methods for forecast models using the BS and BSS, could be applied to evaluate other forecast models and/or variables.