• Title/Summary/Keyword: Large Scale Data

Search Result 2,773, Processing Time 0.028 seconds

Analysis of Performance Requirement for Large-Scale InfiniBand-based DVSM System (대용량의 InfiniBand 기반 DVSM 시스템 구현을 위한 성능 요구 분석)

  • Cho, Myeong-Jin;Kim, Seon-Wook
    • The KIPS Transactions:PartA
    • /
    • v.14A no.4
    • /
    • pp.215-226
    • /
    • 2007
  • For past years, many distributed virtual shared-memory(DVSM) systems have been studied in order to develop a low-cost shared memory system with a fast interconnection network. But the DVSM needs a lot of data and control communication between distributed processing nodes in order to provide memory consistency in software, and this communication overhead significantly dominates the overall performance. In general, the communication overhead also increases as the number of processing nodes increase, so communication overhead is a very important performance factor for developing a large-scale DVSM system. In this paper, we study the performance scalability quantitatively and qualitatively for developing a large-scale DVSM system based on the next generation interconnection network, called the InfiniBand. Based on the study, we analyze a performance requirement of the next-coming interconnection network to be used for developing a performance-scalable DVSM system in the future.

Spring Forest-Fire Variability over Korea Associated with Large-Scale Climate Factors (대규모 기후인자와 관련된 우리나라 봄철 산불위험도 변동)

  • Jeong, Ji-Yoon;Woo, Sung-Ho;Son, Rack-Hun;Yoon, Jin-Ho;Jeong, Jee-Hoon;Lee, Suk-Jun;Lee, Byung-Doo
    • Atmosphere
    • /
    • v.28 no.4
    • /
    • pp.457-467
    • /
    • 2018
  • This study investigated the variability of spring (March-May) forest fire risk in Korea for the period 1991~2017 and analyzed its relationship with large-scale climate factors. The Forest Weather Index (FWI) representing the meteorological risk for forest fire occurrences calculated based on observational data and its relationship with large-scale climate factors were analyzed. We performed the empirical orthogonal function (EOF) analysis on the spring FWI. The leading EOF mode of FWI accounting for about 70% of total variability was found to be highly correlated with total number of forest fire occurrences in Korea. The high FWI, forest fire occurrence risk, in Korea, is associated with warmer atmosphere temperature in midwest Eurasia-China-Korea peninsula, cyclonic circulation anomaly in northeastern China-Korea peninsula-northwest pacific, westerly wind anomaly in central China-Korea peninsula, and low humidity in Korea. These are further related with warmer sea surface temperature and enhanced outgoing longwave radiation over Western Pacific, which represents a typical condition for a La $Ni\tilde{n}a$ episode. This suggests that large-scale climate factors over East Asia and ENSO could have a significant influence on the occurrence of spring forest fires in Korea.

Threshold based User-centric Clustering for Cell-free MIMO Network (셀프리 다중안테나 네트워크를 위한 임계값 기반 사용자 중심 클러스터링)

  • Ryu, Jong Yeol;Lee, Woongsup;Ban, Tae-Won
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.26 no.1
    • /
    • pp.114-121
    • /
    • 2022
  • In this paper, we consider a user centric clustering in order to guarantee the performance of the users in cell free multiple-input multiple-output (MIMO) network. In the user centric clustering scheme, by using large scale fading coefficients of the connected access points (APs), each user decides own cluster with the APs having the higher the large scale fading coefficients than threshold value compared to the highest large scale fading coefficient. In the determined user centric clusters, the APs design the beamformers and power allocations in the distributed manner and the APs cooperatively transmit data to users by using beamformers and power allocations. In the simulation results, we verify the performance of user centric clustering in terms of the spectral efficiency and we also find the optimal threshold value in the given configuration.

Testing Gravity with Cosmic Shear Data from the Deep Lens Survey

  • Sabiu, Cristiano G.;Yoon, Mijin;Jee, M. James
    • The Bulletin of The Korean Astronomical Society
    • /
    • v.43 no.1
    • /
    • pp.62.2-62.2
    • /
    • 2018
  • From the gaussian, near scale-invariant density perturbations observed in the CMB to the late time clustering of galaxies, CDM provides a minimal theoretical explanation for a variety of cosmological data. However accepting this explanation, requires that we include within our cosmic ontology a vacuum energy that is ~122 orders of magnitude lower than QM predictions, or alternatively a new scalar field (dark energy) that has negative pressure. Alternatively, modifications to Einstein's General Relativity have been proposed as a model for cosmic acceleration. Recently there have been many works attempting to test for modified gravity using the large scale clustering of galaxies, ISW, cluster abundance, RSD, 21cm observations, and weak lensing. In this work, we compare various modified gravity models using cosmic shear data from the Deep Lens Survey as well as data from CMB, SNe Ia, and BAO. We use the Bayesian Evidence to quantify the comparison robustly, which naturally penalizes complex models with weak data support. In this poster we present our methodology and preliminary constraints on f(R) gravity.

  • PDF

Implementation of AIoT Edge Cluster System via Distributed Deep Learning Pipeline

  • Jeon, Sung-Ho;Lee, Cheol-Gyu;Lee, Jae-Deok;Kim, Bo-Seok;Kim, Joo-Man
    • International journal of advanced smart convergence
    • /
    • v.10 no.4
    • /
    • pp.278-288
    • /
    • 2021
  • Recently, IoT systems are cloud-based, so that continuous and large amounts of data collected from sensor nodes are processed in the data server through the cloud. However, in the centralized configuration of large-scale cloud computing, computational processing must be performed at a physical location where data collection and processing take place, and the need for edge computers to reduce the network load of the cloud system is gradually expanding. In this paper, a cluster system consisting of 6 inexpensive Raspberry Pi boards was constructed to perform fast data processing. And we propose "Kubernetes cluster system(KCS)" for processing large data collection and analysis by model distribution and data pipeline method. To compare the performance of this study, an ensemble model of deep learning was built, and the accuracy, processing performance, and processing time through the proposed KCS system and model distribution were compared and analyzed. As a result, the ensemble model was excellent in accuracy, but the KCS implemented as a data pipeline proved to be superior in processing speed..

Building Hierarchical Bitmap Indices in Space Constrained Environments (저장 공간이 제약된 환경에서 계층적 비트맵 인덱스 생성에 관한 연구)

  • Kim, Jong Wook
    • Journal of Digital Contents Society
    • /
    • v.16 no.1
    • /
    • pp.33-41
    • /
    • 2015
  • Since bitmap indices are useful for OLAP queries over low-cardinality data columns, they are frequently used in data warehouses. In many data warehouse applications, the domain of a column tends to be hierarchical, such as categorical data and geographical data. When the domain of a column is hierarchical, hierarchical bitmap index is able to significantly improve the performance of queries with conditions on that column. This strategy, however, has a limitation in that when a large scale hierarchy is used, building a bimamp for each distinct node leads to a large space overhead. Thus, in this paper, we introduce the way to build hierarchical bitmap index on an attribute whose domain is organized into a large-scale hierarchy in space-constrained environments. Especially, in order to figure out space overhead of hierarchical bitmap indices, we propose the cut-selection strategy which divides the entire hierarchy into two exclusive regions.

Sensor placement selection of SHM using tolerance domain and second order eigenvalue sensitivity

  • He, L.;Zhang, C.W.;Ou, J.P.
    • Smart Structures and Systems
    • /
    • v.2 no.2
    • /
    • pp.189-208
    • /
    • 2006
  • Monitoring large-scale civil engineering structures such as offshore platforms and high-large buildings requires a large number of sensors of different types. Innovative sensor data information technologies are very extremely important for data transmission, storage and retrieval of large volume sensor data generated from large sensor networks. How to obtain the optimal sensor set and placement is more and more concerned by researchers in vibration-based SHM. In this paper, a method of determining the sensor location which aims to extract the dynamic parameter effectively is presented. The method selects the number and place of sensor being installed on or in structure by through the tolerance domain statistical inference algorithm combined with second order sensitivity technology. The method proposal first finds and determines the sub-set sensors from the theoretic measure point derived from analytical model by the statistical tolerance domain procedure under the principle of modal effective independence. The second step is to judge whether the sorted out measured point set has sensitive to the dynamic change of structure by utilizing second order characteristic value sensitivity analysis. A 76-high-building benchmark mode and an offshore platform structure sensor optimal selection are demonstrated and result shows that the method is available and feasible.

The Study on the Digital Orthophoto Generation and Improvement of it's Quality (수치정사영상 제작 및 개선에 관한 연구)

  • 김감래;전호원
    • Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography
    • /
    • v.17 no.2
    • /
    • pp.97-104
    • /
    • 1999
  • Digital elevation models(DEMs) represent an important data base for orthophoto generation The quality of a DEM depends on the geometrical accuracy of the original point or line data. This study analyzes the effects of grid space and scanning resolution in DEM creation with image matching method. The less standard deviation of DEM error was introduced when we adopted small grid space, but no effects in scanning resolution. Based on the bias error analysis of the DEM, we found that the error of a large scale of aerial photograph was bigger than that of a small scale case, and that such error mainly came from the closed area in large scale photographs. In order to reduce the closed area, the experiment has been conducted using multi scale and different overlap of aerial photo images. The result shows that the size of closed area and the shaded area has been dramatically decreased due to the adoption of multi scale aerial images instead of a couple of stereo images.

  • PDF

Analysis of the Influence Factors of Data Loading Performance Using Apache Sqoop (아파치 스쿱을 사용한 하둡의 데이터 적재 성능 영향 요인 분석)

  • Chen, Liu;Ko, Junghyun;Yeo, Jeongmo
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.4 no.2
    • /
    • pp.77-82
    • /
    • 2015
  • Big Data technology has been attracted much attention in aspect of fast data processing. Research of practicing Big Data technology is also ongoing to process large-scale structured data much faster in Relatioinal Database(RDB). Although there are lots of studies about measuring analyzing performance, studies about structured data loading performance, prior step of analyzing, is very rare. Thus, in this study, structured data in RDB is tested the performance that loads distributed processing platform Hadoop using Apache sqoop. Also in order to analyze the influence factors of data loading, it is tested repeatedly with different options of data loading and compared with data loading performance among RDB based servers. Although data loading performance of Apache Sqoop in test environment was low, but in large-scale Hadoop cluster environment we can expect much better performance because of getting more hardware resources. It is expected to be based on study improving data loading performance and whole steps of performance analyzing structured data in Hadoop Platform.

A Study on the Consecutive Renewal of Road and Building Information in the Multi-scale Digital Maps (다축척 수치지도의 도로 및 건물정보 일괄갱신 연구)

  • Park, Kyeong-Sik
    • Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography
    • /
    • v.29 no.1
    • /
    • pp.21-28
    • /
    • 2011
  • In the existing digital map of the Ver.1.0, it is impossible to make a small scale digital map, which is under the 1/5000 scale map, by using the 1/1000 digital map which is the most large scale one. Because of this reason, the existing digital maps are produced into a 1/1000 and a 1/5000 map by means of two different scale aerial photos. The next generation digital map should be successively related to a small scale digital map based on the most large scale digital one. This is so important from the aspects of data share and the consecutive renewal. Ever since the development of the digital map of the Ver. 2.0, the possibility of making a multi-scale consecutive digital map has been presented and the related research has been done again. The most basic thing in the multi-scale digital maps is to decide the criteria of the generalization between the two scales. In this study, I try to formulate the criteria of the generalization required to make the 1/5000 digital map by using the 111000 digital one. In addition, I by to explore the application possibility of the consecutive renewal by carrying out auto-generalization.