• 제목/요약/키워드: Data loading

검색결과 2,108건 처리시간 0.032초

아파치 스쿱을 사용한 하둡의 데이터 적재 성능 영향 요인 분석 (Analysis of the Influence Factors of Data Loading Performance Using Apache Sqoop)

  • ;고정현;여정모
    • 정보처리학회논문지:소프트웨어 및 데이터공학
    • /
    • 제4권2호
    • /
    • pp.77-82
    • /
    • 2015
  • 빅데이터 기술은 데이터 처리 속도가 빠르다는 면에서 주목을 받고 있다. 그리고 관계형 데이터베이스(Relational Database: RDB)에 저장되어있는 대용량 정형 데이터를 더 빠르게 처리하기 위해서 빅데이터 기술을 활용하는 연구도 진행되고 있다. 다양한 분산 처리 도구들을 사용하여 분석 성능을 측정하는 연구는 많지만 분석하기 전 단계인 정형 데이터 적재의 성능에 관한 연구는 미미하다. 때문에 본 연구에서는 RDB 안에 저장되어있는 정형 데이터를 아파치 스쿱(Apache Sqoop)을 사용하여 분산 처리 플랫폼 하둡(Hadoop)으로 적재하는 성능을 측정하였다. 그리고 적재에 영향을 미치는 요인을 분석하기 위해 여러 가지 영향 요소를 변경해가면서 반복적으로 실험을 수행하였고 RDB 기반으로 구성된 서버 간의 적재 성능과 비교하였다. 실험 환경에서 아파치 스쿱의 적재 속도가 낮았지만 실제 운영하고 있는 대규모 하둡 클러스터 환경에서는 더 많은 하드웨어 자원이 확보되기 때문에 훨씬 더 좋은 성능을 기대할 수 있다. 이는 향후 진행할 적재 성능 개선 및 하둡 환경에서 정형 데이터를 분석하는 전체적인 단계의 성능을 향상시킬 수 있는 방법에 대한 연구의 기반이 될 것으로 예상한다.

Prediction of Settlement Based on Field Monitoring Data under Preloading Improvement with Ramp Loading

  • Woo, Sang-Inn;Yune, Chan-Young;Chung, Choong-Ki
    • 한국지반공학회:학술대회논문집
    • /
    • 한국지반공학회 2008년도 추계 학술발표회
    • /
    • pp.436-452
    • /
    • 2008
  • In this study, the settlement prediction method based on field monitoring data under preloading improvement with ramp loading is developed. Settlement behavior can be predicted with field monitored settlement throughout the entire preloading process including ramp loading followed by constant loading. The developed method is verified by comparing its predicted results with results from physical model tests and field monitoring data.

  • PDF

광주 도심지역에서 측정한 Aethalometer 검댕입자 자료의 보정 (Compensation of Aethalometer Black Carbon Data Observed at a Gwangju Site)

  • 박승식;정정훈;조성용;김승재
    • 한국대기환경학회지
    • /
    • 제25권6호
    • /
    • pp.571-578
    • /
    • 2009
  • $PM_{2.5}$ black carbon (BC) concentrations were measured to investigate the filter spot loading effect in raw BC data at 5-minute time-based resolution using a single-wavelength aethalometer at a Gwangju site. Also the elemental carbon (EC) concentrations from 24-hr integrated filter-based measurements of $PM_{2.5}$ particles were determined to compare with the loading compensated BC values. Close examination of the time-series BC data showed clearly the "gaps" when the filter tape advances, suggesting the correction of raw BC data. Therefore, we calculated the average BC concentration in each range of attenuation (ATN) to decide if there was (or was not) an effect on the aethalometer data according to the loading of the filter spot. A consistent decrease of average BC concentration was found with increasing ATN values for every month, suggesting there was a consistent "spot loading effect" in the raw BC data. The loading compensated BC concentration according to a simple compensation model with loading effect was 1.01~1.15 times greater than the raw BC data. The 24-hr average concentration of EC observed during summer sampling period was about 3% higher than the original 24-hr average BC value and 2% lower than the loading compensated BC concentration.

우리나라 하수도시설의 첨두부하율 영향요소 분석 (Analysis of Factors Affecting Peak Loading Coefficient of Sewer Works in Korea)

  • 현인환;이영호
    • 상하수도학회지
    • /
    • 제25권6호
    • /
    • pp.877-884
    • /
    • 2011
  • Although peak loading coefficient is one of critical design factors for sewer works, its detailed affecting factors were not analyzed because of limited data availability. This study analyzed the affecting factors on peak loading coefficient with plenty data obtained from several newly constructed sewer works. Simple and multiple regression analysis methods were adopted to analyze the relationships of each variable with or without data filtering. Drainage population, drainage area, population density, and daily sewage flow per person showed very weak relationships under diverse characteristics of cities. However, daily sewage flow per person showed stronger relationships with peak loading when daily sewage flow per person was splitted into two ranges. Population density (i.e., drainage population divided by drainage area) and daily sewage flow per person considerably were related with peak loading coefficient when daily sewage flow per person is less than about 400 Lpcd.

점증 재하를 고려한 선행재하 공법 적용 연약지반의 현장 계측을 통한 침하량 예측 방법의 개발 (Prediction Method of Settlement Based on Field Monitoring Data for Soft Ground Under Preloading Improvement with Ramp Loading)

  • 우상인;윤찬영;백승경;정충기
    • 한국지반공학회:학술대회논문집
    • /
    • 한국지반공학회 2008년도 춘계 학술발표회 초청강연 및 논문집
    • /
    • pp.452-461
    • /
    • 2008
  • Previous settlement prediction method based on settlement monitoring such as hyperbolic, monden method were developed under instantaneous loading condition and have restriction to be applied to soft ground under ramp loading condition. In this study, settlement prediction method under ramp loading was developed. New settlement prediction method under ramp loading considers influence factors of consolidation settlement and increase accuracy of settlement prediction using field monitoring data after ramp loading. Large consolidation tests for ideally controlled one dimensional consolidation under ramp loading condition were performed and the settlement behavior was predicted based on the monitoring data. As a result, new prediction method is expected to have great applicability and practicability for the prediction of settlement behavior.

  • PDF

랜덤하중하의 피로균열진전 데이터를 이용한 ${\Delta}K_{eff}$ 평가법의 정량적 평가 (A Quantitative Evaluation of ${\Delta}K_{eff}$ Estimation Methods Based on Random Loading Crack Growth Data.)

  • 구자석;송지호;강재윤
    • 대한기계학회:학술대회논문집
    • /
    • 대한기계학회 2004년도 추계학술대회
    • /
    • pp.208-213
    • /
    • 2004
  • Methods for estimation of the effective stress intensity factor range (${\Delta}K_{eff}$) are evaluated for narrow and wide band random loading crack growth test data of 2024-T351 aluminum alloy. Three methods of determining $K_{op}$, visual measurement, ASTM offset compliance method, and the neural network method proposed by Kang and Song, and three methods of estimating ${\Delta}K_{eff}$, conventional, the 2/PI0 and 2/PI methods proposed by Donald and Paris, are compared in a quantitative manner by using the results of fatigue crack growth life prediction under random loading. For all $K_{op}$ determination methods discussed, the 2/PI0 and 2/PI methods of estimating ${\Delta}K_{eff}$ provide better results than conventional method for narrow and wide band random loading data.

  • PDF

Assessment of whipping and springing on a large container vessel

  • Barhoumi, Mondher;Storhaug, Gaute
    • International Journal of Naval Architecture and Ocean Engineering
    • /
    • 제6권2호
    • /
    • pp.442-458
    • /
    • 2014
  • Wave induced vibrations increase the fatigue and extreme loading, but this is normally neglected in design. The industry view on this is changing. Wave induced vibrations are often divided into springing and whipping, and their relative contribution to fatigue and extreme loading varies depending on ship design. When it comes to displacement vessels, the contribution from whipping on fatigue and extreme loading is particularly high for certain container vessels. A large modern design container vessel with high bow flare angle and high service speed has been considered. The container vessel was equipped with a hull monitoring system from a recognized supplier of HMON systems. The vessel has been operating between Asia and Europe for a few years and valuable data has been collected. Also model tests have been carried out of this vessel to investigate fatigue and extreme loading, but model tests are often limited to head seas. For the full scale measurements, the correlation between stress data and wind data has been investigated. The wave and vibration damage are shown versus heading and Beaufort strength to indicate general trends. The wind data has also been compared to North Atlantic design environment. Even though it has been shown that the encountered wind data has been much less severe than in North Atlantic, the extreme loading defined by IACS URS11 is significantly exceeded when whipping is included. If whipping may contribute to collapse, then proper seamanship may be useful in order to limit the extreme loading. The vibration damage is also observed to be high from head to beam seas, and even present in stern seas, but fatigue damage in general is low on this East Asia to Europe trade.

A Research on Development of Bills of Material Using Web Grid for Product Lifecycle Management

  • Yoo, Ji-Hyun
    • 한국컴퓨터정보학회논문지
    • /
    • 제22권12호
    • /
    • pp.131-136
    • /
    • 2017
  • PLM(Product Lifecycle Management) is an information management system that can integrate data, processes, business systems and human resources throughout the enterprise. BOM(Bills Of Material) is key data for designing, purchasing materials, manufacturing planning and management, which is basic for product development throughout the product life cycle. In this paper, we propose the efficient system to increase the data loading speed and the processing speed when using such large BOM data. We present the performance and usability of IMDG (In Memory Data Grid) for data processing when loading large amounts of data. In the UI, using the pure web grid of JavaScript instead of the existing data loading method can be improve performance of data managing.

MEAN LOAD EFFECT ON FATIGUE OF WELDED JOINTS USING STRUCTURAL STRESS AND FRACTURE MECHANICS APPROACH

  • Kim, Jong-Sung;Kim, Cheol;Jin, Tae-Eun;Dong, P.
    • Nuclear Engineering and Technology
    • /
    • 제38권3호
    • /
    • pp.277-284
    • /
    • 2006
  • In order to ensure the structural integrity of nuclear welded structures during design life, the fatigue life has to be evaluated by fatigue analysis procedures presented in technical codes such as ASME B&PV Code Section III. However, existing fatigue analysis procedures do not explicitly consider the presence of welded joints. A new fatigue analysis procedure based on a structural stress/fracture mechanics approach has been recently developed in order to reduce conservatism by erasing uncertainty in the analysis procedure. A recent review of fatigue crack growth data under various mean loading conditions using the structural stress/fracture mechanics approach, does not consider the mean loading effect, revealed some significant discrepancies in fatigue crack growth curves according to the mean loading conditions. In this paper, we propose the use of the stress intensity factor range ${\Delta}K$ characterized with loading ratio R effects in terms of the structural stress. We demonstrate the effectiveness in characterizing fatigue crack growth and S-N behavior using the well-known data. It was identified that the S-N data under high mean loading could be consolidated in a master S-N curve for welded joints.

짧은 피로균열의 랜덤하중하의 균열닫힘 및 진전거동(Part II : 진전거동 및 진전수명예측) (Crack Closure and Growth Behavior of Short Fatigue Cracks under Random Loading (Part II : Growth Behavior and Growth Life Prediction))

  • 이신영;송지호
    • 대한기계학회:학술대회논문집
    • /
    • 대한기계학회 2000년도 추계학술대회논문집A
    • /
    • pp.141-146
    • /
    • 2000
  • Crack closure and growth behavior of physically short fatigue cracks under random loading are investigated by performing narrow- and wide-band random loading tests for various stress ratios. The importance of the crack closure phenomenon is examined by predicting the growth lives of short cracks using obtained crack opening behavior. Artificially prepared two-dimensional, short through-thickness cracks are used. The crack opening load of short cracks is much lower under random loading than under constant-amplitude loading corresponding to the largest load cycle in a random load history. This result indicates that the largest load cycle in a random load history has an effect to enhance crack opening of short cracks. Most of the life prediction ratios are within the factor of 2 scatter band except several data at very short crack sizes, indicating that crack growth predictions based on the measured crack opening data are excellent. From the results obtained in this study, it can be concluded that crack closure is the primary factor governing fatigue crack growth of short cracks under random loading as well as under constant-amplitude loading.

  • PDF