• Title/Summary/Keyword: 대규모 클러스터

Search Result 154, Processing Time 0.026 seconds

Hot Spot and Data Distribution Process in Specific Situation for Cloud Computing (클라우드 컴퓨팅을 위한 특정한 상황에서의 병목현상과 데이터 분산 처리)

  • Oh, Sung-Jun;Hwang, Myung-Jin;Youn, Hee-Young
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2009.11a
    • /
    • pp.95-96
    • /
    • 2009
  • 클라우드 컴퓨팅은 대규모의 분산 시스템을 기반으로 페타바이트급 이상의 처리가 가능해진 최근에 각광받고 있는 기술이다. 대부분 비 대칭 분산 구조로 객체기반 클러스터 파일 시스템으로 이루어져있다. 이런 환경에서 효율성과 신뢰성 등을 고려하여 많은 연구가 진행 중이다. 본 논문에서는 특정한 상황에서 병목현상과 메타 데이터 서버간의 부하분산을 처리함으로써, 한 특징인 효율적인 측면에서 향상이 가능한 기법을 제안하였다.

A study on the method of acquiring IO statistical information for each user task of the KISTI-5 supercomputer (슈퍼컴퓨터 5호기 사용자의 작업별 IO 통계정보 획득 방안에 대한 연구)

  • Kwon, Min-Woo;Yoon, JunWeon;Hong, TaeYoung
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2021.11a
    • /
    • pp.6-8
    • /
    • 2021
  • 슈퍼컴퓨터 5호기 누리온은 8,437대의 계산노드와 33.88PB 규모의 병렬스토리지가 100Gbps의 Omni-Path(OPA) 인터커넥트로 연결되어 있는 초대형 클러스터 시스템이다. 누리온의 계산자원은 PBS 작업스케줄러를 통해 관리되고 있고 병렬 스토리지는 DDN사의 Exascaler Monitoring System(ESMON)을 통해 influxDB에 read/write IO 통계 데이터를 수집하고 있다. 본 논문에서는 PBS의 과금 데이터와 ESMON influxDB의 IO 통계 데이터를 활용하여 사용자의 작업별 IO 통계 정보를 생성하는 방안에 대하여 소개한다.

Optimization and Performance Analysis of Cloud Computing Platform for Distributed Processing of Big Data (대용량 데이터의 분산 처리를 위한 클라우드 컴퓨팅 환경 최적화 및 성능평가)

  • Hong, Seung-Tae;Shin, Young-Sung;Chang, Jae-Woo
    • Spatial Information Research
    • /
    • v.19 no.4
    • /
    • pp.55-71
    • /
    • 2011
  • Recently, interest in cloud computing which provides IT resources as service form in IT field is increasing. As a result, much research has been done on the distributed data processing that store and manage a large amount of data in many servers. Meanwhile, in order to effectively utilize the spatial data which is rapidly increasing day by day with the growth of GIS technology, distributed processing of spatial data using cloud computing is essential. Therefore, in this paper, we review the representative distributed data processing techniques and we analyze the optimization requirements for performance improvement of the distributed processing techniques for a large amount of data. In addition, we uses the Hadoop and we evaluate the performance of the distributed data processing techniques for their optimization requirements.

Detection and Recovery of Failure Node in SAN-based Cluster Shared File System $SANique^{TM}$ (SAN 기반 클러스터 공유 파일 시스템 $SANique^{TM}$의 오류 노드 탐지 및 회복 기법)

  • Lee, Kyu-Woong
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.13 no.12
    • /
    • pp.2609-2617
    • /
    • 2009
  • This paper describes the design overview of shared file system $SANique^{TM}$ and proposes the method for detection of failure node and recovery management algorithm. We also illustrate the characteristics and system architecture of shared file system based on SAN. In order to provide uninterrupted service, the detection and recovery methods are proposed under the all possible system failures and natural disasters. The various kinds of system failures and disasters are characterized and then the detection and recovery method are proposed in each disconnected computing node group.

A Study on Developing the Simulation Model of Micro-Sources (마이크로소스의 EMTDC 시뮬레이션 모델 개발에 관한 연구)

  • Son, Kwang-Myoung
    • Journal of the Korean Institute of Illuminating and Electrical Installation Engineers
    • /
    • v.19 no.8
    • /
    • pp.24-32
    • /
    • 2005
  • Micro-source units having power ratings in thousands of watts can provide power quality with higher reliability and efficiency than the conventional large scale units. Since these units are small and easy to install, they are clustered with loads creating micro-grid services to customer sites such as office buildings, industrial parks and homes. Micro-sources adopt voltage source inverter to ensure the power quality of sensitive loads. This paper deals with the connection of micro-sources into the system grid EMTDC modeling of the grid connected micro-sources at the power frequency range are proposed and the characteristics of the control system parameters are investigated. Simulation results show that the micro-grid system with two micro-sources has good dynamic characteristics.

Support Vector Machine Using Parallel Hyperplane for Reduction of Training Data (트레이닝 데이터 감소를 위한 병렬 평면 기반의 Support Vector Machine)

  • Lee, Tae-Ho;Kim, Min-Woo;Lee, Byung-Jun;Kim, Kyung-Tae;Youn, Hee-Yong
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2019.07a
    • /
    • pp.115-116
    • /
    • 2019
  • SVM (Support Vector Machine)은 견고성으로 인해 다양한 분류 문제에 적용 할 수 있는 효율적인 기계 학습 기술이다. 그러나 훈련 데이터의 수가 증가함에 따라 시간 복잡도가 급격히 증가하므로 대규모 데이터 세트의 경우 SVM이 비실용적이다. 본 논문에서는 SVM을 사용하여 중복 된 학습 데이터를 효율적으로 제거하는 새로운 병렬 평면(Parallel Hyperplane) 기법을 소개한다. 제안 기법에서 PH는 재귀 적으로 형성되는 반면 PH의 외부에 있는 데이터 포인트의 클러스터는 매 반복마다 제거된다. 시뮬레이션 결과 제안 기법은 기존의 클러스터링 기반 감축 기법과 SMO 기법에 비해 학습 시간을 크게 단축시키면서 데이터 축소 없이 분류의 정확성을 높일 수 있음을 확인 하였다.

  • PDF

Face Search Method Based on Face Feature Extraction and Clustering (얼굴 특징 추출 및 클러스터링을 활용한 얼굴 검색 기법)

  • Shin, Junho;Kim, Jong-hwan;Cho, Sukhee;Kim, Junghak;Koh, Yeong Jun
    • Proceedings of the Korean Society of Broadcast Engineers Conference
    • /
    • fall
    • /
    • pp.95-96
    • /
    • 2021
  • 최근 미디어의 발전으로 빠른 속도로 많은 양의 사람들의 얼굴이 포함된 사진, 동영상들이 인터넷에 업로드 되고 있다. 이러한 현상에 맞춰 인공지능을 활용한 얼굴 인식 기술의 놀라운 발전이 있었으나, 대규모 데이터셋에서 임의의 인물을 검색하는 경우에서는 연산량과 저장공간의 부담이 존재한다. 특히, 인터넷에 존재하는 수많은 불법 촬영물에서 피해자를 정확하고 신속하게 검색하기 위해서는 효율적인 얼굴 검색 시스템이 필요하다. 따라서, 본 논문은 얼굴 특징 추출과 클러스터링을 활용하여 방대한 양의 불법 촬영물 셋에서 피해자 동영상을 효율적으로 검색할 수 있는 기법을 제안한다. 불법 촬영물 동영상 검색 실험 환경을 만들기 위해 YouTube Faces [1] 데이터셋으로 유사 동영상 셋을 만들고 이 환경에서 실험을 진행한다. 얼굴 특징 추출 모델은 ResNet100 네트워크를 CosFace 손실함수와 Glint360K 데이터셋으로 학습시킨 모델 [2]을 사용한다. 추출된 얼굴 특징들을 HAC(Hierarchical Agglomerative Clustering) 알고리즘으로 클러스터링 한 후, 클러스터 대푯값을 통해 얼굴 검색 실험을 했을 때의 실험 결과를 분석한다.

  • PDF

A Study on Reliable Multicast Transmission using Recovery Cluster (복구 클러스터를 이용한 신뢰성 있는 멀티캐스트 전송에 관한 연구)

  • Gu, Myeong-Mo;Kim, Bong-Gi
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.20 no.11
    • /
    • pp.1-6
    • /
    • 2019
  • Multicast is an efficient method for real-time transmission in many multimedia applications. It is important to recover lost packets and to manage multicast groups according to the network status in order to improve the reliability of multicast transmissions. In this paper, we propose a method that can efficiently recover lost packets in a large multicast group. In the proposed method, we create a recovery cluster (RC) using a multicast domain (MD) for recovery of lost packets. In the conventional methods, clusters send a request message for lost packets to the senders in order to recover the packets lost from many multicast applications. This increases packet delay time and overhead because of the feedback messages and retransmitted packets. In the proposed method, we improve these problems using the RC, which consists of many MDs (which have overlay multicast senders), and many cluster heads (CHs). We divide the message into blocks, and divide each block into many segments for packet recovery using the CHs. When packet loss occurs, all CHs share the segment information and recover the lost segments at the same time. Simulation results show that the proposed method could improve the packet recovery ratio by about 50% compared to the conventional methods.

Development of Improved Clustering Harmony Search and its Application to Various Optimization Problems (개선 클러스터링 화음탐색법 개발 및 다양한 최적화문제에 적용)

  • Choi, Jiho;Jung, Donghwi;Kim, Joong Hoon
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.19 no.3
    • /
    • pp.630-637
    • /
    • 2018
  • Harmony search (HS) is a recently developed metaheuristic optimization algorithm. HS is inspired by the process of musical improvisation and repeatedly searches for the optimal solution using three operations: random selection, memory recall (or harmony memory consideration), and pitch adjustment. HS has been applied by many researchers in various fields. The increasing complexity of real-world optimization problems has created enormous challenges for the current technique, and improved techniques of optimization algorithms and HS are required. We propose an improved clustering harmony search (ICHS) that uses a clustering technique to group solutions in harmony memory based on their objective function values. The proposed ICHS performs modified harmony memory consideration in which decision variables of solutions in a high-ranked cluster have higher probability of being selected than those in a low-ranked cluster. The ICHS is demonstrated in various optimization problems, including mathematical benchmark functions and water distribution system pipe design problems. The results show that the proposed ICHS outperforms other improved versions of HS.

Growth of Korean Shipbuilding-Allied Industries and Regional Implications (우리 나라 조선관련산업의 성장과 지역적 함의)

  • Woo, Youn-Sub
    • Journal of the Korean association of regional geographers
    • /
    • v.9 no.1
    • /
    • pp.53-65
    • /
    • 2003
  • The aim of this study is to examine the regional features of shipbuilding-allied industries comprehensively. Inaddition, this study would offer groundwork to construct spatial networks of shipbuilding-allied industries. Relatively in short term, large modem shipbuilding firms of South Korea have developed under the national strategic support, around southeast coast region. Although small shipbuilding firms and marine equipment firms have supported regional economy, these firms have scaled down their functions and roles due to the national policy. In the long tenn, Korean shipbuilding-allied industries have limitations in stable development, because they have separated their roles. Consequently it is necessary the balanced development of large, medium, and small shipbuilding firms and linkage system with marine equipment firms for overall development of shipbuilding-allied industries. This study suggests two spatial strategy for shipbuilding-allied industries development. First, we need to construct a shipbuilding specialization area which agglomerates competent parts firms adjacent to large shipyards. Second, in national strategy, a long term strategy is required to utilize extremely synergy and external effects induced from a shipbuilding duster of southeast coast region which systematically combines shipbuilding-allied industries, universities, the government authorities, and research institutions.

  • PDF