• Title/Summary/Keyword: Cluster computing environment

Search Result 109, Processing Time 0.024 seconds

Applying TIPC Protocol for Increasing Network Performance in Hadoop-based Distributed Computing Environment (Hadoop 기반 분산 컴퓨팅 환경에서 네트워크 I/O의 성능개선을 위한 TIPC의 적용과 분석)

  • Yoo, Dae-Hyun;Chung, Sang-Hwa;Kim, Tae-Hun
    • Journal of KIISE:Computer Systems and Theory
    • /
    • v.36 no.5
    • /
    • pp.351-359
    • /
    • 2009
  • Recently with increase of data in the Internet, platform technologies that can process huge data effectively such as Google platform and Hadoop are regarded as worthy of notice. In this kind of platform, there exist network I/O overheads to send task outputs due to the MapReduce operation which is a programming model to support parallel computation in the large cluster system. In this paper, we suggest applying of TIPC (Transparent Inter-Process Communication) protocol for reducing network I/O overheads and increasing network performance in the distributed computing environments. TIPC has a lightweight protocol stack and it spends relatively less CPU time than TCP because of its simple connection establishment and logical addressing. In this paper, we analyze main features of the Hadoop-based distributed computing system, and we build an experimental model which can be used for experiments to compare the performance of various protocols. In the experimental result, TIPC has a higher bandwidth and lower CPU overheads than other protocols.

AutoScale: Adaptive QoS-Aware Container-based Cloud Applications Scheduling Framework

  • Sun, Yao;Meng, Lun;Song, Yunkui
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.13 no.6
    • /
    • pp.2824-2837
    • /
    • 2019
  • Container technologies are widely used in infrastructures to deploy and manage applications in cloud computing environment. As containers are light-weight software, the cluster of cloud applications can easily scale up or down to provide Internet-based services. Container-based applications can well deal with fluctuate workloads by dynamically adjusting physical resources. Current works of scheduling applications often construct applications' performance models with collected historical training data, but these works with static models cannot self-adjust physical resources to meet the dynamic requirements of cloud computing. Thus, we propose a self-adaptive automatic container scheduling framework AutoScale for cloud applications, which uses a feedback-based approach to adjust physical resources by extending, contracting and migrating containers. First, a queue-based performance model for cloud applications is proposed to correlate performance and workloads. Second, a fuzzy Kalman filter is used to adjust the performance model's parameters to accurately predict applications' response time. Third, extension, contraction and migration strategies based on predicted response time are designed to schedule containers at runtime. Furthermore, we have implemented a framework AutoScale with container scheduling strategies. By comparing with current approaches in an experiment environment deployed with typical applications, we observe that AutoScale has advantages in predicting response time, and scheduling containers to guarantee that response time keeps stable in fluctuant workloads.

Wireless Channel Selection Considering Network Characteristics in Cluster-based Sensor Networks (클러스터 기반 센서 네트워크에서의 네트워크 특성 정보를 고려한 무선 채널 선택 기법)

  • Kim, Dae-Young;Kim, BeomSeok;Cho, Jinsung
    • Journal of Internet Computing and Services
    • /
    • v.16 no.2
    • /
    • pp.7-17
    • /
    • 2015
  • To provide scalability, wireless sensor network has cluster-based architecture. Wireless sensor network can be implemented based on the IEEE 802.15.4 which is exploited in 2.4GHz ISM frequency band. Since this frequency band is used for various data communication, network status of wireless sensor networks frequently changes according to wireless environment. Thus, wireless channel selection to avoid reduction of transmission efficiency is required. This paper estimates network status using the information that a cluster-head collects in a cluster. Through objective function with throughput, RSSI level and reliability as input parameters, this paper proposes proper wireless channel selection. Simulation results show that the proposed method maintains transmission efficiency even though network status changes.

A Design of Infrastructure for Control/Monitoring System in the Distributed Computing Environment (분산 컴퓨팅 환경에서의 제어/감시 시스템 개발을 위한 기반 구조 설계)

  • 이원구;박재현
    • 제어로봇시스템학회:학술대회논문집
    • /
    • 2000.10a
    • /
    • pp.547-547
    • /
    • 2000
  • Recently, due to the advance of computer, network and Internet technology, control/monitoring systems are required to process the massive data, At the same time, the software development environment uses more and more component-based methodology. This paper proposes the services for the control /monitoring domain. Especially we define domain-specific interfaces and categories to acquire compatibility between products, and implement architecture for lightweight event service. As it is very important to support compatibility between heterogeneous systems, the proposed system provides modules for the web service and communication protocols based on the XML. And as proposed architecture consists of cluster of servers and Windows 2000's NLB service, it can guarantee more stable operation,

  • PDF

Load Balancing of Heterogeneous Workstation Cluster based on Relative Load Index (상대적 부하 색인을 기반으로 한 이기종 워크스테이션 클러스터의 부하 균형)

  • Ji, Byoung-Jun;Lee, Kwang-Mo
    • Journal of KIISE:Computing Practices and Letters
    • /
    • v.8 no.2
    • /
    • pp.183-194
    • /
    • 2002
  • The clustering environment with heterogeneous workstations provides the cost effectiveness and usability for executing applications in parallel. Load balancing is considered a necessary feature for a cluster of heterogeneous workstations to minimize the turnaround time. Previously, static load balancing that assigns a predetermined weight for the processing capability of each workstation, or dynamic approaches which execute a benchmark program to get relative processing capability of each workstation were proposed. The execution of the benchmark program, which has nothing to do with the application being executed, consumes the computation time and the overall turnaround time is delayed. In this paper, we present efficient methods for task distribution and task migration, based on the relative load index. We designed and implemented a load balancing system for the clustering environment with heterogeneous workstations. Turnaround times of our methods and the round-robin approach, as well as the load balancing method using a benchmark program, were compared. The experimental results show that our methods outperform all the other methods that we compared.

Design and Implementation of HPC Job Management Framework for Computational Scientific Simulation (계산과학 시뮬레이션을 위한 HPC 작업 관리 프레임워크의 설계 및 구현)

  • Yu, Jung-Lok;Kim, Han-Gi;Byun, Hee-Jung
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2016.05a
    • /
    • pp.554-557
    • /
    • 2016
  • Recently, supercomputer has been increasingly adopted as a computing environment for scientific simulation as well as education, healthcare and national defence. Especially, supercomputing system with heterogeneous computing resources is gaining resurgence of interest as a next-generation problem solving environment, allowing theoretical and/or experimental research in various fields to be free of time and spatial limits. However, traditional supercomputing services have only been handled through a simple form of command-line based console, which leads to the critical limit of accessibility and usability of heterogeneous computing resources. To address this problem, in this paper, we provide the design and implementation of web-based HPC (High Performance Computing) job management framework for computational scientific simulation. The proposed framework has highly extensible design principles, providing the abstraction interfaces of job scheduler (as well as bundle scheduler plug-ins for LoadLeveler, Sun Grid Engine, OpenPBS scheduler) in order to easily incorporate the broad spectrum of heterogeneous computing resources such as cluster, computing cloud and grid. We also present the detailed specification of HTTP standard based RESTful endpoints, which manage simulation job's life-cycles such as job creation, submission, control and status monitoring, etc., enabling various 3rd-party applications to be newly created on top of the proposed framework.

  • PDF

Design and implementation of a Shared-Concurrent File System in distributed UNIX environment (분산 UNIX 환경에서 Shared-Concurrent File System의 설계 및 구현)

  • Jang, Si-Ung;Jeong, Gi-Dong
    • The Transactions of the Korea Information Processing Society
    • /
    • v.3 no.3
    • /
    • pp.617-630
    • /
    • 1996
  • In this paper, a shared-concurrent file system (S-CFS) is designed and implemented using conventional disks as disk arrays on a Workstation Cluster which can be used as a small-scale server. Since it is implemented on UNIX operating systems, S_CFS is not only portable and flexible but also efficient in resource usage because it does not require additional I/O nodes. The result of the research shows that on small-scale systems with enough disks, the performance of the concurrent file system on transaction processing applications is bounded by the bottleneck of CPUs computing powers while the performance of the concurrent file system on massive data I/Os is bounded by the time required to copy data between buffers. The concurrent file system,which has been implemented on a Workstation Cluster with 8 disks,shows a throughput of 388 tps in case of transaction processing applications and can provide the bandwidth of 15.8 Mbytes/sec in case of massive data processing applications. Moreover,the concurrent file system has been dsigned to enhance the throughput of applications requirring high performance I/O by controlling the paralleism of the concurrent file system on user's side.

  • PDF

A design of GPU container co-execution framework measuring interference among applications (GPU 컨테이너 동시 실행에 따른 응용의 간섭 측정 프레임워크 설계)

  • Kim, Sejin;Kim, Yoonhee
    • KNOM Review
    • /
    • v.23 no.1
    • /
    • pp.43-50
    • /
    • 2020
  • As General Purpose Graphics Processing Unit (GPGPU) recently plays an essential role in high-performance computing, several cloud service providers offer GPU service. Most cluster orchestration platforms in a cloud environment using containers allocate the integer number of GPU to jobs and do not allow a node shared with other jobs. In this case, resource utilization of a GPU node might be low if a job does not intensively require either many cores or large size of memory in GPU. GPU virtualization brings opportunities to realize kernel concurrency and share resources. However, performance may vary depending on characteristics of applications running concurrently and interference among them due to resource contention on a node. This paper proposes GPU container co-execution framework with multiple server creation and execution based on Kubernetes, container orchestration platform for measuring interference which may be occurred by sharing GPU resources. Performance changes according to scheduling policies were investigated by executing several jobs on GPU. The result shows that optimal scheduling is not possible only considering GPU memory and computing resource usage. Interference caused by co-execution among applications is measured using the framework.

A Resource Clustering Method Considering Weight of Application Characteristic in Hybrid Cloud Environment (하이브리드 클라우드 환경에서의 응용 특성 가중치를 고려한 자원 군집화 기법)

  • Oh, Yoori;Kim, Yoonhee
    • KIISE Transactions on Computing Practices
    • /
    • v.23 no.8
    • /
    • pp.481-486
    • /
    • 2017
  • There are many scientists who want to perform experiments in a cloud environment, and pay-per-use services allow scientists to pay only for cloud services that they need. However, it is difficult for scientists to select a suitable set of resources since those resources are comprised of various characteristics. Therefore, classification is needed to support the effective utilization of cloud resources. Thus, a dynamic resource clustering method is needed to reflect the characteristics of the application that scientists want to execute. This paper proposes a resource clustering analysis method that takes into account the characteristics of an application in a hybrid cloud environment. The resource clustering analysis applies a Self-Organizing Map and K-means algorithm to dynamically cluster similar resources. The results of the experiment indicate that the proposed method can classify a similar resource cluster by reflecting the application characteristics.

Crack Identification Using Evolutionary Algorithms in Parallel Computing Environment (병렬 환경하의 진화 이론을 이용한 결함인식)

  • Sim, Mun-Bo;Seo, Myeong-Won
    • Transactions of the Korean Society of Mechanical Engineers A
    • /
    • v.26 no.9
    • /
    • pp.1806-1813
    • /
    • 2002
  • It is well known that a crack has an important effect on the dynamic behavior of a structure. This effect depends mainly on the location and depth of the crack. To identify the location and depth of a crack in a structure, a classical optimization technique was adopted by previous researchers. That technique overcame the difficulty of finding the intersection point of the superposed contours that correspond to the eigenfrequency caused by the crack presence. However, it is hard to select a trial solution initially for optimization because the defined objective function is heavily multimodal. A method is presented in this paper, which uses continuous evolutionary algorithms(CEAs). CEAs are effective for solving inverse problems and implemented on PC clusters to shorten calculation time. With finite element model of the structure to calculate eigenfrequencies, it is possible to formulate the inverse problem in optimization format. CEAs are used to identify the crack location and depth minimizing the difference from the measured frequencies. We have tried this new idea on a simple beam structure and the results are promising with high parallel efficiency over about 94%.