• Title/Summary/Keyword: Cluster computing environment

Search Result 109, Processing Time 0.026 seconds

A Study on Scalability of Profiling Method Based on Hardware Performance Counter for Optimal Execution of Supercomputer (슈퍼컴퓨터 최적 실행 지원을 위한 하드웨어 성능 카운터 기반 프로파일링 기법의 확장성 연구)

  • Choi, Jieun;Park, Guenchul;Rho, Seungwoo;Park, Chan-Yeol
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.9 no.10
    • /
    • pp.221-230
    • /
    • 2020
  • Supercomputer that shares limited resources to multiple users needs a way to optimize the execution of application. For this, it is useful for system administrators to get prior information and hint about the applications to be executed. In most high-performance computing system operations, system administrators strive to increase system productivity by receiving information about execution duration and resource requirements from users when executing tasks. They are also using profiling techniques that generates the necessary information using statistics such as system usage to increase system utilization. In a previous study, we have proposed a scheduling optimization technique by developing a hardware performance counter-based profiling technique that enables characterization of applications without further understanding of the source code. In this paper, we constructed a profiling testbed cluster to support optimal execution of the supercomputer and experimented with the scalability of the profiling method to analyze application characteristics in the built cluster environment. Also, we experimented that the profiling method can be utilized in actual scheduling optimization with scalability even if the application class is reduced or the number of nodes for profiling is minimized. Even though the number of nodes used for profiling was reduced to 1/4, the execution time of the application increased by 1.08% compared to profiling using all nodes, and the scheduling optimization performance improved by up to 37% compared to sequential execution. In addition, profiling by reducing the size of the problem resulted in a quarter of the cost of collecting profiling data and a performance improvement of up to 35%.

Clustering Analysis by Customer Feature based on SOM for Predicting Purchase Pattern in Recommendation System (추천시스템에서 구매 패턴 예측을 위한 SOM기반 고객 특성에 의한 군집 분석)

  • Cho, Young Sung;Moon, Song Chul;Ryu, Keun Ho
    • Journal of the Korea Society of Computer and Information
    • /
    • v.19 no.2
    • /
    • pp.193-200
    • /
    • 2014
  • Due to the advent of ubiquitous computing environment, it is becoming a part of our common life style. And tremendous information is cumulated rapidly. In these trends, it is becoming a very important technology to find out exact information in a large data to present users. Collaborative filtering is the method based on other users' preferences, can not only reflect exact attributes of user but also still has the problem of sparsity and scalability, though it has been practically used to improve these defects. In this paper, we propose clustering method by user's features based on SOM for predicting purchase pattern in u-Commerce. it is necessary for us to make the cluster with similarity by user's features to be able to reflect attributes of the customer information in order to find the items with same propensity in the cluster rapidly. The proposed makes the task of clustering to apply the variable of featured vector for the user's information and RFM factors based on purchase history data. To verify improved performance of proposing system, we make experiments with dataset collected in a cosmetic internet shopping mall.

A Reservation based Network Resource Provisioning Testbed Using the Integrated Resource Management System (통합자원관리시스템을 이용한 예약 기반의 네트워크 자원 할당 테스트베드 망)

  • Lim, Huhn-Kuk;Moon, Jeong-Hoon;Kong, Jong-Uk;Han, Jang-Soo;Cha, Young-Wook
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.36 no.12B
    • /
    • pp.1450-1458
    • /
    • 2011
  • The HPcN (Hybrid & high Performance Convergence Network) in research networks means environment which can provide both computing resource such as supercomputer, cluster and network resource to application researchers in the field of medical, bio, aerospace and e-science. The most representative research network in Korea, KREONET has been developing following technologies through the HERO (Hybrid Networking project for research oriented infrastructure) from 200S. First, we have constructed and deployed a control plane technology which can provide a connection oriented network dynamically. Second, the integrated resource management system technology has been developing for reservation and allocation of both computing and network resources, whenever users want to utilize them. In this paper, a testbed network is presented, which is possible to reserve and allocate network resource using the integrated resource management system. We reserve network resource through GNSI (Grid Network Service Interface) messages between GRS (Global Resource Scheduler) and NRM (Network Resource Manager) and allocate network resource through GUNI (Grid User Network Interface) messages between the NRM (network resource manager) and routers, based on reservation information provided from a user on the web portal. It is confirmed that GUNI interface messages are delivered from the NRM to each router at the starting of reservation time and traffic is transmitted through LSP allocated by the NRM.

Broadcasting and Caching Schemes for Location-dependent Queries in Urban Areas (도심환경에서 위치의존 질의를 위한 방송과 캐싱 기법)

  • Jung Il-dong;Yu Young-ho;Lee Jong-hwan;Kim Kyongsok
    • Journal of KIISE:Databases
    • /
    • v.32 no.1
    • /
    • pp.56-70
    • /
    • 2005
  • The results of location-dependent queries(LDQ) generally depend on the current locations of query issuers. Many mechanisms, e.g. broadcast scheme, hoarding, or racking policy, have been developed to improve the system peformance and provide better services, which are specialized for LDQs. Considering geographical adjacency of data and characteristics oi target area, caching policy and broadcast scheme affect the overall performance in LDQ. For this reason, we propose both the caching policy and broadcast scheme, which these features are reflected in. Based on the adjacency of data in LDQ, our broadcast scheme use Hilbert curve to cluster data. Moreover, in order to develop the caching policy suitable for LDQ on urban area, we apply the moving distance of a MH(Mobile Host) to our caching policy We evaluate the performance of the caching policy measuring the workload of MHs and the correctness of LDQ results and the performance of the broadcast scheme measuring the average setup-time of MHs in our experiments. Finally, we expect that our caching policy Provides more correct answers when executing LDQ in focal cache and leads significant improvement of the performance of MHs. It also seems quite probable that our broadcast scheme leads improvement of battery life of the MH.

A step-by-step service encryption model based on routing pattern in case of IP spoofing attacks on clustering environment (클러스터링 환경에 대한 IP 스푸핑 공격 발생시 라우팅 패턴에 기반한 단계별 서비스 암호화 모델)

  • Baek, Yong-Jin;Jeong, Won-Chang;Hong, Suk-Won;Park, Jae-Hung
    • The Journal of Korea Institute of Information, Electronics, and Communication Technology
    • /
    • v.10 no.6
    • /
    • pp.580-586
    • /
    • 2017
  • The establishment of big data service environment requires both cloud-based network technology and clustering technology to improve the efficiency of information access. These cloud-based networks and clustering environments can provide variety of valuable information in real-time, which can be an intensive target of attackers attempting illegal access. In particular, attackers attempting IP spoofing can analyze information of mutual trust hosts constituting clustering, and attempt to attack directly to system existing in the cluster. Therefore, it is necessary to detect and respond to illegal attacks quickly, and it is demanded that the security policy is stronger than the security system that is constructed and operated in the existing single system. In this paper, we investigate routing pattern changes and use them as detection information to enable active correspondence and efficient information service in illegal attacks at this network environment. In addition, through the step-by -step encryption based on the routing information generated during the detection process, it is possible to manage the stable service information without frequent disconnection of the information service for resetting.

Utilization of Social Media Analysis using Big Data (빅 데이터를 이용한 소셜 미디어 분석 기법의 활용)

  • Lee, Byoung-Yup;Lim, Jong-Tae;Yoo, Jaesoo
    • The Journal of the Korea Contents Association
    • /
    • v.13 no.2
    • /
    • pp.211-219
    • /
    • 2013
  • The analysis method using Big Data has evolved based on the Big data Management Technology. There are quite a few researching institutions anticipating new era in data analysis using Big Data and IT vendors has been sided with them launching standardized technologies for Big Data management technologies. Big Data is also affected by improvements of IT gadgets IT environment. Foreran by social media, analyzing method of unstructured data is being developed focusing on diversity of analyzing method, anticipation and optimization. In the past, data analyzing methods were confined to the optimization of structured data through data mining, OLAP, statics analysis. This data analysis was solely used for decision making for Chief Officers. In the new era of data analysis, however, are evolutions in various aspects of technologies; the diversity in analyzing method using new paradigm and the new data analysis experts and so forth. In addition, new patterns of data analysis will be found with the development of high performance computing environment and Big Data management techniques. Accordingly, this paper is dedicated to define the possible analyzing method of social media using Big Data. this paper is proposed practical use analysis for social media analysis through data mining analysis methodology.

A Visualization of Traffic Accidents Hotspot along the Road Network (도로 네트워크를 따른 교통사고 핫스팟의 시각화)

  • Cho, Nahye;Jun, Chulmin;Kang, Youngok
    • Journal of Cadastre & Land InformatiX
    • /
    • v.48 no.1
    • /
    • pp.201-213
    • /
    • 2018
  • In recent years, the number of traffic accidents caused by car accidents has been decreasing steadily due to traffic accident prevention activities in Korea. However, the number of accidents in Seoul is higher than that of other regions. Various studies have been conducted to prevent traffic accidents, which are human disasters. In particular, previous studies have performed the spatial analysis of traffic accidents by counting the number of traffic accidents by administrative districts or by estimating the density through kernel density method in order to identify the traffic accident cluster areas. However, since traffic accidents take place along the road, it would be more meaningful to investigate them concentrated on the road network. In this study, traffic accidents were assigned to the nearest road network in two ways and analyzed by hotspot analysis using Getis-Ord Gi* statistics. One of them was investigated with a fixed road link of 10m unit, and the other by computing the average traffic accidents per unit length per road section. As a result by the first method, it was possible to identify the specific road sections where traffic accidents are concentrated. On the other hand, the results by the second method showed that the traffic accident concentrated areas are extensible depending on the characteristic of the road links. The methods proposed here provide different approaches for visualizing the traffic accidents and thus, make it possible to identify those sections clearly that need improvement as for the traffic environment.

Real-Time GPU Task Monitoring and Node List Management Techniques for Container Deployment in a Cluster-Based Container Environment (클러스터 기반 컨테이너 환경에서 실시간 GPU 작업 모니터링 및 컨테이너 배치를 위한 노드 리스트 관리기법)

  • Jihun, Kang;Joon-Min, Gil
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.11 no.11
    • /
    • pp.381-394
    • /
    • 2022
  • Recently, due to the personalization and customization of data, Internet-based services have increased requirements for real-time processing, such as real-time AI inference and data analysis, which must be handled immediately according to the user's situation or requirement. Real-time tasks have a set deadline from the start of each task to the return of the results, and the guarantee of the deadline is directly linked to the quality of the services. However, traditional container systems are limited in operating real-time tasks because they do not provide the ability to allocate and manage deadlines for tasks executed in containers. In addition, tasks such as AI inference and data analysis basically utilize graphical processing units (GPU), which typically have performance impacts on each other because performance isolation is not provided between containers. And the resource usage of the node alone cannot determine the deadline guarantee rate of each container or whether to deploy a new real-time container. In this paper, we propose a monitoring technique for tracking and managing the execution status of deadlines and real-time GPU tasks in containers to support real-time processing of GPU tasks running on containers, and a node list management technique for container placement on appropriate nodes to ensure deadlines. Furthermore, we demonstrate from experiments that the proposed technique has a very small impact on the system.

An Installation and Model Assessment of the UM, U.K. Earth System Model, in a Linux Cluster (U.K. 지구시스템모델 UM의 리눅스 클러스터 설치와 성능 평가)

  • Daeok Youn;Hyunggyu Song;Sungsu Park
    • Journal of the Korean earth science society
    • /
    • v.43 no.6
    • /
    • pp.691-711
    • /
    • 2022
  • The state-of-the-art Earth system model as a virtual Earth is required for studies of current and future climate change or climate crises. This complex numerical model can account for almost all human activities and natural phenomena affecting the atmosphere of Earth. The Unified Model (UM) from the United Kingdom Meteorological Office (UK Met Office) is among the best Earth system models as a scientific tool for studying the atmosphere. However, owing to the expansive numerical integration cost and substantial output size required to maintain the UM, individual research groups have had to rely only on supercomputers. The limitations of computer resources, especially the computer environment being blocked from outside network connections, reduce the efficiency and effectiveness of conducting research using the model, as well as improving the component codes. Therefore, this study has presented detailed guidance for installing a new version of the UM on high-performance parallel computers (Linux clusters) owned by individual researchers, which would help researchers to easily work with the UM. The numerical integration performance of the UM on Linux clusters was also evaluated for two different model resolutions, namely N96L85 (1.875° ×1.25° with 85 vertical levels up to 85 km) and N48L70 (3.75° ×2.5° with 70 vertical levels up to 80 km). The one-month integration times using 256 cores for the AMIP and CMIP simulations of N96L85 resolution were 169 and 205 min, respectively. The one-month integration time for an N48L70 AMIP run using 252 cores was 33 min. Simulated results on 2-m surface temperature and precipitation intensity were compared with ERA5 re-analysis data. The spatial distributions of the simulated results were qualitatively compared to those of ERA5 in terms of spatial distribution, despite the quantitative differences caused by different resolutions and atmosphere-ocean coupling. In conclusion, this study has confirmed that UM can be successfully installed and used in high-performance Linux clusters.