• Title/Summary/Keyword: 분산병렬시스템

Search Result 380, Processing Time 0.031 seconds

Design of InfiniBand RDMA-based Network Structure of Apache Storm (InfiniBand RDMA 기반 Apache Storm의 네트워크 구조 설계)

  • Yang, Seokwoo;Son, Siwoon;Choi, Seong-Yun;Choi, Mi-Jung;Moon, Yang-Sae
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2017.11a
    • /
    • pp.679-681
    • /
    • 2017
  • Apache Storm은 대용량 데이터 스트림을 처리하기 위한 실시간 분산 병렬 처리 프레임워크이며, 이를 사용해 다수의 프로세스 및 스레드를 동시에 동작시킬 수 있다. 하지만, 이러한 멀티 프로세스 및 스레드 환경을 제공하는 Storm은 많은 네트워크 시스템 호출을 수행하고, 이는 잦은 문맥 전환(context switch), 운영체제로의 버퍼 복사, 운영체제 내의 버퍼 복사 등으로 인해 CPU 과부하 문제를 발생시킬 수 있다. 이러한 문제는 고성능 네트워크 장비인 InfiniBand의 IPoIB(IP over InfiniBand) 통신을 사용할 때, InfiniBand가 지원하는 대역폭(bandwidth) 대비 저용량 데이터의 송수신으로 인해 더 잦은 문맥 전환과 버퍼 복사가 발생하여 CPU 과부하 문제가 더욱 심각해진다. 따라서, 본 논문에서는 InfiniBand의 RDMA(Remote Direct Memory Access)를 Storm에 적용하는 설계안을 제시함으로써 CPU 과부하 문제를 해결한다.

Intra Transcoding from DV to MPEG-2 and chrominance format conversion H/W implementation (DV에서 MPEG-2의 인트라 변환 부호화 방식의 연구 및 색차포맷 변환부의 H/W구현)

  • Lee, Sun-Hang;Kim, Don-Yeon
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2001.10a
    • /
    • pp.735-738
    • /
    • 2001
  • 디지털 캠코더에서 이용하는 영상 압축 방식인 DV 부호화방식은 DCT와 가변장 부호화 방식을 이용한다. DV 방식은 하드웨어 복잡도가 낮은 반면 압축된 비트 율이 약 26Mbps로 높은 편이다. 따라서 스튜디오에서 낮은 복잡도로 영상을 부호화 한 후 VOD 시스템에서 이용하기 위하여 MPEG-2로 변환부호화 할 필요가 있다. 이때의 두 압축방식이 DCT를 이용하므로, DCT영역에서 변환부호화 하면 중간과정을 줄일 수 있어서 계산상의 복잡도를 줄일 수 있다. 본 논문에서는 DV방식에서 MPEG-2의 인트라로 변환부호화시, DV방식의 4:1:1 색차포맷을 MPEG-2의 4:2:2 색차 포맷으로 변환할 때 변환영역에 있는 데이터에 미리 계산된 행렬을 곱하여 병렬처리가 가능하게 설계하였다. 또한 MPEG-2 율제어는 중요한 서브 블록의 분산을 완전히 DCT영역에서 계산하여 하드웨어 복잡도를 줄였다. 색차포맷변환부 하드웨어 구현을 위하여 VHDL로 코딩한 후 FPGA-EXPRESS(synopsys), ALTERA MAX-PLUS II를 사용하여 모의실험을 하였다. 각 모듈별로 기능을 검증한 후, FPGA EXPRESS(synopsys)를 사용하여 합성 및 검증을 하였다.

  • PDF

Sim-Hadoop : Leveraging Hadoop Distributed File System and Parallel I/O for Reliable and Efficient N-body Simulations (Sim-Hadoop : 신뢰성 있고 효율적인 N-body 시뮬레이션을 위한 Hadoop 분산 파일 시스템과 병렬 I / O)

  • Awan, Ammar Ahmad;Lee, Sungyoung;Chung, Tae Choong
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2013.05a
    • /
    • pp.476-477
    • /
    • 2013
  • Gadget-2 is a scientific simulation code has been used for many different types of simulations like, Colliding Galaxies, Cluster Formation and the popular Millennium Simulation. The code is parallelized with Message Passing Interface (MPI) and is written in C language. There is also a Java adaptation of the original code written using MPJ Express called Java Gadget. Java Gadget writes a lot of checkpoint data which may or may not use the HDF-5 file format. Since, HDF-5 is MPI-IO compliant, we can use our MPJ-IO library to perform parallel reading and writing of the checkpoint files and improve I/O performance. Additionally, to add reliability to the code execution, we propose the usage of Hadoop Distributed File System (HDFS) for writing the intermediate (checkpoint files) and final data (output files). The current code writes and reads the input, output and checkpoint files sequentially which can easily become bottleneck for large scale simulations. In this paper, we propose Sim-Hadoop, a framework to leverage HDFS and MPJ-IO for improving the I/O performance of Java Gadget code.

Countinuous k-Nearest Neighbor Query Processing Algorithm for Distributed Grid Scheme (분산 그리드 기법을 위한 연속 k-최근접 질의처리 알고리즘)

  • Kim, Young-Chang;Chang, Jae-Woo
    • Journal of Korea Spatial Information System Society
    • /
    • v.11 no.3
    • /
    • pp.9-18
    • /
    • 2009
  • Recently, due to the advanced technologies of mobile devices and wireless communication, there are many studies on telematics and LBS(location-based service) applications. because moving objects usually move on spatial networks, their locations are updated frequently, leading to the degradation of retrieval performance. To manage the frequent updates of moving objects' locations in an efficient way, a new distributed grid scheme, called DS-GRID (distributed S-GRID), and k-NN(k-nearest neighbor) query processing algorithm was proposed[1]. However, the result of k-NN query processing technique may be invalidated as the location of query and moving objects are changed. Therefore, it is necessary to study on continuous k-NN query processing algorithm. In this paper, we propose both MCE-CKNN and MBP(Monitoring in Border Point)-CKNN algorithmss are S-GRID. The MCE-CKNN algorithm splits a query route into sub-routes based on cell and seproves retrieval performance by processing query in parallel way by. In addition, the MBP-CKNN algorithm stores POIs from the border points of each grid cells and seproves retrieval performance by decreasing the number of accesses to the adjacent cells. Finally, it is shown from the performance analysis that our CKNN algorithms achieves 15-53% better retrieval performance than the Kolahdouzan's algorithm.

  • PDF

Spatial Computation on Spark Using GPGPU (GPGPU를 활용한 스파크 기반 공간 연산)

  • Son, Chanseung;Kim, Daehee;Park, Neungsoo
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.5 no.8
    • /
    • pp.181-188
    • /
    • 2016
  • Recently, as the amount of spatial information increases, an interest in the study of spatial information processing has been increased. Spatial database systems extended from the traditional relational database systems are difficult to handle large data sets because of the scalability. SpatialHadoop extended from Hadoop system has a low performance, because spatial computations in SpationHadoop require a lot of write operations of intermediate results to the disk, resulting in the performance degradation. In this paper, Spatial Computation Spark(SC-Spark) is proposed, which is an in-memory based distributed processing framework. SC-Spark is extended from Spark in order to efficiently perform the spatial operation for large-scale data. In addition, SC-Spark based on the GPGPU is developed to improve the performance of the SC-Spark. SC-Spark uses the advantage of the Spark holding intermediate results in the memory. And GPGPU-based SC-Spark can perform spatial operations in parallel using a plurality of processing elements of an GPU. To verify the proposed work, experiments on a single AMD system were performed using SC-Spark and GPGPU-based SC-Spark for Point-in-Polygon and spatial join operation. The experimental results showed that the performance of SC-Spark and GPGPU-based SC-Spark were up-to 8 times faster than SpatialHadoop.

Design and Implementation of MongoDB-based Unstructured Log Processing System over Cloud Computing Environment (클라우드 환경에서 MongoDB 기반의 비정형 로그 처리 시스템 설계 및 구현)

  • Kim, Myoungjin;Han, Seungho;Cui, Yun;Lee, Hanku
    • Journal of Internet Computing and Services
    • /
    • v.14 no.6
    • /
    • pp.71-84
    • /
    • 2013
  • Log data, which record the multitude of information created when operating computer systems, are utilized in many processes, from carrying out computer system inspection and process optimization to providing customized user optimization. In this paper, we propose a MongoDB-based unstructured log processing system in a cloud environment for processing the massive amount of log data of banks. Most of the log data generated during banking operations come from handling a client's business. Therefore, in order to gather, store, categorize, and analyze the log data generated while processing the client's business, a separate log data processing system needs to be established. However, the realization of flexible storage expansion functions for processing a massive amount of unstructured log data and executing a considerable number of functions to categorize and analyze the stored unstructured log data is difficult in existing computer environments. Thus, in this study, we use cloud computing technology to realize a cloud-based log data processing system for processing unstructured log data that are difficult to process using the existing computing infrastructure's analysis tools and management system. The proposed system uses the IaaS (Infrastructure as a Service) cloud environment to provide a flexible expansion of computing resources and includes the ability to flexibly expand resources such as storage space and memory under conditions such as extended storage or rapid increase in log data. Moreover, to overcome the processing limits of the existing analysis tool when a real-time analysis of the aggregated unstructured log data is required, the proposed system includes a Hadoop-based analysis module for quick and reliable parallel-distributed processing of the massive amount of log data. Furthermore, because the HDFS (Hadoop Distributed File System) stores data by generating copies of the block units of the aggregated log data, the proposed system offers automatic restore functions for the system to continually operate after it recovers from a malfunction. Finally, by establishing a distributed database using the NoSQL-based Mongo DB, the proposed system provides methods of effectively processing unstructured log data. Relational databases such as the MySQL databases have complex schemas that are inappropriate for processing unstructured log data. Further, strict schemas like those of relational databases cannot expand nodes in the case wherein the stored data are distributed to various nodes when the amount of data rapidly increases. NoSQL does not provide the complex computations that relational databases may provide but can easily expand the database through node dispersion when the amount of data increases rapidly; it is a non-relational database with an appropriate structure for processing unstructured data. The data models of the NoSQL are usually classified as Key-Value, column-oriented, and document-oriented types. Of these, the representative document-oriented data model, MongoDB, which has a free schema structure, is used in the proposed system. MongoDB is introduced to the proposed system because it makes it easy to process unstructured log data through a flexible schema structure, facilitates flexible node expansion when the amount of data is rapidly increasing, and provides an Auto-Sharding function that automatically expands storage. The proposed system is composed of a log collector module, a log graph generator module, a MongoDB module, a Hadoop-based analysis module, and a MySQL module. When the log data generated over the entire client business process of each bank are sent to the cloud server, the log collector module collects and classifies data according to the type of log data and distributes it to the MongoDB module and the MySQL module. The log graph generator module generates the results of the log analysis of the MongoDB module, Hadoop-based analysis module, and the MySQL module per analysis time and type of the aggregated log data, and provides them to the user through a web interface. Log data that require a real-time log data analysis are stored in the MySQL module and provided real-time by the log graph generator module. The aggregated log data per unit time are stored in the MongoDB module and plotted in a graph according to the user's various analysis conditions. The aggregated log data in the MongoDB module are parallel-distributed and processed by the Hadoop-based analysis module. A comparative evaluation is carried out against a log data processing system that uses only MySQL for inserting log data and estimating query performance; this evaluation proves the proposed system's superiority. Moreover, an optimal chunk size is confirmed through the log data insert performance evaluation of MongoDB for various chunk sizes.

Performance Improvement for PVM by Zero-copy Mechanism (Zero-copy 기술을 이용한 PVM의 성능 개선)

  • 임성택;심재홍;최경희;정기현;김재훈;문성근
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.25 no.5B
    • /
    • pp.899-912
    • /
    • 2000
  • PVM provides users with a single image of high performance parallel computing machine by collecting machines distributed over a network. Low communication overhead is essential to effectively run applications on PVM based platforms. In the original PVM, three times of memory copies are required for a PVM task to send a message to a remote task, which results in performance degradation. We propose a zero-copy model using global shared memory that can be accessed by PVM tasks, PVM daemon, and network interface card(NIC). In the scheme, a task packs data into global shared memory, and notify daemon that the data is ready to be sent, then daemon routes the data to a remote task to which it is sent with no virtual data copy overhead. Experimental result reveals that the message round trip time between two machines is reduced significantly in the proposed zero-copy scheme.

  • PDF

A Study on implementation model for security log analysis system using Big Data platform (빅데이터 플랫폼을 이용한 보안로그 분석 시스템 구현 모델 연구)

  • Han, Ki-Hyoung;Jeong, Hyung-Jong;Lee, Doog-Sik;Chae, Myung-Hui;Yoon, Cheol-Hee;Noh, Kyoo-Sung
    • Journal of Digital Convergence
    • /
    • v.12 no.8
    • /
    • pp.351-359
    • /
    • 2014
  • The log data generated by security equipment have been synthetically analyzed on the ESM(Enterprise Security Management) base so far, but due to its limitations of the capacity and processing performance, it is not suited for big data processing. Therefore the another way of technology on the big data platform is necessary. Big Data platform can achieve a large amount of data collection, storage, processing, retrieval, analysis, and visualization by using Hadoop Ecosystem. Currently ESM technology has developed in the way of SIEM (Security Information & Event Management) technology, and to implement security technology in SIEM way, Big Data platform technology is essential that can handle large log data which occurs in the current security devices. In this paper, we have a big data platform Hadoop Ecosystem technology for analyzing the security log for sure how to implement the system model is studied.

Design and Implementation of KDSM(KAIST Distributed Shared Memory) System (KDSM(KAIST Distributed Shared Memory) 시스템의 설계 및 구현)

  • Lee, Sang-Kwon;Yun, Hee-Chul;Lee, Joon-Won;Maeng, Seung-Ryoul
    • Journal of KIISE:Computer Systems and Theory
    • /
    • v.29 no.5
    • /
    • pp.257-264
    • /
    • 2002
  • In this paper, we give a detailed description of KDSM(KAIST Distributed Shared Memory) system. KDSM is implemented as a user-level library running on Linux 2.2.13, and TCP/IP is used for communication. KDSM uses page-based invalidation protocol, multiple-writer protocol, and supports HLRC(Home-based Lazy Release Consistency) memory consistency model. To evaluate performance of KDSM, we executed 4 scientific applications and compared the result to JLAJLA. The results showed that performance of KDSM almost equal to JIAJIA for 2 applications and performance of KDSM is better than JIAJIA for 2 applications.

The Study of the Object Replication Management using Adaptive Duplication Object Algorithm (적응적 중복 객체 알고리즘을 이용한 객체 복제본 관리 연구)

  • 박종선;장용철;오수열
    • Journal of the Korea Society of Computer and Information
    • /
    • v.8 no.1
    • /
    • pp.51-59
    • /
    • 2003
  • It is effective to be located in the double nodes in the distributed object replication systems, then object which nodes share is the same contents. The nodes store an access information on their local cache as it access to the system. and then the nodes fetch and use it, when it needed. But with time the coherence Problems will happen because a data carl be updated by other nodes. So keeping the coherence of the system we need a mechanism that we managed the to improve to improve the performance and availability of the system effectively. In this paper to keep coherence in the shared memory condition, we can set the limited parallel performance without the additional cost except the coherence cost using it to keep the object at the proposed adaptive duplication object(ADO) algorithms. Also to minimize the coherence maintenance cost which is the bi99est overhead in the duplication method, we must manage the object effectively for the number of replication and location of the object replica which is the most important points, and then it determines the cos. And that we must study the adaptive duplication object management mechanism which will improve the entire run time.

  • PDF