KIPS Transactions on Computer and Communication Systems (정보처리학회논문지:컴퓨터 및 통신 시스템)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Design and Implementation of Initial OpenSHMEM Based on PCI Express

PCI Express 기반 OpenSHMEM 초기 설계 및 구현

  • 주영웅 (충북대학교 정보통신공학부) ;
  • 최민 (충북대학교 정보통신공학부)
  • Received : 2016.09.05
  • Accepted : 2016.12.13
  • Published : 2017.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

PCI Express is a bus technology that connects the processor and the peripheral I/O devices that widely used as an industry standard because it has the characteristics of high-speed, low power. In addition, PCI Express is system interconnect technology such as Ethernet and Infiniband used in high-performance computing and computer cluster. PGAS(partitioned global address space) programming model is often used to implement the one-sided RDMA(remote direct memory access) from multi-host systems, such as computer clusters. In this paper, we design and implement a OpenSHMEM API based on PCI Express maintaining the existing features of OpenSHMEM to implement RDMA based on PCI Express. We perform experiment with implemented OpenSHMEM API through a matrix multiplication example from system which PCs connected with NTB(non-transparent bridge) technology of PCI Express. The PCI Express interconnection network is currently very expensive and is not yet widely available to the general public. Nevertheless, we actually implemented and evaluated a PCI Express based interconnection network on the RDK evaluation board. In addition, we have implemented the OpenSHMEM software stack, which is of great interest recently.

PCI Express는 고속, 저전력 등의 특성으로 인하여 프로세서와 주변 I/O 장치들을 연결하는 업계 표준의 버스 기술이다. PCI Express는 최근 고성능 컴퓨터나 클러스터/클라우드 컴퓨팅 등의 분야에서 시스템 인터커넥션 네트워크로서 그 활용가능성을 검증하고 있는 추세이다. PCI Express가 시스템 인터커넥션 네트워크로서 활용가능하게 된 계기는 PCI Express에 NTB(non-transparent bridge) 기술이 도입되면서부터이다. NTB 기술은 물리적으로 두 PCI Express subsystem을 연결가능하도록 하지만, 필요할 경우 논리적인 격리(isolation)를 제공하는 특징이 있다. 또한, PGAS(partitioned global address space)와 같은 공유 주소 공간(shared address space) 프로그래밍 모델은 최근 멀티코어 프로세서의 보편화로 인하여 병렬컴퓨팅 프레임워크로 각광받고 있다. 따라서, 본 논문에서는 차세대 병렬컴퓨팅 플랫폼을 위하여 PCI Express 환경에서 OpenSHMEM을 구현하기 위한 초기 OpenSHMEM API를 설계 및 구현하였다. 본 연구에서 구현한 15가지 OpenSHMEM API의 정확성을 검증하기 위해서 Github의 openshmem-example 벤치마크의 수행을 통하여 확인하였다. 현재 시중에서는 PCI Express 기반 인터커넥션 네트워크는 가격이 매우 비싸고 아직 일반인이 사용하기 용이하도록 NIC형태로 널리 보급되지 않은 실정이다. 이러한 기술개발 초기단계에서 본 연구는 PCI Express 기반 interconnection network를 RDK(evaluation board) 수준에서 실제로 동작하는 실험환경을 구축하고, 여기에 추가로 최근 각광받는 OpenSHMEM software stack를 자체적으로 구현하였다는 데 의의가 있다.

Keywords

References

  1. Interconnect family statistics of supercomputer top 500 [Internet], https://www.top500.org/statistics/list. [Accessed: August, 28, 2016].
  2. Y. W. Kim, Y. Ren, and W. Choi, "Design and Implementation of an Alternate System Interconnect based on PCI Express," Journal of the Institute of Electronics and Information Engineers, Vol.52, No.8, pp.74-85, Aug., 2015. https://doi.org/10.5573/ieie.2015.52.8.074
  3. V. Krishnan, "Towards an Integrated IO and Clustering Solution using PCI Express," Cluster Computing, 2007 IEEE International Conference on, pp.259-266, Sept., 2007.
  4. Jong Min Lee, Jung Hwa Lee, and Seong Woo Kim, "Implementation of a GPU Cluster System using Inexpensive Graphics Devices," Journal of Korea Multimedia Society, Vol.14, Issue 11, pp. 1458-1466, Nov., 2011. https://doi.org/10.9717/kmms.2011.14.11.1458
  5. J. Dinan, P. Balaji, Jeff R. Hammond, S. Krishnamoorthy, and V. Tipparaju, "Supporting the Global Arrays PGAS Model Using MPI One-Sided Communication," Parallel & Distributed Processing Symposium, 2012 IEEE 26th International, pp.739-750, May, 2012.
  6. J. Breitbart, M. Schmidtobreick, and V. Heuveline, "Evaluation of the Global Address Space Programming interface(GASPI)," Parallel & Distributed Processing Symposium Workshops (IPDPSW), 2014 IEEE International, pp.717-726, May, 2014.
  7. Hao Wang, Sreeram Potluri, Devendar Bureddy, Carlos Rosales, and Dhabaleswar K. Panda, "GPU-Aware MPI on RDMA-Enabled Clusters: Design, Implementation and Evaluation," IEEE Transactions on Parallel and Distributed Systems, pp.2595-2605, Oct., 2014.
  8. Ryan E. Grant, Mohammad J. Rashti, Ahmad Afsahi, and Pavan Balaji, "RDMA Capable iWARP over Datagrams," Parallel & Distributed Processing Symposium (IPDPS), 2011 IEEE International, pp.628-639, May, 2011.
  9. Weihang Jiang, Jiuxing Liu, hyun-Wook Jin, D.K. Panda, W. Gropp, and R. Thakur, "High performance MPI-2 one-sided communication over InfiniBand," Cluster Computing and the Grid, 2004. CCGrid 2004. IEEE International Symposium on, pp.531-538, Apr., 2004.
  10. Yong-Hwan Lee, Do-Suk Kim and Sang Yoon Oh, "QoS and Flow Control Support on PCI Express Interface Architecture," Korea Institute of Information Technology Magazine, pp.45-52, Dec., 2009.
  11. NTB white papers, AVAGO TECHNOLOGIES [Internet], http://www.avagotech.com/support/download-search. [Acce ssed: August, 29, 2016].
  12. OpenSHMEM Specification document [Internet], http://open shmem.org/site/Specification. [Accessed: August, 29, 2016].
  13. IXS600 PCI Express Gen 2 Switch, http://www.dolphinics.com.
  14. Non-transparent Bridging with IDT 89HPES32NT24G2 PCI Express NTB Switch, Application Note AN-724, IDT.