Browse > Article
http://dx.doi.org/10.3745/JIPS.01.0085

Shared Memory Model over a Switchless PCIe NTB Interconnect Network  

Lim, Seung-Ho (Division of Computer Engineering, Hankuk University of Foreign Studies)
Cha, Kwangho (Division of Supercomputing, Korea Institute of Science and Technology Information)
Publication Information
Journal of Information Processing Systems / v.18, no.1, 2022 , pp. 159-172 More about this Journal
Abstract
The role of the interconnect network, which connects computing nodes to each other, is important in high-performance computing (HPC) systems. In recent years, the peripheral component interconnect express (PCIe) has become a promising interface as an interconnection network for high-performance and cost-effective HPC systems having the features of non-transparent bridge (NTB) technologies. OpenSHMEM is a programming model for distributed shared memory that supports a partitioned global address space (PGAS). Currently, little work has been done to develop the OpenSHMEM library for PCIe-interconnected HPC systems. This paper introduces a prototype implementation of the OpenSHMEM library through a switchless interconnect network using PCIe NTB to provide a PGAS programming model. In particular, multi-interrupt, multi-thread-based data transfer over the OpenSHMEM shared memory model is applied at the implementation level to reduce the latency and increase the throughput of the switchless ring network system. The implemented OpenSHMEM programming model over the PCIe NTB switchless interconnection network provides a feasible, cost-effective HPC system with a PGAS programming model.
Keywords
Interconnect Network; HPC; Multi-Thread; NTB; OpenSHMEM; PCIe;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 Top500.org, "Interconnect family statistics," 2018 [Online]. Available: http://top500.org/statistics/list.
2 Y. W. Kim, Y. Ren, and W. Choi, "Design and implementation of an alternate system interconnect based on PCI express," Journal of the Institute of Electronics and Information Engineers, vol. 52, no. 8, pp. 74-85, 2015.   DOI
3 L. Mohrmann, J. Tongen, M. Friedman, and M. Wetzel, "Creating multicomputer test systems using PCI and PCI Express," in Proceedings of 2009 IEEE International Automatic Testing Conference (AUTOTESTCON), Anaheim, CA, 2009, pp. 7-10.
4 H. Wong, PCI express multi-root switch reconfiguration during system operation," Ph.D. dissertation, Massachusetts Institute of Technology, Cambridge, MA, 2011.
5 K. Kong, "Non-transparent Bridging with IDT 89HPES32NT24G2 PCI Express NTB Switch," 2019 [Online]. Available: https://www.renesas.com/kr/en/document/apn/724-non-transparent-bridging-idt-pes32nt24g2-pcie-switch?language=en.
6 B. Chapman, T. Curtis, S. Pophale, S. Poole, J. Kuehn, C. Koelbel, and L. Smith, "Introducing OpenSHMEM: SHMEM for the PGAS community," in Proceedings of the 4th Conference on Partitioned Global Address Space Programming Model, New York, NY, 2010, pp. 1-3.
7 W. C. C. Tu and T. C. Chiueh, "Seamless fail-over for PCIe switched networks," in Proceedings of the 11th ACM International Systems and Storage Conference, Haifa, Israel, 2018, pp. 101-111).
8 J. R. Hammond, S. Ghosh, and B. M. Chapman, "Implementing OpenSHMEM using MPI-3 one-sided communication," in OpenSHMEM and Related Technologies. Cham, Switzerland: Springer, 2014, pp. 44-58.
9 R. E. Kessler and J. L. Schwarzmeier, "CRAY T3D: a new dimension for Cray Research," in Proceedings of COMPCON Spring Digest of Papers, San Francisco, CA, 1993, pp. 176-182.
10 T. Stitt, "An introduction to the Partitioned Global Address Space (PGAS) programming model," 2009 [Online]. Available: https://cnx.org/contents/gtg1AzdI@7/An-Introduction-to-the-Partitioned-Global-Address-Space-PGAS-Programming-Model.
11 V. Krishnan, "Towards an integrated IO and clustering solution using PCI express," in Proceedings of 2007 IEEE International Conference on Cluster Computing, Austin, TX, 2007, pp. 259-266.
12 S. H. Lim, K. W. Park, and K. Cha, "Developing an OpenSHMEM model over a switchless PCIe non-transparent bridge interface," in Proceedings of 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Rio de Janeiro, Brazil, 2019, pp. 593-602.
13 J. Liu, A. Mamidala, A. Vishnu, and D. K. Panda, "Evaluating infiniband performance with PCI express," IEEE Micro, vol. 25, no. 1, pp. 20-29, 2005.   DOI
14 Broadcom, "PEX 8749," 2011 [Online]. Available: https://www.broadcom.com/products/pcie-switches-bridges/pcie-switches/pex8749.
15 L. Rota, M. Caselle, S. Chilingaryan, A. Kopmann, and M. Weber, "A new DMA PCIe architecture for Gigabyte data transmission," in Proceedings of 2014 19th IEEE-NPSS Real Time Conference, Nara, Japan, 2014, pp. 1-2.
16 M. J. Sullivan, "Intel Xeon Processor C5500/C3500 Series Non-Transparent Bridge," 2010 [Online]. Available: https://www.yumpu.com/en/document/view/34121833/xeon-processor-c5500-c3500-series-non-transparent-bridge-intel.
17 C. Shim, K. H. Cha, and M. Choi, "Design and implementation of initial OpenSHMEM on PCIe NTB based cloud computing," Cluster Computing, vol. 22, no. 1, pp. 1815-1826, 2019.   DOI
18 J. M. Mellor-Crummey and M. L. Scott, "Algorithms for scalable synchronization on shared-memory multiprocessors," ACM Transactions on Computer Systems, vol. 9, no. 1, pp. 21-65, 1991.   DOI
19 PCI-SIG, "Peripheral Component Interconnect Special Interest Group," [Online]. Available: https://pcisig.com/.
20 A. Richter, C. Herber, T. Wild, and A. Herkersdorf, "Resolving performance interference in SR-IOV setups with PCIe Quality-of-Service extensions," in Proceedings of 2016 Euromicro Conference on Digital System Design (DSD), Limassol, Cyprus, 2016, pp. 454-462.
21 P. Onufryk, "PCIe Networked Flash Storage," 2018 [Online]. Available: https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2018/20180809_FNET-301B-1_Onufryk.pdf.
22 S. Pophale, R. Nanjegowda, T. Curtis, B. Chapman, H. Jin, S. Poole, and J. Kuehn, "OpenSHMEM performance and potential: a NPB experimental study," in Proceedings of the 6th Conference on Partitioned Global Address Space Programming Models (PGAS), Santa Barbara, CA, 2012,
23 M. Choi and J. H. Park, "Feasibility and performance analysis of RDMA transfer through PCI Express," Journal of Information Processing Systems, vol. 13, no. 1, pp. 95-103, 2017.   DOI