ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지
DOI QR코드

DOI QR Code

Distributed memory access architecture and control for fully disaggregated datacenter network

  • Kyeong-Eun, Han (Network Research Division, Electronics and Telecommunications Research Institute) ;
  • Ji Wook, Youn (Network Research Division, Electronics and Telecommunications Research Institute) ;
  • Jongtae, Song (Network Research Division, Electronics and Telecommunications Research Institute) ;
  • Dae-Ub, Kim (Network Research Division, Electronics and Telecommunications Research Institute) ;
  • Joon Ki, Lee (Network Research Division, Electronics and Telecommunications Research Institute)
  • Received : 2021.09.15
  • Accepted : 2022.02.05
  • Published : 2022.12.10
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose novel disaggregated memory module (dMM) architecture and memory access control schemes to solve the collision and contention problems of memory disaggregation, reducing the average memory access time to less than 1 ㎲. In the schemes, the distributed scheduler in each dMM determines the order of memory read/write access based on delay-sensitive priority requests in the disaggregated memory access frame (dMAF). We used the memory-intensive first (MIF) algorithm and priority-based MIF (p-MIF) algorithm that prioritize delay-sensitive and/or memory-intensive (MI) traffic over CPU-intensive (CI) traffic. We evaluated the performance of the proposed schemes through simulation using OPNET and hardware implementation. Our results showed that when the offered load was below 0.7 and the payload of dMAF was 256 bytes, the average round trip time (RTT) was the lowest, ~0.676 ㎲. The dMM scheduling algorithms, MIF and p-MIF, achieved delay less than 1 ㎲ for all MI traffic with less than 10% of transmission overhead.

Keywords

Acknowledgement

We give special thanks to Dr. Kwangjoon Kim for valuable discussions and help in the preparation of the manuscript. This work was supported by the Institute for Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00002, Development of Optical Cloud Networking Core Technology).

References

  1. P. X. Gao, A. Narayan, S. Karandikar, J. Carreira, S. Han, R. Agarwal, S. Ratnasamy, and S. Shenker, Network requirements for resource disaggregation, (OSDI'16: Proceedings of the 12th USENIX conference on Operating Systems Design and Implementation, Berkeley, CA, USA), 2016, pp. 249-264.
  2. Y. Cheng, R. Lin, M. De Andrade, L. Wosinska, and J. Chen, Disaggregated data centers: Challenges and tradeoffs, IEEE Commun. Mag. 58 (2019), no. 2, 20-26.
  3. Y. Cheng, M. De Andrade, L. Wosinska, and J. Chen, Resource disaggregation versus integrated servers in data center: Impact of internal transmission capacity limitation, (European Conference on Optical Communication, Rome, Italy), Sept. 2018. https://doi.org/10.1109/ECOC.2018.8535214
  4. S. Han, N. Egi, A. Panda, S. Ratnasamy, G. Shi, and S. Shenker, Network support for resource disaggregation in nextgeneration datacenters, (HotNets-XII: Proceedings of the Twelfth ACM Workshop on Hot Topics in Networks, New York, NY, USA), Nov. 2013. https://doi.org/10.1145/2535771.2535778
  5. S. Di, D. Kondo, and F. Cappello, Characterizing cloud applications on a Google Data Center, (42nd International Conference on Parallel Processing, Lyon, France), Oct. 2013. https://doi. org/10.1109/ICPP.2013.56
  6. G. Zervas, H. Yuan, A. Saljoghei, Q. Chen, and V. Mishra, Optically disaggregated data centers with minimal remote memory latency: Technologies, architectures, and resource allocation [invited], J. Opt. Commun. Netw. 10 (2018), no. 2, A270-A285. https://doi.org/10.1364/JOCN.10.00A270
  7. A. Klimovic, C. Kozyrakis, E. Thereska, B. John, and S. Kumar, Flash storage disaggregation, (EuroSys '16: Proceedings of the Eleventh European Conference on Computer Systems, New York, NY, USA), Apr. 2016. https://doi.org/10.1145/ 2901318.2901337
  8. M. K. Aguilera, N. Amit, I. Calciu, X. Deguillard, J. Gandhi, P. Subrahmanyam, L. Suresh, K. Tati, R. Venkatasubramanian, and M. Wei, Remote memory in the age of fast network, (SoCC '17: Proceedings of the 2017 Symposium on Cloud Computing, Santa Clara, CA, USA), Sept. 2017. https://doi.org/10.1145/ 3127479.3131612
  9. D. Buragohain, A. Ghogare, T. Patel, M. Vutukuru, and P. Kulkarni, DiME: A performance emulator for disaggregated memory architectures, (APSys '17: Proceedings of the 8th AsiaPacific Workshop on Systems, New York, NY, USA), Sept. 2017. https://doi.org/10.1145/3124680.3124731
  10. C. Li, H. Franke, C. Parris, B. Abali, M. Kesavan, and V. Chang, Composable architecture for rack scale big data computing, Future Gener. Comput. Syst. 67 (2016), 180-193. https://doi.org/10.1016/j.future.2016.07.014
  11. K. Shi, S. Lange, I. Haller, D. Cletheroe, R. Behrendt, B. Thomsen, F. Karinou, K. Jozwik, P. Costa, and H. Ballani, System demonstration of nanosecond wavelength switching with burst-mode PAM4 transceiver, (45th European Conference on Optical Communication, Dublin, Ireland), Sept. 2019. https://doi.org/10.1049/cp.2019.1034
  12. K. Clark, H. Ballani, P. Bayvel, D. Cletheroe, T. Gerard, I. Haller, K. Jozwik, K. Shi, B. Thomsen, P. Watts, and H. Williams, Sub-nanosecond clock and data recovery in an optically-switched data centre network, (European Conference on Optical Communication, Rome, Italy), Sept. 2018. https://doi. org/10.1109/ECOC.2018.8535333
  13. X. Guo, F. Yan, X. Xue, G. Exarchakos, and N. Calabretta, Performance assessment of a novel rack-scale disaggregated data center with fast optical switch, (Optical Fiber Communications Conference and Exhibition, Sand Diego, CA, USA), Mar. 2019.
  14. CCIX Consortium, https://www.ccixconsortium.com, (2019).
  15. J. Weiss, R. Dangel, J. Hofrichter, F. Horst, D. Jubin, N. Meier, A. La Porta, and B. J. Offrein, Optical interconnects for disaggregated resources in future datacenters, (The European Conference on Optical Communication, Cannes, France), Sept. 2014. https://doi.org/10.1109/ECOC.2014.6964255
  16. J. Youn, K. E. Han, H. Lee, J. Song, and J. Lee, Experimental demonstration of optically disaggregated memory for edge data centers, (Opto-Electronics and Communications Conference, Taipei, Taiwan), Oct. 2020. https://doi.org/10.1109/ OECC48412.2020.9273657
  17. N. Terzenidis, M. Moralis-Pegios, G. Mourgias-Alexandris, T. Alexoudi, K. Vyrsokinos, and N. Pleros, High-port and lowlatency optical switches for disaggregated data centers: The Hipoλaos switch architecture, J. Opt. Comm. Netw. 10 (2018), no. 7, B102-B116. https://doi.org/10.1364/JOCN.10.00B102
  18. D. Syrivelis, A. Reale, M. Gazzetti, and P. Hofstee, Dynamic synthesis of disaggregated hardware platforms via cache coherent interconnect optical bridge, (European Conference on Optical Communication, Dublin, Ireland), Sept. 2019, pp. 1-3.
  19. G. Zervas, F. Jiang, Q. Chen, V. Mishra, H. Yuan, K. Katrinis, D. Syrivelis, A. Reale, D. Pnevmatikatos, M. Enrico, and N. Parsons, Disaggregated compute, memory and network systems: A new era for optical data centre architectures, (Optical Fiber Communications Conference and Exhibition, Los Angeles, CA, USA), Mar. 2017.
  20. M. Bielski, I. Syrigos, K. Katrinis, D. Syrivelis, A. Reale, D. Theodoropoulos, N. Alachiotis, D. Pnevmatikatos, E. H. Pap, G. Zervas, and V. Mishra, dReDBox: Materializing a full-stack rack-scale system prototype of a next-generation disaggregated datacenter, (Design, Automation & Test in Europe Conference & Exhibition, Dresden, Germany), Mar. 2018. https://doi.org/10.23919/DATE.2018.8342174
  21. N. Terzenidis, M. Moralis-Pegios, G. Mourgias-Alexandris, K. Vyrsokinos, and N. Pleros, High-port low-latency optical switch architecture with optical feed-forward buffering for 256-node disaggregated data centers, Opt. Express 26 (2018), no. 7, 8756-8766. https://doi.org/10.1364/OE.26.008756
  22. F. Yan, W. Miao, O. Raz, and N. Calabretta, OPSquare: A flat DCN architecture based on flow-controlled optical packet switches, Opt. Commun. Netw. 9 (2019), no. 4, 291-303.
  23. S. McCreary and K. C. Claffy, Trends in wide area IP traffic patterns, Cooperative Association for Internet Data Analysis, USA. Available from: http://www.caida.org [last accessed 2017].