Journal of KIISE (정보과학회 논문지)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지
DOI QR코드

DOI QR Code

Dynamic Memory Allocation for Scientific Workflows in Containers

컨테이너 환경에서의 과학 워크플로우를 위한 동적 메모리 할당

  • Received : 2016.07.23
  • Accepted : 2017.02.11
  • Published : 2017.05.15
⟨ Previous Next ⟩

Abstract

The workloads of large high-performance computing (HPC) scientific applications are steadily becoming "bursty" due to variable resource demands throughout their execution life-cycles. However, the over-provisioning of virtual resources for optimal performance during execution remains a key challenge in the scheduling of scientific HPC applications. While over-provisioning of virtual resources guarantees peak performance of scientific application in virtualized environments, it results in increased amounts of idle resources that are unavailable for use by other applications. Herein, we proposed a memory resource reconfiguration approach that allows the quick release of idle memory resources for new applications in OS-level virtualized systems, based on the applications resource-usage pattern profile data. We deployed a scientific workflow application in Docker, a light-weight OS-level virtualized system. In the proposed approach, memory allocation is fine-tuned to containers at each stage of the workflows execution life-cycle. Thus, overall memory resource utilization is improved.

대규모 HPC 과학 응용의 워크로드가 전체 실행시간 동안 다양하게 변화하는 자원 요구사항을 갖게 되면서 특정 시점에 갑자기 요구사항이 증가하는(bursty) 형태가 되고 있다. 그러나 이러한 응용 워크로드를 고려하지 않고, 최대 자원 요구사항만을 반영한 가상 자원의 오버-프로비저닝은 과학 응용의 성능을 보장하지만 다른 응용이 사용할 수 없는 유휴 자원을 늘리는 문제로 남아있다. 본 논문에서는 OS-level 가상화 환경에서 응용의 자원 사용 패턴에 대한 프로파일링 데이터를 기반으로 메모리 자원 재구성 기법을 제안한다. 이는 유휴 상태의 메모리 자원을 신속하게 풀어주어 새로운 응용이 자원을 사용하여 수행할 수 있도록 한다. 본 연구에서는 경량화된 OS-level 가상화 시스템의 하나인 Docker에서 과학 워크플로우 응용을 이용하여 제안하는 알고리즘을 검증하였다. 실험을 통해 과학 응용을 실행하는 동안 컨테이너에 대한 메모리 할당 미세 조정이 전반적인 메모리 자원 활용을 향상시킬 수 있음을 보였다. 또한 응용의 메모리 사용 프로파일 데이터를 기반으로 하는 시뮬레이션 실험을 통해, 제안하는 동적 메모리 할당 기법을 사용하는 경우 대기 작업에 유휴상태의 메모리를 할당하여 전체 대기 작업의 수를 줄이고 시스템 작업 대기 시간이 줄어들었음을 보였다.

Keywords

Acknowledgement

Supported by : National Research Foundation of Korea(NRF)

References

  1. R. Duan, R. Prodan, L. Xiaorong, "Multi-objective game theoretic scheduling of bag-of-tasks workflows on hybrid clouds," IEEE transaction on cloud computing, Vol. 2, No. 1, pp. 29-42, Jan.-Mar. 2014. https://doi.org/10.1109/TCC.2014.2303077
  2. Nimbus cloud [Online]. Available: http://www.nimbusproject.org/
  3. OpenStack [Online]. Available: http://www.OpenStack.org
  4. Amazon Web Services [Online] Available: http://aws.amazon.com
  5. E. Kalyvianaki, C.K Toh, "Ad hoc mobile wireless networks: protocols and systems," pp. 336, Prentice Hall PTR, New Jersey, 2002.
  6. E. Lee, "Cooperative Virtual Data Center: Sharing Data and Resources among Multiple Computing Entities," International Journal of Software Engineering and Its Applications, Vol. 9, No. 11, pp. 137-152, Nov. 2015.
  7. T. Adufu, J. Choi, K. Yoonhee, "Is container-based technology a winner for high performance scientific applications?" Network Operations and Management Symposium (APNOMS), Proc. of the 17th Asia-Pacific. IEEE, pp. 507-510, 2015.
  8. Docker [Online]. Available: http://www.docker.com
  9. N. Memari, S. Jahari, B. Hashim, B. Khairulmizam Samsudin, "Towards virtual honeynet based on LXC virtualization," Proc. of the Region 10 Symposium, 2014 IEEE, pp. 496-501, 2014.
  10. M. G. Xavier, M. V. Neves, C. A. F. De Rose, "Performance Evaluation of Container-based Virtualization for High Performance," Proc. of the 21st Euromicro International Conference for Computing Environments Parallel, Distributed and Network- Based Processing (PDP), pp. 233-240, 2013.
  11. M. G Xavier, M. V. Neves, C A. F. De Rose, "A Performance Comparison of Container-based Virtualization for MapReduce," Proc. of the 22nd Euromicro International Conference for Clusters Parallel, Distributed and Network-Based Processing (PDP), pp. 299-306, 2014.
  12. W. Hong, K. Wang, Y. H. Hsu, "Application-aware Resource Allocation for SDN-based Cloud Datacenters," Proc. of the 2013 International Conference for Cloud Computing and Big Data, pp. 106-110, 2013.
  13. N. Vasic, D. Novakovic, S. Miucin, D. Kostic, and R. Bianchini. DejaVu: Accelerating Resource Allocation in Virtualized Environments, ACM SIGARCH computer architecture news, Vol. 40, No. 1, 2012.
  14. M. Mao, Maria Humphrey, "Auto-scaling to minimize cost and meet application deadlines in cloud workflows," Proc. of the 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-12, 2011.
  15. Yu, "Resource Allocation in Virtualized Systems Based on Try-Before-Buy Approach," Proc. of the 2nd International conference for Challenges in Environmental Science and Computer Engineering, pp. 193-199, 2011.
  16. Kundu, Sajib, et al., "Revenue Driven Resource Allocation for Virtualized Data Centers," Proc. of the 2015 IEEE International Conference for autonomic Computing (ICAC), pp. 197-206, 2015.
  17. A. Wolke, M. Bichler, and T. Setzer, "Planning vs. dynamic control: Resource allocation in corporate clouds," IEEE Transactions on Cloud Computing (TCC), Vol. 4, No. 3, pp. 322-335, 2014. https://doi.org/10.1109/TCC.2014.2360399
  18. T. Lorido-Bortran, J. Miguel-Alonso and J.A. Lozano, "A review of autoscaling techniques for elastic applications in cloud environments," Journal of Grid Computing, Vol. 12, No. 4, pp. 559-592, 2014. https://doi.org/10.1007/s10723-014-9314-7
  19. Peek et al., The GALFA-HI Survey: Data Release 1, The Astrophysical Journal Supplement, Vol. 194, No. 2, article id. 20, pp. 1-13. 2011. https://doi.org/10.1088/0067-0049/194/1/1
  20. Montage [Online] Available: http://montage.ipac.caltech.edu/
  21. Valgrind [Online] Available: http://www.valgrind.org/