Convergence of HPC, big data and machine learning applications and containerized infrastructures

Abstract

The convergence of HPC, BD and ML in the computing continuum is being pursued in earnest across the academic and industry. We envision virtualization and containerization technologies can be the basis for the convergence, because they reside as bridges between applications and infrastructures and provide well-known advantages, such as the encapsulation of specific software environments, which allows for customization, portability, and reproducibility; the isolation of users from the underlying system and from other users, which allows for security and fault protection; and the agile and finegrain resource allocation and balancing, which allows for efficient cluster utilization and failure recovery. However, challenges remain for this convergence at containerization level due to the diversity of applications and hardware heterogeneity. In this talk I will present previous and ongoing work, (1) Enable deployments and understand the performance of HPC, BD and ML applications using containers. (2) Provide an autonomic management platform for containerized HPC, BD, and ML applications. (3) Optimize container management and scheduling for containerized HPC, BD, and ML applications.