Enhancing HPC with serverless computing: Lithops on MareNostrum5

Abstract

Serverless computing offers a novel alternative for developing and deploying applications. By abstracting backend architecture from the user, developers are encouraged to write code without worrying about server management, scaling, or maintenance. The Function-as-a-Service (FaaS) model optimizes this by allowing developers to deploy discrete functions that scale automatically in response to demand without deep concerns about the scalability of the execution infrastructure or platform. Lithops, a multi-cloud serverless computing framework, follows this trend and enables developers to execute Python code across thousands of cloud cores without modifying local scripts. Despite its potential to deploy parallel jobs, Lithops has been designed for running big data jobs over cloud environments, and its applicability to High-Performance Computing (HPC) systems has been unexplored. This paper introduces a novel architecture enabling Lithops deployment in HPC systems like the MareNostrum 5 supercomputer. By leveraging the immense computational power of the HPC-MN5 supercomputer and the FaaS model of Lithops, the architecture aims to offer high performance and scalability while simplifying application coding and deployment. Our evaluations display Lithops' benchmarks over the MareNostrum 5 HPC scale with the number of nodes, outperforming other commercial cloud platforms in terms of Floating Point Operations Per Second (FLOPS) and read-write bandwidth, and avoiding CPU wastage.

This work has been partially financed by the European Commission (EU-HORIZON NEARDATA GA 101092644), Ministry of Science and Innovation under grant MCIN AEI/10.13039/ 501100011033/FEDER and PID GA PID2019- 107255GB-C21, and the Generalitat de Catalunya (AGAUR) under grant agreements 2021-SGR-00478 and 2021-SGR01626. The authors thankfully acknowledge RES resources provided by Barcelona Supercomputing Center in Data-Centric Computing Group to BCV-2024-2-0004.

Peer Reviewed

Postprint (published version)