2026-04-13T07:15:44Zhttps://recercat.cat/oai/request

oai:recercat.cat:2117/3283122026-01-30T07:29:26Zcom_2072_1033col_2072_452950

Managing failures in task-based parallel workflows in distributed computing environments Ejarque, Jorge Bertran, Marta Álvarez Cid-Fuentes, Javier Conejero, Javier Badia Sala, Rosa Maria Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors Algorithms Workflow -- Software Failure management Scientific workflows Parallel programming Distributed computing Sistemes operatius distribuïts (Ordinadors) Processament en paral·lel (Ordinadors) Cicle de treball -- Programari Current scientific workflows are large and complex. They normally perform thousands of simulations whose results combined with searching and data analytics algorithms, in order to infer new knowledge, generate a very large amount of data. To this end, workflows comprise many tasks and some of them may fail. Most of the work done about failure management in workflow managers and runtimes focuses on recovering from failures caused by resources (retrying or resubmitting the failed computation in other resources, etc.) However, some of these failures can be caused by the application itself (corrupted data, algorithms which are not converging for certain conditions, etc.), and these fault tolerance mechanisms are not sufficient to perform a successful workflow execution. In these cases, developers have to add some code in their applications to prevent and manage the possible failures. In this paper, we propose a simple interface and a set of transparent runtime mechanisms to simplify how scientists deal with application-based failures in task-based parallel workflows. We have validated our proposal with use-cases from e-science and machine learning to show the benefits of the proposed interface and mechanisms in terms of programming productivity and performance. This work has been supported by the Spanish Government (contracts SEV2015-0493 and TIN2015-65316-P), by the Generalitat de Catalunya (contract 2014-SGR-1051), and by the European Commission’s Horizon 2020 Framework program through BioExcel Center of Excellence (contracts 823830, and 675728). The research leading to these results has received funding from the collaboration between Fujitsu and BSC (Script Language Platform). Peer Reviewed Postprint (author's final draft) 2020 Part of book or chapter of book https://link.springer.com/chapter/10.1007/978-3-030-57675-2_26 info:eu-repo/grantAgreement/EC/H2020/823830/EU/BioExcel Centre of Excellence for ComputationalBiomolecular Research/BioExcel-2 info:eu-repo/grantAgreement/EC/H2020/675728/EU/Centre of Excellence for Biomolecular Research/BioExcel https://doi.org/10.6084/m9.figshare.12556445 Open Access Springer, Cham