2026-04-18T07:06:08Zhttps://recercat.cat/oai/request

oai:recercat.cat:2117/202702026-02-04T08:17:42Zcom_2072_1033col_2072_452950

A study on data deduplication in HPC storage systems Meister, Dirk Kaiser, Jürgen Brinkmann, Andre Cortés, Toni Kuhn, Michael Kunkel, Julian Universitat Politècnica de Catalunya. Departament d'Arquitectura de Computadors Universitat Politècnica de Catalunya. CAP - Grup de Computació d'Altes Prestacions Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors Information storage and retrieval systems Back-up procedures Computer centres Cryptography Internet Replicated databases Storage management Informació -- Sistemes d'emmagatzematge i recuperació Deduplication is a storage saving technique that is highly successful in enterprise backup environments. On a file system, a single data block might be stored multiple times across different files, for example, multiple versions of a file might exist that are mostly identical. With deduplication, this data replication is localized and redundancy is removed – by storing data just once, all files that use identical regions refer to the same unique data. The most common approach splits file data into chunks and calculates a cryptographic fingerprint for each chunk. By checking if the fingerprint has already been stored, a chunk is classified as redundant or unique. Only unique chunks are stored. This paper presents the first study on the potential of data deduplication in HPC centers, which belong to the most demanding storage producers. We have quantitatively assessed this potential for capacity reduction for 4 data centers (BSC, DKRZ, RENCI, RWTH). In contrast to previous deduplication studies focusing mostly on backup data, we have analyzed over one PB (1212 TB) of online file system data. The evaluation shows that typically 20% to 30% of this online data can be removed by applying data deduplication techniques, peaking up to 70% for some data sets. This reduction can only be achieved by a subfile deduplication approach, while approaches based on whole-file comparisons only lead to small capacity savings. Peer Reviewed Postprint (published version) 2012 Conference report Meister, D. [et al.]. A study on data deduplication in HPC storage systems. A: International Conference for High Performance Computing, Networking, Storage and Analysis. "2012 International Conference for High Performance Computing, Networking, Storage and Analysis (SC)". Salt Lake City, Utah: Institute of Electrical and Electronics Engineers (IEEE), 2012, p. 1-11. 978-1­4673­0806­9 https://hdl.handle.net/2117/20270 10.1109/SC.2012.14 eng Restricted access - publisher's policy 11 p. application/pdf Institute of Electrical and Electronics Engineers (IEEE)