Title:
|
Scientific Big Data Visualization: a Coupled Tools Approach
|
Author:
|
Artigues, Antoni; Cucchietti, Fernando M.; Tripiana, Carlos; Vicente, David; Calmet, Hadrien; Marín, Guillermo; Houzeaux, Guillaume; Vázquez, Mariano
|
Other authors:
|
Barcelona Supercomputing Center |
Abstract:
|
We designed and implemented a parallel visualisation system for the analysis of large scale time-dependent particle type data. The particular challenge we address is how to analyse a high performance computation style dataset when a visual representation of the full set is not possible or useful, and one is only interested in finding and inspecting smaller subsets that fulfil certain complex criteria. We used Paraview as the user interface, which is a familiar tool for many HPC
users, runs in parallel, and can be conveniently extended. We distributed the data in a supercomputing environment using the Hadoop file system. On top of it, we run Hive or Impala, and implemented a connection between Paraview and them that allows us to launch programmable SQL queries in the database directly from within Paraview. The queries return a Paraview-native VTK object that fits directly into the Paraview pipeline. We find good scalability and response times. In the typical supercomputer environment (like the one we used for implementation) the queue and management system make it difficult to keep local data in between sessions, which imposes a bottleneck in the data loading stage. This makes our system most useful when permanently installed on a dedicated cluster. |
Abstract:
|
This work was partially supported by the Spanish Severo Ochoa Program, grant SEV-2011-00067. We acknowledge PRACE for awarding us access to the resource FERMI based in Italy at CINECA. |
Abstract:
|
Peer Reviewed |
Subject(s):
|
-Àrees temàtiques de la UPC::Enginyeria electrònica -Large scale systems -Database management Computer programs -Big data -Parallel visualisation system -Hadoop file system -Paraview-native VTK -Macrodades -Supercomputadors |
Rights:
|
|
Document type:
|
Article - Published version Article |
Published by:
|
South Ural State University (Chelyabinsk, Russia)
|
Share:
|
|