Simet: Synthetic image metrics - a synthetic image evaluation framework

Abstract

Simet provides a modular framework designed for the rigorous evaluation of synthetic image datasets. The framework integrates data provisioning, preprocessing, feature extraction, and complementary metrics, including Fréchet Inception Distance (FID), generative Precision/Recall, and classifier two-sample area under the receiver operating characteristic curve (ROC-AUC), within a single GPU-accelerated pipeline. A restraint mechanism enables declarative pass or fail gating. YAML- and command-line (CLI)-driven orchestration, shared feature caches, and structured logs facilitate reproducible, continuous-integration (CI)-ready workflows. Extensible abstractions, including providers, transforms, feature extractors, and metrics, allow practitioners to add new data sources or tests with minimal code. Templates support downstream utility evaluations, such as training on synthetic data and testing on real data (TSTR). Simet is positioned relative to existing toolkits, and protocols are outlined to demonstrate scalable, multidimensional evaluation of synthetic image data.