High-performance computing techniques for efficient training and inference of AI models

Abstract

This thesis investigates the computational efficiency and scalability of biologically inspired neural network architectures-Liquid Time-Constant (LTC) and Closed-form Continuous-time (CfC) models-under varying hardware and training configurations. While traditional sequence models such as RNNs, LSTMs, and Transformers have demonstrated strong performance in temporal tasks, they often suffer from high computational costs and limited scalability. LTC and CfC networks have emerged as promising alternatives due to their dynamic temporal modeling capabilities and lower parameter counts, but their real-world efficiency in training remains underexplored. To evaluate the training behavior and parallelization potential of these models, experiments were conducted on three datasets of increasing complexity: a synthetic sine-cosine signal, the Human Activity Recognition (HAR) dataset, and the Metro Interstate Traffic dataset. Both LTC and CfC were trained under various conditions, including single-CPU, single-GPU, and distributed multi-GPU environments using PyTorch's Distributed Data Parallel (DDP) framework. Metrics such as training time, accuracy, loss, GPU utilization, and memory consumption were systematically collected and analyzed. The results demonstrate that CfC models consistently train faster than LTC models across all scenarios, achieving comparable or higher accuracy, particularly in larger datasets. Moreover, distributed training significantly reduces training time for both models, with optimal gains observed at two GPUs, beyond which communication overheads begin to affect scalability. These findings highlight the practical trade-offs between architectural complexity and parallel efficiency, and support the use of biologically inspired models in high-performance AI pipelines.