Scalable Deep Neural Network Training: Overcoming Memory Constraints with Performance Preservation

Abstract

This paper introduces a novel method for training Deep Neural Networks (DNNs) with batch sizes that surpass memory constraints, without compromising model performance. Conventional methods struggle with memory limitations when using large batch sizes, often resulting in reduced efficiency and degraded training outcomes. Our proposed solution, Small-Batch Processing (SBP), overcomes these challenges by leveraging a batch streaming mechanism and a loss normalization strategy based on gradient accumulation. SBP processes smaller micro-batches sequentially, enabling large-batch training on a single device without requiring memory augmentation or additional GPUs. Experimental evaluations confirm that SBP matches the performance of traditional approaches while enabling the use of batch sizes that were previously infeasible due to memory constraints. This technique significantly enhances the potential for training DNNs in resource-limited environments, driving improvements in both scalability and training efficiency for deep learning applications.