Towards Zero Memory Footprint Spiking Neural Network Training

التفاصيل البيبلوغرافية
العنوان: Towards Zero Memory Footprint Spiking Neural Network Training
المؤلفون: Lei, Bin, Lin, Sheng, Lin, Pei-Hung, Liao, Chunhua, Ding, Caiwen
سنة النشر: 2023
المجموعة: Computer Science
مصطلحات موضوعية: Computer Science - Neural and Evolutionary Computing, Computer Science - Artificial Intelligence
الوصف: Biologically-inspired Spiking Neural Networks (SNNs), processing information using discrete-time events known as spikes rather than continuous values, have garnered significant attention due to their hardware-friendly and energy-efficient characteristics. However, the training of SNNs necessitates a considerably large memory footprint, given the additional storage requirements for spikes or events, leading to a complex structure and dynamic setup. In this paper, to address memory constraint in SNN training, we introduce an innovative framework, characterized by a remarkably low memory footprint. We \textbf{(i)} design a reversible SNN node that retains a high level of accuracy. Our design is able to achieve a $\mathbf{58.65\times}$ reduction in memory usage compared to the current SNN node. We \textbf{(ii)} propose a unique algorithm to streamline the backpropagation process of our reversible SNN node. This significantly trims the backward Floating Point Operations Per Second (FLOPs), thereby accelerating the training process in comparison to current reversible layer backpropagation method. By using our algorithm, the training time is able to be curtailed by $\mathbf{23.8\%}$ relative to existing reversible layer architectures.
نوع الوثيقة: Working Paper
URL الوصول: http://arxiv.org/abs/2308.08649
رقم الأكسشن: edsarx.2308.08649
قاعدة البيانات: arXiv