论文标题
重新访问批次归一化，以训练低延迟的深度尖峰神经网络
Revisiting Batch Normalization for Training Low-latency Deep Spiking Neural Networks from Scratch
论文作者
论文摘要
由于稀疏，异步和二进制事件（或尖峰）驱动的处理，尖峰神经网络（SNN）最近成为了深度学习的替代方法，可以对神经形态硬件产生巨大的能量效率。然而，培训高临界性和从头开始的低延迟SNN遭受了尖峰神经元的非差异性质。为了解决SNN中的这个培训问题，我们重新访问了批处理标准化，并提出了通过时间（BNTT）技术进行时间批归一化的。到目前为止，大多数先前的SNN都忽略了批处理规范化，认为这是无效的训练时间SNN。与以前的工作不同，我们提出的BNTT沿时间轴将BNTT层中的参数解散，以捕获尖峰的时间动力学。 BNTT中的时间不断发展的可学习参数使神经元可以通过不同的时间步长控制其尖峰速率，从而从头开始实现低延迟和低能训练。我们在CIFAR-10，CIFAR-100，微型IMAGENET和事件驱动的DVS-CIFAR10数据集上进行实验。 BNTT允许我们第一次在仅25-30个时步中从头开始训练深SNN体系结构。我们还建议使用BNTT中参数的分布来减少推断时的潜伏期，提出一种提前出口算法，从而进一步提高了能源效率。
Spiking Neural Networks (SNNs) have recently emerged as an alternative to deep learning owing to sparse, asynchronous and binary event (or spike) driven processing, that can yield huge energy efficiency benefits on neuromorphic hardware. However, training high-accuracy and low-latency SNNs from scratch suffers from non-differentiable nature of a spiking neuron. To address this training issue in SNNs, we revisit batch normalization and propose a temporal Batch Normalization Through Time (BNTT) technique. Most prior SNN works till now have disregarded batch normalization deeming it ineffective for training temporal SNNs. Different from previous works, our proposed BNTT decouples the parameters in a BNTT layer along the time axis to capture the temporal dynamics of spikes. The temporally evolving learnable parameters in BNTT allow a neuron to control its spike rate through different time-steps, enabling low-latency and low-energy training from scratch. We conduct experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and event-driven DVS-CIFAR10 datasets. BNTT allows us to train deep SNN architectures from scratch, for the first time, on complex datasets with just few 25-30 time-steps. We also propose an early exit algorithm using the distribution of parameters in BNTT to reduce the latency at inference, that further improves the energy-efficiency.
