ROBUST AND ACCELERATED SINGLE-SPIKE SPIKING NEU-RAL NETWORK TRAINING WITH APPLICABILITY TO CHAL-LENGING TEMPORAL TASKS

Abstract

Spiking neural networks (SNNs), particularly the single-spike variant in which neurons spike at most once, are considerably more energy efficient than standard artificial neural networks (ANNs). However, single-spike SSNs are difficult to train due to their dynamic and non-differentiable nature, where current solutions are either slow or suffer from training instabilities. These networks have also been critiqued for their limited computational applicability such as being unsuitable for time-series datasets. We propose a new model for training single-spike SNNs which mitigates the aforementioned training issues and obtains competitive results across various image and neuromorphic datasets, with up to a 13.98× training speedup and up to an 81% reduction in spikes compared to the multi-spike SNN. Notably, our model performs on par with multispike SNNs in challenging tasks involving neuromorphic time-series datasets, demonstrating a broader computational role for single-spike SNNs than previously believed.

1. INTRODUCTION

Artificial neural networks (ANNs) have achieved impressive feats over recent years, obtaining human-level performance on visual and auditory tasks (Hinton et al., 2012; He et al., 2016) , natural language processing (Brown et al., 2020) and challenging games (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019) . However, as the difficulty and complexity of the tasks increase, so has the size of the networks required to solve them, demanding a substantial and unsustainable amount of energy (Strubell et al., 2019; Schwartz et al., 2020) . Inspired by the extreme energy efficiency of the brain (Sokoloff, 1960) , spiking neural networks (SNNs) emulated on neuromorphic computers attempt to solve this dilemma, requiring significantly less energy than ANNs (Wunderlich et al., 2019) . These networks are of growing interest, obtaining noteworthy results on visual (Fang et al., 2021; Zhou & Li, 2021 ), auditory (Yin et al., 2020; Yao et al., 2021) and reinforcement learning problems (Patel et al., 2019; Tang et al., 2020; Bellec et al., 2020) . A particular class of SNNs in which individual neurons respond with at most one spike aims to further amplify the energy and scaling advantages of standard SNNs and ANNs. Inspired by the sparse spike processing shown to exist at least for certain stimuli in the auditory and visual systems (Heil, 2004; Gollisch & Meister, 2008) , and forming a class of universal function approximator (Comsa et al., 2020) , these networks obtain extreme energy efficiency due to their singlespike nature (Oh et al., 2021; Liang et al., 2021) . Although providing a promising path toward building very large and energy-efficient networks, we are yet to understand how to properly train these SNNs. The success of the backprop training algorithm in ANNs does not naturally transfer to single-and multi-spike SNNs due to their non-differentiable activation function. Current attempts at training are either slow (as time is sequentially simulated) or suffer from training instabilities (e.g. the dead neuron problem) and idiosyncrasies (e.g. requiring particular regularisation) (Eshraghian et al., 2021) . Additionally, it has been argued that single-spike networks have limited applicability and are not suited for temporal problems, as recently pointed out by Eshraghian et al. (2021) : "[...] it enforces stringent priors upon the network (e.g., each neuron must fire only once) that are incompatible with dynamically changing input data" and Zenke et al. ( 2021): "[...] only using single spikes in each neuron has its limits and is less suitable for processing temporal stimuli, such as electroencephalogram (EEG) signals, speech, or videos". In this work we address these shortcomings by proposing a new model for training single-spike networks, for which the main contributions are summarised as follows. 1. Our model for training single-spike SNNs eschews all sequential dependence on time and exclusively relies on GPU parallelisable non-sequential operations. We experimentally validate this to obtain faster training times over sequentially trained control models on synthetic benchmarks (up to 16.77× speedup) and real datasets (up to 13.98× speedup). 2. We obtain competitive accuracies on various image and neuromorphic datasets with extreme spike sparsity (up to 81% fewer spikes than standard multi-spike SNNs), with our model being insensitive to the dead neuron problem and not requiring careful network regularisation. In other single-spike training methods, but not in our model, the dead neuron problem tends to halt learning due to reduced network activity. 3. We showcase our model's applicability in deeper and convolutional networks, and through the inclusion of trainable membrane time constants manage to solve difficult temporal problems otherwise thought to be unsolvable by single-spike networks. 

2.1. SINGLE-SPIKE MODEL

A spiking neural network (SNN) consists of artificial neurons which output binary signals known as spikes (Figure 1a ). Assume a feedforward network architecture of L fully connected layers, where each layer l consists of N (l ) neurons that are fully connected to the next layer l + 1 via synaptic weights W (l +1) ∈ R N (l +1) ×N (l ) . Neuron i in layer l emits a spike S (l ) i [t ] ∈ {0, 1} at time t if its membrane potential V (l ) i [t ] ∈ R reaches firing threshold V t h . S (l ) i [t ] = f (V (l ) i [t ]) = ! 1, if V (l ) i [t ] > V t h 0, otherwise Membrane potentials evolve according to the leaky integrate and fire (LIF) model τ dV (l ) i (t ) d t = -V (l ) i (t ) + V r est + R I (l ) i (t ) (2) where τ ∈ R is the membrane time constant and R ∈ R is the input resistance (Gerstner et al., 2014) .foot_0 Without loss of generality the LIF model is normalised (V (l ) i (t ) ∈ [0, 1] by V r est = 0,V t h = 1, R = 1; see Appendix) and discretised using the forward Euler method (see Appendix), from



Note, we use () to refer to continuous time and [] to refer to discrete time.



Figure 1: Spiking neuron dynamics. a. Left: A multi-spike neuron emitting and receiving (per presynaptic terminal) multiple spikes. Right: Input and output activity of the neuron (bottom panel: Input raster, middle panel: Input current I and top panel: Membrane potential V . Dotted line represents the firing threshold and a dot above denotes a spike). b. Left: A single-spike neuron emitting and receiving (per presynaptic terminal) at most one spike per stimulus. Right: Input and output activity of the neuron).

