HETEROGENEOUS NEURONAL AND SYNAPTIC DYNAM-ICS FOR SPIKE-EFFICIENT UNSUPERVISED LEARNING: THEORY AND DESIGN PRINCIPLES

Abstract

This paper shows that the heterogeneity in neuronal and synaptic dynamics reduces the spiking activity of a Recurrent Spiking Neural Network (RSNN) while improving prediction performance, enabling spike-efficient (unsupervised) learning. We analytically show that the diversity in neurons' integration/relaxation dynamics improves an RSNN's ability to learn more distinct input patterns (higher memory capacity), leading to improved classification and prediction performance. We further prove that heterogeneous Spike-Timing-Dependent-Plasticity (STDP) dynamics of synapses reduce spiking activity but preserve memory capacity. The analytical results motivate Heterogeneous RSNN design using Bayesian optimization to determine heterogeneity in neurons and synapses to improve E, defined as the ratio of spiking activity and memory capacity. The empirical results on time series classification and prediction tasks show that optimized HRSNN increases performance and reduces spiking activity compared to a homogeneous RSNN.

1. INTRODUCTION

Spiking neural networks (SNNs) (Ponulak & Kasinski, 2011) use unsupervised bio-inspired neurons and synaptic connections, trainable with either biological learning rules such as spike-timingdependent plasticity (STDP) (Gerstner & Kistler, 2002) or supervised statistical learning algorithms such as surrogate gradient (Neftci et al., 2019) . Empirical results on standard SNNs also show good performance for various tasks, including spatiotemporal data classification, (Lee et al., 2017; Khoei et al., 2020) , sequence-to-sequence mapping (Zhang & Li, 2020) , object detection (Chakraborty et al., 2021; Kim et al., 2020) , and universal function approximation (Gelenbe et al., 1999; Iannella & Back, 2001 ). An important motivation for the application of SNN in machine learning (ML) is the sparsity in the firing (activation) of the neurons, which reduces energy dissipation during inference (Wu et al., 2019) . Many prior works have empirically shown that SNN has lower firing activity than artificial neural networks and can improve energy efficiency (Kim et al., 2022; Srinivasan & Roy, 2019) . However, there are very few analytical studies on how to reduce the spiking activity of an SNN while maintaining its learning performance. Understanding and optimizing the relations between spiking activity and performance will be key to designing energy-efficient SNNs for complex ML tasks. In this paper, we derive analytical results and present design principles from optimizing the spiking activity of a recurrent SNN (RSNN) while maintaining prediction performance. Most SNN research in ML considers a simplified network model with a homogeneous population of neurons and synapses (homogeneous RSNN (MRSNN)) where all neurons have uniform integration/relaxation dynamics, and all synapses use the same long-term potentiation (LTP) and long-term depression (LTD) dynamics in STDP learning rules. On the contrary, neurobiological studies have shown that a brain has a wide variety of neurons and synapses with varying firing and plasticity dynamics, respectively (Destexhe & Marder, 2004; Gouwens et al., 2019; Hansel et al., 1995; Prescott et al., 2008) . We show that optimizing neuronal and synaptic heterogeneity will be key to simultaneously reducing spiking activity while improving performance. We define the spike efficiency E of an RSNN as the ratio of its memory capacity C and average spiking activity S. Given a fixed number of neurons and synapses, a higher C implies a network can learn more patterns and hence, perform better in classification or prediction tasks (Aceituno et al., 2020; Goldmann et al., 2020) ; a lower spiking rate implies that a network is less active, and hence, will consume less energy while making inferences (Sorbaro et al., 2020; Rathi et al., 2021) . We analytically show that a Heterogeneous Recurrent SNN (HRSNN) model leads to a more spike-efficient learning architecture by reducing spiking activity while improving C (i.e., performance) of the learning models. In particular, we make the following contributions to the theoretical understanding of an HRSNN. • We prove that for a finite number of neurons, models with heterogeneity among the neuronal dynamics has higher memory capacity C. • We prove that heterogeneity in the synaptic dynamics reduces the spiking activity of neurons while maintaining C. Hence, a model with heterogeneous synaptic dynamics has a lesser firing rate than a model with homogeneous synaptic dynamics. • We connect the preceding results to prove that simultaneously using heterogeneity in neurons and synapses, as in an HRSNN, improves the spike efficiency of a network. We empirically characterize HRSNN considering the tasks of (a) classifying time series ( Spoken Heidelberg Digits (SHD)) and (b) predicting the evolution of a dynamical system (a modified chaotic Lorenz system). The theoretical results are used to develop an HRSNN architecture where a modified Bayesian Optimization (BO) is used to determine the optimal distribution of neuron and synaptic parameters to maximize E. HRSNN exhibits a better performance (higher classification accuracy and lower NRMSE loss) with a lesser average spike count S than MRSNN. Related Works Inspired by the biological observations, recent empirical studies showed potential for improving SNN performance with heterogeneous neuron dynamics (Perez-Nieves et al., 2021; Chakraborty & Mukhopadhyay, 2023) . However, there is a lack of theoretical understanding of why heterogeneity improves SNN performance, which is critical for optimizing SNNs for complex tasks. She et al. ( 2022) have analytically studied the universal sequence approximation capabilities of a feedforward network of neurons with varying dynamics. However, they did not consider heterogeneity in plasticity dynamics, and the results are applicable only for a feed-forward SNN and do not extend to recurrent SNNs (RSNN). The recurrence is not only a fundamental component of a biological brain (Soures & Kudithipudi, 2019), but as a machine learning (ML) model, RSNN also shows good performance in modeling spatiotemporal and nonlinear dynamics (Pyle & Rosenbaum, 2017; Gilra & Gerstner, 2017) . Hence, it is critical to understand whether heterogeneity can improve learning in an RSNN. To the best of our knowledge, this is the first work that analytically studies the impact of heterogeneity in synaptic and neuronal dynamics in an RSNN. This work shows that only using neuronal heterogeneity improves performance and does not impact spiking activity. The number of spikes required for the computation increases exponentially with the number of neurons. Therefore, simultaneously analyzing and optimizing neuronal and synaptic heterogeneity, as demonstrated in this work, is critical to design an energy-efficient recurrent SNN.



Notations Table

