HEBBIAN AND GRADIENT-BASED PLASTICITY EN-ABLES ROBUST MEMORY AND RAPID LEARNING IN RNNS

Abstract

Rapidly learning from ongoing experiences and remembering past events with a flexible memory system are two core capacities of biological intelligence. While the underlying neural mechanisms are not fully understood, various evidence supports that synaptic plasticity plays a critical role in memory formation and fast learning. Inspired by these results, we equip Recurrent Neural Networks (RNNs) with plasticity rules to enable them to adapt their parameters according to ongoing experiences. In addition to the traditional local Hebbian plasticity, we propose a global, gradient-based plasticity rule, which allows the model to evolve towards its self-determined target. Our models show promising results on sequential and associative memory tasks, illustrating their ability to robustly form and retain memories. In the meantime, these models can cope with many challenging few-shot learning problems. Comparing different plasticity rules under the same framework shows that Hebbian plasticity is well-suited for several memory and associative learning tasks; however, it is outperformed by gradient-based plasticity on fewshot regression tasks which require the model to infer the underlying mapping.

1. INTRODUCTION

Biological neural networks can dynamically adjust their synaptic weights when faced with various real-world tasks. The ability of synapses to change their strength over time is called synaptic plasticity, a critical mechanism that underlies animals' memory and learning (Abbott & Regehr, 2004; Stuchlik, 2014; Abraham et al., 2019; Magee & Grienberger, 2020) . For example, synaptic plasticity is essential for memory formation and retrieval in the hippocampus (Martin et al., 2000; Neves et al., 2008; Rioult-Pedotti et al., 2000; Kim & Cho, 2017; Nabavi et al., 2014; Nakazawa et al., 2004) . Furthermore, recent results show that some forms of synaptic plasticity could be induced within seconds, enabling animals to form memory quickly and do one-shot learning (Bittner et al., 2017; Magee & Grienberger, 2020; Milstein et al., 2021) . To test whether plasticity rules could also aid the memory performance and few-shot learning ability in artificial models, we incorporate plasticity rules into Recurrent Neural Networks (RNNs). These plastic RNNs work like the vanilla ones, except that a learned plasticity rule would update network weights according to ongoing experiences at each time step. Historically, Hebb's rule is a classic model for long-term synaptic plasticity; it states that a synapse is strengthened when there is a positive correlation between the pre-and post-synaptic activity (Hebb, 1949) . Several recent papers utilize generalized versions of Hebb's rule and apply it to Artificial Neural Networks (ANNs) in different settings (Miconi et al., 2018; Najarro & Risi, 2020; Limbacher & Legenstein, 2020; Tyulmankov et al., 2022; Rodriguez et al., 2022) . With a redesigned framework, we apply RNNs with neuromodulated Hebbian plasticity to a range of memory and few-shot learning tasks. Consistent with the understanding in neuroscience (Magee & Grienberger, 2020; Martin et al., 2000; Neves et al., 2008) , we find these plastic RNNs excel in memory and few-shot learning tasks. Despite being simple and elegant, classical Hebbian plasticity comes with limitations. In multi-layer networks, the lack of feedback signals to previous layers could impede networks' ability to configure their weights in a fine-grained manner and evolve to the desired target (Magee & Grienberger, 2020; Marblestone et al., 2016) . In recent years, some authors argue that other forms of plasticity rules in the brain could produce similar effects as the back-propagation algorithm, although the underlying mechanisms are probably different (Sacramento et al., 2018; Whittington & Bogacz, 2019; Roelfsema & Holtmaat, 2018) . Inspired by these results, we attempt to model the synaptic plasticity in RNNs as self-generated gradient updates: at each time step, the RNN updates its parameters with a self-determined target. Allowing the RNN to generate and evolve to a customized target enables the RNN to configure its weights in a flexible and coordinated fashion. Like Hebb's rule, the proposed gradient-based plasticity rule is task-agnostic. It operates in an unsupervised fashion, allowing us to compare these two plasticity rules under the same framework. In machine learning, learning a plasticity rule is one of the many meta-learning approaches (Schmidhuber et al., 1997; Bengio et al., 2013) . Although a diverse collection of meta-learning methods have been proposed over the years (Huisman et al., 2021) , these meta-learning methods are typically built upon specific assumptions on the task structure (e.g., assume the supervising signals are explicitly given; see Sec. 2 for more detailed discussion). They thus could not be applied to arbitrary learning problems. In contrast, in our networks, the evolving direction of network parameters dW/dt solely depends on the current network state, i.e., current network parameters and the activity of neurons. Since the designed plasticity rules do not rely on task-specific information (e.g., designated loss function and labels), they could be naturally applied to any learning problems as long as the input is formulated as time series. Therefore, modeling biological plasticity rules also allows us to build more general meta-learners. Our contribution can be summarized as follows. Based on previous work (Miconi et al., 2019) , we formulate a framework that allows us to incorporate different plasticity rules into RNNs. In addition to the local Hebbian plasticity, we propose a novel gradient-based plasticity rule that allows the model to evolve towards self-determined targets. We show that both plasticity rules improve memory performance and enable rapid learning, suggesting that ANNs could benefit from synaptic plasticity similarly to animals. On the other hand, as computational models simulating biological plasticity, our models give insights into the roles of different forms of plasticity in animals' intelligent behaviors. We find that Hebbian plasticity is well-suited for many memory and associative learning tasks. However, the gradient-based plasticity works better in the few-shot regression task, which requires the model to infer the underlying mapping instead of learning direct associations.

2. RELATED WORK

Meta-Learning. Meta-learning, or "learning to learn", is an evolving field in ML that aims to build models that can learn from their ongoing experiences (Schmidhuber et al., 1997; Bengio et al., 2013) . A surprisingly diverse set of meta-learning approaches have been proposed in recent years (Hospedales et al., 2021; Finn et al., 2017; Santoro et al., 2016; Mishra et al., 2018; Lee et al., 2019) . In particular, one line of work proposes to meta-learn a learning rule capable of configuring network weights to adapt to different learning problems. This idea could be implemented by training an optimizer for gradient descent (Andrychowicz et al., 2016; Ravi & Larochelle, 2017) , training a Hypernetwork that generates the weights of another network (Ha et al., 2017) , or meta-learning a plasticity rule which allows RNNs to modify its parameters at each time step (Miconi et al., 2019; Ba et al., 2016; Miconi et al., 2018) . Our method belongs to the last category. Compared to other meta-learning approaches, training plastic RNNs has some unique advantages. Plastic RNNs are general meta-learners that could learn from any sequential input. In contrast, most meta-learning methods cannot deal with arbitrary learning problems due to their assumptions about task formulation. For example, methods that utilize gradient descent in the inner loop (e.g., MAML (Finn et al., 2017) , LSTM meta-learner (Ravi & Larochelle, 2017) and GD 2 (Andrychowicz et al., 2016)) typically assume that there exist explicit supervising signals (e.g., ground truth) and a loss function that is used to update the base learner. However, such information is often implicit in the real world (e.g., when humans do few-shot learning from natural languages (Brown et al., 2020) ). In contrast, plastic RNNs are task-agnostic: they can adapt their weights in an unsupervised manner,

availability

//github.com/yuvenduan/PlasticRNNs.

