SIMPLICIAL HOPFIELD NETWORKS

Abstract

Hopfield networks are artificial neural networks which store memory patterns on the states of their neurons by choosing recurrent connection weights and update rules such that the energy landscape of the network forms attractors around the memories. How many stable, sufficiently-attracting memory patterns can we store in such a network using N neurons? The answer depends on the choice of weights and update rule. Inspired by setwise connectivity in biology, we extend Hopfield networks by adding setwise connections and embedding these connections in a simplicial complex. Simplicial complexes are higher dimensional analogues of graphs which naturally represent collections of pairwise and setwise relationships. We show that our simplicial Hopfield networks increase memory storage capacity. Surprisingly, even when connections are limited to a small random subset of equivalent size to an all-pairwise network, our networks still outperform their pairwise counterparts. Such scenarios include non-trivial simplicial topology. We also test analogous modern continuous Hopfield networks, offering a potentially promising avenue for improving the attention mechanism in Transformer models.

1. INTRODUCTION

Hopfield networks (Hopfield, 1982) foot_0 store memory patterns in the weights of connections between neurons. In the case of pairwise connections, these weights translate to the synaptic strength between pairs of neurons in biological neural networks. In such a Hopfield network with N neurons, there will be N 2 of these pairwise connections, forming a complete graph. Each edge is weighted by a procedure which considers P memory patterns and which, based on these patterns, seeks to minimise a defined energy function such that the network's dynamics are attracted to and ideally exactly settles in the memory pattern which is nearest to the current states of the neurons. The network therefore acts as a content addressable memory -given a partial or noise-corrupted memory, the network can update its states through recurrent dynamics to retrieve the full memory. Since its introduction, the Hopfield network has been extended and studied widely by neuroscientists (Griniasty et al., 1993; Schneidman et al., 2006; Sridhar et al., 2021; Burns et al., 2022 ), physicists (Amit et al., 1985; Agliari et al., 2013; Leonetti et al., 2021), and computer scientists (Widrich et al., 2020; Millidge et al., 2022) . Of particular interest to the machine learning community is the recent development of modern Hopfield networks (Krotov & Hopfield, 2016) and their close correspondence (Ramsauer et al., 2021) to the attention mechanism of Transformers (Vaswani et al., 2017 ). An early (Amit et al., 1985; McEliece et al., 1987) and ongoing (Hillar & Tran, 2018) theme in the study of Hopfield networks has been their memory storage capacity, i.e., determining the number of memory patterns which can be reliably stored and later recalled via the dynamics. As discussed in Appendix A.1, this theoretical and computational exercise serves two purposes: (i) improving the memory capacity of such models for theoretical purposes and computational applications; and (ii) gaining an abstract understanding of neurobiological mechanisms and their implications for biological memory systems. Traditional Hopfield networks with binary neuron states, in the limit of N → ∞ and P → ∞, maintain associative memories for up to approximately 0.14N patterns (Amit et al., 1985; McEliece et al., 1987) , and fewer if the patterns are statistically or spatially correlated (Löwe, 1998) . However, by a clever reformulation of the update rule based on the network energy, this capacity can be improved to N d-1 , where d ≥ 2 (Krotov & Hopfield, 2016), and even further to 2 N/foot_1 (Demircigil et al., 2017). Networks using these types of energy-based update rules are called modern Hopfield networks. Krotov & Hopfield (2016) (like Hopfield (1984) ) also investigated neurons which took on continuous states. Upon generalising this model by using the softmax activation function, Ramsauer et al. (2021) showed a connection to the attention mechanism of Transformers (Vaswani et al., 2017) . However, to the best of our knowledge, these modern Hopfield networks have not been extended further to include explicit setwise connections between neurons, as has been studied and shown to improve memory capacity in traditional Hopfield networks (Peretto & Niez, 1986; Lee et al., 1986; Baldi & Venkatesh, 1987; Newman, 1988) . Indeed, Krotov & Hopfield (2016) , who introduced modern Hopfield networks, make a mathematical analogy between their energy-based update rule and setwise connections given their energy-based update rule can be interpreted as allowing individual pairs of pre-and post-synaptic neurons to make multiple synapses with each other -making pairwise connections mathematically as strong as equivalently-ordered setwise connections 2 . Demircigil et al. ( 2017) later proved this analogy to be accurate in terms of theoretical memory capacity. By adding explicit setwise connections to modern Hopfield networks, we essentially allow all connections (pairwise and higher) to increase their strength -following the same interpretation, this can be thought of as allowing both pairwise and setwise connections between all neurons, any of which may be precisely controlled. Functionally, setwise connections appear in abundance in biological neural networks. What's more, these setwise interactions often modulate and interact with one another in highly complex and nonlinear fashions, adding to their potential computational expressiveness. We discuss these biological mechanisms in Appendix A.2. There are many contemporary models in deep learning which implicitly model particular types of setwise interactions (Jayakumar et al., 2020) . To explicitly model such interactions, we have multiple options. For reasons we discuss in Appendix A.3, we choose to model our setwise connections using a simplicial complex. We therefore develop and study Simplicial Hopfield Networks. We weight the simplices of the simplicial complex to store memory patterns and generalise the energy functions and update rules of traditional and modern Hopfield networks. Our main contributions are: • We introduce extensions of various Hopfield networks with setwise connections. In addition to generalising Hopfield networks to include explicit, controllable setwise connections based on an underlying simplicial structure, we also study whether the topological features of the underlying structure influences performance. • We prove and discuss higher memory capacity in the general case of simplicial Hopfield networks. For the fully-connected simplicial Hopfield network, we prove a larger memory capacity than previously shown by Newman (1988); Demircigil et al. (2017) for higherdegree Hopfield networks. • We empirically show improved performance under parameter constraints. By restricting the total number of connections to that of pairwise Hopfield networks with a mixture of pairwise and setwise connections, we show simplicial Hopfield networks retain a surprising amount of improved performance over pairwise networks but with fewer parameters, and are robust to topological variability.

2. SIMPLICIAL HOPFIELD NETWORKS

2.1 SIMPLICIAL COMPLEXES Simplicial complexes are mathematical objects which naturally represent collections of setwise relationships. Here we use the combinatorial form, called an abstract simplicial complex. Although, to build intuition and visualise the simplicial complex, we also refer to their geometric realisations. Definition 2.1. Let K be a subset of 2 [N ] . The subset K is an abstract simplicial complex if for any σ ∈ K, the condition ρ ⊆ σ gives ρ ∈ K, for any ρ ⊆ σ.



After the proposal ofMarr (1971), many similar models of associative memory were proposed, e.g., those ofNakano (1972), Amari (1972), Little (1974), and Stanley (1976) -all before Hopfield (1982). Nevertheless, much of the research literature refers to and seems more proximally inspired byHopfield (1982). Many of these models can also be considered instances of the Lenz-Ising model(Brush, 1967) with infinite-range interactions. Work by Horn, D. & Usher, M. (1988) study almost the same system but with an slight modification to the traditional update rule, whereas Krotov & Hopfield (2016) use their modern, energy-based update rule.

