CIRCUITNET: A GENERIC NEURAL NETWORK TO RE-ALIZE UNIVERSAL CIRCUIT MOTIF MODELING

Abstract

The successes of artificial neural networks (ANNs) are largely attributed to mimicking the human brain structures. Recent advances in neuroscience revealed that neurons interact with each other through various kinds of connectivity patterns to process information, in which the common connectivity patterns are also called circuit motifs. However, many existing ANNs can only model one or two circuit motifs in their architectures, so that their performance may drastically vary among different types of machine learning tasks. In this paper, we propose a new type of neural network inspired by the architectures of neuronal circuits, namely Circuit Neural Network (CircuitNet). In CircuitNet, a group of densely connected neurons, namely circuit motif unit (CMU), form the basic unit of the network, which is capable of modeling universal circuit motifs by adjusting the weights within the CMUs. Compared with traditional feed-forward networks, CircuitNet has the ability to model more types of neuron connections such as feed-back and lateral motifs. Inspired by the locally dense and globally sparse structure of the human brain, several iterations of signal transmission among different CMUs are achieved by sparse connections through the input ports and output ports of different CMUs. Experiments have demonstrated that CircuitNet can outperform popular neural network architectures in function approximation, reinforcement learning, image classification, and time series forecasting tasks.

1. INTRODUCTION

In the past decades, artificial neural networks (ANN) (McCulloch & Pitts, 1943) have been widely used as function estimators to solve regression and classification problems, which massively push deep learning forward in vastly different fields such as computer vision (He et al., 2016) , natural language processing (Vaswani et al., 2017) , deep reinforcement learning (Mnih et al., 2015) , etc. The successes of ANN are largely attributed to mimicking the simplified human brain structures. For example, the original multi-layer perceptrons (MLP) are collections of neurons organized as layers (Minsky & Papert, 1969; Rosenblatt, 1958) , and signals are controlled and transmitted between layers via linear transformations and non-linear activation functions, just like the synapses in a biological brain. Recently, the network architectures become rather complex, but their basic units such as convolutional layers and recurrent layers are still different abstractions of human nervous systems (Fukushima & Miyake, 1982; Hubel & Wiesel, 1968; Lindsay, 2021; Kietzmann et al., 2019; van Bergen & Kriegeskorte, 2020) . In recent years, understanding on both deep learning and neuroscience has made great advance, and it is time to rethink how artificial neural network designs can be further inspired by neuroscience. Following previous ANNs, we simplify the signal from a neuron as a real number, and focus more on how to model the signal transmission and connectivity patterns among neurons. Recent findings in neuroscience (Luo, 2021; Peters, 1991; Standring, 2021; Swanson, 2012) emphasized the role of specific patterns of synaptic connectivity in neuron communication across different brain regions. These patterns are analogous to the connection of neurons in ANNs, and in the rest of this section we will introduce how these findings can inspire the ANN design. One line of advances in neuroscience revealed that neurons interact with each other through various kinds of connectivity patterns, namely the circuit motifs (Luo, 2021). There are four types of most common circuit motifs, including feed-forward excitation and inhibition, feed-back inhibition, lateral inhibition, and mutual inhibition, as shown in Figure 1 . However, neurons in most existing ANNs (including those with residual connections (He et al., 2016) ) that incorporate merely feedforward networks can only function as the feed-forward excitation and inhibition patterns. Even if some recurrent structures, e.g., RNNs (Hochreiter & Schmidhuber, 1997; Cho et al., 2014) , are able to model cyclic patterns, they may fail with complex interactions among upstream neurons before information comes in. Another typical property of the biological nervous system is the local density and global sparsity (Swanson, 2012) . Although a single neuron can have thousands of synapses, most of them are located within a small region and form a functional group for specific tasks (Hawkins & Ahmad, 2016) . Only a few of them travel to other functional groups as bridges between regions (Peters, 1991; Standring, 2021; Swanson, 2012) . In conclusion, the biological nervous system has a locally dense and globally sparse connection structure. This is typically not true in ANNs where if we consider the vectors in a network as functional groups with some specific meanings, connections within a group are mostly missing but those between groups are of high density. For instance, the attentions between vectors in the original type of transformer form a complete graph and are of the highest density (Vaswani et al., 2017) . On the other hand, many parameters within a neural network have been proved to be redundant (Glorot et al., 2011; Schwarz et al., 2021; Wen et al., 2016; Yoon & Hwang, 2017) and the global sparsity of the biological neurons can provide insight for us to reduce the complexity of the neural network (Mocanu et al., 2018; Pessoa, 2014; Strogatz, 2001) . As a result, the design of our proposed ANN follows two principles that previous work does not fully consider: 1) the low-level interaction pattern among neurons should contain different types of circuit motifs (universal circuit motifs), and 2) neuron connections should follow a locally-dense and globally-sparse pattern to form the high-level structure. In this paper, we propose a new type of architecture named Circuit Neural Network (CircuitNet) for neural network design. CircuitNet uses densely connected neurons as basic functional structure called Circuit Motif Unit (CMU) to support universal circuit motifs. The communication among different CMUs is achieved by the sparse connections between subsets of special neurons in CMUs, namely input/output ports. Experiments on both synthetic and real-world datasets prove that, with comparable or even fewer parameters, CircuitNet can outperform popular neural network architectures in various types of tasks, demonstrating its effectiveness and generalizability in machine learning. Notably, CircuitNet outperforms multi-layer perceptron in function approximation, image classification and reinforcement learning tasks, outperforms MLP-based, CNN-based and Transformer-based neural networks in image classification tasks on CIFAR-10, CIFAR-100 and ImageNet datasets, and outperforms CNN-based, RNN-based and Transformer-based neural networks in time series forecasting tasks.

2. CIRCUIT NEURAL NETWORK

In this section, we introduce the detailed implementation of the proposed Circuit Neural Network, including how to model the signal transmission between neurons in a Circuit Motif Unit (CMU) to support universal circuit motifs and the overall locally dense globally sparse connection structure.

2.1. NEURON SIGNAL TRANSMISSION FOR UNIVERSAL CIRCUIT MOTIFS

In this paper, the signal from a neuron is simplified as a real number. Given two or more connected neurons, their interaction is achieved by signal transmission, which is modeled as a transformation function g(•; •). As shown in Figure 1 , we aim at designing a transformation function that can model four types of neuron connection patterns (i.e., circuit motifs), including feed-forward, mutual, feedback and lateral motifs, corresponding to the four patterns in recent biological findings (Luo, 2021) . In feed-forward and mutual motifs, the input signal depends merely on the state of the previous neuron, while in the other two motifs, an input signal may be affected by the state of another neuron or the target neuron itself. In the following paragraphs, we present the design of our proposed transformation function and elaborate on why this design can model different circuit motifs. Linear Transformation One of the simplest transformation used to pass signals from neuron x j to x i can be written as: g(x i ; x j ) = W ij x j . (1)

