ARCHITECTURE AGNOSTIC NEURAL NETWORKS

Abstract

In this paper, we explore an alternate method for synthesizing neural network architectures, inspired by the brain's stochastic synaptic pruning. During a person's lifetime, numerous distinct neuronal architectures are responsible for performing the same tasks. This indicates that biological neural networks are, to some degree, architecture agnostic. However, artificial networks rely on their fine-tuned weights and hand-crafted architectures for their remarkable performance. This contrast begs the question: Can we build artificial architecture agnostic neural networks? To ground this study we utilize sparse, binary neural networks that parallel the brain's circuits. Within this sparse, binary paradigm we sample many binary architectures to create families of architecture agnostic neural networks not trained via backpropagation. These high-performing network families share the same sparsity, distribution of binary weights, and succeed in both static and dynamic tasks. In summation, we create an architecture manifold search procedure to discover families of architecture agnostic neural networks.

1. INTRODUCTION

Fascinated by the developmental algorithms and stochasticity inherent in the developmental synaptic pruning process, in this paper, we will explore architecure agnostic neural networks via the lens of binary, sparse, networks. We ground our study using sparse binary neural networks because these networks capture many of the most salient aspects of biological networks: • distinct neuronal units implementing non-linear functions in constrain an output to (-1, +1) • synaptic connections that are restricted to (-1, +1) • inhibatory and excitatory connections are represented by (-1, +1) respectively In this paper we demonstrate that (i) AANNs exist in silico, (ii) high-performance sparse binary neural networks on static (MNIST classification) and dynamic (imitation learning on car-racing) tasks exist, and (iii) that our stochastic search and succeed (SENSE) algorithm explores the architecture manifold.

2. RELATED WORK

Biological neural networks endow organisms with the ability to perform a multitude of tasks, ranging from sensory processing (Glickfeld & Olsen, 2017; Peirce, 2015) , to memory storage and retrieval (Tan et al., 2017; Denny et al., 2017) , to decision making (Hanks & Summerfield, 2017; Padoa-Schioppa & Conen, 2017). Remarkably, these complex tasks persist throughout our lives despite neuronal pruning, and synapse deletion up until adulthood. This partially stochastic process of neuronal refinement is known as developmental synaptic pruning. Developmental synaptic pruning occurs when the physical connection between a neuron's dendrite and another neuron's axon is eliminated (Riccomagno & Kolodkin, 2015) , preventing any further relay of information. Interestingly, between infancy and adulthood mammals lose roughly 50% of their neuronal synapses (Chechik et al., 1999) . A study in humans estimated that our prefrontal cortex dendritic spine density, a proxy for synaptic density, is on average more than two times higher in childhood than adulthood (Petanjek et al., 2011) . This evolved process is also partially stochastic (Vogt, 2015) . One of the main manifestations of stochastic developmental variation in the brain

