REPRESENTING LATENT DIMENSIONS USING COM-PRESSED NUMBER LINES

Abstract

Humans use log-compressed number lines to represent different quantities, including elapsed time, traveled distance, numerosity, sound frequency, etc. Inspired by recent cognitive science and computational neuroscience work, we developed a neural network that learns to construct log-compressed number lines. The network computes a discrete approximation of a real-domain Laplace transform using an RNN with analytically derived weights giving rise to a log-compressed timeline of the past. The network learns to extract latent variables from the input and uses them for global modulation of the recurrent weights turning a timeline into a number line over relevant dimensions. The number line representation greatly simplifies learning on a set of problems that require learning associations in different spaces -problems that humans can typically solve easily. This approach illustrates how combining deep learning with cognitive models can result in systems that learn to represent latent variables in a brain-like manner and exhibit human-like behavior manifested through Weber-Fechner law.

1. INTRODUCTION

The human ability to map sensory inputs onto number lines is critical for rapid learning, reasoning, and generalizing. Recordings of activity from individual neurons in mammalian brains suggest a particular form of representation that could give rise to mental number lines over different variables. For instance, the presentation of a salient stimulus to an animal triggers sequential activation of neurons called time cells which are characterized by temporally tuned unimodal basis functions (MacDonald et al., 2011; Tiganj et al., 2017; Eichenbaum, 2014) . Each time cell reaches its peak activity at a particular time after the onset of some salient stimulus. Together, a population of time cells constitutes a temporal number line or a timeline of the stimulus history (Howard et al., 2015; Tiganj et al., 2018) . Similarly, as animals navigate spatial environments neurons called place cells exhibit spatially tuned unimodal basis functions (Moser et al., 2008) . A population of place cells constitutes a spatial number line that can be used for navigation (Bures et al., 1997; Banino et al., 2018) . The same computational strategy seems to be used to represent other variables as well, including numerosity (Nieder & Miller, 2003) , integrated evidence (Morcos & Harvey, 2016), pitch of tones (Aronov et al., 2017) , and conjunctions of these variables (Nieh et al., 2021) . Critically, many of these "neural number lines" appear to be log-compressed (Cao et al., 2021; Nieder & Miller, 2003) , providing a natural account of the Weber-Fechner law observed in psychophysics (Chater & Brown, 2008; Fechner, 1860 Fechner, /1912 ). Here we present a method by which deep neural networks can construct continuous, log-compressed number lines of latent task-relevant dimensions. Modern deep neural networks are excellent function approximators that learn in a distributed manner: weights are adjusted individually for each neuron. Neural activity in the brain suggests a representation where a population of neurons together encodes a distribution over a latent variable in the form of a number line. In other words, a latent variable is not represented as a scalar (e.g., a count of objects could be encoded with a single neuron with a firing rate proportional to the count), but as a function supported by a population of neurons, each tuned to a particular magnitude of the latent variable. To build deep neural networks with this property, we use global modulation such that recurrent weights of a population of cells are adjusted simultaneously. We show that this gives rise to the log-compressed number lines and can greatly facilitate associative learning in the latent space.

