REPRESENTATIONAL DISSIMILARITY METRIC SPACES FOR STOCHASTIC NEURAL NETWORKS

Abstract

Quantifying similarity between neural representations-e.g. hidden layer activation vectors-is a perennial problem in deep learning and neuroscience research. Existing methods compare deterministic responses (e.g. artificial networks that lack stochastic layers) or averaged responses (e.g., trial-averaged firing rates in biological data). However, these measures of deterministic representational similarity ignore the scale and geometric structure of noise, both of which play important roles in neural computation. To rectify this, we generalize previously proposed shape metrics (Williams et al., 2021) to quantify differences in stochastic representations. These new distances satisfy the triangle inequality, and thus can be used as a rigorous basis for many supervised and unsupervised analyses. Leveraging this novel framework, we find that the stochastic geometries of neurobiological representations of oriented visual gratings and naturalistic scenes respectively resemble untrained and trained deep network representations. Further, we are able to more accurately predict certain network attributes (e.g. training hyperparameters) from its position in stochastic (versus deterministic) shape space.

1. INTRODUCTION

Comparing high-dimensional neural responses-neurobiological firing rates or hidden layer activations in artificial networks-is a fundamental problem in neuroscience and machine learning (Dwivedi & Roig, 2019; Chung & Abbott, 2021) . There are now many methods for quantifying representational dissimilarity including canonical correlations analysis (CCA; Raghu et al., 2017) , centered kernel alignment (CKA; Kornblith et al., 2019) , representational similarity analysis (RSA; Kriegeskorte et al., 2008a ), shape metrics (Williams et al., 2021) , and Riemannian distance (Shahbazi et al., 2021) . Intuitively, these measures quantify similarity in the geometry of neural responses while removing expected forms of invariance, such as permutations over arbitrary neuron labels. However, these methods only compare deterministic representations-i.e. networks that can be represented as a function f : Z → R n , where n denotes the number of neurons and Z denotes the space of network inputs. For example, each z ∈ Z could correspond to an image, and f (z) is the response evoked by this image across a population of n neurons (Fig. 1A ). Biological networks are essentially never deterministic in this fashion. In fact, the variance of a stimulus-evoked neural response is often larger than its mean (Goris et al., 2014) . Stochastic responses also arise in the deep learning literature in many contexts, such as in deep generative modeling (Kingma & Welling, 2019) , Bayesian neural networks (Wilson, 2020) , or to provide regularization (Srivastava et al., 2014) . Stochastic networks may be conceptualized as functions mapping each z ∈ Z to a probability distribution, F (• | z), over R n (Fig. 1B , Kriegeskorte & Wei 2021). Although it is easier to study the representational geometry of the average response, it is well understood that this provides an incomplete and potentially misleading picture (Kriegeskorte & Douglas, 2019) . For instance, the ability to discriminate between two inputs z, z ′ ∈ Z depends on the overlap of F (z) and F (z ′ ), and not simply the separation of their means (Fig. 1C-D ). A rich literature in neuroscience has built on top of this insight (Shadlen et al., 1996; Abbott & Dayan, 1999; Rumyantsev et al., 2020) . However,

