ALGORITHMIC DETERMINATION OF THE COMBINATO-RIAL STRUCTURE OF THE LINEAR REGIONS OF RELU NEURAL NETWORKS

Abstract

We algorithmically determine the regions and facets of the canonical polyhedral complex, the universal object into which a ReLU network decomposes its input space. We show that the locations of the vertices of the canonical polyhedral complex along with their signs with respect to layer maps determine the full facet structure across all dimensions. Our algorithm which implements this approach makes use of our theorems that the dual complex to the canonical polyhedral complex is cubical, and it possesses a multiplication compatible with its facet structure. The resulting algorithm is numerically stable, polynomial time in the number of intermediate neurons, and obtains accurate information across all dimensions. This permits us to obtain, for example, the true topology of the decision boundaries of networks with low-dimensional inputs for binary classification tasks. We run empirics on such networks at initialization, finding that width alone does not increease observed topology, but width in the presence of depth does.

1. INTRODUCTION

For fully-connected ReLU networks (Nair & Hinton, 2010) , the canonical polyhedral complex of the network, as defined by Grigsby & Lindsey (2020), encodes its decomposition of input space into linear regions and determines key structures such as the decision boundary for a binary classification task. Investigation of properties and characterizations of this decomposition of input space are ongoing, in particular with respect to counting the top-dimensional linear regions (Serra et al., 2018; Hanin & Rolnick, 2019a; Montufar et al., 2014; Serra & Ramalingam, 2020; Xiong et al., 2020) , since these bounds give one measure of the expressivity of the associated network architecture. However, a theoretical understanding of adjacency between regions and more generally the connectivity of lower-dimensional facets are to our knowledge generally undocumented. Understanding the face relations in the canonical polyhedral complex-for example, for tiled surfaces, understanding all inclusions between vertices (0-faces), edges (1-faces) and polygons (2-faces) -is necessary to relate combinatorial properties of the polyhedral complex of a network to the topology of regions into which the decision boundary partitions input space, geometric measurements such as the presence of critical points (Grigsby et al., 2022) , or other notions of topological expressivity, as explored by Guss & Salakhutdinov (2018); Bianchini & Scarselli (2014). It is common to describe linear regions of the input space, R n0 , using "activation patterns" or "neural codes" recorded as vectors in {0, 1} N (Itskov et al., 2020). Unfortunately, having a list of which activation patterns are present in the interiors of the linear regions does not determine their pairwise intersection properties (Theorem 15), and computing the intersections of these regions directly is not numerically stable. Furthermore, the polyhedra comprising the linear regions appear, at first glance, arbitrarily complicated. In this work, we establish a simpler representation by proving the theorem that the geometric dualfoot_0 of any network's canonical polyhedral complex is a union of n 0 -dimensional cubes (see Figure 1 ). Inspired by the theory of oriented matroids and hyperplane arrangements (Anders et al., 2000; Aguiar & Mahajan, 2017) , we demonstrate how the notion of "sign vectors" from oriented matroids, which are vectors with entries in {-1, 0, 1}, can serve as a labeling scheme for vertices, edges, and higher-dimensional regions. These sign vectors have properties that track



Recall, for example, that the dual of the icosahedron is the dodecahedron. 1

