DISCRETIZATION INVARIANT LEARNING ON NEURAL FIELDS

Abstract

While neural fields have emerged as powerful representations of continuous data, there is a need for neural networks that can perform inference on such data without being sensitive to how the field is sampled, a property called discretization invariance. We develop DI-Net, a framework for learning discretization invariant operators on neural fields of any type. Whereas current theoretical analyses of discretization invariant networks are restricted to the limit of infinite samples, our analysis does not require infinite samples and establishes upper bounds on the variation in DI-Net outputs given different finite discretizations. Our framework leads to a family of neural networks driven by numerical integration via quasi-Monte Carlo sampling with discretizations of low discrepancy. DI-Nets manifest desirable theoretical properties such as universal approximation of a large class of maps between L 2 functions, and gradients that are also discretization invariant. DI-Nets can also be seen as generalizations of many existing network families as they bridge discrete and continuous network classes, such as convolutional neural networks (CNNs) and neural operators respectively. Experimentally, DI-Nets derived from CNNs are demonstrated to classify and segment visual data represented by neural fields under various discretizations, and sometimes even generalize to new types of discretizations at test time. Code: supplementary materials (URL to be released).

1. INTRODUCTION

Neural fields (NFs), which encode signals as the parameters of a neural network, have many useful properties. NFs can efficiently store and stream continuous data (Sitzmann et al., 2020b; Dupont et al., 2022; Gao et al., 2021; Takikawa et al., 2022; Cho et al., 2022) , represent and render detailed 3D scenes at lightning speeds (Müller et al., 2022) , and integrate data from a wide range of modalities (Gao et al., 2022) . NFs are thus an appealing data representation for many applications. However, current approaches for training networks on a dataset of NFs have major limitations. The sampling-based approach converts such data to pixels or voxels as input to discrete networks (Vora et al., 2021) , but it incurs interpolation errors and does not leverage the ability to evaluate the NF anywhere on its domain. The hypernetwork approach trains a model to predict NF parameters (or a lower dimensional "modulation" of such parameters) which can be tailored for downstream tasks (Tancik et al., 2020a; Dupont et al., 2022; Mehta et al., 2021) , but hypernetworks based on the parameter space of one type of NF are incompatible with other types. Moreover, hypernetworks are unsuitable for important classes of NFs whose parameters extend beyond a neural network, such as those with voxel (Sun et al., 2021; Alex Yu and Sara Fridovich-Keil et al., 2021), octree (Yu et al., 2021) or hash table (Müller et al., 2022; Takikawa et al., 2022) components. We seek to strengthen the sampling-based approach with the notion of discretization invariance: the output of an operator that processes a continuous signal by sampling it at a set of discrete points should be largely independent of how the sample points are chosen, particularly as the number of points becomes large. In this paper we propose the DI-Net, a discretization invariant neural network for learning and inference on neural fields (Fig. 1 ). By parameterizing layers as integrals over parametric functions of the input field, DI-Nets have access to powerful numerical integration techniques that yield strong convergence properties, including a universal approximation theorem for a wide class of maps between function spaces. DI-Nets can be applied to any type of NF, or in fact any data that can be represented as integrable functions on a bounded measurable set. Thus DI-Nets are a broad class of neural networks that encompass other continuous networks such as neural operators, and also extend Figure 1 : The DI-Net processes a neural field by evaluating it on a point set (discretization) which is used to perform numerical integration throughout the network. DI-Nets are interoperable between all types of NFs and can be trained on a broad range of tasks. discrete networks that act on pixels, point clouds, and meshes. They can be applied to classification, segmentation, and many other tasks. Our contributions are as follows: • We show that discretization invariance gives rise to a family of neural networks based on numerical integration, which we call DI-Nets. • Backpropagation through DI-Nets is discretization invariant, and they universally approximate a large class of maps between function spaces. • DI-Nets generalize a wide class of discrete models to the continuous domain, and we derive continuous analogues of convolutional neural networks for inference on neural fields that encode visual data. • We demonstrate convolutional DI-Nets on NF classification and dense prediction tasks, and show it can perform well under a range of discretization schemes. • We probe the limits of discretization invariance in practice, finding that DI-Net has some ability to generalize to new discretizations at test time, modulated by the task and the type of discretizations it was trained on.

2. RELATED WORK

Neural fields Multilayer perceptrons (MLPs) can be trained to capture a wide range of continuous data with high fidelity. The most prominent domains include shapes (Park et al., 2019; Mescheder et al., 2018 ), objects (Niemeyer et al., 2020; Müller et al., 2022), and 3D scenes (Mildenhall et al., 2020; Sitzmann et al., 2021) , but previous works also apply NFs to gigapixel images (Martel et al., 2021) , volumetric medical images (Corona-Figueroa et al., 2022 ), acoustic data (Sitzmann et al., 2020b; Gao et al., 2021) , tactile data (Gao et al., 2022) , depth and segmentation maps (Kundu et al., 2022), and 3D motion (Niemeyer et al., 2019) . Hypernetworks and modulation networks were developed for learning directly with NFs, and have been demonstrated on tasks including generative modeling, data imputation, novel view synthesis and classification (Sitzmann et al., 2020b; 2021; Tancik et al., 2020a; Sitzmann et al., 2019; 2020a; Mehta et al., 2021; Chan et al., 2021; Dupont et al., 2021; 2022) . Hypernetworks use meta-learning to learn to produce the MLP weights of desired output NFs, while modulation networks predict modulations that can be used to transform the parameters of an existing NF or generate a new NF. Another approach for learning NF→NF maps evaluates an input NF at grid points, produces features at the same points via a U-Net, and passes interpolated features through an MLP to produce output values at arbitrary query points (Vora et al., 2021) . Discretization invariant networks Networks that are agnostic to the discretization of the data domain has been explored in several contexts. Hilbert space PCA, DeepONets and neural operators learn discretization invariant maps between function spaces (Bhattacharya et al., 2020; Lu et al., 2021; Li et al., 2020; Kovachki et al., 2021b) , and are tailored to solve partial differential equations efficiently. On surface meshes, DiffusionNet (Sharp et al., 2022) uses the diffusion operator to achieve convergent behavior under mesh refinement. These previous works define discretization invariance as convergent behavior in the limit of infinite sample points, but do not characterize how different discretizations yield different behaviors in the finite case. In this work, we choose a stronger

