RANDOM WEIGHT FACTORIZATION IMPROVES THE TRAINING OF CONTINUOUS NEURAL REPRESENTA-TIONS

Abstract

Continuous neural representations have recently emerged as a powerful and flexible alternative to classical discretized representations of signals. However, training them to capture fine details in multi-scale signals is difficult and computationally expensive. Here we propose random weight factorization as a simple drop-in replacement for parameterizing and initializing conventional linear layers in coordinate-based multi-layer perceptrons (MLPs) that significantly accelerates and improves their training. We show how this factorization alters the underlying loss landscape and effectively enables each neuron in the network to learn using its own self-adaptive learning rate. This not only helps with mitigating spectral bias, but also allows networks to quickly recover from poor initializations and reach better local minima. We demonstrate how random weight factorization can be leveraged to improve the training of neural representations on a variety of tasks, including image regression, shape representation, computed tomography, inverse rendering, solving partial differential equations, and learning operators between function spaces.

1. INTRODUCTION

Some of the recent advances in machine learning can be attributed to new developments in the design of continuous neural representations, which employ coordinate-based multi-layer perceptrons (MLPs) to parameterize discrete signals (e.g. images, videos, point clouds) across space and time. Such parameterizations are appealing because they are differentiable and much more memory efficient than grid-sampled representations, naturally allowing smooth interpolations to unseen input coordinates. As such, they have achieved widespread success in a variety of computer vision and graphics tasks, including image representation (Stanley, 2007; Nguyen et al., 2015) , shape representation (Chen & Zhang, 2019; Park et al., 2019; Genova et al., 2019; 2020 ), view synthesis (Sitzmann et al., 2019;; Saito et al., 2019; Mildenhall et al., 2020; Niemeyer et al., 2020) Despite their flexibility, it has been shown both empirically and theoretically that coordinate-based MLPs suffer from "spectral bias" (Rahaman et al., 2019; Cao et al., 2019; Xu et al., 2019) . This manifests as a difficulty in learning the high frequency components and fine details of a target function. A popular method to resolve this issue is to embed input coordinates into a higher dimensional space, for example by using Fourier features before the MLP (Mildenhall et al., 2020; Tancik et al., 2020) . Another widely used approach is the use of SIREN networks (Sitzmann et al., 2020) , which employs MLPs with periodic activations to represent complex natural signals and their derivatives. One main limitation of these methods is that a number of associated hyper-parameters (e.g. scale factors) need to be carefully tuned in order to avoid catastrophic generalization/interpolation errors. Unfortunately, the selection of appropriate hyper-parameters typically requires some prior knowledge about the target signals, which may not be available in some applications.



, texture generation(Oechsle et al., 2019; Henzler et al., 2020), etc. Coordinate-based MLPs have also been applied to scientific computing applications such as physics-informed neural networks (PINNs) for solving forward and inverse partial differential equations (PDEs) Raissi et al. (2019; 2020); Karniadakis et al. (2021), and Deep Operator networks (DeepONets) for learning operators between infinitedimensional function spaces Lu et al. (2021); Wang et al. (2021e).

