NOISE INJECTION NODE REGULARIZATION FOR RO-BUST LEARNING

Abstract

We introduce Noise Injection Node Regularization (NINR), a method of injecting structured noise into Deep Neural Networks (DNN) during the training stage, resulting in an emergent regularizing effect. We present theoretical and empirical evidence for substantial improvement in robustness against various test data perturbations for feed-forward DNNs when trained under NINR. The novelty in our approach comes from the interplay of adaptive noise injection and initialization conditions such that noise is the dominant driver of dynamics at the start of training. As it simply requires the addition of external nodes without altering the existing network structure or optimization algorithms, this method can be easily incorporated into many standard architectures. We find improved stability against a number of data perturbations, including domain shifts, with the most dramatic improvement obtained for unstructured noise, where our technique outperforms existing methods such as Dropout or L 2 regularization, in some cases. Further, desirable generalization properties on clean data are generally maintained.

1. INTRODUCTION

Nonlinear systems often display dynamical instabilities which enhance small initial perturbations and lead to cumulative behavior that deviates dramatically from a steady-state solution. Such instabilities are prevalent across physical systems, from hydrodynamic turbulence to atomic bombs (see Jeans & Darwin (1902); Parker (1958); Chandrasekhar (1961) ; Drazin & Reid (2004) ; Strogatz (2018) for just a few examples). In the context of deep learning (DL), DNNs, once optimized via stochastic gradient descent (SGD), suffer from similar instabilities as a function of their inputs. While remarkably successful in a multitude of real world tasks, DNNs are often surprisingly vulnerable to perturbations in their input data as a result (Szegedy et al., 2014) . Concretely, after training, even small changes to the inputs at deployment can result in total predictive breakdown. One may classify such perturbations with respect to the distribution from which training data is implicitly drawn. This data is typically assumed to have support over (the vicinity of) some lowdimensional submanifold of potential inputs, which is only learned approximately due to the discrete nature of the training set. To perform well during training, a network need only have well-defined behavior on the data manifold, accomplished through training on a given data distribution. However, data seen on deployment can display other differences with respect to the training set, as illustrated § Equal contribution 1

