NOISE INJECTION NODE REGULARIZATION FOR RO-BUST LEARNING

Abstract

We introduce Noise Injection Node Regularization (NINR), a method of injecting structured noise into Deep Neural Networks (DNN) during the training stage, resulting in an emergent regularizing effect. We present theoretical and empirical evidence for substantial improvement in robustness against various test data perturbations for feed-forward DNNs when trained under NINR. The novelty in our approach comes from the interplay of adaptive noise injection and initialization conditions such that noise is the dominant driver of dynamics at the start of training. As it simply requires the addition of external nodes without altering the existing network structure or optimization algorithms, this method can be easily incorporated into many standard architectures. We find improved stability against a number of data perturbations, including domain shifts, with the most dramatic improvement obtained for unstructured noise, where our technique outperforms existing methods such as Dropout or L 2 regularization, in some cases. Further, desirable generalization properties on clean data are generally maintained.

1. INTRODUCTION

Nonlinear systems often display dynamical instabilities which enhance small initial perturbations and lead to cumulative behavior that deviates dramatically from a steady-state solution. Such instabilities are prevalent across physical systems, from hydrodynamic turbulence to atomic bombs (see Jeans & Darwin (1902) ; Parker (1958) ; Chandrasekhar (1961) ; Drazin & Reid (2004) ; Strogatz (2018) for just a few examples). In the context of deep learning (DL), DNNs, once optimized via stochastic gradient descent (SGD), suffer from similar instabilities as a function of their inputs. While remarkably successful in a multitude of real world tasks, DNNs are often surprisingly vulnerable to perturbations in their input data as a result (Szegedy et al., 2014) . Concretely, after training, even small changes to the inputs at deployment can result in total predictive breakdown. One may classify such perturbations with respect to the distribution from which training data is implicitly drawn. This data is typically assumed to have support over (the vicinity of) some lowdimensional submanifold of potential inputs, which is only learned approximately due to the discrete nature of the training set. To perform well during training, a network need only have well-defined behavior on the data manifold, accomplished through training on a given data distribution. However, data seen on deployment can display other differences with respect to the training set, as illustrated § Equal contribution in Fig. 1 . These distortions introduce vulnerabilities that are a crucial drawback of trained DNNs, making them susceptible to commonly occurring noise which is ubiquitous in real-world tasks. By studying how networks dynamically act to mitigate the negative effects of input noise, we identify a novel dynamical regularization method starting in a noise-dominated regime, leading to more robust behavior for a range of data perturbations. This is the central contribution of this work. Background: Regularization involves introducing additional constraints in order to solve an illposed problem or to prevent over-fitting. In the context of DL problems, different regularization schemes have been proposed (for a review, see Kukačka et al. ( 2018) and references therein). These methods are designed to constrain the network parameters during training, thereby reducing sensitivity to irrelevant features in the input data, as well as avoiding overfitting. For instance, weight norm regularization (L 2 , L 1 , etc.) (Cortes & Vapnik, 1995; Zheng et al., 2003) can be used to reduce overfitting to the training data, and is often found to improve generalization performance (Hinton, 1987; Krogh & Hertz, 1991; Zhang et al., 2018) . Alternatively, introducing stochasticity during training (e.g., Dropout (Srivastava et al., 2014) ), has become a standard addition to many DNN architectures, for similar reasons. These methods are mostly optimized to reduce the generalization error from training to test data, under the assumption that both are sampled from the same underlying distribution (Srivastava et al., 2014) . Here, we propose a new method which is instead tailored for robustness. Our method relies on noise-injection, that actively reduces the sensitivity to uncorrelated input perturbations. Our contribution: In this paper, we employ Noise Injection Nodes (NINs), which feed random noise through designated optimizable weights, forcing the network to adapt to layer inputs which contain no useful information. Since the amount of injected noise is a free parameter, at initialization we can set it to be anything from a minor perturbation to the dominant effect, leading to a system breakdown for extreme values. The general behavior of NINs and how they probe the network is the main goal of Levi et al. (2022) , while we focus here on their regularizing properties in different noise injection regimes. The results of Levi et al. (2022) are explicitly recast in the context of regularization in App. C for a linear model, which captures the main insights. Our study suggests that within a certain range of noise injection parameter values, this procedure can substantially improve robustness against subsequent input corruption and partially against other forms of distributional shifts, where the maximal improvement occurs for large noise injection magnitudes approaching the boundary of this window, above which the training accuracy degrades to random guessing. To the best of our knowledge, this regime has not been previously explored. In the following, we analyze how the addition of NINs produces a regularization scheme which we call Noise Injection Node Regularization (NINR). The main features of NINR are enhanced stability, simplicity, and flexibility, without drastically compromising generalization performance. In order to demonstrate these features, we consider two types of feed-forward architectures: Fully Connected Networks (FCs) and Convolutional Neural Networks (CNNs), and use various datasets to train the systems. We compare NINR robustness improvement with standard regularization methods, as well as performance of these systems when using input corruption during training (CDT). Our results



Figure 1: Illustration of perturbations to data inputs with respect to the joint probability distribution manifold of features and labels. Points indicate {sample, label} pairs {x, y}, where different colored points correspond to samples drawn from different marginal distributions. Black points represent pairs from a training dataset {xi, yi} N i=1 , with the red spheres indicating corrupted inputs, determined by shifted distribution functions f corrupted (x + ϵ, y). The gray arrow represents an adversarial attack, performed by ascending up the gradient of the network output to reach the closest decision boundary, while generalization from training to test data is depicted as interpolation from black to blue points. Finally, domain shift is a shift in the underlying distribution on the same manifold, depicted by the green arrow and points.

