REGULARIZED LINEAR CONVOLUTIONAL NETWORKS INHERIT FREQUENCY SENSITIVITY FROM IMAGE STATISTICS

Abstract

It is widely acknowledged that trained convolutional neural networks (CNNs) have different levels of sensitivity to signals of different frequency. In particular, a number of empirical studies have documented CNNs sensitivity to low-frequency signals. In this work we show with theory and experiments that this observed sensitivity is a consequence of the frequency distribution of natural images, which is known to have most of its power concentrated in low-to-mid frequencies. Our theoretical analysis relies on representations of the layers of a CNN in frequency space, an idea that has previously been used to accelerate computations and study implicit bias of network training algorithms, but to the best of our knowledge has not been applied in the domain of model robustness.

1. INTRODUCTION

Since their rise to prominence in the early 1990s, convolutional neural networks (CNNs) have formed the backbone of image and video recognition, object detection, and speech to text systems (Lecun et al., 1998) . The success of CNNs has largely been attributed to their "hard priors" of spatial translation invariance and local receptive fields (Goodfellow et al., 2016, §9.3 ). On the other hand, more recent research has revealed a number of less desirable and potentially data-dependent biases of CNNs, such as a tendency to make predictions on the basis of texture features (Geirhos et al., 2019) . Moreover, it has been repeatedly observed that CNNs are sensitive to perturbations in targeted ranges of the Fourier frequency spectrum (Guo et al., 2019; Sharma et al., 2019) and further investigation has shown that these frequency ranges are dependent on training data (Yin et al., 2019; Bernhard et al., 2021; Abello et al., 2021; Maiya et al., 2022) . In this work, we provide a mathematical explanation for these frequency space phenomena, showing with theory and experiments that neural network training causes CNNs to be most sensitive to frequencies that are prevalent in the training data distribution. Our theoretical results rely on representing an idealized CNN in frequency space, a strategy we borrow from (Gunasekar et al., 2018) . This representation is built on the classical convolution theorem, w * x = ŵ • x (1.1) where x and ŵ denote the Fourier transform of x and w respectively, and * denotes a convolution. Equation 1.1 demonstrates that a Fourier transform converts convolutions into products. As such, in a "cartoon" representation of a CNN in frequency space, the convolution layers become coordinate-wise multiplications (a more precise description is presented in section 3). This suggests that in the presence of some form of weight decay, the weights ŵ for high-power frequencies in the training data distribution will grow during training, while weights corresponding to low-power frequencies in the training data will be suppressed. The resulting uneven magnitude of the weights ŵ across frequencies can thus account for the observed uneven perturbation-sensitivity of CNNs in frequency space. We formalize this argument for linear CNNs (without biases) in sections 3 and 4. One interesting feature of the framework set up in section 4 is that the discrete Fourier transform (DFT) representation of a linear CNN is precisely a feedforward network with block diagonal weight matrices, where each block corresponds to a frequency index. We show in theorem 4.9 that a learning objective for such a network of depth L with an 2 -norm penalty on weights is equivalent to an objective for the associated linear model with an p penalty on the singular values of each of its blocks, i.e. each frequency index -this result is new for CNNs with multiple channels and outputs. In particular, the latter penalty is highly sparsity-encouraging, suggesting as depth increases these linearly-activated CNNs have an even stronger incentive to prioritize frequencies present in the training data. It has long been known that the frequency content of natural images is concentrated in low-to-mid frequencies, in the sense that the power in Fourier frequency f is well-described by et al., 2001) . Hence, when specialized to training data distributions of natural images, our results explain findings that CNNs are more susceptible to low frequency perturbations in practice (Guo et al., 2019; Sharma et al., 2019) . 1/|f | α for a coefficient α ≈ 1 (Lee We use our theoretical results to derive specific predictions: CNN frequency sensitivity aligns with the frequency content of training data, and deeper models, as well as models trained with substantial weight decay, exhibit frequency sensitivity more closely reflecting the statistics of the underlying images. We confirm these predictions for nonlinear CNNs trained on the CIFAR10 and ImageNette datasets. Figure 1 shows our experimental results for a variety of CNN models trained on CIFAR10 as well as a variant of CIFAR10 preprocessed with high pass filtering (more experimental details will be provided in section 5). To the best of our knowledge, ours is the first work to connect the following research threads (see section 2 for further discussion): • equivalences between linear neural networks and sparse linear models, • classical data-dependent "shrinkage" properties of sparse linear models, • statistical properties of natural images, and • sensitivity of CNNs to perturbations in certain frequency ranges.

2. RELATED WORK

The following a brief synopsis of work most closely related to this paper; a more through survey can be found in appendix A. Perturbations in frequency components: (Guo et al., 2019) found that adversarial perturbations constrained to low frequency Fourier components allowed for greater query efficiency and higher transferability between different neural networks, and (Sharma et al., 2019) demonstrated that constraining to high or or midrange frequencies did not produce similar effects. (Jo & Bengio, 2017) , (Yin et al., 2019 ), (Bernhard et al., 2021 ), (Abello et al., 2021 ) and (Maiya et al., 2022) all found in one way or another that model frequency sensitivity depends on the underlying training data. Our work began as an attempt to explain this phenomenon mathematically. Implicit bias and representation cost of CNNs: Our analysis of (linear) convolutional networks leverages prior work on implicit bias and representational cost of CNNs, especially (Gunasekar et al., 2018) . There it was found that for a linear CNN trained on a binary linear classification task with exponential loss, with linear effective predictor β, the Fourier transformed predictor β converges (in direction) to a first-order stationary point of min 1 2 | β| 2/L such that y n βT x n ≥ 1 for all n. (2.1) Our general setup in section 3 closely follows these authors', and our theorem 4.9 partially confirms a suspicion of (Gunasekar et al., 2018, §6) that "with multiple outputs, as more layers are added, even fully connected networks exhibit a shrinking sparsity penalty on the singular values of the effective linear matrix predictor ..." While the above result describes a form of implicit regularization imposed by gradient descent, we instead consider explicit regularization imposed by auxiliary 2 norm penalties in objective functions, and prove equivalences of minimization problems. In this sense our analysis is technically more closely related to that of (Dai et al., 2021) , which considers parametrized families of functions f (x, w) and defines the representation cost of a function g(x) appearing in the parametric family as R(g) := min{|w| 2 2 | f (x, w) = g(x) for all x}. (2.2)

