IMPLICIT CONVEX REGULARIZERS OF CNN ARCHI-TECTURES: CONVEX OPTIMIZATION OF TWO-AND THREE-LAYER NETWORKS IN POLYNOMIAL TIME

Abstract

We study training of Convolutional Neural Networks (CNNs) with ReLU activations and introduce exact convex optimization formulations with a polynomial complexity with respect to the number of data samples, the number of neurons, and data dimension. More specifically, we develop a convex analytic framework utilizing semi-infinite duality to obtain equivalent convex optimization problems for several two-and three-layer CNN architectures. We first prove that two-layer CNNs can be globally optimized via an 2 norm regularized convex program. We then show that multi-layer circular CNN training problems with a single ReLU layer are equivalent to an 1 regularized convex program that encourages sparsity in the spectral domain. We also extend these results to three-layer CNNs with two ReLU layers. Furthermore, we present extensions of our approach to different pooling methods, which elucidates the implicit architectural bias as convex regularizers.

1. INTRODUCTION

Convolutional Neural Networks (CNNs) have shown a remarkable success across various machine learning problems (LeCun et al., 2015) . However, our theoretical understanding of CNNs still remains restricted, where the main challenge arises from the highly non-convex and nonlinear structure of CNNs with nonlinear activations such as ReLU. Hence, we study the training problem for various CNN architectures with ReLU activations and introduce equivalent finite dimensional convex formulations that can be used to globally optimize these architectures. Our results characterize the role of network architecture in terms of equivalent convex regularizers. Remarkably, we prove that the proposed methods are polynomial time with respect to all problem parameters. Bengio et al. (2006) ; Bach (2017). However, these studies are restricted to two-layer fully connected networks with infinite width, thus, the optimization problem involves infinite dimensional variables. Moreover, it has been shown that even adding a single neuron to a neural network leads to a non-convex optimization problem which cannot be solved efficiently (Bach, 2017) 2019) proved that the minimum 2 norm two-layer network that perfectly fits a one dimensional dataset outputs the linear spline interpolation. Moreover, Gunasekar et al. (2018) studied certain linear convolutional networks and revealed an implicit non-convex quasi-norm regularization. However, as the number of layers increases, the regularization approaches to 0 quasi-norm, which is not computationally tractable. Recently, Pilanci & Ergen (2020) showed that two-layer CNNs with linear activations can be equivalently optimized as nuclear and 1 norm regularized convex problems. Although all the norm characterizations provided by these studies are insightful for future research, existing results are quite restricted due to linear activations, simple settings or intractable problems. 1



. Another line of research in Parhi & Nowak (2019); Ergen & Pilanci (2019; 2020a;b;c;d); Pilanci & Ergen (2020); Savarese et al. (2019); Gunasekar et al. (2018); Maennel et al. (2018); Blanc et al. (2019); Zhang et al. (2016) focuses on the effect of implicit and explicit regularization in neural network training and aims to explain why the resulting network generalizes well. Among these studies, Parhi & Nowak (2019); Ergen & Pilanci (2020b;c;d); Savarese et al. (

