ADAPTIVE GRADIENT METHODS CONVERGE FASTER WITH OVER-PARAMETERIZATION (AND YOU CAN DO A LINE-SEARCH)

Abstract

Adaptive gradient methods are typically used for training over-parameterized models capable of exactly fitting the data; we thus study their convergence in this interpolation setting. Under an interpolation assumption, we prove that AMSGrad with a constant step-size and momentum can converge to the minimizer at the faster O(1/T ) rate for smooth, convex functions. Furthermore, in this setting, we show that AdaGrad can achieve an O(1) regret in the online convex optimization framework. When interpolation is only approximately satisfied, we show that constant step-size AMSGrad converges to a neighbourhood of the solution. On the other hand, we prove that AdaGrad is robust to the violation of interpolation and converges to the minimizer at the optimal rate. However, we demonstrate that even for simple, convex problems satisfying interpolation, the empirical performance of these methods heavily depends on the step-size and requires tuning. We alleviate this problem by using stochastic line-search (SLS) and Polyak's step-sizes (SPS) to help these methods adapt to the function's local smoothness. By using these techniques, we prove that AdaGrad and AMSGrad do not require knowledge of problem-dependent constants and retain the convergence guarantees of their constant step-size counterparts. Experimentally, we show that these techniques help improve the convergence and generalization performance across tasks, from binary classification with kernel mappings to classification with deep neural networks.

1. INTRODUCTION

Adaptive gradient methods such as AdaGrad (Duchi et al., 2011 ), RMSProp (Tieleman & Hinton, 2012) , AdaDelta (Zeiler, 2012) , Adam (Kingma & Ba, 2015), and AMSGrad (Reddi et al., 2018) are popular optimizers for training deep neural networks (Goodfellow et al., 2016) . These methods scale well and exhibit good performance across problems, making them the default choice for many machine learning applications. Theoretically, these methods are usually studied in the non-smooth, online convex optimization setting (Duchi et al., 2011; Reddi et al., 2018) with recent extensions to the strongly-convex (Mukkamala & Hein, 2017; Wang et al., 2020; Xie et al., 2020) and non-convex settings (Li & Orabona, 2019; Ward et al., 2019; Zhou et al., 2018; Chen et al., 2019; Wu et al., 2019; Défossez et al., 2020; Staib et al., 2019 ). An online-batch reduction gives guarantees similar to stochastic gradient descent (SGD) in the offline setting (Cesa-Bianchi et al., 2004; Hazan & Kale, 2014; Levy et al., 2018) . However, there are several discrepancies between the theory and application of these methods. Although the theory advocates for using decreasing step-sizes for Adam, AMSGrad and its variants (Kingma & Ba, 2015; Reddi et al., 2018) , a constant step-size is typically used in practice (Paszke et al., 2019) . Similarly, the standard analysis of these methods assumes a decreasing momentum parameter, however, the momentum is fixed in practice. On the other hand, AdaGrad (Duchi et al., 2011) has been shown to be "universal" as it attains the best known convergence rates in both the stochastic smooth and non-smooth settings (Levy et al., 2018) , but its empirical performance is rather disappointing when training deep models (Kingma & Ba, 2015) . Improving the empirical performance was indeed the main motivation behind Adam and other methods (Tieleman & Hinton, 2012; Zeiler, 2012) that followed AdaGrad. Although these methods have better empirical performance, they are not guaranteed to converge to the solution with a constant step-size and momentum parameter.

