CRITICAL BATCH SIZE MINIMIZES STOCHASTIC FIRST-ORDER ORACLE COMPLEXITY OF DEEP LEARNING OPTIMIZER USING HYPERPARAMETERS CLOSE TO ONE Anonymous authors Paper under double-blind review

Abstract

Practical results have shown that deep learning optimizers using small constant learning rates, hyperparameters close to one, and large batch sizes can find the model parameters of deep neural networks that minimize the loss functions. We first show theoretical evidence that the momentum method (Momentum) and adaptive moment estimation (Adam) perform well in the sense that the upper bound of the theoretical performance measure is small with a small constant learning rate, hyperparameters close to one, and a large batch size. Next, we show that there exists a batch size called the critical batch size minimizing the stochastic first-order oracle (SFO) complexity, which is the stochastic gradient computation cost, and that SFO complexity increases once the batch size exceeds the critical batch size. Finally, we provide numerical results that support our theoretical results. That is, the numerical results indicate that Adam using a small constant learning rate, hyperparameters close to one, and the critical batch size minimizing SFO complexity has faster convergence than Momentum and stochastic gradient descent (SGD).

1. INTRODUCTION

1.1 BACKGROUND Useful deep learning optimizers have been proposed to find the model parameters of the deep neural networks that minimize loss functions called the expected risk and empirical risk, such as stochastic gradient descent (SGD) (Robbins & Monro, 1951; Zinkevich, 2003; Nemirovski et al., 2009; Ghadimi & Lan, 2012; 2013 ), momentum methods (Polyak, 1964;; Nesterov, 1983) Theoretical analyses of adaptive methods for nonconvex optimization were presented in (Zaheer et al., 2018; Zou et al., 2019; Chen et al., 2019; Zhou et al., 2020; Zhuang et al., 2020; Chen et al., 2021) (see (Jain et al., 2018; Fehrman et al., 2020; Chen et al., 2020; Scaman & Malherbe, 2020; Loizou et al., 2021) for convergence analyses of SGD). A particularly interesting feature of adaptive methods is the use of hyperparameters, denoted by β 1 and β 2 , that can be set to influence the method performance P(K) : = 1 K K k=1 E[∥∇f (θ k )∥ 2 ], where ∇f is the gradient of a loss function f : R d → R, (θ k ) K k=1 is the sequence generated by an optimizer, and K is the number of steps. The previous results are summarized in Table 1 indicating that using β 1 and/or β 2 close to 0 makes the upper bound of P(K) small (see also Appendix A.1). Meanwhile, practical results for adaptive methods were presented in (Kingma & Ba, 2015; Reddi et al., 2018; Zaheer et al., 2018; Zou et al., 2019; Chen et al., 2019; Zhuang et al., 2020; 1 



, and adaptive methods. The various adaptive methods include Adaptive Gradient (AdaGrad) (Duchi et al., 2011), Root Mean Square Propagation (RMSProp) (Tieleman & Hinton, 2012), Adaptive Moment Estimation (Adam) (Kingma & Ba, 2015), Adaptive Mean Square Gradient (AMSGrad) (Reddi et al., 2018), Yogi (Zaheer et al., 2018), Adam with decoupled weight decay (AdamW) (Loshchilov & Hutter, 2019), and AdaBelief (named for adapting stepsizes by the belief in observed gradients) (Zhuang et al., 2020).

