CONVERGENCE ANALYSIS OF HOMOTOPY-SGD FOR NON-CONVEX OPTIMIZATION

Abstract

First-order stochastic methods for solving large-scale non-convex optimization problems are widely used in many big-data applications, e.g. training deep neural networks as well as other complex and potentially non-convex machine learning models. Their inexpensive iterations generally come together with slow global convergence rate (mostly sublinear), leading to the necessity of carrying out a very high number of iterations before the iterates reach a neighborhood of a minimizer. In this work, we present a first-order stochastic algorithm based on a combination of homotopy methods and SGD, called Homotopy-Stochastic Gradient Descent (H-SGD), which finds interesting connections with some proposed heuristics in the literature, e.g. optimization by Gaussian continuation, training by diffusion, mollifying networks. Under some mild assumptions on the problem structure, we conduct a theoretical analysis of the proposed algorithm. Our analysis shows that, with a specifically designed scheme for the homotopy parameter, H-SGD enjoys a global linear rate of convergence to a neighborhood of a minimum while maintaining fast and inexpensive iterations. Experimental evaluations confirm the theoretical results and show that H-SGD can outperform standard SGD.

1. INTRODUCTION

This paper focuses on the theoretical development and analysis of a stochastic optimization algorithm, called Homotopy-Stochastic Gradient Descent (H-SGD), based on the combination of homotopy methods and stochastic gradient descent (SGD). The algorithm we propose is specifically designed to solve finite-sum problems of the following form w * ∈ arg min w∈R d    f (w) := 1 N N j=1 f j (w)    , where f : R d → R is continuously differentiable, bounded below and not necessarily convex. In particular, we assume that we only have access to noisy function values and gradients of the objective function in equation 1 via a stochastic first-order oracle, as in (Nemirovski et al., 2009) and (Ghadimi & Lan, 2013) . Problems of this form typically arise in machine learning and deep learning applications, where the dimensionality of the datasets makes the full function and gradient evaluations too expensive. This class of problems is generally approximately solved by stochastic first-order iterative algorithms, e.g. SGD (Bottou et al., 2018 ), Adagrad (Duchi et al., 2011 ), Adam (Kingma & Ba, 2015) . At the iteration t, the algorithms of this class acquire a stochastic estimate of the function value f (w t , ξ t ) and the gradient g(w t , ξ t ) by calling the oracle with input w t , where ξ t is a random variable, i.e. when the noise comes from subsampling as in the mini-batch scenarios, then ξ t ∈ {0, 1} N with ξ t 1 = M and g(w t , ξ t ) = 1 M N j=1 ξ t,j • ∇f j (w t ). In the case of SGD, for a given w 0 ∈ R d and α > 0, the iterates are generated as follows w t+1 := w t -αg(w t , ξ t ) . (2) Consequently, the iterate w t+1 = w t+1 (ξ [t] ) is a function of the history ξ [t] := (ξ 0 , . . . , ξ t ) (also w 0 should be included in case it is a random initial point) of the generated random process and hence is itself random.

