ON THE LOWER BOUND OF MINIMIZING POLYAK-ŁOJASIEWICZ FUNCTIONS Anonymous authors Paper under double-blind review

Abstract

Polyak-Łojasiewicz (PL) (Polyak, 1963) condition is a weaker condition than the strong convexity but suffices to ensure a global convergence for the Gradient Descent algorithm. In this paper, we study the lower bound of algorithms using first-order oracles to find an approximate optimal solution. We show that any first-order algorithm requires at least Ω L µ log 1 ε gradient costs to find an εapproximate optimal solution for a general L-smooth function that has an µ-PL constant. This result demonstrates the optimality of the Gradient Descent algorithm to minimize smooth PL functions in the sense that there exists a "hard" PL function such that no first-order algorithm can be faster than Gradient Descent when ignoring a numerical constant. In contrast, it is well-known that the momentum technique, e.g. (Nesterov, 2003, chap. 2) can provably accelerate Gradient Descent to O L μ log 1 ε gradient costs for functions that are L-smooth and μ-strongly convex. Therefore, our result distinguishes the hardness of minimizing a smooth PL function and a smooth strongly convex function as the complexity of the former cannot be improved by any polynomial order in general.

1. INTRODUCTION

We consider the problem min x∈R d f (x), where the function f is L-smooth and satisfies the Polyak-Łojasiewicz condition. A function f is said to satisfy the Polyak-Łojasiewicz condition if (2) holds for some µ > 0: ∥∇f (x)∥ 2 ≥ 2µ f (x) -inf y∈R d f (y) , ∀x ∈ R d . We refer to (2) as the µ-PL condition and simply denote inf y∈R d f (y) by f * . The PL condition may be originally introduced by Polyak (Polyak, 1963) and Łojasiewicz (Lojasiewicz, 1963) independently. The PL condition is strictly weaker than strong convexity as one can show that any μ-strongly convex function which by definition satisfies: f (x) ≥ f (y) + ⟨∇f (y), x -y⟩ + μ 2 ∥x -y∥ 2 is also μ-PL by minimizing both sides with respect to x (Karimi et al., 2016). However, the PL condition does not even imply convexity. From a geometric view, the PL condition suggests that the sum of the squares of the gradient dominates the optimal function value gap, which implies that any local stationary point is a global minimizer. Because it is relatively easy to obtain an approximate local stationary point by first-order algorithms, the PL condition serves as an ideal and weaker alternative to strong convexity. In machine learning, the PL condition has received wide attention recently. Lots of models are found to satisfy this condition under different regimes. Examples include, but are not limited to, matrix decomposition and linear neural networks under a specific initialization (Hardt & Ma, 2016; Li et al., 2018) , nonlinear neural networks in the so-called neural tangent kernel regime (Liu et al., 1

