LOCAL SEARCH ALGORITHMS FOR RANK-CONSTRAINED CONVEX OPTIMIZATION

Abstract

We propose greedy and local search algorithms for rank-constrained convex optimization, namely solving min rank(A)≤r * R(A) given a convex function R : R m×n → R and a parameter r * . These algorithms consist of repeating two steps: (a) adding a new rank-1 matrix to A and (b) enforcing the rank constraint on A. We refine and improve the theoretical analysis of Shalev-Shwartz et al. (2011), and show that if the rank-restricted condition number of R is κ, a solution A with rank O(r * • min{κ log R(0)-R(A * ) , κ 2 }) and R(A) ≤ R(A * ) + can be recovered, where A * is the optimal solution. This significantly generalizes associated results on sparse convex optimization, as well as rank-constrained convex optimization for smooth functions. We then introduce new practical variants of these algorithms that have superior runtime and recover better solutions in practice. We demonstrate the versatility of these methods on a wide range of applications involving matrix completion and robust principal component analysis.

1. INTRODUCTION

Given a real-valued convex function R : R m×n → R on real matrices and a parameter r * ∈ N, the rank-constrained convex optimization problem consists of finding a matrix A ∈ R m×n that minimizes R(A) among all matrices of rank at most r * : min rank(A)≤r * R(A) Even though R is convex, the rank constraint makes this problem non-convex. Furthermore, it is known that this problem is NP-hard and even hard to approximate (Natarajan (1995); Foster et al. ( 2015)). In this work, we propose efficient greedy and local search algorithms for this problem. Our contribution is twofold: 1. We provide theoretical analyses that bound the rank and objective value of the solutions returned by the two algorithms in terms of the rank-restricted condition number, which is the natural generalization of the condition number for low-rank subspaces. The results are significantly stronger than previous known bounds for this problem. 2. We experimentally demonstrate that, after careful performance adjustments, the proposed general-purpose greedy and local search algorithms have superior performance to other methods, even for some of those that are tailored to a particular problem. Thus, these algorithms can be considered as a general tool for rank-constrained convex optimization and a viable alternative to methods that use convex relaxations or alternating minimization. The rank-restricted condition number Similarly to the work in sparse convex optimization, a restricted condition number quantity has been introduced as a reasonable assumption on R. If we let ρ + r be the maximum smoothness bound and ρ - r be the minimum strong convexity bound only along rank-r directions of R (these are called rank-restricted smoothness and strong convexity respectively), the rank-restricted condition number is defined as κ r = 



If this quantity is bounded, 1

