DIFFERENTIALLY PRIVATE OPTIMIZATION FOR SMOOTH NON-CONVEX ERM

Abstract

We develop simple differentially private optimization algorithms that move along directions of (expected) descent to find an approximate second-order necessary solution for non-convex ERM. We use line search, mini-batching, and a two-phase strategy to improve the speed and practicality of the algorithm. Numerical experiments demonstrate the effectiveness of these approaches.

1. INTRODUCTION

Privacy protection has become a central issue in machine learning algorithms, and differential privacy (Dwork & Roth, 2014 ) is a rigorous and popular framework for quantifying privacy. In our paper, we propose a differentially private optimization algorithm that finds an approximate second-order necessary solution for ERM problems. We proposed several techniques to improve the practical performance of the method, including backtracking line search, mini-batching, and a heuristic to avoid the effects of conservative assumptions made in the analysis. For given f : R d → R, consider the following minimization problem, min w∈R d f (w). We want to find an approximate second-order necessary solution, defined formally as follows. Definition 1 ((ϵ g , ϵ H )-2NS). For given positive values of ϵ g and ϵ H , We say that w is an (ϵ g , ϵ H )-approximate second-order necessary solution (abbreviated as (ϵ g , ϵ H )-2NS) if ∥∇f (w)∥ ≤ ϵ g , λ min ∇ 2 f (w) ≥ -ϵ H . (1) We are mostly interested in the case of ϵ g = α and ϵ H = √ M α, that is, we seek an (α, √ M α)-2NS, where M is the Lipschitz constant for ∇ 2 f . We will focus on the empirical risk minimization (ERM) problem, defined as follows, Definition 2 (ERM). Given a dataset D = {x 1 , . . . , x n } and a loss function l(w, x), we seek the parameter w ∈ R d that minimizes the empirical risk f (w) = L(w, D) := 1 n n i=1 l (w, x i ) . ERM is a classical problem in machine learning that has been studied extensively; proposes a trust-region type (DP-TR) algorithm that gives an approximate second-order necessary solution for ERM, satisfying both conditions in (1), for particular choices of ϵ g and ϵ H . This work requires the trust-region subproblem to be solved exactly at each iteration, and fixes the radius of the trust region at a small value, akin to a "short step" in a line-search method. An earlier work (Wang et al., 2019) proposed the DP-GD algorithm, which takes short steps in a noisy gradient direction, then sorts through all the iterates so generated to find one that satisfies second-order



see, for exampleShalev-Shwartz &  Ben-David (2014). In this paper, we describe differentially private (DP) techniques for solving ERM.

