LOWER BOUNDS FOR DIFFERENTIALLY PRIVATE ERM: UNCONSTRAINED AND NON-EUCLIDEAN

Abstract

We consider the lower bounds of differentially private empirical risk minimization (DP-ERM) for convex functions in both constrained and unconstrained cases concerning the general p norm beyond the 2 norm considered by most of the previous works. We provide a simple black-box reduction approach that can generalize lower bounds in constrained to unconstrained cases. Moreover, for ( , δ)-DP, we achieve the optimal Ω( ) lower bounds for both constrained and unconstrained cases and any p geometry where p ≥ 1 by considering 1 loss over the ∞ ball. ) for 1 ≤ p ≤ 2 and O(

1. INTRODUCTION

Since the seminal work of Dwork et al. (2006) , differential privacy (DP), defined below, has become the standard and rigorous notion of privacy guarantee for machine learning algorithms. In DP-ERM, we are given a family convex functions where each function (•; z) is defined on a convex set K ⊆ R d , and a data-set D = {z 1 , • • • , z n } to design a differentially private algorithm that can minimize the loss function L(θ; D) = 1 n n i=1 (θ; z i ), and the value L(θ; D) -min θ ∈K L(θ ; D) is called the excess empirical loss with respect to solution θ, measuring how it compares with the best solution in K. DP-ERM in the constrained case and Euclidean geometry (with respect to 2 norm) was studied first, well-studied, and most of the previous literature belongs to this case. More specifically, the Euclidean constrained case considers convex loss functions defined on a bounded convex set C R d , assuming the functions are 1-Lipschitz over the convex set of diameter 1 with respect to the 2 norm.  √ d log(1/δ) n ) Ours both 1 ≤ p ≤ ∞ general Ω( d n ) Ω( √ d log(1/δ) n ) Table 1 : Comparison of lower bounds for private convex ERM. One can easily extend our lower bounds in the unconstrained case to the constrained case. The lower bound of Song et al. ( 2021) is weaker than ours in the important over-parameterized d n setting, as rank ≤ min{n, d}.

1.1. OUR CONTRIBUTIONS

This paper considers the lower bounds for DP-ERM under unconstrained and/or non-euclidean settings. We summarize our main results as follows: • We propose a black-box reduction approach, which directly generalizes the lower bounds in constrained cases to the unconstrained case. Such a method is beneficial for its simplicity. Nearly all exiting lower bounds in the constrained case can be extended to the unconstrained case directly, and any new progress in the constrained case can be of immediate use due to its black-box nature. • We achieve Ω( √ d log(1/δ) n ) lower bounds for both constrained and unconstrained cases and any p geometry for p ≥ 1 at the same time by considering 1 loss over the ∞ ball. This bound improves previous results, exactly matches the upper bounds for 1 < p ≤ 2 and obtains novel bounds for p > 2.foot_1 



When δ > 0, we may refer to it as approximate-DP, and we name the particular case when δ = 0 pure-DP sometimes. The current best upper bound is O(min{log d, 1 p-1 } √ d log(1/δ) n



Definition 1.1 (Differential privacy). A randomized mechanism M is ( , δ)-differentially private 1 if for any event O ∈ Range(M) and for any neighboring databases D and D that differ by a single data element, one has Pr[M(D) ∈ O] ≤ exp( ) Pr[M(D ) ∈ O] + δ. Among the rich literature on DP, many fundamental problems are based on empirical risk minimization (ERM), and DP-ERM becomes one of the most well-studied problems in the DP community. See e.g., Chaudhuri & Monteleoni (2008); Rubinstein et al. (2009); Chaudhuri et al. (2011); Kifer et al. (2012); Song et al. (2013); Bassily et al. (2014); Jain & Thakurta (2014); Talwar et al. (2015); Kasiviswanathan & Jin (2016); Fukuchi et al. (2017); Wu et al. (2017); Zhang et al. (2017); Wang et al. (2017); Iyengar et al. (2019); Bassily et al. (2020); Kulkarni et al. (2021); Asi et al. (2021); Bassily et al. (2021b); Wang et al. (2021); Bassily et al. (2021a); Gopi et al. (2022); Arora et al. (2022); Ganesh et al. (2022).

For pure-DP (i.e. ( , 0)-DP), the seminal work Bassily et al. (2014) achieved tight upper and lower bounds Θ( d n ). As for approximate-DP (i.e. ( , δ)-DP when δ > 0), previous works Bassily et al. (2014); Steinke & Ullman (2016); Wang et al. (2017); Bassily et al. (2019) achieved the tight bound in the unconstrained case was neglected before and gathered people's attention recently. Jain & Thakurta (2014); Song et al. (2021) found a tight bound Õ( √ rank n ) for minimizing the excess empirical risk of Generalized Linear Models (GLMs, see Definition A.1 in Appendix) in the unconstrained case and evaded the curse of dimensionality, where rank is the rank of the feature matrix in the GLM problem. As a comparison, the tight bound Θ( √ d n ) holds for the constrained DP-GLM, even for the overparameterized case when rank ≤ n d. The dimension-independent result is intriguing, as modern machine learning models are usually huge, with millions to billions of parameters (dimensions). A natural question arises whether one can get similar dimension-independent results for a more general family of functions beyond GLMs. Unfortunately, Asi et al. (2021) provided a negative answer and gave an Ω( √ d n log d ) lower bound for some general convex functions. Their method chooses appropriate objective functions and utilizes one-way marginals, but the extra logarithmic term in their bound seems nontrivial to remove in the unconstrained case. Another aspect is DP-ERM in non-Euclidean settings. Most previous works in the literature consider the constrained Euclidean setting where the convex domain and (sub)gradients of objective functions have bounded 2 norms, and DP-ERM concerning the general p norm is much less well-understood. Motivated by the importance and wide applications of non-Euclidean settings, some previous works Talwar et al. (2015); Asi et al. (2021); Bassily et al. (2021b) analyzed constrained DP-ERM with respect to the general p norm with many exciting results, and there is still room for improvement in many regimes

