INDIVIDUAL PRIVACY ACCOUNTING FOR DIFFEREN-TIALLY PRIVATE STOCHASTIC GRADIENT DESCENT

Abstract

Differentially private stochastic gradient descent (DP-SGD) is the workhorse algorithm for recent advances in private deep learning. It provides a single privacy guarantee to all datapoints in the dataset. We propose an efficient algorithm to compute privacy guarantees for individual examples when releasing models trained by DP-SGD. We use our algorithm to investigate individual privacy parameters across a number of datasets. We find that most examples enjoy stronger privacy guarantees than the worst-case bound. We further discover that the training loss and the privacy parameter of an example are well-correlated. This implies groups that are underserved in terms of model utility are simultaneously underserved in terms of privacy guarantee. For example, on CIFAR-10, the average ε of the class with the lowest test accuracy is 43.6% higher than that of the class with the highest accuracy.

1. INTRODUCTION

Differential privacy is a strong notion of data privacy, enabling rich forms of privacy-preserving data analysis (Dwork & Roth, 2014) . Informally speaking, it quantitatively bounds the maximum influence of any datapoint using a privacy parameter ε, where a small value of ε corresponds to stronger privacy guarantees. Training deep models with differential privacy is an active research area (Papernot et al., 2017; Bu et al., 2020; Yu et al., 2022; Anil et al., 2021; Li et al., 2022; Golatkar et al., 2022; Mehta et al., 2022; De et al., 2022; Bu et al., 2022) . Models trained with differential privacy not only provide theoretical privacy guarantee to their data but also are more robust against empirical attacks (Bernau et al., 2019; Carlini et al., 2019; Jagielski et al., 2020; Nasr et al., 2021) . Differentially private stochastic gradient descent (DP-SGD) is the de-facto choice for differentially private deep learning (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) . DP-SGD first clips individual gradients and then adds Gaussian noise to the average of clipped gradients. Standard privacy accounting takes a worst-case approach, and provides all examples with the same privacy parameter ε. However, from the perspective of machine learning, different examples can have very different impacts on a learning algorithm (Koh & Liang, 2017; Feldman & Zhang, 2020) . For example, consider support vector machines: removing a non-support vector has no effect on the resulting model, and hence that example would have perfect privacy. In this paper, we give an efficient algorithm to accurately estimate individual privacy parameters of models trained by DP-SGD. Our privacy guarantee adapts to the training trajectory of one execution of DP-SGD to provide a precise characterization of privacy cost (see Section 2.1 for more details). Inspecting individual privacy parameters allows us to better understand example-wise impacts. It turns out that, for common benchmarks, many examples experience much stronger privacy guarantee than the worst-case bound. To illustrate this, we plot the individual privacy parameters of MNIST (LeCun et al., 1998) , CIFAR-10 (Krizhevsky, 2009), and UTKFace (Zhang et al., 2017) in Figure 1 . Experimental details, as well as more results, can be found in Section 4 and 5. The disparity in individual privacy guarantees naturally arises when running DP-SGD. To the best of our knowledge, our investigation is the first to explicitly reveal such disparity. We propose two techniques to make individual privacy accounting viable for DP-SGD. First, we maintain estimates of the gradient norms for all examples so the individual privacy costs can be computed accurately at every update. Second, we round the gradient norms with a small precision r to control the number of different privacy costs, which need to be computed numerically. We explain 0.9 1.6 2.3 3.0 3.7 4.4 5.1 5.8 6.5 7.2 7.9 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 22000 24000 Count CIFAR-10, test acc.=74.2, max i=7.8, min i=1.1 0.1 0.3 0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9 2.1 2. UTKFace-Gender, test acc.=88.2, max i=4.5, min i=0.7 0.5 0.9 1.3 1.7 2.1 2.5 2.9 3.3 3. why these two techniques are necessary in Section 2. More details of the proposed algorithm, as well as methods to release individual privacy parameters, are in Section 3. We further demonstrate a strong correlation between the privacy parameter of an example and its final training loss. We find that examples with higher training loss also have higher privacy parameters in general. This suggests that the same examples suffer a simultaneous unfairness in terms of worse privacy and worse utility. While prior works have shown that underrepresented groups experience worse utility (Buolamwini & Gebru, 2018) , and that these disparities are amplified when models are trained privately Bagdasaryan et al. ( 2019 2022), we are the first to show that the privacy guarantee and utility are negatively impacted concurrently. This is in comparison to prior work in the differentially private setting which took a worst-case perspective for privacy accounting, resulting in a uniform privacy guarantee for all training examples. For instance, when running gender classification on UTKFace, the average ε of the race with the lowest test accuracy is 25% higher than that of the race with the highest accuracy. We also study the disparity in privacy when models are trained without differential privacy, which may be of independent interest to the community. We use the success rates of membership inference attacks to measure privacy in this case and show groups with worse accuracy suffer from higher privacy risks. 1.1 RELATED WORK Several works have explored example-wise privacy guarantees in differentially private learning. Jorgensen et al. (2015) propose personalized differential privacy that provides pre-specified individual privacy parameters which are independent of the learning algorithm, e.g., users can choose different levels of privacy guarantees based on their sensitivities to privacy leakage (Mühl & Boenisch, 2022) . A recent line of works also uses the variation in example-wise sensitivities that naturally arise in learning to study example-wise privacy. Per-instance differential privacy captures the privacy parameter of a target example with respect to a fixed training set (Wang, 2019; Redberg & Wang, 2021; Golatkar et al., 2022) . Feldman & Zrnic (2021) design an individual Rényi differential privacy filter. The filter stops when the accumulated cost reaches a target budget that is defined before training. It allows examples with smaller per-step privacy costs to run for more steps. The final privacy guarantee offered by the filter is still the worst-case over all possible outputs as the predefined budget has to be independent of the algorithm outcomes. In this work, we propose output-specific differential privacy and give an efficient algorithm to compute individual guarantees of DP-SGD. We further discover that the disparity in individual privacy parameters correlates well with the disparity in utility.



acc.=97.1, max i=2.4, min i=0.3 0.5 0.9 1.3 1.7 2.1 2.5 2.9 3.3 3.7 4.1 4.5

Figure 1: Individual privacy parameters of models trained by DP-SGD. The value of δ is 1 × 10 -5 . The dashed lines indicate 10%, 30%, and 50% of datapoints. The black solid line shows the privacy parameter of the original analysis.

); Suriyakumar et al. (2021); Hansen et al. (2022); Noe et al. (

