INDIVIDUAL PRIVACY ACCOUNTING FOR DIFFEREN-TIALLY PRIVATE STOCHASTIC GRADIENT DESCENT

Abstract

Differentially private stochastic gradient descent (DP-SGD) is the workhorse algorithm for recent advances in private deep learning. It provides a single privacy guarantee to all datapoints in the dataset. We propose an efficient algorithm to compute privacy guarantees for individual examples when releasing models trained by DP-SGD. We use our algorithm to investigate individual privacy parameters across a number of datasets. We find that most examples enjoy stronger privacy guarantees than the worst-case bound. We further discover that the training loss and the privacy parameter of an example are well-correlated. This implies groups that are underserved in terms of model utility are simultaneously underserved in terms of privacy guarantee. For example, on CIFAR-10, the average ε of the class with the lowest test accuracy is 43.6% higher than that of the class with the highest accuracy.

1. INTRODUCTION

Differential privacy is a strong notion of data privacy, enabling rich forms of privacy-preserving data analysis (Dwork & Roth, 2014) . Informally speaking, it quantitatively bounds the maximum influence of any datapoint using a privacy parameter ε, where a small value of ε corresponds to stronger privacy guarantees. Training deep models with differential privacy is an active research area (Papernot et al., 2017; Bu et al., 2020; Yu et al., 2022; Anil et al., 2021; Li et al., 2022; Golatkar et al., 2022; Mehta et al., 2022; De et al., 2022; Bu et al., 2022) . Models trained with differential privacy not only provide theoretical privacy guarantee to their data but also are more robust against empirical attacks (Bernau et al., 2019; Carlini et al., 2019; Jagielski et al., 2020; Nasr et al., 2021) . Differentially private stochastic gradient descent (DP-SGD) is the de-facto choice for differentially private deep learning (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) . DP-SGD first clips individual gradients and then adds Gaussian noise to the average of clipped gradients. Standard privacy accounting takes a worst-case approach, and provides all examples with the same privacy parameter ε. However, from the perspective of machine learning, different examples can have very different impacts on a learning algorithm (Koh & Liang, 2017; Feldman & Zhang, 2020) . For example, consider support vector machines: removing a non-support vector has no effect on the resulting model, and hence that example would have perfect privacy. In this paper, we give an efficient algorithm to accurately estimate individual privacy parameters of models trained by DP-SGD. Our privacy guarantee adapts to the training trajectory of one execution of DP-SGD to provide a precise characterization of privacy cost (see Section 2.1 for more details). Inspecting individual privacy parameters allows us to better understand example-wise impacts. It turns out that, for common benchmarks, many examples experience much stronger privacy guarantee than the worst-case bound. To illustrate this, we plot the individual privacy parameters of MNIST (LeCun et al., 1998) , CIFAR-10 (Krizhevsky, 2009), and UTKFace (Zhang et al., 2017) in Figure 1 . Experimental details, as well as more results, can be found in Section 4 and 5. The disparity in individual privacy guarantees naturally arises when running DP-SGD. To the best of our knowledge, our investigation is the first to explicitly reveal such disparity. We propose two techniques to make individual privacy accounting viable for DP-SGD. First, we maintain estimates of the gradient norms for all examples so the individual privacy costs can be computed accurately at every update. Second, we round the gradient norms with a small precision r to control the number of different privacy costs, which need to be computed numerically. We explain 1

