

Abstract

We show that with improved training, the standard approach for differentially private GANs -updating the discriminator with noisy gradients -achieves or competes with state-of-the-art results for private image synthesis. Existing instantiations of this approach neglect to consider how adding noise only to discriminator updates disrupts the careful balance between generator and discriminator necessary for successful GAN training. We show that a simple fix -taking more discriminator steps between generator steps -restores parity and improves training. Furthermore, with the goal of restoring parity between the generator and discriminator, we experiment with further modifications to improve discriminator training and see further improvements in generation quality. For MNIST at ε = 10, our private GANs improve the record FID from 48.4 to 13.0, and record downstream classifier accuracy from 83.2% to 95.0%.

1. INTRODUCTION

Differential privacy (DP) (Dwork et al., 2006b) has emerged as a compelling approach for training machine learning models on sensitive data. However, incorporating DP requires significant changes to the training process. Notably, it prevents the modeller from working directly with private data, complicating debugging and exploration. Furthermore, the modeller can no longer interact with a private dataset after exhausting their allocated privacy budget. One approach to alleviate these issues is by producing differentially private synthetic data, which can be plugged directly into existing machine learning pipelines, without further concern for privacy. A recent line of work studies leveraging deep generative models to produce DP synthetic data. Early efforts focused on privatizing generative adversarial networks (GANs) (Goodfellow et al., 2014 ) by using differentially private stochastic gradient descent (DPSGD) (Abadi et al., 2016) to update the GAN discriminator -an approach referred to as DPGAN (Xie et al., 2018; Beaulieu-Jones et al., 2019; Torkzadehmahani et al., 2019) . However, follow-up work has significantly departed from this baseline DPGAN approach, either in terms of: (a) the privatization scheme, in favor of approaches based on subsample-and-aggregate which divide the data into ≥ 1000 disjoint partitions and train teacher discriminators separately on each one (Jordon et al., 2019; Long et al., 2021; Chen et al., 2020; Wang et al., 2021) ; or (b) the generative modelling framework altogether, opting instead to minimize notions of statistical distance between real and generated data, such as maximum mean discrepancy (Harder et al., 2021; Vinaroz et al., 2022) , or Sinkhorn divergences (Cao et al., 2021) . For labelled image synthesis, these custom generative models designed specifically for privacy fall short of GANs when evaluated at their non-private limits (ε → ∞), suggesting limited scalability to larger, higher-resolution datasets.foot_0 On the other hand, the literature corroborates that under modest privacy budgets, these departures from the baseline DPGAN lead to significant improvements in generation quality. Proposed explanations attribute these results to inherent limitations of the DPGAN framework, suggesting that either: (a) privatizing discriminator training is sufficient for privacy, but may be overkill when only the generator needs to be released (Long et al., 2021) ; or (b) adversarial objectives may be unsuited for training under privacy (Cao et al., 2021) .



For example, the record FID for MNIST at ε = 10 is 48.4 (Cao et al., 2021). When evaluated at ε = ∞, their method achieves an FID of 43.4. Our non-private GANs obtain an FID of 3.2.

