DISPARATE IMPACT IN DIFFERENTIAL PRIVACY FROM GRADIENT MISALIGNMENT

Abstract

As machine learning becomes more widespread throughout society, aspects including data privacy and fairness must be carefully considered, and are crucial for deployment in highly regulated industries. Unfortunately, the application of privacy enhancing technologies can worsen unfair tendencies in models. In particular, one of the most widely used techniques for private model training, differentially private stochastic gradient descent (DPSGD), frequently intensifies disparate impact on groups within data. In this work we study the fine-grained causes of unfairness in DPSGD and identify gradient misalignment due to inequitable gradient clipping as the most significant source. This observation leads us to a new method for reducing unfairness by preventing gradient misalignment in DPSGD.

1. INTRODUCTION

The increasingly widespread use of machine learning throughout society has brought into focus social, ethical, and legal considerations surrounding its use. In highly regulated industries, such as healthcare and banking, regional laws and regulations require data collection and analysis to respect the privacy of individuals. 1 Other regulations focus on the fairness of how models are developed and used. 2 As machine learning is progressively adopted in highly regulated industries, the privacy and fairness aspects of models must be considered at all stages of the modelling lifecycle. There are many privacy enhancing technologies including differential privacy (Dwork et al., 2006) , federated learning (McMahan et al., 2017) , secure multiparty computation (Yao, 1986) , and homomorphic encryption (Gentry, 2009) that are used separately or jointly to protect the privacy of individuals whose data is used for machine learning (Choquette-Choo et al., 2020; Adnan et al., 2022; Kalra et al., 2021) . The latter three technologies find usage in sharing schemes and can allow data to be analysed while preventing its exposure to the wrong parties. However, the procedures usually return a trained model which itself can leak private information (Carlini et al., 2019) . On the other hand, differential privacy (DP) focuses on quantifying the privacy cost of disclosing aggregated information about a dataset, and can guarantee that nothing is learned about individuals that could not be inferred from population-level correlations (Jagielski et al., 2019) . Hence, DP is often used when the results of data analysis will be made publicly available, for instance when exposing the outputs of a model, or the results of the most recent US census (Abowd, 2018) . Not only must privacy be protected for applications in regulated industries, models must be fair. While there is no single definition that captures what it means to be fair, with regards to modelbased decision making fairness may preclude disparate treatment or disparate impact (Mehrabi et al., 2021) . Disparate treatment is usually concerned with how models are applied across populations, whereas disparate impact can arise from biases in datasets that are amplified by the greedy nature of loss minimization algorithms (Buolamwini & Gebru, 2018) . Differences in model performance across protected groups can result in a significant negative monetary, health, or societal impact for individuals who are discriminated against (Chouldechova & Roth, 2020). Unfortunately, it has been observed that disparate impact can be exacerbated by applying DP in machine learning (Bagdasaryan et al., 2019) . Applications of DP always come with a privacy-utility tradeoff, where stronger guarantees of privacy negatively impact the usefulness of results -model performance in this context (Dwork & Roth, 2014) . Underrepresented groups within the population can experience disparity in the cost of adding privacy, hence, fairness concerns are a major obstacle to deploying models trained with DP. The causes of unfairness in DP depend on the techniques used, but are not fully understood. For the most widely used technique, differentially private stochastic gradient descent (DPSGD), two sources of error are introduced that impact model utility. Per-sample gradients are clipped to a fixed upper bound on their norm, then noise is added to the averaged gradient. Disparate impact from DPSGD was initially hypothesized to be rooted in unbalanced datasets (Bagdasaryan et al., 2019) , though counterexamples were found by Xu et al. (2021) . Recent research claims disparate impact to be caused by incommensurate clipping errors across groups, in turn effected by a large difference in average group gradient norms (Xu et al., 2021; Tran et al., 2021a) . In this work we highlight the disparate impact of gradient misalignment. In particular, we claim that the most significant cause of disparate impact is the difference in the direction of the unclipped and clipped gradients, which in turn can be caused by aggressive clipping and imbalances of gradient norms between groups. Our analysis of direction errors leads to a variant of DPSGD with properly aligned gradients. We explore this alternate method in relation to disparate impact and show that it not only significantly reduces the cost of privacy across all protected groups, it also reduces the difference in cost of privacy for all groups. Hence, it removes disparate impact and is more effective than previous proposals in doing so. On top of this, it is the only approach which does not require access to protected group labels, and thereby avoids disparate treatment of groups. In summary we: • Conduct a more fine-grained analysis of disparate impact in DPSGD, and demonstrate gradient misalignment to be the most significant cause; • Identify an existing algorithm, previously undiscussed in the fairness context, which properly aligns gradients, and show it reduces disparate impact and disparate treatment; • Improve the utility of said algorithm via two alterations; • Experimentally verify that aligning gradients is more successful at mitigating disparate impact than previous approaches.

2. RELATED WORK

Privacy and Fairness: While privacy and fairness have been extensively studied separately, recently their interactions have come into focus. et al., 2019; Farrand et al., 2020) . Chang & Shokri (2021) empirically found that imposing fairness constraints on private models could lead to higher privacy loss for certain groups. We consider crossmodel fairness where the cost of adding privacy to a non-private model must be fairly distributed between groups. Adaptive Clipping: Many variations on the clipping procedure in DPSGD have been proposed to improve properties other than fairness. Adaptive clipping comes in many forms, but usually tunes the clipping threshold during training to provide better privacy-utility tradeoffs and convergence (Andrew et al., 2021; Pichapati et al., 2019) . The convergence of DPSGD connects to the symmetry properties of the distribution of gradients (Chen et al., 2020) which are affected by clipping.



Examples of laws governing data privacy include the General Data Protection Regulation in Europe, Health Insurance Portability and Accountability Act in the USA, and Personal Information Protection and Electronic Documents Act in Canada. In the USA, fair lending laws including the Fair Housing Act, and Equal Credit Opportunity Act prohibit discrimination based on protected characteristics such as race, age, and sex.



Ekstrand et al. (2018) considered the intersection of privacy and fairness for several definitions of privacy. This research gained new urgency when Bagdasaryan et al. (2019) observed that DPSGD exacerbated existing disparity in model accuracy on underrepresented groups. Disparate impact due to DP was further observed in Pujol et al. (2020) and Farrand et al. (2020) for varying levels of group imbalance. Using an adversarial definition of privacy, Jaiswal & Mower Provost (2020) found that overrepresented groups can incur higher privacy costs. Similar examples were shown in Xu et al. (2021) for DPSGD, and disparate impact was linked to groups having larger gradient norms.

