INDIVIDUALLY FAIR GRADIENT BOOSTING

Abstract

We consider the task of enforcing individual fairness in gradient boosting. Gradient boosting is a popular method for machine learning from tabular data, which arise often in applications where algorithmic fairness is a concern. At a high level, our approach is a functional gradient descent on a (distributionally) robust loss function that encodes our intuition of algorithmic fairness for the ML task at hand. Unlike prior approaches to individual fairness that only work with smooth ML models, our approach also works with non-smooth models such as decision trees. We show that our algorithm converges globally and generalizes. We also demonstrate the efficacy of our algorithm on three ML problems susceptible to algorithmic bias.

1. INTRODUCTION

In light of the ubiquity of machine learning (ML) methods in high-stakes decision-making and support roles, there is concern about ML models reproducing or even exacerbating the historical biases against certain groups of users. These concerns are valid: there are recent incidents in which algorithmic bias has led to dire consequences. For example, Amazon recently discovered its ML-based resume screening system discriminates against women applying for technical positions (Dastin, 2018) . In response, the ML community has proposed a myriad of formal definitions of algorithmic fairness. Broadly speaking, there are two types of fairness definitions: group fairness and individual fairness (Chouldechova & Roth, 2018) . In this paper, we focus on enforcing individual fairness. At a highlevel, the idea of individual fairness is the requirement that a fair algorithm should treat similar individuals similarly. For a while, individual fairness was overlooked in favor of group fairness because there is often no consensus on which users are similar for many ML tasks. Fortunately, there is a flurry of recent work that addresses this issue (Ilvento, 2019; Wang et al., 2019; Yurochkin et al., 2020; Mukherjee et al., 2020a) . In this paper, we assume there is a fair metric for the ML task at hand and consider the task of individually fair gradient boosting. Gradient boosting, especially gradient boosted decision trees (GBDT), is a popular method for tabular data problems (Chen & Guestrin, 2016) . Unfortunately, existing approaches to enforcing individual fairness are either not suitable for training non-smooth ML models (Yurochkin et al., 2020) or perform poorly with flexible non-parametric ML models. We aim to fill this gap in the literature. Our main contributions are: 1. We develop a method to enforce individual fairness in gradient boosting. Unlike other methods for enforcing individually fairness, our approach handles non-smooth ML models such as (boosted) decision trees. 2. We show that the method converges globally and leads to ML models that are individually fair. We also show that it is possible to certify the individual fairness of the models a posteriori. 3. We show empirically that our method preserves the accuracy of gradient boosting while improving widely used group and individual fairness metrics. Consider a supervised learning problem. Let X ⊂ R d be the input space and Y be the output space. To keep things simple, we assume Y = {0, 1}, but our method readily extends to multi-class classification problems. Define Z = X × {0, 1}. We equip X with a fair metric d x that measures the similarity between inputs. The fair metric is application specific, and we refer to the literature on fair metric learning (Ilvento, 2019; Wang et al., 2019; Yurochkin et al., 2020) for ways of picking the fair metric. Our goal is to learn an ML model f : X → {0, 1} that is individually fair. Formally, we enforce distributionally robust fairness (Yurochkin et al., 2020) , which asserts that an ML model has similar accuracy/performance (measured by the loss function) on similar samples (see Definition 2.1). One  c((x 1 , y 1 ), (x 2 , y 2 )) d 2 x (x 1 , x 2 ) + ∞ • 1 {y1 =y2} . We see that c((x 1 , y 1 ), (x 2 , y 2 )) is small iff x 1 and x 2 are similar (in the fair metric d x ) and y 1 = y 2 . In other words, c is small iff two similar examples are assigned the same output. Define the optimal transport distance W (with transport cost c) on probability distributions on Z: W (P 1 , P 2 ) inf Π∈C(P1,P2) Z×Z c(z 1 , z 2 )dΠ(z 1 , z 2 ), where C(P 1 , P 2 ) is the set of couplings between P 1 and P 2 (distributions on Z × Z whose marginals are P 1 and P 2 ). This optimal transport distance lifts the fair metric on (points in) the sample space to distributions on the sample space. Two distributions are close in this optimal transport distance iff they assign mass to similar areas of the sample space Z. Finally, define the adversarial risk function L r (f ) sup P : W (P,P * )≤ E P [ (f (X), Y )], where P * is the data generating distribution and > 0 is a small tolerance parameter. The adversarial risk function looks for distributions on the sample space that (i) are similar to the data generating distribution and (ii) increases the risk of the ML model f . This reveals differential performance of the ML model on similar samples. This search for differential performance is captured by the notion of distributionally robust fairness: Definition 2.1 (distributionally robust fairness (DRF) (Yurochkin et al., 2020) ). An ML model h : X → Y is ( , δ)-distributionally robustly fair (DRF) WRT the fair metric d x iff sup P :W (P,Pn)≤ Z (z, h)dP (z) ≤ δ. (2.3) In light of the preceding developments, a natural cost function for training individually fair ML models is the adversarial cost function: L e (f ) sup P : W (P,Pn)≤ E P [ (f (X), Y )], (2.4) where P n is the empirical distribution of the training data. This is the empirical counterpart of (2.2), and it works well for training smooth ML models (Yurochkin et al., 2020) . Unfortunately, (2.4) is hard to evaluate for non-smooth ML models: it is defined as the optimal value of an optimization problem, but the gradient ∂ x (f (x), y) is not available because the ML model f is non-smooth.



way to accomplish this is adversarial training(Yurochkin et al., 2020; Yurochkin & Sun, 2020). Unfortunately, adversarial training relies on the smoothness of the model (with respect to the inputs), so it cannot handle non-smooth ML models (e.g. decision trees). We address this issue by considering a restricted adversarial cost function that only searches over the training examples (instead of the entire input space) for similar examples that reveal violations of individual fairness. As we shall see, this restricted adversarial cost function is amenable to functional gradient descent, which allows us to develop a gradient boosting algorithm.

