SIMPLICITY BIAS LEADS TO AMPLIFIED PERFORMANCE DISPARITIES

Abstract

The simple idea that not all things are equally difficult has surprising implications when applied in a fairness context. In this work we explore how "difficulty" is model-specific, such that different models find different parts of a dataset challenging. When difficulty correlates with group information, we term this difficulty disparity. Drawing a connection with recent work exploring the inductive bias towards simplicity of SGD-trained models, we show that when such a disparity exists, it is further amplified by commonly-used models. We quantify this amplification factor across a range of settings aiming towards a fuller understanding of the role of model bias. We also present a challenge to the simplifying assumption that "fixing" a dataset is sufficient to ensure unbiased performance.

1. INTRODUCTION

Without actually training, understanding what a model will find challenging is far from trivial. A certain dataset may be hard for one model but not for another (Wolpert & Macready, 1997) . For a given model, two classes may be easily separable, while for another they may be hard to distinguish. Given this, it follows naturally that "difficulty" is a function of both data and model, such that we can't properly account for difficulty by analyzing the dataset alone. In the context of fairness in machine learning, for a given task, a data-model pair may be more difficult for one social group than another, leading to disparate impact (Barocas & Selbst, 2016). 



For example, Buolamwini & Gebru's 2018 audit of commercial image recognition systems finds

Figure 1: What is difficulty amplification? (a) Consider a binary classification of circles and triangles. Above y = 0 (light gray background) we have a simple group which is linearly separable. Below y = 0 (dark gray) we have a more complex group with a non-linear decision boundary. (b) Illustration of test accuracy when training on the simple group only (light gray) and the complex group only (dark gray). As expected we obtain better accuracy on the simple group. (c) However, when training on both groups at once, our model exacerbates the difference: the observed accuracy disparity d (height of pink area) exceeds the estimated accuracy disparity from individual group training d (height of green area). When d > d, we call this difficulty amplification.

