LEARNING WITH INSTANCE-DEPENDENT LABEL NOISE: MAINTAINING ACCURACY AND FAIRNESS

Abstract

Incorrect labels hurt model performance when the model overfits to noise. Many state-of-the-art approaches that address label noise assume that label noise is independent from the input features. In practice, however, label noise is often feature or instance-dependent, and therefore is biased (i.e., some instances are more likely to be mislabeled than others). Approaches that ignore this dependence can produce models with poor discriminative performance, and depending on the task, can exacerbate issues around fairness. In light of these limitations, we propose a two-stage approach to learn from datasets with instance-dependent label noise. Our approach utilizes alignment points, a small subset of data for which we know the observed and ground truth labels. On many tasks, our approach leads to consistent improvements over the state-of-the-art in discriminative performance (AU-ROC) while maintaining model fairness (area under the equalized odds curve, AUEOC). For example, when predicting acute respiratory failure onset on the MIMIC-III dataset, the harmonic mean of the AUROC and AUEOC of our approach is 0.84 (SD 0.01) while that of the next best baseline is 0.81 (SD 0.01). Overall, our approach leads to learning more accurate and fair models compared to existing approaches in the presence of instance-dependent label noise.

1. INTRODUCTION

Datasets used to train machine learning models can contain incorrect labels (i.e., label noise). While label noise is widely studied, the majority of past work focuses on when the noise is independent from an instance's features (i.e., instance-independent label noise) [Song et al. (2020) ]. However, label noise is sometimes biased and depends on an instance's features (i.e., instance-dependent) [Wei et al. (2022b) ], leading to different noise rates within subsets of the data. This results in model overfitting, and in tasks where the dataset contains instances from different groups corresponding to some sensitive attribute, this can also lead to disparities in performance [Liu (2021) ]. For example, consider the task of predicting cardiovascular disease among patients admitted to a hospital. Compared to male patients, female patients may be more likely to be misdiagnosed [Maserejian et al. (2009) ] and thus mislabeled, potentially leading to worse predictions for female patients. [Berthon et al. (2021) ]. In some settings, this approach can have a negative effect on model fairness. For example, when instances represent individuals belonging to subgroups defined by a sensitive attribute, approaches that filter out mislabeled individuals could ignore a disproportionately higher number of individuals from subgroups with more label noise. While relabeling approaches use all available data, they can be sensitive to assumptions around the noise distribution [Ladouceur et al. (2007) ]. In the second category, current approaches rely on objective functions that are less prone to overfitting to the noise, while using all of the data and



Although instance-dependent label noise has recently received more attention [Cheng et al. (2020b); Xia et al. (2020); Wang et al. (2021a)], the effect of these approaches on model fairness has been relatively understudied [Liu (2021)]. Here, we address the limitations of current approaches and propose a novel method for learning with instance-dependent label noise, specifically examining how modeling assumptions affect existing issues around model fairness. Broadly, current work addressing instance-dependent label noise falls into one of two categories: 1) that which learns to identify mislabeled instances [Cheng et al. (2020a); Xia et al. (2022); Zhu et al. (2022a)], and 2) that which learns to optimize a noise-robust objective function [Feng et al. (2020); Wei et al. (2022a)]. In the first category, instances identified as mislabeled are either filtered out [Kim et al. (2021)] or relabeled

