FAIRNESS-AWARE CONTRASTIVE LEARNING WITH PARTIALLY ANNOTATED SENSITIVE ATTRIBUTES

Abstract

Learning high-quality representation is important and essential for visual recognition. Unfortunately, traditional representation learning suffers from fairness issues since the model may learn information of sensitive attributes. Recently, a series of studies have been proposed to improve fairness by explicitly decorrelating target labels and sensitive attributes. Most of these methods, however, rely on the assumption that fully annotated labels on target variable and sensitive attributes are available, which is unrealistic due to the expensive annotation cost. In this paper, we investigate a novel and practical problem of Fair Unsupervised Representation Learning with Partially annotated Sensitive labels (FURL-PS). FURL-PS has two key challenges: 1) how to make full use of the samples that are not annotated with sensitive attributes; 2) how to eliminate bias in the dataset without target labels. To address these challenges, we propose a general Fairness-aware Contrastive Learning (FairCL) framework consisting of two stages. Firstly, we generate contrastive sample pairs, which share the same visual information apart from sensitive attributes, for each instance in the original dataset. In this way, we construct a balanced and unbiased dataset. Then, we execute fair contrastive learning by closing the distance between representations of contrastive sample pairs. Besides, we also propose an unsupervised way to balance the utility and fairness of learned representations by feature reweighting. Extensive experimental results illustrate the effectiveness of our method in terms of fairness and utility, even with very limited sensitive attributes and serious data bias.

1. INTRODUCTION

Learning powerful representation takes an important role in visual recognition, and there are a lot of works proposed to learn visual representations (Bengio et al., 2013; Kolesnikov et al., 2019; Wang et al., 2020a; Liu et al., 2022) . Among them, contrastive learning achieves state-of-the-art performance on various vision tasks (Tian et al., 2020; Chuang et al., 2020) . Contrastive learning first generates views from original images by random data augmentation, and the views from the same image are defined as positive samples. Then the model can learn effective representations by closing the distance between representations of positive samples, while being protected from mode collapse via an additional module such as negative samples (Chen et al., 2020a; He et al., 2020; Chen et al., 2020b) , momentum update (Grill et al., 2020) , and stopping gradient (Chen & He, 2021). Unfortunately, traditional representation learning methods ignore potential fairness issues, which becomes an increasing concern as recognition systems are widely used in the real world (Zemel et al., 2013; Madras et al., 2018; Creager et al., 2019; Lv et al., 2023) . For example, the model trained by contrastive learning may learn the information of sensitive attributes (e.g., gender, race) by using it as a shortcut to minimize the distance between representations of positive samples in the training stage, since the positive samples have the same sensitive attributes. As a result, decisions based on biased representation models may discriminate against certain groups or individuals in practice, by using spurious correlations between predictive target and sensitive attributes (Wang

