FAIRNESS-AWARE CONTRASTIVE LEARNING WITH PARTIALLY ANNOTATED SENSITIVE ATTRIBUTES

Abstract

Learning high-quality representation is important and essential for visual recognition. Unfortunately, traditional representation learning suffers from fairness issues since the model may learn information of sensitive attributes. Recently, a series of studies have been proposed to improve fairness by explicitly decorrelating target labels and sensitive attributes. Most of these methods, however, rely on the assumption that fully annotated labels on target variable and sensitive attributes are available, which is unrealistic due to the expensive annotation cost. In this paper, we investigate a novel and practical problem of Fair Unsupervised Representation Learning with Partially annotated Sensitive labels (FURL-PS). FURL-PS has two key challenges: 1) how to make full use of the samples that are not annotated with sensitive attributes; 2) how to eliminate bias in the dataset without target labels. To address these challenges, we propose a general Fairness-aware Contrastive Learning (FairCL) framework consisting of two stages. Firstly, we generate contrastive sample pairs, which share the same visual information apart from sensitive attributes, for each instance in the original dataset. In this way, we construct a balanced and unbiased dataset. Then, we execute fair contrastive learning by closing the distance between representations of contrastive sample pairs. Besides, we also propose an unsupervised way to balance the utility and fairness of learned representations by feature reweighting. Extensive experimental results illustrate the effectiveness of our method in terms of fairness and utility, even with very limited sensitive attributes and serious data bias.

1. INTRODUCTION

Learning powerful representation takes an important role in visual recognition, and there are a lot of works proposed to learn visual representations (Bengio et al., 2013; Kolesnikov et al., 2019; Wang et al., 2020a; Liu et al., 2022) . Among them, contrastive learning achieves state-of-the-art performance on various vision tasks (Tian et al., 2020; Chuang et al., 2020) . Contrastive learning first generates views from original images by random data augmentation, and the views from the same image are defined as positive samples. Then the model can learn effective representations by closing the distance between representations of positive samples, while being protected from mode collapse via an additional module such as negative samples (Chen et al., 2020a; He et al., 2020; Chen et al., 2020b ), momentum update (Grill et al., 2020) , and stopping gradient (Chen & He, 2021). Unfortunately, traditional representation learning methods ignore potential fairness issues, which becomes an increasing concern as recognition systems are widely used in the real world (Zemel et al., 2013; Madras et al., 2018; Creager et al., 2019; Lv et al., 2023) . For example, the model trained by contrastive learning may learn the information of sensitive attributes (e.g., gender, race) by using it as a shortcut to minimize the distance between representations of positive samples in the training stage, since the positive samples have the same sensitive attributes. As a result, decisions based on biased representation models may discriminate against certain groups or individuals in practice, by using spurious correlations between predictive target and sensitive attributes (Wang et al., 2020b; Park et al., 2021; Zhang et al., 2021) . Therefore, how to develop a fair representation model is of paramount importance for both academic research and real applications. Most of existing works achieve fairness via decorrelating target labels and sensitive attributes explicitly, which rely on the data annotations (Mehrabi et al., 2021; Wu et al., 2022; Zhang et al., 2022; Zhu et al., 2022) . However, assuming that all data have fully annotated labels can be unrealistic (Liu et al., 2016; Zhang et al., 2020b; Shao et al., 2021; Jung et al., 2022; Song et al., 2023) . In many real scenarios, the target labels and even downstream tasks are not provided, and all we have are images and limited annotations of sensitive attributes. Data labels require additional expensive cost of human annotations, which naturally leads us to ask the following question: Can we train a fair unsupervised representation model with only partially annotated sensitive attributes? In this paper, we investigate a practical and novel problem of Fair Unsupervised Representation Learning with Partially annotated Sensitive attributes (FURL-PS). Our goal is to utilize the images and limited sensitive labels to learn visual representations that can be used for various downstream tasks of visual recognition, while achieving fairness by being minimally correlated with sensitive attributes. It is challenging to solve the proposed problem. Firstly, most samples are not labeled with sensitive attributes. A natural idea is to pseudo-label the unlabeled data by a sensitive attribute classifier. However, it is not advisable to train a representation model on the data with pseudo-sensitive labels, since the noises in pseudo labels may severely affect the fairness performance. Secondly, there may be data imbalance between demographic groups. Assuming that the female group has a large proportion of samples of blond hair, while the male group has the opposite proportion. As a result, the models trained on the above biased data may learn spurious correlation between gender and blond hair. Unfortunately, it is difficult to balance the data distribution of different groups without the prior of downstream tasks or annotated target labels. Generally, FURL-PS problem has two main challenges: 1) How to make full use of the data that are not annotated with sensitive attributes? 2) How to balance the possible agnostic bias in data without target labels? To address these challenges, our idea is to construct a balanced dataset annotated with sensitive labels based on the original dataset, and then train a representation model with fair contrastive learning on the unbiased dataset. We propose a two-stage Fairness-aware Contrastive Learning (FairCL) framework to implement the above idea. In the first stage, we design a semi-supervised learning algorithm to train the image attribute editor with limited sensitive labels, which is used to edit the pre-defined sensitive attributes of a given image. In the second stage, we train a representation model by fair contrastive learning with balanced augmentation. Specifically, based on the image attribute editor, we can generate contrastive sample pairs, which share the same visual information apart from sensitive attributes (e.g., male and female), for each sample in the original dataset. By closing the distance between representations of contrastive sample pairs, the model can learn powerful and fair representations. Our approach has two advantages: 1) we can get the utmost out of unlabeled images by generating samples with given sensitive attributes from them; 2) the augmented dataset is unbiased, since it consists of contrastive sample pairs and thus the data proportions are naturally balanced for different demographic groups. Furthermore, we also develop an unsupervised way to balance the utility and fairness of learned representations by feature reweighting. We validate the effectiveness of our method on two facial attribute recognition datasets: CelebA (Liu et al., 2018) and UTK-Face (Zhang et al., 2017) . Extensive experimental results show that the proposed method outperforms the existing unsupervised learning methods in terms of both classification accuracy and fairness, and even achieves comparable performance with the semi-supervised methods that require annotations on the target labels. Besides, our method is robust to the ratio of sensitive labels and severity of data bias. Furthermore, we also show the extensibility of our general framework to different contrastive learning algorithms through experiments. Main Contributions: 1) To the best our knowledge, we are the first one to propose the practical and challenging problem of Fair Unsupervised Representation Learning with only Partially annotated Sensitive attributes (FURL-PS). 2) We develop the Fairness-aware Contrastive Learning (FairCL) framework to solve the proposed problem, which can be compatible with all of contrastive learning algorithms to learn a fair and powerful representation model. 3) Extensive experiments illustrate the effectiveness of our proposed method in terms of fairness and utility.

