FIGHTING FIRE WITH FIRE: CONTRASTIVE DEBIASING WITHOUT BIAS-FREE DATA VIA GENERATIVE BIAS-TRANSFORMATION

Abstract

Despite their remarkable ability to generalize with over-capacity networks, deep neural networks often abuse bias instead of using the actual task-related information for discriminative tasks. Since such shortcuts are only effective within the collected dataset, the resulting biased model underperforms on real-world inputs. To counteract the influence of bias, existing methods either exploit auxiliary information which is rarely obtainable in practice or sift handful bias-free samples to emphasize them for debiasing. However, such presumptions are not always guaranteed, and the existing methods could break down due to the unmet presumptions. In this paper, we propose Contrastive Debiasing via Generative Bias-transformation (CDvG) which is capable of operating without exploiting bias labels and bias-free samples explicitly. Motivated by our observation that not only discriminative models but also image translation models tend to focus on the easy-to-learn bias, CDvG employs a image translation model to transform the bias to another mode of bias while preserving task-relevant information. Through contrastive learning, we set transformed biased views against another, learning bias-invariant representations. Especially, as the bias has a stronger correlation or is easier to perceive compared to the signal, the translation model is more likely to be a bias translation model, resulting in better debiasing effect. Experimental results demonstrate that CDvG outperforms the state-of-the-arts, especially when bias-free samples are extremely scarce.

1. INTRODUCTION

Recent advances in deep learning have showcased that deep neural networks (DNN) are capable of reaching state-of-the-art performance in various fields of machine learning, such as computer vision (He et al., 2015) , natural language processing (Brown et al., 2020) , reinforcement learning (Mnih et al., 2016) and more. However, it is also known that the over-parameterized nature of DNNs not only exposes them to general overfitting but also renders them susceptible to biases present in collected datasets (Torralba & Efros, 2011) which are detrimental to the generalizability. In supervised learning, neural networks tend to prefer shortcut solutions based on biases rather than real signal (Zhu et al., 2017b; Li et al., 2018) . Since spurious correlations do not provide task-related information, DNNs that use these biases will ultimately fail on future data. For instance, a classifier trained to identify car racing images using a dataset dominated by track will exploit the track road information. However, the classifier will fail to exhibit the same performance on images of off-road rallies. To this end, debiasing is imperative in utilizing DNNs for real-world applications. A tautological solution to the bias problem is to construct a bias-free dataset from the start. However, curating a dataset devoid of all bias is extremely costly at best, and generally infeasible. A more practical attempt at neutralizing dataset bias is to fortify a dataset with explicit supervision with regards to the bias (Kim et al., 2019a; Sagawa et al., 2019) . However, additional expenditure of human labor in procuring such information cannot be avoided, which renders the option less appealing. In most cases where such explicit supervision for bias is absent, the following two lines of works are recently proposed. One line of works mitigates the influence of bias by leveraging the bias type (e.g. texture) (Bahng et al., 2020; Geirhos et al., 2019; Wang et al., 2019; Hong & Yang, 2021) to design bias-oriented auxiliary models or to augment texture-perturbed samples. However, such prior knowledge of bias is by no means guaranteed, and even with such information, designing bias-oriented architectures is not always straightforward. Instead, another line of works leverages an empirical prior knowledge that malignant biases are usually learned faster than task-relevant features (Li & Vasconcelos, 2019; Nam et al., 2020; Lee et al., 2021; Kim et al., 2021) . However, these methods tend to break down in regimes where bias-free samples are extremely scarce or absent (See Section 4.1). To tackle these shortcomings, we develop a general debiasing method capable of operating even in regimes devoid of bias-free data, thus not reliant on any presumptions regarding the biases ¯. In this paper, we propose Contrastive Debiasing via Generative Bias-transformation (CDvG), which contrasts the bias modes within the dataset against each other to attenuate the influence of bias while effectively learning the task-relevant information contained in all samples. Through preliminary experiments, we find that image-to-image translation models favor learning malignant biases over task-relevant signals, as discriminative models are known to do so (See Section 3.2). Motivated by the observation, we train a biased image translation model without bias labels that learns the bias distribution over the signal, obtaining capabilities to transform the bias feature of a given input to another bias. Using the learned bias-translation model, we synthesize novel views with altered bias features. Fighting fire with fire, we pit one bias type against another via contrastive learning. By maximizing agreement between the views with different biases, the model is encouraged to learn bias-invariant representations. Unlike existing methods, CDvG does not require explicit supervision, domain knowledge, or other meta-knowledge -the existence of bias-free samples. Our contributions are three-fold: • We experimentally observe that certain image translation models are also prone to consider conspicuous but incidental information such as background or texture first rather than task-related information, as discriminative models do (Sec. 3). • We propose Contrastive Debiasing via Generative Bias-transformation (CDvG) with generative view transformation that transforms bias factors of input images (Sec. 4). CDvG does not presume the existence of bias labels, bias type information, or even the existence



Figure 1: Illustration of our Contrastive Debiasing via Generative Bias-transformation (CDvG).

