NUISANCES VIA NEGATIVA: ADJUSTING FOR SPURIOUS CORRELATIONS VIA DATA AUGMENTATION

Abstract

There exist features that are related to the label in the same way across different settings for that task; these are semantic features or semantics. Features with varying relationships to the label are nuisances. For example, in detecting cows from natural images, the shape of the head is a semantic and because images of cows often have grass backgrounds but not always, the background is a nuisance. Relationships between a nuisance and the label are unstable across settings and, consequently, models that exploit nuisance-label relationships face performance degradation when these relationships change. Direct knowledge of a nuisance helps build models that are robust to such changes, but knowledge of a nuisance requires extra annotations beyond the label and the covariates. In this paper, we develop an alternative way to produce robust models by data augmentation. These data augmentations corrupt semantic information to produce models that identify and adjust for where nuisances drive predictions. We study semantic corruptions in powering different robust-modeling methods for multiple out-of distribution (OOD) tasks like classifying waterbirds, natural language inference, and detecting Cardiomegaly in chest X-rays.

1. INTRODUCTION

Relationships between the label and the covariates can change across data collected at different places and times. For example, in classifying animals, data collected in natural habitats have cows appear on grasslands, while penguins appear on backgrounds of snow; these animal-background relationships do not hold outside natural habitats (Beery et al., 2018; Arjovsky et al., 2019) . Some features, like an animal's shape, are predictive of the label across all settings for a task; these are semantic features, or semantics in short. Other features with varying relationships with the label, like the background, are nuisances. Even with semantics present, models trained via empirical risk minimization (ERM) can predict using nuisances and thus fail to generalize (Geirhos et al., 2020) . Models that rely only on the semantic features perform well even when the nuisance-label relationship changes, unlike models that rely on nuisances. Many methods exist to build models robust to changing nuisance-label relationships (Mahabadi et al., 2019; Makar et al., 2022; Liu et al., 2021; Puli et al., 2022; He et al., 2019) ; we call these spurious-correlation avoiding methods (SCAMs). These methods broadly fall into two classes: 1) methods that assume access to nuisances, like Nuisance-Randomized Distillation (NURD) (Puli et al., 2022) , debiased focus loss (DFL), product of experts (POE) (Mahabadi et al., 2019) , and 2) methods that rely on assumptions about ERMtrained models relying on nuisances, like Just Train Twice (JTT) (Liu et al., 2021) . We point out a commonality between the two classes of methods: a model that predicts the label from the nuisance called a biased model, that are built using extra annotations or assumptions. Intuitively, biased models play a role in building robust predictive models by providing a way to detect when the nuisance can influence predictions. How do we build biased models without extra annotations in the form of nuisances being known in the training data or assumptions about ERM-trained models relying on nuisances? In this work, we build robust models from a different and complementary source of assumptions: knowledge about semantics. Imagine using data augmentation to corrupt semantics in the covariates -if the resulting semantic-corrupted input can still predict the label, the prediction must rely on nuisances, thereby providing a window into nuisances that can be used to build a biased model. Designing a data augmentation that corrupts semantics is easy. For example, replacing the covariates with random noise would fully corrupt the semantics. However, after such a corruption there is nothing that predicts the label meaning no nuisance information would be identified. A better semantic corruption would corrupt the semantics, while preserving some nuisances. This preservation is possible when semantics and nuisances appear differently in the covariates; we call such a difference a separation. We identify different separations to develop semantic corruptions for object recognition and natural language inference (NLI). The first separation is when semantics are global and nuisances are local. Formally, global semantics are position-dependent functions of the subsets of the covariates (patches in images, or words in sentences), while local nuisances are position-independent. For example, in recognizing cows, the shape of the animal structures the distant patches where the cow's eyes, ears, tail, hooves appear; nuisances like grass can appear anywhere without structure. Due to positional-dependence, randomizing positions of subsets of covariates corrupts global semantics; however, position-independent local nuisances are retained. Under the global/local separation, we corrupt semantics via patch randomization (PATCH-RAND) for images and n-gram randomization (NGRAM-RAND) for NLI. The second separation is when certain parts of the input are required for semantics. For example in chest X-rays, lungs appear in the center, while nuisances like the scanner affect the border. In NLI, the premise sets up the context for detecting entailment. Without the premise, entailment cannot be determined by semantics, but the hypothesis retains some nuisances. For this separation, masking parts of the covariates corrupts semantics for object recognition via region-of-interest masking (ROI-MASK) and for the semantic context in NLI via premise masking (PREM-MASK). The last two separations are when semantics and nuisances are signals of different frequencies or different pixel-intensities. For example, in detecting Cardiomegaly in chest X-rays, semantic features like the heart are low-frequency features with high pixel-intensity; see fig. 2 . However, nuisances like noise due to the X-ray scanner can be high-frequency or low-intensity signals. For such separations, frequency filtering (FREQ-FILTER) and intensity filtering (INT-FILTER) corrupt semantics. We demonstrate the value of semantic corruption by using it to power a variety of methods: NURD (Puli et al., 2022) , DFL, POE (Mahabadi et al., 2019) , and JTT (Liu et al., 2021) . We run these methods by building biased models using nuisances produced by semantic corruption. These methods with semantic corruptions outperform ERM on out-of distribution (OOD) generalization tasks like waterbirds (Sagawa et al., 2019) , cardiomegaly detection from chest X-rays, and NLI. The performance of NURD, DFL, POE run with semantic corruption is similar to what the methods achieve with extra observed nuisance variables. Finally, JTT with semantic corruptions outperforms vanilla JTT.

2. WHAT DO METHODS NEED TO REDUCE SPURIOUS CORRELATIONS?

A spurious correlation is a relationship between the covariates and the label that changes across settings like time and location (Geirhos et al., 2020) . Models that exploit a spurious correlation can perform poorly outside the training distribution. We focus on the class of methods that correct models using knowledge of nuisances or where they might appear (Mahabadi et al., 2019; Liu et al., 2021; Puli et al., 2022) ; we call these spurious-correlation avoiding methods (SCAMs). With label y, a vector of nuisances z, and covariates x, the goal is to predict well on data regardless of the nuisance-label relationship. Next, we establish that the central part of several SCAMs is a model that predicts the label using nuisances, which we call the biased model. Let 



p tr and p te be the training and test distributions respectively, and let a |= b denote that the random variables a, b are independent. NURD. In tackling spurious correlations, Puli et al. (2022) identify a conditional that has performance guarantees on test distribution p te with an unknown nuisance-label relationship. They develop NURD to learn the conditional using data from p tr ̸ = p te . NURD uses 1) the nuisancerandomized distribution, p |= (y, z, x) = p(y)p |= (z)p(x | y, z), where z |= p |= y, and 2) an uncorrelating representation r(x) for which z |= p |= y | r(x). In p |= , the nuisance alone cannot predict the label; this helps avoid features that depend only on the nuisance. Next, features that are mixed functions of the label and the nuisance (e.g. x 1 = y + z) can also be spurious. Uncorrelating r(x) avoid such features. With these insights, NURD builds models of the form p |= (y | r(x)) that are most informative of the label. We work with reweighting-NURD, which estimates p |= by weighting samples as p(y) /ptr(y | z)p tr (y, z, x). See appendix A for more details. End-to-end bias mitigation. Mahabadi et al. (2019) consider two methods to train a biased model and a base predictive model jointly to make the base model predict without relying on the biases. The

