EFFICIENT CONDITIONALLY INVARIANT REPRESEN-TATION LEARNING

Abstract

We introduce the Conditional Independence Regression CovariancE (CIRCE), a measure of conditional independence for multivariate continuous-valued variables. CIRCE applies as a regularizer in settings where we wish to learn neural features φ(X) of data X to estimate a target Y , while being conditionally independent of a distractor Z given Y . Both Z and Y are assumed to be continuous-valued but relatively low dimensional, whereas X and its features may be complex and high dimensional. Relevant settings include domain-invariant learning, fairness, and causal learning. The procedure requires just a single ridge regression from Y to kernelized features of Z, which can be done in advance. It is then only necessary to enforce independence of φ(X) from residuals of this regression, which is possible with attractive estimation properties and consistency guarantees. By contrast, earlier measures of conditional feature dependence require multiple regressions for each step of feature learning, resulting in more severe bias and variance, and greater computational cost. When sufficiently rich features are used, we establish that CIRCE is zero if and only if φ(X) ⊥ ⊥ Z | Y . In experiments, we show superior performance to previous methods on challenging benchmarks, including learning conditionally invariant image features.

1. INTRODUCTION

We consider a learning setting where we have labels Y that we would like to predict from features X, and we additionally observe some metadata Z that we would like our prediction to be 'invariant' to. In particular, our aim is to learn a representation function φ for the features such that φ(X) ⊥ ⊥ Z | Y . There are at least three motivating settings where this task arises. 1. Fairness. In this context, Z is some protected attribute (e.g., race or sex) and the condition φ(X) ⊥ ⊥ Z | Y is the equalized odds condition (Mehrabi et al., 2021). 2. Domain invariant learning. In this case, Z is a label for the environment in which the data was collected (e.g., if we collect data from multiple hospitals, Z i labels the hospital that the ith datapoint is from). The condition φ(X) ⊥ ⊥ Z | Y is sometimes used as a target for invariant learning (e.g., Long et al., 2018; Tachet des Combes et al., 2020; Goel et al., 2021; Jiang & Veitch, 2022) . Wang & Veitch (2022) argue that this condition is well-motivated in cases where Y causes X. 3. Causal representation learning. Neural networks may learn undesirable "shortcuts" for their tasks -e.g., classifying images based on the texture of the background. To mitigate this issue, various schemes have been proposed to force the network to use causally relevant factors in its decision (e.g., Veitch et al., 2021; Makar et al., 2022; Puli et al., 2022) . The structural causal assumptions used in such approaches imply conditional independence relationships between the features we would like the network to use, and observed metadata

availability

data experiments is available at github.com/namratadeka/circe

