DAVA: DISENTANGLING ADVERSARIAL VARIATIONAL AUTOENCODER

Abstract

The use of well-disentangled representations offers many advantages for downstream tasks, e.g. an increased sample efficiency, or better interpretability. However, the quality of disentangled interpretations is often highly dependent on the choice of dataset-specific hyperparameters, in particular the regularization strength. To address this issue, we introduce DAVA, a novel training procedure for variational auto-encoders. DAVA completely alleviates the problem of hyperparameter selection. We compare DAVA to models with optimal hyperparameters. Without any hyperparameter tuning, DAVA is competitive on a diverse range of commonly used datasets. Underlying DAVA, we discover a necessary condition for unsupervised disentanglement, which we call PIPE. We demonstrate the ability of PIPE to positively predict the performance of downstream models in abstract reasoning. We also thoroughly investigate correlations with existing supervised and unsupervised metrics. The code is available at github.com/besterma/dava.

1. INTRODUCTION

Real-world data tends to be highly structured, full of symmetries and patterns. This implies that there exists a lower-dimensional set of ground truth factors that is able to explain a significant portion of the variation present in real-world data. The goal of disentanglement learning is to recover these factors, so that changes in a single ground truth factor are reflected only in a single latent dimension of a model (see Figure 1 If the generative ground-truth factors are known and labeled data is available, one can train a model in a supervised manner to extract the ground-truth factors. What if the generative factors are unknown, but one still wants to profit from the aforementioned benefits for a downstream task? This may be necessary when the amount of labeled data for the downstream task is limited or training is computationally expensive. Learning disentangled representations in an unsupervised fashion is generally impossible without the use of some priors (Locatello et al., 2019b) . These priors can be present both implicitly in the model architecture and explicitly in the loss function (Tschannen et al., 2018) . An example of such a prior present in the loss function is a low total correlation between



for an example). Such an abstraction allows for more efficient reasoning(Van Steenkiste et al., 2019)  and improved interpretability(Higgins et al., 2017a). It further shows positive effects on zero-shot domain adaption(Higgins et al., 2017b)  and data efficiency(Duan et al.,  2020; Schott et al., 2022).

Figure 1: Latent traversals of a single latent dimension (hair fringes) of DAVA trained on CelebA. DAVA visibly disentangles the fringes from all other facial properties.

