ADDRESSING THE TOPOLOGICAL DEFECTS OF DISEN-TANGLEMENT Anonymous authors Paper under double-blind review

Abstract

A core challenge in Machine Learning is to disentangle natural factors of variation in data (e.g. object shape vs pose). A popular approach to disentanglement consists in learning to map each of these factors to distinct subspaces of a model's latent representation. However, this approach has shown limited empirical success to date. Here, we show that this approach to disentanglement introduces topological defects (i.e. discontinuities in the encoder) for a broad family of transformations acting on images -encompassing simple affine transformations such as rotations and translations. Moreover, motivated by classical results from group representation theory, we propose an alternative, more flexible approach to disentanglement which relies on distributed equivariant operators, potentially acting on the entire latent space. We theoretically and empirically demonstrate the effectiveness of our approach to disentangle affine transformations. Our work lays a theoretical foundation for the recent success of a new generation of models using distributed operators for disentanglement

1. INTRODUCTION

Learning disentangled representations is arguably key to build robust, fair, and interpretable ML systems (Bengio et al., 2013; Lake et al., 2017; Locatello et al., 2019a) . However, it remains unclear how to achieve disentanglement in practice. Current approaches aim to map different factors of variations in the data to distinct subspaces of a latent representation, but have achieved only limited empirical success (Higgins et al., 2016; Burgess et al., 2018) . More work on the theoretical foundations of disentanglement could provide the key to the development of more successful approaches. In its original formulation, disentanglement consists in isolating statistically independent factors of variation in data into independent latent dimensions. This perspective has led to a range of theoretical studies investigating the conditions under which these factors are identifiable (Locatello et al., 2019b; Shu et al., 2020; Locatello et al., 2020; Hauberg, 2019; Khemakhem et al., 2020) . More recently, Higgins et al. (2018) has proposed an alternative perspective connecting disentanglement to group theory (see Appendix A for a primer on group theory). In this framework, the factors of variation are different subgroups acting on the dataset, and the goal is to learn representations where separated subspaces are equivariant to distinct subgroups -a promising formalism since many transformations found in the physical world are captured by group structures (Noether, 1915) . However, the fundamental principles for how to design models capable of learning such equivariances remain to be discovered (but see Caselles-Dupré et al. (2019) ). Here we attack the problem of disentanglement through the lens of topology (Munkres, 2014) . We show that for a very broad class of transformations acting on images -encompassing all affine transformations (e.g. translations, rotations), an encoder that would map these transformations into dedicated latent subspaces would necessarily be discontinuous. With this assurance, we reframe disentanglement by distinguishing its objective from its traditional implementation, resolving the discontinuities of the encoder. Guided by classical results from group representation theory (Scott & Serre, 1996), we then theoretically and empirically demonstrate the capacity of a model equipped with distributed equivariant operators in latent space to disentangle a range of affine image transformations including translations, rotations and combinations thereof.

2. EMPIRICAL LIMITATIONS OF TRADITIONAL DISENTANGLEMENT

In this section we empirically explore the limitations of traditional disentanglement approaches, in both unsupervised (variational autoencoder and variants) and supervised settings.

VAE, beta-VAE and CCI-VAE

We show that, consistent with results from prior literature, a variational autoencoder model (VAE) and its variants are successful at disentangling the factors of variation on a simple dataset. We train a VAE, beta-VAE and CCI-VAE (Kingma & Welling, 2014; Higgins et al., 2016; Burgess et al., 2018) on a dataset composed of a single class of MNIST digits (the "4s"), augmented with 10 evenly spaced rotations (all details of the models and datasets are in App. B). After training, we qualitatively assess the success of the models to disentangle the rotation transformation through traditional latent traversals: we feed an image of the test set to the network and obtain its corresponding latent representation. We then sweep a range of values for each latent dimension while freezing the other dimensions, obtaining a sequence of image reconstructions for each of these sweeps. We present in Fig. 1A examples of latent traversals along a single latent dimension, selected to be visually closest to a rotation (see Fig. 5 for latent traversals along all other latent dimensions). We find that all these models are mostly successful at the task of disentangling rotation for this simple dataset, in the sense that a sweep along a single dimension of the latent maps to diverse orientations of the test image. We then show that on a slightly richer dataset (MNIST with all digits classes), a VAE model and its variants fail to disentangle shape from pose. We train all three models studied (VAE,



Figure 1: Failure modes of common disentanglement approaches. A. Latent traversal best capturing rotation for a VAE, β-VAE, and CCI-VAE for rotated MNIST restricted to a single digit class ("4"). B. Same as panel A for all 10 MNIST classes. C. Variance of single latents in response to image rotation, averaged over many test images. D. Ranked eigenvalues of the latent covariance matrix in response to image rotation, averaged over many test images. E. A supervised disentangling model successfully reconstructs some digits (top) but fails on other examples (bottom). F. Failure cases of the supervised model trained on a dataset of 2000 rotated shapes (see also Fig. 8).

availability

All code is available at https://anonymous.4open. science/r/5b7e2cbb-54dc-4fde-bc2c-8f75d29fc15a/.

