ON DISENTANGLED REPRESENTATIONS EXTRACTED FROM PRETRAINED GANS

Abstract

Constructing disentangled representations is known to be a difficult task, especially in the unsupervised scenario. The dominating paradigm of unsupervised disentanglement is currently to train a generative model that separates different factors of variation in its latent space. This separation is typically enforced by training with specific regularization terms in the model's objective function. These terms, however, introduce additional hyperparameters responsible for the trade-off between disentanglement and generation quality. While tuning these hyperparameters is crucial for proper disentanglement, it is often unclear how to tune them without external supervision. This paper investigates an alternative route to disentangled representations. Namely, we propose to extract such representations from the state-of-the-art GANs trained without disentangling terms in their objectives. This paradigm of post hoc disentanglement employs little or no hyperparameters when learning representations, while achieving results on par with existing state-of-the-art, as shown by comparison in terms of established disentanglement metrics, fairness, and the abstract reasoning task. All our code and models are publicly available 1 .

1. INTRODUCTION

Unsupervised learning of disentangled representations is currently one of the most important challenges in machine learning. Identifying and separating the factors of variation for the data at hand provides a deeper understanding of its internal structure and can bring new insights into the data generation process. Furthermore, disentangled representations are shown to benefit certain downstream tasks, e.g., fairness (Locatello et al., 2019a) and abstract reasoning (van Steenkiste et al., 2019) . Since the seminal papers on disentanglement learning, such as InfoGAN (Chen et al., 2016) and β-VAE (Higgins et al., 2017) , a large number of models were proposed, and this problem continues to attract much research attention (Alemi et al., 2016; Chen et al., 2018; Burgess et al., 2017; Kim & Mnih, 2018; Kumar et al., 2018; Rubenstein et al., 2018; Esmaeili et al., 2019; Mathieu et al., 2019; Rolinek et al., 2019; Nie et al., 2020; Lin et al., 2020) . The existing models achieve disentanglement in their latent spaces via specific regularization terms in their training objectives. Typically, these terms determine the trade-off between disentanglement and generation quality. For example, for β-VAE (Higgins et al., 2017) , one introduces the KLdivergence regularization term that constrains the VAE bottleneck's capacity. This term is weighted by the β multiplier that enforces better disentanglement for β > 1 while resulting in worse reconstruction quality. Similarly, InfoGAN utilized a regularization term approximating the mutual information between the generated image and factor codes. As has been shown in the large scale study Locatello et al. (2019b) , hyperparameter values can critically affect the obtained disentanglement. In the unsupervised setting, the values of ground truth latent factors utilized by disentanglement metrics are unknown, and thus selection of correct hyperparameters becomes a nontrivial task. In this paper, we investigate if disentangled representations can be extracted from the pretrained nondisentangled GAN models, which currently provide the state-of-the-art generation quality (Karras et al., 2020) . These GANs are trained without disentanglement terms in their objectives; therefore, we do not need to tune the hyperparameters mentioned above. Our study is partially inspired by a very recent line of works on controllable generation (Voynov & Babenko, 2020; Shen & Zhou, 2020; Härkönen et al., 2020; Peebles et al., 2020) , which explore the latent spaces of pretrained GANs and identify the latent directions useful for image editing. The mentioned methods operate without external supervision, therefore, are valid to use in the unsupervised disentanglement. As shown by the comparison on the common benchmarks, the proposed post hoc disentanglement is competitive to the current state-of-the-art in terms of existing metrics, becoming an important alternative to the established "end-to-end" disentanglement. Overall, our contributions are the following: • We investigate an alternative paradigm to construct disentangled representations by extracting them from non-disentangled models. In this setting, one does not need to tune hyperparameters for disentanglement regularizers. • We bridge the fields of unsupervised controllable generation and disentanglement learning by using the developments of the former to benefit the latter. As a separate technical contribution, we propose a new simple technique, which outperforms the existing prior methods of controllable generation. • We extensively evaluate all the methods on several popular benchmarks employing commonly used metrics. In most of the operating points, the proposed post hoc disentanglement successfully reaches competitive performance.

2.1. DISENTANGLED REPRESENTATIONS

Learning disentangled representation is a long-standing goal in representation learning (Bengio et al., 2013) useful for a variety of downstream tasks (LeCun et al., 2004; Higgins et al., 2018; Tschannen et al., 2018; Locatello et al., 2019a; van Steenkiste et al., 2019) . While there is no strict definition of disentangled representation, we follow the one considered in (Bengio et al., 2013) : disentangled representation is a representation where a change in one dimension corresponds to the change only in one factor of variation while leaving other factors invariant. Natural data is assumed to be generated from independent factors of variations, and well-learned disentangled representations should separate these explanatory sources. The most popular approaches so far were based on variational autoencoders (VAEs). Usually, to make representations "more disentangled", VAEs objectives are enriched with specific regularizers (Alemi et al., 2016; Higgins et al., 2017; Chen et al., 2018; Burgess et al., 2017; Kim & Mnih, 2018; Kumar et al., 2018; Rubenstein et al., 2018; Esmaeili et al., 2019; Mathieu et al., 2019; Rolinek et al., 2019) . The general idea behind these approaches is to enforce an aggregated posterior to be factorized, thus providing disentanglement. Another line of research on disentangled representations is based on the InfoGAN model (Chen et al., 2016) . InfoGAN is an unsupervised model, which adds an extra regularizer to GAN loss to maximize the mutual information between the small subset of latent variables (factor codes) and observations. In practice, the mutual information loss is approximated using an encoder network via Variational Information Maximization. InfoGAN-CR(Lin et al., 2020) is a modification of In-foGAN that employs the so-called contrastive regularizer (CR), which forces the elements of the latent code set to be visually perceptible and distinguishable between each other. A very recently proposed InfoStyleGAN model (Nie et al., 2020) incorporates similar ideas into the state-of-the-art StyleGAN architecture, allowing for producing both disentangled representations and achieving excellent visual quality of samples. In contrast to these approaches, we use no regularizers or additional loss functions and simply study state-of-the-art GANs trained in a conventional manner.

2.2. CONTROLLABLE GENERATION

Based on rich empirical evidence, it is believed that the latent space of GANs can encode meaningful semantic transformations, such as orientation, appearance, or presence of objects in scenes, of generated images via vector arithmetic (Radford et al., 2016; Zhu et al., 2016; Bau et al., 2019; Chen et al., 2016) . This means that for an image produced by some latent code, such a transformation can be obtained by simply shifting this latent code in a certain carefully constructed direction, independent from the chosen latent code. E.g., in the case of human faces, we may have separate directions



https://bit.ly/3ipb6dW

