ON THE OPTIMAL PRECISION OF GANS

Abstract

Generative adversarial networks (GANs) are known to face model misspecification when learning disconnected distributions. Indeed, continuous mapping from a unimodal latent distribution to a disconnected one is impossible, so GANs necessarily generate samples outside of the support of the target distribution. In this paper, we make the connection between the performance of GANs and their latent space configuration. In particular, we raise the following question: what is the latent space partition that minimizes the measure of out-of-manifold samples? Building on a recent result of geometric measure theory, we prove a sufficient condition for GANs to be optimal when the dimension of the latent space is larger than the number of modes. In particular, we show the optimality of generators that structure their latent space as a 'simplicial cluster' -a Voronoi partition where centers are equally distant. We derive both an upper and a lower bound on the optimal precision of GANs learning disconnected manifolds. Interestingly, these two bounds have the same order of decrease: √ log m, m being the number of modes. Finally, we perform several experiments to exhibit the geometry of the latent space and experimentally show that GANs have a geometry with similar properties to the theoretical one.

1. INTRODUCTION

GANs (Goodfellow et al., 2014) , a family of deep generative models, have shown great capacities to generate photorealistic images (Karras et al., 2019) . State-of-the-art models, like StyleGAN (Karras et al., 2019) or TransformerGAN (Jiang et al., 2021) , show empirical benefits from relying on overparametrized networks with high-dimensional latent spaces. Besides, manipulating the latent representation of a GAN is also helpful for diverse tasks such as image editing (Shen et al., 2020; Wu et al., 2021) or unsupervised learning of image segmentation (Abdal et al., 2021) . However, there is still a poor theoretical understanding of how GANs organize their latent space. We argue that this is a crucial step in better apprehending the behavior of GANs. To better understand GANs, the setting of disconnected distributions learning is enlightening. Experimental and theoretical works (Khayatkhoei et al., 2018; Tanielian et al., 2020) have shown a fundamental limitation of GANs when dealing with such distributions. Since the distribution modeled by GANs is connected, some areas of GANs' support are necessarily mapped outside the true data distribution. When covering modes of a disconnected distribution, GANs try to minimize the measure of the generated distribution lying outside the true modes (e.g. the purple area on the right of Figure 1 ). In other words, GANs need to minimize the measure of the borders between the modes in the latent space. Considering a Gaussian latent space, minimizing this measure is closely linked to the field of Gaussian isoperimetric inequalities (Ledoux, 1996) . This field aims at deriving the partitions that decompose a Gaussian space with a minimal Gaussian-weighted perimeter. We argue that the optimal partitions derived in Gaussian isoperimetric inequalities cast a light on the structure of the latent space of GANs. Most notably, a recent result (Milman and Neeman, 2022) shows that, as long as the number of components m in the partition and the number of dimensions d of the Gaussian space are such that m ≤ d + 1, the optimal partition is a 'simplicial cluster': a Voronoi diagram obtained from the cells of equidistant points (see left of Figure 1 for m = 3 and d = 3). In this paper, we apply this result to the field of GANs and show, both experimentally and theoretically, that GANs with 'simplicial cluster' latent space minimize out-of-distribution generated samples. We draw the connection between GANs and Gaussian isoperimetric inequalities by using the precision metric (Sajjadi et al., 2018; Kynkäänniemi et al., 2019) , which quantifies the portion of generated 1

