SAMPLE WEIGHTING AS AN EXPLANATION FOR MODE COLLAPSE IN GENERATIVE ADVERSARIAL NETWORKS

Abstract

Generative adversarial networks were introduced with a logistic MiniMax cost formulation, which normally fails to train due to saturation, and a Non-Saturating reformulation. While addressing the saturation problem, NS-GAN also inverts the generator's sample weighting, implicitly shifting emphasis from higher-scoring to lower-scoring samples when updating parameters. We present both theory and empirical results suggesting that this makes NS-GAN prone to mode dropping. We design MM-nsat, which preserves MM-GAN sample weighting while avoiding saturation by rescaling the MM-GAN minibatch gradient such that its magnitude approximates NS-GAN's gradient magnitude. MM-nsat has qualitatively different training dynamics, and on MNIST and CIFAR-10 it is stronger in terms of mode coverage, stability and FID. While the empirical results for MM-nsat are promising and favorable also in comparison with the LS-GAN and Hinge-GAN formulations, our main contribution is to show how and why NS-GAN's sample weighting causes mode dropping and training collapse.

1. INTRODUCTION

Generative adversarial networks have come a long way since their introduction (Goodfellow et al., 2014) and are currently state of the art for some tasks, such as generating images. A combination of deep learning developments, GAN specific advances and vast improvements in data sets and computational resources have enabled GANs to generate high resolution images that require some effort to distinguish from real photos (Zhang et al., 2018; Brock et al., 2018; Karras et al., 2018) . 1



Figure 1: Median Fréchet Inception Distance during training for ten runs on MNIST, CIFAR-10, CAT 128 2 and FFHQ 512 2 , using very simple convolutional GANs. The shaded areas show minimum and maximum value during training for the cost formulations. MM-nsat is best overall, suffers less from gradual mode dropping and trains reliably on the more challenging datasets.

