A TEXT GAN FOR LANGUAGE GENERATION WITH NON-AUTOREGRESSIVE GENERATOR Anonymous authors Paper under double-blind review

Abstract

Despite the great success of Generative Adversarial Networks (GANs) in generating high-quality images, GANs for text generation still face two major challenges: first, most text GANs are unstable in training mainly due to ineffective optimization of the generator, and they heavily rely on maximum likelihood pretraining; second, most text GANs adopt autoregressive generators without latent variables, which largely limits the ability to learn latent representations for natural language text. In this paper, we propose a novel text GAN, named NAGAN, which incorporates a non-autoregressive generator with latent variables. The non-autoregressive generator can be effectively trained with gradient-based methods and free of pretraining. The latent variables facilitate representation learning for text generation applications. Experiments show that our model is competitive compared with existing text GANs in unconditional text generation, and it outperforms existing methods on sentence manipulation in latent space and unsupervised text decipherment.

1. INTRODUCTION

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have achieved great success in generating continuous data, such as images with high resolution and fidelity (Brock et al., 2019) . Unsurprisingly, GANs are also widely studied for text generation, but the adaptation is by no means trivial. The mainstream text GANs (Yu et al., 2017b; Guo et al., 2018) apply a different framework tailored for discrete sequence data, but there are remaining several unsolved research problems. One problem lies in ineffective optimization. Most text GANs resort to gradient-free RL (reinforcement learning) algorithms, mainly due to the nature of discrete text data. However, since RL methods abandon the gradient information, they suffer from unstable training processes (Ke et al., 2019) . Though some works (Chen et al., 2018) explored the feasibility of gradient-based methods, the optimization is still ineffective. As a result, most text GANs heavily rely on MLE pretraining, and some even report worse performance after GAN training (Caccia et al., 2020) . Another problem can be attributed to the generative model. Most text GANs adopt an autoregressive generator, which defines an explicit likelihood without any latent variable. Latent variables have empowered image GANs with various applications, such as unsupervised style transfer (Taigman et al., 2017) and image editing (Brock et al., 2017) . However, most text GANs merely generate sentences from the learned distribution with autoregressive decoding, thereby hardly applicable to text style transfer or controlled text generation which may require latent representations. We therefore challenge the conventional design of existing text GANs and argue that incorporating a non-autoregressive generator can benefit from both efficient gradient-based optimization methods and the use of latent variables. Our proposed model, named Non-Autoregressive GAN (NAGAN), consists of a non-autoregressive generator and a regularized discriminator. The non-autoregressive generator naturally translates latent variables to the tokens in parallel, and the gradient-based optimization on our feed-forward structure is significantly more effective than the same method on an autoregressive generator. The discriminator is regularized by Max Gradient Penalty (Zhou et al., 2019) , which is another key to effective optimization. Our contributions are summarized as follows: • We propose non-autoregressive GAN, which characterizes itself by employing a non-autoregressive generator and latent variables, and efficiently training from scratch with gradient-based methods. To our knowledge, NAGAN is the first text GAN which learns latent representations from scratch.

