TAILORING LANGUAGE GENERATION MODELS UNDER TOTAL VARIATION DISTANCE

Abstract

The standard paradigm of neural language generation adopts maximum likelihood estimation (MLE) as the optimizing method. From a distributional view, MLE in fact minimizes the Kullback-Leibler divergence (KLD) between the distribution of the real data and that of the model. However, this approach forces the model to distribute non-zero (sometimes large) probability mass to all training samples regardless of their quality. Moreover, in the attempt to cover the low-probability regions in the data distribution, the model systematically overestimates the probability of corrupted text sequences, which we conjecture is one of the main reasons for text degeneration during autoregressive decoding. To remedy this problem, we leverage the total variation distance (TVD) with its robustness to outliers, and develop practical bounds to apply it to language generation. Then, we introduce the TaiLr 1 objective that balances the tradeoff of estimating TVD. Intuitively, TaiLr downweights real data samples that have low model probabilities with tunable penalization intensity. Experimental results show that our method alleviates the overestimation of degenerated sequences without sacrificing diversity and improves generation quality on a wide range of text generation tasks.

1. INTRODUCTION

The dominant approach to train language generation models is to maximize the likelihood of text samples in training data. With the development of pre-training techniques, the quality of texts generated by current models has been improved by a large margin (Radford et al., 2019; Brown et al., 2020) . However, the text degeneration phenomena, e.g., repetitions (Holtzman et al., 2020; Welleck et al., 2020 ), incoherence (Guan et al., 2021; Ji & Huang, 2021) , and other ill-formed generation results sampled from the noisy long tail (Dou et al., 2022; LeBrun et al., 2022) , are still widely observed in large pre-trained models. These results indicate that using MLE as the optimizing method has theoretical limitations that are hard to be compensated by increasing the model size. Given the real data distribution p(x) and the model distribution q(x) defined by a learned generation model, we can view MLE as minimizing the KLD between p(x) and q(x). However, minimizing D KL (p, q) will lead to a zero-avoiding solution of q(x) that spreads itself to cover all the modes in the real data (Minka, 2005; Malinin & Gales, 2019) . As the model is forced to take into account all the modes regardless of their quality and saliency, this behavior could deteriorate the overall generation quality when (i) the data inherently exhibits too many variations, e.g., in open-ended generation, the model often over-presents unrelated words in the unreliable long tail of its distribution (Holtzman et al., 2020) . (ii) the data contains flawed or noisy references, e.g., hallucination and missing contents in text summarization (Zhao et al., 2020) degrade the generation quality of the model. In language generation, the attempt to cover all the non-zero probability regions in the data distribution would lead to a problem directly related to text degeneration, which we term as data void

