RAINPROOF: AN UMBRELLA TO SHIELD TEXT GENER-ATORS FROM OUT-OF-DISTRIBUTION DATA

Abstract

As more and more conversational and translation systems are deployed in production, it is essential to implement and to develop effective control mechanisms guaranteeing their proper functioning and security. An essential component to ensure safe system behavior is out-of-distribution (OOD) detection, which aims at detecting whether an input sample is statistically far from the training distribution. Although OOD detection is a widely covered topic in classification tasks, it has received much less attention in text generation. This paper addresses the problem of OOD detection for machine translation and dialog generation from an operational perspective. Our contributions include: (i) RAINPROOF a Relative informAItioN Projection ODD detection framework; and (ii) a more operational evaluation setting for OOD detection. Surprisingly, we find that OOD detection is not necessarily aligned with task-specific measures. The OOD detector may filter out samples that are well processed by the model and keep samples that are not, leading to weaker performance. Our results show that RAINPROOF breaks this curse and achieve good results in OOD detection while increasing performance.

1. INTRODUCTION

Significant progress have been made in Natural Language Generation (NLG) in recent years with the development of powerful generic (e.g., GPT (Radford et al., 2018; 2019; Brown et al., 2020) ) and task-specific (e.g., Grover (Zellers et al., 2019) , Pegasus (Zhang et al., 2020) and DialogGPT (Zhang et al., 2019 )) text generators. Text generators power machine translation systems or chat bots that are by definition exposed to the public and whose reliability is therefore a prerequisite for adoption. Text generators are trained in the context of a so-called closed world (Antonucci et al., 2021; Fei & Liu, 2016) , where training and test data are assumed to be drawn i.i.d. from a single distribution, known as the in-distribution. However, when deployed, these models operate in an open world (Parmar et al., 2021; Zhou, 2022) where the i.i.d. assumption is often violated. This change in data distribution is detrimental and induces a drop in performance as illustrated in Tab. 3 and Tab. 4. Thus, to ensure the trustworthiness and adoption, it is necessary to develop tools to protect them from harmful distribution shifts. For example, a trained translation model is not expected to be reliable when presented with another language (e.g. a Spanish model exposed to Catalan, or a Dutch model exposed to Afrikaans) or unexpected technical language (e.g., a colloquial translation model exposed to rare technical terms from the medical field). Most of the existing research, which aims to protect models from Out-Of-Distribution (OOD) data, focuses on classification. Despite their importance, (conditional) text generation has received much less attention even though it is among the most exposed applications. Existing solutions fall into two categories. The first one called training-aware methods (Zhu et al., 2022; Vernekar et al., 2019a; b) modifies the classifier training by exposing the neural network to OOD samples during training. The second one called plug-in methods aims at distinguishing regular samples in the in distribution (IN) from OOD samples based on the behavior of the model on a new input. Plug-in methods include Maximum Softmax Prediction (MSP) (Hendrycks & Gimpel, 2016) or Energy (Lee et al., 2018a) or feature-based anomaly detectors that compute a per-class anomaly score (Ming et al., 2022; Ryu et al., 2017; Huang et al., 2020; Ren et al., 2021a) . Although plug-in methods seem attractive, their adaptation to text generation may not be straightforward. The sheer number of words present in the vocabulary prevents it to be used directly within the classification framework. In this work, we aim at developing new tools to build more reliable text generators, which can be used in practical systems. First, we work in the unsupervised detection setting where we do not assume that we have access to OOD samples as they are often not available. Second, we work in the black-box scenario, which is the most common in the Software as a Service framework Rudin & Radin (2019). In the black-box setting detection methods only have access to the output of the DNN architecture. Third, we want an easy-to-use and effective method to ensure adoptability. Last, we argue that OOD detection impacts on tasks specific performance of the whole system should be taken into account when choosing OOD detectors in an operational setting. Our contributions. Our main contributions can be summarized as follows: 1. A more operational benchmark for text generation OOD detection. We present LOFTER the Language Out oF disTribution pErformance benchmaRk. Existing works on OOD detection for language modeling (Arora et al., 2021) focus on (i) english language only, (ii) the GLUE benchmark and (iii) measure performance solely in terms of OOD detection. LOFTER is, in our view, a more operational setting with a strong focus on neural machine translation (NMT) and dialog generation. First, it introduces more realistic data shifts that go beyond English Fan et al. ( 2021): language shifts induced by closely related language pairs (e.g., Spanish and Catalan or Dutch and Afrikaansfoot_0 ) and domain change (e.g., medical vs news data or different types of dialogs). In addition, LOFTER comes with an updated evaluation setting: detectors' performance are jointly evaluated w.r.t the overall system's performance on the end task. 2. Novel information theoretic-based detectors. We present RAINPROOF: a Relative informAItioN Projection Out OF distribution detector. RAINPROOF is fully unsupervised. It is flexible and can be applied both when no reference samples (IN) are available (corresponding to scenario s 0 ) and when they are (corresponding to scenario s 1 ). RAINPROOF tackles s 0 by computing the models' predictions negentropy (Brillouin, 1953) . For s 1 , it relies its natural extension: the Information Projection (Kullback, 1954; Csiszár, 1967) , an information-theoretic tool that remains overlooked by the machine learning community. 3. New insights on the operational value of OOD detectors Our extensive experiments on LOFTER show that OOD detectors may filter out samples that are well processed by the model and keep samples that are not, leading to weaker performance. Our results show that RAINPROOF breaks this curse and achieve good results in OOD detection while increasing performance. 4. Code and reproductibility. After acceptance, we will publish the open-source code on github. com and the data to facilitate future research, ensure reproducibility and reduce computational costs.

2.1. NOTATIONS & CONDITIONAL TEXT GENERATION

Let us denote Ω a vocabulary of size |Ω| and Ω * its Kleene closure (Fletcher et al., 1990)  2 . We denote P(Ω) = p ∈ [0, 1] |Ω| : |Ω| i=1 p i = 1 the set of probability distributions defined over Ω. Let D train be the training set, composed of N ⩾ 1 i.i.d. samples {(x i , y i )} N i=1 ∈ (X Y) N with probability law p XY . We denote p X and p Y the associated marginal laws of p XY . Each x i is a sequence of tokens and we denote x i j ∈ Ω the jth token of the ith sequence. x i ⩽t = {x i 1 , • • • , x i t } ∈ Ω * denotes the prefix of length t. The same notations hold for y. Conditional textual generation. In conditional textual generation, the goal is to model a probability distribution p ⋆ (x, y) over variable-length text sequences (x, y) by finding p θ ≈ p ⋆ (x, y) for any (x, y). In this work, we assume to have access to a pretrained conditional language model f θ : X × Y → R |Ω| where the output is the (unormalized) logits scores. f θ parameterized p θ , i.e., for any (x, y), p θ (x, y) = softmax(f θ (x, y)/T ) where T ∈ R denotes the temperature. Given an input sequence x, the pretrained language f θ can recursively generate an output sequence ŷ by



Afrikaans is a daughter language of Dutch (Jansen et al., 2007). The Dutch sentence: Appelen zijn gewoonlijk groen, geel of rood can be translated in "Appels is gewoonlik groen, geel of rooi."2 The Kleene closure corresponds to sequences of arbitrary size written with words in Ω. Formally:Ω * = ∞ i=0 Ω i .

