UNSUPERVISED ANOMALY DETECTION FROM SEMAN-TIC SIMILARITY SCORES

Abstract

In this paper we present SemSAD, a simple and generic framework for detecting examples that lie out-of-distribution (OOD) for a given training set. The approach is based on learning a semantic similarity measure to find for a given test example the semantically closest example in the training set and then using a discriminator to classify whether the two examples show sufficient semantic dissimilarity such that the test example can be rejected as OOD. We are able to outperform previous approaches for anomaly, novelty, or out-of-distribution detection in the visual domain by a large margin. In particular we obtain AUROC values close to one for the challenging task of detecting examples from CIFAR-10 as out-of-distribution given CIFAR-100 as in-distribution, without making use of label information.

1. INTRODUCTION

Anomaly detection or novelty detection aims at identifying patterns in data that are significantly different to what is expected. This problem is inherently a binary classification problem that classifies examples either as in-distribution or out-of-distribution, given a sufficiently large sample from the in-distribution (training set). A natural approach to OOD detection is to learn a density model from the training data and compute the likelihood ratio of OOD examples. However, in practice this approach frequently fails for high-dimensional data (Nalisnick et al. (2019)) , where it has been shown that deep generative models can assign higher likelihood to OOD examples than to in-distribution examples. This surprising result is likely the consequence of how existing deep generative models generalise. For example, Variational Autoencoders (Kingma & Welling (2014)) generalise by superposition of examples, which is a consequence of the stochastic nature of the posterior that can map different examples to the same point in latent space. As superposition is an averaging process that reduces the information content it can be expected that examples of lower complexity than the training examples can map to high likelihood regions in latent space. Note that it is possible for a datapoint to have high likelihood under a distribution yet be nearly impossible to be sampled, a property known as asymptotic equipartition property in information theory Cover & Thomas (2001) . As a consequence, most of the current OOD detection approaches make use of a score function s(x) to classify test examples as in-distribution or OOD. In case that the examples of the training set are labelled, a simple score can be given by s(x) = max y p(y|x), with p(y|x) the softmax probability for predicting class labels, y ∈ {1, .., K} (Hendrycks & Gimpel (2017)). If s(x) is below a threshold the test example is classified as OOD. Labelled data allows to learn representations that are associated with the semantic information shared by the examples in the training set, which can be used for OOD detection. However, the approach suffers from the problem that the scores for in-distribution examples can be widely distributed across the interval of possible score values, s(x) ∈ [1/K, 1], especially if the number of labels are low and the classification task is hard, which strongly increases the false-positive rate. Consequently, better performance was found for approaches that use labeled data for learning a higher dimensional representation that encodes for semantic information (Lee et al. (2018b) ). In this representation space the in-distribution occupies just a small volume and a random feature vector would be most likely classified as OOD. Another simplification arises if the OOD detection problem is supervised, with some OOD examples labelled as such and contribute to the training set. In this case the OOD detection problem boils down to an unbalanced classification problem (Chalapathy & Chawla (2019)). In general OOD detection benefits from separating the factors of variation for the in-distribution in either relevant (e.g. object identity) or irrelevant (e.g. compression artefacts) using prior knowledge, where the relevant factors are typically those that carry salient semantic information. In line with the arguments put forward by Ahmed & Courville (2020) , this separation helps an OOD model to systematically generalise, e.g. whether we are allowed to re-colour or add noise to images for data augmentation. Generalisation over the training set is necessary, as learning under insufficient inductive bias would result in misclassification of examples from an in-distribution test set as OOD. Labeled data provide this additional information, as relevant factors can be defined as those that help the classification task, with the limitation that there might be more factors involved in characterising the in-distribution than those needed to predict the labels. In this work, we introduce a general framework for OOD detection problems that does not require label information. Our framework can be widely applied to OOD detection tasks, including visual, audio, and textual data with the only limitation that transformations must be a priori known that conserve the semantics of training examples, such as geometric transformations for images, proximity of time intervals for audio recordings (van den Oord et al. ( 2018)), or randomly masking a small fraction of words in a sentence or paragraph (Devlin et al. ( 2019)). For visual data we show new state-of-the-art OOD classification accuracies for standard benchmark data sets, surpassing even the accuracies that include labels as additional information. The key contributions of this work are 



For autoregressive generative models, such as PixelCNN (van den Oord et al. (2016)), it has been shown that the pixel-by-pixel generation process is strongly determined by the local surrounding of pixels (Chen et al. (2018)), where the fact that nearby pixels of training examples frequently share the same color can explain why mono-chromatic images are assigned a high likelihood (Nalisnick et al. (2019)). Local pixel correlations also seem to be responsible for the failure of generative models based on Normalising Flows to assign correct likelihood values to OOD examples Schirrmeister et al. (2020).

We propose a new OOD detection framework that is applicable in absence of labeled indistribution data or OOD examples that are labeled as such.• We show that our approach strongly improves OOD detection for challenging tasks in the visual domain • We find that identifying semantically close examples in the training set is central for reliable OOD detection 2 RELATED WORK Unsupervised Methods using in-distribution labels. Many OOD detection methods make use of labels to generate scores that are either based on class prediction probabilities or on intermediate representations for an in-distribution classification task. For example, Hendrycks & Gimpel (Hendrycks & Gimpel (2017)) used the maximum of the softmax probabilities (MSP) to discriminate between OOD and in-distribution. More recent approaches Lee et al. (2018a); Winkens et al. (2020); Zhang et al. (2020) use labels to learn an intermediate representation on which a density distribution (e.g. multivariate normal distribution or deep generative network) can be fitted, which then can be used to compute the likelihood of OOD examples. As labels implicitly provide information about the semantic relation of examples in the training set, approaches using label information typically show higher accuracy than unsupervised methods. These approaches can be improved by introducing additional parameters or training strategies. For example, MSP was improved by introducing a tem-General Unsupervised Methods. If label information is absent, other means must be found to impose an inductive bias on the OOD detection model to generalise over the training set. Existing approaches can be separated in methods that learn generalisable features based on (i) self-supervised learning tasks Golan & El-Yaniv (2018), transformations that destroy semantics Choi & Chung (2020), match of encoder-decoder architectures Xiao et al. (2020), or make use of a semantically related auxiliary outlier distribution Schirrmeister et al. (2020). The work that is most related to ours is Geometric-Transformation Classification (GEOM), proposed by Golan & El-Yaniv (2018)

