SSD: A UNIFIED FRAMEWORK FOR SELF-SUPERVISED OUTLIER DETECTION

Abstract

We ask the following question: what training information is required to design an effective outlier/out-of-distribution (OOD) detector, i.e., detecting samples that lie far away from the training distribution? Since unlabeled data is easily accessible for many applications, the most compelling approach is to develop detectors based on only unlabeled in-distribution data. However, we observe that most existing detectors based on unlabeled data perform poorly, often equivalent to a random prediction. In contrast, existing state-of-the-art OOD detectors achieve impressive performance but require access to fine-grained data labels for supervised training. We propose SSD, an outlier detector based on only unlabeled in-distribution data. We use self-supervised representation learning followed by a Mahalanobis distance based detection in the feature space. We demonstrate that SSD outperforms most existing detectors based on unlabeled data by a large margin. Additionally, SSD even achieves performance on par, and sometimes even better, with supervised training based detectors. Finally, we expand our detection framework with two key extensions. First, we formulate few-shot OOD detection, in which the detector has access to only one to five samples from each class of the targeted OOD dataset. Second, we extend our framework to incorporate training data labels, if available. We find that our novel detection framework based on SSD displays enhanced performance with these extensions, and achieves state-of-the-art performance 1 .

1. INTRODUCTION

Deep neural networks are at the cornerstone of multiple safety-critical applications, ranging from autonomous driving (Ramanagopal et al., 2018) to biometric authentication (Masi et al., 2018; Günther et al., 2017) . When trained on a particular data distribution, referred to as in-distribution data, deep neural networks are known to fail against test inputs that lie far away from the training distribution, commonly referred to as outliers or out-of-distribution (OOD) samples (Grubbs, 1969; Hendrycks & Gimpel, 2017) . This vulnerability motivates the use of an outlier detector before feeding the input samples to the downstream neural network modules. However, a key question is to understand what training information is crucial for effective outlier detection? Will the detector require fine-grained annotation of training data labels or even access to a set of outliers in the training process? Since neither data labels nor outliers are ubiquitous, the most compelling option is to design outlier detectors based on only unlabeled in-distribution data. However, we observe that most of the existing outlier detectors based on unlabeled data fail to scale up to complex data modalities, such as images. For example, autoencoder (AE) (Hawkins et al., 2002) based outlier detectors have achieved success in applications such as intrusion detection (Mirsky et al., 2018) , and fraud detection (Schreyer et al., 2017) . However, this approach achieves close to chance performance on image datasets. Similarly, density modeling based methods, such as PixelCNN++ (Salimans et al., 2017) and Glow (Kingma & Dhariwal, 2018) are known to assign even a higher likelihood to outliers in comparison to indistribution data (Nalisnick et al., 2019) . Published as a conference paper at ICLR 2021 Bendale & Boult, 2016; Liang et al., 2018; Dhamija et al., 2018; Winkens et al., 2020) . This is a strong assumption since labels, in-particular fine-grained labels, can be very costly to collect in some applications (Google AI Pricing, 2020), which further motivates the use of unlabeled data. The inability of supervised detectors to use unlabeled data and poor performance of existing unsupervised approaches naturally give rise to the following question. Can we design an effective out-of-distribution (OOD) data detector with access to only unlabeled data from training distribution? A framework for outlier detection with unlabeled datafoot_1 involves two key steps: 1) Learning a good feature representation with unsupervised training methods 2) Modeling features of in-distribution data without requiring class labels. For example, autoencoders attempt to learn the representation with a bottleneck layer, under the expectation that successful reconstruction requires learning a good set of representations. Though useful for tasks such as dimensionality reduction, we find that these representations are not good enough to sufficiently distinguish in-distribution data and outliers. We argue that if unsupervised training can develop a rich understanding of key semantics in in-distribution data then absence of such semantics in outliers can cause them to lie far away in the feature space, thus making it easy to detect them. Recently, self-supervised representation learning methods have made large progress, commonly measured by accuracy achieved on a downstream classification task (Chen et al., 2020; He et al., 2020; Oord et al., 2018; Misra & Maaten, 2020; Tian et al., 2020) . We leverage these representations in our proposed cluster-conditioned framework based on the Mahalanobis distance (Mahalanobis, 1936) . Our key result is that self-supervised representations are highly effective for the task of outlier detection in our self-supervised outlier detection (SSD) framework where they not only perform far better than most of the previous unsupervised representation learning methods but also perform on par, and sometimes even better, than supervised representations. What if access to a fraction of OOD data or training data labels is available? How do we move past a detector based on unlabeled data and design a framework which can take advantage of such information? Though access to outliers during training is a strong assumption, it may be feasible to obtain a few prior instances of such outliers (Görnitz et al., 2013) . We characterize this setting as few-shot OOD detection, where we assume access to very few, often one to five, samples from the targeted set of outliers. While earlier approaches (Liang et al., 2018; Lee et al., 2018b) mostly use such data to calibrate the detector, we find that access to just a few outliers can bring an additional boost in the performance of our detector. Crucial to this success is the reliable estimation of first and second order statistics of OOD data in the high dimensional feature space with just a few samples. Finally, if class labels are available in the training phase, how can we incorporate them in the SSD framework for outlier detection? Recent works have proposed the addition of the supervised crossentropy and self-supervised learning loss with a tunable parameter, which may require tuning for optimal parameter setting for each dataset (Hendrycks et al., 2019b; Winkens et al., 2020) . We demonstrate that incorporating labels directly in the contrastive loss achieves 1) a tuning parameterfree detector, and 2) state-of-the-art performance.

1.1. KEY CONTRIBUTIONS

SSD for unlabeled data. We propose SSD, an unsupervised framework for outlier detection based on unlabeled in-distribution data. We demonstrate that SSD outperforms most existing unsupervised outlier detectors by a large margin while also performing on par, and sometimes even better than supervised training based detection methods. We validate our observation across four different datasets: CIFAR-10, CIFAR-100, STL-10, and ImageNet. Extensions of SSD. We provide two extensions of SSD to further improve its performance. First, we formulate few-shot OOD detection and propose detection methods which can achieve a significantly large gain in performance with access to only a few targeted OOD samples. Next, we extend SSD, without using any tuning parameter, to also incorporate in-distribution data labels and achieve state-of-the-art performance.



Our code is publicly available at https://github.com/inspire-group/SSD We refer to OOD detection without using class labels of in-distribution data as unsupervised OOD detection.

