HARNESSING OUT-OF-DISTRIBUTION EXAMPLES VIA AUGMENTING CONTENT AND STYLE

Abstract

Machine learning models are vulnerable to Out-Of-Distribution (OOD) examples, and such a problem has drawn much attention. However, current methods lack a full understanding of different types of OOD data: there are benign OOD data that can be properly adapted to enhance the learning performance, while other malign OOD data would severely degenerate the classification result. To Harness OOD data, this paper proposes a HOOD method that can leverage the content and style from each image instance to identify benign and malign OOD data. Particularly, we design a variational inference framework to causally disentangle content and style features by constructing a structural causal model. Subsequently, we augment the content and style through an intervention process to produce malign and benign OOD data, respectively. The benign OOD data contain novel styles but hold our interested contents, and they can be leveraged to help train a style-invariant model. In contrast, the malign OOD data inherit unknown contents but carry familiar styles, by detecting them can improve model robustness against deceiving anomalies. Thanks to the proposed novel disentanglement and data augmentation techniques, HOOD can effectively deal with OOD examples in unknown and open environments, whose effectiveness is empirically validated in three typical OOD applications including OOD detection, open-set semi-supervised learning, and open-set domain adaptation. * Corresponding to Tongliang Liu and Chen Gong. 1 We follow (Bengio et al., 2011) to regard the augmented data as a type of OOD data

1. INTRODUCTION

Learning in the presence of Out-Of-Distribution (OOD) data has been a challenging task in machine learning, as the deployed classifier tends to fail if the unseen data drawn from unknown distributions are not properly handled (Hendrycks & Gimpel, 2017; Pan & Yang, 2009) . Such a critical problem ubiquitously exists when deep models meet domain shift (Ganin et al., 2016; Tzeng et al., 2017) and unseen-class data (Hendrycks & Gimpel, 2017; Scheirer et al., 2012) , which has drawn a lot of attention in some important fields such as OOD detection (Hein et al., 2019; Hendrycks & Gimpel, 2017; Lee et al., 2018; Liang et al., 2018; Liu et al., 2020; Wang et al., 2022a; 2023; 2022b) , Open-Set Domain Adaptation (DA) (Liu et al., 2019; Saito et al., 2018) , and Open-Set Semi-Supervised Learning (SSL) (Huang et al., 2021b; 2022b; a; Oliver et al., 2018; Saito et al., 2021; Yu et al., 2020) . In the above fields, OOD data can be divided into two types, namely benign OOD data 1 and malign OOD data. The benign OOD data can boost the learning performance on the target distribution through DA techniques (Ganin & Lempitsky, 2015; Tzeng et al., 2017) , but they can be misleading if not being properly exploited. To improve model generalization, many positive data augmentation techniques (Cubuk et al., 2018; Xie et al., 2020) have been proposed. For instance, the performance of SSL (Berthelot et al., 2019; Sohn et al., 2020) has been greatly improved thanks to the augmented benign OOD data. On the contrary, malign OOD data with unknown classes can damage the The green lines and red lines denote the augmentation of benign OOD data X and malign OOD data X, respectively. In all figures, the blank variables are observable and the shaded variables are latent. classification results, but they are deceiving and hard to detect (Hendrycks & Gimpel, 2017; Liang et al., 2018; Wei et al., 2022b; a) . To train a robust model against malign OOD data, some works (Kong & Ramanan, 2021; Sinha et al., 2020) conduct negative data augmentation to generate "hard" malign data which resemble in-distribution (ID) data. By separating such "hard" data from ID data, the OOD detection performance can be improved. When presented with both malign and benign OOD data, it is more challenging to decide which to separate and which to exploit. As a consequence, the performance of existing open-set methods could be sub-optimal due to two drawbacks: 1) radically exploiting too much malign OOD data, and 2) conservatively denying too much benign OOD data. In this paper, we propose a HOOD framework (see Fig. 2 ) to properly harness OOD data in several OOD problems. To distinguish benign and malign OOD data, we model the data generating process by following the structural causal model (SCM) (Glymour et al., 2016; Pearl, 2009; Gao et al., 2022) in Fig. 1 (a). Particularly, we decompose an image instance X into two latent components: 1) content variable C which denotes the interested object, and 2) style variable S which contains other influential factors such as brightness, orientation, and color. The content C can indicate its true class Y , and the style S is decisive for the environmental condition, which is termed as domain D. Intuitively, malign OOD data cannot be incorporated into network training, because they contain unseen contents, thus their true classes are different from any known class; and benign OOD data can be adapted because they only have novel styles but contain the same contents as ID data. Therefore, we can distinguish the benign and malign OOD data based on the extracted the content and style features. In addition, we conduct causal disentanglement through maximizing an approximated evidence lower-bound (ELBO) (Blei et al., 2017; Yao et al., 2021; Xia et al., 2022b) of joint distribution P (X, Y, D). As a result, we can effectively break the spurious correlation (Pearl, 2009; Glymour et al., 2016; Hermann et al., 2020; Li et al., 2021b; Zhang et al., 2022) between content and style which commonly occurs during network training (Arjovsky et al., 2019) , as shown by the dashed lines in Fig. 1 (b ). In the ablation study, we find that HOOD can correctly disentangle content and style, which can correspondingly benefit generalization tasks (such as open-set DA and open-set SSL) and detection task (such as OOD detection). To further improve the learning performance, we conduct both positive and negative data augmentation by solely intervening the style and content, respectively, as shown by the blue and red lines in Fig. 1  (c ). Such process is achieved through backpropagating the gradient computed from an intervention objective. As a result, style-changed data X must be identified as benign OOD data, and contentchanged data X should be recognized as malign OOD data. Without including any bias, the benign OOD data can be easily harnessed to improve model generalization, and the malign OOD data can be directly recognized as harmful ones which benefits the detection of unknown anomalies. By conducting extensive experiments on several OOD applications, including OOD detection, open-set SSL, and open-set DA, we validate the effectiveness of our method on typical benchmark datasets. To sum up, our contributions are three-fold: • We propose a unified framework dubbed HOOD which can effectively disentangle the content and style features to break the spurious correlation. As a result, benign OOD data and malign OOD data can be correctly identified based on the disentangled features. • We design a novel data augmentation method which correspondingly augments the content and style features to produce benign and malign OOD data, and further leverage them to enhance the learning performance. • We experimentally validate the effectiveness of HOOD on various OOD applications, including OOD detection, open-set SSL, and open-set DA.



Figure 1: (a) An ideal causal diagram which reveals the data generating process. (b) Illustration of our disentanglement. The brown-edged variables C and S are approximations of content C and style S. The dashed lines indicate the unwanted causal relations that should be broken. (c) Illustration of the data augmentation of HOOD.The green lines and red lines denote the augmentation of benign OOD data X and malign OOD data X, respectively. In all figures, the blank variables are observable and the shaded variables are latent.

