WHAT ARE EFFECTIVE LABELS FOR AUGMENTED DATA? IMPROVING ROBUSTNESS WITH AUTOLABEL

Abstract

A wide breadth of research has devised data augmentation approaches that can improve both accuracy and generalization performance for neural networks. However, augmented data can end up being far from the clean data and what is the appropriate label is less clear. Despite this, most existing work simply reuses the original label from the clean data, and the choice of label accompanying the augmented data is relatively less explored. In this paper, we propose AutoLabel to automatically learn the labels for augmented data, based on the distance between the clean distribution and augmented distribution. AutoLabel is built on label smoothing and is guided by the calibration-performance over a hold-out validation set. We show that AutoLabel is a generic framework that can be easily applied to existing data augmentation methods, including AugMix, mixup, and adversarial training. Experiments on CIFAR-10, CIFAR-100 and ImageNet show that AutoLabel can improve models' accuracy and calibration performance, especially under distributional shift. Additionally, we demonstrate that AutoLabel can help adversarial training by bridging the gap between clean accuracy and adversarial robustness.

1. INTRODUCTION

Deep neural networks are increasingly being used in high-stakes applications such as healthcare and autonomous driving. For safe deployment, we not only want models to be accurate on expected test cases (independent and identically distributed samples), but we also want models to be robust to distribution shift (Amodei et al., 2016) and to not be vulnerable to adversarial attacks (Goodfellow et al., 2014; Carlini & Wagner, 2017; Madry et al., 2017; Qin et al., 2020b) . Recent work has shown that the accuracy of state-of-the-art models drops significantly when tested on corrupted data (Hendrycks & Dietterich, 2019) . Furthermore, these models are not just wrong on these unexpected examples, but also overconfident -Ovadia et al. (2019) showed that calibration of models degrades under shift. Calibration measures the gap between a model's own estimate of correctness (a.k.a. confidence) versus the empirical accuracy, which measures the actual probability of correctness. Building models that are accurate and robust, i.e. can be trusted under unexpected inputs from both distributional shift and adversarial attacks, is a challenging but important research problem. Improving both calibration under distribution shift and adversarial robustness has been the focus of numerous research directions. While there are many approaches to addressing these problems, one of the fundamental building blocks is data augmentation: (Hendrycks et al., 2020) or adding adversarially generated, imperceptible noise (Madry et al., 2017; Zhang et al., 2019) . Approaches like these have been shown to improve the robustness and calibration of overparametrized neural networks as they alleviate the issue of neural networks overfitting to spurious features that do not generalize beyond the i.i.d. test set. In the broad amount of research on data augmentation, most of it attempts to apply transformations that do not change the true label such that the label of the original example can also be assumed to be the label of the transformed example, without expensive manual review. While there has been a significant amount of work in how to construct such pseudo-examples in input space, there has been relatively little attention on whether this assumption of label-preservation holds in practice and what label should be assigned to such augmented inputs. For instance, many popular methods assign one-hot targets to both training data as well as augmented inputs that can be quite far away from the training data where even human raters may not be 100% sure of the label. This runs the risk of adding noise to the training process and degrading accuracy. With this observation, in this paper we investigate the choice of target labels for augmented inputs and propose AutoLabel, a method that automatically adapts the confidence assigned to augmented labels, assigning high confidence to inputs close to the training data and lowering the confidence as we move farther away from the training data. • We show that AutoLabel significantly improves the calibration of models (and accuracy, although less dramatically) on both clean and corrupted data for CIFAR10, CIFAR100 and ImageNet. In addition, AutoLabel also helps bridge the gap between accuracy and adversarial robustness.

2. RELATED WORK

Data Augmentation. Recent work has shown that introducing additional training examples can further improve a model's accuracy and generalization (Devries & Taylor, 2017; Cubuk et al., 2018; Yun et al., 2019; Takahashi et al., 2019; Lopes et al., 2019; Zhong et al., 2020) 



generating synthetic examples, typically by modifying existing training examples, that provide additional training data outside the empirical training distribution. A wide breadth of literature has explored what are effective ways to modify training examples, such as making use of domain knowledge through label-preserving transformations

Figure 1: (a): An example showing AutoLabel assigning different labels to augmented images (e.g., by AugMix (Hendrycks et al., 2020)) based on their transformation distances to the clean image. The label for the true class is automatically learned based on the calibration performance on validation set. (b): Examples of images augmented by AugMix with different distances to the original image.

Figure 1 (left) gives a high-level overview of our proposed AutoLabel along with examples of augmented images of varying distances generating by AugMix (Hendrycks et al., 2020) on the right. Our key contributions are as follows: • We propose AutoLabel, a distance-based approach where the training labels are smoothed to different extents based on the distance between the augmented data and the clean data. • We show that AutoLabel is complementary to methods which focus on generating augmented inputs by combining it with popular methods for data augmentation, such as AugMix (Hendrycks et al., 2020), mixup (Zhang et al., 2018) as well as adversarial training (Madry et al., 2017).

. For example, AugMix(Hendrycks et al., 2020)  utilizes stochasticity and diverse augmentations, together with a consistency loss over the augmentations, to achieve state-of-the-art corruption robustness.Mixup (Zhang et al.,  2018), on the other hand, trains a neural network over convex combinations of pairs of examples and shows improved generalization of neural networks. Furthermore, adversarial training(Goodfellow  et al., 2014; Madry et al., 2017; Zhang et al., 2019)  can also be thought as a special data augmentation technique aiming for improving model's adversarial robustness. In this paper, we investigate the choice of the target labels for augmented inputs and show how to apply AutoLabel to these existing data augmentation techniques to further improve model's robustness.Kull et al., 2019). Model's predictive uncertainty can also be quantified using Bayesian neural networks and approximate Bayesian approaches, e.g., variational inference(Graves, 2011; Blundell et al., 2015), MCMC sampling based on stochastic gradients(Welling & Teh,  2011), and dropout-based variational inference(Kingma et al., 2015; Gal & Ghahramani, 2016). In addition to calibration over in-distribution data, more recently,Ovadia et al. (2019)  show that model calibration can further degrade under unseen data shifts, where ensemble of deep neural networks(Lakshminarayanan et al., 2017) is shown to be most robust to dataset shift.

