WHAT ARE EFFECTIVE LABELS FOR AUGMENTED DATA? IMPROVING ROBUSTNESS WITH AUTOLABEL

Abstract

A wide breadth of research has devised data augmentation approaches that can improve both accuracy and generalization performance for neural networks. However, augmented data can end up being far from the clean data and what is the appropriate label is less clear. Despite this, most existing work simply reuses the original label from the clean data, and the choice of label accompanying the augmented data is relatively less explored. In this paper, we propose AutoLabel to automatically learn the labels for augmented data, based on the distance between the clean distribution and augmented distribution. AutoLabel is built on label smoothing and is guided by the calibration-performance over a hold-out validation set. We show that AutoLabel is a generic framework that can be easily applied to existing data augmentation methods, including AugMix, mixup, and adversarial training. Experiments on CIFAR-10, CIFAR-100 and ImageNet show that AutoLabel can improve models' accuracy and calibration performance, especially under distributional shift. Additionally, we demonstrate that AutoLabel can help adversarial training by bridging the gap between clean accuracy and adversarial robustness.

1. INTRODUCTION

Deep neural networks are increasingly being used in high-stakes applications such as healthcare and autonomous driving. For safe deployment, we not only want models to be accurate on expected test cases (independent and identically distributed samples), but we also want models to be robust to distribution shift (Amodei et al., 2016) and to not be vulnerable to adversarial attacks (Goodfellow et al., 2014; Carlini & Wagner, 2017; Madry et al., 2017; Qin et al., 2020b) . Recent work has shown that the accuracy of state-of-the-art models drops significantly when tested on corrupted data (Hendrycks & Dietterich, 2019) . Furthermore, these models are not just wrong on these unexpected examples, but also overconfident -Ovadia et al. (2019) showed that calibration of models degrades under shift. Calibration measures the gap between a model's own estimate of correctness (a.k.a. confidence) versus the empirical accuracy, which measures the actual probability of correctness. Building models that are accurate and robust, i.e. can be trusted under unexpected inputs from both distributional shift and adversarial attacks, is a challenging but important research problem. Improving both calibration under distribution shift and adversarial robustness has been the focus of numerous research directions. While there are many approaches to addressing these problems, one of the fundamental building blocks is data augmentation: (Hendrycks et al., 2020) or adding adversarially generated, imperceptible noise (Madry et al., 2017; Zhang et al., 2019) . Approaches like these have been shown to improve the robustness and calibration of overparametrized neural networks as they alleviate the issue of neural networks overfitting to spurious features that do not generalize beyond the i.i.d. test set. In the broad amount of research on data augmentation, most of it attempts to apply transformations that do not change the true label such that the label of the original example can also be assumed to be the label of the transformed example, without expensive manual review. While there has been a significant amount of work in how to construct such pseudo-examples in input space, there has been relatively little attention on whether this assumption of label-preservation holds in practice and what label should be assigned to such augmented inputs. For instance, many popular methods assign



generating synthetic examples, typically by modifying existing training examples, that provide additional training data outside the empirical training distribution. A wide breadth of literature has explored what are effective ways to modify training examples, such as making use of domain knowledge through label-preserving transformations

