TRANSFERABLE UNSUPERVISED ROBUST REPRESEN-TATION LEARNING

Abstract

Robustness is an important, and yet, under-explored aspect of unsupervised representation learning, which has seen a lot of recent developments. In this work, we address this gap by developing a novel framework: Unsupervised Robust Representation Learning (URRL), which combines unsupervised representation learning's pretext task and robust supervised learning (e.g., AugMix). Moreover, it is commonly assumed that there needs to be a trade-off between natural accuracy (on clean data) and robust accuracy (on corrupted data). We upend this view and show that URRL improves both the natural accuracy of unsupervised representation learning and its robustness to corruptions and adversarial noise. A further challenge is that the robustness of a representation might not be preserved in the transfer learning process after fine-tuning on downstream tasks. We develop transferable robustness by proposing a task-agnostic similarity regularization during the fine-tuning process. We show that this improves the robustness of the resulting model without the need for any adversarial training or further data augmentation during fine-tuning.

1. INTRODUCTION

Recently, there has been an increasing interest in unsupervised visual representation learning, where the goal is to learn effective representations of images without using human annotations (Bachman et al., 2019; Chen et al., 2020b; He et al., 2020; Misra & Maaten, 2020) . In this work, we consider a relatively under-explored aspect: the robustness of these representations. Here, we use robustness to encompass the representation's resilience to common corruptions (Hendrycks & Dietterich, 2019) , adversarial examples (Madry et al., 2018) , and its ability to measure the uncertainty of its output in the face of such perturbations (Kumar et al., 2019) . More specifically, we develop new learning methods for transferable robustness of these representations, which improves and preserves the representation's robustness after fine-tuning on downstream tasks. While recent works have only evaluated the virtue of a representation by its accuracy on clean data after transfer learning and fine-tuning, we argue that transferable robustness should also be an integral part of a good representation. Recently, Rezaei & Liu (2020) have shown that downstream task models are vulnerable to adversarial attacks that are based solely on the pre-trained representation before fine-tuning. This highlights the importance of transferable robustness, without which downstream task models are vulnerable and lack generalization. Moreover, it is commonly assumed that there needs to be a trade-off between natural accuracy (on clean data) and robust accuracy (on corrupted data) (Zhang et al., 2019) . While there have been works showing that such a trade-off is not necessary for supervised learning (Yang et al., 2020) , it is unclear how the principles they leverage is applicable to unsupervised representation learning.

Summary of Contributions:

(1) We show that the accuracy-robustness trade-off is not necessary for unsupervised representation learning. ( 2 URRL improves both the clean accuracy and robustness of state-of-the-art representation learning under linear evaluation, and our full framework, URRL with Similarity Regularization (URRL-SR), further improves its robust accuracy by over 10% on 12 datasets. The proposed URRL is shown in Figure 1 (a). Our approach is motivated by the fact that in previous works, data augmentation has played an important role in both supervised learning robustness and unsupervised representation learning (Yun et al., 2019; Chen et al., 2020b) . URRL randomly samples a pretext task of unsupervised representation learning and a robust data augmentation (AugMix, Hendrycks et al. ( 2020b)), and then optimizes them both under a contrastive learning framework. We show that URRL improves both the clean accuracy (on the original images without corruptions) and the robustness of the learned representation under the standard linear evaluation protocol. Nevertheless, using the representation under linear evaluation limits the performance of transfer learning, and ultimately the usefulness of representation learning in downstream tasks. It is thus important to evaluate the representations' transferable robustness after fine-tuning the entire network to the downstream task. The biggest challenge is to preserve the inherited robustness of the representations: an aggressive fine-tuning schedule on a small dataset can significantly overwrite the robustness of the learned representations. We address this challenge by proposing a task-agnostic similarity regularization that is applicable for the fine-tuning of the learned representation on any downstream task. We show that this improves the robustness of the fine-tuned model. The proposed regularization is shown in Figure 1(b) . f θ is initialized by the robust representation f * learned in Figure 1 (a), and is now being fine-tuned by the downstream task loss L task . f * is robust because it is trained on augmentation of corrupted versions of an image. We hypothesize that this form of robustness is not just about how f * represents an image individually, but also about how f * measures the similarities between images. This implies that we can preserve the robustness of f * by maintaining its similarity measures between images. We achieve this by regularizing the similarity matrix of a representation during fine-tuning to be similar to that of the original representation before fine-tuning. This allows the downstream task model to better utilize the robustness of the learned representation, while not capping the performance, which would be the case if the representation is fixed. Our full approach, URRL with similarity regularization (URRL-SR), further improves URRL's robust accuracy by over 10% on 12 datasets without any adversarial training or further data augmentation.

2. METHOD

In this work, we advocate the transferable robustness of unsupervised representation learning, which measures the robustness of a representation after fine-tuning on downstream tasks. The two challenges are: (i) It is commonly assumed that there is an accuracy-robustness trade-off, and the promotion of transferable robustness could potentially set back the performance gain from unsupervised



Figure 1: (a) We show that by integrating AugMix with pretext tasks of state-of-the-art approaches. URRL leads to improvement of both clean accuracy and robustness and does not have the accuracyrobustness tradeoff. (b) We propose a task-agnostic similarity regularization, where the pairwise similarity s i,j of f θ is regularized by that of the frozen representation f * during fine-tuning by L task . This encourages f θ to capture f * 's robustness without the need to train on corrupted data.

