LEARNING FROM NOISY DATA WITH ROBUST REPRE-SENTATION LEARNING

Abstract

Learning from noisy data has attracted much attention, where most methods focus on label noise. In this work, we propose a new framework which simultaneously addresses three types of noise commonly seen in real-world data: label noise, outof-distribution input, and input corruption. In contrast to most existing methods, we combat noise by learning robust representation. Specifically, we embed images into a low-dimensional subspace by training an autoencoder on the deep features. We regularize the geometric structure of the subspace with robust contrastive learning, which includes an unsupervised consistency loss and a supervised mixup prototypical loss. Furthermore, we leverage the structure of the learned subspace for noise cleaning, by aggregating information from neighboring samples. Experiments on multiple benchmarks demonstrate state-of-the-art performance of our method and robustness of the learned representation. Our code will be released 1 .

1. INTRODUCTION

Data in real life is noisy. However, deep models with remarkable performance are mostly trained on clean datasets with high-quality human annotations. Manual data cleaning and labeling is an expensive process that is difficult to scale. On the other hand, there exists almost infinite amount of noisy data online. It is crucial that deep neural networks (DNNs) could harvest noisy training data. However, it has been shown that DNNs are susceptible to overfitting to noise (Zhang et al., 2017) . As shown in Figure 1 , a real-world noisy image dataset often consists of multiple types of noise. Label noise refers to samples that are wrongly labeled as another class (e.g. flower labeled as orange). Out-of-distribution input refers to samples that do not belong to any known classes. Input corruption refers to image-level distortion (e.g. low brightness) that causes data shift between training and test. Most of the methods in literature focus on addressing the more detrimental label noise. Two dominant approaches include: (1) find clean samples as those with smaller loss and assign larger weights to them (Han et al., 2018; Yu et al., 2019; Shen & Sanghavi, 2019; Arazo et al., 2019) ; (2) relabel noisy samples using model's predictions (Reed et al., 2015; Ma et al., 2018; Tanaka et al., 2018; Yi & Wu, 2019) . The recently proposed DivideMix (Li et al., 2020a) integrates both approaches in a co-training framework, but it also increases computation cost. Previous methods that focus on addressing label noise do not consider out-of-distribution input or input corruption, which limits their performance in real-world scenarios. Furthermore, using a model's own prediction to relabel samples could cause confirmation bias, where the prediction error accumulates and harms performance. We propose a new direction for effective learning from noisy data. Our method embeds images into noise-robust low-dimensional representations, and regularizes the geometric structure of the representations with contrastive learning. Specifically, our algorithmic contributions include: • We propose noise-robust contrastive learning, which introduces two contrastive losses. The first is an unsupervised consistency contrastive loss. It enforces inputs with perturbations to have similar normalized embeddings, which helps learn robust and discriminative representation. • Our second contrastive loss is a weakly-supervised mixup prototypical loss. We compute class prototypes as normalized mean embeddings, and enforces each sample's embedding to be closer to 1 Code is in the supplementary material 1

