TEACH ME HOW TO INTERPOLATE A MYRIAD OF EM-BEDDINGS

Abstract

Mixup refers to interpolation-based data augmentation, originally motivated as a way to go beyond empirical risk minimization (ERM). Yet, its extensions focus on the definition of interpolation and the space where it takes place, while the augmentation itself is less studied: For a mini-batch of size m, most methods interpolate between m pairs with a single scalar interpolation factor λ. In this work, we make progress in this direction by introducing MultiMix, which interpolates an arbitrary number n of tuples, each of length m, with one vector λ per tuple. On sequence data, we further extend to dense interpolation and loss computation over all spatial positions. Overall, we increase the number of tuples per mini-batch by orders of magnitude at little additional cost. This is possible by interpolating at the very last layer before the classifier. Finally, to address inconsistencies due to linear target interpolation, we introduce a self-distillation approach to generate and interpolate synthetic targets. We empirically show that our contributions result in significant improvement over state-of-the-art mixup methods on four benchmarks. By analyzing the embedding space, we observe that the classes are more tightly clustered and uniformly spread over the embedding space, thereby explaining the improved behavior.

1. INTRODUCTION

Mixup (Zhang et al., 2018) is a data augmentation method that interpolates between pairs of training examples, thus regularizing a neural network to favor linear behavior in-between examples. Besides improving generalization, it has important properties such as reducing overconfident predictions and increasing the robustness to adversarial examples. Several follow-up works have studied interpolation in the latent or embedding space, which is equivalent to interpolating along a manifold in the input space (Verma et al., 2019) , and a number of nonlinear and attention-based interpolation mechanisms (Yun et al., 2019; Kim et al., 2020; 2021; Uddin et al., 2021; Hong et al., 2021) . However, little progress has been made in the augmentation process itself, i.e., the number of examples being interpolated and the number of interpolated examples being generated. Mixup was originally motivated as a way to go beyond empirical risk minimization (ERM) (Vapnik, 1999) through a vicinal distribution expressed as an expectation over an interpolation factor λ, which is equivalent to the set of linear segments between all pairs of training inputs and targets. In practice however, in every training iteration, a single scalar λ is drawn and the number of interpolated pairs is limited to the size of the mini-batch, as illustrated in Figure 1(a) . This is because, if interpolation takes place in the input space, it would be expensive to increase the number of examples per iteration. To our knowledge, these limitations exist in all mixup methods. In this work, we argue that a data augmentation process should augment the data seen by the model, or at least by its last few layers, as much as possible. In this sense, we follow manifold mixup (Verma et al., 2019) and generalize it in a number of ways to introduce MultiMix, as illustrated in Figure 1(b) . First, rather than pairs, we interpolate tuples that are as large as the mini-batch. Effectively, instead of linear segments between pairs of examples in the mini-batch, we sample on their entire convex hull. Second, we draw a different vector λ for each tuple. Third, and most important, we increase the number of interpolated tuples per iteration by orders of magnitude by only slightly decreasing the actual training throughput in examples per second. This is possible by interpolating at the deepest

