UNSUPERVISED META-LEARNING THROUGH LATENT-SPACE INTERPOLATION IN GENERATIVE MODELS

Abstract

Several recently proposed unsupervised meta-learning approaches rely on synthetic meta-tasks created using techniques such as random selection, clustering and/or augmentation. In this work, we describe a novel approach that generates metatasks using generative models. The proposed family of algorithms generate pairs of in-class and out-of-class samples from the latent space in a principled way, allowing us to create synthetic classes forming the training and validation data of a meta-task. We find that the proposed approach, LAtent Space Interpolation Unsupervised Meta-learning (LASIUM), outperforms or is competitive with current unsupervised learning baselines on few-shot classification tasks on the most widely used benchmark datasets.

1. INTRODUCTION

Few-shot meta-learning algorithms for neural networks such as Mishra et al. (2018); Finn et al. (2017) ; Snell et al. (2017) prepare networks to quickly adapt to unseen tasks. This is done in a meta-training phase that typically involves a large number of supervised learning tasks. Generating supervised tasks depends on large, labeled datasets and hand-specified task distribution. Very recently, several approaches had been proposed that perform the meta-training by generating synthetic training tasks from an unsupervised dataset. This requires us to generate samples with specific pairwise information: in-class pairs of samples that are with high likelihood in the same class, and out-of-class pairs that are with high likelihood not in the same class. For instance, UMTRA by Khodadadeh et al. (2019) and AAL by Antoniou & Storkey (2019) achieve this through random selection from a domain with many classes for out-of-class pairs and by augmentation for in-class pairs. Hsu et al. (2019) propose CACTUs that creates synthetic labels through unsupervised clustering of the domain. In this paper, we rely on recent advances in the field of generative models, such as the variants of generative adversarial networks (GANs) and variational autoencoders (VAEs), to generate the in-class and out-of-class pairs of meta-training data. The fundamental idea of our approach is that in-class pairs are close while out-of-class pairs are far away in the latent space representation of the generative model. Thus, we can generate in-class pairs by interpolating between two out-of-class samples in the latent space and choosing interpolation ratios that put the new sample close to one of the objects. From this latent sample, the generative model creates the new in-class object. Our approach requires little domain-specific tweaking, and the necessary tweaks are human-comprehensible. For instance, we need to choose thresholds for latent space distance that ensure that classes are in different domains, as well as interpolation ratio thresholds that ensure that the sample is in the same class as the nearest edge. Another advantage of the approach is that we can utilize off-the-shelf, pre-trained generative models. The main contributions of this paper can be summarized as follows: • We describe an algorithm, LAtent Space Interpolation Unsupervised Meta-learning (LA-SIUM), that creates training data for a downstream meta-learning algorithm starting from an unlabeled dataset by leveraging interpolation in the latent space of a generative model. • We show that on the most widely used few-shot learning datasets, LASIUM outperforms or performs competitively with other unsupervised meta-learning algorithms, significantly outperforms transfer learning in all cases, and in a number of cases approaches the performance of supervised meta-learning algorithms.

2. RELATED WORK

Meta-learning or "learning to learn" in the field of neural networks is an umbrella term that covers a variety of techniques that involve training a neural network over the course of a meta-training phase, such that when presented with the target task, the network is able to learn it much more efficiently than a randomly initialized network would. Most approaches use labeled data during the meta-learning phase. While in some domains there is an abundance of labeled datasets, in many domains such labeled data is difficult to acquire. Unsupervised meta-learning approaches aim to learn from an unsupervised dataset from a domain similar from that of the target task. Typically these approaches generate synthetic few-shot learning tasks for the meta-learning phase through a variety of techniques. CACTUs (Hsu et al. ( 2019 The accuracy of these approaches was shown to be comparable with but lower than supervised meta-learning approaches, though with the advantage of requiring orders of magnitude less labeled training data. A common weakness of these approaches is that the techniques used to generate the synthetic tasks (clustering, augmentation, random sampling) are highly domain dependent. Our proposed approach, LASIUM, takes advantage of generative models trained on the specific 2019)) aims to solve the missing mode problem of conditional GANs through a regularization term that maximizes the distance between the generated images with respect to the distance between their corresponding input latent codes. Progressive GAfNs (Karras et al. ( 2018)) are growing both the generator and discriminator progressively, and approach resembling the layer-wise training of autoencoders.

3. LATENT SPACE INTERPOLATION UNSUPERVISED META-LEARNING

Preliminaries: We define an N -way, K (tr) -shot supervised classification task, T , as a set D (tr) T composed of i ∈ {1, . . . , N ×K (tr) } data points (x i , y i ) such that there are exactly K (tr) samples for each categorical label y i ∈ {1, . . . , N }. During meta-learning, an additional set ,D (val) T , is attached to each task that contains another N × K (val) data points separate from the ones in D (tr) T . We have exactly K (val) samples for each class in D (val) T as well.



)) uses a progressive clustering method. UMTRA (Khodadadeh et al. (2019)) utilizes the statistical diversity properties and domain-specific augmentations to generate synthetic training and validation data. AAL (Antoniou & Storkey (2019)) uses augmentation of the unlabeled training set to generate the validation data.

domain to create the in-class and out-of-class pairs of meta-training data. By creating new training data through interpolation between training samples, LASIUM-OC has similarities with mixup Zhang et al. (2017). The most successful neural-network based generative models in recent years are variational autoencoders (VAE) (Diederik & Welling (2014)) and generative adversarial networks (GANs) (Goodfellow et al. (2014)). The implementation variants of the LASIUM algorithm described in this paper rely on the original VAE model and on two specific variations of the GAN concept, respectively. MSGAN (aka Miss-GAN) (Mao et al. (

