DATASET CONDENSATION WITH GRADIENT MATCHING

Abstract

As the state-of-the-art machine learning methods in many fields rely on larger datasets, storing datasets and training models on them become significantly more expensive. This paper proposes a training set synthesis technique for data-efficient learning, called Dataset Condensation, that learns to condense large dataset into a small set of informative synthetic samples for training deep neural networks from scratch. We formulate this goal as a gradient matching problem between the gradients of deep neural network weights that are trained on the original and our synthetic data. We rigorously evaluate its performance in several computer vision benchmarks and demonstrate that it significantly outperforms the state-of-the-art methods 1 . Finally we explore the use of our method in continual learning and neural architecture search and report promising gains when limited memory and computations are available.

1. INTRODUCTION

Large-scale datasets, comprising millions of samples, are becoming the norm to obtain state-ofthe-art machine learning models in multiple fields including computer vision, natural language processing and speech recognition. At such scales, even storing and preprocessing the data becomes burdensome, and training machine learning models on them demands for specialized equipment and infrastructure. An effective way to deal with large data is data selection -identifying the most representative training samples -that aims at improving data efficiency of machine learning techniques. While classical data selection methods, also known as coreset construction (Agarwal et al., 2004; Har-Peled & Mazumdar, 2004; Feldman et al., 2013) , focus on clustering problems, recent work can be found in continual learning (Rebuffi et al., 2017; Toneva et al., 2019; Castro et al., 2018; Aljundi et al., 2019) and active learning (Sener & Savarese, 2018) where there is typically a fixed budget in storing and labeling training samples respectively. These methods commonly first define a criterion for representativeness (e.g. in terms of compactness (Rebuffi et al., 2017; Castro et al., 2018) , diversity (Sener & Savarese, 2018; Aljundi et al., 2019) , forgetfulness (Toneva et al., 2019) ), then select the representative samples based on the criterion, finally use the selected small set to train their model for a downstream task. Unfortunately, these methods have two shortcomings: they typically rely on i) heuristics (e.g. picking cluster centers) that does not guarantee any optimal solution for the downstream task (e.g. image classification), ii) presence of representative samples, which is neither guaranteed. A recent method, Dataset Distillation (DD) (Wang et al., 2018) goes beyond these limitations by learning a small set of informative images from large training data. In particular, the authors model the network parameters as a function of the synthetic training data and learn them by minimizing the training loss over the original training data w.r.t. synthetic data. Unlike in the coreset methods, the synthesized data are directly optimized for the downstream task and thus the success of the method does not rely on the presence of representative samples. Inspired from DD (Wang et al., 2018) , we focus on learning to synthesize informative samples that are optimized to train neural networks for downstream tasks and not limited to individual samples in original dataset. Like DD, our goal is to obtain the highest generalization performance with a model trained on a small set of synthetic images, ideally comparable performance to that of a model trained on the original images (see Figure 1(a) ). In particular, we investigate the following Figure 1 : Dataset Condensation (left) aims to generate a small set of synthetic images that can match the performance of a network trained on a large image dataset. Our method (right) realizes this goal by learning a synthetic set such that a deep network trained on it and the large set produces similar gradients w.r.t. its weights. The synthetic data can later be used to train a network from scratch in a small fraction of the original computational load. CE denotes Cross-Entropy. questions. Is it possible to i) compress a large image classification dataset into a small synthetic set, ii) train an image classification model on the synthetic set that can be further used to classify real images, iii) learn a single set of synthetic images that can be used to train different neural network architectures? To this end, we propose a Dataset Condensation method to learn a small set of "condensed" synthetic samples such that a deep neural network trained on them obtains not only similar performance but also a close solution to a network trained on the large training data in the network parameter space. We formulate this goal as a minimization problem between two sets of gradients of the network parameters that are computed for a training loss over a large fixed training set and a learnable condensed set (see Figure 1 (b)). We show that our method enables effective learning of synthetic images and neural networks trained on them, outperforms (Wang et al., 2018) and coreset methods with a wide margin in multiple computer vision benchmarks. In addition, learning a compact set of synthetic samples also benefits other learning problems when there is a fixed budget on training images. We show that our method outperforms popular data selection methods by providing more informative training samples in continual learning. Finally, we explore a promising use case of our method in neural architecture search, and show that -once our condensed images are learned -they can be used to train numerous network architectures extremely efficiently. Our method is related to knowledge distillation (KD) techniques (Hinton et al., 2015; Buciluǎ et al., 2006; Ba & Caruana, 2014; Romero et al., 2014 ) that transfer the knowledge in an ensemble of models to a single one. Unlike KD, we distill knowledge of a large training set into a small synthetic set. Our method is also related to Generative Adversarial Networks (Goodfellow et al., 2014a; Mirza & Osindero, 2014; Radford et al., 2015) and Variational AutoEncoders (Kingma & Welling, 2013) that synthesize high-fidelity samples by capturing the data distribution. In contrast, our goal is to generate informative samples for training deep neural networks rather than to produce "real-looking" samples. Finally our method is related to the methods that produce image patches by projecting the feature activations back to the input pixel space (Zeiler & Fergus, 2014) , reconstruct the input image by matching the feature activations (Mahendran & Vedaldi, 2015) , recover private training images for given training gradients (Zhu et al., 2019; Zhao et al., 2020) , synthesize features from semantic embeddings for zero-shot learning (Sariyildiz & Cinbis, 2019). Our goal is however to synthesize a set of condensed training images not to recover the original or missing training images. In the remainder of this paper, we first review the problem of dataset condensation and introduce our method in section 2, present and analyze our results in several image recognition benchmarks in section 3.1, showcase applications in continual learning and network architecture search in section 3.2, and conclude the paper with remarks for future directions in section 4.

2. METHOD

2.1 DATASET CONDENSATION Suppose we are given a large dataset consisting of |T | pairs of a training image and its class label T = {(x i , y i )}| |T | i=1 where x ∈ X ⊂ R d , y ∈ {0, . . . , C -1}, X is a d-dimensional input space and C is the number of classes. We wish to learn a differentiable function φ (i.e. deep neural network) case of large-scale models, one can adopt the back-optimization approach in (Domke, 2012) which re-defines θ S as the output of an incomplete optimization: θ S (S) = opt-alg θ (L S (θ), ς) (6) where opt-alg is a specific optimization procedure with a fixed number of steps (ς). In practice, θ T for different initializations can be trained first in an offline stage and then used as the target parameter vector in eq. ( 5). However, there are two potential issues by learning to regress θ T as the target vector. First the distance between θ T and intermediate values of θ S can be too big in the parameter space with multiple local minima traps along the path and thus it can be too challenging to reach. Second opt-alg involves a limited number of optimization steps as a tradeoff between speed and accuracy which may not be sufficient to take enough steps for reaching the optimal solution. These problems are similar to those of (Wang et al., 2018) , as they both involve parameterizing θ S with S and θ 0 .

2.3. DATASET CONDENSATION WITH CURRICULUM GRADIENT MATCHING

Here we propose a curriculum based approach to address the above mentioned challenges. The key idea is that we wish θ S to be close to not only the final θ T but also to follow a similar path to θ T throughout the optimization. While this can restrict the optimization dynamics for θ, we argue that it also enables a more guided optimization and effective use of the incomplete optimizer. We can now decompose eq. ( 5) into multiple subproblems: min S E θ0∼P θ 0 [ T -1 t=0 D(θ S t , θ T t )] subject to θ S t+1 (S) = opt-alg θ (L S (θ S t ), ς S ) and θ T t+1 = opt-alg θ (L T (θ T t ), ς T ) (7) where T is the number of iterations, ς S and ς T are the numbers of optimization steps for θ S and θ T respectively. In words, we wish to generate a set of condensed samples S such that the network parameters trained on them (θ S t ) are similar to the ones trained on the original training set (θ T t ) at each iteration t. In our preliminary experiments, we observe that θ S t+1 , which is parameterized with S, can successfully track θ T t+1 by updating S and minimizing D(θ S t , θ T t ) close to zero. In the case of one step gradient descent optimization for opt-alg, the update rule is: θ S t+1 ← θ S t -η θ ∇ θ L S (θ S t ) and θ T t+1 ← θ T t -η θ ∇ θ L T (θ T t ), where η θ is the learning rate. Based on our observation (D(θ S t , θ T t ) ≈ 0), we simplify the formulation in eq. ( 7) by replacing θ T t with θ S t and use θ to denote θ S in the rest of the paper: min S E θ0∼P θ 0 [ T -1 t=0 D(∇ θ L S (θ t ), ∇ θ L T (θ t ))]. We now have a single deep network with parameters θ trained on the synthetic set S which is optimized such that the distance between the gradients for the loss over the training samples L T w.r.t. θ and the gradients for the loss over the condensed samples L S w.r.t. θ is minimized. In words, our goal reduces to matching the gradients for the real and synthetic training loss w.r.t. θ via updating the condensed samples. This approximation has the key advantage over (Wang et al., 2018) and eq. ( 5) that it does not require the expensive unrolling of the recursive computation graph over the previous parameters {θ 0 , . . . , θ t-1 }. The important consequence is that the optimization is significantly faster, memory efficient and thus scales up to the state-of-the-art deep neural networks (e.g. ResNet (He et al., 2016) ). Discussion. The synthetic data contains not only samples but also their labels (s, y) that can be jointly learned by optimizing eq. ( 9) in theory. However, their joint optimization is challenging, as the content of the samples depend on their label and vice-versa. Thus in our experiments we learn to synthesize images for fixed labels, e.g. one synthetic image per class. Algorithm. We depict the optimization details in Alg. 1. At the outer level, it contains a loop over random weight initializations, as we want to obtain condensed images that can later be used to train previously unseen models. Once θ is randomly initialized, we use φ θ to first compute the loss over both the training samples (L T ), synthetic samples (L S ) and their gradients w.r.t. θ, then optimize the synthetic samples S to match these gradients ∇ θ L S to ∇ θ L T by applying ς S gradient descent steps with learning rate η S . We use the stochastic gradient descent optimization for both opt-alg θ and opt-alg S . Next we train θ on the updated synthetic images by minimizing the loss L S with learning rate η θ for ς θ steps. Note that we sample each real and synthetic batch pair from T and S containing samples from a single class and the synthetic data for each class are separately (or parallelly) updated at each iteration (t) for the following reasons: i) this reduces memory use at train time, ii) imitating the mean gradients w.r.t. the data from single class is easier compared to those of multiple classes. This does not bring any extra computational cost. Algorithm 1: Dataset condensation with gradient matching Input: Training set T Required: Randomly initialized set of synthetic samples S for C classes, probability distribution over randomly initialized weights P θ 0 , deep neural network φ θ , number of outer-loop steps K, number of inner-loop steps T , number of steps for updating weights ς θ and synthetic samples ςS in each inner-loop step respectively, learning rates for updating weights η θ and synthetic samples ηS . for k = 0, • • • , K -1 do Initialize θ0 ∼ P θ 0 for t = 0, • • • , T -1 do for c = 0, • • • , C -1 do 6 Sample a minibatch pair B T c ∼ T and B S c ∼ S B T c and B S c are of the same class c. Compute L T c = 1 |B T c | (x,y)∈B T c (φ θ t (x), y) and L S c = 1 |B S c | (s,y)∈B S c (φ θ t (s), y) Update Sc ← opt-alg S (D(∇ θ L S c (θt), ∇ θ L T c (θt)), ςS , ηS ) Update θt+1 ← opt-alg θ (L S (θt), ς θ , η θ ) Use the whole S Output: S Gradient matching loss. The matching loss D(•, •) in eq. ( 9) measures the distance between the gradients for L S and L T w.r.t. θ. When φ θ is a multi-layered neural network, the gradients correspond to a set of learnable 2D (out×in) and 4D (out×in×h×w) weights for each fully connected (FC) and convolutional layer resp where out, in, h, w are number of output and input channels, kernel height and width resp. The matching loss can be decomposed into a sum of layerwise losses as D(∇ θ L S , ∇ θ L T ) = L l=1 d(∇ θ (l) L S , ∇ θ (l) L T ) where l is the layer index, L is the number of layers with weights and d(A, B) = out i=1 1 - A i• • B i• A i• B i• where A i• and B i• are flattened vectors of gradients corresponding to each output node i, which is in dimensional for FC weights and in×h×w dimensional for convolutional weights. In contrast to (Lopez-Paz et al., 2017; Aljundi et al., 2019; Zhu et al., 2019) that ignore the layer-wise structure by flattening tensors over all layers to one vector and then computing the distance between two vectors, we group them for each output node. We found that this is a better distance for gradient matching (see the supplementary) and enables using a single learning rate across all layers.

3.1. DATASET CONDENSATION

First we evaluate classification performance with the condensed images on four standard benchmark datasets: digit recognition on MNIST (LeCun et al., 1998) , SVHN (Netzer et al., 2011) and object classification on FashionMNIST (Xiao et al., 2017) , CIFAR10 (Krizhevsky et al., 2009) . We test our method using six standard deep network architectures: MLP, ConvNet (Gidaris & Komodakis, 2018) , LeNet (LeCun et al., 1998) , AlexNet (Krizhevsky et al., 2012) learning (Snell et al., 2017; Vinyals et al., 2016; Gidaris & Komodakis, 2018) with D duplicate blocks, and each block has a convolutional layer with W (3 × 3) filters, a normalization layer N , an activation layer A and a pooling layer P , denoted as [W, N, A, P ] × D. The default ConvNet (unless specified otherwise) includes 3 blocks, each with 128 filters, followed by InstanceNorm (Ulyanov et al., 2016) , ReLU and AvgPooling modules. The final block is followed by a linear classifier. We use Kaiming initialization (He et al., 2015) for network weights. The synthetic images can be initialized from Gaussian noise or randomly selected real training images. More details about the datasets, networks and hyper-parameters can be found in the supplementary. The pipeline for dataset condensation has two stages: learning the condensed images (denoted as C) and training classifiers from scratch on them (denoted as T). Note that the model architectures used in two stages might be different. For the coreset baselines, the coreset is selected in the first stage. We investigate three settings: 1, 10 and 50 image/class learning, which means that the condensed set or coreset contains 1, 10 and 50 images per class respectively. Each method is run for 5 times, and 5 synthetic sets are generated in the first stage; each generated synthetic set is used to train 20 randomly initialized models in the second stage and evaluated on the test set, which amounts to evaluating 100 models in the second stage. In all experiments, we report the mean and standard deviation of these 100 testing results. Baselines. We compare our method to four coreset baselines (Random, Herding, K-Center and Forgetting) and also to DD (Wang et al., 2018) . In Random, the training samples are randomly selected as the coreset. Herding baseline, which selects closest samples to the cluster center, is based on (Welling, 2009) and used in (Rebuffi et al., 2017; Castro et al., 2018; Wu et al., 2019; Belouadah & Popescu, 2020) . K-Center (Wolf, 2011; Sener & Savarese, 2018) picks multiple center points such that the largest distance between a data point and its nearest center is minimized. For Herding and K-Center, we use models trained on the whole dataset to extract features, compute l 2 distance to centers. Forgetting method (Toneva et al., 2019) selects the training samples which are easy to forget during training. We do not compare to GSS-Greedy (Aljundi et al., 2019) , because it is also a similarity based greedy algorithm like K-Center, but GSS-Greedy trains an online learning model to measure the similarity of samples, which is different from general image classification problem. More detailed comparisons can be found in the supplementary. Comparison to coreset methods. We first compare our method to the coreset baselines on MNIST, FashionMNIST, SVHN and CIFAR10 in Table 1 using the default ConvNet in classification accuracy. Whole dataset indicates training on the whole original set which serves as an approximate upper-bound performance. First we observe that our method outperforms all the baselines significantly and achieves a comparable result (98.8%) in case of 50 images per class to the upper bound (99.6%) in MNIST which uses two orders of magnitude more training images per class (6000). We also obtain promising results in FashionMNIST, however, the gap between our method and upper bound is bigger in SVHN and CIFAR10 which contain more diverse images with varying foregrounds and backgrounds. We also observe that, (i) the random selection baseline is competitive to other coreset methods in 10 and 50 images per class and (ii) herding method is on average the best coreset technique. We visualize the condensed images produced by our method under 1 image/class setting in Figure 2 . Interestingly they are interpretable and look like "prototypes" of each class. Comparison to DD (Wang et al., 2018) . Unlike the setting in Table 1 , DD (Wang et al., 2018) reports results only for 10 images per class on MNIST and CIFAR10 over LeNet and AlexCifarNet (a customized AlexNet). We strictly follow the experimental setting in (Wang et al., 2018) , use the same architectures and report our and their original results in Table 3 for a fair comparison. Our method achieves significantly better performance than DD on both benchmarks; obtains 5% higher accuracy with only 1 synthetic sample per class than DD with 10 samples per class. In addition, our method obtains consistent results over multiple runs with a standard deviation of only 0.6% on MNIST, while DD's performance significantly vary over different runs (8.1%). Finally our method trains 2 times faster than DD and requires 50% less memory on CIFAR10 experiments. More detailed runtime and qualitative comparison can be found in the supplementary. Cross-architecture generalization. Another key advantage of our method is that the condensed images learned using one architecture can be used to train another unseen one. Here we learn 1 condensed image per class for MNIST over a diverse set of networks including MLP, ConvNet (Gidaris & Komodakis, 2018) , LeNet (LeCun et al., 1998) , AlexNet (Krizhevsky et al., 2012) , VGG-11 (Simonyan & Zisserman, 2014) and ResNet-18 (He et al., 2016) (see Table 2 ). Once the condensed sets are synthesized, we train every network on all the sets separately from scratch and evaluate their cross architecture performance in terms of classification accuracy on the MNIST test set. Table 2 shows that the condensed images, especially the ones that are trained with convolutional networks, perform well and are thus architecture generic. MLP generated images do not work well for training convolutional architectures which is possibly due to the mismatch between translation invariance properties of MLP and convolutional networks. Interestingly, MLP achieves better performance with convolutional network generated images than the MLP generated ones. The best results are obtained in most cases with ResNet generated images and ConvNet or ResNet as classifiers which is inline with their performances when trained on the original dataset. Number of condensed images. We also study the test performance of a ConvNet trained on them for MNIST, FashionMNIST, SVHN and CIFAR10 for various number of condensed images per class in Figure 3 in absolute and relative terms -normalized by its upper-bound. Increasing the number of condensed images improves the accuracies in all benchmarks and further closes the gap with the upper-bound performance especially in MNIST and FashionMNIST, while the gap remains larger in SVHN and CIFAR10. In addition, our method outperforms the coreset method -Herding by a large margin in all cases. Activation, normalization & pooling. We also study the effect of various activation (sigmoid, ReLU (Nair & Hinton, 2010; Zeiler et al., 2013) , leaky ReLU (Maas et al., 2013)), pooling (max, average) and normalization functions (batch (Ioffe & Szegedy, 2015) , group (Wu & He, 2018) , layer (Ba et al., 2016) , instance norm (Ulyanov et al., 2016) ) and have the following observations: i) leaky ReLU over ReLU and average pooling over max pooling enable learning better condensed images, as they allow for denser gradient flow; ii) instance normalization obtains better classifica- tion performance than its alternatives when used in the networks that are trained on a small set of condensed images. We refer to the supplementary for detailed results and discussion.

3.2. APPLICATIONS

Continual Learning First we apply our method to a continual-learning scenario (Rebuffi et al., 2017; Castro et al., 2018) where new tasks are learned incrementally and the goal is to preserve the performance on the old tasks while learning the new ones. We build our model on E2E method in (Castro et al., 2018) that uses a limited budget rehearsal memory (we consider 10 images/class here) to keep representative samples from the old tasks and knowledge distillation (KD) to regularize the network's output w.r.t. to previous predictions. We replace its sample selection mechanism (herding) with ours such that a set of condensed images are generated and stored in the memory, keep the rest of the model same and evaluate this model on the task-incremental learning problem on the digit recognition datasets, SVHN (Netzer et al., 2011) , MNIST (LeCun et al., 1998) and USPS (Hull, 1994) in the same order. MNIST and USPS images are reshaped to 32 × 32 RGB images. We compare our method to E2E (Castro et al., 2018) , depicted as herding in Figure 4 , with and without KD regularization. The experiment contains 3 incremental training stages (SVHN→MNIST→USPS) and testing accuracies are computed by averaging over the test sets of the previous and current tasks after each stage. The desired outcome is to obtain high mean classification accuracy at T3. The results indicate that the condensed images are more data-efficient than the ones sampled by herding and thus our method outperforms E2E in both settings, while by a larger margin (2.3% at T3) when KD is not employed. Neural Architecture Search. Here we explore the use of our method in a simple neural architecture search (NAS) experiment on CIFAR10 which typically requires expensive training of numerous architectures multiple times on the whole training set and picking the best performing ones on a validation set. Our goal is to verify that our condensed images can be used to efficiently train multiple networks to identify the best network. To this end, we construct the search space of 720 ConvNets as described in Section 3.1 by varying hyper-parameters W , N , A, P , D over an uniform grid (see supplementary for more details), train them for 100 epochs on three small proxy datasets (10 images/class) that are obtained with Random sampling, Herding and our method. Note that we train the condensed images for once only with the default ConvNet architecture and use them to train all kinds of architectures. We also compare to early-stopping (Li & Talwalkar, 2020) in which the model is trained on whole training set but with the same number of training iterations as the one required for the small proxy datasets, in other words, for the same amount of computations. Table 4 depicts i) the average test performance of the best selected model over 5 runs when trained on the whole dataset, ii) Spearman's rank correlation coefficient between the validation accuracies obtained by training the selected top 10 models on the proxy dataset and whole dataset, iii) time for training 720 architectures on a NVIDIA GTX1080-Ti GPU, and iv) memory print of the training images. Our method achieves the highest testing performance (84.5%) and performance correlation (0.79), meanwhile significantly decreases the the searching time (from 8604.3 to 18.8 minutes) and storage space (from 5 × 10 4 to 1 × 10 2 images) compared to whole-dataset training. The competitive early-stopping baseline achieves on par performance for the best performing model with ours, however, the rank correlation (0.42) of top 10 models is significantly lower than ours (0.79) which indicates unreliable correlation of performances between early-stopping and whole-dataset training. Furthermore, early-stopping needs 100 times as many training images as ours needs. Note that the training time for synthetic images is around 50 minutes (for K = 500) which is one time off and negligible cost when training thousands even millions of candidate architectures in NAS.

4. CONCLUSION

In this paper, we propose a dataset condensation method that learns to synthesize a small set of informative images. We show that these images are significantly more data-efficient than the same number of original images and the ones produced by the previous method, and they are not architecture dependent, can be used to train different deep networks. Once trained, they can be used to lower the memory print of datasets and efficiently train numerous networks which are crucial in continual learning and neural architecture search respectively. For future work, we plan to explore the use of condensed images in more diverse and thus challenging datasets like ImageNet (Deng et al., 2009 ) that contain higher resolution images with larger variations in appearance and pose of objects, background.

A IMPLEMENTATION DETAILS

In this part, we explain the implementation details for the dataset condensation, continual learning and neural architecture search experiments. Dataset condensation. The presented experiments involve tuning of six hyperparameters -the number of outer-loop K and inner-loop steps T , learning rates η S and number of optimization steps ς S for the condensed samples, learning rates η θ and number of optimization steps ς θ for the model weights. In all experiments, we set K = 1000, η S = 0.1, η θ = 0.01, ς S = 1 and employ Stochastic Gradient Descent (SGD) as the optimizer. The only exception is that we set η S to 0.01 for synthesizing data with MLP in cross-architecture experiments (Table 2 ), as MLP requires a slightly different treatment. Note that while K is the maximum number of outer-loop steps, the optimization can early-stop automatically if it converges before K steps. For the remaining hyperparameters, we use different sets for 1, 10 and 50 image(s)/class learning. We set T = 1, ς θ = 1 for 1 image/class, T = 10, ς θ = 50 for 10 images/class, T = 50, ς θ = 10 for 50 images/class learning. Note that when T = 1, it is not required to update the model parameters (Step 9 in Algorithm 1), as this model is not further used. For those experiments where more than 10 images/class are synthesized, we set T to be the same number as the synthetic images per class and ς θ = 500/T , e.g. T = 20, ς θ = 25 for 20 images/class learning. The ablation study on hyper-parameters are given in Appendix B which shows that our method is not sensitive to varying hyper-parameters. We do separate-class mini-batch sampling for Step 6 in Algorithm 1. Specifically, we sample a mini-batch pair B T c and B S c that contain real and synthetic images from the same class c at each inner iteration. Then, the matching loss for each class is computed with the sampled mini-batch pair and used to update corresponding synthetic images S c by back-propogation (Step 7 and 8). This is repeated separately (or parallelly given enough computational resources) for every class. Training as such is not slower than using mixed-class batches. Although our method still works well when we randomly sample the real and synthetic mini-batches with mixed labels, we found that separateclass strategy is faster to train as matching gradients w.r.t. data from single class is easier compared to those of multiple classes. In experiments, we randomly sample 256 real images of a class as a mini-batch to calculate the mean gradient and match it with the mean gradient that is averaged over all synthetic samples with the same class label. The performance is not sensitive to the size of real-image mini-batch if it is greater than 64. In all experiments, we use the standard train/test splits of the datasets -the train/test statistics are shown in Table T5 . We apply data augmentation (crop, scale and rotate) only for experiments (coreset methods and ours) on MNIST. The only exception is that we also use data augmentation when compared to DD (Wang et al., 2018) on CIFAR10 with AlexCifarNet, and data augmentation is also used in (Wang et al., 2018) . For initialization of condensed images, we tried both Gaussian noise and randomly selected real training images, and obtained overall comparable performances in different settings and datasets. Then, we used Gaussian noise for initialization in experiments. In the first stage -while training the condensed images -, we use Batch Normalization in the VGG and ResNet networks. For reliable estimation of the running mean and variance, we sample many real training data to estimate the running mean and variance and then freeze them ahead of Step 7. In the second stage -while training a deep network on the condensed set -, we replace Batch Normalization layers with Instance Normalization in VGG and ResNet, due to the fact that the batch statistics are not reliable when training networks with few condensed images. Another minor modification that we apply to the standard network ResNet architecture in the first stage is replacing the strided convolutions where stride = 2 with convolutional layers where stride = 1 coupled with an average pooling layer. We observe that this change enables more detailed (per pixel) gradients w.r.t. the condensed images and leads to better condensed images. Continual learning. In this experiment, we focus on a task-incremental learning on SVHN, MNIST and USPS with the given order. The three tasks share the same label space, however have significantly different image statistics. The images of the three datasets are reshaped to 32×32 RGB size for standardization. We use the standard splits for training sets and randomly sample 2,000 test images for each datasets to obtain a balanced evaluation over three datasets. Thus each model is tested on a growing test set with 2,000, 4,000 and 6,000 images at the three stages respectively. We use the default ConvNet in this experiment and set the weight of distillation loss to 1.0 and the temperature to 2. We run 5,000 and 500 iterations for training and balanced finetuning as in (Castro et al., 2018 ) with the learning rates 0.01 and 0.001 respectively. We run 5 experiments and report the mean and standard variance in Figure 4 . Neural Architecture Search. To construct the searching space of 720 ConvNets, we vary hyperparameters W ∈ {32, 64, 128, 256}, D ∈ {1, 2, 3, 4}, N ∈ {None, BatchNorm, LayerNorm, InstanceNorm, GroupNorm}, A ∈ {Sigmoid, ReLu, LeakyReLu}, P ∈ {None, MaxPooling, AvgPooling}. We randomly sample 5,000 images from the 50,000 training images in CIFAR10 as the validation set. Every candidate ConvNet is trained with the proxy dataset, and then evaluated on the validation set. These candidate ConvNets are ranked by the validation performance. 10 architectures with top validation accuracies are selected to calculate Spearman's rank correlation coefficient, because the best model that we want will come from the top 10 architectures. We train each ConvNet for 5 times to get averaged validation and testing accuracies. We visualize the performance correlation for different proxy datasets in Figure F5 . Obviously, the condensed proxy dataset produced by our method achieves the highest performance correlation (0.79) which significantly higher than early-stopping (0.42). It means our method can produce more reliable results for NAS.

B FURTHER ANALYSIS

Next we provide additional results on ablative studies over various deep network layers including activation, pooling and normalization functions and also over depth and width of deep network architecture. We also study the selection of hyper-parameters and the gradient distance metric. An additional qualitative analysis on the learned condensed images is also given. Ablation study on activation functions. Here we study the use of three activation functions -Sigmoid, ReLU, LeakyReLu (negative slope is set to 0.01) -in two stages, when training condensed images (denoted as C) and when training a ConvNet from scratch on the learned condensed images (denoted as T). The experiments are conducted in MNIST dataset for 1 condensed image/class setting. Table T6 shows that all three activation functions are good for the first stage while generating good condensed images, however, Sigmoid performs poor in the second stage while learning a classifier on the condensed images -its testing accuracies are lower than ReLu and LeakyReLu by around 5%. This suggests that ReLU can provide sufficiently informative gradients for learning condensed images, though the gradient of ReLU w.r.t. to its input is typically sparse. Ablation study on pooling functions. Next we investigate the performance of two pooling functions -average pooling and max pooling -also no pooling for 1 image/class dataset condensation with ConvNet in MNIST in terms of classification accuracy. Table T7 shows that max and average pooling both perform significantly better than no pooling (None) when they are used in the second stage. When the condensed samples are trained and tested on models with average pooling, the best testing accuracy (91.7 ± 0.5%) is obtained, possibly, because average pooling provides more informative and smooth gradients for the whole image rather than only for its discriminative parts. Ablation study on normalization functions. Next we study the performance of four normalization options -No normalization, Batch (Ioffe & Szegedy, 2015) , Layer (Ba et al., 2016) , Instance (Ulyanov et al., 2016) and Group Normalization (Wu & He, 2018) (number of groups is set to be four) -for 1 image/class dataset condensation with ConvNet architecture in MNIST classification accuracy. Table T8 shows that the normalization layer has little influence for learning the condensed set, while the choice of normalization layer is important for training networks on the condensed set. LayerNorm and GroupNorm have similar performance, and InstanceNorm is the best choice for training a model on condensed images. BatchNorm obtains lower performance which is similar to None (no normalization), as it is known to perform poorly when training models on few condensed samples as also observed in (Wu & He, 2018) . Note that Batch Normalization does not allow for a stable training in the first stage (C); thus we replace its running mean and variance for each batch with those of randomly sampled real training images. Ablation study on network depth and width. Here we study the effect of network depth and width for 1 image/class dataset condensation with ConvNet architecture in MNIST in terms of classification accuracy. To this end we conduct multiple experiments by varying the depth and width of the networks that are used to train condensed synthetic images and that are trained to classify testing data in ConvNet architecture and report the results in Table T9 and Table T10 . In Table T9 , we observe that deeper ConvNets with more blocks generate better condensed images that results in better classification performance when a network is trained on them, while ConvNet with 3 blocks performs best as classifier. Interestingly, Table T10 shows that the best results are obtained with the classifier that has 128 filters at each block, while network width (number of filters at each block) in generation has little overall impact on the final classification performance. Ablation study on hyper-parameters. Our performance is not sensitive to hyper-parameter selection. The testing accuracy for various K and T , when learning 10 images/class condensed sets, is depicted in Figure F6 . The results show that the optimum K and T are around similar values across all datasets. Thus we simply set K to 1000 and T to 10 for all datasets. Similarly, for the remaining ones including learning rate, weight decay, we use a single set of hyperparameters that are observed to work well for all datasets and architectures in our preliminary experiments. Ablation study on gradient distance metric. To prove the effectiveness and robustness of the proposed distance metric for gradients (or weights), we compare to the traditional ones (Lopez-Paz et al., 2017; Aljundi et al., 2019; Zhu et al., 2019) image/class learning experiment on MNIST with different architectures. For simplicity, the synthetic images are learned and tested on the same architecture in this experiment. Further qualitative analysis We first depict the condensed images that are learned on MNIST, FashionMNIST, SVHN and CIFAR10 datasets in one experiment using the default ConvNet in 10 images/class setting in Figure F7 . It is interesting that the 10 images/class results in Figure F7 are diverse which cover the main variations, while the condensed images for 1 image/class setting (see Figure 2 ) look like the "prototype" of each class. For example, in Figure F7 (a), the ten images of "four" indicate ten different styles. The ten "bag" images in Figure F7 (b) are significantly different from each other, similarly "wallet" (1st row), "shopping bag" (3rd row), "handbag" (8th row) and "schoolbag" (10th row). Figure F7 (c ) also shows the diverse house numbers with different shapes, colors and shadows. Besides, different poses of a "horse" have been learned in Figure F7 (d) .

C COMPARISON TO MORE BASELINES

Optimal random selection. One interesting and strong baseline is Optimal Random Selection (ORS) that we implement random selection experiments for 1,000 times and pick the best ones. Table T12 presents the performance comparison to the selected Top 1000 (all), Top 100 and Top 10 coresets. These optimal coresets are selected by ranking their performance. Obviously, the condensed set generated by our method surpasses the selected Top 10 of 1000 coresets with a large margin on all four datasets. Generative model. We also compare to the popular generative model, namely, Conditional Generative Adversarial Networks (cGAN) (Mirza & Osindero, 2014) . The generator has two blocks which consists of the Up-sampling (scale factor=2), Convolution (stride=1), BatchNorm and LeakyReLu layers. The discriminator has three blocks which consists of Convolution (stride=2), BatchNorm and LeakyReLu layers. In additional to the random noise, we also input the class label as the condition. We generate 1 and 10 images per class for each dataset with random noise. Table T12 shows that the images produced by cGAN have similar performances to those randomly selected coresets (i.e. Top 1000). It is reasonable, because the aim of cGAN is to generate real-look images. In contrast, our method aims to generate images that can train deep neural networks efficiently.

Analysis of coreset performances

We find that K-Center (Wolf, 2011; Sener & Savarese, 2018) and Forgetting (Toneva et al., 2019) Random and Herding (Rebuffi et al., 2017) , in this experimental setting. After analyzing the algorithms and coresets, we find two main reasons. 1) K-Center and Forgetting are not designed for training deep networks from scratch, instead they are for active learning and continual learning respectively. 2) The two algorithms both tend to select "hard" samples which are often outliers when only a small number of images are selected. These outliers confuse the training, which results in worse performance. Specifically, the first sample per class in K-Center coreset is initialized by selecting the one closest to each class center. The later ones selected by the greedy criterion that pursues maximum coverage are often outliers which confuse the training. Performance on CIFAR100. We supplement the performance comparison on CIFAR100 dataset which includes 10 times as many classes as other benchmarks. More classes while fewer images per class makes CIFAR100 significantly more challenging than other datasets. We use the same set of hyper-parameters for CIFAR100 as other datasets. Table T13 depicts the performances of coreset selection methods, Label Distillation (LD) Bohdal et al. (2020) and ours. Our method achieves 12.8% and 25.2% testing accuracies on CIFAR100 when learning 1 and 10 images per class, which are the best compared with others. D FURTHER COMPARISON TO DD (WANG ET AL., 2018) Next we compare our method to DD (Wang et al., 2018) first quantitatively in terms of crossarchitecture generalization, then qualitatively in terms of synthetic image quality, and finally in terms of computational load for training synthetic images. Note that we use the original source code to obtain the results for DD that is provided by the authors of DD in the experiments.

Generalization ability comparison.

Here we compare the generalization ability across different deep network architectures to DD. To this end, we use the synthesized 10 images/class data learned with LeNet on MNIST to train MLP, ConvNet, LeNet, AlexNet, VGG11 and ResNet18 and report the results in Table T14 . We see that that the condensed set produced by our method achieves good classification performances with all architectures, while the synthetic set produced by DD perform poorly when used to trained some architectures, e.g. AlexNet, VGG and ResNet. Note that DD generates learning rates to be used in every training step in addition to the synthetic data. This is in contrast to our method which does not learn learning rates for specific training steps. Although the tied learning rates improve the performance of DD while training and testing on the same architecture, they will hinder the generalization to unseen architectures. 

E EXTENDED RELATED WORK

Variations of Dataset Distillation. There exists recent work that extends Dataset Distillation (Wang et al., 2018) . For example, (Sucholutsky & Schonlau, 2019; Bohdal et al., 2020) aim to improve DD by learning soft labels with/without synthetic images. (Such et al., 2020) utilizes a generator to synthesize images instead of directly updating image pixels. However, the reported quantitative and qualitative improvements over DD are minor compared to our improvements. In addition, none of these methods have thoroughly verified the cross-architecture generalization ability of the synthetic images. Zero-shot Knowledge Distillation. Recent zero-shot KD methods (Lopes et al., 2017; Nayak et al., 2019) aim to perform KD from a trained model in the absence of training data by generating synthetic data as the intermediate production to further use. Unlike them, our method does not require pretrained teacher models to provide the knowledge, i.e. to obtain the features and labels. Data Privacy & Federated Learning. Synthetic dataset is also a promising solution to protecting data privacy and enabling safe federated learning. There exists some work that uses synthetic dataset to protect the privacy of medical dataset (Li et al., 2020) and reduce the communication rounds in federated learning (Zhou et al., 2020) . Although transmitting model weights or gradients (Zhu et al., 2019; Zhao et al., 2020) may increase the transmission security, the huge parameters of modern deep neural networks are prohibitive to transmit frequently. In contrast, transmitting small-scale synthetic dataset between clients and server is low-cost (Goetz & Tewari, 2020) .



The implementation is available at https://github.com/VICO-UoE/DatasetCondensation. Local smoothness is frequently used to obtain explicit first-order local approximations in deep networks (e.g. see(Rifai et al., 2012; Goodfellow et al., 2014b;Koh & Liang, 2017)).



Figure 3: Absolute and relative testing accuracies for varying the number of condensed images/class for MNIST, FashionMNIST, SVHN and CIFAR10. The relative accuracy means the ratio compared to its upperbound, i.e. training with the whole dataset.

Figure F6: Ablation study on the hyper-parameters K and T when learning 10 images/class condensed sets.

The performance comparison to coreset methods. This table shows the testing accuracies (%) of different methods on four datasets. ConvNet is used for training and testing. Img/Cls: image(s) per class, Ratio (%): the ratio of condensed images to whole training set.

5±1.2 63.9±6.5 77.3±5.8 70.9±11.6 53.2±7.0 80.9±3.6 ConvNet 69.6±1.6 91.7±0.5 85.3±1.8 85.1±3.0 83.4±1.8 90.0±0.8 LeNet 71.0±1.6 90.3±1.2 85.0±1.7 84.7±2.4 80.3±2.7 89.0±0.8 AlexNet 72.1±1.7 87.5±1.6 84.0±2.8 82.7±2.9 81.2±3.0 88.9±1.1 VGG 70.3±1.6 90.1±0.7 83.9±2.7 83.4±3.7 81.7±2.6 89.1±0.9 ResNet 73.6±1.2 91.6±0.5 86.4±1.5 85.4±1.9 83.4±2.4 89.4±0.9 Cross-architecture performance in testing accu-

Comparison to DD(Wang et al., 2018) in terms of testing accuracy (%).

Neural Architecture Search. Methods are compared in performance, ranking correlation, time and memory cost.

Train/test statistics for USPS, MNIST, FashionMNIST, SVHN, CIFAR10 and CIFAR100 datasets.

The performance correlation between the training on proxy dataset and whole-dataset. For each proxy dataset, the best 10 models are selected based on validation set performance. In the figure, each point represents an architecture. Cross-activation experiments in accuracy (%) for 1 condensed image/class in MNIST.

Cross-pooling experiments in accu-

Cross-normalization experiments in accuracy (%) for 1 condensed image/class in MNIST.

Cross-depth performance in accuracy (%) for 1 condensed image/class in MNIST.

TableT11shows that the proposed gradient distance metric remarkably outperforms others on complex architectures (e.g. LeNet, AlexNet, VGG and ResNet) and achieves the best performances in most settings, which means it is more effective and robust than the traditional ones. Note that we set η S = 0.1 for MLP-Euclidean and MLP-Cosine because it works better than η S = 0.01. Ablation study on different gradient distance metrics. Obviously, the proposed distance metric is more effective and robust. Euclidean: squared Euclidean distance, Cosine: Cosine distance.

The

The performance comparison on CIFAR100. This table shows the testing accuracies (%) of different methods. ConvNet is used for training and testing except that LD † uses AlexNet. Img/Cls: image(s) per class, Ratio (%): the ratio of condensed images to whole training set.

Generalization ability comparison to DD. The 10 condensed images per class are trained with LeNet, and tested on various architectures. It shows that condensed images generated by our method have better generalization ability.

Time and memory use for training DD and our method in 10 images/class setting.

acknowledgement

Acknowledgment. This work is funded by China Scholarship Council 201806010331 and the EPSRC programme grant Visual AI EP/T028572/1. We thank Iain Murray and Oisin Mac Aodha for their valuable feedback.

annex

Qualitative comparison. We also provide a qualitative comparison to to DD in terms of image quality in Figure F8 . Note that both of the synthetic sets are trained with LeNet on MNIST and AlexCifarNet on CIFAR10. Our method produces more interpretable and realistic images than DD, although it is not our goal. The MNIST images produced by DD are noisy, and the CIFAR10 images produced by DD do not show any clear structure of the corresponding class. In contrast, the MNIST and CIFAR10 images produced by our method are both visually meaningful and diverse.Training memory and time. One advantage of our method is that we decouple the model weights from its previous states in training, while DD requires to maintain the recursive computation graph which is not scalable to large models and inner-loop optimizers with many steps. Hence, our method requires less training time and memory cost. We compare the training time and memory cost required by DD and our method with one NVIDIA GTX1080-Ti GPU. Table T15 shows that our method requires significantly less memory and training time than DD and provides an approximation reduction of 17% and 55% in memory and 71% and 51% in train time to learn MNIST and CIFAR10 datasets respectively. Furthermore, our training time and memory cost can be significantly decreased by using smaller hyper-parameters, e.g. K, T and the batch size of sampled real images, with a slight performance decline (refer to Figure F6 ).

