THEORETICAL CHARACTERIZATION OF THE GENERALIZATION PERFORMANCE OF OVERFITTED META-LEARNING

Abstract

Meta-learning has arisen as a successful method for improving training performance by training over many similar tasks, especially with deep neural networks (DNNs). However, the theoretical understanding of when and why overparameterized models such as DNNs can generalize well in meta-learning is still limited. As an initial step towards addressing this challenge, this paper studies the generalization performance of overfitted meta-learning under a linear regression model with Gaussian features. In contrast to a few recent studies along the same line, our framework allows the number of model parameters to be arbitrarily larger than the number of features in the ground truth signal, and hence naturally captures the overparameterized regime in practical deep meta-learning. We show that the overfitted min ℓ 2 -norm solution of model-agnostic meta-learning (MAML) can be beneficial, which is similar to the recent remarkable findings on "benign overfitting" and "double descent" phenomenon in the classical (single-task) linear regression. However, due to the uniqueness of meta-learning such as task-specific gradient descent inner training and the diversity/fluctuation of the ground-truth signals among training tasks, we find new and interesting properties that do not exist in single-task linear regression. We first provide a high-probability upper bound (under reasonable tightness) on the generalization error, where certain terms decrease when the number of features increases. Our analysis suggests that benign overfitting is more significant and easier to observe when the noise and the diversity/fluctuation of the ground truth of each training task are large. Under this circumstance, we show that the overfitted min ℓ 2 -norm solution can achieve an even lower generalization error than the underparameterized solution.

1. INTRODUCTION

Meta-learning is designed to learn a task by utilizing the training samples of many similar tasks, i.e., learning to learn (Thrun & Pratt, 1998) . With deep neural networks (DNNs), the success of meta-learning has been shown by many works using experiments, e.g., (Antoniou et al., 2018; Finn et al., 2017) . However, theoretical results on why DNNs have a good generalization performance in meta-learning are still limited. Although DNNs have so many parameters that can completely fit all training samples from all tasks, it is unclear why such an overfitted solution can still generalize well, which seems to defy the classical knowledge bias-variance-tradeoff (Bishop, 2006; Hastie et al., 2009; Stein, 1956; James & Stein, 1992; LeCun et al., 1991; Tikhonov, 1943) . The recent studies on the "benign overfitting" and "double-descent" phenomena in classical (singletask) linear regression have brought new insights on the generalization performance of overfitted solutions. Specifically, "benign overfitting" and "double descent" describe the phenomenon that the test error descends again in the overparameterized regime in linear regression setup (Belkin et al., 2018; 2019; Bartlett et al., 2020; Hastie et al., 2019; Muthukumar et al., 2019; Ju et al., 2020; Mei & Montanari, 2019) . Depending on different settings, the shape and the properties of the descent curve of the test error can differ dramatically. For example, Ju et al. (2020) showed that the min ℓ 1 -

