NASOA: TOWARDS FASTER TASK-ORIENTED ONLINE FINE-TUNING

Abstract

Fine-tuning from pre-trained ImageNet models has been a simple, effective, and popular approach for various computer vision tasks. The common practice of finetuning is to adopt a default hyperparameter setting with a fixed pre-trained model, while both of them are not optimized for specific tasks and time constraints. Moreover, in cloud computing or GPU clusters where the tasks arrive sequentially in a stream, faster online fine-tuning is a more desired and realistic strategy for saving money, energy consumption, and CO2 emission. In this paper, we propose a joint Neural Architecture Search and Online Adaption framework named NASOA towards a faster task-oriented fine-tuning upon the request of users. Specifically, NASOA first adopts an offline NAS to identify a group of training-efficient networks to form a pretrained model zoo. We propose a novel joint block and macro level search space to enable a flexible and efficient search. Then, by estimating fine-tuning performance via an adaptive model by accumulating experience from the past tasks, an online schedule generator is proposed to pick up the most suitable model and generate a personalized training regime with respect to each desired task in a one-shot fashion. The resulting model zoo 1 is more training efficient than SOTA NAS models, e.g. 6x faster than RegNetY-16GF, and 1.7x faster than EfficientNetB3. Experiments on multiple datasets also show that NASOA achieves much better fine-tuning results, i.e. improving around 2.1% accuracy than the best performance in RegNet series under various time constraints and tasks; 40x faster compared to the BOHB method.

1. INTRODUCTION

Fine-tuning using pre-trained models becomes the de-facto standard in the field of computer vision because of its impressive results on various downstream tasks such as fine-grained image classification (Nilsback & Zisserman, 2008; Welinder et al., 2010 ), object detection (He et al., 2019; Jiang et al., 2018; Xu et al., 2019) and segmentation (Chen et al., 2017; Liu et al., 2019) . Kornblith et al. (2019); He et al. (2019) verified that fine-tuning pre-trained networks outperform training from scratch. It can further help to avoid over-fitting (Cui et al., 2018) as well as reduce training time significantly (He et al., 2019) . Due to those merits, many cloud computing and AutoML pipelines provide fine-tuning services for an online stream of upcoming users with new data, different tasks and time limits. In order to save the user's time, money, energy consumption, or even CO2 emission, an efficient online automated fine-tuning framework is practically useful and in great demand. Thus, in this work, we propose to explore the problem of faster online fine-tuning. The conventional practice of fine-tuning is to adopt a set of predefined hyperparameters for training a predefined model (Li et al., 2020) . It has three drawbacks in the current online setting: 1) The design of the backbone model is not optimized for the upcoming fine-tuning task and the selection of the backbone model is not data-specific. 2) A default setting of hyperparameters may not be optimal across tasks and the training settings may not meet the time constraints provided by users. 3) With the incoming tasks, the regular diagram is not suitable for this online setting since it cannot memorize and accumulate experience from the past fine-tuning tasks. Thus, we propose to decouple our faster fine-tuning problem into two parts: finding efficient fine-tuning networks and generating optimal fine-tuning schedules pertinent to specific time constraints in an online learning fashion.



The efficient training model zoo (ET-NAS) has been released at: https://github.com/NAS-OA/ NASOA

