FEED-FORWARD LATENT DOMAIN ADAPTATION

Abstract

We study the highly practical but comparatively under-studied problem of latentdomain adaptation, where a source model should be adapted to a target dataset that contains a mixture of unlabelled domain-relevant and domain-irrelevant examples. Furthermore, motivated by the requirements for data privacy and the need for embedded and resource-constrained devices of all kinds to adapt to local data distributions, we focus on the setting of feed-forward source-free domain adaptation, where adaptation should not require access to the source dataset, and also be back propagation-free. Our solution is to meta-learn a network capable of embedding the mixed-relevance target dataset and dynamically adapting inference for target examples using cross-attention. The resulting framework leads to consistent improvement on strong ERM baselines. We also show that our framework sometimes even improves on the upper bound of domain-supervised adaptation, where only domain-relevant instances are provided for adaptation. This suggests that human annotated domain labels may not always be optimal, and raises the possibility of doing better through automated instance selection.

1. INTRODUCTION

Domain shift presents a real-world challenge for the application of machine learning models because performance degrades when deployment data are not from the training data distribution. This issue is ubiquitious as it is often impossible or prohibitively costly to pre-collect and annotate training data that is sufficiently representative of test data statistics. The field of domain adaptation (Kouw & Loog, 2021; Csurka et al., 2022) has therefore attracted a lot of attention with the promise of adapting models during deployment to perform well using only unlabeled deployment data. The main body of work in deep domain adaptation assumes that there is a pre-specified source domain and a pre-specified target domain. An unlabeled adaptation set is provided from the target domain, and various methods define different learning objectives that update a deep model on the unlabeled adaptation set, with the aim of improving performance on new test data drawn from the target domain. In this paper we make two main contributions: A conceptual contribution, of a highly practical variant of the domain adaptation problem; and an algorithm for effective domain adaptation in this condition. A motivating scenario Let us introduce one potential illustrative application scenario that motivates the variant of the domain adaptation problem that we propose here. Suppose that a robot or other mobile embedded vision system needs to recognise objects. Because it is mobile, it may encounter objects in different unconstrained contexts, e.g., indoor or outdoor backgrounds, sunny or rainy weather, rooms with lights on or lights off, etc. The robot's object recognition model should adapt in order to maintain strong performance across all these conditions, for example by adapting based on a buffer of recently experienced unlabelled images. However, unlike standard pre-defined domain adaptation benchmarks with neatly curated domains, there are two new challenges: 1) Using such a buffer as the adaptation set means that the adaptation data can be of mixed relevance to the test image to be processed at any given instant. For example the recent history used for adaptation may span multiple rooms, while any individual test image comes from a specific room. 2) The adaptation needs to happen on-board the robot and ideally happen in real-time as the adaptation set itself is updated over time. The first challenge is the latent domain challenge, wherein uncurated adaptation sets do not have consistent relevance to a given test image (Fig. 1 ) The second challenge requires adaptation to take place without back-propagation (which is too costly and not supported on most embedded platforms). It means adaptation should be feed-forward.

