NON-INHERENT FEATURE COMPATIBLE LEARNING

Abstract

The need of Feature Compatible Learning (FCL) arises from many large scale retrieval-based applications, where updating the entire library of embedding vectors is expensive. When an upgraded embedding model shows potential, it is desired to transform the benefit of the new model without refreshing the library. While progresses have been made along this new direction, existing approaches for feature compatible learning mostly rely on old training data and classifiers, which are not available in many industry settings. In this work, we introduce an approach for feature compatible learning without inheriting old classifier and training data, i.e., Non-Inherent Feature Compatible Learning. Our approach requires only features extracted by old model's backbone and new training data, and makes no assumption about the overlap between old and new training data. We propose a unified framework for FCL, and extend it to handle the case where the old model is a black-box. Specifically, we learn a simple pseudo classifier in lieu of the old model, and further enhance it with a random walk algorithm. As a result, the embedding features produced by the new model can be matched with those from the old model without sacrificing performance. Experiments on ImageNet ILSVRC 2012 and Places365 data proved the efficacy of the proposed approach.

1. INTRODUCTION

In recent years, deep learning based methods achieved huge success in various of computer vision tasks, especially for visual searching since they could provide powerful feature representations. In a typical visual search system, the deployed deep learning model extracts the features of both gallery and query images as discriminate representations. During the retrieval stage, gallery images will be ranked based on their feature distances (e.g. Euclidean distance) to query images. In conventional approaches, the query and gallery features are generated by the same model. Once the deployed model of retrieval system is updated, the entire set of gallery features need to be 'backfilled' or 'reindexed' (Shen et al., 2020) . As time goes by, the gallery becomes extremely large and 'backfilling' could be a painful process since millions even billions of images need to be re-processed by the new model, which is computationally expensive. There has to be a new mechanism that processes gallery images and the query image with two different models, while still maintaining the retrieval accuracy. In other words, the new deployed model extracted features should be 'compatible' to the existing ones without sacrificing accuracy. Such feature compatible learning problem is also named as 'Backward-Compatible Training' (Shen et al., 2020) , or 'Asymmetric Metric Learning' (Budnik & Avrithis, 2020) . Existing approaches for feature compatible learning assumed significant overlap between new and old training sets. In Shen et al. (2020) , the training set for new embedding model is a superset of the old set. In Budnik & Avrithis (2020), the training set for large and small models is the same, which means obtaining new model in an incremental way is not possible. Besides, in Shen et al. (2020) , the classifier for old model is also needed for computing the influence loss, which is a strong requirement in real applications. As an example, a model deployed in a recommendation system as a black-box API takes images as input and returns the processed features, but the parameters of the model are not accessible. In addition, its classifier and training details are not available, neither does the formula of the loss function. This kind of setting is quite common for various practical reasons in search, recommendation, content understanding and review applications.

