CONTRASTIVE ALIGNMENT OF VISION TO LANGUAGE THROUGH PARAMETER-EFFICIENT TRANSFER LEARN-ING

Abstract

Contrastive vision-language models (e.g. CLIP) are typically created by updating all the parameters of a vision model and language model through contrastive training. Can such models be created by a small number of parameter updates to an already-trained language model and vision model? The literature describes techniques that can create vision-language models by updating a small number of parameters in a language model, but these require already aligned visual representations and are non-contrastive, hence unusable for latency-sensitive applications such as neural search. We explore the feasibility and benefits of parameter-efficient contrastive vision-language alignment through transfer learning: creating a model such as CLIP by minimally updating an already-trained vision and language model. We find that a minimal set of parameter updates (<7%) can achieve the same performance as full-model training, and updating specific components (<1% of parameters) can match 75% of full-model training. We describe a series of experiments: we show that existing knowledge is conserved more strongly in parameter-efficient training and that parameter-efficient scaling scales with model and dataset size. Where paired-image text data is scarce but strong multilingual language models exist (e.g. low resource languages), parameter-efficient training is even preferable to full-model training. Given a fixed compute budget, parameter-efficient training allows training larger models on the same hardware, achieving equivalent performance in less time. Parameter-efficient training hence constitutes an energyefficient and effective training strategy for contrastive vision-language models that may be preferable to the full-model training paradigm for common use cases. Code and weights at https://github.com/codezakh/LilT.

1. INTRODUCTION

Advances in transfer learning within the field of natural language processing (Houlsby et al., 2019b; Ben Zaken et al., 2022) have shown that when adapting to a novel task, updates to a small percentage of neurons (< 1%) in large, pretrained transformer-based language models can achieve nearly equivalent results to finetuning the entire model. Sung et al. (2021) showed that given the existence of already-aligned visual representations (e.g. CLIP's visual encoder) only a small number (4%) of parameters in a pretrained language model need to be updated for the language model to complete tasks such as visual question answering using the already-aligned visual representations. However, the creation of aligned vision and language representations typically involves updating all the parameters of a language model and a vision model, often randomly initialized (Radford et al., 2021) . Zhai et al. (2021) find that if the weights of a pretrained vision model are used as an initialization, only the neurons of the language model need to be updated to align the visual and language representations and match or exceed the performance of full-model training, resulting in a 50% reduction in trainable parameters. We take this line of investigation to its natural conclusion, asking -given that strong, pretrained vision and language models both exist, can we minimally update both of their parameters to align their representations? Answering this question is valuable for two reasons. From a practical perspective, contrastive vision-language alignment constitutes a form of large-scale pretraining and hence a heavy energy 1

