ON PRE-TRAINED LANGUAGE MODELS FOR ANTIBODY

Abstract

Antibodies are vital proteins offering robust protection for the human body from pathogens. The development of general protein and antibody-specific pre-trained language models both facilitate antibody prediction tasks. However, there have been limited studies that comprehensively explore the representation capability of distinct pre-trained language models on different antibody tasks. To investigate the problem, we aim to answer several key questions in this paper, such as how pre-trained language models perform in antibody tasks with different specificity and how introducing specific biological mechanisms to the pre-training process can benefit the model. Additionally, we evaluate if the learned antibody pre-trained representations can be applied to real-world antibody problems, like drug discovery and immune process understanding. Previously, no benchmark available largely hindered the study to answer these questions. To aid in our investigation, we provide an AnTibody Understanding Evaluation (ATUE) benchmark. We comprehensively evaluate the performance of protein pre-trained language models by empirical study along with conclusions and new insights. Our ATUE and code are released at https://github.com/dqwang122/EATLM.

1. INTRODUCTION

Antibodies are a type of protein that is useful for diagnosing and treating a variety of diseases, including SARS- CoV-2 (Zhu et al., 2022) . It is crucial to understand the information contained in antibody sequences to develop effective therapeutic antibodies and advance our understanding of the immune system (Greiff et al., 2020; Lu et al., 2018; Yermanos et al., 2018) . Recent advances in general Pre-trained Protein Language Models (PPLM) and specific Pre-trained Antibody Language Models (PALM) offer new possibilities for antibody-related tasks. For example, PPLMs have shown promising results in transferring learned representations to antibody tasks (Kim et al., 2021; Zaslavsky et al., 2022) and PALMs have been found to improve model performance in antibody paratope predictions (Leem et al., 2022) . Figure 1 : Performance of pre-trained language models on tasks with different antibody specificity. Despite these successes, few studies have thoroughly examined the capability of different pre-trained language models (e.g. general PPLMs and specific PALMs) on various antibody tasks, which hinders the development of better architectures for antibody discovery and modification. To investigate this problem, we compared the performance of the pre-trained protein language model ESM (Rives et al., 2021) , the pre-trained antibody language model AntiBERT (Leem et al., 2021) , a pre-trained antibody language model EATLM by introducing antibody specific mechanisms, and a model trained from scratch (No Pretrain) on three antibody tasks with varying levels of specificity. The result is illustrated in Figure 1 . Here,

