ON PRE-TRAINED LANGUAGE MODELS FOR ANTIBODY

Abstract

Antibodies are vital proteins offering robust protection for the human body from pathogens. The development of general protein and antibody-specific pre-trained language models both facilitate antibody prediction tasks. However, there have been limited studies that comprehensively explore the representation capability of distinct pre-trained language models on different antibody tasks. To investigate the problem, we aim to answer several key questions in this paper, such as how pre-trained language models perform in antibody tasks with different specificity and how introducing specific biological mechanisms to the pre-training process can benefit the model. Additionally, we evaluate if the learned antibody pre-trained representations can be applied to real-world antibody problems, like drug discovery and immune process understanding. Previously, no benchmark available largely hindered the study to answer these questions. To aid in our investigation, we provide an AnTibody Understanding Evaluation (ATUE) benchmark. We comprehensively evaluate the performance of protein pre-trained language models by empirical study along with conclusions and new insights. Our ATUE and code are released at https://github.com/dqwang122/EATLM.

1. INTRODUCTION

Antibodies are a type of protein that is useful for diagnosing and treating a variety of diseases, including SARS- CoV-2 (Zhu et al., 2022) . It is crucial to understand the information contained in antibody sequences to develop effective therapeutic antibodies and advance our understanding of the immune system (Greiff et al., 2020; Lu et al., 2018; Yermanos et al., 2018) . Recent advances in general Pre-trained Protein Language Models (PPLM) and specific Pre-trained Antibody Language Models (PALM) offer new possibilities for antibody-related tasks. For example, PPLMs have shown promising results in transferring learned representations to antibody tasks (Kim et al., 2021; Zaslavsky et al., 2022) and PALMs have been found to improve model performance in antibody paratope predictions (Leem et al., 2022) . Figure 1 : Performance of pre-trained language models on tasks with different antibody specificity. Despite these successes, few studies have thoroughly examined the capability of different pre-trained language models (e.g. general PPLMs and specific PALMs) on various antibody tasks, which hinders the development of better architectures for antibody discovery and modification. To investigate this problem, we compared the performance of the pre-trained protein language model ESM (Rives et al., 2021) , the pre-trained antibody language model AntiBERT (Leem et al., 2021) , a pre-trained antibody language model EATLM by introducing antibody specific mechanisms, and a model trained from scratch (No Pretrain) on three antibody tasks with varying levels of specificity. The result is illustrated in Figure 1 . Here, specificity refers to the antibody's unique evolution processes distinct from that of protein to obtain functionality, such as the ability to bind antigen (The definition is discussed in detail in §3.1). We can see that while ESM performs well in tasks that are less antibody specific, its performance decreases significantly in tasks that are more specific. Additionally, AntiBERT does not demonstrate a clear advantage over the non-pre-trained model in the high-specificity task. These results highlight the limitations of current pre-training language models for antibody-related studies. Using general PPLM representations directly may harm performance, and current pre-training strategies for PALMs may not fit the specific biological functions of antibodies. This emphasizes the need for a comprehensive model design guideline for various antibody tasks. Our main focus is to address the following questions: (I) How well will pre-trained language models perform on antibody tasks with varying specificity? Addressing of the question is mainly hindered by two challenges: the lack of a reliable antibodyspecific benchmark for performance evaluation and comprehensive studies of current PPLMs and PALMs. (II) Can incorporating biological mechanisms, specifically antibody-specific evolution, into the pre-training process provide additional benefits for antibody representation learning? This idea has been explored in several computational biology studies, which have demonstrated promising results in antibody-related tasks such as disease diagnosis and therapeutic antibody development (Yermanos et al., 2018; Miho et al., 2019) . Then, it is interesting to know whether antibody representation learning can benefit from the incorporation of antibody-specific evolution information. (III) Are the pre-trained antibody representations useful in practical applications, such as drug discovery and immune process understanding? Antibodies are critical in drug development, and it is essential to determine whether pre-training representations can be beneficial for biologists to comprehend antibody functions or develop drugs. To investigate these questions, we first propose antibody study benchmark AnTibody Understanding Evaluation (ATUE). This is the first antibody benchmark with four real-world supervised tasks related to therapeutic antibody engineering, B cell analysis, and antibody discovery. These tasks cover a range of specificity levels to evaluate models on different aspects of antibody biological functions. Based on ATUE, we conduct empirical studies to investigate the representation ability of distinct pre-trained language models. To explore the impact of incorporating specific biological mechanisms in antibody pre-training, two objectives are introduced to tailor masked language modeling for evolution: (1) Ancestor germline prediction guides the model to discriminate the evolutionary relationship between antibody and ancestral sequences. (2) Mutation position prediction mimics hypermutation during the evolution. These methods are used to investigate the representation ability of antibody evolution-tailored language model. Finally, we take a close look at the SARS-CoV-2 antibody discovery to investigate the pre-trained representation under a real-world scenario. We have three main contributions in this study: • We created the first comprehensive antibody benchmark called ATUE to help with antibody application studies, which includes four real-world supervised tasks ranging from low to high specificity. We also introduce two new objectives for antibody pretraining that incorporate antibody-specific evolutionary information. • We made key observations for providing guidelines for better antibody representation. Firstly, PPLMs perform well on antibody tasks that have a high relationship with structure, but they perform poorly on tasks with high antibody specificity. Secondly, in most cases, PALMs perform as well as or even better than PPLMs with less pre-training data. Thirdly, PALMs can be improved by incorporating the evolution process, but the evolution information from MSAs does not always benefit antibody tasks. • We identified 11 potential SARS-CoV-2 binders that have highly identical sequences to existing therapeutic antibodies that bind to the virus, which could accelerate real-world antibody discovery.

2. RELATED WORK

Our work focuses on researching the effectiveness of protein and pre-trained antibody language models for antibody-specific tasks. Below we review the representative existing methods. We list the details in Table 1 .

