

Abstract

We propose a new kind of embedding for natural language text that deeply represents semantic meaning. Standard text embeddings use the outputs from hidden layers of a pretrained language model. In our method, we let a language model learn from the text and then literally pick its brain, taking the actual weights of the model's neurons to generate a vector. We call this representation of the text a neural embedding. We confirm the ability of this representation to reflect semantics of the text by an analysis of its behavior on several datasets, and by a comparison of neural embedding with state of the art sentence embeddings.

1. INTRODUCTION

Capturing the semantic meaning of text as a vector is a fundamental challenge for natural language processing (NLP) and an area of active research (Giorgi et al., 2021; Zhang et al., 2020; Gao et al., 2021; Huang et al., 2021; Yan et al., 2021; Zhang et al., 2021; Muennighoff, 2022; Alexander Liu, 2022; Chuang et al., 2022) . Recent work has focused on fine-tuning pretrained language models with contrastive learning, either supervised (e. Motivated by the need for deeper semantic representations of text, we propose a different kind of embedding. We allow a language model to fine-tune on a text input, and then measure the resulting changes to the model's own neuronal weights as a neural embedding. We demonstrate that neural embeddings do indeed represent the semantic differences between samples of text. We evaluate neural embeddings on several datasets and compare them with several state of the art sentence embeddings. We observe that neural embeddings correlate better specifically with semantics, while being comparable in other evaluations. We find that neural embeddings behave differently from the known embeddings we considered. Our contribution: 1. We propose a new kind of text representation: neural embeddingsfoot_0 (Section 2). 2. We evaluate embeddings by using several datasets and several criteria (Section 3). We show that by these criteria the neural embeddings are (1) better correlated with semantic similarity and consistency, and (2) strongly differ by the errors they do and by how they represent the qualities of the text.

2. NEURAL EMBEDDING METHOD

The technique for generating neural embeddings is using micro-tuning, first introduced for the BLANC-tune method of document summary quality evaluationfoot_1 Vasilyev et al. (2020) . It is a tuning on one sample only, and the tuned model is used for the sample only. Tuning a pretrained model on a specific narrow domain is a common practice to improve performance. Micro-tuning takes this to extreme, narrowing down the 'domain' to a 'dataset' consisting of just one sample. For each text sample, we start with the original language model and fine-tune only a few selected layers L 0 , L 1 , ..., L m while keeping all other layers frozen. Once the fine-tuning on the text sample is complete, we measure the difference between the new weights W ′ j and the original weights W j



github url will be provided here. The code is in supplementary material. https://github.com/PrimerAI/blanc 1



g. Reimers & Gurevych (2019); Zhang et al. (2021); Yan et al. (2021)) or unsupervised (e.g. Giorgi et al. (2021); Gao et al. (2021)). The embedding is generated by pooling the outputs of certain layers of the model as it processes a text.

