LOGIC PRE-TRAINING OF LANGUAGE MODELS

Abstract

Pre-trained language models (PrLMs) have been shown useful for enhancing a broad range of natural language understanding (NLU) tasks. However, the capacity for capturing logic relations in challenging NLU still remains a bottleneck even for state-of-the-art PrLM enhancement, which greatly stalls their reasoning abilities. To bridge the gap, we propose logic pre-training of language models to equip PrLMs with logical reasoning ability. To let logic pre-training perform on a clear, accurate, and generalized knowledge basis, we introduce fact instead of the plain language unit in previous PrLMs. The fact is extracted through syntactic parsing in avoidance of unnecessary complex knowledge injection. Meanwhile, it enables training logic-aware models to be conducted on a more general language text. To explicitly guide the PrLM to capture logic relations, three complementary selfsupervised pre-training objectives are introduced: 1) logical structure completion to accurately capture fact-level logic from the original context, 2) logical path prediction on a logical graph to uncover global logic relationships among facts, 3) logical connectives masking to capture discourse-level for fact groups. We evaluate our model on a broad range of NLP tasks, including natural language inference, relation extraction, and machine reading comprehension with logical reasoning. Experimental results show that our model achieves significant performance in all the downstream tasks, especially in logical reasoning related tasks.

1. INTRODUCTION

Machine reasoning in natural language understanding (NLU) aims to teach machines to understand human languages by building and analyzing the connections between the facts, events, and observations using logical analysis techniques like deduction and induction, which is one of the ultimate goals towards human-parity intelligence. Although pre-trained language models (PrLMs), such as BERT (Devlin et al., 2018) , GPT (Radford et al., 2018) , XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019) , have established state-of-the-art performance on various aspects in NLU, they are still short in complex language understanding tasks that involve reasoning (Helwe et al., 2021) . The major reason behind this is that they are insufficiently capable of capturing logic relations such as negation (Kassner & Schütze, 2019), factual knowledge (Poerner et al., 2019) , events (Rogers et al., 2020) , and so on. Many previous studies (Sun et al., 2021; Xiong et al., 2019; Wang et al., 2020) are then motivated to inject knowledge into pre-trained models like BERT and RoBERTa. However, they too much rely on massive external knowledge sources and ignore that language itself is a natural knowledge carrier as the basis of acquiring logic reasoning ability (Ouyang et al., 2021) . Taking the context in Figure 1 as an example, previous approaches tend to focus on entities such as the definition of "government" and the concepts related to it like "governor", but overlook the exact relations inherent in this example, thus failing to model the complex reasoning process. Given the fact that PrLMs are the key supporting components in natural language understanding, in this work, we propose a fundamental solution by empowering the PrLMs with the capacity of capturing logic relations, which is necessary for logical reasoning. However, logical reasoning can only be implemented on the basis of clear, accurate, and generalized knowledge. Therefore, we leverage fact as the conceptual knowledge unit to serve the basis for logic relation extraction. Fact is organized as a triple, i.e., in the form of predicate-argument structures, to represent the meaning such as "who-did-what-to-whom" and "who-is-what". Compared with existing studies that inject complex knowledge like knowledge graphs, the knowledge structure based on fact is far less complicated and more general in representing events and relations in languages. On top of the fact-based knowledge structure, we present PROPHET, a logic-aware pre-trained language model to learn the logical relations in a universal way from very large corpus. Given syntactically extracted fact, instead of simply syntactic-level structure of text (Xu et al., 2020) , we leverage multi-dimensional logical information for better representations. In detail, we introduce three novel pre-training objectives based on the newly introduced knowledge structure basis fact: 1) logical connectives masking for learning sentence-level logic connection. 2) logical structure completion task on top of facts for regularization, aligning extracted fact with the original context. 3) logical path prediction to capture the logic relationship between facts. PROPHET is based on Transformer (Vaswani et al., 2017) neural architecture. We pre-train PROPHET on the traditional English Wikipedia corpus, which includes 1.1M articles in total. We evaluate the model on a broad range of language understanding tasks: natural language inference, semantic similarity, machine reading comprehension, etc. Experimental results show that the fact is useful as the carrier for knowledge modeling, and the newly introduced pre-training tasks can improve PROPHET and achieves significant performance on downstream tasks. 1In summary, the contributions of this paper are: (1) PROPHET takes the initial attempt to pretrain language models that leverages the conceptual knowledge unit fact to equip PrLMs with logical reasoning ability. (2) We introduce three novel pre-training tasks for learning logic-aware representations from texts and facts. (3) PROPHET provides significant improvement on various downstream tasks, including logical reasoning of texts, general natural language understanding, and document-level relation extraction.

2.1. PRE-TRAINED LANGUAGE MODELS IN NLP

Large pre-trained language models (Devlin et al., 2018; Liu et al., 2019; Radford et al., 2018) have brought dramatic empirical improvements on almost every NLP task in the past few years. A classical norm of pre-training is to train neural models on a large corpus with self-supervised pre-training objectives. "Self-supervised" means that the supervision provided in the training process is automatically generated from the raw text instead of manually generation. Designing effective criteria for language modeling is one of the major topics in training pre-trained models, which decides how the model captures the knowledge from large-scale unlabeled data. The most popular pre-training objective used today is masked language modeling (MLM), initially used in BERT (Devlin et al., 2018) , which randomly masks out tokens, and the model is asked to uncover it given surrounding context. Recent studies have investigated diverse variants of denoising strategies (Raffel et al., 2020; Lewis et al., 2020 ), model architecture (Yang et al., 2019) , and auxiliary objectives (Lan et al., 2019; Joshi et al., 2020) to improve the model strength during pre-training. Although the existing techniques have shown effectiveness in capturing syntactic and semantic information after large-scale pre-training, they perform sensitivity to role reversal and struggles with pragmatic inference and role-based event knowledge (Rogers et al., 2020) , which are critical to the ultimate goal of complex reasoning that requires to uncover logical structures. However, it is difficult for pre-trained language models to capture the logical structure inherent in the texts since logical supervision is rarely available during pre-training. Therefore, we are motivated to explicitly guide the model to capture such clues via our newly introduced self-supervised tasks. There is also a lot of work in the research line of enhancing general reasoning abilities in pre-trained language models. The existing approaches mainly design novel pre-training objectives and leverage abundant knowledge sources such as WordNet (Miller, 1995 ). Notably, ERNIE 3.0 (Sun et al., 2021) 



Our codes have been uploaded as supplemental material, which will be open after the double review period.



REASONING ABILITY FOR PRE-TRAINED LANGUAGE MODELS Previous works have extensively investigated the possibility of injecting specific type of reasoning via pre-training. For reasoning types, Pi et al. (2022); Geva et al. (2020) focuses on numerical reasoning; Staliūnaitė et al. (2021); Tamborrino et al. (2020) target at commonse reasoning; Zhong et al. (2022) deals with multi-hop reasoning. Different from them, PROPHET concentrates on logical reasoning, which plays a fundamental role in general reasoning via natural language.

