LEARNING CONTEXTUALIZED KNOWLEDGE STRUC-TURES FOR COMMONSENSE REASONING

Abstract

Recently, neural-symbolic architectures have achieved success on commonsense reasoning through effectively encoding relational structures retrieved from external knowledge graphs (KGs) and obtained state-of-the-art results in tasks such as (commonsense) question answering and natural language inference. However, these methods rely on quality and contextualized knowledge structures (i.e., fact triples) that are retrieved at the pre-processing stage but overlook challenges caused by incompleteness of a KG, limited expressivity of its relations, and retrieved facts irrelevant to the reasoning context. In this paper, we present a novel neural-symbolic model, named Hybrid Graph Network (HGN), which jointly generates feature representations for new triples (as a complement to existing edges in the KG), determines the relevance of the triples to the reasoning context, and learns graph module parameters for encoding the relational information. Our model learns a compact graph structure (comprising both extracted and generated edges) through filtering edges that are unhelpful to the reasoning process. We show marked improvement on three commonsense reasoning benchmarks and demonstrate the superiority of the learned graph structures with user studies. 1

1. INTRODUCTION

Commonsense knowledge is essential for developing human-level artificial intelligence systems that can understand and interact with the real world. However, commonsense knowledge is assumed by humans and thus rarely written down in text corpora for machines to learn from and make inferences with. Fig. 1 shows an example in a popular commonsense reasoning benchmark named Common-senseQA (Talmor et al., 2019) . The knowledge about the relations between concepts, e.g., the fact triple (print, Requires, use paper), is not explicitly given in the question and answer. Without important background knowledge as clues, natural language understanding (NLU) models may fail to answer such simple commonsense questions that are trivial to humans. Current commonsense reasoning models can be classified into retrieval-augmented methods (Banerjee et al., 2019; Pan et al., 2019) and KG-augmented methods (Wang et al., 2019b; Kapanipathi et al., 2020) . Retrieval-augmented methods retrieve relevant sentences from an external corpus such as Wikipedia. The retrieved sentences are usually not interconnected, and their unstructured nature makes it inherently difficult for models to do complex reasoning over them (Zhang et al., 2018) . On the other hand, symbolic commonsense KGs such as ConceptNet (Speer et al., 2017) provide structured representation of the relational knowledge between concepts, which is of critical importance for effective (multi-hop) reasoning and making interpretable predictions. Therefore, recent advances (Lin et al., 2019; Feng et al., 2020; Malaviya et al., 2020; Bosselut & Choi, 2019) have focused on KG-augmented neural-symbolic commonsense reasoning -integrating the symbolic commonsense knowledge with the pre-trained neural language models such as BERT (Devlin et al., 2019) . One of the key challenges for KG-augmented commonsense reasoning is how to obtain relevant and useful facts for the model to reason over. These supporting facts are usually not readily available to the model and require explicit annotation by humans. Most existing works (Lin et al., 2019; Wang et al., 2019b; Lv et al., 2020) follow heuristic procedures to extract supporting fact triples from KGs, e.g., by finding connections between concepts mentioned in the question and answer. This simplified extraction process may be sub-optimal because the commonsense KGs are usually incomplete (Min 1 Code has been uploaded and will be published. 1

